KR102505725B1 - Thermal image system and method of processing the image - Google Patents

Abstract

The present invention is a thermal image detector; a detector interface unit converting analog image data detected by the thermal image detector into digital image data; a main control unit for correcting and processing digital image data received from the detector interface unit; and a power unit generating power and supplying power to the detector interface unit and the main control unit.

Description

Thermal image system and method of processing the image

The present invention relates to a thermal imaging device, and more particularly, to a thermal imaging device and an image processing method capable of supplementing the disadvantages of a conventional FPGA-based thermal imaging device and applying the advantages of a commercial processor.

delete

A thermal image system is a device that images infrared rays emitted from an object. Since such a thermal imaging device can identify objects that cannot be identified with a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) based imaging device, it has been used in various military fields. Recently, as the demand for thermal imaging devices increases, the price of the thermal imaging sensor, which is a core component, is lowered and performance is improved, so that its utilization is increasing not only in the military field but also in the civilian field. Thermal imaging devices in the civilian sector are mainly used for vehicle night vision, autonomous driving, and security systems. Currently, the scope of application of thermal imagers in the civilian sector is used in high-end vehicles and special environments, but as the market for night vision in vehicles is expected to increase steadily, related demand is expected to increase explosively.

On the other hand, the infrared detector can be divided into a cooling detector having a cooler outside and a non-cooling detector equipped with a thermoelectric element instead of the cooler function. Since uncooled detectors do not have an external cooler, they have various advantages such as cost, size, and power consumption. However, since there is no cooling function, additional work is required for video signal processing. Accordingly, until now, uncooled infrared detectors have been developed as field programmable gate array (FPGA)-based thermal imaging devices for image signal processing.

Currently, most thermal imagers are designed and built using FPGA-based processors. The FPGA-based thermal imager can configure a system optimized for requirements, and has the advantage of being able to optimize various algorithms according to the desired system configuration.

An internal structure diagram of such an FPGA-based thermal imaging device is shown in FIG. 1 . As shown in FIG. 1 , an FPGA-based thermal imaging device may include a multi-processor core, programmable logic, an internal data bus, and a memory. These FPGA-based thermal imaging devices modularize algorithms such as NUC, TEC-less, CEM, and DPC for thermal image correction in programmable logic, and are equipped with a real-time operation system (Real-Time OS; RTOS), video input and output drivers (Drivers) are composed.

In the case of the current video system, various video input standards (MIPI-CSI-2, CPI, Analog) and video output standards (HDMI, Camera Link, BT1120, BT656, NTSC/PAL, USB) and image processing algorithms to improve image quality are used. demand a variety of However, in the case of an FPGA-based system, it is difficult to reflect various requirements, and if a chipset is changed according to requirements, the existing system cannot be used and additional development is required. Therefore, the recycling rate of system modules is low or impossible, resulting in unnecessary development costs and time.

delete

Korean Patent Registration No. 10-2093917

The present invention provides a thermal imaging device and an image processing method capable of supplementing the disadvantages of existing FPGA-based thermal imaging devices and applying the advantages of a commercial processor.

The present invention provides a thermal imaging device based on a commercial processor capable of supporting various standard image input/output interfaces and thus being utilized in various imaging devices and devices, and a processing method thereof.

delete

A thermal imaging device according to an aspect of the present invention includes a thermal image detector; a detector interface unit converting analog image data detected by the thermal image detector into digital image data; a main control unit for correcting and processing digital image data received from the detector interface unit; and a power supply unit generating power and supplying power to the detector interface unit and the main control unit.

The thermal image detector includes an uncooled detector.

The detector interface unit, the main control unit, and the power supply unit are implemented on the same substrate or on different substrates and are connected to each other through wires.

The detector interface unit may include: a first detector controller that receives a control signal of the thermal image detector from the main controller and controls the thermal image detector; and an A/D converter that receives analog image data from the thermal image detector and converts it into digital image data. and an interface unit controlling a connection between the thermal image detector and the main control unit.

The main control unit includes a data correction unit for correcting the image data received from the detector interface unit, an image processing unit for inputting the corrected image data and processing an image, and a second device for controlling the thermal image detector through the detector interface unit. It includes a detector control unit, an interface unit for controlling connection with the detector interface unit, and an image output unit for outputting a corrected and processed image.

The data compensating unit removes fixed pattern noise of the thermal image sensor by correcting an output value according to a temperature change of the thermal image detector and correcting non-uniform values of each pixel of the thermal image.

The image processing unit improves the contrast ratio of the image corrected by the data correction unit, and corrects it using normal pixels around the dead pixel.

The power supply unit includes a power generation unit for generating driving power for the detector interface unit and the main control unit, an image conversion unit for converting image data transmitted from the main control unit, and the detector interface unit, the main control unit, and an external interface. It includes an interface unit that controls the connection.

The video conversion unit converts the format of the video transmitted from the main control unit in order to deliver it to the outside.

delete

A thermal imaging device according to another aspect of the present invention includes a thermal image detector; a detector interface unit converting analog image data detected by the thermal image detector into digital image data; and a main control unit for receiving, correcting, and processing digital image data from the detector interface unit, wherein the main control unit is connected to and controls an image input unit for inputting image data from the detector interface unit and the detector interface unit. A communication unit for transmitting and receiving signals, a processing unit for controlling the thermal image detector through the detector interface unit and receiving, correcting and processing digital image data from the detector interface unit, and an image output unit for outputting the processed image to the outside; , and a memory unit for storing a plurality of data including image data and storing an image processing sequence.

The video input unit includes at least one of MIPI CSI-2, CPI, BT.1120, and BT.656, the communication unit includes at least one of UART, I2C, SPI, and 1Gbit Ethernet, and the video output unit includes Camera Link, It includes at least one of HDMI, NTSC/PAL, and LCD.

The processing unit may include a first ARM core for controlling and managing the main control unit, a second ARM core for controlling a thermal image detector and external connection, correcting an output value according to a temperature change of the thermal image detector, and compensating for each thermal image. A first DSP core that removes fixed pattern noise of a thermal image sensor by correcting non-uniform values of pixels, and a contrast ratio of an image corrected by the first DSP core is improved and corrected using normal pixels around dead pixels. It includes a second DSP core that does.

delete

An image processing method of a thermal imaging device according to another aspect of the present invention includes converting analog image data into digital image data; correcting an output value according to a temperature change of image data; removing fixed pattern noise from image data; The process of converting to a standard image protocol for using a commercial image processor; Pseudo color conversion process for black and white images; and converting RGB data to YUV to compress an image.

correcting dead pixels after removing the fixed pattern noise; removing noise from image data; and converting into RGB Bayer pattern data.

an image resize process for outputting an image size according to a user setting after converting the RGB data to YUV; OSD insertion process for outputting text on images according to user settings; and converting to a standard video output protocol.

delete

The present invention is manufactured based on a commercial image processor, compensates for the disadvantages of existing FPGA-based thermal imaging devices, and can be used in imaging devices and various devices by supporting various standard image input and output interfaces.

In addition, the core image processing algorithm is modularized to operate in each core (DSP), and the algorithm required for image processing is changed to a standard image format so that it can be performed in a commercial image processor, so that the existing CCD/CMOS-based image processing It may be possible to perform on the device. Therefore, it is possible to increase the development period and cost reduction through system recycling in the future.

delete

1 is an internal structure diagram of a conventional FPGA-based thermal imaging device;
2 is a block diagram for explaining the configuration of a thermal imaging device according to an embodiment of the present invention;
3 is a schematic diagram illustrating a configuration of a main controller of a thermal imaging device according to an embodiment of the present invention;
4 is a block diagram for explaining the structure of a main processor constituting a main control unit of a thermal imaging apparatus according to an embodiment of the present invention;
5 is a block diagram for explaining the relationship between a main processor and its peripheral devices according to an embodiment of the present invention;
6 to 8 are block diagrams for explaining the main functions of each core constituting the main processor.
9 is a flowchart illustrating a data processing method for image processing of a thermal imaging device according to an embodiment of the present invention.
10 is a photograph in which dead pixels generated in a thermal image are processed;
11 is a comparison photograph before and after applying edge enhancement.
12 is a picture to which digital zoom and PIP functions are applied.
13 is a graph showing processor usage for each function;
14 is a graph comparing power consumption of a conventional FPGA-based thermal imaging device and a thermal imaging device according to the present invention.
15 is a picture of a thermal imaging device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and the scope of the invention to those skilled in the art. It is provided for complete information. In order to clearly express the various layers and each region in the drawing, the thickness is enlarged and expressed, and the same reference number in the drawing refers to the same element.

2 is a block diagram for explaining the configuration of a thermal imaging device according to an embodiment of the present invention.

Referring to FIG. 2 , a thermal imaging apparatus according to an embodiment of the present invention does not use a conventional FPGA-based processor but uses a commercial multi-core processor. The thermal imaging device of the present invention is largely composed of three functional units, and may include a detector interface unit 100, a main control unit 200, and a power supply unit 300. In addition, the detector interface unit 100, the main control unit 200, and the power supply unit 300 may be implemented on a predetermined substrate, and the substrate includes a printed circuit board (PCB) on which a predetermined circuit is printed. can do. That is, the detector interface unit 100, the main control unit 200, and the power supply unit 300 may be implemented on a PCB. In this case, the detector interface unit 100, the main control unit 200, and the power supply unit 300 may be implemented on one PCB or on different PCBs. For example, the detector interface unit 100, the main control unit 200, and the power supply unit 300 may be implemented on first, second, and third PCBs, respectively, and these first to third PCBs have predetermined wires. can be connected to each other through

The detector interface unit 100 converts analog image data detected by the thermal sensor 10 into digital image data and transmits the converted digital image data to the main controller 200 . Also, the detector interface unit 100 controls the thermal image detector 10 and controls connection with the thermal image detector 10 . To this end, the detector interface unit 100 includes a first detector controller 110 that controls the thermal image detector 10 and an A/D converter 120 that receives analog image data from the thermal image detector 10 and converts it into digital image data. ) and an interface unit 130 connected to the thermal image detector 10 and connected to the main controller 200 to transmit the digital image data converted by the A/D converter 310 to the main controller 200. can do. The interface unit 130 may be connected to the thermal image detector 10 and the main controller 200 to communicate with them. That is, the interface unit 130 includes an image input unit that is connected to the thermal image detector 10 and inputs an image from the thermal image detector 10, an image output unit that outputs digital image data to the main controller 200, and a thermal image detector ( 10) may be provided with a communication unit that outputs a control signal for controlling. As described above, the detector interface unit 100 is connected to the thermal image detector 10 to control the thermal image detector 10, receives analog image data captured by the thermal image detector 10, and converts it into digital image data. It is transmitted to the main controller 200.

The main control unit 200 corrects digital image data, that is, raw data, received from the detector interface unit 100, and processes the corrected image data. Also, the main control unit 200 controls the thermal image detector 10 through the detector interface unit 100 . To this end, the main control unit 200 includes a raw data correction unit 210 that corrects raw data received from the detector interface unit 100, an image processing unit 220 that processes the corrected image, and a detector interface unit. A second detector controller 230 for controlling the thermal image detector 10 through 100 may be included. In addition, the main control unit 200 is an interface unit ( 240) and an image output unit 250 for outputting an image. Also, the interface unit 240 may be connected to the power supply unit 300 . That is, the interface unit 240 may be connected to the detector interface unit 100 and the power supply unit 300 to communicate with them. The interface unit 240 is connected to the detector interface unit 100 and transmits an image input unit for inputting image raw data from the detector interface unit 100 and a control signal for controlling the thermal image detector 10 to the detector interface unit 100. ) may be provided with a communication unit for outputting. As described above, the main control unit 200 is connected to the detector interface unit 100, receives raw data from the detector interface unit 100, corrects it, processes the corrected image data, and outputs it to the outside, and the detector interface unit ( The thermal image detector 10 is controlled through 100). That is, in order to control the thermal image detector 10, the second detector controller 230 of the main controller 200 generates a control signal and transmits the control signal to the first detector controller 110 of the detector interface 100, The detector controller 110 controls the thermal image detector 10 according to a control signal from the main controller 200 . On the other hand, the raw data corrector 210 corrects the TEC (Thermal Electric Cooler)-less for correcting the output value according to the temperature change of the uncooled thermal image sensor and the non-uniform value of each pixel of the thermal image to correct the thermal image It can perform the function of NUC (Non-Uniformity Correction) to remove the fixed pattern noise of the sensor. In addition, the image processing unit 220 may perform functions of Contrast Enhancement Mapping (CEM) for improving the contrast ratio after NUC and TEC-less and Defect Pixel Correction (DPC) for correcting using normal pixels around dead pixels.

The power supply unit 300 generates power consumed by the detector interface unit 100 and the main control unit 200 and protects the power. That is, the power supply unit 300 generates power consumed by the detector interface unit 100 and the main control unit 200 . Also, the power supply unit 300 may convert image data transmitted from the main control unit 200 and transmit the converted data to the outside. To this end, the power supply unit 300 includes a power generation unit 310 for generating driving power for the detector interface unit 100 and the main control unit 200, a power protection unit 320 for protecting the generated power, and a main control unit ( 200) may include an image conversion unit 330 that converts image data transmitted, a detector interface unit 100, a main controller 200, and an interface unit 340 connected to an external interface. That is, the power supply unit 300 generates at least one predetermined power source and transmits it to the detector interface unit 100 and the main control unit 200, converts image data transmitted from the main control unit 200, and transmits it to the outside. Here, the video data conversion of the power supply unit 300 is a format conversion for transferring the video transmitted from the main control unit 200 to the outside. That is, the video data format-converted by the power supply unit 300 is transferred to the outside through the interface unit 340, and the power supply unit 300 converts the video data into several video output methods according to the output requirements of the interface unit 340.

delete

As described above, in the thermal imaging device according to an embodiment of the present invention, the detector interface unit 100 receives analog image data photographed by the thermal image detector 10 and converts it into digital image data, and then the main control unit 200 , and the main controller 200 corrects and processes the image raw data received from the detector interface unit 100. In addition, the power supply unit 300 converts the image processed by the main control unit 200 and outputs it to the outside. Meanwhile, the main control unit 200 generates a control signal for controlling the thermal image detector 10 and transmits the generated control signal to the detector interface unit 100, and the detector interface unit 100 controls the thermal image detector 10 according to the control signal. do.

delete

3 is a schematic diagram for explaining the configuration of a thermal imaging device according to an embodiment of the present invention, and is a schematic diagram for explaining the configuration of a main control unit.

Referring to FIG. 3 , the main control unit of the thermal imaging device according to an embodiment of the present invention includes an image input unit 241, a communication unit 242, an image output unit 250, a memory unit 561, 522; 560, and a processing unit. (ie, the main processor) (271, 272, 273, 274; 270). That is, the video input unit 241 and the communication unit 242 constitute the interface unit 240 with respect to the main control unit 200 of FIG. 2 . In addition, the main processor 270 may configure the raw data correction unit 210, the image processing unit 220, and the detector control unit 230. That is, the main processor 270 may perform the functions of the raw data corrector 210, the image processor 220, and the detector controller 230. The image input unit 241 may be provided to input image data through the detector interface unit 100, and may include MIPI CSI-2, CPI, BT.1120, BT.656, and the like. That is, the video input unit 241 can be driven by selecting at least one of MIPI CSI-2, CPI, BT.1120, and BT.656. The communication unit 242 may be provided to transmit and receive control signals to and from the detector interface unit 100 and the power supply unit 300, and may include UART, I2C, SPI, 1Gbit Ethernet, and the like. That is, the communication unit 242 may be driven by selecting at least one of UART, I2C, SPI, and 1Gbit Ethernet. The image output unit 250 may be provided to output the processed image to the outside, and may include Camera Link, HDMI, NTSC/PAL, LCD, and the like. That is, the video output unit 250 can be driven by selecting at least one of Camera Link, HDMI, NTSC/PAL, and LCD. The memory unit 260 stores data, image processing algorithms, and the like, and may include a DDR memory 261 and a NAND memory 262. Also, the processor 270 may include ARM cores 271 and 272 and DSP cores 273 and 274. Here, data correction is performed by the first DSP core 273, image processing is performed by the first ARM core 271 and the second DSP core 274, and the detector control and external control are performed by the second ARM core ( 272). That is, the processor 270 is composed of four cores, and the four cores perform functions of data correction, image processing, detector control, and external control, respectively.

The structure of the main controller of this thermal imaging device is designed to ensure versatility. Therefore, unlike the existing FPGA structure, the main controller interface can be designed as a standard video input/output interface so that various sensors and video output devices can be mounted. In addition, various inputs (MIPI CSI-2, CPI, BT.1120, BT.656) and outputs (HDMI, Camera Link, NTSC/PAL, LCD) are possible according to system requirements. That is, internal hardware can be configured to select input (MIPI CSI-2, CPI, BT.1120, BT.656) and output (HDMI, Camera Link, NTSC/PAL, LCD) inside a commercial processor. Therefore, one of the input/output formats can be selected by processor register setting and software setting.

delete

4 is a block diagram for explaining the structure of a main processor of a thermal imaging apparatus according to an embodiment of the present invention. That is, FIG. 4 is a block diagram functionally illustrating the main processor 270 of FIG. 3 constituting the main control unit of the thermal imaging apparatus according to an embodiment of the present invention. 3 and 4, the main processor 270 may include a first ARM core 271, a second ARM core 272, a first DSP core 273, and a second DSP core 274. can

The first ARM core 271 is in charge of real-time OS (RTOS) and image signal process (ISP) functions that control/manage the system. That is, the first ARM core 271 performs image processing. The first ARM core 571 includes a bootloader, a bootloader utility, a BIOS, a BIOS device driver, a network support driver, and a codec. , It may be configured including a user interface.

The second ARM core 272 is in charge of network and communication. Specifically, the second ARM core 272 performs detector control and external control. The second ARM core 272 includes a network driver, a BIOS support driver, a network device driver, an audio device driver, and a communication device driver. can be configured to include

The first and second DSP cores 273 and 274 are in charge of image processing for thermal images. That is, the first and second DSP cores 273 and 274 process the thermal image received from the detector interface unit 100, and one of the first and second DSP cores 273 and 274 performs data correction. , the other one performs image processing. Of course, the first and second DSP cores 273 and 274 may alternately perform correction and image processing on sequentially input image data. For example, the first DSP core 273 performs data correction and image processing on first image data, and the second DSP core 273 performs data correction on second image data input next to the first image data. Calibration and image processing can be performed. These first and second DSP cores 273 and 274 are TEC (Thermal Electric Cooler)-less for correcting the output value according to the temperature change of the uncooled thermal image sensor, and the non-uniform value of each pixel of the thermal image. NUC (Non-Uniformity Correction) to remove fixed pattern noise of the thermal image sensor by correction, CEM (Contrast Enhancement Mapping) to improve the contrast ratio after NUC and TEC-less, DPC to correct using normal pixels around dead pixels (Defect Pixel Correction), Vision Processing Library, and BIOS Support Driver.

The thermal image processing unit, that is, the first and second DSP cores 273 and 274, a core function of the current system, is modularized so that it can be used in other systems.

Meanwhile, NUC, TEC-less, and CEM performed by the DSP cores 273 and 274 are described in detail as follows.

In the multi-dimensional array thermal image sensor, that is, the thermal image detector 10, a fixed pattern appears due to a geometric difference between each pixel element or a difference in gain of a transmission and amplification stage, which is called Fixed Pattern Noise (FPN). This phenomenon appears more severely in infrared detectors than in CCDs that detect visible light bands, more severely in 2-dimensional detectors than in 1-dimensional detectors, and more severely in uncooled detectors than in cooled detectors. Due to the nature of ROIC (Read-Out Integrated Circuit), which processes a two-dimensional array of uncooled detectors in units of rows and columns, FPN occurs in horizontal and vertical directions, which is called image non-uniformity. The response characteristics of one pixel of the thermal image detector are not linear. Non-linear quadratic or higher-order equations are approximated with first-order equations to facilitate modeling and reduce the amount of computation. In order to obtain NUC, two reference inputs are required. For example, for HOT and COLD, data (Raw) is received by using a black body (black body), and G (gain) and O (offset) of Equation 1 below save Equation 1 below is an equation for obtaining a basic NUC.

TEC-less is an algorithm that compensates and maintains the change according to temperature at the output of an uncooled detector without TEC as a constant value. Since there is no constant temperature maintaining device, image output values for the same object are not constant, resulting in noise and fixed patterns. Therefore, after observing the value of each pixel of the image data at a constant temperature according to the temperature, the output value that changes per 1 ° C is stored and applied in real time according to each temperature. In general, there is a difference in the amount of change depending on the temperature in a large range, so it is used with NUC and applied using several tables. Equation 2 below is an equation obtained by adding a TEC operation method to Equation 1.

CEM (Contrast Enhancement Mapping) will be described as follows. In the output image after NUC and TEC-less image processing, the difference in image output values between objects is very small, so it is difficult to distinguish the difference. In a state in which image data values are concentrated only at a certain point, the image is output with a poor contrast ratio. Therefore, it is necessary to improve the contrast by evenly rearranging the distribution of image data values with slight differences, and for this purpose, Contrast Enhancement Mapping (CEM) is performed.

delete

5 is a block diagram for explaining the relationship between a main processor and its peripheral devices according to an embodiment of the present invention, and is a structural diagram of a thermal image processing module based on a commercial processor. That is, FIG. 5 is a block diagram showing the relationship between a main processor and its peripheral devices for processing thermal images. 6 to 8 are block diagrams for main functions of each core constituting the main processor. That is, FIGS. 6 to 8 are functional block diagrams of each core of the main processor for processing thermal images.

Referring to FIG. 5 , the structure of a commercial processor-based thermal image processing module according to the present invention is composed of a total of three thermal image processing units. That is, it may be composed of a memory unit 260, an addressing unit 280, and a main processor (273, 274, 280; 270).

The memory unit 260 may store image input data, lookup table data, CEM table data, primary image data, and secondary image data. For example, a plurality of input data and a plurality of lookup table data may be stored in a DDR memory.

The addressing unit 280 performs a data addressing function between the memory unit 260 and the main processor 270 . That is, the addressing unit 280 performs an addressing function of each core constituting the main processor 270 with a plurality of image data and a plurality of lookup table data stored in the memory unit 260 . The addressing unit 280 includes input line addressing for addressing image input data to the first DSP core 273, look-up table addressing for addressing look-up table data to the first DSP core 273, and CEM table data. 2 CEM table addressing for addressing the DSP core 274, output line addressing for addressing the image data of the second DSP core 274 to the primary image data of the memory unit 260, and EVE ( First image data addressing for addressing to the embedded vision engine 290, second image data addressing for storing image data of the EVE 290 as secondary image data in the memory unit 260, and memory unit 260 It may include third image data addressing to output secondary image data of through the display device 291 .

The first DSP core 273 performs thermal image processing by inputting image input data and lookup table data of the memory unit 260 through input line addressing and lookup table addressing. The first DSP core 273 performs NUC and TEC-less image processing, which are the most core functions of thermal image processing. To this end, the 1st DSP core 273, as shown in FIG. 6, uses an internal line buffer to prevent loss of input image frames, NUC & TEC-less where NUC & TEC-less operations are performed. It consists of a LUT (Look-Up Table) line for transferring gain and offset data required for block, NUC & TEC operation. In addition, the first DSP core 273 may include an output line buffer that transfers NUC and TEC-less image data to the second DSP core 274 .

The second DSP core 274 inputs CEM table data of the memory unit 260 through CEM table addressing and inputs image data subjected to NUC and TEC-less processing through the first DSP core 273 to obtain a thermal image. Improve the contrast ratio of the image. To this end, the second DSP core 274, as shown in FIG. 7, uses an internal line buffer to prevent loss of an input image frame and pixels for applying a histogram to an input frame. It consists of a pixel count table, a data count for collecting input image data, a histogram block for performing a histogram, and a relocate block for rearranging the contrast ratio. In addition, the second DSP core 274 may include an output line buffer that transfers image data having an improved contrast ratio of a thermal image to the memory unit 260 through output line addressing.

As shown in FIG. 8, the EVE (Embedded Vision Engine) 280 embedded in the first ARM core for additional image enhancement performs a Defect Pixel Correction (DPC) block for dead pixel processing and image noise improvement. It is composed of a noise filter block for thermal imaging and a color converter block for applying pseudo color to thermal images expressed in black and white.

Here, each core shares the memory unit 260 through Link API, and communication between each core is performed through IPC and SYS BIOS. In addition, all input/output hardware control and communication are performed through the BIOS driver of the main core, that is, the first ARM core 271.

delete

9 is a flowchart illustrating a data processing method for image processing of a thermal imaging device according to an embodiment of the present invention. That is, FIG. 9 is a flowchart illustrating an image processing method of a thermal imaging device.

Referring to FIG. 9 , the data processing sequence for image processing of the thermal imaging device includes a process of receiving raw image (RAW) data (S110), a process of image correction (S120), a process of removing noise (S130), and a process of converting a standard image protocol ( S140), dead pixel correction (DPC) process (S150), noise filtering process (S160), CFA interpolation process (S170), pseudo color conversion process (S180), YUV conversion process (S190), an image resize process (S200), an OSD insertion process (S210), and a video output protocol conversion process (S220).

S110 : The detector interface unit 100 converts the image data detected by the thermal image detector 10 into digital image data and then transfers the converted image data to the main controller 200 . In this case, the thermal image detector 10 may be an uncooled thermal image detector, and the raw data correction unit 210 of the main controller 200 receives the changed digital image data through the detector interface unit 100 .

S120: Perform thermal image TEC-less to correct the output value according to the temperature change of the uncooled thermal image sensor. TEC-less is implemented to compensate and maintain a constant value for the change according to temperature in the output of an uncooled detector without TEC. Since there is no constant temperature maintaining device, the image output value for the same object is not constant, resulting in noise and fixed patterns. Therefore, the value of each pixel of the image data at a constant temperature is observed according to the temperature, and the output value that changes per 1 ° C is stored and applied in real time according to each temperature. In general, there is a difference in the amount of change depending on the temperature in a large range, so it is used with NUC and applied using several tables.

S130: Perform thermal image NUC to remove fixed pattern noise of uncooled thermal image sensor. All multi-dimensional array thermal image sensors exhibit fixed pattern noise (FPN) due to geometrical differences between pixel elements or differences in gains of transmission and amplification stages. This phenomenon appears more severely in an infrared detector than a CCD detecting a visible light band, in a two-dimensional detector rather than a one-dimensional detector, and in an uncooled detector rather than a cooled detector. Due to the nature of the ROIC (Read-Out Integrated Circuit), which processes the two-dimensional array of uncooled detectors in rows and columns, FPN occurs in the horizontal and vertical directions, resulting in non-uniformity of the image. The response characteristic of one pixel of the detector is not linear. Non-linear quadratic or higher-order equations are approximated with first-order equations to facilitate modeling and reduce the amount of computation. To obtain NUC, two reference inputs are required. For example, for HOT and COLD, a black body is used to receive data (Raw) and obtain G (gain) and O (offset).

S140: Convert standard video protocol to use a commercial image processor. In the case of thermal image data, as the thermal energy of the image is converted into data and used, it is composed of basic image data that is not used in the existing image processor. Therefore, in order to use the image processing function of the image processor, it is necessary to convert the video protocol into an image protocol used by the existing image processor. It converts the thermal energy numerical data composed of 14 bits into a layer format (Bayer format) capable of RGB conversion.

S150: Perform DPC (Dead Pixel Correction) for pixels whose output is not normal due to the characteristics of the sensor. In the thermal image sensor, dead pixels occur in the active pixel line during the manufacturing process. If this dead pixel is not processed, it is output as a small dot when outputting an image, and it has a great effect on the image quality. Therefore, image processing is required to operate this dead pixel as an active pixel through an adjacent pixel. In the present invention, a dead pixel correction method used in a CMOS sensor-based camera is used. If the coordinates of the dead pixel are d0, the adjacent pixels (d1 to d8) are summed.

S160: A noise filter is performed to remove noise from image data. The noise filter is a method of restoring an image by filtering noise of an image generated as the gain of a power source or a sensor increases. A commonly used noise filter method was used. That is, it is a method of replacing the downsampled values of adjacent 2x2 pixel blocks with the noise-occurring part.

S170: Performs CFA interpolation to convert RAW data into RGB Bayer pattern data. In the present invention, the data converted to the standard video protocol in step 140 is converted into RGB format through CFA interpolation for color processing. The method of artificially configuring all colors by interpolating the color values of neighboring pixels for each pixel was implemented as it is.

S180: Pseudo color conversion is performed to colorize the black and white image. Thermal image data is basically output in black and white. It is to artificially output a color value according to the temperature by adjusting the output RGB color according to the user setting. Change the R and B values among the existing RGB values to apply color values.

S190: Convert RGB data to YCbCr for image compression. That is, the existing RGB data format is converted to YUV data format. When converting to YUV, there is an advantage in that the amount of data can be reduced and the frame rate can be increased compared to the existing data format.

S200: Image resizing is performed to output an image size according to user settings. It should be possible to output the output image in various resolutions according to the user's requirements. Accordingly, the image resize function enlarges the image by combining adjacent pixels according to the image magnification required by the user and then reducing the size, or conversely, by copying and arranging the pixels according to the image magnification.

S210: OSD insertion is performed to output text to the image according to user settings. In order to output menu characters or system messages on the basic output image, a transparent frame buffer is overlaid on the video output frame buffer and the text is output to the frame buffer, thereby implementing the function of outputting text on the output image.

S220: Converts to standard video output protocol (Camera Link, HDMI, NTSC/PAL). It is necessary to change the video data converted to YUV video output in step 190 into a format that the user can visually check. Typically, outputs such as HDMI and NTSC/PAL are required. The conversion method is output using HDMI Encoder IC and NTSC/PAL Encoder.

The above data processing sequence is performed in the main controller 200 shown in FIG. 2 of the present invention. That is, data processing is performed through the raw data correction unit 210 , the image processing unit 220 , the interface unit 240 and the image output unit 250 . In addition, the data processing sequence is performed through a multi-core processor constituting the main controller 200. That is, it is performed through the ARM cores 272 and 272 and the DSP cores 273 and 274 of the multi-core processors shown in FIGS. 3 to 8 . Specifically, the data processing process is performed through at least one of the first and second DSP cores 273 and 274. At this time, the data processing process may be alternately performed by the first and second DSP cores 273 and 274. For example, the first DSP core 273 may process one image data and the second DSP core 274 may process the next image data. Of course, for one image data, the first and second DSP cores 273 and 274 can be alternately performed. For example, data correction is performed by the first DSP core, and other thermal image TEC-less, Thermal image NUC and the like can be performed in the 2nd DSP core.

delete

The thermal imaging device according to the present invention can process thermal images by utilizing the structure of a currently used CCD/CMOS image processing processor. Since the CCD/CMOS image processing method and the thermal image processing method have different image expression methods, the algorithms and device structures used are also different. However, the thermal imaging device proposed in the present invention is designed to enable common use of ISP (Image Signal Processing) through analysis of two different structures. In order to apply a general image processing algorithm, after applying the algorithm necessary for thermal image processing, it was changed to a standard image format so that the previously used CCD/CMOS image processing was possible. Therefore, CCD/CMOS image processing and thermal image processing can be selectively performed according to system requirements.

On the other hand, the reason why it was not applied to commercial processors compared to the existing FPGA-based is that most image processors are composed of visible light sensor-based image processing systems that receive and image visible light image sensor data input, so it is difficult to process thermal images. It was impossible. However, in the present invention, the reason why a commercial processor could be used without using an FPGA is to optimize the system design by utilizing the digital signal processor (DSP) structure that can configure logic similar to that of an FPGA and the RISC processor structure, which is an advantage of a commercial processor. This is because it enables the use of an existing visible light sensor-based image processing processor.

The present invention has changed the structure of image processing of an existing commercial multi-core processor to enable thermal imaging. The difference from conventional image processing is that it is designed to first perform thermal image processing such as TEC-less and NUC before performing general image processing after receiving image data, and it is changed to a standard image processing format through image protocol conversion to improve the existing system. The system is configured for use.

In the case of the existing system, it is composed of data having RGB color values, but in the case of the thermal imaging system, since the data is composed only of the column values of the object rather than the RGB color values, imaging is impossible because the processing structure of the existing system does not match. In the present invention, in order to solve this problem, the system is reconfigured by changing the processing method. In most commercial image processors, the image processing block is composed of internal hardware and cannot be modified. Therefore, the present invention designs an internal system by using multiple processors to make the most of the existing method and reconstructing blocks requiring addition or modification in the DSP. For example, the Black Level Adjustment function is not necessary for creating thermal images. Therefore, the system is reconfigured by replacing the TEC-less function with a DSP instead of this block. In order to reduce the system load, blocks that are commonly used are used as they are. The reason why thermal image data can be used in the common block is that the image protocol converter block converts thermal image data in the same way as data output from a black and white camera.

delete

Performances of the thermal imaging device according to the present invention and the conventional FPGA-based thermal imaging device were compared, and the results are as follows. The internal structure of the thermal imaging device according to the present invention is designed to apply the ISP function used in CCD/CMOS. Currently applied functions include DPC, Edge Enhancement, Digital Zoom/PIP, and Contrast/Brightness.

10 is a view of processing dead pixels generated in a thermal image. As shown in FIG. 10 (a), it can be seen that the part where the white dot occurred due to the occurrence of dead pixels at the top and center was processed by applying the DPC function as shown in FIG. 10 (b). can Meanwhile, FIG. 10(c) is an enlarged view of the top dead pixel of FIG. 10(a), and FIG. 10(d) is an enlarged view of FIG. 10(b) of the same portion.

11(a) and 11(b) are comparison photos before and after applying edge enhancement. Unlike CCD/CMOS, it is not easy to distinguish edges due to the characteristics of thermal images. Therefore, by applying the edge enhancement function, an effect of clearing the boundary of the image can be seen.

12 is a digital zoom and PIP function. These two functions were not even tried because they consumed a lot of resources to implement in FPGA, but based on the multi-core image processor, the existing functions were changed to fit the thermal image and implemented.

13 is a graph showing processor usage for each function.

System 1 is CCD/CMOS sensor based image processing system, System 2 is thermal image sensor based NUC algorithm application system, System 3 is thermal image sensor based NUC, TEC-less algorithm applied system, System 4 is thermal image sensor based NUC, TEC -less, CEM algorithm application system. System 1 uses only the first ARM core, and System 2 mainly uses the first ARM core and the first DSP core. In System 3 and System 4, the 1st and 2nd DSP cores use more ARM cores. In System 3, the 2nd DSP core uses more than the 1st DSP core. In System 4, the 1st DSP core uses more than the 2nd DSP core. use more

Meanwhile, the biggest advantage of using a multi-core processor is reduced power consumption. In the case of existing FPGA-based systems, it is impossible to switch unused logic to a standby state, but in the proposed system, only the main core can be activated and the unused cores can be changed to a standby state. 14 is a graph comparing power consumption of the conventional FPGA-based thermal imaging system and the proposed system. When the system (without Display) has the same output as FLIR's TAU-2 and BAE's TWV640, power consumption is reduced by about 10% compared to the proposed system. In addition, if you additionally configure a system (Full Function) that includes various video output functions such as standard video output, OSD (On Screen Display), and PIP, the power consumption increases by about 20% compared to the power consumption of other companies (TAU-2, TWV640) can see However, in order to add the above function to the systems of other companies (TAU-2, TWV640), an additional video output processor is required, and when added, power consumption is expected to increase compared to the proposed system.

[Table 1] is a table comparing the size of the proposed system and the existing FPGA-based system. Compared to other products, the hardware size of the proposed system is reduced by about 20% compared to the TWV640.

Product Size(mm) TAU-20 44.45(W)x44.45(H)x29.9(D) TWV640 26.2(W)x33.3(H)x22.9(D) Proposed System 26(W)x26(H)x18.5(D)

15 is an actual photograph of the developed product. As shown, the size of the thermal imager is about 26 cm, the diameter of a coin. As power consumption and size are reduced, it can be used in portable equipment or low-power equipment, so that the usability is increased compared to conventional systems.

delete

Although the technical spirit of the present invention as described above has been specifically described according to the above embodiments, it should be noted that the above embodiments are for explanation and not for limitation. In addition, those skilled in the art will understand that various embodiments are possible within the scope of the technical spirit of the present invention.

delete

10: thermal image detector 100: detector interface unit
200: main control unit 300: power unit

Claims (15)

thermal imaging detector;
a detector interface unit converting analog image data detected by the thermal image detector into digital image data;
a main control unit for correcting and processing digital image data received from the detector interface unit; and
A power supply unit generating power and supplying power to the detector interface unit and the main control unit;
The main control unit,
A first ARM core for controlling and managing the main control unit;
a second ARM core for controlling the thermal image detector and controlling an external connection;
a first DSP core for correcting an output value according to a temperature change of the thermal image detector and correcting non-uniform values of each pixel of the thermal image to remove fixed pattern noise of the thermal image sensor;
and a second DSP core that improves the contrast ratio of the image corrected by the first DSP core and corrects it using normal pixels around dead pixels.
The thermal imaging device of claim 1 , wherein the thermal image detector includes an uncooled detector.
The thermal imaging device of claim 1 , wherein the detector interface unit, the main control unit, and the power supply unit are implemented on the same substrate or on different substrates and connected to each other through wires.
The method according to claim 1, wherein the detector interface unit,
a first detector controller configured to receive a control signal of the thermal image detector from the main controller and control the thermal image detector;
An A/D converter for receiving analog image data from the thermal image detector and converting it into digital image data;
A thermal imaging device including an interface unit controlling a connection between the thermal image detector and a main control unit.
delete delete delete The method according to claim 1, wherein the power supply unit,
A power generation unit for generating driving power for the detector interface unit and the main control unit;
an image converter for converting the image data transmitted from the main controller;
A thermal imaging device including an interface unit controlling a connection with the detector interface unit, a main controller, and an external interface.
The thermal imaging device of claim 8 , wherein the image conversion unit converts a format of the image transmitted from the main controller to deliver it to the outside.
thermal imaging detector;
a detector interface unit converting analog image data detected by the thermal image detector into digital image data; and
A main control unit for correcting and processing digital image data received from the detector interface unit;
The main control unit,
an image input unit for inputting image data from the detector interface unit;
A communication unit connected to the detector interface unit to transmit and receive a control signal;
a processing unit that controls the thermal image detector through the detector interface unit and receives, corrects, and processes digital image data from the detector interface unit;
An image output unit for outputting the processed image to the outside;
A memory unit for storing a plurality of data including image data and storing an image processing sequence;
The processing unit,
A first ARM core for controlling and managing the main control unit;
a second ARM core that controls the thermal image detector and controls external connections;
a first DSP core for correcting an output value according to a temperature change of the thermal image detector and correcting non-uniform values of each pixel of the thermal image to remove fixed pattern noise of the thermal image sensor;
and a second DSP core that improves the contrast ratio of the image corrected by the first DSP core and corrects it using normal pixels around dead pixels.
The method according to claim 10, wherein the video input unit includes at least one of MIPI CSI-2, CPI, BT.1120, and BT.656,
The communication unit includes at least one of UART, I2C, SPI, and 1Gbit Ethernet,
The image output unit includes at least one of Camera Link, HDMI, NTSC / PAL, and LCD.
delete thermal imaging detector; a detector interface unit converting analog image data detected by the thermal image detector into digital image data; a main control unit for correcting and processing digital image data received from the detector interface unit; and a power supply unit generating power and supplying power to the detector interface unit and the main control unit, wherein the main control unit controls a first ARM core that controls and manages the main control unit, the thermal imaging detector, and controls an external connection. a 2nd ARM core that corrects an output value according to temperature change of the thermal image detector, and a 1st DSP core that corrects non-uniform values of each pixel of the thermal image to remove fixed pattern noise of the thermal image sensor; An image processing method of a thermal imaging device including a second DSP core that improves the contrast ratio of an image corrected by the DSP core and corrects it using normal pixels around dead pixels, the method comprising the steps of:
converting analog image data into digital image data;
correcting an output value according to a temperature change of image data;
removing fixed pattern noise from image data;
The process of converting to a standard image protocol for using a commercial image processor;
Pseudo color conversion process for black and white images; and
Including the process of converting RGB data to YUV and compressing the image,
The image processing method of the thermal imaging device, wherein the correction, noise removal, protocol conversion, color conversion, and image compression processes are performed by the first and second DSP cores.
The method according to claim 13, after removing the fixed pattern noise,
process of correcting dead pixels;
removing noise from image data; and
An image processing method of a thermal imaging device, further comprising converting raw data into RGB Bayer pattern data.
The method according to claim 13 or 14, after converting the RGB data to YUV,
An image resize process for outputting an image size according to user settings;
OSD insertion process for outputting text on images according to user settings; and
An image processing method of a thermal imaging device, further comprising converting a standard image output protocol into a standard image output protocol.

Images