CN116402679A - A Lightweight Infrared Super-resolution Adaptive Reconstruction Method - Google Patents

Abstract

The invention belongs to the technical field of image processing, in particular to a light-weight infrared super-resolution adaptive reconstruction method, comprising the following steps: Step 1, constructing a network model: the infrared image super-resolution reconstruction model includes an input initialization layer, image feature extraction Module and output image reconstruction module; step 2, prepare the data set: prepare the infrared image data set, and perform analog downsampling and data augmentation on it for subsequent network training; step 3, train the network model: train the infrared image super-resolution rate reconstruction model. The adaptive image feature processing unit proposed by the present invention, by limiting the self-attention mechanism in the sliding window, relies on the features in the sliding window to calculate and update the feature values in the window adaptively, avoiding the use of the same image in the local window. The convolution kernel improves the expressive ability and reduces the amount of computation generated during the training and reasoning process of the self-attention mechanism.

Description

A Lightweight Infrared Super-resolution Adaptive Reconstruction Method

technical field

The invention relates to the technical field of image processing, in particular to a light-weight infrared super-resolution self-adaptive reconstruction method.

Background technique

The imaging mechanism of infrared images is to perform imaging by sensing the thermal radiation emitted by objects in the environment. It does not rely on the reflection of ambient light or artificial light sources. It has strong anti-interference and all-weather working capabilities; , are widely used in the fields of military, automatic driving, and security; however, the manufacturing process of infrared imaging sensors is relatively complicated, and dense arrays require the support of refrigerators, so their resolution is generally low and expensive; compared to directly improving imaging sensors , through the method of image super-resolution, restore the high-frequency information in part of the infrared image, improve the resolution and quality of the image, can effectively improve the imaging quality and low cost, have important practical significance and broad application prospects; infrared image super-resolution Resolution is a highly underdetermined problem. Lost details need to be estimated through a large number of image structure relationships, which makes super-resolution reconstruction of infrared images more difficult; the current mainstream solution is to use convolutional neural networks to complete from low-resolution The mapping from high-resolution infrared images to high-resolution infrared images is limited by the principle of multiplexing convolution kernel parameters in convolutional networks.

The Chinese patent publication number is "CN112308772B", and the title is "Super-resolution reconstruction method based on local and non-local information of deep learning". The screening network, including two modules of local network and non-local enhancement network, restores the lost details in the image through very deep convolution operation; convolution operation uses a fixed convolution kernel in each layer, which makes the shallow network The ability to express is very poor, so the network is often designed to be very deep and wide, which makes the computational complexity and storage capacity occupancy rate high; therefore, how to overcome the limitations of convolution operations, through a small number of learnable parameters and multiply-add It is an urgent problem for those skilled in the art to realize high-quality super-resolution reconstruction through calculation.

Contents of the invention

(1) Solved technical problems

Aiming at the deficiencies of the prior art, the present invention provides a light-weight infrared super-resolution adaptive reconstruction method, which solves the problems raised in the background art above.

(2) Technical solution

The present invention specifically adopts the following technical solutions in order to achieve the above object:

A lightweight infrared super-resolution adaptive reconstruction method, comprising the following steps:

Step 1, constructing a network model: the infrared image super-resolution reconstruction model includes an input initialization layer, an image feature extraction module and an output image reconstruction module;

Step 2, prepare the data set: prepare the infrared image data set, and perform simulated downsampling and data augmentation on it for subsequent network training;

Step 3, training the network model: training the infrared image super-resolution reconstruction model, inputting the data set prepared in step 2 into the network model constructed in step 1 for training;

Step 4, minimize the loss function and select the optimal evaluation index: by minimizing the loss function of the network output image and label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been predicted. After the training is completed, the model parameters are saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system;

Step 5, fine-tuning the model: prepare multiple additional infrared image data sets, train and fine-tune the model, obtain better model parameters, and further improve the generalization ability of the model; finally, the model can cope with various types of infrared imagers maintain good reconstruction quality;

Step 6, save the model: solidify the final model parameters, and then when the infrared image super-resolution reconstruction operation is required, directly input the image into the network to obtain the final reconstructed image.

In the aforementioned lightweight infrared super-resolution adaptive reconstruction method, the input in the infrared image super-resolution reconstruction model in step 1 is initialized as a single-layer convolutional layer, which is used to map the input image into the feature space to For further refinement and processing of subsequent features; the image feature extraction module consists of four layers of adaptive image feature processing units, specifically, the adaptive image feature processing unit consists of convolutional layer 1, self-attention layer and convolutional layer 2 The self-attention layer is composed of linear feature disassembly, self-attention mechanism, relative position encoding layer, fully connected layer 1, fully connected layer 2 and feature reorganization; the output image reconstruction module is composed of channel compression layer, global skip connection, and pixel reorganization layer composition.

Above-mentioned a kind of lightweight infrared super-resolution self-adaptive reconstruction method, the infrared image data set in the training process in the described step 2 uses FLIRADAS data set; The infrared image in the data set is simulated and down-sampled by 2, 3, 4 times respectively , for supervised training of super-resolution reconstruction models at different super-resolution scales;

In the aforementioned light-weight infrared super-resolution adaptive reconstruction method, in the step 4, the loss function is selected to use an adaptive loss function during the training process, and when the deviation value is very high, pixel loss is introduced to stabilize and quickly Optimize the network parameters to avoid the problem of gradient explosion; when the deviation value is reduced below the threshold, the structural loss is used to focus on the restoration of the texture details of the image when optimizing the network parameters; the choice of loss function affects the quality of the model, which can be realistic It can accurately reflect the difference between the predicted value and the real value, and can correctly feedback the quality of the model.

The above-mentioned light-weight infrared super-resolution adaptive reconstruction method, in the step 4, the appropriate evaluation index selection peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in the training process can effectively evaluate the algorithm super-resolution The quality of the resolution reconstruction results and the degree of distortion between the real high-resolution image, which measures the performance of the network model.

In the aforementioned light-weight infrared super-resolution adaptive reconstruction method, MFNet and TNO data sets are used in the process of fine-tuning model parameters in step 5.

The present invention also provides a light-weight infrared super-resolution electronic device, which includes: a multifunctional video stream input and output interface, a central processing unit, a plurality of graphics processing units, a storage device and stored in the memory and A computer program that can run on a processor; wherein, when the central processing unit and multiple image processing units execute the computer program, the steps of the above method are realized.

The present invention also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of the above method are realized.

(3) Beneficial effects

Compared with the prior art, the present invention provides a lightweight infrared super-resolution adaptive reconstruction method, which has the following beneficial effects:

The adaptive image feature processing unit proposed by the present invention, by limiting the self-attention mechanism in the sliding window, relies on the features in the sliding window to calculate and update the feature values in the window adaptively, avoiding the use of the same image in the local window. The convolution kernel improves the expressive ability and reduces the amount of computation generated during training and reasoning of the self-attention mechanism.

In the self-adaptive image feature processing unit proposed by the present invention, relative position coding is added in the sliding window to avoid repeated calculation of the overlapping part when calculating self-attention; when the overlapping part is reorganized, the mathematical expectation of the corresponding area in each window is used to update, There is no need to additionally design information interaction means between windows.

The present invention no longer uses the layer normalization operation in the self-attention calculation mechanism to ensure the integrity of the image structure information and contrast information; at the same time, the input feature vector and the new feature vector are spliced and input into the feedforward network for update to update Preserves the low-frequency structure of the image well.

The present invention proposes an adaptive loss function, which can monitor the state of the network model in real time during the training process, and automatically select the overall similarity or image texture details for the network to learn, thereby improving the reconstruction performance of the finally obtained network model.

Description of drawings

Fig. 1 is a flowchart of the present invention;

Fig. 2 is the network model structure of the present invention;

Fig. 3 is the processing flowchart of adaptive image processing unit of the present invention;

Fig. 4 is a schematic diagram of the working principle of the feature map in the sliding window self-attention mechanism of the present invention;

Fig. 5 is a schematic diagram of the working principle of pixel reorganization in the present invention;

Fig. 6 is a comparison result diagram of main performance indicators between the light-weight infrared super-resolution method of the present invention and the prior art;

FIG. 7 is a schematic diagram of the internal structure of an electronic device implementing a lightweight infrared super-resolution method according to the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Example 1

As shown in Figure 1, a flow chart of a lightweight infrared super-resolution adaptive reconstruction method, the method specifically includes the following steps:

Step 1, build a network model: the infrared image super-resolution reconstruction model includes an input initialization layer, an image feature extraction module and an output image reconstruction module; the input is initialized as a single-layer convolutional layer, which is used to map the input image into the feature space, to For further refinement and processing of subsequent features; the image feature extraction module consists of four layers of adaptive image feature processing units, specifically, the adaptive image feature processing unit consists of convolution layer 1, feature disassembly layer, and relative position encoding layer , self-attention layer, feature reorganization and convolution layer two, wherein the self-attention layer is composed of linear self-attention mechanism, fully connected layer 1, and fully connected layer 2; the output image reconstruction module is composed of channel compression layer and global skip connection , and a pixel reorganization layer;

Step 2, prepare the data set: prepare the FLIRADAS infrared image data set; augment the infrared image in the data set, simulate downsampling by 2, 3, and 4 times respectively, and use it for super-resolution reconstruction models of different super-resolution scales conduct supervised training;

Step 3, training the network model: training the infrared image super-resolution reconstruction model, inputting the data set prepared in step 2 into the network model constructed in step 1 for training;

Step 4, Minimize the loss function and select the optimal evaluation index: During the training process, the loss function is selected to use an adaptive loss function. When the deviation value is high, pixel loss is introduced to optimize the network parameters stably and quickly to avoid gradient explosion. The problem; when the deviation value is reduced below the threshold, the structural loss is used to focus on the restoration of the texture details of the image when optimizing the network parameters; by minimizing the loss function of the network output image and label, until the number of training times reaches the set threshold or loss When the value of the function reaches the set range, it can be considered that the model parameters have been pre-trained, and the model parameters are saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system;

Step 5, fine-tuning the model: prepare MFNet and TNO infrared image data sets, train and fine-tune the model, obtain better model parameters, and further improve the generalization ability of the model; finally, the model can cope with various types of infrared imagers maintain good reconstruction quality;

Step 6, save the model: solidify the final model parameters, and then input the image directly into the network to obtain the final reconstructed image when infrared super-resolution operation is required.

Example 2:

As shown in Figure 1, a flow chart of a lightweight infrared super-resolution adaptive reconstruction method, the method specifically includes the following steps:

Step 1, build a network model;

The entire infrared image super-resolution reconstruction model in the step 1 includes an input initialization layer, an image feature extraction module, and an output image reconstruction module; the input initialization layer is a convolution kernel of 3×3, a step size of 1, and a padding of 1 and For the convolutional layer with bias parameters set, the process of transforming the input I _ir ∈ ^1×H×W into the feature space to obtain the initial feature f ₁ ∈ ^C×H×W can be expressed as:

f ₁ =W ₁ *I _ir +B ₁

In the formula, W ₁ is the convolution kernel in the input initialization layer, B ₁ is the bias in the convolution operation, and * indicates the convolution operation;

Then, the feature is further processed by the input image feature extraction module. The image feature extraction module includes 4 adaptive image processing units, each unit is responsible for processing the feature map output by the previous layer, and combining the output feature map of this layer with the input feature map Output after splicing in the channel dimension; input feature f ₁ ∈ ^C×H×W into the image feature extraction module, and obtain the output feature f _n ∈ ^(n+1)C×H×W of each unit, n=1,2,3 , the specific process of 4 can be expressed as:

为第n个自适应图像处理单元，其工作原理如图2所示；在自适应图像处理单元中，特征图f_i′∈^C×H×W先经过一个卷积核为1×1、步长为1的卷积将通道改变为自注意力机制所需的特征向量长度，得到新特征图f₁′∈^C'^×H×W，该过程可表示为：In the formula,

is the nth adaptive image processing unit, and its working principle is shown in Figure 2; in the adaptive image processing unit, the feature map f _i ′∈ ^C×H×W firstly passes through a convolution kernel of 1×1, step The convolution with a length of 1 changes the channel to the feature vector length required by the self-attention mechanism, and obtains a new feature map f ₁ ′∈ ^C ′ ^×H×W , the process can be expressed as:

f ₁ ′=σ(W _i ′*f _i ′+B _i ′)

其中i＝1,2,......,H/m,j＝1,2,......,W/m分别为高宽方向切分的窗口编号；然后，将索引权重W_Q、查询权重W_K和内容权重W_V分别与各向量相乘，将特征向量分化为索引向量Q、查询向量K和内容向量V，该过程可以表达为：In the formula, σ(x)=max(x,0)+min(x,p) is a parameter linear rectification function; due to its high efficiency and excellent fitting ability, in the present invention, all activation functions are designed as parameter linear rectification function; next, f ₁ ′ will be processed by the sliding window self-attention mechanism shown in Fig. 3, which divides the feature into n×n n×n A vector of length C', get the set of vectors

Where i=1,2,...,H/m,j=1,2,...,W/m are the window numbers of the height and width directions; then, the index weight W _Q , query weight W _K and content weight W _V are respectively multiplied by each vector, and the feature vector is differentiated into index vector Q, query vector K and content vector V. This process can be expressed as:

Q=W _Q w _i,j ,K=W _K w _i,j ,V=W _V w _i,j

Multiply the index vector Q and the transpose K ^T of the query vector by matrix, which is equivalent to calculating the inner product in the vector set, and then calculate the correlation between different vectors in the vector set; perform softmax normalization on the correlation matrix The output of the self-attention mechanism can be obtained by multiplying it with the content vector V after optimization

可以表示为：In the formula, _BP is the relative position encoding, which is used to reduce the repeated self-attention calculation introduced in the sliding window process, d _k is the length of the feature vector; then, the feature vector first passes through the fully connected layer 1, and then spliced with the input image Get the output vector through the fully connected layer 2

It can be expressed as:

In the formula, W ₁ ′, W ₂ ′ are weight parameters of fully connected layer 1 and 2, B ₁ ′, B ₂ ′ are bias parameters of fully connected layer 1 and fully connected layer 2 respectively; after obtaining the output vector, Reassemble it into a feature map f ₂ ′∈ ^C ′ ^×H×W in the original order, where the overlapping pixels are replaced by the expectation of the gray value of the pixel in each window; finally, the feature map f ₂ ′ passes a kernel size The convolution operation of 1×1 with a step size of 1 is added to the input feature map f _i ′ to realize the local residual connection, and the output feature f _o ′ is obtained. This process can be expressed as:

f _o ′=σ(W _o ′*f ₂ ′+B _o ′)+f _i ′

After the image feature extraction module obtains the output feature f ₄ ∈ ^5C×H×W , it is input to the output image reconstruction module. First, it passes through a channel compression layer, and uses a convolution with a kernel size of 1×1 and a step size of 1 to compress the number of channels. to be the same as the initial feature f ₁ ∈ ^C×H×W , and then added to the initial feature, the channel is further reduced to the square of the scale with a convolutional layer with a kernel size of 1×1 and a step size of 1, and finally used as shown in the figure The pixel reorganization shown in 5 outputs the final super-resolution reconstructed image I _SR ∈ ^1×sH×sW (s is the super-resolution multiple); this operation can be specifically expressed as:

I _SR ＝G _pixelshuffle (W _c2 *σ(W _c1 *f ₄ )+f ₁ )

In the formula, W _c1 and W _c2 are the weight parameters of the channel compression layer and the convolution layer, and G _pixelshuffle ( ) represents the pixel shuffling operation;

Step 2, prepare the data set;

The data set in Step 2 uses the FLIR ADAS data set, which includes 8862 thermal infrared images with a resolution of 512×640; first, these images are cut into 256×256 image blocks, and a total of 37,976 image blocks are obtained. Then use bicubic downsampling to obtain low-resolution images and combine them into high- and low-resolution image pairs; in order to expand the amount of data, the images are then horizontally flipped, vertically flipped, rotated, translated, and scaled and cropped;

Step 3, train the network model;

The training scheme in the described step 3 is specifically: the number of times of training is set to be 100, and the number of pictures input to the network is about 16-32 each time, and the upper limit of the number of pictures input to the network is mainly based on the performance of the computer graphics processor. It is decided that the number of pictures input to the network each time is generally in the range of 16-32, which can make the network training more stable and the training results better; the learning rate of the training process is set to 0.001, which can not only ensure the speed of training, but also avoid gradients The problem of explosion; when the training reaches 100, 150 and 175 times, the learning rate is reduced to 0.1 of the current learning rate, which can better approach the optimal value of the parameters; the network parameter optimizer chooses the adaptive moment estimation algorithm, and its advantages are mainly After the bias correction, the learning rate of each iteration has a certain range, so that the parameters are relatively stable; the threshold of the loss function value is set to 0.01, and the training of the entire network can be considered to be basically completed if it is less than this threshold;

Step 4, minimize the loss function and select the optimal evaluation index;

In the step 4, the loss value will be calculated on the output and label of the network, and a better super-resolution reconstruction effect can be achieved by minimizing the loss function; the loss function selects structural similarity and pixel loss, and adjusts the loss function according to the current training effect of the model The use of; the structural similarity calculation formula is as follows:

SSIM(x,y)=[l(x,y)] ^α ·[c(x,y)] ^β ·[s(x,y)] ^γ

Among them, l(x,y) represents the brightness contrast function, c(x,y) represents the contrast contrast function, s(x,y) represents the structure contrast function, and the definitions of the three functions are as follows:

In practical applications, α, β and γ all take the value of 1, and C ₃ is 0.5C ₂ , so the structural similarity formula can be expressed as:

x and y respectively represent the pixels of the window of size N×N in the two images, μ _x and μ _y represent the mean values of x and y respectively, which can be used as brightness estimation; σ _x and σ _y represent the variance of x and y respectively , can be used as a contrast estimate; σ _xy represents the covariance of x and y, which can be used as a structural similarity measure; c1 and c2 are minimum value parameters, which can avoid the denominator from being 0, and usually take 0.01 and 0.03 respectively; so by definition, the entire The structural similarity of images is calculated as follows:

X and Y respectively represent the two images to be compared, MN is the total number of windows, x _ij and y _ij are the local windows in the two images; the structural similarity is symmetrical, and its value range is between [0,1] , the closer the value is to 1, the greater the structural similarity, and the smaller the difference between the two images; in general, it is enough to directly reduce the difference between it and 1 through network optimization, and the structural similarity loss is as follows:

SSIM _loss ＝1-MSSIM(I _ir ,I _SR )

By optimizing the structural similarity loss, the structural difference between the output image and the input image can be gradually reduced, making the image more similar in brightness and contrast, and more similar in intuitive perception, and the quality of the generated image is higher;

The pixel loss function is defined as follows:

At the beginning of network training or when there are severe fluctuations, pixel loss can stably optimize network parameters and allow the network to continue training in the correct direction; but pixel loss mainly comes from the difference in the low-frequency part of the energy concentration, even if the difference is small, so Structural Similarity Loss This loss that focuses on structural differences in images is more suitable for fine-tuning the network; based on this, the overall loss function is defined as:

Appropriate evaluation index selection peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in the described step 4, peak signal-to-noise ratio is based on the error between corresponding pixel points, promptly based on error-sensitive image quality evaluation; Structural similarity It measures image similarity from three aspects: brightness, contrast, and structure. It is an index used to measure the similarity of two digital images; the definition of structural similarity is the same as that of the loss function peak signal-to-noise ratio quality evaluation. The definition is as follows:

Step 5, fine-tune the model;

In the step 5, the infrared image data of the MFNet and TNO data sets are used, including a total of about 2000 infrared images with a resolution of 640×480; the images are subjected to the image preprocessing operation in the step 2 to obtain the model fine-tuning data Set; load the model weight parameters obtained in step 4, adjust the learning rate to 0.000001, input the image pairs of the model fine-tuning data set into the model, and continue training for 10 training cycles;

Step 6, save the model and parameters;

After the network training is completed in the step 6, it is necessary to save all the parameters in the network, and then input an image of any size to obtain the super-resolution reconstruction result;

Among them, the realization of operations such as convolution, splicing, and up-and-down sampling are algorithms well known to those skilled in the art, and specific processes and methods can be found in corresponding textbooks or technical documents.

By constructing a light-weight infrared super-resolution adaptive reconstruction method, the present invention can obtain a higher-quality super-resolution reconstruction effect. Due to its lightweight structure, it has fewer parameters than the previous complex network, and can Applied to various mobile devices; by calculating and obtaining the related indicators of the image with the existing method, the feasibility and superiority of the method are further verified; the comparison of the related indicators of the prior art and the method proposed by the present invention is shown in Figure 6;

Based on the same inventive concept as the above image super-resolution reconstruction method, the embodiment of the present application also provides an electronic device, which can specifically be a desktop computer, a portable computer, an edge Computing equipment, tablet computers, smart phones, etc., as shown in Figure 7, the electronic equipment can be composed of main components processor, memory and communication interface;

The processor can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processor (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable Logic devices, discrete gate or transistor logic devices, and discrete hardware components can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application; the general-purpose processor can be a microprocessor or any conventional processor; The method steps disclosed in the embodiments of the present application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor;

As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer-executable programs and modules; the memory can include at least one type of storage medium, for example, it can include random Access memory (RAM), static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, optical disk, etc.; memory is a memory that can be used to carry or store desired data in the form of instructions or data structures program code and any other medium that can be accessed by a computer, but is not limited thereto; the memory in this embodiment of the application can also be a circuit or any other device that can implement a storage function for storing program instructions or data;

The communication interface can be used for data transmission between the computing device and other computing devices, terminals or imaging devices. The communication interface can use common protocols, such as Universal Serial Bus (USB), Synchronous/Asynchronous Serial Receiver/Transmitter (USART), control device area network (CAN), etc.; the communication interface can be used to transfer data between different devices and its communication protocol, but is not limited to this; the communication interface in the embodiment of the application can also be optical communication or any other capable The method or protocol to realize the information transmission;

The present invention also provides a computer-readable storage medium for lightweight infrared super-resolution adaptive reconstruction, the computer-readable storage medium may be the computer-readable storage medium included in the device described in the above implementation manner; It can be a computer-readable storage medium that exists independently and is not assembled into the device; the computer-readable storage medium stores one or more programs, and the program is used by one or more processors to execute the method described in the present invention ;

It should be noted that although the electronic device shown in FIG. 7 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device also includes other devices necessary for normal operation; At the same time, according to specific needs, those skilled in the art should understand that the device can also include components to realize other additional functions; in addition, those skilled in the art should understand that the device can also only include the necessary devices to realize the embodiments of the present invention , without necessarily including all the devices shown in Figure 7.

Finally, it should be noted that: the above is only a preferred embodiment of the present invention, and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it still The technical solutions recorded in the foregoing embodiments may be modified, or some technical features thereof may be equivalently replaced. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1. A lightweight infrared super-resolution adaptive reconstruction method, characterized in that: comprising the steps: Step 1, constructing a network model: the infrared image super-resolution reconstruction model includes an input initialization layer, an image feature extraction module and an output image reconstruction module; Step 2, prepare the data set: prepare the infrared image data set, and perform simulated downsampling and data augmentation on it for subsequent network training; Step 3, training the network model: training the infrared image super-resolution reconstruction model, inputting the data set prepared in step 2 into the network model constructed in step 1 for training; Step 4, minimize the loss function and select the optimal evaluation index: by minimizing the loss function of the network output image and label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been predicted. After the training is completed, the model parameters are saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system; Step 5, fine-tuning the model: prepare multiple additional infrared image data sets, train and fine-tune the model, obtain better model parameters, and further improve the generalization ability of the model; finally, the model can cope with various types of infrared imagers maintain good reconstruction quality; Step 6, save the model: solidify the final model parameters, and then when the infrared image super-resolution reconstruction operation is required, directly input the image into the network to obtain the final reconstructed image. 2. a kind of light-weight infrared super-resolution adaptive reconstruction method according to claim 1, is characterized in that: in the infrared image super-resolution reconstruction model in described step 1, input initialization is single-layer convolutional layer, uses It is used to map the input image into the feature space for further refinement and processing of subsequent features; the image feature extraction module consists of four layers of adaptive image feature processing units, specifically, the adaptive image feature processing unit consists of convolutional layers 1. Self-attention layer and convolution layer 2. The self-attention layer is composed of linear feature disassembly, self-attention mechanism, relative position encoding layer, fully connected layer 1, fully connected layer 2 and feature reorganization; output image The reconstruction module consists of a channel compression layer, a global skip connection, and a pixel reorganization layer. 3. A light-weight infrared super-resolution adaptive reconstruction method according to claim 1, characterized in that: the self-attention mechanism in the step 1. 4. a kind of light-weight infrared super-resolution adaptive reconstruction method according to claim 1, is characterized in that: in described step 1, Transformer module is made of efficient global local multi-head self-attention (EGLMSA) and multi-layer perceptron ( The MLP) block consists of two layer normalization layers and two summation operations, in which the efficient global local multi-head self-attention layer extracts the global context and local context respectively. The global context is crucial for the semantic segmentation of complex urban scenes, but the local Information For preserving rich spatial details, the proposed effective global-local attention constructs two parallel branches. The local branch is a relatively shallow structure that uses two parallel convolutional layers to extract local context. Then add two batch normalization operations before the final sum operation; the global branch first deploys a depthwise convolution to reduce the image resolution, thereby compressing the amount of computation and memory, and then uses the vector as the layer normalization Input, three vectors Q, K, and V are sent into three linear predictions, Q, K, and V are obtained by linear transformation of the input word vector X, and each matrix W can be obtained through learning, and this transformation can improve The fitting ability of the model, the obtained Q, K, V can be understood as Q: the information to be queried, K: the vector to be queried, V: the value obtained by the query, perform matrix multiplication on the Q and K vectors, and then pass the volume The product layer, Softmax activation function and instance normalization operation perform attention mapping, perform matrix multiplication on the obtained attention map and V vector, and finally aggregate the global context in the global branch and the local context in the local branch to generate a global - Local context, using depthwise convolutions, batch normalization operations, and standard convolutions to represent fine-grained global-local contexts. 5. a kind of lightweight infrared super-resolution adaptive reconstruction method according to claim 1, is characterized in that: in described step 2, semantic segmentation data set uses MFNet data set; The picture cropping of training set and verification set into several block pictures, and the resolution and dimension of each block picture are the initial resolution and initial dimension; perform semantic segmentation and labeling on the block picture category. 6. a kind of lightweight infrared super-resolution adaptive reconstruction method according to claim 1, is characterized in that: in described step 3, semantic segmentation data set uses MFNet data set in pre-training process; The four-channel image channel is separated to obtain visible light color images and infrared images. Images with complex scenes, various details, and complete categories are selected as training samples, and the rest of the images are used as test set samples. Visible light images and infrared images are respectively used as input networks for training. 7. A kind of lightweight infrared super-resolution adaptive reconstruction method according to claim 1, is characterized in that: in described step 4, loss function selects DiceLoss loss function in training process; The selection of loss function affects model It can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model. 8. A light-weight infrared super-resolution adaptive reconstruction method according to claim 1, characterized in that: SODA is used in the process of fine-tuning model parameters in said step 5.

Images