CN118822850A - Multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system - Google Patents

Abstract

The present invention relates to a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system. The method comprises: acquiring an infrared thermal imaging image for processing, obtaining a high-resolution image and a low-resolution image as an image pair and inputting them into a multi-scale dense residual super-resolution reconstruction network model; extracting shallow low-frequency features and deep high-frequency features in the image pair for feature fusion, and inputting the fused features into an image reconstruction module to reconstruct a super-resolution image. A multi-scale dense residual super-resolution reconstruction network model is established through training, and the use of a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module can improve the reconstruction effect of the temperature information of the infrared thermal image, and can restore the texture details of the infrared thermal imaging image, thereby improving the overall quality of the reconstructed super-resolution image and improving the reconstruction effect of the infrared thermal image.

Description

Multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system

Technical Field

The present invention relates to the field of infrared detection technology, and in particular to a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system.

Background Art

With the continuous development of image technology, the scope of image application is getting wider and wider. For example, medical imaging, satellite imaging, security monitoring and other fields all require more detailed images for diagnosis or monitoring. In the traditional image acquisition process, due to the influence of traditional sensors, noise and other factors, the acquired images have problems such as distortion and blur, which affects the difficulty and accuracy of image processing, analysis, and recognition. With the continuous development and popularization of deep learning technology, a deep learning-based image super-resolution reconstruction method has been proposed and has been widely used and studied. Super-resolution reconstruction refers to the process of reconstructing high-resolution images from low-resolution images through a series of algorithms and techniques. The application of this technology can effectively improve the clarity and details of the image, thereby improving the effects of image processing, analysis, and recognition tasks. Early super-resolution reconstruction methods reconstructed images through interpolation and reconstruction methods, and performed pixel-level interpolation in low-resolution images to restore the detailed information in high-resolution images as much as possible. The application of super-resolution reconstruction of infrared thermal imaging is very wide, such as in the fields of medical imaging, industrial detection, and military fields. For example, in the medical field, in medical imaging, infrared thermal imaging technology is used to detect the temperature distribution on the human body surface for the diagnosis of disease or injury. By applying super-resolution reconstruction technology, the resolution of infrared thermal imaging images can be improved, so that subtle temperature changes can be displayed more clearly, helping doctors to diagnose diseases or monitor treatment effects more accurately. At the same time, in the industrial field, infrared thermal imaging technology is widely used to detect the heat distribution of equipment and structures to prevent equipment failure or detect structural defects. By applying infrared thermal imaging super-resolution reconstruction technology, the clarity and details of infrared thermal imaging images can be improved, so as to more accurately detect abnormal conditions in equipment or structures and improve the efficiency and safety of industrial production.

Existing super-resolution reconstruction technology requires a lot of computing resources and time for reconstruction, and has high requirements for device performance, which limits its use in real-time applications and low-energy devices. At the same time, as far as the current super-resolution reconstruction technology is concerned, the main application scenario is visible light images, and there is little corresponding research on infrared thermal imaging images. The existing algorithms may not be able to accurately restore the texture details in the image to a certain extent, and the reconstructed images will have problems such as blur and artifacts. In addition, for low-quality, blurred, and noisy input images, the current super-resolution reconstruction algorithms have limited effect on reconstructing images. Therefore, since the resolution of traditional infrared thermal imaging systems is often limited by hardware or cost constraints, their image resolution is relatively low.

Summary of the invention

Based on this, in order to solve the above technical problems, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system are provided, which can provide clearer and richer image information and reconstruct clear images.

A multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method, the method comprising:

Acquire an infrared thermal imaging image, and perform image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

Inputting the high-resolution image and the low-resolution image as image pairs into a multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model includes a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module;

Extracting shallow low-frequency features in the image pair by the shallow feature extraction module, and extracting deep high-frequency features in the image pair by the deep feature extraction module;

Based on the feature fusion module, the shallow low-frequency features and the deep high-frequency features are fused, and the fused features are input into the image reconstruction module to reconstruct a super-resolution image.

In one of the embodiments, the shallow feature extraction module is provided with a two-dimensional convolution in a convolutional neural network;

Extracting shallow low-frequency features from the image pair by the shallow feature extraction module includes:

The shallow feature extraction module extracts a first feature in the image pair through a first convolution, and extracts a second feature in the image pair through a second convolution;

The first feature and the second feature are used as shallow low-frequency features;

The first feature is used for residual learning, and the second feature is used for deep high-frequency feature extraction.

In one of the embodiments, the deep feature extraction module is provided with a plurality of multi-scale dense asymmetric modules;

Extracting deep-level high-frequency features from the image pair by the deep-layer feature extraction module includes:

Based on the Densenet network, each of the multi-scale dense asymmetric modules is densely stacked;

The second feature is used as input, and each of the multi-scale dense asymmetric modules is progressively input and output in sequence to obtain each local feature.

In one of the embodiments, the multi-scale dense asymmetric module is provided with a multi-scale dense residual module and a plurality of asymmetric convolution modules;

The asymmetric convolution module is provided with different types of convolution kernels for performing two-dimensional convolution operations;

The asymmetric convolution module includes a plurality of asymmetric convolution blocks, which are used to decompose a two-dimensional convolution into two one-dimensional convolutions.

In one embodiment, the feature fusion module is used to fuse the shallow low-frequency features and the deep high-frequency features, including:

Through the feature fusion module, a concatenate feature fusion operation is used to fuse the shallow low-frequency features and the deep high-frequency features;

Different feature maps are spliced together through the feature fusion module to obtain shallow and deep feature information.

In one of the embodiments, the image reconstruction module is provided with a sub-pixel convolution module and a convolution layer;

Inputting the fused features into the image reconstruction module to reconstruct a super-resolution image includes:

Upsampling the fused features through the sub-pixel convolution module to obtain processed data;

According to the processed data, image reconstruction is performed through the convolution layer to obtain a super-resolution image.

In one embodiment, the training process of the multi-scale dense residual super-resolution reconstruction network model includes:

Collect infrared thermal image photovoltaic panel data set images and perform image degradation processing to obtain low-resolution blurred images;

Annotating the low-resolution blurred image, and taking the low-resolution image and the high-resolution image as an image pair;

Inputting the image pair into a multi-scale dense residual super-resolution reconstruction network model, and outputting a super-resolution reconstructed image through the multi-scale dense residual super-resolution reconstruction network model;

The loss function is calculated, and parameters are updated based on the super-resolution reconstructed image to obtain optimal weight parameters, thereby completing the training of the multi-scale dense residual super-resolution reconstruction network model.

In one embodiment, the collecting of infrared thermal image photovoltaic panel data set images and performing image degradation processing to obtain a low-resolution blurred image includes:

Performing bicubic interpolation processing on the infrared thermal image photovoltaic panel data set image to obtain an image after bicubic interpolation processing;

The image after the bicubic interpolation processing is scaled to obtain a low-resolution blurred image.

A multi-scale dense residual network infrared thermal imaging super-resolution reconstruction system, the system comprising:

An image processing unit, used for acquiring an infrared thermal imaging image, and performing image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

An image input unit, used to input the high-resolution image and the low-resolution image as an image pair into a multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model includes a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module;

A feature extraction unit, configured to extract shallow-level low-frequency features in the image pair through the shallow-level feature extraction module, and to extract deep-level high-frequency features in the image pair through the deep-level feature extraction module;

The image reconstruction unit is used to perform feature fusion on the shallow-level low-frequency features and the deep-level high-frequency features based on the feature fusion module, and input the fused features into the image reconstruction module to reconstruct a super-resolution image.

The above-mentioned multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method and system establishes a multi-scale dense residual super-resolution reconstruction network model through training, and uses a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module to improve the temperature information of the infrared thermal image and restore the texture details of the image, thereby improving the overall quality of the reconstructed super-resolution image and improving the reconstruction effect of the infrared thermal image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram showing an application environment of a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method in one embodiment;

FIG2 is a schematic diagram of a process of a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method in one embodiment;

FIG3 is a schematic diagram of the structure of a multi-scale dense asymmetric module in one embodiment;

FIG4 is a schematic diagram of the structure of an asymmetric convolution module (ACB) in one embodiment;

FIG5 is a schematic diagram of the principle of asymmetric convolution in one embodiment;

FIG6 is a schematic diagram of the working principle of a sub-pixel convolution module in one embodiment;

FIG7 is a schematic diagram of a training process of a multi-scale dense residual super-resolution reconstruction network model in one embodiment;

FIG8 is a schematic diagram of a framework of a multi-scale dense residual super-resolution reconstruction network model in one embodiment;

FIG9 is a schematic diagram of the main training process of a multi-scale dense residual super-resolution reconstruction network model in one embodiment;

FIG10 and FIG11 are schematic diagrams showing a comparison of visual effects on high-resolution images with a magnification of x2 reconstructed by three traditional model algorithms in Experiment 1 according to an embodiment;

FIG. 12 and FIG. 13 are schematic diagrams showing a comparison of visual effects of Experiment 2 in one embodiment on high-resolution images reconstructed with two structurally similar models at a magnification of x2;

FIG14 is a structural block diagram of a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction system in one embodiment;

FIG. 15 is a diagram showing the internal structure of a computer device in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

It is understood that the terms "first", "second", etc. used in this application may be used herein to describe features, but these features are not limited by these terms. These terms are only used to distinguish a first feature from another feature. For example, without departing from the scope of this application, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature. Both the first feature and the second feature are features, but they are not the same feature.

The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. As shown in Figure 1, the application environment includes a computer device 110. The computer device 110 can obtain infrared thermal imaging images and perform image processing on the infrared thermal imaging images to obtain high-resolution images and low-resolution images; the computer device 110 can input the high-resolution images and low-resolution images as image pairs into the multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model includes a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module; the computer device 110 can extract shallow low-frequency features in the image pair through the shallow feature extraction module, and extract deep high-frequency features in the image pair through the deep feature extraction module; the computer device 110 can perform feature fusion on shallow low-frequency features and deep high-frequency features based on the feature fusion module, and input the fused features into the image reconstruction module to reconstruct a super-resolution image. Among them, the computer device 110 can be, but not limited to, various personal computers, laptops, robots, unmanned aerial vehicles, tablet computers and other devices.

In one embodiment, as shown in FIG2 , a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method is provided, comprising the following steps:

Step 202, acquiring an infrared thermal imaging image, and performing image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

Step 204, inputting the high-resolution image and the low-resolution image as image pairs into a multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model includes a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module;

Step 206, extracting shallow low-frequency features in the image pair through a shallow feature extraction module, and extracting deep high-frequency features in the image pair through a deep feature extraction module;

Step 208: Based on the feature fusion module, the shallow low-frequency features and the deep high-frequency features are fused, and the fused features are input into the image reconstruction module to reconstruct the super-resolution image.

In this embodiment, based on residual network theory and convolutional neural network, a multi-scale dense residual super-resolution reconstruction network model is constructed. Specifically, the multi-scale dense residual super-resolution reconstruction network model is mainly divided into four parts: a shallow feature extraction module, a deep feature extraction module, a feature fusion module and an image reconstruction module.

Among them, the shallow feature extraction module uses the two-dimensional convolution in the convolutional neural network to extract shallow detail features and simply map low-dimensional features to high-dimensional features; the deep feature extraction module is composed of a stack of multi-scale dense asymmetric modules MDAB to extract deeper and richer detail features; the feature fusion module fuses the features of different levels globally through the channel fusion operation Concat, thereby producing a richer and more comprehensive feature representation, reducing the loss of features of the entire model and enhancing the complementarity of information at different levels; the image reconstruction module is mainly composed of sub-pixel convolution modules and convolution layers, which reconstructs high-resolution images after upsampling low-resolution images.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include a shallow low-frequency feature extraction process, and the specific process includes: a two-dimensional convolution in a convolutional neural network is set in the shallow feature extraction module; the shallow feature extraction module extracts the first feature in the image pair through the first convolution, and extracts the second feature in the image pair through the second convolution; the first feature and the second feature are used as shallow low-frequency features; the first feature is used for residual learning, and the second feature is used for deep high-frequency feature extraction.

Among them, two two-dimensional convolution layers with two 3x3 convolution kernels are used in the shallow feature extraction module to perform shallow feature extraction on low-resolution images, simply mapping the input from low-dimensional space to high-dimensional space.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include a process of deep-level high-frequency feature extraction, and the specific process includes: a number of multi-scale dense asymmetric modules are set in the deep feature extraction module; based on the Densenet network, each multi-scale dense asymmetric module is densely stacked; the second feature is taken as input, and each multi-scale dense asymmetric module is progressively input and output in sequence to obtain each local feature.

In one embodiment, a multi-scale dense asymmetric module is provided with a multi-scale dense residual module and several asymmetric convolution modules; the asymmetric convolution module is provided with different types of convolution kernels for performing two-dimensional convolution operations; the asymmetric convolution module includes several asymmetric convolution blocks for decomposing the two-dimensional convolution into two one-dimensional convolutions.

Since the traditional feature extraction module mainly extracts features by stacking simple two-dimensional convolutions, the effect of extracting features is not ideal. Therefore, a multi-scale dense asymmetric MDAB module is proposed in this embodiment, wherein the structure of the multi-scale dense asymmetric module is shown in Figure 3. In the deep feature extraction module, multiple MDAB multi-scale dense blocks are stacked. A multi-scale dense residual module is added to the MDAB module, which is mainly composed of N asymmetric convolution modules (ACB) to extract deep features. At the same time, the idea of the Densenet network is adopted, and dense connections are used to utilize the extracted feature information to avoid gradient disappearance. At the same time, different depth information is integrated to improve the fluidity of information transmission, avoid information loss in the process, and improve feature extraction efficiency. The feature extraction effect of the overall module is better than that of previous models.

In this embodiment, the structure of the asymmetric convolution module (ACB) is shown in FIG4 , wherein AC-1, 3, and 5 respectively represent the use of convolution kernel sizes of 1, 3, and 5, and the following modules are asymmetric convolution block modules. Specifically, the ACB module combines the ideas of multi-scale feature extraction and asymmetric convolution, and utilizes the combination of asymmetric convolution blocks (AC) and multi-scale feature extraction ideas to improve the extraction efficiency of deep-level features and the retention of temperature information of infrared images.

In the process of feature extraction, as shown in Figure 5, firstly based on the multi-scale feature extraction block, the ACB module contains three convolution kernels (3x3, 5x5, 7x7), which is improved from the original traditional DenseNet network idea of using a single two-dimensional convolution to three two-dimensional convolution operations using 3x3, 5x5, 7x7 convolution kernels, and at the same time performs channel fusion with its own features; different convolution kernels have different receptive fields, which can capture information at different scales, and reduce information loss while retaining different scale features. In addition, temperature information based on infrared thermal imaging may be more contained in low-frequency information, and larger convolution kernels are helpful to identify thermal distribution pattern information in a larger range, thereby improving the effect of feature extraction.

Secondly, based on the asymmetric convolution block (AC), this embodiment uses two one-dimensional asymmetric convolutions and a traditional two-dimensional convolution, and uses residual learning to influence the traditional 3x3 two-dimensional convolution and enrich the feature space, and converts it into nonlinear features through the RELU activation function, so that the neural network can learn and express complex feature relationships. Specifically, the asymmetric convolution is equivalent to decomposing the two-dimensional convolution into two one-dimensional convolutions, and enhances the extraction of local key features by extracting features in the vertical and horizontal directions. At the same time, it can also reduce the convolution parameters while maintaining the effective receptive field, achieve more effective parameter sharing, and improve the model's expression of complex features.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include a feature concatenation process, the specific process including: through a feature fusion module, using a concatenate feature fusion operation to perform feature fusion on shallow low-frequency features and deep high-frequency features; through a feature fusion module, different feature maps are spliced to obtain shallow and deep feature information.

In this embodiment, the computer device adopts a concatenate feature fusion operation to fuse features at different levels and splice different feature maps to obtain feature information at all levels and improve the performance and robustness of the model.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include an image reconstruction process, and the specific process includes: a sub-pixel convolution module and a convolution layer are provided in the image reconstruction module; the fused features are up-sampled by the sub-pixel convolution module to obtain processed data; based on the processed data, the image is reconstructed by the convolution layer to obtain a super-resolution image.

In this embodiment, a sub-pixel convolution module is used in the image reconstruction module for upsampling operation, and finally the convolution layer is used to realize the final image reconstruction to obtain a high-resolution image. The working principle of the sub-pixel convolution module is shown in FIG6. Sub-pixel convolution is an upsampling method that can effectively amplify the reduced feature map. The basic idea is to add a pixel rearrangement layer after the convolution layer, which expands the number of channels and rearranges the original pixels into a higher-resolution image. Through sub-pixel convolution, the resolution of the image can be effectively improved and more detail information can be retained.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include a training process of a multi-scale dense residual super-resolution reconstruction network model, the specific process including: collecting infrared thermal image photovoltaic panel data set images for image degradation processing to obtain a low-resolution blurred image; annotating the low-resolution blurred image, and taking the low-resolution image and the high-resolution image as an image pair; inputting the image pair into the multi-scale dense residual super-resolution reconstruction network model, and outputting a super-resolution reconstructed image through the multi-scale dense residual super-resolution reconstruction network model; calculating the loss function, and updating the parameters based on the super-resolution reconstructed image to obtain the optimal weight parameters, and completing the training of the multi-scale dense residual super-resolution reconstruction network model.

In this embodiment, the training process of the multi-scale dense residual super-resolution reconstruction network model is shown in Figure 7, including: obtaining a training data set image, inputting an HR-LR image pair consisting of a low-resolution image (LR) and a high-resolution image (HR), and LR as a model input; a shallow feature extraction module extracts shallow low-frequency features, and at the same time maps the input from a low-dimensional space to a high-dimensional space; a high-frequency feature extraction module extracts deep-level high-frequency features; a feature fusion module fuses features at different levels; an image reconstruction module reconstructs a super-resolution image SR; outputs SR, and obtains the optimal weight parameters by calculating the loss with HR and supervising back propagation to update the training parameters, thereby completing the training of the multi-scale dense residual super-resolution reconstruction network model.

During the model training process, the loss function used in the training is the L1 mean absolute error (MAE), which is used to supervise the training of the model. The calculation formula is: Among them, xi is the true value of the i-th sample, _yi is the predicted value of the i-th sample, and n is the number of samples; the smaller the loss value, the closer the prediction result of the model is to the true value.

During the model training process, Peak Signal-to-Noise Ratio (PSNR) is one of the commonly used indicators to measure image or audio quality. It represents the ratio of the peak signal power to the noise power between the original signal and the distorted signal. The higher the PSNR value, the smaller the distortion and the better the image or audio quality. The calculation formula is:

During the model training process, the Structural Similarity Index (SSIM) is an indicator used to measure the similarity between two images. It not only considers the similarity of brightness information, but also the similarity of contrast and structural information, so it is more in line with human perception of image quality. SSIM can better reflect the human eye's perception of image quality, so it is considered to be a more effective image quality evaluation indicator in many cases.

In one embodiment, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided may also include a process of image degradation processing, the specific process including: performing bicubic interpolation processing on the infrared thermal image photovoltaic panel data set image to obtain a bicubic interpolation processed image; performing scaling processing on the bicubic interpolation processed image to obtain a low-resolution blurred image.

In this embodiment, the dataset used is a private infrared thermal image photovoltaic panel dataset, which mainly consists of 130 thermal images, of which 100 images are used as training sets. Among them, the degradation model of infrared thermal images mainly uses the bicubic linear interpolation method to obtain a blurred image, and then scales it to obtain a low-resolution blurred image of a specified size. The image after scaling to x2 and interpolation is compared with the original image as shown in Figure 7.

In one embodiment, in a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method provided, the framework of the multi-scale dense residual super-resolution reconstruction network model is shown in FIG8 .

Assuming that I _LR and I _HR are the input and output images in the input model, the first convolution extracts features from the I _LR image to obtain F _-1 , and then the second convolution extracts features to obtain F ₀ : F _-1 = H _CONV (I _LR ); F ₀ = H _CONV (F _-1 ). Among them, H _CONV (.) represents the 3x3 convolution operation, F _-1 is used for subsequent residual learning, and F ₀ is used for further deep feature extraction as the input of the multi-scale dense asymmetric module MDAB. It is assumed that there are N MDAB modules, so there are:

F _n =H _MDAB,n (F _n-1 ) =H _MDAB,n (H _MDAB,n-1 (...(H _MDAB,1 (F ₀ ))...))

Among them, H _MDAB,n (.) represents the nth MDAB module, and the output F _n is obtained. Each MDAB module is input and output progressively and sequentially, while retaining the outputs of local features F ₁ , F ₂ ...F _n at all levels, providing input for subsequent global feature fusion, that is, F _concat =H _concat (F ₁ ,F ₂ ,...F _n ); F _RL =F _concat +F _-1 . Among them, F _concat (.) represents global feature fusion, and F _concat is obtained after global fusion, and feature F _RL is obtained after residual learning with F _-1 . After extracting local and global features, the final high-resolution image I _HR is obtained after reconstruction module, that is, I _HR = _HRB (F _RL ) =H _MDRN (I _LR ); among them, H _RB (.) represents the reconstruction operation of the network, and H _MDRN (.) represents the function of the entire MDRN.

Specifically, the main training process of the multi-scale dense residual super-resolution reconstruction network model is shown in Figure 9, including: obtaining infrared thermal image photovoltaic panel data set images, and obtaining low-resolution blurred images through image degradation processing; after obtaining the low-resolution image, annotating the low-resolution image and the high-resolution image as an image pair as the basic data set for training; training through the multi-scale dense residual super-resolution reconstruction network model to obtain the optimal weight parameters for mapping the low-resolution image and the high-resolution image; finally, through testing, reconstructing a high-resolution clear image from the low-resolution blurred image.

In one embodiment, the multi-scale dense residual super-resolution reconstruction network model adopts 8 MDAB modules, and 8 ACB modules are used in the MDAB module. After training on a self-built infrared photovoltaic data set, it can reconstruct low-resolution images into high-resolution images with excellent effects. The low-resolution images can be well reconstructed, and the detailed features of the reconstructed images can be clearly observed, including light spots, edge textures and other detailed information. At the same time, the overall image quality is greatly improved, providing a good image foundation for subsequent target detection research.

In one embodiment, in order to illustrate the difference between the present invention and other technologies, two experiments are designed to illustrate its improvements. Experiment 1 illustrates the superiority of the present invention by comparing it with the traditional interpolation reconstruction algorithm and SRCNN and FSRCNN networks; Experiment 2 further demonstrates the advantages of the present invention in reconstructing images by comparing it with the deep residual network RDN and the RDN network containing an asymmetric convolution module. The specific experimental process is as follows:

Experiment 1: The superiority of the image reconstruction effect of this application will be quantitatively explained. Compared with traditional interpolation algorithms, this application can learn complex image features and high-level representations, so it can better capture the correlation and structural information between images, thereby generating clearer and more realistic high-resolution images, while traditional interpolation algorithms usually simply increase the resolution by pixel interpolation, which cannot effectively improve the image quality. In addition, compared with the traditional SRCNN and FSRCNN network models, this application is different from the previous cubic convolutional network structure. It uses more convolutional layers and feature extraction blocks MDAB, which can better extract the details and edge features of the image. At the same time, residual connections and feature fusion blocks are added to fully extract features at different levels and retain the overall information. Finally, sub-pixel convolution is used to upsample and reconstruct the image to achieve the effect of reconstructing a good high-resolution image. The overall model is optimized, and the reconstructed image is clearer than the image reconstructed by the SRCNN model, and the detailed edge information is more obvious, and the overall image quality is significantly improved.

Experiment 1 trains and tests on a self-built infrared photovoltaic panel dataset, analyzes the test images, and evaluates the indicators. The results of the validation set indicators under the comparison of three different models are shown in the following table:

Experiment 1 compares the visual effects of three traditional model algorithms on high-resolution images with a magnification of x2, as shown in Figures 10 and 11.

Experiment 2: Experiment 2 will compare with some models with similar structures to further illustrate the superiority of the module proposed in this application. It includes the deep residual network RDN and the network with asymmetric convolution added to the RDN network (RDN+AC). Through this experiment, the superior performance of the MDAB module in this application can be proved, and the improvement of the overall reconstruction effect of the entire MDRN model can be proved. The test results of the validation set indicators are shown in the following table:

Experiment 2 compares the visual effects on high-resolution images with a magnification of x2 reconstructed from two models with similar structures, as shown in Figures 12 and 13.

The newly created multi-scale dense residual super-resolution reconstruction network model in this application focuses on proposing a new feature extraction module, namely the multi-scale dense asymmetric module (MDAB). By combining the idea of deep residual network, N asymmetric convolution blocks (ACB) are used as basic units to effectively retain the temperature information and detail information of low-resolution images of infrared thermal images, and realize the process of transforming infrared low-resolution blurred images into high-resolution clear images. Through this model, the transformation of infrared thermal images into clearer images can be achieved, providing a good image foundation for other fields such as image detection and recognition.

A private infrared thermal imaging photovoltaic panel data set is used. The data set used can cover specific scenes and objects. The scenes and objects of this application focus on the light spots of industrial infrared photovoltaic panel thermal images. Through this data set, the field of infrared thermal imaging image reconstruction can be greatly improved. At the same time, the reconstruction effect of infrared thermal images can be further improved and can be applied in specific scenes.

The purpose of this application is to improve the quality of high-resolution images reconstructed from low-resolution infrared thermal imaging images of photovoltaic panels, and to apply them to subsequent spot detection, thereby improving the overall quality of the reconstructed image, with significant improvements in the two indicators of peak signal-to-noise ratio PSNR and structural similarity SSIM, while retaining good image detail information. The reconstructed image has significantly improved quality compared to the original image, and can provide an image of better quality.

When performing super-resolution image reconstruction, prepare the training data of the model, use the low-resolution images of the self-built photovoltaic panel data set after downsampling by the interpolation algorithm and the real original images as image pairs as the training set, of which 100 pairs of images are used as the training set and the other 30 pairs of images are used as the verification set; build the super-resolution reconstruction algorithm model, and use Python as the assembly language for training; start training the model, and print the corresponding training loss function loss and analyze the peak signal-to-noise ratio PSNR and structural similarity SSIM of each training iteration number; print out the weights of each round of training iteration and save the corresponding model; use the trained weight parameter results to test the test image, use the evaluation index to evaluate the reconstructed image and print the results, and finally obtain a high-resolution image, which can provide clearer and richer image information.

It should be understood that, although the steps in the above-mentioned flowcharts are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowcharts may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

In one embodiment, as shown in FIG14 , a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction system is provided, comprising: an image processing unit 1410, an image input unit 1420, a feature extraction unit 1430 and an image reconstruction unit 1440, wherein:

The image processing unit 1410 is used to obtain an infrared thermal imaging image and perform image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

The image input unit 1420 is used to input the high-resolution image and the low-resolution image as an image pair into the multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model includes a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an image reconstruction module;

A feature extraction unit 1430, configured to extract shallow low-frequency features in the image pair through a shallow feature extraction module, and to extract deep high-frequency features in the image pair through a deep feature extraction module;

The image reconstruction unit 1440 is used to perform feature fusion on shallow-level low-frequency features and deep-level high-frequency features based on the feature fusion module, and input the fused features into the image reconstruction module to reconstruct a super-resolution image.

In one embodiment, a two-dimensional convolution in a convolutional neural network is provided in the shallow feature extraction module; the feature extraction unit 1430 is also used for: the shallow feature extraction module extracts the first feature in the image pair through the first convolution, and extracts the second feature in the image pair through the second convolution; the first feature and the second feature are used as shallow low-frequency features; the first feature is used for residual learning, and the second feature is used for deep high-frequency feature extraction.

In one embodiment, a plurality of multi-scale dense asymmetric modules are provided in the deep feature extraction module; the feature extraction unit 1430 is also used to: densely stack the multi-scale dense asymmetric modules based on the Densenet network; take the second feature as input, and progressively input and output the multi-scale dense asymmetric modules in sequence to obtain various local features.

In one embodiment, a multi-scale dense asymmetric module is provided with a multi-scale dense residual module and several asymmetric convolution modules; the asymmetric convolution module is provided with different types of convolution kernels for performing two-dimensional convolution operations; the asymmetric convolution module includes several asymmetric convolution blocks for decomposing the two-dimensional convolution into two one-dimensional convolutions.

In one embodiment, the image reconstruction unit 1440 is also used to perform feature fusion on shallow low-frequency features and deep high-frequency features through a feature fusion module and a concatenate feature fusion operation; different feature maps are spliced through the feature fusion module to obtain shallow and deep feature information.

In one embodiment, the image reconstruction unit 1440 is also used to input the fused features into the image reconstruction module to reconstruct a super-resolution image, including: upsampling the fused features through a sub-pixel convolution module to obtain processed data; and reconstructing the image through a convolution layer based on the processed data to obtain a super-resolution image.

In one embodiment, a model training module is also included, which is used to collect infrared thermal image photovoltaic panel data set images for image degradation processing to obtain low-resolution blurred images; annotate the low-resolution blurred images, and use the low-resolution images and high-resolution images as image pairs; input the image pairs into a multi-scale dense residual super-resolution reconstruction network model, and output a super-resolution reconstructed image through the multi-scale dense residual super-resolution reconstruction network model; calculate the loss function, and update the parameters based on the super-resolution reconstructed image to obtain the optimal weight parameters, and complete the training of the multi-scale dense residual super-resolution reconstruction network model.

In one embodiment, the model training module is also used to perform bicubic interpolation processing on the infrared thermal image photovoltaic panel data set image to obtain a bicubic interpolation processed image; and to perform scaling processing on the bicubic interpolation processed image to obtain a low-resolution blurred image.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG15. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

Those skilled in the art will understand that the structure shown in FIG. 15 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, steps of a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method are implemented.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method are implemented.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims (10)

1. A multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method, characterized in that the method comprises the following steps:

Acquiring an infrared thermal imaging image, and performing image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

Inputting the high-resolution image and the low-resolution image as image pairs into a multi-scale dense residual super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model comprises a shallow layer feature extraction module, a deep layer feature extraction module, a feature fusion module and an image reconstruction module;

Extracting shallow secondary low-frequency features in the image pair through the shallow feature extraction module, and extracting deep high-frequency features in the image pair through the deep feature extraction module;

and carrying out feature fusion on the shallow sub-low frequency features and the deep high frequency features based on the feature fusion module, and inputting the fused features into the image reconstruction module to reconstruct a super-resolution image.

2. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method according to claim 1, wherein a two-dimensional convolution in a convolution neural network is arranged in the shallow feature extraction module;

Extracting shallow sub-low frequency features in the image pair by the shallow feature extraction module, including:

The shallow feature extraction module extracts first features in the image pair through a first convolution, and extracts second features in the image pair through a second convolution;

taking the first characteristic and the second characteristic as shallow low-frequency characteristics;

the first feature is used for residual learning, and the second feature is used for deep high-frequency feature extraction.

3. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method according to claim 2, wherein a plurality of multi-scale dense asymmetric modules are arranged in the deep feature extraction module;

extracting deep high-frequency features in the image pair by the deep feature extraction module, including:

Stacking the multi-scale dense asymmetric modules in a dense block manner based on Densenet networks;

and taking the second characteristic as input, gradually and sequentially inputting and outputting the multi-scale dense asymmetric modules to obtain each local characteristic.

4. The infrared thermal imaging super-resolution reconstruction method of the multi-scale dense residual error network according to claim 3, wherein a multi-scale dense residual error module and a plurality of asymmetric convolution modules are arranged in the multi-scale dense asymmetric modules;

different types of convolution kernels are arranged in the asymmetric convolution module and are used for performing two-dimensional convolution operation;

the asymmetric convolution module comprises a plurality of asymmetric convolution blocks which are used for decomposing two-dimensional convolution into two one-dimensional convolution.

5. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method according to claim 1, wherein the feature fusion module performs feature fusion on the shallow sub-low frequency features and the deep high frequency features, and the method comprises:

Performing feature fusion on the shallow secondary low-frequency features and the deep high-frequency features by adopting a concatate feature fusion operation through the feature fusion module;

And splicing different feature graphs through the feature fusion module to obtain shallow-level and deep-level feature information.

6. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method according to claim 1, wherein a sub-pixel convolution module and a convolution layer are arranged in the image reconstruction module;

inputting the fused features into the image reconstruction module to reconstruct a super-resolution image, wherein the method comprises the following steps of:

Up-sampling the fused features through the sub-pixel convolution module to obtain processed data;

and carrying out image reconstruction through the convolution layer according to the processed data to obtain a super-resolution image.

7. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction method according to claim 1, wherein the training process of the multi-scale dense residual super-resolution reconstruction network model comprises:

acquiring an infrared thermal image photovoltaic panel dataset image for image degradation treatment to obtain a low-resolution blurred image;

labeling the low-resolution blurred image, and taking the low-resolution image and the high-resolution image as an image pair;

Inputting the image pair into a multi-scale dense residual super-resolution reconstruction network model, and outputting a super-resolution reconstruction image through the multi-scale dense residual super-resolution reconstruction network model;

And calculating a loss function, and carrying out parameter updating based on the super-resolution reconstructed image to obtain an optimal weight parameter, thereby completing the training of the multi-scale dense residual error super-resolution reconstructed network model.

8. The method for reconstructing a multiscale dense residual network infrared thermal imaging super-resolution according to claim 7, wherein the acquiring an image of an infrared thermal image photovoltaic panel dataset image for image degradation processing to obtain a low-resolution blurred image comprises:

performing bicubic interpolation processing on the infrared thermal image photovoltaic panel dataset image to obtain an image subjected to bicubic interpolation processing;

And performing scaling treatment on the images subjected to bicubic interpolation treatment to obtain low-resolution blurred images.

9. A multi-scale dense residual network infrared thermal imaging super-resolution reconstruction system, the system comprising:

The image processing unit is used for acquiring an infrared thermal imaging image and carrying out image processing on the infrared thermal imaging image to obtain a high-resolution image and a low-resolution image;

The image input unit is used for inputting the high-resolution image and the low-resolution image as image pairs into a multi-scale dense residual error super-resolution reconstruction network model; the multi-scale dense residual super-resolution reconstruction network model comprises a shallow layer feature extraction module, a deep layer feature extraction module, a feature fusion module and an image reconstruction module;

the feature extraction unit is used for extracting shallow secondary low-frequency features in the image pair through the shallow feature extraction module and extracting deep high-frequency features in the image pair through the deep feature extraction module;

And the image reconstruction unit is used for carrying out feature fusion on the shallow secondary low-frequency features and the deep high-frequency features based on the feature fusion module, and inputting the fused features into the image reconstruction module to reconstruct a super-resolution image.

10. The multi-scale dense residual network infrared thermal imaging super-resolution reconstruction system according to claim 9, wherein a two-dimensional convolution in a convolutional neural network is arranged in the shallow feature extraction module; the feature extraction unit is further configured to: the shallow feature extraction module extracts first features in the image pair through a first convolution, and extracts second features in the image pair through a second convolution; taking the first characteristic and the second characteristic as shallow low-frequency characteristics; the first feature is used for residual learning, and the second feature is used for deep high-frequency feature extraction.