CN111861886B - An image super-resolution reconstruction method based on multi-scale feedback network - Google Patents

Abstract

The invention relates to an image super-resolution reconstruction method based on a multi-scale feedback network, which comprises the following steps: (1) creating an image dataset; (2) Extracting features of an input image, recursively realizing low-resolution and high-resolution feature mapping by using a multi-scale upper projection unit and a multi-scale lower projection unit to obtain high-resolution feature images with different depths, performing convolution calculation on the high-resolution feature images to obtain residual images, and finally interpolating the low-resolution images and adding the residual images to obtain output images; (3) Training a multi-scale feedback network by using the data set, and generating a trained network model; (4) And inputting the low-resolution image to be processed into a trained network to obtain an output high-resolution image. The method can train networks with different depths and expand to other amplification factors through small parameter adjustment, saves training cost, can realize amplification with larger amplification factors, and improves peak signal-to-noise ratio and structural similarity of reconstructed images.

Description

An image super-resolution reconstruction method based on multi-scale feedback network

technical field

The invention relates to an image super-resolution reconstruction method based on a multi-scale feedback network, belonging to the fields of computer vision and deep learning.

Background technique

Image super-resolution (SR) reconstruction technology is an important image processing technology in the field of computer vision, which is widely used in medical imaging, security monitoring, improving remote sensing image quality, image compression and target detection. Image super-resolution reconstruction aims to establish a suitable model to convert a low-resolution (Low Resolution, LR) image into a corresponding high-resolution image (High Resolution, HR). Since a given LR image input corresponds to multiple possible HR images, the SR reconstruction problem is a challenging ill-conditioned inverse problem.

Currently, the proposed SR reconstruction methods are mainly divided into three categories, which are interpolation-based methods, reconstruction-based methods, and learning-based methods. Among them, the SR method based on deep learning has attracted extensive attention in recent years due to its superior reconstruction performance. As a pioneering work in the field of deep learning technology SR, SRCNN fully demonstrates the superiority of convolutional neural networks. Therefore, many networks have proposed a series of SR methods based on convolutional neural networks based on the SRCNN architecture. As an important factor, depth can provide the network with a larger receptive field and more contextual information. However, increasing the depth can easily cause two problems: gradient disappearance/explosion and a large number of network parameters.

In order to solve the gradient problem, researchers proposed residual learning and successfully trained deeper networks. In addition, some networks introduced dense connections to alleviate the problem of gradient disappearance and encourage feature reuse; in order to reduce parameters, researchers proposed recursive learning to help Weight sharing. Thanks to these mechanisms, many networks tend to construct deeper and more complex network structures to obtain higher evaluation indicators. However, research has found that many networks currently have the following problems:

First, although many SR methods achieve the high performance of the deep network, they ignore the difficulty of network training, resulting in the need to spend a huge training set and invest more training skills and time.

Second, most SR methods directly learn hierarchical feature representations from LR inputs and map to the output space in a feed-forward manner, and this one-way mapping relies on limited features in LR images. And many feedforward networks that require preprocessing operations are only suitable for a single magnification factor, and migrating to other multiples requires cumbersome operations and is extremely inflexible.

Contents of the invention

In order to solve the problems existing in the prior art, the present invention provides an image super-resolution reconstruction method based on a multi-scale feedback network. It is characterized in that, comprising the steps of:

Step 1, using the image degradation model to establish a data set;

Step 2, building a multi-scale feedback network, the multi-scale feedback network includes an image feature extraction module, an image feature mapping module and a high-resolution image calculation module;

Step 2.1, image feature extraction;

The LR image I _LR input by the network is input to the feature extraction module f ₀ to generate the initial LR feature map L ⁰ ;

L ⁰ =f ₀ (I _LR )

Let conv(f,n) represent the convolutional layer, f is the size of the convolution kernel, and n is the number of channels; in the above formula, f ₀ is composed of two convolutional layers conv(3,n ₀ ) and conv(1,n) , n0 represents the number of channels of the initial low-resolution feature extraction layer, n represents the number of input channels in the feature mapping module; first use conv(3,n ₀ ) to generate shallow features L with low-resolution image information from the input ⁰ , then use conv(1,n) to reduce the number of channels from n ₀ to n;

Step 2.2, image feature mapping;

The low-resolution feature map Lg ^-1 is input to the recursive feedback module to generate the high-resolution feature map ^Hg :

Among them, G represents the number of multi-scale projection groups, that is, the number of recursions; Denotes the feature mapping process of the multi-scale projection group in the g-th recursion. When g is equal to 1, it means that the initial feature map L ⁰ is used as the input of the first multi-scale projection group, and when g is greater than 1, it means that the LR feature map L ^g-1 generated by the previous multi-scale projection group is used as the current input;

Step 2.3, calculating the high-resolution image;

Concatenate multiple HR feature maps in depth to calculate the residual image using the following formula:

I ^Res ＝f _RM ([H ¹ ,H ² ,…,H ^g ])

Among them, [H ¹ ,H ² ,…,H ^g ] represents the depth concatenation of multiple HR feature maps, f _RM represents the conv(3,3) operation, and I ^Res is the residual image.

Adding the interpolated image of the LR image to the residual image I ^Res to obtain the reconstructed high-resolution image I ^SR ;

I ^SR ＝ I ^Res +f _US (I _LR )

Among them, f _US represents an interpolation operation.

Step 3, train the multi-scale feedback network;

Step 4, image reconstruction.

The further design of the above-mentioned technical solution is: the process of using the image degradation model to establish a data set in the first step is, given that I _LR represents an LR image, and I _HR represents a corresponding HR image, the degradation process is expressed as:

I _LR = D(I _HR ; δ)

Model the degraded map that generates the LR image from the HR image, and model the degradation as a single downsampling operation:

Among them, ↓ _s represents the magnification factor s for downsampling operation, and δ is the scaling factor.

The interpolation algorithm is a bilinear interpolation algorithm or a bicubic interpolation algorithm.

The loss function of training the multi-scale feedback network in the step 3 is:

Among them, x is a set of weight parameters and bias parameters, i represents the sequence number of iterative training in the whole training process, and m represents the number of training images.

The beneficial effects of the present invention are:

The modularized end-to-end architecture of the present invention can flexibly train networks of different depths and arbitrarily expand to other magnifications through small parameter adjustments, greatly saving training costs, and can successfully achieve greater multiples ( 8 times) magnification to improve the peak signal-to-noise ratio and structural similarity of the reconstructed image. This method can also alleviate the ringing effect and checkerboard artifacts based on the convolutional neural network method, predict more high-frequency details and suppress smooth components, so that the reconstructed image has clearer and sharper edge features, which is closer to the real high-frequency image. resolution image.

Description of drawings

Fig. 1 is the flowchart of the inventive method;

Figure 2 is a structural diagram of a multi-scale feedback network;

Figure 3 is a structural diagram of multi-scale projection units in the network;

Figure 4 is a structural diagram of the multi-scale projection unit in the network.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Example

As shown in Figure 1, the image super-resolution reconstruction method based on the multi-scale feedback network of the present embodiment includes the following steps:

Step 1, use the image degradation model to establish a data set;

Let I _LR denote the LR image, I _HR denote the corresponding HR image, and express the degradation process as:

I _LR ＝ D(I _HR ; δ) (1)

Model the degraded map that generates the LR image from the HR image, and model the degradation as a single downsampling operation:

Among them, ↓ _s represents the magnification factor s for downsampling operation, and δ is the scaling factor.

In this embodiment, bicubic interpolation with anti-aliasing is used as a downsampling operation, and m training images in DIV2K are obtained as a training set. Select Set5, Set14, Urban100, BSD100 and Manga109 as the standard test set, and use the bicubic interpolation algorithm to perform 2 times, 3 times, 4 times and 8 times downsampling respectively.

Step 2, build a multi-scale feedback network; the network structure is shown in Figure 2, including the following steps:

Step 2.1, image feature extraction;

Input the initial LR image I _LR into the feature extraction module f ₀ to generate the initial LR feature map L ⁰ :

L ⁰ ＝f ₀ (I _LR ) (3)

Let conv(f,n) denote the convolution layer, f is the convolution kernel size, and n is the number of channels. Among them, f ₀ consists of 2 convolutional layers conv(3,n ₀ ) and conv(1,n), n0 represents the number of channels of the initial LR image feature extraction layer, and n represents the number of input channels in the feature mapping module. First use conv(3,n ₀ ) to generate shallow feature L ⁰ with LR image information from the input, and then use conv(1,n) to reduce the number of channels from n ₀ to n.

Step 2.2, image feature mapping;

The multi-scale upper projection unit and the multi-scale lower projection unit are used to form a projection group to recursively realize low-resolution and high-resolution feature maps, and obtain high-resolution feature maps of different depths. The low-resolution feature map Lg ^-1 is input to the recursive feedback module to generate the high-resolution feature map ^Hg :

Among them, G represents the number of multi-scale projection groups, that is, the number of recursions; Denotes the feature mapping process of the multi-scale projection group in the g-th recursion. When g is equal to 1, it means that the initial feature map L ⁰ is used as the input of the first multi-scale projection group, and when g is greater than 1, it means that the LR feature map L ^g-1 generated by the previous multi-scale projection group is used as the current input.

The operation includes mapping LR features to HR features and mapping HR features to LR features. The structures are shown in Figure 3 and Figure 4.

The multi-scale projection unit maps LR features to HR features through the following six steps (the structure is shown in Figure 3):

(1): The LR feature map L ^g-1 calculated in the previous cycle is used as input, and deconvolution with different kernel sizes is used respectively and /> Perform an upsampling operation on the two branches to obtain two HR feature maps /> and />

and /> Represents Deconv1(k ₁ ,n) and Deconv2(k ₂ ,n), respectively, k ₁ and k ₂ represent the size of the deconvolution kernel, and n represents the number of channels.

(2): The HR feature map and /> Concatenation, using convolutions with different kernel sizes /> and /> Perform downsampling operations on both branches and generate two LR feature maps /> and />

and /> Denote Conv1(k ₁ ,2n) and Conv2(k ₂ ,2n) respectively, and the number of channels of each branch changes from n to 2n.

(3): The LR feature map and /> Cascading, using 1×1 convolution for pooling and dimensionality reduction, /> and /> Mapped to an LR feature map />

C _u represents Conv(1,n), and the number of channels of each branch changes from 2n to n. And all 1×1 convolutions add non-linear excitation on the learned representation of the previous layer.

(4): Calculate the input LR feature map L ^g-1 and the reconstructed LR feature map Residuals between />

(5): Deconvolution using different kernel sizes and /> Residuals /> Perform an upsampling operation, and the residual in the LR feature is mapped to the HR feature to generate a new HR residual feature/> and />

and /> represent the deconvolution layers Deconv1(k ₁ ,n) and Deconv2(k ₂ ,n) respectively, and the number of channels of each branch is still n.

(6): The residual HR feature and /> Concatenated, and superimposed with the HR features concatenated in step (2), the final HR feature map H ^g of the upper projection unit is output through 1×1 convolution.

C _h represents Conv(1,n), the total number of channels after addition is 2n, and the number of output channels is reduced to n by Conv(1,n), which is consistent with the number of input channels.

The multi-scale projection unit maps HR features to LR features through the following six steps (the structure is shown in Figure 4):

Step (1): The HR feature map H ^g output by the multi-scale projection unit of the previous cycle is used as input, and convolutions with different kernel sizes are used respectively. and /> Perform the downsampling operation on the two branches to obtain two LR feature maps /> and

and /> Denote Conv1(k ₁ ,n) and Conv2(k ₂ ,n) respectively.

Step (2): The LR feature map and /> Cascade, using deconvolutions with different kernel sizes respectively /> and /> Perform an upsampling operation on both branches and generate two HR feature maps /> and />

and /> represent Deconv1(k ₁ ,2n) and Deconv2(k ₂ ,2n) respectively, and the number of channels of each branch is changed from n to 2n.

Step (3): The HR feature map and /> Concatenate, and get HR feature map by 1×1 convolution />

C _d represents Conv(1,n), and the number of channels of each branch changes from 2n to n.

Step (4): Calculate the input HR feature map ^Hg and the reconstructed HR feature map Residuals between />

Step (5): Convolution with different kernel sizes and /> Residuals /> The downsampling operation is performed, and the residual in the HR feature is mapped to the LR feature to generate a new LR residual feature/> and />

and /> represent the convolutional layers Conv1(k ₁ ,n) and Conv2(k ₂ ,n) respectively, and the number of channels of each branch is still n.

Step (6): The residual LR feature and /> Concatenate and superimpose with the LR features concatenated in step 2, and output the final LR feature map L ^g of the lower projection unit through 1×1 convolution.

C _l represents Conv(1,n), the total number of channels after addition is 2n, and the number of output channels is reduced to n by Conv(1,n), which is consistent with the number of input channels.

Step 2.3, calculating the high-resolution image;

Multiple high-resolution feature map depth cascades are used to calculate the residual image using the following formula;

I ^Res =f _RM ([H ¹ ,H ² ,…,H ^g ]) (23)

Among them, [H ¹ ,H ² ,…,H ^g ] represents the deep concatenation of multiple HR feature maps, f _RM represents the conv(3,3) operation, and a series of concatenated HR feature maps are input into conv(3, 3) Generate a residual image I ^Res .

Add the interpolated low-resolution image to the residual image I ^Res to generate the reconstructed high-resolution image I ^SR :

I ^SR ＝ I ^Res +f _US (I _LR ) (24)

Wherein, f _US represents an interpolation up-sampling operation, and a bilinear interpolation algorithm may be used, or a bicubic interpolation algorithm or other interpolation algorithms may be used.

Step 3, train the multi-scale feedback network;

Set the batch of the network to 16, and employ rotation and flipping for data augmentation. Input LR images and corresponding HR images of different sizes according to the magnification factor. Use Adam to optimize the network parameters, the momentum factor is 0.9, and the weight decay is 0.0001. The initial value of the learning rate is set to 0.0001, and every 200 iterations, the learning rate decays to half of the original value.

Different kernel sizes and paddings are designed in each branch of the multi-scale projection unit and the kernel size and step size are adjusted according to the corresponding magnification. Both input and output use the RGB channels of the color image. PReLU is used as the activation function after all convolutional and deconvolutional layers except the reconstruction layer at the end of the network. Use the image data set of step 1 to train the network according to the process of step 2 until the cost loss is reduced to the set value and the training reaches the maximum number of iterations. Using the _L1 function as the loss function, the expression is as follows:

Where x is a set of weight parameters and bias parameters, i represents the sequence number of iterative training during the entire training process, and m represents the number of training images.

Step 4, image reconstruction;

Input the low-resolution image to be processed into the trained network to obtain the output high-resolution image.

The peak signal-to-noise ratio and structural similarity are used as evaluation indicators to evaluate the performance of the model in five standard test sets of Set5, Set14, Urban100, BSD100 and Manga109, and all tests use the y channel.

In order to verify the effectiveness and reliability of this method, it is compared with several existing reconstruction methods at different magnifications. At low magnifications (×2,×3,×4), the present method is compared with 21 existing state-of-the-art methods. Since many models are not suitable for high magnification (×8), the present method was compared with 12 advanced methods. For ×2 upscaling, our method achieves the best PSNR results in five benchmark datasets. However, for the scale-ups of ×3, ×4 and ×8, the peak signal-to-noise ratio and structural similarity of our method outperform all other models. As the magnification factor increases, the advantages are relatively more obvious, especially for ×8, which proves the effectiveness of this method in dealing with high magnification. In these five datasets, our method has higher objective evaluation metrics in terms of peak signal-to-noise ratio and structural similarity. It is proved that our method not only tends to construct regular artificial patterns, but also is good at reconstructing irregular natural patterns. Our method has advantages in adapting to various scene features and has amazing super-resolution reconstruction results for images with different features.

The multi-scale feedback network method designed in this embodiment only uses m (m=800) training images from DIV2K, and can still obtain reconstruction superior to other existing methods in 8 times magnification through a relatively small training set performance. Combining multi-scale convolution with a feedback mechanism enables this method to not only learn rich hierarchical feature representations at multiple context scales and capture image features at different scales, but also use high-level features to refine low-level representations to better represent HR and Interrelationships between LR images. In addition to combining high-level information and low-level information, local information and global information are also combined through global residual learning and local residual feedback fusion to better improve the quality of reconstructed images. Moreover, the modular end-to-end architecture enables our method to flexibly train networks of different depths and arbitrarily expand to other magnifications with only minor parameter adjustments. This method can effectively alleviate the influence of ringing effect and checkerboard artifacts. Compared with many current advanced methods, it has excellent reconstruction performance, especially in high magnification where many methods are not good at.

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the technical principle of the present invention, some improvements and modifications can also be made. It should also be regarded as the scope of protection of the present invention, including but not limited to using this method and its improvement and deformation methods for other image processing aspects, such as image classification, detection, denoising, enhancement, etc.

Claims (4)

1. The image super-resolution reconstruction method based on the multi-scale feedback network is characterized by comprising the following steps of:

step one, an image data set is established by utilizing an image degradation model;

step two, constructing a multi-scale feedback network, wherein the multi-scale feedback network comprises an image feature extraction module, an image feature mapping module and a high-resolution image calculation module;

step 2.1, extracting image features;

inputting a network into a low resolution image I _LR Input feature extraction module f ₀ Generating an initial low resolution feature map L ⁰ ；

L ⁰ ＝ ₀ (I _LR )

Let conv (f, n) represent the convolution layer, f is the convolution kernel size, n is the channel number; f in the above ₀ Consists of 2 convolutional layers conv (3, n ₀ ) And conv (1, n), n ₀ The number of channels representing the initial low resolution feature extraction layer, n representing the number of input channels in the feature mapping module; firstly using conv (3, n) ₀ ) Generating shallow features L with low resolution image information from an input ⁰ Then the conv (1, n) is used to change the channel number from n ₀ Reducing to n;

step 2.2, mapping image features;

the multi-scale upper projection unit and the multi-scale lower projection unit form a projection group to recursively realize low-resolution and high-resolution feature mapping, so as to obtain high-resolution feature graphs with different depths; low resolution feature map L ^g-1 Input recursive feedback module for generating high resolution feature map H ^g ：

Wherein G represents the number of multi-scale projection groups, i.e., the number of recursions;representing a feature mapping process of the multi-scale projection group in the g-th recursion; when g is equal to 1, the initial characteristic diagram L is represented ⁰ As input to the first multiscale projection group, when g is greater than 1, the LR signature L to be generated by the previous multiscale projection group is represented ^g-1 As a current input; />The operations include two operations of mapping LR features to HR features and mapping HR features to LR features;

the mapping of LR features to HR features proceeds as follows:

(1): LR profile L calculated from previous cycle ^g-1 As input, deconvolution with different kernel sizes is used, respectivelyAndup-sampling is performed on both branches to obtain two HR profile +.>And->

And->Respectively Deconv1 (k) ₁ N) and Deconv2 (k) ₂ ,n)，k ₁ And k ₂ Indicating the size of the deconvolution kernel, n indicating the number of channels;

(2): HR feature mapAnd->Concatenating, using convolutions of different kernel sizes, respectively>And->Performing downsampling operations on both branches and generating two LR profiles>And->

And->Respectively Conv1 (k) ₁ 2 n) and Conv2 (k) ₂ 2 n), the number of channels of each branch is changed from n to 2n;

(3): will LR characteristic diagramAnd->Cascade, pooling and dimension reduction operations using 1 x 1 convolution, +.>And->Mapping to an LR profile +.>

C _u Representing Conv (1, n), the general state of each branchThe number of tracks is changed from 2n to n; and all 1 x 1 convolutions add nonlinear excitation to the learning representation of the previous layer;

(4): calculate the LR feature map L of the input ^g-1 And reconstructed LR feature mapsResidual error between->

(5): deconvolution with different kernel sizesAnd->Respectively->Performing up-sampling operation, wherein residual error in LR characteristic is mapped into HR characteristic, thereby generating new HR residual error characteristic +.>And->

And->Respectively represent deconvolution layer Deconv1 (k) ₁ N) and Deconv2 (k) ₂ N), the number of channels of each branch is still n;

(6): residual HR featureAnd->Cascading and overlapping with the HR features cascaded in the step (2), outputting a final HR feature map H of the upper projection unit through 1X 1 convolution ^g ；

Ch represents Conv (1, n), the total channel number after addition is 2n, the output channel number is reduced to n by Conv (1, n), and the output channel number is consistent with the input channel number;

the mapping HR features to LR features proceeds as follows:

(1): HR characteristic diagram H outputted by the projection unit on multiple scales of previous cycle ^g As input, convolutions of different kernel sizes are used, respectivelyAnd->Performing downsampling operations on the two branches to obtain two LR profiles +.>And->

And->Respectively Conv1 (k) ₁ N) and Conv2 (k) ₂ ,n)；

(2): will LR characteristic diagramAnd->Cascade, respectively using deconvolution +.>And->Performing up-sampling operations on both branches and generating two HR profile +.>And->

And->Respectively Deconv1 (k) ₁ 2 n) and Deconv2 (k) ₂ 2 n), the number of channels of each branch is changed from n to 2n;

step (3): HR feature mapAnd->Cascading and obtaining HR profile +.1 by 1X 1 convolution>

C _d Representing Conv (1, n), the number of channels per branch is changed from 2n to n;

step (4): computing an input HR feature map H ^g And reconstructed HR feature mapsResidual error between->

Step (5): convolution with different kernel sizesAnd->Respectively->Downsampling, wherein residuals in the HR feature are mapped into the LR feature to generate a new LR residual feature +.>And->

And->Respectively are provided withRepresenting the convolutional layer Conv1 (k) ₁ N) and Conv2 (k) ₂ N), the number of channels of each branch is still n;

(6): residual LR characterizationAnd->Cascading and overlapping with the LR characteristic cascaded in the step (2), outputting a final LR characteristic diagram L of the lower projection unit through 1X 1 convolution ^g ；

C _l Conv (1, n) is represented, the total channel number after addition is 2n, and the output channel number is reduced to n through Conv (1, n) and is consistent with the input channel number;

step 2.3, calculating a high-resolution image;

the depth cascade of a plurality of high-resolution feature images is used for calculating residual images;

l ^Res ＝f _RM ([H ¹ ,H ² ,…,H ^g ])

wherein, [ H ] ¹ ,H ² ,…,H ^g ]Depth concatenation representing multiple high resolution feature maps, f _RM Representing conv (3, 3) operation, I ^Res Is a residual image;

image obtained by interpolating low resolution image and residual image I ^Res Adding to generate a reconstructed high resolution image I ^SR ；

I ^SR ＝I ^Res +f _US (I _LR )

Wherein f _US Representing an interpolation operation;

training a multi-scale feedback network;

step four, reconstructing an image;

and inputting the low-resolution image to be processed into a trained network to obtain an output high-resolution image.

2. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the process of establishing the data set by using the image degradation model in the first step is that,

given I _LR Representing low resolution images, I _HR Representing the corresponding high resolution image, the degradation process is represented as:

I _LR ＝D(I _HR ；δ)

modeling a degradation map that generates a low resolution image from a high resolution image and modeling degradation as a single downsampling operation:

wherein ∈ _s Indicating that the amplification factor s is downsampled and delta is a scale factor.

3. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the interpolation algorithm is a bilinear interpolation algorithm or a bicubic interpolation algorithm.

4. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the loss function of the training multi-scale feedback network in the third step is as follows:

wherein x is a set of weight parameters and bias parameters, i represents a serial number of iterative training in the whole training process, and m represents the number of training images.