CN111861886B - Image super-resolution reconstruction method based on multi-scale feedback network - Google Patents
Image super-resolution reconstruction method based on multi-scale feedback network Download PDFInfo
- Publication number
- CN111861886B CN111861886B CN202010682515.XA CN202010682515A CN111861886B CN 111861886 B CN111861886 B CN 111861886B CN 202010682515 A CN202010682515 A CN 202010682515A CN 111861886 B CN111861886 B CN 111861886B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- feature
- scale
- conv
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000012549 training Methods 0.000 claims abstract description 26
- 238000013507 mapping Methods 0.000 claims abstract description 23
- 230000003321 amplification Effects 0.000 claims abstract description 12
- 238000003199 nucleic acid amplification method Methods 0.000 claims abstract description 12
- 238000004364 calculation method Methods 0.000 claims abstract description 3
- 238000006731 degradation reaction Methods 0.000 claims description 14
- 238000010586 diagram Methods 0.000 claims description 13
- 230000015556 catabolic process Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 3
- 238000012512 characterization method Methods 0.000 claims description 2
- 230000005284 excitation Effects 0.000 claims description 2
- 238000011176 pooling Methods 0.000 claims description 2
- 230000009467 reduction Effects 0.000 claims description 2
- 230000008901 benefit Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008713 feedback mechanism Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to an image super-resolution reconstruction method based on a multi-scale feedback network, which comprises the following steps: (1) creating an image dataset; (2) Extracting features of an input image, recursively realizing low-resolution and high-resolution feature mapping by using a multi-scale upper projection unit and a multi-scale lower projection unit to obtain high-resolution feature images with different depths, performing convolution calculation on the high-resolution feature images to obtain residual images, and finally interpolating the low-resolution images and adding the residual images to obtain output images; (3) Training a multi-scale feedback network by using the data set, and generating a trained network model; (4) And inputting the low-resolution image to be processed into a trained network to obtain an output high-resolution image. The method can train networks with different depths and expand to other amplification factors through small parameter adjustment, saves training cost, can realize amplification with larger amplification factors, and improves peak signal-to-noise ratio and structural similarity of reconstructed images.
Description
Technical Field
The invention relates to an image super-resolution reconstruction method based on a multi-scale feedback network, and belongs to the field of computer vision and the field of deep learning.
Background
The Super-resolution (SR) reconstruction technology is an important image processing technology in the field of computer vision, and is widely applied to the fields of medical imaging, security monitoring, remote sensing image quality improvement, image compression and target detection. Image super-Resolution reconstruction aims at building a suitable model to convert a Low Resolution (LR) image into a corresponding High Resolution (HR) image. Since a given LR image input corresponds to multiple possible HR images, the SR reconstruction problem is a challenging, pathologic inverse problem.
Currently, the proposed SR reconstruction methods are mainly divided into three categories, namely an interpolation-based method, a reconstruction-based method and a learning-based method. Among them, the SR method based on deep learning has received attention in recent years with its superior reconstruction performance. SRCNN is taken as an mountain-opening operation in the SR field of deep learning technology, and fully demonstrates the superiority of the convolutional neural network. Therefore, many networks have proposed a series of SR methods based on convolutional neural networks based on the srcan architecture. Depth as an important factor can provide a larger receptive field and more contextual information to the network, however increasing depth is extremely prone to two problems: gradient vanishing/exploding and a large number of network parameters.
In order to solve the gradient problem, researchers have proposed residual learning and successfully trained deeper networks, and in addition, some networks introduce dense connections to alleviate the gradient vanishing problem and encourage feature reuse; to reduce the parameters, researchers have proposed recursive learning to aid in weight sharing. Thanks to these mechanisms, many networks tend to construct deeper and more complex network structures to obtain higher evaluation indexes, however, many networks have the following problems:
the first and many SR methods, although realizing high performance of the deep network, neglect the training difficulty of the network, so that a huge training set is required to be spent, and more training skills and time are input.
Second, most SR methods learn hierarchical feature representations directly from LR inputs in a feed-forward manner and map to output space, this one-way mapping being dependent on limited features in the LR image. And many feedforward networks requiring preprocessing operations are only suitable for a single magnification, and the transition to other magnification requires cumbersome operations and extremely inflexible.
Disclosure of Invention
The invention provides an image super-resolution reconstruction method based on a multi-scale feedback network in order to solve the problems in the prior art. The method is characterized by comprising the following steps of:
step one, establishing a data set by using an image degradation model;
step two, constructing a multi-scale feedback network, wherein the multi-scale feedback network comprises an image feature extraction module, an image feature mapping module and a high-resolution image calculation module;
step 2.1, extracting image features;
network-input LR image I LR Input feature extraction module f 0 Generating an initial LR profile L 0 ；
L 0 ＝f 0 (I LR )
Let conv (f, n) represent the convolution layer, f is the convolution kernel size, n is the channel number; f in the above 0 Consists of 2 convolutional layers conv (3, n 0 ) And conv (1, n), n0 represents the number of channels of the initial low resolution feature extraction layer, n represents the number of input channels in the feature mapping module; firstly using conv (3, n) 0 ) Generating shallow features L with low resolution image information from an input 0 Then the conv (1, n) is used to change the channel number from n 0 Reducing to n;
step 2.2, mapping image features;
low resolution feature map L g-1 Input recursive feedback module for generating high resolution feature map H g ：
Wherein G represents the number of multi-scale projection groups, i.e., the number of recursions;representing the feature mapping process for the multi-scale projection group in the g-th recursion. When g is equal to 1, the initial characteristic diagram L is represented 0 As input to the first multiscale projection group, when g is greater than 1, the LR signature L to be generated by the previous multiscale projection group is represented g-1 As a current input;
step 2.3, calculating a high-resolution image;
computing a residual image from the plurality of HR feature map depth cascades using:
I Res ＝f RM ([H 1 ,H 2 ,…,H g ])
wherein, [ H ] 1 ,H 2 ,…,H g ]Depth concatenation representing multiple HR feature maps, f RM Representing conv (3, 3) operation, I Res Is a residual image.
Image obtained by interpolating LR image and residual image I Res Adding to obtain reconstructed high resolution image I SR ；
I SR ＝I Res +f US (I LR )
Wherein f US Representing an interpolation operation.
Training a multi-scale feedback network;
and step four, reconstructing an image.
The technical scheme is further designed as follows: the process of establishing the data set by using the image degradation model in the first step is that I is given LR Representing LR images, I HR Representing the corresponding HR image, the degradation process is expressed as:
I LR ＝D(I HR ；δ)
modeling a degradation map that generates an LR image from HR images and modeling degradation as a single downsampling operation:
wherein ∈ s Indicating that the amplification factor s is downsampled and delta is a scale factor.
The interpolation algorithm is a bilinear interpolation algorithm or a bicubic interpolation algorithm.
The loss function of the training multi-scale feedback network in the third step is as follows:
wherein x is a set of weight parameters and bias parameters, i represents a serial number of iterative training in the whole training process, and m represents the number of training images.
The beneficial effects of the invention are as follows:
the modularized end-to-end system structure not only can flexibly train networks with different depths through small parameter adjustment and be arbitrarily expanded to other amplification factors, greatly saves training cost, but also can successfully realize amplification of a larger multiple (8 times), and improves peak signal-to-noise ratio and structural similarity of reconstructed images. The method can also relieve the influence of ringing effect and chessboard artifact based on the convolutional neural network method, predicts more high-frequency details and suppresses smooth components, so that the reconstructed image has clearer and sharper edge characteristics and is closer to a real high-resolution image.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a block diagram of a multi-scale feedback network;
FIG. 3 is a block diagram of a projection unit on multiple scales in a network;
fig. 4 is a block diagram of a multi-scale projection unit in a network.
Detailed Description
The invention will now be described in detail with reference to the accompanying drawings and specific examples.
Examples
As shown in fig. 1, the image super-resolution reconstruction method based on the multi-scale feedback network of the present embodiment includes the following steps:
step 1, establishing a data set by using an image degradation model;
set I LR Representing LR images, I HR Representing the corresponding HR image, the degradation process is expressed as:
I LR ＝D(I HR ；δ) (1)
modeling a degradation map that generates an LR image from HR images and modeling degradation as a single downsampling operation:
wherein ∈ s Indicating that the amplification factor s is downsampled and delta is a scale factor.
In this embodiment, bicubic interpolation with antialiasing is used as a downsampling operation, and m training images in DIV2K are acquired as a training set. Set5, set14, urban100, BSD100, and Manga109 were chosen as standard test sets and downsampled 2-fold, 3-fold, 4-fold, and 8-fold, respectively, using bicubic interpolation algorithms.
Step 2, constructing a multi-scale feedback network; the network structure is shown in fig. 2, and comprises the following steps:
step 2.1, extracting image features;
will initial LR image I LR Input feature extraction module f 0 Generating an initial LR profile L 0 ：
L 0 ＝f 0 (I LR ) (3)
Let conv (f, n) denote the convolution layer, f be the convolution kernel size, n be the number of channels. Wherein f 0 Consists of 2 convolutional layers conv (3, n 0 ) And conv (1, n), n0 represents the number of channels of the initial LR image feature extraction layer, and n represents the number of input channels in the feature mapping module. Firstly using conv (3, n) 0 ) Generating shallow features L with LR image information from input 0 Then the conv (1, n) is used to change the channel number from n 0 Down to n.
Step 2.2, mapping image features;
and recursively realizing low-resolution and high-resolution feature mapping by using the multi-scale upper projection unit and the multi-scale lower projection unit to form a projection group, so as to obtain high-resolution feature graphs with different depths. Low resolution feature map L g-1 Input recursive feedback module for generating high resolution feature map H g ：
Wherein G represents multiple scalesThe number of degree projection sets, i.e., the number of recursions;representing the feature mapping process for the multi-scale projection group in the g-th recursion. When g is equal to 1, the initial characteristic diagram L is represented 0 As input to the first multiscale projection group, when g is greater than 1, the LR signature L to be generated by the previous multiscale projection group is represented g-1 As the current input.
The operations include two operations of mapping LR features to HR features and mapping HR features to LR features, the structure of which is shown in fig. 3 and 4.
The multiscale projection unit maps LR features to HR features (structure shown in fig. 3) by six steps:
(1): LR profile L calculated from previous cycle g-1 As input, deconvolution with different kernel sizes is used, respectivelyAnd->Up-sampling is performed on both branches to obtain two HR profile +.>And->
And->Respectively Deconv1 (k) 1 N) and Deconv2 (k) 2 ,n)，k 1 And k 2 The size of the deconvolution kernel is represented, and n represents the number of channels.
(2): HR feature mapAnd->Concatenating, using convolutions of different kernel sizes, respectively>And->Performing downsampling operations on both branches and generating two LR profiles>And->
And->Respectively Conv1 (k) 1 2 n) and Conv2 (k) 2 2 n), the number of channels per branch is changed from n to 2n.
(3): will LR characteristic diagramAnd->Cascade, pooling and dimension reduction operations using 1 x 1 convolution, +.>And->Mapping to an LR profile +.>
C u Conv (1, n) is represented, and the number of channels per branch is changed from 2n to n. And all 1 x 1 convolutions add nonlinear excitation to the learned representation of the previous layer.
(4): calculate the LR feature map L of the input g-1 And reconstructed LR feature mapsResidual error between->
(5): deconvolution with different kernel sizesAnd->Respectively->Performing up-sampling operation, wherein residual error in LR characteristic is mapped into HR characteristic, thereby generating new HR residual error characteristic +.>And->
And->Respectively represent deconvolution layer Deconv1 (k) 1 N) and Deconv2 (k) 2 N), the number of channels per branch remains n.
(6): residual HR featureAnd->Cascading and overlapping with the HR features cascaded in the step (2), outputting a final HR feature map H of the upper projection unit through 1X 1 convolution g 。
C h Conv (1, n) is shown, the total number of channels after addition is 2n, and the number of output channels is reduced to n by Conv (1, n) to keep the same as the number of input channels.
The multiscale downscaling unit maps HR features to LR features (structure shown in fig. 4) by six steps:
step (1): HR characteristic diagram H outputted by the projection unit on multiple scales of previous cycle g As input, convolutions of different kernel sizes are used, respectivelyAnd->Performing downsampling operations on the two branches to obtain two LR profiles +.>And
and->Respectively Conv1 (k) 1 N) and Conv2 (k) 2 ,n)。
Step (2): will LR characteristic diagramAnd->Cascade, respectively using deconvolution +.>And->Performing up-sampling operations on both branches and generating two HR profile +.>And->
And->Respectively Deconv1 (k) 1 2 n) and Deconv2 (k) 2 2 n), the number of channels per branch is changed from n to 2n.
Step (3): HR feature mapAnd->Cascading and obtaining HR profile +.1 by 1X 1 convolution>
C d Conv (1, n) is represented, and the number of channels per branch is changed from 2n to n.
Step (4): computing an input HR feature map H g And reconstructed HR feature mapsResidual error between->
Step (5): convolution with different kernel sizesAnd->Respectively->Downsampling, wherein residuals in the HR feature are mapped into the LR feature to generate a new LR residual feature +.>And->
And->Respectively represent the convolution layers Conv1 (k) 1 N) and Conv2 (k) 2 N), the number of channels per branch remains n.
Step (6): residual LR characterizationAnd->Cascading and overlapping with the LR features cascaded in the step 2, and outputting a final LR feature map L of the lower projection unit through 1X 1 convolution g 。
C l Conv (1, n) is shown, the total number of channels after addition is 2n, and the number of output channels is reduced to n by Conv (1, n) to keep the same as the number of input channels.
Step 2.3, calculating a high-resolution image;
the depth cascade of a plurality of high-resolution feature images is used for calculating residual images;
I Res ＝f RM ([H 1 ,H 2 ,…,H g ]) (23)
wherein, [ H ] 1 ,H 2 ,…,H g ]Depth concatenation representing multiple HR feature maps, f RM Representing conv (3, 3) operation, a series to be cascadedColumn HR feature map input conv (3, 3) generates residual image I Res 。
Image obtained by interpolating low resolution image and residual image I Res Adding to generate a reconstructed high resolution image I SR ：
I SR ＝I Res +f US (I LR ) (24)
Wherein f US Indicating an interpolation up-sampling operation, a bilinear interpolation algorithm, a bicubic interpolation algorithm, or other interpolation algorithm may be used.
Step 3, training a multi-scale feedback network;
the batch of networks is set to 16 and data enhancement is performed using rotation and flipping. LR images with different sizes and corresponding HR images are input according to the magnification coefficients. The network parameters were optimized using Adam with a momentum factor of 0.9 and a weight decay of 0.0001. The initial value of the learning rate was set to 0.0001, and the learning rate decayed to half of the original value 200 times per iteration.
Different kernel sizes and fills are designed in each branch of the multi-scale projection unit and the kernel sizes and step sizes are adjusted according to the corresponding magnification. Both input and output use RGB channels of a color image. The PReLU is used as an activation function after all convolution and deconvolution layers except for the reconstruction layer at the end of the network. Training the network according to the process of the step 2 by using the image data set of the step 1 until the cost loss is reduced to a set value and the training reaches the maximum number of iterations. By L 1 The function is taken as a loss function and expressed as follows:
wherein x is a set of weight parameters and bias parameters, i represents a serial number of iterative training in the whole training process, and m represents the number of training images.
Step 4, reconstructing an image;
and inputting the low-resolution image to be processed into a trained network to obtain an output high-resolution image.
The peak signal-to-noise ratio and the structural similarity are used as evaluation indexes to evaluate the model performance in 5 standard test sets of Set5, set14, urban100, BSD100 and Manga109, and all tests use y channels.
To verify the effectiveness and reliability of the method, it is compared with existing reconstruction methods at different magnifications. In the low magnification (×2, ×3, ×4), the present method was compared with the 21 advanced methods currently existing. Since many models are not suitable for high magnification (×8), the method is compared with 12 advanced methods. For x 2 amplification, the method achieves the best peak signal-to-noise results in the five reference data sets. However, for x 3, x 4 and x 8 amplifications, the peak signal to noise ratio and structural similarity of the method are superior to all other models. The advantages are relatively more pronounced with increasing amplification factor, especially for x 8, demonstrating the effectiveness of the present method in handling high magnification. In the five data sets, the method has higher objective evaluation indexes in terms of peak signal-to-noise ratio and structural similarity. It has been demonstrated that the present method not only tends to construct regular artificial patterns, but also is good at reconstructing irregular natural patterns. The method has advantages in adapting to various scene features and has surprising super-resolution reconstruction results for images with different features.
The multi-scale feedback network method designed in this embodiment uses only m (m=800) training images from DIV2K, and can still obtain superior reconstruction performance in 8-fold amplification over other existing methods through a relatively small training set. The multi-scale convolution is combined with a feedback mechanism, so that the method not only can learn rich hierarchical feature representations on multiple context scales and capture image features of different scales, but also can refine low-level representations by utilizing high-level features, and better characterize the interrelationship between HR and LR images. In addition to combining the high-level information and the low-level information, the local information and the global information are combined through global residual learning and local residual feedback fusion, so that the quality of the reconstructed image is better improved. In addition, the modularized end-to-end architecture enables the method to train the network with different depths flexibly and to be arbitrarily expanded to other magnification factors through small parameter adjustment. Compared with many advanced methods at present, the method has excellent reconstruction performance, and particularly has more obvious advantages in high magnification that many methods are not good.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and variations can be made without departing from the technical principles of the present invention, and these modifications and variations should also be regarded as the scope of the invention, including but not limited to using the present method and its modifications and variations for other image processing aspects, such as image classification, detection, denoising, enhancement, etc.
Claims (4)
1. The image super-resolution reconstruction method based on the multi-scale feedback network is characterized by comprising the following steps of:
step one, an image data set is established by utilizing an image degradation model;
step two, constructing a multi-scale feedback network, wherein the multi-scale feedback network comprises an image feature extraction module, an image feature mapping module and a high-resolution image calculation module;
step 2.1, extracting image features;
inputting a network into a low resolution image I LR Input feature extraction module f 0 Generating an initial low resolution feature map L 0 ；
L 0 ＝ 0 (I LR )
Let conv (f, n) represent the convolution layer, f is the convolution kernel size, n is the channel number; f in the above 0 Consists of 2 convolutional layers conv (3, n 0 ) And conv (1, n), n 0 The number of channels representing the initial low resolution feature extraction layer, n representing the number of input channels in the feature mapping module; firstly using conv (3, n) 0 ) Generating shallow features L with low resolution image information from an input 0 Then the conv (1, n) is used to change the channel number from n 0 Reducing to n;
step 2.2, mapping image features;
the multi-scale upper projection unit and the multi-scale lower projection unit form a projection group to recursively realize low-resolution and high-resolution feature mapping, so as to obtain high-resolution feature graphs with different depths; low resolution feature map L g-1 Input recursive feedback module for generating high resolution feature map H g ：
Wherein G represents the number of multi-scale projection groups, i.e., the number of recursions;representing a feature mapping process of the multi-scale projection group in the g-th recursion; when g is equal to 1, the initial characteristic diagram L is represented 0 As input to the first multiscale projection group, when g is greater than 1, the LR signature L to be generated by the previous multiscale projection group is represented g-1 As a current input; />The operations include two operations of mapping LR features to HR features and mapping HR features to LR features;
the mapping of LR features to HR features proceeds as follows:
(1): LR profile L calculated from previous cycle g-1 As input, deconvolution with different kernel sizes is used, respectivelyAndup-sampling is performed on both branches to obtain two HR profile +.>And->
And->Respectively Deconv1 (k) 1 N) and Deconv2 (k) 2 ,n)，k 1 And k 2 Indicating the size of the deconvolution kernel, n indicating the number of channels;
(2): HR feature mapAnd->Concatenating, using convolutions of different kernel sizes, respectively>And->Performing downsampling operations on both branches and generating two LR profiles>And->
And->Respectively Conv1 (k) 1 2 n) and Conv2 (k) 2 2 n), the number of channels of each branch is changed from n to 2n;
(3): will LR characteristic diagramAnd->Cascade, pooling and dimension reduction operations using 1 x 1 convolution, +.>And->Mapping to an LR profile +.>
C u Representing Conv (1, n), the general state of each branchThe number of tracks is changed from 2n to n; and all 1 x 1 convolutions add nonlinear excitation to the learning representation of the previous layer;
(4): calculate the LR feature map L of the input g-1 And reconstructed LR feature mapsResidual error between->
(5): deconvolution with different kernel sizesAnd->Respectively->Performing up-sampling operation, wherein residual error in LR characteristic is mapped into HR characteristic, thereby generating new HR residual error characteristic +.>And->
And->Respectively represent deconvolution layer Deconv1 (k) 1 N) and Deconv2 (k) 2 N), the number of channels of each branch is still n;
(6): residual HR featureAnd->Cascading and overlapping with the HR features cascaded in the step (2), outputting a final HR feature map H of the upper projection unit through 1X 1 convolution g ；
Ch represents Conv (1, n), the total channel number after addition is 2n, the output channel number is reduced to n by Conv (1, n), and the output channel number is consistent with the input channel number;
the mapping HR features to LR features proceeds as follows:
(1): HR characteristic diagram H outputted by the projection unit on multiple scales of previous cycle g As input, convolutions of different kernel sizes are used, respectivelyAnd->Performing downsampling operations on the two branches to obtain two LR profiles +.>And->
And->Respectively Conv1 (k) 1 N) and Conv2 (k) 2 ,n)；
(2): will LR characteristic diagramAnd->Cascade, respectively using deconvolution +.>And->Performing up-sampling operations on both branches and generating two HR profile +.>And->
And->Respectively Deconv1 (k) 1 2 n) and Deconv2 (k) 2 2 n), the number of channels of each branch is changed from n to 2n;
step (3): HR feature mapAnd->Cascading and obtaining HR profile +.1 by 1X 1 convolution>
C d Representing Conv (1, n), the number of channels per branch is changed from 2n to n;
step (4): computing an input HR feature map H g And reconstructed HR feature mapsResidual error between->
Step (5): convolution with different kernel sizesAnd->Respectively->Downsampling, wherein residuals in the HR feature are mapped into the LR feature to generate a new LR residual feature +.>And->
And->Respectively are provided withRepresenting the convolutional layer Conv1 (k) 1 N) and Conv2 (k) 2 N), the number of channels of each branch is still n;
(6): residual LR characterizationAnd->Cascading and overlapping with the LR characteristic cascaded in the step (2), outputting a final LR characteristic diagram L of the lower projection unit through 1X 1 convolution g ；
C l Conv (1, n) is represented, the total channel number after addition is 2n, and the output channel number is reduced to n through Conv (1, n) and is consistent with the input channel number;
step 2.3, calculating a high-resolution image;
the depth cascade of a plurality of high-resolution feature images is used for calculating residual images;
l Res ＝f RM ([H 1 ,H 2 ,…,H g ])
wherein, [ H ] 1 ,H 2 ,…,H g ]Depth concatenation representing multiple high resolution feature maps, f RM Representing conv (3, 3) operation, I Res Is a residual image;
image obtained by interpolating low resolution image and residual image I Res Adding to generate a reconstructed high resolution image I SR ；
I SR ＝I Res +f US (I LR )
Wherein f US Representing an interpolation operation;
training a multi-scale feedback network;
step four, reconstructing an image;
and inputting the low-resolution image to be processed into a trained network to obtain an output high-resolution image.
2. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the process of establishing the data set by using the image degradation model in the first step is that,
given I LR Representing low resolution images, I HR Representing the corresponding high resolution image, the degradation process is represented as:
I LR ＝D(I HR ；δ)
modeling a degradation map that generates a low resolution image from a high resolution image and modeling degradation as a single downsampling operation:
wherein ∈ s Indicating that the amplification factor s is downsampled and delta is a scale factor.
3. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the interpolation algorithm is a bilinear interpolation algorithm or a bicubic interpolation algorithm.
4. The image super-resolution reconstruction method based on a multi-scale feedback network according to claim 1, wherein: the loss function of the training multi-scale feedback network in the third step is as follows:
wherein x is a set of weight parameters and bias parameters, i represents a serial number of iterative training in the whole training process, and m represents the number of training images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010682515.XA CN111861886B (en) | 2020-07-15 | 2020-07-15 | Image super-resolution reconstruction method based on multi-scale feedback network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010682515.XA CN111861886B (en) | 2020-07-15 | 2020-07-15 | Image super-resolution reconstruction method based on multi-scale feedback network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111861886A CN111861886A (en) | 2020-10-30 |
| CN111861886B true CN111861886B (en) | 2023-08-08 |
Family
ID=72983037
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010682515.XA Active CN111861886B (en) | 2020-07-15 | 2020-07-15 | Image super-resolution reconstruction method based on multi-scale feedback network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111861886B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112418418B (en) * | 2020-11-11 | 2024-08-06 | 无锡禹空间智能科技有限公司 | Data processing method and device based on neural network, storage medium and server |
| CN112767427A (en) * | 2021-01-19 | 2021-05-07 | 西安邮电大学 | Low-resolution image recognition algorithm for compensating edge information |
| CN112927159B (en) * | 2021-03-11 | 2022-08-02 | 清华大学深圳国际研究生院 | True image denoising method based on multi-scale selection feedback network |
| CN113191949B (en) * | 2021-04-28 | 2023-06-20 | 中南大学 | Multi-scale super-resolution pathology image digitizing method, system and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108550115A (en) * | 2018-04-25 | 2018-09-18 | 中国矿业大学 | A kind of image super-resolution rebuilding method |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN108921789A (en) * | 2018-06-20 | 2018-11-30 | 华北电力大学 | Super-resolution image reconstruction method based on recurrence residual error network |
| CN109035163A (en) * | 2018-07-09 | 2018-12-18 | 南京信息工程大学 | A kind of adaptive denoising method based on deep learning |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
| CN110197468A (en) * | 2019-06-06 | 2019-09-03 | 天津工业大学 | A kind of single image Super-resolution Reconstruction algorithm based on multiple dimensioned residual error learning network |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
| CN111192200A (en) * | 2020-01-02 | 2020-05-22 | 南京邮电大学 | Image super-resolution reconstruction method based on fusion attention mechanism residual error network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9369621B2 (en) * | 2010-05-03 | 2016-06-14 | Invisage Technologies, Inc. | Devices and methods for high-resolution image and video capture |
-
2020
- 2020-07-15 CN CN202010682515.XA patent/CN111861886B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108550115A (en) * | 2018-04-25 | 2018-09-18 | 中国矿业大学 | A kind of image super-resolution rebuilding method |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN108921789A (en) * | 2018-06-20 | 2018-11-30 | 华北电力大学 | Super-resolution image reconstruction method based on recurrence residual error network |
| CN109035163A (en) * | 2018-07-09 | 2018-12-18 | 南京信息工程大学 | A kind of adaptive denoising method based on deep learning |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
| CN110197468A (en) * | 2019-06-06 | 2019-09-03 | 天津工业大学 | A kind of single image Super-resolution Reconstruction algorithm based on multiple dimensioned residual error learning network |
| CN111192200A (en) * | 2020-01-02 | 2020-05-22 | 南京邮电大学 | Image super-resolution reconstruction method based on fusion attention mechanism residual error network |
Non-Patent Citations (1)
| Title |
|---|
| 黄思炜.基于深度学习的超分辨率图像重建算法研究.《中国优秀硕士学位论文全文数据库信息科技辑》.2018,(第11期),第I138-528页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111861886A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112215755B (en) | Image super-resolution reconstruction method based on back projection attention network | |
| Yang et al. | DRFN: Deep recurrent fusion network for single-image super-resolution with large factors | |
| CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN109903228B (en) | Image super-resolution reconstruction method based on convolutional neural network | |
| CN109389552B (en) | Image super-resolution algorithm based on context-dependent multitask deep learning | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| CN106228512A (en) | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method | |
| CN106204447A (en) | The super resolution ratio reconstruction method with convolutional neural networks is divided based on total variance | |
| CN109636721B (en) | Video super-resolution method based on countermeasure learning and attention mechanism | |
| CN112837224A (en) | Super-resolution image reconstruction method based on convolutional neural network | |
| CN105631807A (en) | Single-frame image super resolution reconstruction method based on sparse domain selection | |
| Guo et al. | Multiscale semilocal interpolation with antialiasing | |
| Cao et al. | New architecture of deep recursive convolution networks for super-resolution | |
| CN113538246A (en) | Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network | |
| CN104899835A (en) | Super-resolution processing method for image based on blind fuzzy estimation and anchoring space mapping | |
| CN116188272B (en) | Two-stage depth network image super-resolution reconstruction method suitable for multiple fuzzy cores | |
| CN111353938A (en) | Image super-resolution learning method based on network feedback | |
| CN116797456A (en) | Image super-resolution reconstruction method, system, device and storage medium | |
| CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
| CN118447031A (en) | Skin mirror image segmentation method based on diffusion fuzzy learning | |
| CN116611995B (en) | Super-resolution reconstruction method of handwritten text image based on depth expansion network | |
| CN116188273A (en) | Uncertainty-oriented bimodal separable image super-resolution method | |
| CN110211059A (en) | A kind of image rebuilding method based on deep learning | |
| CN113538231B (en) | Single image super-resolution reconstruction system and method based on pixel distribution estimation | |
| CN114140317A (en) | Image animation method based on cascade generation confrontation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |