CN108764250A - A method of extracting essential image with convolutional neural networks - Google Patents

Abstract

The invention provides a method for extracting essential images by using a convolutional neural network. First, construct a two-stream convolutional network with a parallel structure from image to image; then, use a specific training data set to train the network, optimize network parameters to extract multi-layer features with environmental invariance, and directly reconstruct Generate the essential image (reflection map and light map). Since the two-stream convolutional neural network built on the basis of deep learning theory has powerful feature extraction capabilities, it can directly separate the reflection map and illumination map from the original image. At the same time, the model is a fully convolutional network model from image to image, including two branch flows, which are used to generate illumination maps and reflection maps respectively, and the network structure combines the convolution results of higher layers with the deconvolution operation The combination of the final results can reduce the reconstruction error of the illumination map and reflection map to a certain extent, and improve the ability of network feature reconstruction.

Description

A Method for Extracting Essential Images Using Convolutional Neural Networks

technical field

The invention belongs to the technical field of image processing, and in particular relates to a method for extracting essential images by using a convolutional neural network.

Background technique

Image understanding and analysis is one of the most fundamental tasks in image processing. Since the image imaging process is affected by many factors such as the characteristics of the target object itself, the shooting environment, and the acquisition equipment, the image processing process needs to fully consider the interference of many factors, such as shadows, color discontinuities, and changes in target postures. Wait. These changing factors have brought great challenges to image processing algorithms, which have greatly affected the performance of existing image analysis algorithms in complex environments. Therefore, how to improve the robustness of image analysis algorithms in complex environments has become a research hotspot in recent years. In fact, if the essential features in the image can be analyzed based on the existing observation image, the above-mentioned problems encountered in the image analysis process can be well solved. Essential features refer to the inherent features of the target object that have nothing to do with the surrounding environment, that is, the reflection characteristics of the object (including information such as color and texture) and the shape characteristics of the object. Although for the target object, these two inherent features will not change with the change of the surrounding environment, but the collected observation image of the target object will be affected by the environment. Therefore, if the essential characteristics of the object can be directly analyzed from the observation image, the inherent shape, color, texture and other information of the object can be extracted, and the influence of environmental changes on the image can be eliminated, so that a more accurate understanding of the image can be realized. At the same time, it also provides a more reliable information basis for further realizing robust image analysis in complex environments.

The existing algorithms can be divided into three categories according to the way of extracting the essential features of the target: one type of algorithm is an implicit essential feature analysis algorithm, that is, the multimodal analysis of the target object (the target object under different lighting conditions and representation under different postures) for learning. In the process of learning, this type of algorithm does not particularly consider the internal relationship between various appearances, but directly conducts pattern analysis on various observation results, thus trying to obtain the distribution function of the target in the feature space. A serious problem encountered by this type of algorithm is the generalization of the target description function. That is to say, the distribution of training samples seriously affects the final learned distribution function. If the learning samples are only images of the target object in a single lighting state or pose, it is difficult to generalize the results of training and learning to images of the target object in different lighting states or new poses. Therefore, in the case of imperfect samples, it is difficult to generalize this type of algorithm to the situation where the target object is in various complex environments. Another type of algorithm is the explicit essential feature analysis algorithm. This type of algorithm will analyze the internal connection of the target object according to its appearance in different states. Compared with the first type of implicit algorithm, this type of algorithm directly analyzes the reflection characteristics and shape of the object according to the physical imaging principle and the prior knowledge of reflection characteristics and shape. Therefore, the image of the target object in a new state can be directly calculated according to these inherent features. Therefore, the results obtained by explicit analysis algorithms are more accurate and more generalizable. However, this type of algorithm usually uses constraints such as structure, texture, and color to realize the estimation of essential features, that is, following the theoretical framework of Retinex, the signal classification problem is transformed into an energy optimization problem, and the calculation and analysis are completed on a single scale. Therefore, the accuracy of the analysis results largely depends on the performance of the optimization algorithm. At the same time, because the convexity of the established function to be optimized cannot be guaranteed, the process of solving often falls into a local minimum and cannot obtain the optimal solution, or It is required that the initialization step should be as close to the optimal solution as possible. These all limit the performance of this type of algorithm. There is another type of algorithm that uses a deep learning-based neural network to extract the essential image, and directly predicts the essential image from an RGB image by training a convolutional neural network. However, the existing algorithm network structure is relatively simple, and the training set is an artificial image synthesized by computer graphics software, so the extracted essential image is not very clear, especially when it is applied to natural images.

Contents of the invention

In order to overcome the lack of feature extraction capabilities of existing implicit and explicit essential feature analysis algorithms, and the deep learning-based neural network algorithm is mainly aimed at problems such as artificial images, the present invention provides a method for extracting essential images using a convolutional neural network. Firstly, a two-stream convolutional network with a parallel structure from image to image is constructed, and then the network is trained to optimize network parameters to extract multi-layer features with environmental invariance, and directly reconstruct the essential image (reflection map and illumination) picture). Using a multi-stream structure, on the one hand, tasks can be separated, and different streams can extract different features; on the other hand, the two are mutually restrictive conditions, which can improve the accuracy of the algorithm.

A method for extracting essential images using a convolutional neural network, characterized in that the steps are as follows:

Step 1: Construct a dual-stream convolutional neural network structure model with a parallel structure, which is divided into a public branch and two proprietary branches.

Among them, the public branch is composed of 5 convolutional layers, and each convolutional layer is followed by a pooling layer. The convolution kernels of the convolutional layers are all 3×3, and each layer outputs a feature image. The first convolutional layer outputs a feature image with a dimension of 64, the second convolutional layer outputs a feature image with a dimension of 128, and the third convolutional layer outputs a feature image with a dimension of 128. The dimension of the output feature image of the layer is 256, the dimension of the output feature image of the fourth convolutional layer and the fifth convolutional layer is 512, and the pooling layer adopts the average pooling with a size of 2×2.

The two proprietary branches have the same structure, each containing 3 deconvolution layers, the convolution kernels are 4×4, one branch is used to reconstruct the illumination image, the other branch is used to reconstruct the reflection image, all deconvolution layers The output dimensions of both are 256.

The feature image output by the third convolution layer of the public branch and the output of the second deconvolution layer of the proprietary branch are used as the input of the third deconvolution layer of the proprietary branch; the fourth convolution layer of the public branch The feature image output by the convolution layer and the output of the first deconvolution layer of the proprietary branch are used as the input of the second deconvolution layer of the proprietary branch.

Step 2: Construct the training data set. The middle part of each image in the BOLD data set created by Jiang et al. intercepts an image with a size of 1280×1280, and divides the intercepted image into five equal parts on the row and column respectively, then the original data For each image in the set, 25 images with a size of 256×256 are obtained, 53720 sets of images are randomly selected to form the test set, and the remaining images form the training set.

Step 3: Use the training set obtained in step 2 to train the two-stream convolutional neural network constructed in step 1. First, randomly initialize the weights of each layer of the network, and then use the supervised error backpropagation training method to train the network Perform training to obtain a trained network. Among them, the basic learning rate of the network is 10 ^-13 , a fixed learning rate strategy is adopted, the batch size of the network is 5, the loss function is SoftmaxwithLoss, and the network convergence condition is that the difference between the loss function values of the two iterations before and after is within ±5 of its value % range.

Step 4: Use the trained network to process the test set obtained in step 2 to obtain the extracted essential images, namely the illumination map and reflection map.

The present invention also tests the method on the MIT Intrinsic Images dataset, a public dataset of intrinsic images, and the result shows that the method is still effective.

The beneficial effects of the present invention are: due to the adoption of the technical route of essential image extraction based on deep learning theory, the reflection map and the illumination map can be directly separated from the original image by using the powerful feature extraction ability of the neural network constructed based on the deep learning theory . In addition, the two-stream convolutional neural network proposed by the present invention is a fully convolutional network model from image to image, which contains two branch flows, which are used to generate illumination maps and reflection maps respectively; and, the network structure combines higher-level Combining the convolution result with the result after the deconvolution operation can enhance the details of the feature map after the deconvolution operation, reduce the reconstruction error of the illumination map and reflection map to a certain extent, and improve the ability of network feature reconstruction.

Description of drawings

Fig. 1 is a flow chart of a method for extracting essential images using a convolutional neural network of the present invention

Fig. 2 is the two-stream convolutional neural network structural diagram that the present invention constructs

Fig. 3 is the partial image example of the dataset constructed by the present invention

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, and the present invention includes but not limited to the following embodiments.

The present invention provides a method for extracting essential images using a convolutional neural network, as shown in Figure 1, the main process is as follows:

1. Construct a two-stream convolutional neural network structure model with a parallel structure

The image reconstruction process actually assigns different weights to the features extracted from the image, and combines the same type of features to complete the goal of reconstructing the illumination map and reflection map from the original image. In other words, all desired features are present in the same original image. Therefore, the feature extraction part can be shared, while the reconstruction of two different types of essential images needs to be done separately. Therefore, the network constructed by the present invention is divided into two parts, the public branch and the private branch. After the convolution operation of the public branch, the size of the feature map output by each layer is gradually reduced. In order to keep the same size of the input image and the output image in the spatial structure, three deconvolution layers are designed on the two proprietary branches respectively, so that the spatial size of the feature map can be gradually restored to the original size. Inspired by the residual network structure, the present invention finds that combining the last two layers of the public branch with the last two layers of the proprietary branch can achieve better optimization results for network parameters. Based on the above reasons, the present invention constructs a two-stream convolutional neural network structure with a parallel structure as shown in FIG. 2 . The network model is divided into one public branch and two private branches.

Among them, the public branch is composed of 5 convolutional layers, and each convolutional layer is followed by a pooling layer. The convolution kernels of the convolutional layers are all 3×3, and each layer outputs a feature image. The first convolutional layer outputs a feature image with a dimension of 64, the second convolutional layer outputs a feature image with a dimension of 128, and the third convolutional layer outputs a feature image with a dimension of 128. The dimension of the output feature image of the layer is 256, the dimension of the output feature image of the fourth convolutional layer and the fifth convolutional layer is 512, and the pooling layer adopts the average pooling with a size of 2×2. The two proprietary branches have the same structure, each containing 3 deconvolution layers, the convolution kernels are 4×4, one branch is used to reconstruct the illumination image, the other branch is used to reconstruct the reflection image, all deconvolution layers The output dimensions of both are 256. Moreover, the feature image output by the third convolutional layer of the public branch and the output of the second deconvolution layer of the proprietary branch are used as the input of the third deconvolution layer of the proprietary branch; the fourth convolutional layer of the public branch The output feature image and the output of the first deconvolution layer of the proprietary branch are used as the input of the second deconvolution layer of the proprietary branch.

2. Data set construction

The network structure proposed by the present invention is relatively complicated, and there are many network parameters to be trained. In order to make the network play its optimal performance, the present invention uses the BOLD dataset created by Jiang et al. (Jiang X Y, Schofield AJ, Wyatt JL. Correlation-Based Intrinsic Image Extraction from a Single Image [C]. European Conference on Computer Vision, 2010:58-71) to construct a dataset for studying essential image extraction algorithms. The dataset contains 268,600 sets of images, each of which contains an original image, an illumination image, and a reflection image. 53,720 groups were randomly selected to form a test set for testing the performance of the essential image extraction algorithm. The remaining 214,880 groups constitute the training set, which is used to train the deep learning neural network. The BOLD database contains a large number of high-resolution image groups, which are all objects shot under well-adjusted lighting conditions, mainly including various complex patterns, faces and outdoor scenes. Figure 3 shows some image examples of the data set. The purpose of building this database by Jiang et al. is to provide a test platform for image processing algorithms. Specifically, it mainly includes essential image extraction algorithms, de-illumination algorithms, and light source estimation algorithms. To this end, they provide maps of lighting conditions and surfaces of objects, namely light maps and reflectance maps, and are both standard RGB color space images with linear luminance characteristics. The present invention comprehensively considers the number of pictures, picture quality and scene complexity, etc., and finally decides to select the picture group with complex patterns as the shooting object to construct the data set used in this research. The original pictures are in each dimension horizontally There are 1280 pixels and 1350 pixels in the vertical direction. The amount of data is too large for ordinary computers, and it is easy to cause the problem of over-learning, which is not conducive to the training of deep learning neural networks. The image category selected by the present invention has an obvious feature: the key information is concentrated in the middle of the image. Therefore, the present invention selects a 1280×1280 feature frame in the middle of the image, intercepts the original image, and then divides the image into five equal parts in rows and columns. In this way, an original image can be divided into 25 smaller images of 256×256. In this way, the original image is cropped, the key information in the original image is preserved, and the data utilization is maximized. At the same time, it also provides a variety of convenient conditions for this research: the amount of data in each group of pictures is moderate, and there is no impact on computer performance. Too high requirements; a relatively reasonable image size makes it more convenient to design a convolutional neural network; the cropped image contains both positive and negative samples, which can avoid overfitting to a certain extent.

3. Network training

In this embodiment, the training set in the data set created based on BOLD is used to train the constructed deconvolutional neural network under the framework of Caffe. Compared with other frameworks, the Caffe framework is not only easy to install, but also supports all operating systems, and has good interface support for Python and Matlab. Since the constructed network structure is relatively complex, the amount of data to be learned is large, and the number of iterations required by the network is also relatively large. At the same time, in order to prevent the network from learning too quickly and missing the optimal solution, in the process of training the network, The present invention decides to set the basic learning rate as 10 ^-13 and the learning rate policy as "fixed", ie, fixed learning rate, after trial and error. Considering the performance of the computer, and in order to avoid too fast network convergence, the batch size of the network is set to 5, and the loss function is SoftmaxwithLoss.

The loss function is used to calculate the difference between the output result and the real label. As the number of iterations increases, the network loss becomes smaller and smaller, that is, the estimated result is getting closer to the real label. The loss function on a single dimension can be written as:

Among them, {(x ¹ ,y ¹ ),...,(x ^m ,y ^m )} represent m groups of labeled training data _{, x} represents the input, y represents the corresponding label, and y∈[0,255]. 1{F} is an indicator function. When F is true, the function value is 1, and when it is false, the function value is 0. θ represents the parameters of the convolutional neural network. During the training process, the error between the estimated image and the real label is back-propagated to the neural network to optimize its parameters, so that the error is gradually reduced. For RGB images, the loss function is the sum of the errors of the above loss function in the three dimensions of the image R, G, and B.

After about 210,000 iterations, the difference before and after the loss function value fluctuates in the range of ±5%, and the network gradually converges, that is, the network parameters tend to be optimal under the current network structure, and the ability of the network to extract the essential image tends to be the best, and it is trained good internet. Although the two proprietary branches look the same in structure, due to the different real labels provided to them, the network parameters they learn will be different during the network training process. Therefore, when using the network to extract essential images, their operations on data will also be different accordingly, so that different branches can extract different types of essential images.

4. Use the trained network to extract the essential image

Use the trained network to process the test set contained in the data set established in step 2, that is, convert the RGB image contained in it into a three-dimensional matrix, as the input of the network, and obtain the extracted essential image after the multi-layer operation of the network, that is Lightmaps and reflection maps. The present invention also tests the method on the MIT IntrinsicImages dataset, a public dataset of intrinsic images, and the result shows that the method is still effective.

Claims (1)

1. a kind of method for extracting essential image with convolutional neural networks, it is characterised in that steps are as follows：

Step 1：The double-current convolutional neural networks structural model with parallel construction is built, which is divided into one publicly-owned point Branch, two proprietary branches；

Wherein, publicly-owned branch is made of 5 convolutional layers, and each convolutional layer is followed by a pond layer；The convolution kernel of convolutional layer is 3 × 3, each layer of one width characteristic image of output, it is 64 that the first convolutional layer, which exports characteristic image dimension, and the second convolutional layer exports feature Image dimension is 128, and it is 256 that third convolutional layer, which exports characteristic image dimension, and Volume Four lamination and the 5th convolutional layer export feature Image dimension is 512, pond layer use size for 2 × 2 average pond；

Two proprietary branched structures are identical, separately include 3 warp laminations, and convolution kernel is 4 × 4, and a branch is for reconstructing Light image, for reconstructing reflected image, the output dimension of all warp laminations is 256 for another branch；

The characteristic image of third convolutional layer output of the publicly-owned branch is made with the output of the second warp lamination of proprietary branch For the input of the third warp lamination of proprietary branch；The publicly-owned branch Volume Four lamination output characteristic image with it is proprietary Input of the output of first warp lamination of branch as the second warp lamination of proprietary branch；

Step 2：Training dataset is built, is intercepted by the middle part of every piece image of Jiang et al. BOLD data sets created big The small image for being 1280 × 1280, and respectively by five decile of interception image in row and column, then each width figure in original data set The image for being 256 × 256 as obtaining 25 width sizes, randomly selects 53720 groups of image construction test sets therein, residual image structure At training set；

Step 3：The training set obtained using step 2 is trained the double-current convolutional neural networks that step 1 is built, first to net The weights of each layer of network carry out random initializtion, then using the training method for the error back propagation for having supervision, are carried out to network Training, obtains trained network；Wherein, the basic learning rate of network is 10^-13, using fixed learning rate strategy, network Batch size are 5, loss function SoftmaxwithLoss, and network convergence condition is the loss function of front and back iteration twice The difference of value is in ± 5% range of its value；

Step 4：The test set that step 2 obtains is handled using trained network, the essential image extracted, i.e. light According to figure and reflectogram.