CN117391973A - Image de-motion blur method based on multi-scale improved residual block CNN - Google Patents

Abstract

The invention discloses an image motion blur removing method based on a multi-scale improved residual block CNN, which comprises the following steps: collecting and storing electric power inspection video data of the transmission line inspection robot, preparing a data set, and marking the data set as a source data set; clipping and compressing the source data set to obtain a clear and fuzzy image pair, and recording the clear and fuzzy image pair as a power inspection deblurring data set; constructing a multi-scale improved residual block convolutional neural network, and training the multi-scale improved residual block convolutional neural network by using a power inspection deblurring data set to obtain a trained multi-scale improved residual block convolutional neural network model; and deblurring the power inspection blurred image data by using a trained multi-scale improved residual block convolutional neural network model. The invention improves the deblurring effect and accuracy of the power inspection image, is suitable for processing the problem of the image shooting blur of the high-voltage power transmission inspection robot, and has the advantages of high operation speed, good deblurring effect and strong environment interference resistance.

Description

Image de-motion blur method based on multi-scale improved residual block CNN

Technical field

The invention belongs to the field of power inspection data processing, relates to power inspection fuzzy image data collection and deep learning technology, and specifically relates to an image de-motion blur method based on multi-scale improved residual block CNN.

Background technique

At present, my country mainly has three inspection methods: manual inspection, drone inspection, and inspection robot to conduct regular inspections of overhead transmission lines. Robot inspection uses a cable-climbing robot to walk along transmission cables at a certain speed, and can cross some obstacles on the cables. It uses its own camera and detection equipment to inspect the cables. Overhead transmission lines Methods. The inspection robot can walk autonomously on the transmission line. Through the inspection equipment such as visible light and infrared cameras it carries, it can inspect the line channel and body at close range. There are no inspection blind spots, and it is not subject to airspace control, which improves inspection efficiency. High efficiency and good effect. Compared with other inspection methods, its advantages are more prominent in areas such as mountains, rivers, lakes, and forests. This method not only reduces the labor cost of inspection, but also improves work efficiency and inspection accuracy.

However, in the actual engineering site, the inspection robot has difficulty in capturing images due to the focal length of the photo, the jitter of the robot body and the gimbal camera in windy weather, the relative movement between the photographed object and the imaging device, the intensity of the light, etc. Blurring often occurs, and image blurring has a great impact on subsequent technical work in obtaining clear image information, and even interrupts some subsequent image-based intelligent technical work, such as image recognition, target detection, and image segmentation. How to make blurry images clear, thereby enhancing the reliability and readability of image data and providing a prerequisite for subsequent technical work, has become an urgent problem to be solved.

Contents of the invention

Purpose of the invention: In order to overcome the deficiencies in the existing technology, an image de-motion blur method based on multi-scale improved residual block CNN is provided, which has the advantages of fast operation speed, good motion blur removal effect, and strong anti-environmental interference ability.

Technical solution: In order to achieve the above objectives, the present invention provides an image de-motion blur method based on multi-scale improved residual block CNN, which includes the following steps:

S1: Collect and save the power inspection video data of the transmission line inspection robot, perform frame processing on the collected power inspection video data, and then perform image sample screening and image preprocessing to obtain clear and blurred image pairs, which are made Data set, and recorded as the source data set;

S2: Complete the image reprocessing of the source data set, crop and compress the source data set, and obtain 133 pairs of clear and blurred images, which are recorded as the power inspection deblurred data set. This data set is unique to the present invention. , At present, there is no deblurred data set in the direction of power inspection. Different from the traditional public data set GoPro production method, the GoPro data set uses a method of merging clear images of more than a dozen consecutive frames into one blurred image. Deblurring of power inspection The data set uses the compression and cropping method of a single blurred image to form a new blurred image;

S3: Build a multi-scale improved residual block convolutional neural network, and use the power inspection deblurred data set to fully train the multi-scale improved residual block convolutional neural network, and obtain the trained multi-scale improved residual block convolution Neural network model. The difference between this model algorithm and the traditional multi-scale convolutional neural network lies in the improvement of the network residual structure and the construction of the loss function;

S4: Use the trained multi-scale improved residual block convolutional neural network model to deblur the power inspection blurred image data, and then perform target detection and recognition on the deblurred image to determine whether the power inspection target is in a defective state, and Record in the background and in the cloud.

Further, the step S1 is specifically:

A1: When the transmission line inspection robot is shaking, collect data from the power inspection video of the inspection robot, and obtain and save the power inspection video data when the transmission line inspection robot is shaking. The power inspection video includes the same scene. clear images and blurred images;

A2: Use the video frame processing method to perform frame processing on the inspection video collected in step A1 to obtain the framed image, and then screen the image samples to obtain images that meet the requirements;

A3: Use the histogram equalization method to preprocess the image collected in step A2 to obtain the preprocessed image data;

A4: Classify and pair the preprocessed data obtained in step A3 to obtain one-to-one corresponding pairs of clear and blurred images, which are recorded as source data sets.

Further, the step A1 includes:

When the transmission line inspection robot is shaking, the data of the power inspection video of the inspection robot is collected. The inspection robot shoots at different inspection line positions, different focal lengths, and different scenes to obtain a series of frame rates of 16 Frame/S short video and save it.

Further, the step A2 is specifically:

The videos collected in step A1 are processed into frames one by one to obtain 16*t images, where t is the time length of the video;

Perform sample screening on the frame-processed images, select images that contain the power inspection target, and discard images that have nothing to do with the power inspection target.

Further, the step A3 is specifically:

The histogram equalization method is used to preprocess the image obtained in step A2 to enhance the local contrast of the image without affecting the overall contrast; image histogram definition: a number with a gray level in the range [0, L-1] An image whose histogram is a discrete function:

p(r _k )＝n _k /n (1)

where p is the probability function, r _k is the k-th gray level, k = 0,1,2,...,L-1, L is the total number of gray levels, n _k is the pixel of the k-th gray level in the image The total number, n is the total number of pixels in the image, and then the preprocessed image is obtained.

Further, the step A4 includes:

Classify the preprocessed images into two categories: clear images and blurred images;

The obtained clear and blurred images are paired in one-to-one correspondence on the condition of the same background and the same inspection target position, and the source data set A ₁ is obtained.

Further, the step S2 is specifically:

B1: Crop the image in the source data set _A1 . The original image resolution is 2688*1520. Use the method of moving the fixed size photo frame to crop the area containing the power inspection target. The resolution and length of the resulting image must be greater than 256 Pixels, the width must be greater than 256 pixels, and the resolution sizes of clear and blurry image pairs must be exactly equal;

B2: Carry out image compression on the image obtained in step B1 to ensure that the image length is greater than 256 pixels and the width is greater than 256 pixels. The resolutions of the clear and blurry image pairs must be completely equal, thus obtaining 133 pairs of clear and blurry image pairs. , and recorded as the power inspection deblurred data set A ₂ .

Further, the step S3 is specifically:

C1: Build a multi-scale improved residual block convolutional neural network and determine the network structure;

C2: Select the network optimizer and use the public data set GoPro and the power inspection deblurred data set _A2 to train the multi-scale improved residual block convolutional neural network to obtain the trained multi-scale improved residual block convolutional neural network. Model.

Further, the structure of the multi-scale improved residual block convolutional neural network in step C1 is composed of three neural network structures of different size scales, and a clear image is generated at each scale as a sub-problem of the deblurring task , this sub-problem takes a blurred image and the initial deblurred result (downsampling from the previous scale) as input, and infers that at this scale, this clear image is:

I ^m , h ^m =Net(B ^m ,I ^m+1 ,h ^m+1 ; θ) (2)

Among them, I ^m is the deblurred image output by the layer network at this level of scale, h ^m is the hidden layer feature at this level of scale, Net is the multi-scale improved residual block convolutional neural network proposed by the present invention, and B ^m is The blurred image input by the network of this layer at this level of scale, I ^m+1 is the deblurred image output by the previous level of scale, h ^m+1 is the hidden layer feature of the previous level of scale, θ is the parameter to be trained, and m is the original Level scale, m+1 is the upper level scale.

Further, the neural network structure at any scale in step C1 is the same, and the specific construction method is as follows:

The first part is the input module, including a convolution layer and three improved residual blocks; any one of the improved residual blocks is different from the traditional improved residual block structure. This invention uses splicing operations to replace the original improved residual blocks. The element-by-element addition operation of the block includes two convolutional layers, a linear rectification function and a skip layer splicing; the splicing operation has the following three advantages compared to the original element-wise addition operation of the improved residual block: 1. Information fusion: By connecting two feature maps in the channel dimension, the fusion and combination of information can be achieved, allowing the network to utilize more contextual information. 2. Representation capability enhancement: The connection operation expands the number of channels of the feature map, increases the representation capability of the network, and helps to learn more complex feature representations. 3. Multi-scale feature fusion: Multi-scale feature fusion can be achieved by connecting feature maps with different spatial resolutions, improving the model's perception of targets at different scales. The input of the first part comes from the public data set GoPro and the power inspection deblurred data set _A2 . Each image in the data set is processed by the OpenCV program to obtain a partial image of 256*256 size, and then sent to the input module. The first convolution layer uses 64 3*3 convolution kernels, the step size is 1, and the number of padding is 1. The output size calculation formula of the convolution layer is as follows:

Among them, Z is the length of the convolution output data, W is the length of the convolution input data, P is the number of padding, F is the length of the convolution kernel, and S represents the step size; for the first coding block in the first part, the The output size of a convolution layer is calculated by formula (3) and the output size of the first convolution layer is 256*256*64; since the convolution kernel size and format used in the first part of the convolution layer are the same, the parameters Same, and so on, the final output size of the first part of the convolutional layer and the three improved residual blocks is also 256*256*64;

The second part is a coding structure composed of two coding blocks, where any coding block contains a convolutional layer, a linear rectification function and three improved residual blocks; for the first coding block in the second part, this layer 128 3*3 convolution kernels are selected, the step size is 2, and the padding amount is 0.5. The output size of the first convolution layer is calculated using the formula (3). The calculated output size of the first convolution layer is 128 *128*128; After the first convolutional layer of the coding block, a linear rectification function is used as the activation function, and the data after the activation function is sent to the second convolutional layer of the coding block. The second convolutional layer uses 64 A 3*3 convolution kernel, the step size is 2, and the padding amount is 0.5. According to the output size calculation formula (3) of the convolution layer, the output size of the second convolution layer in the second part is also 128*128*128 ; Since the convolutional layer in the second part uses the same convolution kernel size and format, the same parameters, and so on, the final output size of the first coding block in the second part is 128*128*128; similarly, the second For the second coding block of the part, this layer uses 256 3*3 convolution kernels, the step size is 2, and the number of padding is 0.5. The output size of the first convolution layer is calculated by the calculation formula (3). The output size of a convolutional layer is 64*64*256; since the second part of the convolutional layer uses the same convolution kernel size and format, the same parameters, and so on, the second coding block of the second part finally The output size is 64*64*256;

The third part consists of two decoding blocks, any of which is composed of three improved residual blocks, a deconvolution layer and a linear rectification function; the output size calculation formula of the deconvolution layer is as follows:

Z ₁ =(W ₁ -1)×S ₁ -2×P ₁ +F ₁ (4)

Among them, Z ₁ is the length of the convolution output data, W ₁ is the length of the convolution input data, P ₁ is the number of padding, F ₁ is the length of the convolution kernel, and S ₁ represents the step size; according to the output of the deconvolution layer Size calculation formula (4), this layer uses 256 3*3 convolution kernels, the step size is 2, and the number of padding is 0.5. The output size of the first deconvolution layer in the third part is 128*128*256 ; Since the deconvolution layer in the third part uses the same convolution kernel size and format, the same parameters, and so on, the final output size of the first decoding block in the third part is 128*128*256; similarly, the The second decoding block of the three parts, this layer uses 128 3*3 convolution kernels, the stride is 2, and the number of padding is 0.5. The output size of the first deconvolution layer is calculated by formula (4) The output size of the first deconvolution layer is 256*256*128; since the deconvolution layer of the second decoding block uses the same convolution kernel size and format, the same parameters, and so on, the third part The final output size of the second decoding block is 256*256*128;

The fourth part is the output module, including three improved residual blocks and a convolution layer; this layer uses 64 3*3 convolution kernels, the step size is 1, and the number of padding is 1. The first part of the fourth part The output size of the convolution layer is 256*256*64; since the convolution kernel size and format used in the fourth part of the convolution layer are the same, the parameters are the same, and so on, the final output size of the decoding block in the fourth part is 256* 256*64.

Further, the step C2 is specifically:

The public data set GoPro and the power inspection deblurred data set _A2 are divided into training sets and test sets according to the ratio of 4:1. During the training process, the loss function uses a four-loss weight balance function, and the four-loss weight balance function is defined :

L＝L _p +k ₁ L ₁ +k ₂ L ₂ +k ₃ L ₃ (5)

Among them, L is the four-loss weight balance function, k ₁ , k ₂ and k ₃ are fusion coefficients. In the present invention, k ₁ , k ₂ and k ₃ all take a numerical value of 0.3, L _p is the perceptual loss function, and L ₁ is L1 Norm loss function, L ₂ is the L2 norm loss function, L ₃ is the multi-scale frequency reconstruction loss function, φ _i,j is the j-th convolution after activation in the VGG19 network before the i-th layer maximum pooling. The _feature map _of Mapping the coordinate value in the height direction, n is the batch of scales, I ⁿ is the deblurred image output by the nth scale in this network, S ⁿ is the clear image output by the nth scale in this network, t _k represents the kth scale The total number of elements under , ||·|| ₁ is the L1 distance calculator, ||·|| ₂ is the L2 distance calculator, and F represents the fast Fourier transform (FFT) that transmits the image signal to the frequency domain;

The network optimizer adopts Adam optimizer, the training batch size is 24, and the learning rate is 5×10 ^-5 . The network is fully trained to obtain the trained multi-scale improved residual block convolutional neural network model.

Further, the step S4 includes:

D1: Collect the inspection image data captured during the inspection process of the power inspection robot in real time, and use the trained model obtained in step S3 to deblur the image data collected in real time to obtain the current status of the inspection target;

D2: Deblur the image based on the inspection target obtained in step D1, and perform target detection and recognition of the image. If the inspection target is defective or damaged, the background and cloud will issue an alarm and record the data.

Beneficial effects: Compared with the existing technology, the present invention has the following advantages:

1. The present invention provides an image de-motion blur method based on multi-scale improved residual block CNN, which fully considers the characteristics of blurred images captured by power inspection robots and designs a multi-scale improved residual block convolutional neural network. The comparison of clear and blurred image data is used for feature comparison, which solves the problem of motion blur in inspection images of current power inspection robots. And it has high portability, can be applied to embedded system platforms, and has broad application prospects. The invention has the advantages of fast calculation speed, good motion blur removal effect and strong anti-environmental interference ability.

2. In order to solve the problem of blurred images of the electric power inspection robot due to jitter, the present invention creates a data set that conforms to the electric power inspection scene, and proposes a multi-scale improved residual block convolutional neural network to remove the motion of the electric power inspection image. Using the fuzzy method, a multi-scale improved residual block convolutional neural network model and algorithm are designed. The invention improves the effect and accuracy of deblurring power inspection images, and is suitable for solving the blurring problem of inspection images taken by high-voltage power transmission inspection robots.

Description of the drawings

Figure 1 is a schematic workflow diagram of an image de-motion blur method based on multi-scale improved residual block CNN provided by an embodiment of the present invention;

Figure 2 is an overall block diagram of an inspection system for removing motion blur from power inspection images based on a multi-scale improved residual block convolutional neural network provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of the coordinate system of an inspection robot provided by an embodiment of the present invention;

Figure 4 is a multi-scale improved residual block convolutional neural network model diagram provided by an embodiment of the present invention;

Figure 5 is an overall structural diagram of a power inspection image motion blur removal network based on a multi-scale improved residual block convolutional neural network provided by an embodiment of the present invention;

Figure 6 is an on-site image of a power inspection image with motion blur removed according to an embodiment of the present invention.

Detailed ways

The present invention will be further clarified below in conjunction with the accompanying drawings and specific examples. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, those skilled in the art will be familiar with various aspects of the present invention. Modifications in the form of equivalents fall within the scope defined by the appended claims of this application.

In order to solve the blurring problem of inspection images captured by high-voltage power transmission inspection robots, as shown in Figures 1 and 2, the present invention provides an image de-motion blur method based on multi-scale improved residual block CNN, which includes the following steps:

S1: Collect and save the power inspection video data of the transmission line inspection robot, perform frame processing on the collected power inspection video data, and then perform image sample screening and image preprocessing to obtain clear and blurred image pairs, which are made Data set, and recorded as the source data set;

S2: Complete the image reprocessing of the source data set, crop and compress the source data set, and obtain 133 pairs of clear and blurred images, which are recorded as the power inspection deblurred data set. This data set is unique to the present invention. , At present, there is no deblurred data set in the direction of power inspection. Different from the traditional public data set GoPro production method, the GoPro data set uses a method of merging clear images of more than a dozen consecutive frames into one blurred image. Deblurring of power inspection The data set uses the compression and cropping method of a single blurred image to form a new blurred image;

S3: Build a multi-scale improved residual block convolutional neural network, and use the power inspection deblurred data set to fully train the multi-scale improved residual block convolutional neural network, and obtain the trained multi-scale improved residual block convolution Neural network model. The difference between this model algorithm and the traditional multi-scale convolutional neural network lies in the improvement of the network residual structure and the construction of the loss function;

S4: Use the trained multi-scale improved residual block convolutional neural network model to deblur the power inspection blurred image data, and then perform target detection and recognition on the deblurred image to determine whether the power inspection target is in a defective state, and Record in the background and in the cloud.

In this embodiment, the coordinate system of the power inspection robot is shown in Figure 3. Referring to Figure 3, the power inspection data is collected, image preprocessed and data sets are produced.

Step S1 in this embodiment is specifically:

A1: When the transmission line inspection robot is shaking, collect data from the power inspection video of the inspection robot, and obtain and save the power inspection video data when the transmission line inspection robot is shaking. The power inspection video includes the same scene. clear images and blurred images;

A2: Use the video frame processing method to perform frame processing on the inspection video collected in step A1 to obtain the framed image, and then screen the image samples to obtain images that meet the requirements;

A3: Use the histogram equalization method to preprocess the image collected in step A2 to obtain the preprocessed image data;

A4: Classify and pair the preprocessed data obtained in step A3 to obtain one-to-one corresponding pairs of clear and blurred images, which are recorded as source data sets.

Step A1 in this embodiment includes:

When the transmission line inspection robot is shaking, the data of the power inspection video of the inspection robot is collected. The inspection robot shoots at different inspection line positions, different focal lengths, and different scenes to obtain a series of frame rates of 16 Frame/S short video and save it.

Step A2 in this embodiment is specifically:

The videos collected in step A1 are processed into frames one by one to obtain 16*t images, where t is the time length of the video;

Perform sample screening on the frame-processed images, select images that contain the power inspection target, and discard images that have nothing to do with the power inspection target.

Step A3 in this embodiment is specifically:

The histogram equalization method [please refer to Gonzalez et al. Digital Image Processing [M]. Electronic Industry Press, 2011: 72-85.] is used to preprocess the image obtained in step A2 to enhance the local image. Contrast without affecting the overall contrast; image histogram definition: a digital image with a gray level in the range [0, L-1], the histogram of the image is a discrete function:

p(r _k )＝n _k /n (1)

where p is the probability function, r _k is the k-th gray level, k = 0,1,2,...,L-1, L is the total number of gray levels, n _k is the pixel of the k-th gray level in the image The total number, n is the total number of pixels in the image, and then the preprocessed image is obtained.

Step A4 in this embodiment includes:

Classify the preprocessed images into two categories: clear images and blurred images;

The obtained clear and blurred images are paired in one-to-one correspondence on the condition of the same background and the same inspection target position, and the source data set A ₁ is obtained.

Step S2 in this embodiment is specifically:

B1: Crop the image in the source data set _A1 . The original image resolution is 2688*1520. Use the method of moving the fixed size photo frame to crop the area containing the power inspection target. The resolution and length of the resulting image must be greater than 256 Pixels, the width must be greater than 256 pixels, and the resolution sizes of clear and blurry image pairs must be exactly equal;

B2: Carry out image compression on the image obtained in step B1 to ensure that the image length is greater than 256 pixels and the width is greater than 256 pixels. The resolutions of the clear and blurry image pairs must be completely equal, thus obtaining 133 pairs of clear and blurry image pairs. , and recorded as the power inspection deblurred data set A ₂ .

Step S3 in this embodiment is specifically:

C1: Build a multi-scale improved residual block convolutional neural network and determine the network structure;

Referring to Figures 4 and 5, the structure of the multi-scale improved residual block convolutional neural network consists of three neural network structures of different size scales, and generates a clear image at each scale as a sub-problem of the deblurring task. This sub-problem takes a blurred image and the initial deblurred result (downsampling from the previous scale) as input, and infers that at this scale, this clear image is:

I ^m , h ^m =Net(B ^m ,I ^m+1 ,h ^m+1 ; θ) (2)

Among them, I ^m is the deblurred image output by the layer network at this level of scale, h ^m is the hidden layer feature at this level of scale, Net is the multi-scale improved residual block convolutional neural network proposed by the present invention, and B ^m is The blurred image input by the network of this layer at this level of scale, I ^m+1 is the deblurred image output by the previous level of scale, h ^m+1 is the hidden layer feature of the previous level of scale, θ is the parameter to be trained, and m is the original Level scale, m+1 is the upper level scale.

The neural network structure at any scale is the same. The specific construction method is as follows:

The first part is the input module, including a convolution layer and three improved residual blocks; any one of the improved residual blocks is different from the traditional improved residual block structure. This invention uses splicing operations to replace the original improved residual blocks. The element-by-element addition operation of the block includes two convolutional layers, a linear rectification function and a skip layer splicing; the splicing operation has the following three advantages compared to the original element-wise addition operation of the improved residual block: 1. Information fusion: By connecting two feature maps in the channel dimension, the fusion and combination of information can be achieved, allowing the network to utilize more contextual information. 2. Representation capability enhancement: The connection operation expands the number of channels of the feature map, increases the representation capability of the network, and helps to learn more complex feature representations. 3. Multi-scale feature fusion: Multi-scale feature fusion can be achieved by connecting feature maps with different spatial resolutions, improving the model's perception of targets at different scales. The input of the first part comes from the public data set GoPro and the power inspection deblurred data set _A2 . Each image in the data set is processed by the OpenCV program to obtain a partial image of 256*256 size, and then sent to the input module. The first convolution layer uses 64 3*3 convolution kernels, the step size is 1, and the number of padding is 1. The output size calculation formula of the convolution layer is as follows:

Among them, Z is the length of the convolution output data, W is the length of the convolution input data, P is the number of padding, F is the length of the convolution kernel, and S represents the step size; for the first coding block in the first part, the The output size of a convolution layer is calculated by formula (3) and the output size of the first convolution layer is 256*256*64; since the convolution kernel size and format used in the first part of the convolution layer are the same, the parameters Same, and so on, the final output size of the first part of the convolutional layer and the three improved residual blocks is also 256*256*64;

The second part is a coding structure composed of two coding blocks, where any coding block contains a convolutional layer, a linear rectification function and three improved residual blocks; for the first coding block in the second part, this layer 128 3*3 convolution kernels are selected, the step size is 2, and the padding amount is 0.5. The output size of the first convolution layer is calculated using the formula (3). The calculated output size of the first convolution layer is 128 *128*128; After the first convolutional layer of the coding block, a linear rectification function is used as the activation function, and the data after the activation function is sent to the second convolutional layer of the coding block. The second convolutional layer uses 64 A 3*3 convolution kernel, the step size is 2, and the padding amount is 0.5. According to the output size calculation formula (3) of the convolution layer, the output size of the second convolution layer in the second part is also 128*128*128 ; Since the convolutional layer in the second part uses the same convolution kernel size and format, the same parameters, and so on, the final output size of the first coding block in the second part is 128*128*128; similarly, the second For the second coding block of the part, this layer uses 256 3*3 convolution kernels, the step size is 2, and the number of padding is 0.5. The output size of the first convolution layer is calculated by the calculation formula (3). The output size of a convolutional layer is 64*64*256; since the second part of the convolutional layer uses the same convolution kernel size and format, the same parameters, and so on, the second coding block of the second part finally The output size is 64*64*256;

The third part consists of two decoding blocks, any of which is composed of three improved residual blocks, a deconvolution layer and a linear rectification function; the output size calculation formula of the deconvolution layer is as follows:

Z ₁ =(W ₁ -1)×S ₁ -2×P ₁ +F ₁ (4)

Among them, Z ₁ is the length of the convolution output data, W ₁ is the length of the convolution input data, P ₁ is the number of padding, F ₁ is the length of the convolution kernel, and S ₁ represents the step size; according to the output of the deconvolution layer Size calculation formula (4), this layer uses 256 3*3 convolution kernels, the step size is 2, and the number of padding is 0.5. The output size of the first deconvolution layer in the third part is 128*128*256 ; Since the deconvolution layer in the third part uses the same convolution kernel size and format, the same parameters, and so on, the final output size of the first decoding block in the third part is 128*128*256; similarly, the The second decoding block of the three parts, this layer uses 128 3*3 convolution kernels, the stride is 2, and the number of padding is 0.5. The output size of the first deconvolution layer is calculated by formula (4) The output size of the first deconvolution layer is 256*256*128; since the deconvolution layer of the second decoding block uses the same convolution kernel size and format, the same parameters, and so on, the third part The final output size of the second decoding block is 256*256*128;

The fourth part is the output module, including three improved residual blocks and a convolution layer; this layer uses 64 3*3 convolution kernels, the step size is 1, and the number of padding is 1. The first part of the fourth part The output size of the convolution layer is 256*256*64; since the convolution kernel size and format used in the fourth part of the convolution layer are the same, the parameters are the same, and so on, the final output size of the decoding block in the fourth part is 256* 256*64.

C2: Select the network optimizer and use the public data set GoPro and the power inspection deblurred data set _A2 to train the multi-scale improved residual block convolutional neural network to obtain the trained multi-scale improved residual block convolutional neural network. Model;

Step C2 in this embodiment is specifically:

The public data set GoPro and the power inspection deblurred data set _A2 are divided into training sets and test sets according to the ratio of 4:1. During the training process, the loss function uses a four-loss weight balance function, and the four-loss weight balance function is defined :

L＝L _p +k ₁ L ₁ +k ₂ L ₂ +k ₃ L ₃ (5)

Among them, L is the four-loss weight balance function, k ₁ , k ₂ and k ₃ are fusion coefficients. In the present invention, k ₁ , k ₂ and k ₃ all take a numerical value of 0.3, L _p is the perceptual loss function, and L ₁ is L1 Norm loss function, L ₂ is the L2 norm loss function, L ₃ is the multi-scale frequency reconstruction loss function, φ _i,j is the j-th convolution after activation in the VGG19 network before the i-th layer maximum pooling. The _feature map _of Mapping the coordinate value in the height direction, n is the batch of scales, I ⁿ is the deblurred image output by the nth scale in this network, S ⁿ is the clear image output by the nth scale in this network, t _k represents the kth scale The total number of elements under , ||·|| ₁ is the L1 distance calculator, ||·|| ₂ is the L2 distance calculator, and F represents the fast Fourier transform (FFT) that transmits the image signal to the frequency domain;

In this embodiment, the network optimizer uses the Adam optimizer [Adam optimizer, English name is AdamOptimizer, please refer to Yang Guanci, Yang Jing, Li Shaobo, Hu Jianjun. Improved CNN algorithm based on Dropout and ADAM optimizer [J]. Journal of Huazhong University of Science and Technology (Natural Science Edition), 2018,46(07):122-127.], the training batch size is 24, and the learning rate is 5×10 ^-5 . The network is fully trained to obtain the trained multi-scale improved residual block convolutional neural network model.

Step S4 in this embodiment includes:

D1: Collect the inspection image data captured during the inspection process of the power inspection robot in real time, and use the trained model obtained in step S3 to deblur the image data collected in real time to obtain the current status of the inspection target;

D2: Deblur the image based on the inspection target obtained in step D1, and perform target detection and recognition of the image. If the inspection target is defective or damaged, the background and cloud will issue an alarm and record the data.

In order to verify the effectiveness of the above-mentioned solution of the present invention, this embodiment takes the Tianjin State Grid inspection and acceptance project as an example. As shown in Figure 6, the comparison between the blurred image taken by the robot on site and the deblurred image shows that the area in the red box is Part, the deblurring effect is particularly obvious. This part is a key part of hardware and an important part of the power inspection target. It shows that the multi-scale improved residual block convolutional neural network provided by the present invention has strong adaptability and superiority for power inspection to remove motion blur.

Through the implementation of the above technical solutions, it can be seen that the advantages of the present invention can be summarized as follows:

(1) Provides the data collection method and data preprocessing process of the power inspection robot, including: frame processing, cropping and compression, graphics pairing and histogram equalization.

(2) Provides a method for creating a data set and a method for building a multi-scale improved residual block convolutional neural network, using the difference between pairs of clear and blurred images to complete the deblurring of power inspection images.

(3) An image de-motion blur method based on multi-scale improved residual block CNN is provided. This method has the advantages of fast operation speed, good motion blur removal effect, and strong resistance to environmental interference.

Claims (10)

1. An image motion blur removing method based on a multi-scale improved residual block CNN is characterized by comprising the following steps:

s1: collecting and storing electric power inspection video data of the electric power line inspection robot, framing the collected electric power inspection video data, then screening an image sample, preprocessing an image to obtain a clear and fuzzy image pair, preparing a data set, and marking the data set as a source data set;

s2: clipping and compressing the source data set to obtain a clear and fuzzy image pair, and recording the clear and fuzzy image pair as a power inspection deblurring data set;

s3: building a multi-scale improved residual block convolutional neural network, and fully training the multi-scale improved residual block convolutional neural network by using a power inspection deblurring data set to obtain a trained multi-scale improved residual block convolutional neural network model;

s4: and deblurring the power inspection blurred image data by using a trained multi-scale improved residual block convolutional neural network model.

2. The image motion blur removal method based on the multi-scale improved residual block CNN according to claim 1, wherein the step S1 is specifically:

a1: in the shaking state of the power transmission line inspection robot, acquiring data of an electric power inspection video of the inspection robot to obtain and store electric power inspection video data of the power transmission line inspection robot in the shaking state, wherein the electric power inspection video comprises a clear image and a blurred image in the same scene;

a2: carrying out framing processing operation on the inspection video acquired in the step A1 by adopting a video framing processing method to obtain a frame processed image, and then carrying out image sample screening to obtain an image meeting the requirements;

a3: performing preprocessing operation on the image acquired in the step A2 by adopting a histogram equalization method to obtain preprocessed image data;

a4: and (3) classifying and pairing the preprocessed data obtained in the step (A3) to obtain clear and fuzzy image pairs corresponding to each other one by one, and recording the clear and fuzzy image pairs as a source data set.

3. The image motion blur removal method based on the multi-scale improved residual block CNN according to claim 2, wherein the step A2 is specifically:

carrying out frame-dividing treatment on the videos acquired in the step A1 one by one to obtain 16 x t images, wherein t is the time length of the videos;

and carrying out sample screening on the image subjected to framing treatment, selecting an image containing the power inspection target, and discarding the image irrelevant to the power inspection target.

4. The image motion blur removing method based on the multi-scale improved residual block CNN according to claim 2, wherein the step A3 is specifically:

preprocessing the image obtained in the step A2 by adopting a histogram equalization method, wherein the preprocessing is used for enhancing the local contrast of the image without affecting the overall contrast; image histogram definition: a digital image having a gray level in the range 0, l-1, the histogram of which is a discrete function:

p(r _k )＝n _k /n (1)

where p is a probability function, r _k Is the kth gray level, k=0, 1,2, …, L-1, L is the total number of gray levels, n _k Is the total number of pixels in the image for the kth gray level, n is the total number of pixels in the image, and then a preprocessed image is obtained.

5. The image motion blur removal method based on the multi-scale improved residual block CNN according to claim 1, wherein the step S2 is specifically:

b1: will source data set A ₁ Cutting the image in the image, wherein the resolution of the original image is 2688 x 1520, a fixed-size photo frame moving method is adopted to cut the area containing the power inspection target, the resolution of the obtained image is greater than 256 pixels in length and greater than 256 pixels in width, and the resolution of the clear and blurred image pairs is completely equal;

b2: b1, performing image compression on the image obtained in the step B1, ensuring that the length of the image is greater than 256 pixels, the width of the image is greater than 256 pixels, and the resolution of the clear and fuzzy image pairs are completely equal, so that the clear and fuzzy image pairs are obtained and are recorded as a power inspection deblurring data set A ₂ 。

6. The image motion blur removal method based on the multi-scale improved residual block CNN according to claim 5, wherein the step S3 is specifically:

c1: constructing a multi-scale improved residual block convolutional neural network, and determining a network structure;

c2: selecting a network optimizer and defuzzifying data set A using public data set GoPro and power inspection ₂ Training the multi-scale improved residual block convolutional neural network to obtain a trained multi-scale improved residual block convolutional neural network model.

7. The image deblurring method based on the multi-scale improved residual block CNN according to claim 6, wherein the structure of the multi-scale improved residual block convolutional neural network in step C1 is composed of 3 different-sized-scale neural network structures, a clear image is generated at each scale, and the clear image is taken as a sub-problem of the deblurring task, the sub-problem takes as input a blurred image and an initial deblurring result, and it is presumed that at this scale, the clear image is:

I ^m ，h ^m ＝Net(B ^m ,I ^m+1 ,h ^m+1 ；θ) (2)

wherein I is ^m For the deblurred image output by the layer of network under the scale of the present level, h ^m For the hidden layer characteristics under the present scale, net is the multi-scale improved residual block convolution neural network provided by the invention, B ^m For the fuzzy image input by the layer of network under the scale of the present level, I ^m+1 A deblurred image output for the previous scale, h ^m+1 And (3) as the hidden layer characteristics of the upper-level scale, θ is a parameter to be trained, m is the current-level scale, and m+1 is the upper-level scale.

8. The image motion blur removing method based on the multi-scale improved residual block CNN according to claim 7, wherein the neural network structure at any scale in the step C1 is the same, and the specific construction method is as follows:

the first part is an input module comprising a convolutional layer and three modified residual blocks;

the second part is a coding structure formed by two coding blocks, wherein any one coding block comprises a convolution layer, a linear rectification function and three improved residual blocks;

the third part consists of two decoding blocks, wherein any one decoding block consists of three improved residual blocks, a deconvolution layer and a linear rectification function;

the fourth part is the output module, comprising three modified residual blocks and one convolutional layer.

9. The image deblurring method based on multi-scale improved residual block CNN according to claim 8, wherein the first portion is the inputIn the module, the invention replaces the element-by-element addition operation of the original improved residual block by using the splicing operation, and comprises two convolution layers, a linear rectification function and a layer jump splicing; the first part of input comes from the public data set GoPro and the power inspection deblurring data set A ₂ Each image in the data set is processed by an OpenCV program to obtain a local image with 256 x 256 sizes, the local image is sent to an input module, a first convolution layer is input, the convolution kernels of 64 3*3 are selected, the step size is 1, the filling quantity is 1, and the calculation formula of the output size of the convolution layer is shown as the following formula:

wherein Z is the length of the convolution output data, W is the length of the convolution input data, P is the filling number, F is the length of the convolution kernel, and S represents the step size; for a first code block in the first part, calculating the output size of the first convolution layer by a calculation formula (3) to obtain that the output size of the first convolution layer is 256×256×64; since the convolution kernel adopted by the convolution layer of the first part has the same size and format, the parameters are the same, and so on, the final output sizes of the convolution layer of the first part and the three improved residual blocks are 256×256×64;

in the coding structure of the second part, for the first coding block in the second part, 128 convolution kernels of 3*3 are selected for the layer, the step size is 2, the number of fills is 0.5, the output size of the first convolution layer calculates equation (3), calculating to obtain a first convolution layer the output size of (2) is 128 x 128; using a linear rectification function as an activation function after the first convolution layer of the coding block, feeding the data subjected to the activation function into a second convolution layer of the coding block, the second convolution layer employing 64 3*3 convolution kernels, step length is 2, the filling quantity is 0.5, and according to the output size calculation formula (3) of the convolution layer, second part of the second convolution layer the output size is also 128×128×128; since the convolution kernel employed by the second portion of the convolution layer is the same size and format, the parameters are the same, and so on, the final output size of the first encoded block of the second portion is 128 x 128; similarly, the second coding block of the second part selects 256 convolution kernels of 3*3, the step length is 2, the filling quantity is 0.5, and the output size of the first convolution layer is calculated by a calculation formula (3) to obtain that the output size of the first convolution layer is 64×64×256; because the convolution kernel adopted by the convolution layer of the second part has the same size and format, the parameters are the same, and so on, the final output size of the second coding block of the second part is 64 x 256;

in the two decoding blocks of the third portion, the calculation formula of the output size of the deconvolution layer is shown as the following formula:

Z ₁ ＝(W ₁ -1)×S ₁ -2×P ₁ +F ₁ (4)

wherein Z is ₁ Is the length of the convolution output data, W ₁ Is the length of the convolved input data, P ₁ Is the filling quantity, F ₁ Is the length of the convolution kernel, S ₁ Representing a step size; according to an output size calculation formula (4) of the deconvolution layer, 256 convolution kernels of 3*3 are selected in the deconvolution layer, the step length is 2, the filling quantity is 0.5, and the output size of the first deconvolution layer of the third part is 128×128×256; because the convolution kernel adopted by the deconvolution layer of the third part has the same size and format, the parameters are the same, and so on, the final output size of the first decoding block of the third part is 128 by 256; similarly, the second decoding block of the third part selects 128 convolution kernels of 3*3, the step length is 2, the filling quantity is 0.5, the output size of the first deconvolution layer is calculated by a calculation formula (4), and the output size of the first deconvolution layer is 256×256×128; because the convolution kernel adopted by the deconvolution layer of the second decoding block has the same size and format, the parameters are the same, and so on, the final output size of the second decoding block of the third part is 256×256×128;

the output module of the fourth part selects 64 3*3 convolution kernels, the step length is 1, the filling quantity is 1, and the output size of the first convolution layer of the fourth part is 256×256×64; since the convolution kernel size and format adopted by the convolution layer of the fourth portion are the same, the parameters are the same, and so on, the final output size of the decoding block of the fourth portion is 256×256×64.

10. The image motion blur removal method based on the multi-scale improved residual block CNN according to claim 7, wherein the step C2 is specifically:

deblurring the public dataset GoPro and the Power inspection data set A ₂ Dividing the training set and the testing set according to the ratio of 4:1, wherein in the training process, the loss function uses a four-loss weight balance function, and the four-loss weight balance function is defined as follows:

L＝L _p +k ₁ L ₁ +k ₂ L ₂ +k ₃ L ₃ (5)

wherein L is a four-loss weight balance function, k ₁ ，k ₂ And k is equal to ₃ L is the fusion coefficient _p To perceive the loss function, L ₁ As L1 norm loss function, L ₂ As L2 norm loss function, L ₃ Reconstructing a loss function, phi, for multi-scale frequencies _i,j Is the feature map, W, taken by the jth convolution after activation in VGG19 networks before the ith layer maximum pooling _i,j Is the width of the feature map, H _i,j Is the height of the feature map, S representsA clear image, I represents the generated deblurred image, x is the coordinate value of the width direction of the feature map, y is the coordinate value of the height direction of the feature map, n is the scale batch, I ⁿ S for deblurring the image output by the nth scale in the network ⁿ T is the clear image output by the n-th scale in the network _k Representing the total number of elements at the kth scale, I.I ₁ Is an L1 distance calculator that is used to calculate the distance between the two devices, I.I ₂ Is an L2 distance calculator, F represents a fast fourier transform that transmits an image signal to a frequency domain;

the network optimizer adopts Adam optimizer, training batch size is 24, and learning rate is 5×10 ^-5 And fully training the network to obtain a trained multi-scale improved residual block convolutional neural network model.