CN116645592B - A crack detection method and storage medium based on image processing - Google Patents

Abstract

The present invention relates to the field of image processing. A crack detection method based on deep learning proposed by this invention adopts the Swin Mask RCNN algorithm in crack detection, and it uses Swin Transformer as its backbone network. Compared with previous target detection models, Swin Transformer has better feature extraction capabilities and higher expression capabilities, making features richer and helping to improve the accuracy of crack detection. The network constructed by the present invention is in the Swin Transformer network. After the sliding window operation, a multi-layer window fusion module is constructed to enable windows with crack information to be fused to prevent the loss of crack information, allowing it to better locate crack information in the image, improving position detection accuracy while reducing calculations. quantity.

Description

A crack detection method and storage medium based on image processing

Technical field

The present invention relates to the technical field of image processing, and in particular, to a crack detection method and storage medium based on image processing.

Background technique

Mask RCNN was proposed in October 2017 and is a deep learning model for target detection and instance segmentation. It inherits the two-stage detection framework of Faster RCNN and introduces an additional "Mask Branch" to solve the instance segmentation problem. This mask branch extracts the binary mask of each instance in the target image by adding a convolution and deconvolution network to the RoI pooling layer of Faster RCNN, combining the convolutional neural network and the region convolutional neural network (RCNN). Integrated with the idea of segmentation network. The network structure consists of four parts:

Backbone: A convolutional neural network for feature extraction, usually ResNet or ResNeXt;

Region Proposal Network (RPN): A network used to generate candidate regions, the same as FasterRCNN;

Bounding Box Head: A network for classifying and regressing candidate regions, the same as Faster RCNN;

Mask Head: A fully convolutional network for generating masks for candidate regions.

However, Mask RCNN performs mediocrely in small target detection. The first is that Mask RCNN uses larger grid features, so it is difficult to capture the detailed information of small targets, and the detection performance is naturally reduced; in addition, in practical applications, there is almost no monotonous background and the only small target that exists, and the collected Images generally have multiple objects. MaskRCNN's classifier adapts well to large targets. Under the same parameter threshold, it is difficult to detect multiple targets of the same type but different sizes; and when there are multiple large targets in a picture with multiple small target sets, Mask RCNN cannot accurately segment, and such pictures are precisely the parts that need to be detected most in crack detection. Therefore, although Mask RCNN has better detection results on large targets, it is difficult to show its advantages in the field of crack detection.

Swin Transformer was proposed by Liu Ze and others from Microsoft Research Asia in March 2021. It uses hierarchical structure, sliding window, and linear calculation respectively to solve the two major challenges of Transformer application in the image field: scale changes of visual entities and images. High computational complexity at high resolution and achieves excellent performance on multiple vision tasks. (Remove "includes image classification (reaching 87.3% top-1 accuracy on ImageNet-1K), object detection and instance segmentation (reaching 58.7% box AP and 51.1% mask AP on COCOtest-dev), semantic segmentation (Achieving 53.5% mIoU on ADE20K val) etc.")

Swin mask RCNN is based on Swin Transformer and uses the Mask RCNN framework to implement target detection and instance segmentation. By using Swin Transformer as the backbone network, the convolutional layer in Mask RCNN is replaced, and moving windows of different sizes and hierarchical structures are used to solve the challenges encountered in the conversion from language to vision, such as scale changes of visual entities and The high resolution of the image pixels.

However, the Swin Transformer sliding window operation may cause small targets to be divided into multiple windows, affecting feature extraction and positioning accuracy, and the hierarchical structure may cause small targets to be lost or blurred in high-level feature maps, affecting classification and segmentation quality. In addition, position encoding is the default setting and is not sensitive to small target detection such as cracks. This also makes SwinTransformer perform poorly in small target detection such as crack detection.

Contents of the invention

To this end, the technical problem to be solved by the present invention is to overcome the existing technology in which Mask RCNN performs poorly in small target detection and the Swin Transformer sliding window operation may cause small targets to be divided into multiple windows, affecting feature extraction and positioning accuracy. Moreover, the hierarchical structure may cause small targets to be lost or blurred in high-level feature maps, affecting the quality of classification and segmentation. In addition, position encoding is the default setting and is not sensitive to small target detection such as cracks. This also makes Swin Transformer perform poorly in small target detection such as crack detection.

In order to solve the above technical problems, the present invention provides a crack detection method based on image processing, including:

S101: Obtain the crack image to be detected;

S102: Use the constructed improved Swin Transformer network as the backbone network to extract the features of the crack image to be detected, and generate a series of candidate areas, including:

S201: Use the built multi-head self-attention mechanism module to calculate the crack image to be detected and output a feature map, including:

Divide the crack image to be detected into multiple n*n small panes according to the same window size to obtain the feature map after the first division, and use the built Shift Window Attention mechanism module to The small pane in the feature map performs window sliding to obtain the feature map after the second division. The feature map after the first division and the feature map after the second division are input into the multi-layer window fusion module constructed. The layer window fusion module normalizes the feature map after the first division and the feature map after the second division, and converts each small window of the normalized feature map after the first division into Each small pane in the feature map after the second division performs feature mapping and calculates a similarity matrix. The small panes in the feature map after the first division are the same as the small panes in the feature map after the second division. The panes have overlapping parts. According to the obtained similarity matrix, it is judged whether the small panes with overlapping parts need to be fused. If the similarity matrix indicates that there is connected crack information, the two small panes are fused to generate a fusion window output. , if the similarity matrix indicates that there is no connected crack information, the two small panes will not be fused and will be output as two independent windows respectively, obtaining multiple fusion windows and multiple independent windows output by the multi-layer window fusion module;

Use the constructed transformation matrix to calculate each window output by the multi-layer window fusion module to obtain the query value Q, key value K, and value V in the self-attention mechanism of each window, and calculate the query value Q, key value K, and value V in the self-attention mechanism of each window. First, a convolution layer is used to linearly transform the pixels in each window to extract pixel data, and the features are extracted based on the query value Q, key value K, and value V of each window, specifically:

in, is the number of columns of the Q, K matrix, that is, the vector dimension, B represents the position offset of each window, and the value of B is given by the set relative position offset parameter table;

Use the relative position index constructed in the multi-head self-attention mechanism module to retrieve the parameters in the relative position encoding table and calculate and fuse the features of each window to output the feature map output by the multi-head self-attention calculation;

S202: Use the constructed feature pyramid network to process the feature map output by multi-head self-attention calculation, extract features of different scales in the feature map output by multi-head self-attention calculation and fuse them, and output the fused feature map. ;

S203: Use the constructed Region Proposal Network to predict the fused feature map and obtain a series of candidate regions. Each candidate region contains a score and a position offset. Each candidate region is sorted and summed according to the score. Screen and retain some high-scoring candidate areas;

S103: Classify and regress each obtained high-scoring candidate area, obtain the final detection frame based on the position offset of each high-scoring candidate area, and predict the pixel-level mask in each detection frame, and output the band There are detection results of target detection frames and image segmentation.

Further, when the built Shift Window Attention mechanism module performs window sliding on the small panes in the feature map after the first division to obtain the window sliding on the small panes in the feature map after the second division, when the When the small pane in the feature map after the first division slides along the horizontal or vertical direction, the sliding distance of the window is less than the side length of the small pane. When the small pane in the feature map after the first division slides along the When the small pane slides diagonally, the sliding distance of the window is smaller than the diagonal distance of the small pane.

Further, using the constructed feature pyramid network to process the feature map output by multi-head self-attention calculation includes:

Perform a convolution operation on the input feature map to obtain multiple feature maps of different levels;

Construct a top-down path and lateral connections to build a feature pyramid. Each pyramid layer contains an upsampling operation and a 1x1 convolution operation to compare the feature map of the previous layer with the feature map of the current layer. Add and perform feature fusion;

Use a 3x3 convolution operation to smooth each pyramid layer, and output the smoothed feature map of each pyramid layer.

Further, the size and number of channels of each feature map in the multiple different levels of feature maps are different.

Further, each candidate area is classified and regressed, and the final detection frame is obtained according to the position offset of each candidate area, and the pixel-level mask in each detection frame is predicted, and the target detection frame is output. And the detection results of image segmentation include:

Use the built RoIAlign layer to pool the candidate regions to the same size;

Output a fixed-size feature vector of the candidate region, representing the characteristics of each candidate region;

Classify and regress the feature vector of each candidate area, and output a category label and a position offset, representing the final detection frame of each RoI;

Perform a masking operation on the feature vector in the final detection frame of each RoI to predict the pixel-level mask in each detection frame, output a binary matrix representing the foreground and background pixels in each detection frame, and compare it with the original image Combined output is a detection result with target detection frame and image segmentation.

Further, the RoIAlign layer uses a bilinear interpolation method to pool the candidate regions.

Further, the classification and regression of the feature vector of each candidate region is specifically: using a fully connected layer to integrate and regress the features of each candidate region, and using a softmax layer to classify the features of each candidate region.

Further, the mask operation is performed on the feature vector in the final detection frame of each candidate area to predict the pixel-level mask in each detection frame. Specifically, a convolution layer is used to perform a mask operation on the feature map in the detection frame. Feature extraction, and a sigmoid activation function is used to binarize the extracted features and output a binary matrix.

Further, the convolution layer in using a convolution layer to extract features from the feature map within the detection frame includes a constructed horizontal edge convolution kernel and a longitudinal edge convolution kernel.

The present invention also provides a storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the above-mentioned image processing-based crack detection method are implemented.

The above technical solution of the present invention has the following advantages compared with the existing technology:

The image processing-based crack detection method and storage medium of the present invention innovatively apply the Swinmask RCNN model to the field of crack detection, and optimize the Swinmask RCNN model parameters and model structure. In order to better extract cracks, we built a multi-layer window fusion module after the Swin Transformer network sliding window operation, so that small windows with crack characteristics can be fused with each other and retain complete crack information. In the model feature extraction process, we constructed a transverse edge convolution kernel and a longitudinal edge convolution kernel, which can better extract information about elongated cracks; while improving the position detection accuracy, the amount of calculation is reduced.

Description of the drawings

In order to make the content of the present invention easier to understand clearly, the present invention will be further described in detail below based on specific embodiments of the present invention and in conjunction with the accompanying drawings, wherein

Figure 1 is a flow chart of a crack detection method based on image processing according to an embodiment of the present invention;

Figure 2 is an algorithm workflow diagram of a crack detection method based on image processing constructed by the present invention;

Figure 3 is a schematic diagram of the sliding window operation in the Swin Transformer network;

Figure 4 is a schematic diagram of the working of the multi-layer window fusion module;

Figure 5 is a working principle diagram of the multi-layer window fusion module;

Figure 6 is the structure diagram of the self-attention mechanism Self-Attention;

Figure 7 is a schematic diagram of the relative position coding table constructed by the present invention;

Figure 8 is a schematic diagram of the working of the self-attention mechanism at both ends;

Figure 9 is a schematic network diagram of the feature pyramid;

Figure 10 is the network structure diagram of the feature pyramid;

Figure 11 is the structure diagram of the Swin Transformer network;

Figure 12 is a comparison chart of the first crack original image and the crack detection results processed by the MaskRCNN network and a crack detection method based on image processing provided by the present invention;

Figure 13 is a comparison chart of the second original crack image and the crack detection results processed by the MaskRCNN network and a crack detection method based on image processing provided by the present invention;

Figure 14 is a comparison diagram of the third original crack image and the crack detection results processed by the MaskRCNN network and a crack detection method based on image processing provided by the present invention;

Figure 15 is a comparison chart of the fourth crack original image and the crack detection results processed by the MaskRCNN network and a crack detection method based on image processing provided by the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific examples, so that those skilled in the art can better understand and implement the present invention, but the examples are not intended to limit the present invention.

Embodiment 1

Referring to Figure 1, the steps of Embodiment 1 of the present invention include:

S101: Obtain the crack image to be detected;

S102: Use the constructed improved Swin Transformer network as the backbone network to extract the features of the crack image to be detected, and generate a series of candidate areas, including:

S201: Use the built multi-head self-attention mechanism module to calculate the crack image to be detected and output a feature map, including:

Divide the crack image to be detected into multiple n*n small panes according to the same window size to obtain the feature map after the first division, and use the built Shift Window Attention mechanism module to The small pane in the feature map performs window sliding to obtain the feature map after the second division. The feature map after the first division and the feature map after the second division are input into the multi-layer window fusion module constructed. The layer window fusion module normalizes the feature map after the first division and the feature map after the second division, and converts each small window of the normalized feature map after the first division into Each small pane in the feature map after the second division performs feature mapping and calculates a similarity matrix. The small panes in the feature map after the first division are the same as the small panes in the feature map after the second division. The panes have overlapping parts. According to the obtained similarity matrix, it is judged whether the small panes with overlapping parts need to be fused. If the similarity matrix indicates that there is connected crack information, the two small panes are fused to generate a fusion window output. , if the similarity matrix indicates that there is no connected crack information, the two small panes will not be fused and will be output as two independent windows respectively, obtaining multiple fusion windows and multiple independent windows output by the multi-layer window fusion module;

Use the constructed transformation matrix to calculate each window output by the multi-layer window fusion module to obtain the query value Q, key value K, and value V in the self-attention mechanism of each window, and calculate the query value Q, key value K, and value V in the self-attention mechanism of each window. First, a convolution layer is used to linearly transform the pixels in each window to extract pixel data, and the features are extracted based on the query value Q, key value K, and value V of each window, specifically:

,

in, is the number of columns of the Q, K matrix, that is, the vector dimension, B represents the position offset of each window, and the value of B is given by the set relative position offset parameter table;

Use the relative position index constructed in the multi-head self-attention mechanism module to retrieve the parameters in the relative position encoding table and calculate and fuse the features of each window to output the feature map output by the multi-head self-attention calculation;

S202: Use the constructed feature pyramid network to process the feature map output by multi-head self-attention calculation, extract features of different scales in the feature map output by multi-head self-attention calculation and fuse them, and output the fused feature map. ;

S203: Use the constructed Region Proposal Network to predict the fused feature map and obtain a series of candidate regions. Each candidate region contains a score and a position offset. Each candidate region is sorted and summed according to the score. Screen and retain some high-scoring candidate areas;

S103: Classify and regress each obtained high-scoring candidate area, obtain the final detection frame based on the position offset of each high-scoring candidate area, and predict the pixel-level mask in each detection frame, and output the band There are detection results of target detection frames and image segmentation.

A crack detection method based on image processing proposed by the present invention adopts the Swin MaskRCNN algorithm in crack detection and uses Swin Transformer as its backbone network. Compared with previous target detection models, SwinTransformer has better feature extraction capabilities and higher expression capabilities, making features richer and helping to improve the accuracy of crack detection. In order to better extract the cracks, we built a multi-layer window fusion module after the sliding window operation of the Swin Transformer network, so that the small panes with crack characteristics can be fused with each other, and further the information between the small panes can be transferred and retained. complete crack information.

Embodiment 2

The algorithm workflow diagram of Embodiment 2 of the present invention is shown in Figure 2. The collected road crack, tunnel crack and bridge crack data are trained based on the Swin mask RCNN deep learning network structure. The input image is a scaled and manually annotated crack. The prediction is mainly divided into two stages, including:

The first stage: Use Swin Transformer as the backbone network to extract the features of the image and generate a series of candidate areas. First, the input image is divided into multiple small patches, called patches, and each patch is converted into a feature vector; in the second step, self-attention is calculated for each patch. In this step, sliding windows and layering are used structure, and during the attention calculation process, this embodiment improves the position encoding matrix to improve the model's detection efficiency and accuracy of cracks; then, multiple scale feature maps are output, each feature map contains different sizes and resolution patches.

Candidate region generation: On the feature map of the last scale, a Region ProposalNetwork (RPN) is used to predict a series of candidate regions, called Regions of Interest (RoI); each RoI contains a score and a position bias. Shift, indicating the confidence and location of the region. RoIs are sorted and filtered according to their scores, and a certain number of high-scoring RoIs are retained as input to the second stage.

The second stage: Classify and regress each candidate area to obtain the final detection frame, and predict the pixel-level mask within each detection frame. First, candidate area alignment is performed, and each RoI is aligned on feature maps of different scales. An RoI Align layer is used to maintain the resolution and position accuracy of the feature map; then a fixed-size feature vector is output, representing Characteristics of each RoI. The next step is to predict the detection frame. Classification and regression operations are performed on the feature vector of each RoI, using a fully connected layer and a softmax layer to predict the category and position offset of each RoI. It then outputs a class label and a position offset representing the final detection box for each RoI. Finally, mask prediction is performed. The feature vector of each RoI is masked, using a convolutional layer improved for cracks and a sigmoid layer to predict the pixel-level mask in each detection frame, and then output a binary matrix representing each The foreground and background pixels within the frame are detected and finally output.

The specific steps of the crack detection method based on image processing provided in Embodiment 2 of the present invention include:

S31: Obtain the crack image to be detected and the annotated crack image;

S32: Use Swin Transformer as the backbone network to extract image features and generate a series of candidate areas, including:

S321: Crop the entire picture, embed the vector, set the crop size to 4*4 pixels, set the output channel after cropping to determine the size of the embedded vector, and finally expand the H and W dimensions and move them to the first dimension; input The picture is (960, 960, 3) (RGB three channels). After cropping, it is (240, 240, 96). 96 is the channel output specified by the model. 240 comes from 960/4=240. The window is set to 7* 7. Obtain the segmented feature map, that is, the feature map after the first division; use the built Shift Window Attention mechanism module to slide the small pane in the feature map after the first division, with the window position downward, Move two units to the right to obtain the feature map after the second division;

In step S321, sliding window operation (Shifted Window Attention): As shown in Figure 3, the standard Transformer architecture and its adapted version for image classification both perform global self-attention, which calculates the relationship between each token and all other tokens. relationship (Attention Map). But Swin Transformer is different, it adopts a sliding window (Shift Window Attention) design.

S322: As shown in Figure 4, input the feature map after the first division and the feature map after the second division into the multi-window fusion module (Multi-window confluence, referred to as MWC). The multi-layer window fusion module is a purely mathematical calculation module added after the original sliding window module. It does not add network parameters for training and does not increase the amount of network calculation. Its basic principle is as follows:

As shown in Figure 5: MWC takes the output fw (feature map after the first division) and fsw (feature map after the second division) of W-MSA and SW-MSA as input. First, the feature map is normalized along the channel direction through softmax to ensure that the vector modulus length is equal to 1. Specific operation: first perform the softmax operation, the formula is as follows , e represents the base of the natural logarithm, that is, the constant 2.71828, /> is the output value of the i-th node, and C is the number of output nodes. Get the normalized eigenvector, then divide it by its modulus length to ensure that the vector modulus length is equal to 1.

Then, each small pane of the first divided feature map after normalization and each small pane of the second divided feature map are processed, using patch as a vector, the two features are Dot multiplication is performed on the same position of the map, both inputs are (240, 240, 96), dot multiplication is performed on the corresponding position, and finally a feature map with the same size as the original feature map is obtained, and the similarity matrix of fw and fsw is obtained by addition. The similarity matrix is the characteristic matrix of small panes. The small panes in W-MSA and SW-MSA have overlapping parts. If the elements in the overlapping parts of the two small panes in the matrix have some crack characteristic parameters, then this If there is possible crack information between two small panes with overlapping parts, the two small panes will be fused. If the elements in the overlapping part of the two small panes in the matrix do not have some characteristic parameters of cracks, the process will not be performed. Fusion.

Finally, when the small panes in the feature map after the first division are merged with the small panes in the feature map after the second division, fw and fsw are obtained by soft pooling and mathematical addition transformation. The feature map is multiplied with the similarity matrix to obtain the feature window fc with contextual information. This allows multiple windows to be combined and prevents information from being lost on slender targets like cracks. The feature map obtained by soft pooling and mathematical addition transformation of the small window is multiplied by the similarity matrix. The soft pooling operation uses the softmax function to calculate the weight in each pooling area, thereby performing a weighted average of the input features. Since the actual image is larger, the specific operation is explained as follows: Assume there is a feature map with an input feature map size of 4x4 and a channel number of 1. Use a 2x2 pooling window for soft pooling operations. The input is a matrix (1 2 3 4, 5 6 7 8, 910 11 12, 13 14 15 16). For each 2x2 pooling area, calculate the weight obtained by the softmax operation, and then weight the features in the pooling area. average. For the first pooling area: (1 2, 5 6), through the softmax operation, the calculated weight is: (0.0474 0.0474, 0.9526 0.9526), and the weighted average of the features in the pooling area is: (1*0.0474 + 2*0.0474 + 5*0.9526 + 6*0.9526) = 5.9992. For other pooling areas, softmax operation and weighted average are also performed. The final pooled feature map is: (5.9992 7.9992, 13.999215.9992). In this example, the size of the original feature map is 4x4. After a 2x2 soft pooling operation, a 2x2 pooled feature map is obtained. The mathematical addition transformation is to add the corresponding positions of fw and fsw.

S323: Use the constructed transformation matrix to calculate each window output by the multi-layer window fusion module to obtain the query value Q, key value K, and value V in the self-attention mechanism of each window, and obtain each window based on The query value Q, key value K, and value V are used to extract features, and the relative position index constructed in the multi-head self-attention mechanism module is used to retrieve the parameters in the relative position encoding table and the features of each window are calculated and fused to output the multi-head self-attention Feature map of force calculation output;

The multi-head attention mechanism constructed by this invention is composed of multiple self-attention mechanisms (Self-Attention). Figure 6 is a structural diagram of a single self-attention mechanism:

The matrices Q (query), K (key value), and V (value) need to be used during calculation. In practice, Self-Attention receives the input (matrix X composed of word representation vectors x) or the output of the previous encoding module. Q, K, and V are obtained by linear transformation of the input of Self-Attention. The input of Self-Attention is represented by the matrix X, and Q, K, V can be calculated using the linear transformation matrices WQ, WK, and WV. in, is the number of columns of the Q and K matrices, that is, the vector dimension. The specific formula is:/> ;

After obtaining the matrices Q, K, V, the output of Self-Attention can be calculated according to the above formula. The specific calculation process is as follows:

get After that, use Softmax to calculate the attention coefficient of each vector for other vectors. Softmax in the formula is to perform Softmax on each row of the matrix, that is, the sum of each row becomes 1. After obtaining the Softmax matrix, it can be multiplied by V to obtain the final Output.

As shown in Figure 7, the relative position coding table example of the present invention includes two parts: a relative position index, and a relative position coding table. The relative position coding table is constructed by: establishing a relative position coding table, and initializing the data in the table. , taking the annotated crack images as sample data, using the detection accuracy of the sample data as a standard, using machine learning methods to train the parameters in the relative position coding table, and obtaining the relative position coding table. The calculation formula is as follows: ;

The sign(x) function is used to return the sign of x. If x is a positive number, sign(x) returns 1; if x is a negative number, sign(x) returns -1; if x is equal to 0, sign(x) returns 0, and the same goes for y. ,/> Represents the x and y deviation values.

As shown in Figure 8: Taking the two-headed attention as an example, the first self-attention part calculates b(i,1), and similarly, b(i,2) can be obtained. Calculate the output bi,/> is the transformation matrix. In the figure, ei is the position encoding.

S324: Use the constructed feature pyramid network to process the feature map output by self-attention calculation, extract features of different scales in the feature map and fuse them, and output the fused feature map;

The network schematic diagram of the feature pyramid constructed by the embodiment of the present invention is shown in Figure 9:

Feature Pyramid Network (FPN for short) is a general network structure used for multi-scale target detection. FPN was originally proposed by Facebook AI Research in 2017. Its goal is to use feature maps at different scales to detect targets of different sizes.

FPN uses two different levels of features: one is the bottom shallow level, which has high resolution but little semantics; the other is the high-level deep level, which has low resolution but has more semantics. They are combined to form multi-scale feature maps that can detect targets at different scales. FPN has two core steps: the first step is to build a pyramid-like structure from the bottom up to generate initial feature maps of different resolutions; the second step is to upsample, fuse and combine the initial feature maps from the top down. Adjust to get better feature representation. Specifically, FPN transforms the bottom feature map to the same resolution as the upper feature map through upsampling, and then performs a weighted summation of them to obtain a richer feature expression.

The specific network structure diagram of the feature pyramid is shown in Figure 10:

A backbone network is used to perform a convolution operation on the input image. The input has 3 channels and multiple convolution kernels. For each convolution kernel, the three input channels are first convolved separately, and then the results of the three channels are added to obtain the convolution output. Multiple feature maps of different levels are obtained. The size and number of channels of each feature map are the same. different.

Then, a top-down path and lateral connections are used to build a feature pyramid. Each pyramid layer contains an upsampling operation and a 1x1 convolution operation to combine the feature map of the previous layer with the features of the current layer. Graph summation, thus merging high-level and low-level information. Here the upsampling operation uses bilinear interpolation, which is a method of calculating the value of each pixel in a high-resolution image based on linear interpolation in two directions. Suppose there is a low-resolution image of size mxn that needs to be upsampled into a high-resolution image of size M x N. For each pixel position in the high-resolution image , its corresponding value needs to be calculated. First, the corresponding position in the low-resolution image is calculated . The specific calculation method is as follows:/> .

Then, find the four closest pixels to (x, y) in the low-resolution image, which are ). Among them,/> and/> is the largest integer not larger than x and y,/> and is the smallest integer not less than x and y. Next, calculate the position in the high-resolution image by the following formula /> pixel value:

in,

,/> Indicates the position in the low-resolution image/> pixel value. With the above calculations, a low-resolution image can be upsampled to a high-resolution image using linear interpolation.

Finally, a 3x3 convolution operation is used to smooth each pyramid layer so that each pyramid layer has the same number of channels and eliminates the aliasing effect caused by upsampling. Using the FPN target detection model can obtain better multi-scale feature maps to detect targets of different sizes, which can effectively improve the target detection accuracy.

S325: Use the constructed Region Proposal Network (RPN) layer to predict the fused feature map and obtain a series of candidate regions.

S33: Classify and regress each candidate area to obtain the final detection frame, and predict the pixel-level mask within each detection frame, including:

S331: Use the built RoIAlign layer to pool the candidate areas. The RoIAlign layer uses bilinear interpolation to pool the candidate areas to the same size. Average pooling is used here. The specific operation is as follows: slide on the input feature map A fixed size 3*3 pooling window; for each pooling window, calculate the average of all values in the window; use the average as the value of the corresponding position in the output feature map; and maintain the resolution and position accuracy of the feature map , the output Rol is the completed candidate area, and a fixed-size feature vector is output to represent the characteristics of Rol.

In the step S331, RoIAlign is used to accurately align the characteristics of the RoI on the feature map; the RoIAlign layer divides the RoI area into small grids of fixed size, and uses the bilinear interpolation method to calculate the characteristics of each small grid on the feature map. Pixel value; the feature map corresponding to RoI is obtained.

S332: Classify and regress the feature vector of each RoI, using a fully connected layer and a softmax layer. The fully connected layer integrates and regresses the features of each candidate region, and the softmax layer integrates the features of each candidate region. Classification, to predict the category and position offset of each Rol. Then output a category label and a position offset, representing the final detection box of each Rol;

S333: Perform mask prediction. The feature vector of each Rol is masked, and a convolution layer improved for cracks and a sigmoid layer are used. The convolution layer extracts features from the feature map in the detection frame. The convolution layer is equipped with a constructed lateral The edge convolution kernel and the longitudinal edge convolution kernel are used to further extract the details of the crack image. The sigmoid activation function binarizes the extracted features and outputs a binary matrix composed of 0 and 1, where 0 and 1 1 respectively represents the two states or categories in the matrix to predict the pixel-level mask in each detection frame, and the binary matrix represents the foreground and background pixels in each detection frame. Finally, it is combined with the original image to output an image segmentation detection result with target detection frame and sum.

The Swin Transformer network structure constructed by this invention is shown in Figure 11:

The Swin Transformer network uses a window translation-based method to model and process visual data. The whole system consists of multi-head self-attention (W-MSA), shifted window multi-head self-attention (SW-MSA) and multi-layer perceptron (MLP). Inserting layernorm (LN) layers in the middle can make the training more stable, and use residual connections after each module. The specific formula is:

The core idea of Swin Transformer is to use a hierarchical window mechanism to model images. Each window processes a part of the image, and then integrates the results of window processing through a Transformer-based method. Swin Transformer divides the image into multiple large windows, and each window is regarded as a feature map. Images of different sizes produce a set of multi-resolution feature maps after being assigned to different numbers of windows, thereby reducing time complexity. , and enhance the feature extraction capability of the model. At the same time, Swin Transformer uses cross-window local convolution to extract features from non-local fields of view, and introduces absolute position encoding and relative position encoding within the window, allowing the model to establish position correlation in the coordinate system when processing images with spatial information. Relationship.

The present invention uses the "Pytorch" framework based on Python to build a network algorithm model based on Swin Mask RCNN, and selects CUDA and MMCV-compatible versions of mmdetection and COCO related toolkits to build a virtual environment suitable for training the model. According to the number and size of images, Select an appropriate weight file and divide the 2000 images into a training set, a test set, and a verification set in a ratio of 6:2:2. The crack location, specific outline, label information, etc. are manually marked. After modifying the model parameters according to the image size, number of detection categories, image training and quantity, the training set is put into the pre-training model for training. After 700 rounds of training, the trained weight file is obtained. Use Python to write a program to call the trained weight file. Test the test set, that is, input a picture and give a test result. The model mAP is about 41%. The comparison results between the crack detection method based on image processing proposed by the present invention and the Mask RCNN network extraction are as shown in Figure 12, Figure 13. As shown in Figures 14 and 15, Figures 12 to 15 respectively show the original images of four different cracks and the crack detection results processed by the MaskRCNN network and a crack detection method based on image processing provided by the present invention. Comparison chart; the left image is the original image of each type of crack, the middle image is the crack detection result image extracted through the Mask RCNN network, and the right image is the crack detection result image processed by a crack detection method based on image processing of the present invention. After comparison It can be seen that the method proposed by the present invention has more obvious advantages in terms of detection accuracy and detail detection of images.

Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and flowcharts of methods and computer program products according to embodiments of the present application. It will be understood that each process in the flowchart illustrations can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device used to implement the functions specified in one process or multiple processes of the flow chart.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing a function specified in a flowchart process or processes.

Claims (10)

1. A crack detection method based on image processing, characterized by: including: S101: Obtain the crack image to be detected; S102: Use the constructed improved Swin Transformer network as the backbone network to extract the features of the crack image to be detected, and generate a series of candidate areas, including: S201: Use the built multi-head self-attention mechanism module to calculate the crack image to be detected and output a feature map, including: Divide the crack image to be detected into multiple n*n small panes according to the same window size to obtain the feature map after the first division, and use the built Shift Window Attention mechanism module to The small pane in the feature map performs window sliding to obtain the feature map after the second division. The feature map after the first division and the feature map after the second division are input into the multi-layer window fusion module constructed. The layer window fusion module normalizes the feature map after the first division and the feature map after the second division, and converts each small window of the normalized feature map after the first division into Each small pane in the feature map after the second division performs feature mapping and calculates a similarity matrix. The small panes in the feature map after the first division are the same as the small panes in the feature map after the second division. The panes have overlapping parts. According to the obtained similarity matrix, it is judged whether the small panes with overlapping parts need to be fused. If the similarity matrix indicates that there is connected crack information, the two small panes are fused to generate a fusion window output. , if the similarity matrix indicates that there is no connected crack information, the two small panes will not be fused and will be output as two independent windows respectively, obtaining multiple fusion windows and multiple independent windows output by the multi-layer window fusion module; Use the constructed transformation matrix to calculate each window output by the multi-layer window fusion module to obtain the query value Q, key value K, and value V in the self-attention mechanism of each window, and calculate the query value Q, key value K, and value V in the self-attention mechanism of each window. First, a convolution layer is used to linearly transform the pixels in each window to extract pixel data, and the features are extracted based on the query value Q, key value K, and value V of each window, specifically: ; in, is the number of columns of the Q, K matrix, that is, the vector dimension, B represents the position offset of each window, and the value of B is given by the set relative position offset parameter table; Use the relative position index constructed in the multi-head self-attention mechanism module to retrieve the parameters in the relative position encoding table and calculate and fuse the features of each window to output the feature map output by the multi-head self-attention calculation; S202: Use the constructed feature pyramid network to process the feature map output by multi-head self-attention calculation, extract features of different scales in the feature map output by multi-head self-attention calculation and fuse them, and output the fused feature map. ; S203: Use the constructed Region Proposal Network to predict the fused feature map and obtain a series of candidate regions. Each candidate region contains a score and a position offset. Each candidate region is sorted and summed according to the score. Screen and retain some high-scoring candidate areas; S103: Classify and regress each obtained high-scoring candidate area, obtain the final detection frame based on the position offset of each high-scoring candidate area, and predict the pixel-level mask in each detection frame, and output the band There are detection results of target detection frames and image segmentation. 2. A crack detection method based on image processing according to claim 1, characterized in that: the constructed Shift Window Attention mechanism module performs window sliding on the small pane in the feature map after the first division. When the small pane in the feature map obtained for the second division performs window sliding, when the small pane in the feature map after the first division performs window sliding along the horizontal or vertical direction, the sliding distance of the window is smaller than that of the small window. The side length of the grid, when the small pane in the feature map after the first division slides along the diagonal line of the small pane, the sliding distance of the window is smaller than the diagonal distance of the small pane. 3. A crack detection method based on image processing according to claim 1, characterized in that: using the constructed feature pyramid network to process the feature map output by multi-head self-attention calculation includes: Perform a convolution operation on the input feature map to obtain multiple feature maps of different levels; Construct a top-down path and lateral connections to build a feature pyramid. Each pyramid layer contains an upsampling operation and a 1x1 convolution operation to compare the feature map of the previous layer with the feature map of the current layer. Add and perform feature fusion; Use a 3x3 convolution operation to smooth each pyramid layer, and output the smoothed feature map of each pyramid layer. 4. A crack detection method based on image processing according to claim 3, characterized in that: the size and number of channels of each feature map in the multiple different levels of feature maps are different. 5. A crack detection method based on image processing according to claim 1, characterized in that: classifying and regressing each candidate area, and obtaining the final detection frame according to the position offset of each candidate area, and Predict the pixel-level mask within each detection frame, and output the detection results with target detection frame and image segmentation including: Use the built RoIAlign layer to pool the candidate regions to the same size; Output a fixed-size feature vector of the candidate region, representing the characteristics of each candidate region; Classify and regress the feature vector of each candidate area, and output a category label and a position offset, representing the final detection frame of each RoI; Perform a masking operation on the feature vector in the final detection frame of each RoI to predict the pixel-level mask in each detection frame, output a binary matrix representing the foreground and background pixels in each detection frame, and compare it with the original image Combined output is a detection result with target detection frame and image segmentation. 6. A crack detection method based on image processing according to claim 5, characterized in that: the RoIAlign layer uses a bilinear interpolation method to pool the candidate areas. 7. A crack detection method based on image processing according to claim 5, characterized in that: the classification and regression of the feature vector of each candidate area is specifically: using a fully connected layer to classify each candidate area The features are integrated and regressed, and a softmax layer is used to classify the features of each candidate area. 8. A crack detection method based on image processing according to claim 5, characterized in that: the feature vector in the final detection frame of each candidate area is masked to predict the feature vector in each detection frame. Specifically, the pixel-level mask uses a convolutional layer to extract features from the feature map within the detection frame, and uses a sigmoid activation function to binarize the extracted features and output a binary matrix. 9. A crack detection method based on image processing according to claim 8, characterized in that: using a convolution layer to extract features from the feature map in the detection frame, the convolution layer includes constructed lateral edges. Convolution kernel and longitudinal edge convolution kernel. 10. A storage medium, characterized in that: a computer program is stored on the storage medium, and when the computer program is executed by a processor, the crack based on image processing according to any one of claims 1 to 9 is implemented. Steps of the detection method.