CN111127493A

CN111127493A - Remote sensing image semantic segmentation method based on attention multi-scale feature fusion

Info

Publication number: CN111127493A
Application number: CN201911098961.XA
Authority: CN
Inventors: 周勇; 何欣; 赵佳琦; 夏士雄; 张迪; 姚睿; 刘兵; 杜文亮
Original assignee: China University of Mining and Technology CUMT
Current assignee: China University of Mining and Technology CUMT
Priority date: 2019-11-12
Filing date: 2019-11-12
Publication date: 2020-05-08

Abstract

The invention discloses a remote sensing image semantic segmentation method based on attention multi-scale feature fusion. And performing semantic segmentation on the data to be tested by using the trained network during testing. The network is a lightweight encoder-decoder architecture. The idea of an image cascade network is introduced, an attention mechanism is utilized to optimize coding features and decoding features, a multi-scale attention optimization module, a multi-scale feature fusion module and a boundary enhancement module are constructed, feature maps of different scales are extracted and fused, multi-scale semantic labels and boundary labels are used for guiding training, and semantic segmentation of remote sensing images can be effectively carried out.

Description

Remote sensing image semantic segmentation method based on attention multi-scale feature fusion

Technical Field

The invention relates to the technical field of image processing, in particular to a remote sensing image semantic segmentation method based on attention multi-scale feature fusion.

Background

Currently, semantic segmentation has become one of the key problems in the field of computer vision. From a macroscopic perspective, semantic segmentation is a high-level task that lays a good foundation for scene understanding. In reality, there are a large number of application scenarios that need to infer relevant knowledge or semantics (i.e., from concrete to abstract processes) from image data. These applications include unmanned driving, medical health, image search engines, augmented reality, and the like. These problems have been well solved by applying various conventional methods of computer vision and machine learning techniques. However, the rapid development of deep learning makes these methods gradually disfavored. In recent years, scholars in the field of semantic segmentation adopt a convolutional neural network CNN to extract high-level semantic features of images, and popular network models such as UNet, ResNet, deep lab and the like. Experiments have shown that it far exceeds the conventional method both in terms of performance and in terms of time consumption.

With the continuous deepening of the application of the geospatial information, how to sufficiently and accurately extract the information of the remote sensing image becomes important. The semantic segmentation of the remote sensing image is taken as a research hotspot in the field of remote sensing, and has wide application prospects in civil fields such as land management, disaster prediction and the like and military.

Different from common natural images such as city street views and the like, the high-resolution remote sensing image generally comprises detailed information with rich categories of ground objects such as roads, water sources, buildings, trees and the like. It has the following characteristics:

(1) the scale of individual targets is small, and a large number of small targets such as buildings, trees and the like exist in the remote sensing image, so that the features are difficult to extract;

(2) similarity exists between classes, heterogeneity exists in the classes, for example, the difference between roads and cement ground is low, and the tops of buildings are different;

(3) the occlusion problem can arise from overlapping boundaries of different objects, such as tree and building shadows.

The characteristics provide new challenges for the learning robust feature representation of the existing deep learning model, and the new challenges are the key for improving the semantic segmentation precision of the remote sensing image. And the remote sensing data sets with labels are very few, so that the development of a deep learning network in semantic segmentation application is limited, and the model is easy to generate an overfitting phenomenon.

Disclosure of Invention

Aiming at the problems, the invention provides a remote sensing image semantic segmentation method based on attention multi-scale feature fusion.

In order to realize the aim of the invention, the invention provides a remote sensing image semantic segmentation method based on attention multi-scale feature fusion, which comprises the following steps:

s10, constructing an initial semantic segmentation network based on attention multi-scale feature fusion; the method specifically comprises the following steps:

s11, establishing a depth semantic segmentation network feature encoder of the preset resolution image;

s12, constructing a feature decoder;

s13, determining the initial semantic segmentation network according to the deep semantic segmentation network feature encoder and the feature decoder;

s20, constructing a training data set, and performing parameter training on the attention multi-scale feature fusion-based semantic segmentation network by adopting the training data set to obtain the attention multi-scale feature fusion-based model semantic segmentation network; the method specifically comprises the following steps:

s21, compressing the semantic truth labels of the original data set by using a nearest interpolation method, carrying out boundary detection on the original data set, and acquiring boundary truth labels to determine a training data set;

s22, inputting the training data set into the initial semantic segmentation network for network parameter training, and determining a model semantic segmentation network according to a training result;

s30, inputting the data to be tested into the model semantic segmentation network to perform semantic segmentation on the data to be tested; the method specifically comprises the following steps:

s31, inputting the test data set into the model semantic segmentation network to obtain the semantic segmentation result of each remote sensing image;

and S32, evaluating the performance of the model semantic segmentation network by using MIoU as an evaluation index.

In one embodiment, step S11 includes:

s1111, down-sampling the original image resolution to 1/2, and inputting a depth residual error network ResNet 50;

s1112, replacing the 7 × 7 convolutional layer of the first stage of the depth residual error network with three 3 × 3 convolutional layers, and adding a dropout layer after each stage;

s1113, adding a down-sampling layer after the output R1 of the first residual block in the third stage of the depth residual error network, further reducing the size of the feature map to 1/32, and continuously inputting the feature map into the depth residual error network to obtain a feature map R2;

s1121, performing four-time average pooling on the characteristic maps R2, wherein the sizes of pooled nuclei are 1 × 1, 3 × 3, 5 × 5 and 7 × 7 respectively;

s1122, obtaining feature maps P1, P2, P3 and P4 which are the same as R2 in size through bilinear interpolation of the output of the pooling layer;

s1123, performing pixel-by-pixel addition so that a is P1+ P2+ P3+ P4+ R2;

s1124, compute the attention map of R2 by averagepoiling, conv1x1, batch norm and sigmoid activation functions and multiply the attention map pixel by pixel with a, forming the final output F1 of the multi-scale attention optimization module.

In one embodiment, step S12 includes:

s121, subjecting the final output F1 to convolution layer with convolution kernel of 1 × 1 to obtain a feature map D1;

s122, inputting D1 and R1 into a multi-scale feature fusion module to obtain a 1/16 feature map O1 of the original image;

s123, inputting the output characteristics of the O1 and ResNet50 network in the second stage into the multi-scale characteristic fusion module again to obtain a 1/8 characteristic diagram O2 of the original image;

s124, the original image with the resolution of 1024 × 1024 is subjected to convolution layers with the step size of 2 and the convolution kernel size of 3 × 3, and thus a characteristic diagram X1 of 1/8 with the size of the original image is obtained;

s125, overlapping X1 and O2 by using a Concatenate layer, and then accessing a 3X 3 convolutional layer to obtain a 1/8 feature map X2 with the size of an original image;

s126, inputting X1 and X2 into a boundary enhancement module to obtain a feature map X3 of 1/4 of the original image;

s127, upsampling the X1 by using a bilinear interpolation value to obtain a characteristic diagram Y1;

s128, overlapping X3 and Y1 by using a Concatenate layer, and then accessing a 3X 3 convolutional layer to obtain a 1/4 feature map Y2 with the size of an original image;

s129, in the training stage, adding a convolutional layer and a fully-connected Softmax classifier after Y2, setting the number of channels of the convolutional layer as the number n _ class of the semantic segmentation of the remote sensing data set, obtaining a feature map Y3 of 1/4 with the size of an original image and generating a semantic loss function

As one embodiment, step S122 includes:

s1221, upsampling D1 by adopting a linear interpolation value, and then performing convolution and normalization processing on a cavity with a convolution kernel of 3 x3 and an expansion rate of 2 to obtain a characteristic diagram F1 with the size of 1/16;

s1222, computing an attention map a1 for each channel of F1 by averaging pooling, convolution, normalization processing and Sigmoid activation functions;

s1223, multiplying the feature R11 obtained by performing a layer of 1 × 1 convolution on R1 by A1 to obtain a channel-level attention feature map C1;

s1224, performing feature fusion O1 ═ F1+ C1+ R1, where O1 is the output of the multi-scale feature fusion module;

s1225, adding a convolution layer and a fully-connected Softmax classifier in a training stage, setting the number of channels of the convolution layer as the number n _ class of the classes of the semantic segmentation of the remote sensing data set, obtaining a feature vector T1 and generating a semantic loss function

S1226, in the second multi-scale feature fusion module, taking O1 as input, repeating the steps S1221 and S1222, taking the output of the second stage of the ResNet50 network as the input of S1223, continuing to repeat the steps S1224 and S1225 to obtain a feature map O2 with the size of 1/8, and generating a feature vector T2 and a semantic loss function in the training stage

As one embodiment, step S126 includes:

s1261, inputting X1 and X2 into three convolutional layers and a 4-step-size deconvolution layer respectively to obtain feature maps DF1 and DF 2; the convolution kernels of the first two convolution layers are 3 multiplied by 3, the number of channels is 32, the convolution kernel of the third layer is 1 multiplied by 1, the number of channels is 16, the step length of the deconvolution layer is 2, and the convolution kernel is 3 multiplied by 3;

s1262, performing feature fusion DF1+ DF 2;

s1263, adding convolution layer with convolution kernel of 3 × 3 to obtain 1/4 feature graph X3 with original image size;

s1264, in the training phase, sampling X3 by 4 times, adding a convolution layer and a fully-connected Softmax classifier, setting the number of channels of the convolution layer to be 2, obtaining a model prediction value E, and generating a boundary loss function L_edge。

As an example, the boundary loss function L_edgeThe method comprises the following steps:

L_edge＝-∑_Xy_Xlogs_X，

in the formula, L_edgeRepresenting the boundary loss function, X representing the input image, y_XIndicating the desired output, s_XRepresenting the actual output.

In one embodiment, step S22 includes:

s221, inputting a training data set, semantic truth labels of three scales and boundary truth labels into an initial semantic segmentation network;

s222, obtaining output model predicted values Y3, T1, T2 and E through initial semantic segmentation network training;

s223, calculating errors between the model predicted values Y3, T1, T2 and E and corresponding true values by using a set loss function L;

s224, optimizing the set loss functions corresponding to the model predicted values Y3, T1, T2 and E respectively by adopting an adaptive moment estimation optimization algorithm;

s225, setting a loss function L to be continuously reduced through iterative training until convergence, wherein the performance of the semantic segmentation network is optimal at the moment, and determining the model semantic segmentation network according to the network parameters currently possessed by the semantic segmentation network.

As an embodiment, the set loss function L includes a sum of semantic loss and boundary loss, and the determination formula of the set loss function L includes:

in the formula (I), the compound is shown in the specification,

the ith semantic loss function (ith semantic loss function) is shown, and in the present embodiment, since there are three semantic loss functions in total, I is 3, L_edgeRepresenting boundary loss function, α_iDenotes semantic loss weight, λ is edge loss weight, α₁、α₂、α₃Weights representing three semantic loss and edge loss, respectively, for the US3D dataset, let α₁＝1，α₂＝0.4，α₃＝0.16，λ＝1。

As an embodiment, the determination formula of each semantic loss function is:

in the formula (I), the compound is shown in the specification,

denotes the ith semantic loss function, i is 1, 2, 3, X denotes the input image X, y_XIndicating the desired output, s_XRepresenting the actual output.

As an example, MIoU is the average cross-over ratio, the cross-over ratio between the true tag value and the segmentation predicted by the system. This ratio may be redefined as the number of true positive cases (intersections) divided by the total number (including true positive cases, false negative cases, and false positive cases). The determination formula of MIoU includes:

wherein p is_iiIndicating the number of true positive examples,

representing the total number (including true positive, false negative, and false positive), k represents the number of categories. The higher the average intersection ratio is than the MIoU, the better the performance of the semantic segmentation network is.

The remote sensing image semantic segmentation method based on attention multi-scale feature fusion constructs a training data set by constructing an initial semantic segmentation network based on attention multi-scale feature fusion, obtains a model semantic segmentation network based on attention multi-scale feature fusion by adopting the training data set and carrying out semantic segmentation network training based on attention multi-scale feature fusion, inputs data to be detected into the model semantic segmentation network to carry out semantic segmentation on the data to be detected, introduces the thought of an image cascade network, greatly reduces the number of model parameters, simultaneously utilizes an attention system to optimize coding features and decoding features, constructs a multi-scale attention optimization module, a multi-scale feature fusion module and a boundary enhancement module, extracts and fuses feature maps of different scales, and uses multi-scale semantic labels and boundary labels to guide training, the performance of the corresponding model is effectively improved.

Drawings

FIG. 1 is a flow chart of a remote sensing image semantic segmentation method based on attention multi-scale feature fusion according to an embodiment;

FIG. 2 is a network architecture diagram of one embodiment;

FIG. 3 is a block diagram of a multi-scale attention optimization module of an embodiment;

FIG. 4 is a block diagram of a multi-scale feature fusion module of an embodiment;

FIG. 5 is a block diagram of a boundary enhancement module of an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

Referring to fig. 1, fig. 1 is a flowchart of a semantic segmentation method for a remote sensing image based on attention multi-scale feature fusion according to an embodiment, and includes the following steps:

s12, constructing a feature decoder;

s13, determining the initial semantic segmentation network according to the deep semantic segmentation network feature encoder and the feature decoder.

The preset resolution image is a low resolution image. In the step S11, the depth residual network ResNet50 may be used for encoding; a multi-scale attention optimization module is constructed to refine the feature map.

and S22, inputting the training data set into the initial semantic segmentation network for network training, and determining a model semantic segmentation network according to a training result.

The training data set may include a vast number of public remote sensing images.

In step S21, the original data set semantic truth labels may be compressed 1/16, 1/8, 1/4 using nearest neighbor interpolation. And (4) adopting a Canny algorithm in Opencv to carry out boundary detection, and obtaining a boundary truth value label.

In one example, a training data set is used and the semantic segmentation network based on attention multi-scale feature fusion is subjected to parameter training, and a structural diagram of the semantic segmentation network based on attention multi-scale feature fusion can be referred to as shown in fig. 2. Specifically, fig. 2 is a design drawing of the present example, and a semantic segmentation network is built according to the design drawing. In the training stage, training set data is input into a network, how to perform semantic segmentation on network learning is achieved through continuous training, and four loss functions are arranged in the network to supervise the learning process. And after the training is finished, obtaining the model semantic segmentation network, namely the trained network. In the testing stage, a test set is input to test the model semantic segmentation network, and MIou is used as an evaluation index. Higher values of MIou indicate better segmentation.

In one example, a block diagram of a multi-scale attention optimization module may be found with reference to FIG. 3.

The data to be measured is a remote sensing image needing semantic segmentation.

Specifically, the MIoU is an average cross-over ratio, and a cross-over ratio between the true tag value and the segmentation predicted by the system. This ratio may be redefined as the number of true positive cases (intersections) divided by the total number (including true positive cases, false negative cases, and false positive cases). The determination formula of MIoU includes:

wherein p is_iiIndicating the number of true positive examples,

In one embodiment, step S11 includes:

s1123, performing pixel-by-pixel addition so that a is P1+ P2+ P3+ P4+ R2;

s1124, calculating an attention map of R2 by averagepoolling, conv1x1, batch norm and sigmoid activation functions, and multiplying the attention map with a pixel by pixel to form the final output F1 of the multi-scale attention optimization module.

As one example, step S12 includes:

The feature map obtained in S128 is the output of the feature decoder; the whole process of S121 to S129 is to construct a feature decoder.

In one example, a block diagram of the multi-scale feature fusion module may be as shown with reference to FIG. 4. The structure of the boundary enhancing module can be seen with reference to fig. 5.

As one embodiment, step S122 includes:

Specifically, in steps S1225, S1226, and S129 described above:

semantic loss function

Semantic loss function

And semantic loss function

Respectively as follows:

in the formula (I), the compound is shown in the specification,

represents the ith semantic loss function (i.e., the ith semantic loss function), X represents the input image X, y_XIndicating the desired output, s_XRepresenting the actual output.

As one embodiment, step S126 includes:

s1262, performing feature fusion DF1+ DF 2;

Specifically, in step S126, the convolution layers are each composed of a convolution, an activation function Relu, and a normalization layer.

L_edge＝-∑_Xy_Xlogs_X，

in the formula, L_edgeRepresenting the boundary loss function, X representing the input image X, y_XIndicating the desired output, s_XRepresenting the actual output.

In one embodiment, step S22 includes:

s222, obtaining output model predicted values Y3, T1, T2 and E through initial semantic segmentation network training; (i.e., model predicted value Y3, model predicted value T1, model predicted value T2, and model predicted value E)

s224, optimizing loss functions corresponding to model predicted values Y3, T1, T2 and E respectively by adopting an adaptive matrix estimation (ADAM) optimization algorithm; namely, an adaptive moment estimation (ADAM) optimization algorithm is adopted to optimize the four loss functions in step S223;

In particular, the following set loss function L can be minimized by back propagation:

where W represents the set of all standard network layer parameters. The semantic segmentation output and the boundary output are both associated with a classifier, wherein the corresponding weights are respectively denoted as w_s、w_edge。

The ith semantic loss function is expressed, and in the present embodiment, since there are three semantic loss functions in total, I is 3. L is_edgeRepresenting boundary loss function, α_i(α₁、α₂、α₃) λ are all continuous hyperparameters, α₁、α₂、α₃Weights representing three semantic losses, λ being the weight of the edge loss, respectively, let α for the US3D dataset₁＝1，α₂＝0.4，α₃＝0.16，λ＝1。

in the formula (I), the compound is shown in the specification,

the ith semantic loss function is expressed, and there are three in the present invention, so I is 3. L is_edgeRepresenting boundary loss function, α₁、α₂、α₃λ is a continuous hyper-parameter, representing the weight of three semantic losses, edge losses, respectively, for the US3D dataset, let α₁＝1，α₂＝0.4，α₃＝0.16，λ＝1。

As an example of the way in which the device may be used,

in the formula (I), the compound is shown in the specification,

representing the ith semantic loss function, X representing the input image X, y_XIndicating the desired output, s_XRepresenting the actual output.

Compared with the prior art, the remote sensing image semantic segmentation method based on attention multi-scale feature fusion has the following advantages:

(1) the method is a lightweight encoder-decoder semantic segmentation network, and the parameter number is only 8.4M.

(2) The constructed multi-scale attention optimization module is combined with a depth residual error network to serve as a low-resolution branched feature extractor, the defect that a pooling layer is prone to losing spatial information is overcome, and global and local features can be effectively extracted.

(3) The idea of image cascade network is introduced, the low-resolution image is used for extracting the features, and then the high-resolution image is used for refining the features, so that the model parameters are greatly reduced.

(4) A multi-scale feature fusion module is provided, features of different scales are fused in a decoder part, and high-level features are used as guidance of low-level features, so that the image detail information can be recovered, and the semantic extraction and segmentation of small-scale targets are improved;

(5) a boundary enhancement module is constructed by using the deconvolution layer and is used for capturing boundary information of the remote sensing image, and the problem of overlapping of different target boundaries of the remote sensing image is solved.

In one embodiment, the remote sensing image semantic segmentation method based on attention multi-scale feature fusion is tested in a hardware environment with a memory of 16GB and a software environment with 1 NVIDIA Geforce RTX 2080 Ti GPU for acceleration and python3.6, tensoflow-GPU 1.13.1, keras-GPU 2.1.4 and CUDA 10.0.

The experiment of this example used the US3D telemetry data set. Urban semantic three-dimensional (US3D), a large public data set containing multiple views of two large cities, multiband satellite images and ground truth geometric and semantic labels. For the semantic segmentation task, the US3D dataset currently includes coverage of approximately 100 square kilometers for jackson wale, florida and omaha, nebraska, usa, with a full color Ground Sample Distance (GSD) of approximately 30 cm. The source remote sensing satellite images of the US3D dataset are provided by DigitalGlobe, and the semantic labels are automatically derived from the published HSIP lidar product. There are 4292 high-resolution remote sensing satellite images and semantic truth labels of 1024 × 1024 in the data set, wherein 5 categories are labeled, namely, ground, trees, buildings, water and roads. In the experiments of the present invention, 80% of the entire data set was used for training of the model and 20% for testing.

And (4) analyzing results:

in the simulation experiment of this embodiment, the MFFANet (selective segmentation network of remote sensing image based on Multi-scale Feature Fusion) and the image cascade network (ICNet) are compared by the method of the present invention, and the segmentation effect is compared and analyzed, where table 1 is the comparison result. As can be seen from table 1, the method of the present invention has a higher MIou than ICNet on the US3D data set.

TABLE 1 comparative results of the experiments

The embodiment introduces the idea of image cascade network, and greatly reduces the number of model parameters. Meanwhile, an attention mechanism is utilized to optimize coding features and decoding features, a multi-scale attention optimization module, a multi-scale feature fusion module and a boundary enhancement module are constructed, feature graphs of different scales are extracted and fused, multi-scale semantic labels and boundary labels are used for guiding training, and the performance of the model is effectively improved under the condition that the parameter quantity of the model is only 8.4M.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

It should be noted that the terms "first \ second \ third" referred to in the embodiments of the present application merely distinguish similar objects, and do not represent a specific ordering for the objects, and it should be understood that "first \ second \ third" may exchange a specific order or sequence when allowed. It should be understood that "first \ second \ third" distinct objects may be interchanged under appropriate circumstances such that the embodiments of the application described herein may be implemented in an order other than those illustrated or described herein.

The terms "comprising" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that comprises a list of steps or modules is not limited to the listed steps or modules but may alternatively include other steps or modules not listed or inherent to such process, method, product, or device.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A remote sensing image semantic segmentation method based on attention multi-scale feature fusion is characterized by comprising the following steps:

s12, constructing a feature decoder;

2. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion according to claim 1, wherein the step S11 includes:

s1123, performing pixel-by-pixel addition so that a is P1+ P2+ P3+ P4+ R2;

3. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion according to claim 2, wherein the step S12 includes:

4. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion according to claim 3, wherein the step S122 comprises:

5. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion according to claim 3, wherein the step S126 comprises:

s1262, performing feature fusion DF1+ DF 2;

6. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion as claimed in claim 5, wherein said boundary loss function L_edgeThe method comprises the following steps:

L_edge＝-∑_Xy_Xlogs_X，

7. The method for semantically segmenting the remote sensing image based on the attention multi-scale feature fusion according to any one of claims 1 to 6, wherein the step S22 comprises:

8. The method for semantically segmenting the remote sensing image based on attention multi-scale feature fusion of claim 7, wherein the set loss function L comprises the sum of semantic loss and boundary loss, and the determination formula of the set loss function L comprises:

in the formula (I), the compound is shown in the specification,

denotes the ith semantic loss function, i 1, 2, 3, L_edgeRepresenting boundary loss function, α_iRepresents a semantic loss weight, and λ is an edge loss weight.

9. The remote sensing image semantic segmentation method based on attention multi-scale feature fusion according to claim 8, characterized in that the determination formula of each semantic loss function is:

in the formula (I), the compound is shown in the specification,

10. The method for semantically segmenting the remote sensing image based on the attention multi-scale feature fusion of any claim 1 to 6, wherein MIoU is an average cross-over ratio, and the determination formula of MIoU comprises:

wherein p is_iiIndicating the number of true positive examples,

representing the total number and k representing the number of categories.