CN114299394A - An Intelligent Interpretation Method of Remote Sensing Image - Google Patents

Abstract

The invention belongs to the field of computer vision, and in particular relates to an intelligent remote sensing image interpretation method. The present invention finally realizes the intelligent interpretation of remote sensing images by using semantic segmentation technology, vectorization technology, and vector simplification and smoothing technology. By using the invention, it is possible to input remote sensing images of any size, and directly output the corresponding vector results. Moreover, the interpretation time of a medium-sized district/county remote sensing image with an area of about 400 square kilometers and a remote sensing resolution of 0.8 meters can be controlled to about 1 hour, while manual interpretation takes 2 weeks, which greatly improves the interpretation. Efficiency, reducing the cost of manual interpretation and realizing the automation of interpretation. In addition, there is no similar automatic interpretation product on the market at present, which fills the market gap of intelligent interpretation of remote sensing images.

Description

An Intelligent Interpretation Method of Remote Sensing Image

technical field

The invention belongs to the field of computer vision, in particular to an intelligent remote sensing image interpretation method.

Background technique

Identifying and extracting different types of ground feature information from remote sensing images is a major subject in the field of remote sensing image processing. In the field of remote sensing, the extraction of object category information from images is called remote sensing image interpretation. The extracted information includes the category, shape, area, etc. of the features. Relevant statistical departments need to count the feature information of the same area every year, and compare it with the information of the past years, so as to understand and track the development and changes of the feature elements in the region, and provide data support for the government, enterprises and other relevant departments to assist decision-making, etc. The traditional method is to use professional remote sensing image interpretation software such as ArcGis/QGIS/Grass, etc., by trained professional interpreters, by comparing remote sensing images, in the software along the contour of the ground objects in the way of drawing points by hand to draw closed lines. Vector polygon (Polygon), usually interpreted areas are divided into administrative areas, such as counties, for a medium-sized county administrative area with an area of about 400 square kilometers and a remote sensing resolution of 0.8 meters, a professional draws about It takes 2 weeks, which is very time-consuming and labor-intensive. At the same time, due to the influence of subjective factors such as individual differences and fatigue, the rendering accuracy will be inconsistent with the classification standards before and after, resulting in the phenomenon of interpretation errors, and the interpretation effect is not good.

Due to the great progress and success of artificial intelligence deep learning technology, especially convolutional neural network technology in the field of computer vision image processing and recognition, many researchers have attracted many researchers to apply artificial intelligence neural network technology to remote sensing image processing. On the technical route, the remote sensing image segmentation task belongs to the semantic segmentation task in computer vision, that is, the different target pixels in the remote sensing image are divided into different categories. In the generated remote sensing segmentation result image, the same feature category uses the same pixel. Value representation, different pixel values represent different categories. In the deep learning semantic segmentation technology, XiaLi, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, Hong Liu. Expectation-Maximization Attention Networks for Semantic Segmentation introduced the expectation maximization attention mechanism into the convolutional neural network, and proposed the EMANet implementation Semantic segmentation operation. Considering that the spatial pyramid pooling module can capture multi-scale information, the encoder-decoder architecture can better capture the edges of sharp objects, Liang-ChiehChen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.Encoder-Decoder with Atrous Separable On the basis of DeepLabv3, Convolution for Semantic Image Segmentation proposes DeepLabv3+ based on Encoder-Decoder network architecture by adding Decoder module and applying depthwise separable convolution to ASPP and Decoder modules. Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-Contextual Representations for Semantic Segmentation adopts HRNetV2+OCR module to obtain the performance of SOTA on the CityScape dataset.

Because the training of convolutional neural networks usually requires a large amount of labeled training data, and remote sensing image data is usually sensitive and involves the security of national geographic information, only a few authorized government departments or secret-related enterprises, universities, and research institutions have these Generally, there are few well-labeled datasets publicly available, which also brings many difficulties and barriers to the improvement of technical research. In recent years, some government departments and enterprises have also held some competitions, which will provide some desensitized and declassified data sets to attract more researchers in the direction of deep learning to participate in order to promote technological progress in this field. At present, the difficulty of the core technology in this field is still the performance of pixel-level classification results. Since there are many foreign objects of the same spectrum in remote sensing images (that is, the pixel values are the same but belong to different categories), and the same objects are different in spectrum (that is, the same target has different pixel values due to shooting equipment, altitude, illumination, weather, date, etc.), which is a remote sensing image. The performance improvement of image segmentation technology has brought great challenges, and it is also a major difficulty to be overcome in the field of remote sensing imagery.

At present, remote sensing image segmentation methods mostly focus on remote sensing image segmentation technology, that is, the improvement of pixel-level classification performance, and there are still no mature commercial products.

SUMMARY OF THE INVENTION

Based on the deep learning semantic segmentation technology, the present invention realizes the intelligent interpretation of remote sensing images by using its own remote sensing image data set. Input the remote sensing image, you can get the interpretation result of the surface vector (Polygon). FIG. 1 is the overall network structure and processing flow chart of the present invention, and FIG. 2 is a superimposed display of the vector result output by the present invention and the input remote sensing image.

For achieving the above object, the technical scheme of the present invention is:

An intelligent remote sensing image interpretation method. After obtaining the remote sensing image, the interpretation method includes the following steps:

S1. Segment the original remote sensing image as needed, and obtain multiple target images after segmentation; wherein, during segmentation, the obtained adjacent target images have overlapping parts;

S2. Use the deep learning method to classify all the obtained target images pixel by pixel, and generate a grayscale image of the predicted classification result; specifically:

The target image is sent to three different deep learning models for semantic segmentation, namely DeepLabv3, HRNetW48+OCR, and EMANet networks, and then the ensemble learning method is used to determine whether any pixel belongs to a certain category by the three networks. The output result is decided by voting, and the decision result obtained by the majority of votes is the final category result of the pixel; at the same time, the semantic segmentation deep learning model also outputs a probability map that each pixel belongs to a certain category, that is, the confidence level;

S3, splicing the predicted classification results of all target images according to their positions in the original image to obtain a grayscale image of the predicted result of the original remote sensing image; similarly, splicing the probability map;

S4. Perform connection processing on the long and narrow strip-shaped fracture elements in the grayscale image of the prediction result;

S5, filtering out the small spots in the grayscale image of the prediction result, specifically filling the independent spots whose number of pixels is less than the specified threshold of the category with the surrounding pixel categories;

S6, converting the grayscale image of the prediction result obtained in step S5 into a vector shp file, and calculating the confidence and area of each spot as an attribute of the spot;

S7. Use the Douglas-Peucker algorithm in the open source software GrassGis to simplify and smooth the area boundary of the vector shp file, eliminate the jagged edge effect of the ground object, and obtain the interpretation result of the remote sensing image.

The beneficial effects of the present invention are as follows: the present invention can realize intelligent interpretation of remote sensing images, input grid (pixel) remote sensing images, and directly output vector interpretation results, which greatly reduces the difficulty and workload of manual interpretation, and greatly reduces the difficulty and workload of manual interpretation. Greatly improve the speed of interpretation and realize the automation of interpretation.

The method of the invention opens up the three links of remote sensing image segmentation, grid vectorization, and vector simplification and smoothing, realizes an automatic remote sensing intelligent interpretation system, outputs the ground object information represented by vectors, and greatly improves the interpretation efficiency and interpretation. Translation automation. Input a remote sensing image of any size, and through the intelligent interpretation method of the present invention, the corresponding vector interpretation result can be directly output.

Description of drawings

FIG. 1 shows the processing flow steps and core processing modules of the remote sensing image intelligent interpretation method of the present invention.

Figure 2 is a superimposed display of the vector result interpreted by the system and the input remote sensing image.

Figure 3 is a schematic diagram of the result after the raster is converted into a vector.

FIG. 4 is an enlarged comparison diagram of a partial result of the result after vector simplification and smoothing.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings, and some processes in the flow process will be shown.

The invention can handle the automatic interpretation of remote sensing images of eight categories and twenty-five categories. The classification standards and corresponding numerical codes are shown in Table 1 (numbers 1-8 represent eight classification standards, and the others are classification standards of 25 classifications):

Table 1 Classification standard and corresponding numerical code

FIG. 1 is the overall framework and processing flow of the intelligent interpretation method of remote sensing images according to the present invention. It is divided into four stages as a whole, the first stage is the remote sensing image segmentation prediction stage. This stage completes the cropping of remote sensing images, independent prediction, generation of prediction result maps and probability maps, and splicing and fusion of sub-image results. The second stage is the post-processing stage of the segmentation grayscale result. It includes two modules, one is a module for connecting fracture elements, and the other is a module for filtering out small patches under the specified threshold of the category. The third stage is the vectorization stage. Complete raster to vector while adding confidence and area attributes to each vector blob. The fourth stage, the vector simplification smoothing module. The implementation of Douglas-Peucker simplification algorithm in GrassGis is used to simplify and smooth the vector, eliminate edge jaggedness, and at the same time ensure the correct topology. Specifically:

The first stage is the semantic segmentation stage of remote sensing images. In this stage, the deep learning algorithm is mainly used to classify remote sensing images pixel by pixel to generate a grayscale image of the classification result, and the pixel value is the class code. In the field of remote sensing image processing, this grayscale image of pixel classification result is also called raster result image. Usually remote sensing images are divided into administrative units, such as remote sensing images of a certain county, which are usually larger in size, such as image width and height of 20000x20000 pixels. Due to the limitation of computing conditions such as computer memory and graphics card memory, the original large-size image cannot be accommodated at one time. Therefore, the large-size image is cropped into a small-scale image of 512x512 (the specific size can be adjusted according to the limitations of computer memory and graphics card memory), They are respectively sent to the network for prediction, and then the prediction results of each small-scale image are stitched together. The present invention proposes to flexibly set small-size images that can be processed by the computer according to the configuration of computer hardware, so that the system is given the ability to process large-size remote sensing images. The format of the input image in the present invention is the original image of the remote sensing image, such as the remote sensing image with geographic coordinates and spatial reference information and the suffix is tif or img format.

When cropping, adjacent small-sized images need to have overlapping parts, because if they do not overlap at all, since each small-sized image is predicted through the network separately, there will be inconsistent prediction effects at the edges of the images. For example, if a building happens to be divided into two images, obvious splicing marks will be produced when splicing. The solution of the present invention is that when cropping, adjacent images overlap, for example, two adjacent small images overlap by 200 pixels, and then average the overlapping area on the predicted output result, so that the obtained prediction result can be very large. To reduce the splicing marks to a certain extent.

In the segmentation core technology, in order to improve the model performance, the present invention uses three models for integration, that is, each small-size remote sensing image passes through three different models, namely DeepLabv3, EMANet and HRNetW48+OCR. Then the three models make predictions respectively, and then vote on whether the same pixel position belongs to a certain category, that is, when two or more models predict that the pixel belongs to a certain category, the position is confirmed as This category, otherwise not this category. Ensemble learning is a common technique in machine learning to improve model performance.

At the same time, in order to facilitate subsequent manual intervention to correct the vector results predicted by the model, the segmentation model will also output a probability map that each pixel belongs to a certain category, also known as the confidence level. After vectorization, the confidence of each vector patch is the average value of the probability of each pixel under the patch. When human intervention is involved in the modification, the vector pattern with low confidence can be focused on, and manual modification can be carried out.

In addition, due to the uneven distribution of various categories in remote sensing images, for example, the eight categories of cultivated land accounted for nearly 40%, while the garden and water area accounted for only about 3-4%, resulting in the model training process, the proportion of The class prediction effect is not good. In order to solve this problem, the present invention adopts the FocalLoss loss function proposed in Focal Loss for Dense Object Detection by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar, etc. , The channel Attention mechanism proposed by Samuel Albanie, Gang Sun, Enhua Wu et al in Squeeze-and-Excitation Networks and Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon et al in CBAM: Convolutional Block Attention Module The idea of using the spatial Attention mechanism and the channel Attention mechanism together further alleviates the impact of uneven categories on model performance.

Finally, the prediction results of each small image at this stage need to be spliced into a grayscale image of the prediction result with the same width and height as the original input remote sensing image.

The second stage is the post-processing stage of the segmentation results. Because the model needs to process the results of remote sensing image output in two aspects. First, it is found that in the predicted results, there are many fractures in long and narrow strip-like elements such as roads and rivers, and they need to be connected. Second, there are some scattered independent elements in the predicted results, which are filtered out below a certain area (number of pixels). The solution adopted in the present invention is to fill with other pixels around the small image spot. Each category filtering standard specified in the present invention is shown in Table 2, Table 3:

Table 2: Corresponding filtering thresholds for eight categories

Table 3: Twenty-five classifications correspond to filtering thresholds

After completing the connection of the fracture elements and filtering out the small patches, the next stage of processing is entered.

The third stage, the raster vectorization stage. That is, the raster classification result of the second stage post-processing is converted into a stage represented by a vector polygon surface (Polygon). For connected regions with the same pixel value, use the region outline polygon representation. That is, the raster is vectorized into a representation of polygon features. Since the output of the second stage is only in image format and does not contain information such as spatial reference and geographic coordinates of the image, it is necessary to add the spatial reference and geographic coordinates of the original remote sensing image to the results of the second stage before vectorization. The present invention uses the open source library Gdal to realize vectorization. Vectorization techniques typically generate vector polygons along pixel edges. This approach will give a lot of jagged edges to the vectorized result. But the advantage is that there are no topological errors. That is, there is no overlap and gap between the polygons of the surface vector, and it is also a mainstream and relatively successful commercial vector method. In the process of vectorization, a confidence attribute is added to each patch. The calculation of the confidence comes from the probability map output by remote sensing image segmentation, which is the mean value of the corresponding pixel probability under the patch. At the same time, the area of each vector patch is calculated as the area attribute of the patch.

The fourth stage, the vector simplification smoothing stage. After the third stage of vectorization, there is jaggedness in the result, and it is necessary to remove unnecessary jagged edges as much as possible. For example, the edge of a straight line represented by stepped jagged teeth is represented by the two vertices of the straight line, and the unnecessary jagged points in the middle are removed. . The arcs represented by the adjacent sawtooth are represented by relatively smooth arcs. The techniques here usually involve two types, one is the simplification technique and the other is the smoothing technique. For the simplification technique, that is, through the algorithm, delete the points as much as possible, and use fewer points to represent the edge, so as to achieve the effect of a straight line with jagged edges. Another technique is smoothing, which starts with smooth curves instead of jagged edge lines. The most common problem between these two techniques is that during the simplification or smoothing process, it will lead to topology errors, that is, gaps or overlaps between adjacent surface vectors. Results with topology errors will cause statistical errors and cannot be used. After a lot of investigations, the present invention is realized by the Douglas-Peucker simplified algorithm provided in the v.generalize module of the open source software GrassGis, which can eliminate aliasing while maintaining the correctness of the topology structure. Figures 3 and 4 show the effect of this algorithm on simplification and smoothing of jagged edges.

Based on the core artificial intelligence algorithm, the invention opens up four stages of remote sensing image segmentation, post-processing of segmentation results, grid vectorization, vector simplification and smoothing, etc., and realizes intelligent interpretation of large-scale remote sensing images, requiring only manual intervention in the later stage. Correcting the prediction results with low confidence greatly reduces the intensity of manual interpretation and improves the efficiency of interpretation. Experiments have verified that it takes about 2 weeks to manually interpret an ordinary and medium-sized county, but using the method of the present invention, the result can be obtained in 2 hours, which greatly improves the interpretation efficiency. It is understood that there is no successful automatic interpretation product of remote sensing images on the market, and the remote sensing intelligent interpretation system of the present invention fills the market gap.

Claims (1)

1. An intelligent remote sensing image interpretation method is characterized in that after a remote sensing image is acquired, the interpretation method comprises the following steps:

s1, segmenting the original remote sensing image as required to obtain a plurality of target images; wherein at the time of segmentation, adjacent target images obtained are caused to have an overlapping portion;

s2, classifying all the obtained target images pixel by adopting a deep learning method to generate a prediction classification result gray level image; the method specifically comprises the following steps:

respectively sending the target image into three different semantic segmentation deep learning models, namely a DeepLabv3, a HRNetW48+ OCR and an EMANet network, then determining whether any pixel belongs to a certain category by using an integrated learning method and voting according to results output by the three networks respectively, wherein the determination result of the number of votes obtained is the final category result of the pixel; meanwhile, the semantic segmentation deep learning model also outputs a probability map, namely confidence coefficient, of each pixel belonging to a certain category;

s3, splicing the prediction classification results of all target images according to the positions of the target images in the original images to obtain a prediction result gray-scale image of the original remote sensing image; splicing the probability graph in the same way;

s4, connecting the long and narrow strip-shaped broken elements in the prediction result gray-scale image;

s5, filtering small patches in the prediction result gray scale image, specifically filling independent patches with the number of pixels smaller than a category designated threshold with the categories of the pixels around the independent patches;

s6, converting the prediction result gray-scale image obtained in the step S5 into a vector shp file, and meanwhile, calculating the confidence coefficient and the area of each image spot as the attributes of the image spots;

s7, simplifying and smoothing the boundary of the vector shp file region by using a Douglas-Peucker algorithm in open source software GrassGis, and eliminating the sawtooth effect of the edges of the ground objects, thereby obtaining the interpretation result of the remote sensing image.

Images