CN115841633A - Power tower and power line associated correction power tower and power line detection method - Google Patents

Abstract

A power tower and power line detection method for power tower and power line correlation correction is characterized in that images of the power tower and the power line are shot from the upper part, and graying pretreatment is carried out on the images; carrying out edge detection on the preprocessed image to obtain a binary edge image; carrying out power line detection and power tower detection by using the edge map to obtain power lines represented by parallel line groups and a power tower area framed by a rectangular frame; mapping the rectangular frame to an edge map, excluding background areas on two sides of the rectangular frame of the power tower, and re-detecting the power lines in the internal area of the rectangular frame of the power tower; and after a new power line detection result is obtained, estimating the ratio of the power tower area in the edge graph according to the left and right widths of the power line distribution area, updating power tower detection parameters, and re-detecting the power tower until the requirements are met or the iteration times are reached. The method does not depend on a large-scale data set, and has higher detection speed and better effect.

Description

A method for detecting power towers and power lines with correlation correction between power towers and power lines

Technical Field

The present invention belongs to the technical field of power facility safety detection, relates to the inspection of power towers and power lines, and in particular to a power tower and power line detection method for correcting the correlation between power towers and power lines.

Background Art

Traditional power line inspections mainly include manual inspections and helicopter inspections. Manual inspections are a way for workers to manually inspect and check the status of power lines after climbing power towers. There are many problems with manual inspections. On the one hand, the accuracy and efficiency are not high, especially for long-distance transmission lines, which require a lot of manpower and material costs; on the other hand, the safety is low, especially for lines in complex environments, coupled with the influence of bad weather conditions, manual inspections will be very difficult and dangerous. Helicopter inspections are another common inspection method for power lines. They mainly use helicopters equipped with professionals and various sensor equipment to hover over power lines for inspections. Helicopter inspections have solved some of the problems of manual inspections to a certain extent. For example, helicopters can reach complex terrains such as canyons that are difficult for people to reach, and the inspection efficiency is greatly improved. However, helicopters are not flexible to use, are too large to enter many specific areas, and have high inspection and maintenance costs, so helicopter inspections are only used in some special cases. In summary, the current traditional inspection methods can no longer meet the growing demand for power line inspections.

In recent years, with the development of artificial intelligence and robotics, there has emerged a technical solution that relies on drones with simple cameras to collect data and use artificial intelligence technology to understand images for power line inspections. The drone inspection method is less affected by climate and weather and can work for a long time. It not only greatly improves the inspection frequency and inspection quality, but also reduces labor costs while ensuring the safety of workers. Therefore, it has obvious advantages in power line inspections. However, in the inspection method of drones with cameras, the key is how to use computer vision recognition technology to detect and identify power lines and power towers.

At present, the main methods used by researchers at home and abroad for detecting power lines in images can be divided into two categories: traditional image processing methods and data-driven machine learning methods. Image processing methods are based on the physical and geometric characteristics of the power lines themselves, and define the characteristic patterns of the power lines based on artificial experience to perform pattern recognition on the images to be detected. For example, based on prior knowledge, the power lines are regarded as a continuous straight line, and the power line detection is completed through classic line segment detection methods, such as Hough transform, Radon transform, directional filtering, and straight line detection algorithms based on gradient and edge information. Traditional image processing methods are easily interfered by noise, and their artificial experience is mostly targeted at specific scenes. In the face of different environments, parameters need to be adjusted frequently, or even the characteristic patterns need to be redefined, in order to achieve better detection results. Data-driven machine learning methods, on the other hand, learn and train machine learning models on a large amount of labeled data to obtain a detection model that can effectively describe the characteristics of power lines. Among machine learning methods, the method based on deep learning models (usually convolutional neural networks) has a better effect. Data-driven machine learning methods usually require a large number of manually labeled data sets to train the model, and the extracted power lines need to be post-processed and optimized using the power line structure information.

At the same time, the main technical route for the detection of power towers is to use the target detection method in the field of computer vision. Moreover, the method with better results is also the target detection method based on deep learning models, such as the two-stage detection method based on FasterR-CNN and Mask R-CNN and the one-stage target detection method based on YOLO. Among them, the two-stage detection algorithm has high accuracy, but the detection speed is slow, and it is not suitable for real-time inspection by drones; the one-stage target detection algorithm based on regression, such as YOLO and SSD algorithms, has high detection efficiency, but the detection accuracy cannot be guaranteed, and the false detection rate and missed detection rate will be higher.

Even so, the overall detection accuracy of power line and power tower detection methods currently used for applications such as power inspection is limited, and none of them fully meet the requirements of high reliability and high availability. The reasons for this are insufficient labeled training data, which leads to insufficient training of machine learning models, and the complexity and diversity of power lines, power towers, and the environment in which they are located, which leads to the poor versatility of traditional image processing methods. In particular, power inspection is a field-specific problem, and there is relatively little image data available, and even less labeled image data. Therefore, although data-driven machine learning methods are theoretically more effective, they lack sufficient data to train highly available detection models. In addition, existing methods usually perform power line detection or power tower detection separately, and do not fully utilize prior knowledge in the field of power transmission, such as the associated structural relationship between power lines and towers.

Summary of the invention

In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method for detecting power towers and power lines with associated correction of power towers and power lines, so as to at least solve at least one of the problems of insufficient training caused by insufficient image data and poor detection accuracy of power towers and power lines in complex environments.

In order to achieve the above object, the technical solution adopted by the present invention is:

A method for detecting power towers and power lines for correcting the correlation between power towers and power lines comprises the following steps:

Step 1, taking an image of a power tower and power lines from above, and performing grayscale preprocessing on the image;

Step 2, performing edge detection on the preprocessed image to obtain a binary edge map;

Step 3, using the edge map to perform power line detection, including line segment detection, merging line segments and parallel line group clustering in sequence, to obtain power lines represented by the parallel line group;

Step 4, using the edge map to detect the power tower, and obtaining the power tower area framed by the rectangular frame;

Step 5, mapping the rectangular frame to the edge map, excluding the background areas on both sides of the rectangular frame of the power tower, and re-detecting the power lines in the area inside the rectangular frame of the power tower; after obtaining the new power line detection results, the proportion of the power tower area in the edge map is estimated by the left and right widths of the power line distribution area, and then the power tower detection parameters are updated, and the power tower is re-detected until the requirements are met or the number of iterations is reached.

In one embodiment, in step 1, each image taken contains only one power tower target and power lines; the preprocessing includes adjusting image resolution, image grayscale, histogram equalization and denoising; wherein in the grayscale process, only the red and blue channels are retained to generate a grayscale image.

In one embodiment, in step 3, the two endpoints of the line segment in the edge map are detected using a probabilistic Hough transform; then the coordinates of the center point of the line segment, the slope of the line segment and the intercept are calculated, and the line segments are grouped according to their geometric distances. After grouping, the line segments in each group are fitted into straight lines using the least squares method and extended to run through the image, and finally the straight lines with similar slopes are grouped as parallel lines; finally, the parallel line groups are clustered according to the slopes, the straight lines with the most counts in the clustering results are retained, and the messy straight lines with large differences in slope from the power lines are filtered out.

In one embodiment, among the power lines represented by the obtained parallel line group, the width of the power tower is inferred from the distance D between the two outermost power lines, and then the area W where the power tower is located is estimated based on the aspect ratio of the power tower. According to the power line detection result, the edge map is rotated as a whole by α to make the power tower area in the map horizontal, where α is the angle between the line group obtained after clustering and the vertical direction.

In one embodiment, in step 4, a corner point detection method based on curvature scale space is used to perform corner point detection on the edge map to obtain a corner point distribution map; then, the corner point distribution map is integrally projected in the horizontal and vertical directions by pixel statistical projection to obtain projection histograms in the horizontal and vertical directions; and each projection value in the horizontal direction is replaced by the median of the n adjacent projections on the left and right, and each projection value in the vertical direction is replaced by the median of the m adjacent projections on the top and bottom to obtain a smoothed projection histogram by median smoothing; the proportion of the power tower in the edge map is estimated by the width and height of the power line area, and the projection histograms in the horizontal and vertical directions are divided into multiple small areas according to the proportion, and the height difference between each small area and the adjacent area is calculated, and the areas with the largest height difference have the left and right boundaries and the upper and lower boundaries of the power tower.

In one embodiment, the updating of the power tower detection parameters is to update the number of divisions of the small area.

In one embodiment, in step 5, the degree of change of the results during the iterative detection of power towers uses IoU as an evaluation indicator, and the number of iterations is determined based on IoU; the IoU is the result obtained by dividing the overlapping part of two areas by the collective part of the two areas, and is used to measure the degree of overlap of two power tower detection results; when IoU is greater than a set threshold t, the two detection results are considered to be consistent, the current result is output as the final result, and the task is terminated; when IoU is less than or equal to the threshold t, the next round of iterative detection is continued. If the number of iterations exceeds the given maximum number of iterations, the task is forced to end and the detection result of the last round is output.

The present invention also provides a power tower and power line detection system for power tower and power line correlation correction, comprising:

A pre-processing module performs grayscale pre-processing on the images of power towers and power lines taken from above;

The edge detection module performs edge detection on the preprocessed image to obtain a binary edge map;

A power line detection module, which uses the edge map to perform power line detection, including line segment detection, line segment merging and parallel line group clustering in sequence, to obtain power lines represented by parallel line groups;

A power tower detection module, which uses the edge map to detect power towers and obtains a power tower area defined by a rectangular frame;

The associated correction module maps the rectangular box to the edge map, excludes the background areas on both sides of the rectangular box of the power tower, and re-detects the power lines in the area inside the rectangular box of the power tower; after obtaining the new power line detection results, the proportion of the power tower area in the edge map is estimated by the left and right widths of the power line distribution area, and then the power tower detection parameters are updated, and the power tower is re-detected until the requirements are met or the number of iterations is reached.

In one embodiment, the detection system is deployed on a drone embedded platform to provide the drone flight control program with detection and positioning functions for the positions of power towers and power lines; the images of the power towers and power lines are captured by the drone.

Compared with the prior art method of independently detecting power lines and power towers, the present invention does not rely on large-scale data sets, has a faster detection speed, and correlates and corrects the detection results of power lines and power towers, resulting in better detection effects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the detection algorithm of the present invention.

FIG. 2 is a schematic diagram of power towers and power lines in an image after clustering according to the present invention.

FIG3 is a schematic diagram of the power tower area obtained after correction by the present invention.

FIG. 4 is a schematic diagram of corner point projection of the present invention.

FIG. 5 is a schematic diagram of power lines re-detected excluding areas on both sides of the power tower area according to the present invention.

FIG6 is a detection area obtained by the IOU (intersection over union) calculation method of the present invention, wherein the left figure is the intersection and the right figure is the union.

DETAILED DESCRIPTION

The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

Traditional target detection methods face the problem of insufficient data on the one hand, and the detection accuracy is still low in complex environments on the other hand. To this end, the present invention provides a power tower and power line detection method, which adopts an image processing route to effectively cope with the challenge of insufficient image data. At the same time, through the associated detection of power lines and towers, it can better cope with the detection accuracy problem of power towers and power lines in complex environments.

The power tower and power line detection of the present invention aims to detect the power tower and power line in the picture, wherein the power tower is framed by a rectangular frame around it, and the power line is marked by a straight line. According to the main processing flow of the present invention, referring to FIG1, it mainly includes the following steps:

Step 1, take images of power towers and power lines, and perform necessary preprocessing on the obtained images, which should at least include grayscale conversion. Generally speaking, the images should be taken from above the power towers and power lines.

Step 2: Perform edge detection on the preprocessed image to obtain a binary edge map.

Step 3, using the edge map to perform power line detection, including line segment detection, merging line segments and clustering parallel line groups in sequence, to obtain power lines represented by parallel line groups. That is, the two ends of the line segment are detected, then fitted into a straight line, and clustered according to the similarity of the slope.

Step 4: Use the edge map to detect the power tower and obtain the power tower area defined by the rectangular frame.

Step 5, mapping the rectangular frame to the edge map, excluding the background areas on both sides of the rectangular frame of the power tower, and re-detecting the power lines in the area inside the rectangular frame of the power tower; after obtaining the new power line detection results, the proportion of the power tower area in the edge map is estimated by the left and right widths of the power line distribution area, and then the power tower detection parameters are updated, and the power tower is re-detected until the requirements are met or the number of iterations is reached.

The power line detection in step 3, the power tower detection in step 4, and the associated correction in step 5 are performed iteratively, as shown in FIG1 .

The method of the present invention is mainly based on the above method. The present invention first uses a more efficient traditional image processing method to achieve the initial detection of power lines and power towers; then uses the power field knowledge such as the relationship between power lines and power towers to achieve the organic integration and mutual correction of line detection and tower detection in a step-by-step iterative manner. Finally, the above two problems are overcome at the same time or at least one of them is overcome.

Correspondingly, the present invention provides a power tower and power line detection system for power tower and power line association correction, comprising: a preprocessing module, an edge detection module, a power line detection module, a power tower detection module and an association correction module. The above steps 1 to 5 are respectively executed. In the following description, the execution content of the preprocessing module is consistent with step 1, the execution content of the edge detection module is consistent with step 2, the execution content of the power line detection module is consistent with step 3, the execution content of the power tower detection module is consistent with step 4, and the execution content of the association correction module is consistent with step 5.

The present invention can be specifically used for the detection of power towers and power lines in the image pictures collected by unmanned aerial vehicle power inspections, that is, the detection method or detection system is deployed on a drone embedded platform to provide the drone flight control program with the function of detecting and locating the positions of power towers and power lines. The unmanned aerial vehicle power inspection scenario described in the present invention is: after the unmanned aerial vehicle carries a camera and flies above the power towers and power lines, it shoots the power towers and power lines below from a bird's-eye view; considering that the distance between power towers is very far in the long-distance power transmission scenario, the drone field of view described in the present invention only includes one tower target, that is, each image taken includes only one power tower target, and power lines should also be included. The following is a detailed description of the present invention taking the execution content of each module as an example.

1. Preprocessing module (execute step 1)

The functions of the preprocessing module include adjusting image resolution, graying, histogram equalization, and denoising. The preprocessing module receives video data shot by the drone and decomposes the video into single-frame images for preprocessing. Usually, the images shot by the drone lens are high-resolution color RGB images, which are slow to process directly. Therefore, they are first scaled according to a certain ratio and adjusted to a resolution that is easy to process.

In the process of image grayscale conversion, the three channels of the RGB image must first be separated. UAV aerial images are usually based on the ground as the background, and the main content includes green vegetation, yellow land, blue-green waters, and silvery-white power lines under normal lighting conditions. The silvery-white color composition of the power line is RGB (192, 192, 192), which contains more blue (B) primary color information than the background color. In order to eliminate the influence of the complex ground environment on the extraction of power lines, the present invention only retains the red and blue channels to generate a grayscale image.

Since the power line detection is carried out outdoors, the environment or light changes may cause overexposure or underexposure, which will cause the overall dark image. It is also possible that the contrast of the target straight line in the image is different, which will affect the later image segmentation and power line extraction. Therefore, in the process of processing digital images, the grayscale histogram is used to count the number of occurrences of pixels with different grayscale levels. The horizontal axis of the histogram is 0-255 grayscale levels, and the vertical axis is the number of pixels with different grayscale levels appearing in the image. The grayscale histogram of the image can be used to observe the grayscale distribution state of the image. Histogram equalization is to stretch the image nonlinearly and redistribute the pixel values in the image to achieve the purpose of roughly equalizing the number of pixel values in a certain area. Because in an image, when the grayscale value distribution is more balanced, the amount of information contained in the image is greater, that is, the color level of the result obtained by histogram equalization is clearer, and the linear features in the image are more recognizable.

The image is then smoothed using methods such as Gaussian filtering. Gaussian filtering is a linear smoothing filter that is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. In layman's terms, Gaussian filtering is the process of weighted averaging the entire image. For each pixel value, the template is convolved and then the value of the template center is replaced by the value calculated by the weighted average.

2. Edge detection module (execute step 2)

Edge detection is a parallel boundary segmentation technology based on grayscale discontinuity, and is the first step of all boundary segmentation methods. The edge is the boundary between the target and the background, and edge extraction is an important step to distinguish the target from the background. In general, edge detection methods use the differences in color, texture, grayscale and other features between the background and the target to achieve it. Edge detection calculations are generally completed using first-order or second-order derivatives, but in actual digital images, derivatives are approximated by differential operations instead of differential operations. The grayscale values of points on both sides of the edge of the image change suddenly, so these points will have larger differential values. When the direction of the differential is perpendicular to the boundary, the differential value is the largest. Based on this feature, the edge of the image can be obtained. This module uses the Canny operator, which includes the following steps:

1) Gradient and direction calculation: Use the Sobel operator to calculate the gradient and direction of each pixel.

2) Non-maximum suppression: eliminates spurious responses caused by edge detection.

3) Double threshold: detect real and potential edges.

4) Hysteresis technique: edge detection and tracking boundaries are completed by suppressing weak edges.

Edge detection has two functions: the edge of the power line image is an important feature of the power line, and edge detection is a necessary step in power line detection; in addition, after extracting the edge of the image, the edge is filled, extended and smoothed, and corner detection can be performed based on the rate of change of pixels on the edge, thereby performing power tower detection.

3. Power line detection module (execute step 3)

The input of this module is the binary edge map obtained after edge detection. The steps of power line detection mainly include line segment detection, line segment merging, and parallel line group clustering.

1) Line segment detection uses probabilistic Hough transform. Hough transform is a feature extraction technology in image processing. It detects objects with specific shapes through a voting algorithm. This process calculates the local maximum of the cumulative results in a parameter space to obtain a set that conforms to the specific shape as the Hough transform result. It is a common method for detecting line segments in images. The standard Hough transform essentially maps the image to its parameter space. It needs to calculate all M edge points, so its computational complexity and required memory space will be very large. If only m (m<M) edge points are processed in the input image, the selection of these m edge points is probabilistic, which is the probabilistic Hough transform. In actual drone images, power lines are usually not displayed as clear, continuous long straight lines, but as many shorter broken line segments. An important feature of the probabilistic Hough transform is that it can detect the two endpoints of the line segment in the image and accurately locate the straight line in the image. Therefore, the present invention uses probabilistic Hough transform combined with distance measurement on the basis of traditional line detection algorithm to complete the grouping and merging of lines, that is, to detect the two endpoints of the line segment in the edge map, so that the broken line segment can be fitted into a straight line, thereby improving the integrity of power line detection.

2) Merge line segments: After obtaining the coordinates of the line segment endpoints through probabilistic Hough transform, calculate the coordinates of the line segment center point, the line segment slope and intercept, and group the line segments according to their geometric distance. After grouping, use the least squares method to fit the line segments in each group into straight lines, and extend the fitted straight lines to run through the image. Finally, the straight lines with similar slopes are grouped as parallel lines.

3) Clustering of parallel line groups: In images taken by drones, power lines usually present the following characteristics: approximately straight lines, with a certain range of inclination angles, approximately parallel to each other, running through the entire image, and vertically passing through the power tower area. Therefore, clustering parallel line groups based on slopes and retaining the straight lines with the most counts in the clustering results can filter out messy straight lines with large differences in slope from the power lines.

In an embodiment of the present invention, the K-means++ algorithm is used for clustering of parallel line groups. K-means is a commonly used clustering algorithm based on Euclidean distance. It believes that the closer the distance between two targets, the greater the similarity. The K-means++ algorithm optimizes the selection of the initial cluster center. The clustering process is roughly as follows: i) first randomly select K samples (K parallel line groups with different slopes) from the sample set (a set of parallel line groups, each sample is a parallel line group) as cluster centers, and calculate the distances between all samples and these K "cluster centers"; ii) for each sample, divide it into the cluster with the "cluster center" closest to it, and calculate the new "cluster center" of each cluster for the new cluster; iii) repeat the above process until the "cluster center" does not move.

As shown in Figure 2, after clustering, the angle α between the detected power line group and the vertical direction is calculated for image correction, and the position and size of the power tower area can be estimated based on the power line distribution area. Specifically, as shown in Figure 3, among the power lines represented by the obtained parallel line group, the width of the power tower can be inferred from the distance D between the two outermost power lines, and then the area W where the power tower is located can be estimated based on the aspect ratio of the power tower. The reason why the image needs to be corrected before power tower detection is that horizontal bounding boxes are usually used in target detection tasks to represent the approximate range of targets in the image, while objects in drone images are usually in any direction. Using horizontal bounding boxes to detect targets will cause this type of object detection box to contain many background areas and cannot reflect the size and aspect ratio of the target object. This not only increases the difficulty of the detection task, but also leads to inaccurate representation of the target range.

According to the structure of power towers and power lines, multiple power lines will vertically pass through a power tower area, while power towers are not densely distributed in unmanned aerial images, and there is only one power tower area in the image. Therefore, the present invention uses this feature to rotate the edge map as a whole by α according to the power line detection result, so that the power tower area in the image can be horizontal, making the subsequent power tower detection more accurate.

4. Power tower detection module (execute step 4)

The input of the power tower detection module is the edge map obtained by the edge detection module. This module uses corner detection (such as Curvature Scale Space Corner Detection, CSS corner detection method) on the edge map to obtain the corner distribution map, and then performs threshold judgment on the horizontal and vertical projections of the corners to determine the upper, lower, left and right boundaries. The basic principle is that there is a complex horizontal and vertical staggered steel frame structure inside the power tower, and the edge and corner density inside the tower is significantly greater than the background area. Therefore, this feature can be used to distinguish the tower area from the background area.

In fact, most existing methods directly use edge information to perform statistics on the density distribution of edge points for power tower detection. However, this method has some shortcomings. For example, the selection of high and low thresholds of the most commonly used Canny edge detection operator is not set according to image characteristics, but relies on prior experience to set the threshold. If the high and low thresholds are set large during the edge detection process, more image edge details will be lost, resulting in discontinuous edge detection results. If the high and low thresholds are set small, false edge detection will occur.

Therefore, in the present invention, for the internal structure of the power tower, on the basis of edge detection, further corner detection based on curvature scale space is performed, that is, Curvature Scale Space Corner Detection, CSS corner detection algorithm. The steps of the algorithm are as follows: first, the image contour extracted by the Canny edge detection operator and other methods is used to fill the gaps in the binary edge contour; second, after filling, the curvature of each pixel on the contour is calculated at a large scale, and if it exceeds the threshold, it is determined as a candidate corner point; finally, each pixel in the candidate corner point set is tracked at a small scale to accurately locate the position of the corner point.

After obtaining the corner distribution map of the image, the corner distribution map is projected in the horizontal direction by pixel statistical projection to obtain the projection histogram in the horizontal direction. In the projection histogram, the height of the projection represents the edge density in the horizontal direction, and the projection height is higher at the position with large edge density. At the same time, each projection value in the horizontal direction is replaced by the median of the n adjacent projections on the left and right to eliminate noise by median smoothing. The smoothing process is mainly to remove isolated noise points in the image while better retaining the details of the image. In this way, a relatively smooth projection histogram as shown in Figure 4 is obtained. In the projection histogram, the height of the projection column inside the power tower is much greater than the background area, and there will be obvious peaks at the edge of the power tower. The proportion of the power tower in the edge map can be estimated by the width of the power line area. The projection histogram is divided into multiple small areas according to the proportion, and the drop between each small area and the adjacent area is calculated. The area with the largest drop has the left and right boundaries of the power tower. Similarly, the projection histogram in the vertical direction can be projected in the vertical direction to determine the upper and lower boundaries of the power tower. That is, the corner point distribution map is integrally projected in the vertical direction through pixel statistical projection to obtain a projection histogram in the vertical direction; then each projection value in the vertical direction is replaced by the median of the m adjacent projections above and below through median smoothing to obtain a smoothed projection histogram; the proportion of the power tower in the edge map is estimated by the height of the power line area, and the projection histogram in the vertical direction is divided into multiple small areas according to the proportion, and the height difference between each small area and the adjacent area is calculated. The upper and lower boundaries of the power tower exist in the area with the largest height difference.

5. Association correction module (execute step 5)

After the above modules complete the initial detection of power lines and power towers, the power lines represented by the parallel line group and the power tower area framed by the rectangular frame can be obtained. Next, the association verification module performs iterative updates (i.e., iterative execution of steps 3, 4, and 5) based on the spatial structural association relationship between the power towers and power lines to correct the detection results of the power lines and power towers.

Specifically, the rectangular box representing the power tower detection result is first mapped to the edge map, the background area on both sides of the power tower rectangular box area is excluded, and the power lines are re-detected in the inner area of the power tower rectangular area, using the same method as the initial detection. Secondly, after obtaining the new power line detection results, the proportion of the power tower area in the image is estimated by the left and right widths of the power line distribution area, and then the parameters of the power tower detection module (that is, the number of small area divisions in the power tower detection module) are updated, and the histogram projection method is used again to detect the power tower.

The degree of change in the results during the iterative detection of power towers uses IoU (Intersection over Union) as an evaluation indicator, and the number of iterations is determined based on IoU. IoU is the result of dividing the overlapping part of two areas by the collective part of the two areas, and is used to measure the degree of overlap of two power tower detection results. When IoU is greater than the set threshold t, the two detection results are considered to be consistent, the current result is output as the final result, and the task is terminated; when IoU is less than or equal to the threshold t, the next round of iterative detection continues. If the number of iterations exceeds the given maximum number of iterations, the task is forced to end and the last round of detection results is output.

The following is a workflow of the detection algorithm. This workflow is a small drone transmitting images in real time, and the algorithm automatically detects power towers and power lines.

First, the drone was started, the camera and lidar were turned on, and it hovered over the power tower and took images with a resolution of 1600×1200 pixels. After receiving the image transmitted by the drone, the preprocessing module scaled the image to 800×600 resolution according to the ratio for subsequent processing. After scaling, the three RGB channels of the image were separated, and the red and blue channels were extracted to generate a grayscale image. The grayscale image was denoised using Gaussian filtering. The two-dimensional Gaussian function formula is as follows:

Two-dimensional Gaussian distribution:

In the formula, δ is the Gaussian distribution parameter, which can be calculated from the filter kernel size, and the image is filtered using a 3*3 template.

The preprocessed grayscale image is input to the edge detection module, and the Canny algorithm is used for edge detection to detect and identify the set of pixels with drastic brightness changes in the image. The Canny algorithm uses the Sobel operator.

The formula for calculating the convolution kernel is:

A is the original image.

The horizontal and vertical grayscale values of each pixel in the image are combined by the following formula to calculate the grayscale of the point:

G＝| _Gx |+| _Gy |

After edge detection, the edge map is used for power line detection and power tower detection. Power line detection uses probabilistic Hough transform to detect line segments, deletes and merges duplicate line segments by calculating the line segment slope and mutual distance, and classifies parallel line segments.

The algorithm flow of line segment detection and merging is as follows:

1) Obtain the coordinates of the line segment endpoints through probabilistic Hough transform.

2) Calculate the coordinates of the center point of the line segment, the slope and intercept of the line segment, and mark the status as UNUSED.

3) Select the line segment with the minimum ordinate among the endpoints as the initial line segment L _j , and the geometric distance from this point to the center point of other line segments _Li is _Di.

4) Traverse the UNUSED line segments in the line segment set, set the distance threshold to W _θ , if D _i <W _θ , store the line segment in the line segment group K _j , and set the line segment status to USED. If all line segments are traversed, repeat step 2) until all line segments are USED.

5) Obtain the endpoint coordinates of each line segment in the line segment group _Kj , fit the line segments using the least squares method, and extend the fitted straight line to run through the entire image.

6) Compare the slopes of the fitted lines one by one. The lines whose slope difference is less than the threshold S _θ are classified as a parallel line group, and the average slope is used to represent the parallel line group.

After classification, the parallel line groups are clustered according to the slope, interfering straight lines are excluded, and finally the power lines are detected. The clustering uses the Kmeans++ algorithm, and the steps are as follows:

1) Select K parallel line groups as the initial cluster center points {C ₁ , C ₂ , C ₃ ,…, C _k }, that is, the initial cluster center, where _Ci represents the slope of the i-th parallel line group.

2) Calculate the Euclidean distance from each sample point _Xi to the K cluster centers represented by each parallel line group, find the cluster center closest to the point, and assign it to the corresponding cluster. The calculation of the Euclidean distance is as follows:

3) After all points are assigned to clusters, all M sample points (parallel line groups) are divided into K clusters {S ₁ ,S ₂ ,S _3, …,S _k }. Then recalculate the centroid (average distance center) of each cluster and set it as the new "cluster center". The calculation of the "cluster center" is as follows:

4) Repeat steps 2-3 until the “cluster center” no longer changes.

Theoretically, in the images taken by drones, the number of line segments belonging to power lines is the largest, and the slopes of these line segments are almost the same. Therefore, through the above clustering process, the cluster with the largest number of line segments should be the set of line segments corresponding to power lines. Next, select a cluster with the most parallel line segments from all the above clusters, and calculate the average slope α of all line segments in the cluster, which is the inclination angle of the power lines in the current picture. Subsequently, the image is rotated by an angle α according to the slope α of the power lines so that the power lines are perpendicular to the horizontal direction. At the same time, since the power lines should be approximately perpendicular to the outer rectangular frame of the power tower in the actual scene, after the above rotation operation, the power tower area should be as shown in Figure 3, with the left and right sides of the power tower parallel to the vertical direction.

At the same time, the edge map of the original image is also rotated accordingly to obtain the rotation-corrected edge map, and then CSS corner point detection is performed. The process of corner point detection is as follows:

1) For the edge contour extracted by the Canny edge detection operator, when the edge point is an endpoint, if there are other endpoints in the neighborhood of the endpoint, the non-edge point between the two endpoints is replaced by the edge point; if there are other edge lines in the neighborhood of the endpoint, it is marked as a T-node.

2) For each edge contour, a large-scale Gaussian filter function is used to calculate the curvature of each pixel on the contour.

3) If the curvature of a pixel on the edge line is greater than the preset threshold and is the only local maximum value, and its curvature value is greater than twice the minimum curvature value in the neighborhood, then the pixel is marked as a candidate corner point. The curvature is calculated as follows:

The edge curve is defined according to the arc length coefficient u:

Γ(u)＝(x(u),y(u))

The extracted edge curve is smoothed and denoised using a one-dimensional Gaussian filter function to obtain a smooth curve:

Γ(u,σ)＝(X(u,σ),Y(u,σ))

表示卷积操作，g(u,σ)表示标准差为σ的一维高斯滤波函数。根据边缘曲线函数可以求得曲线的曲率函数：in

represents the convolution operation, g(u,σ) represents the one-dimensional Gaussian filter function with a standard deviation of σ. The curvature function of the curve can be obtained according to the edge curve function:

where g′(u,σ) and g″(u,σ) are the first-order derivative and second-order partial derivative of g(u,σ) with respect to u, respectively.

4) Since the large-scale filter used in 2) highly blurs the curve, only a roughly screened set of candidate corner points is obtained. Therefore, a small-scale Gaussian filter is needed to track each pixel in the candidate corner point set to accurately locate the position of the corner point and improve the accuracy of corner point positioning.

5) For the obtained T-node and the detected candidate corner point, if the two are adjacent, delete one of them.

After obtaining the binary corner map through corner detection, the corner map is integrally projected in the horizontal and vertical directions to obtain the projection histograms in the horizontal and vertical directions. Taking the vertical direction as an example, the number of corner points in each column of the image is accumulated during projection, and the number of corner points is represented by the column height, which can intuitively represent the distribution of corner points in the vertical direction.

After obtaining the integral projection histogram, the idea of median filtering is used to smooth and denoise the projection histogram. For example, a sliding window of 9 pixels is used to traverse the projection map, and the heights of the projection columns in each column are saved to form a sequence. The heights of the projection columns in the sequence are sorted from large to small, and the height of the projection column in the middle position, i.e., the median value, is saved and assigned to the pixel point in the middle position of the sliding window. In this way, a relatively smooth corner projection map can be obtained.

The width of the power line distribution can be obtained from the power line detection as W, which can be roughly regarded as the width of the power tower. Combined with the adjusted image size X ₀ *Y _0, the projection image will be divided into R (x, y) small areas in the horizontal and vertical directions after denoising. The calculation method is as follows:

的情况为例，将第一个小区域内每列(行)的角点数量{p₀₀,p₀₁,…,p_0m}累加，其中

用区域内角点总量P表示该区域的角点密度。角点数量P的计算如下：by

Take the case of as an example, the number of corner points in each column (row) in the first small area {p ₀₀ ,p ₀₁ ,…,p _0m } is accumulated, where

The total number of corner points in the region, P, is used to represent the corner point density of the region. The number of corner points, P, is calculated as follows:

After obtaining {P ₀ ,P ₁ ,P ₂ ,…,P ₁₉ }, calculate the absolute value of the difference between each small area and the adjacent area:

D _i = |P _i+1 -P _i |

There is one side boundary of the power tower between the two areas corresponding to the maximum difference max{D ₀ ,D ₁ ,D ₂ ,…,D _i }.

Similarly, projection in the vertical direction can determine the upper and lower boundaries of the power tower.

At the same time, several parallel line groups obtained by corner point map and line segment detection are subjected to association analysis to remove irrelevant parallel line groups. The association analysis here mainly takes into account that power lines are generally connected to certain bracket positions of power towers, and these bracket positions are usually a corner point. Therefore, candidate straight lines that cannot be power lines can be further eliminated by judging whether the detected straight line passes through the corner point in the corner point map. Specifically, for all parallel line groups, after being extended into straight lines spanning the entire image through minimum binarization, it is judged in turn whether the straight line passes through any corner point in the corner point map. Parallel lines that do not pass through any corner point are removed and are no longer possible choices for power lines.

After the initial inspection of power lines and power towers is completed, the results are corrected through iterative inspection. The iterative process is as follows:

1) Map the power tower detection results to the edge detection map.

2) As shown in FIG. 5 , the power lines are re-detected after excluding the areas W ₁ and W ₂ on both sides of the power tower area.

3) Estimate the proportion of the power tower area in the image based on the left and right widths of the power line distribution.

4) Based on the results of 3), the projection method is used again to detect power towers.

5) After the power tower is re-detected, the IoU value with the previous power tower detection result is calculated. The IoU threshold is set to 0.7. After correction, compare with the previous detection result. If IoU is greater than 0.7, it means that the two detection results are consistent, output the result, and end the task; if it is less than or equal to 0.7, it means that there is a deviation from the previous detection result, and continue to iterate until the results are consistent or the number of iterations is exceeded. _{B 1} and B ₂ are the areas of the two detections as shown in Figure 6, and the calculation of IoU is as follows:

Claims (10)

1. A method for detecting power towers and power lines for correcting the association between power towers and power lines, characterized by comprising the following steps: Step 1, taking an image of a power tower and power lines from above, and performing grayscale preprocessing on the image; Step 2, performing edge detection on the preprocessed image to obtain a binary edge map; Step 3, using the edge map to perform power line detection, including line segment detection, merging line segments and parallel line group clustering in sequence, to obtain power lines represented by the parallel line group; Step 4, using the edge map to detect the power tower, and obtaining the power tower area framed by the rectangular frame; Step 5, correcting the detection results of power lines and power towers through iterative detection, the method is as follows: mapping the rectangular frame to the edge map, excluding the background areas on both sides of the rectangular frame of the power tower, and re-detecting the power lines in the area inside the rectangular frame of the power tower; after obtaining the new power line detection results, the proportion of the power tower area in the edge map is estimated by the left and right widths of the power line distribution area, and then the power tower detection parameters are updated, and the power tower is re-detected until the requirements are met or the number of iterations is reached. 2. According to the power tower and power line associated correction method of claim 1, it is characterized in that in the step 1, each image taken contains only one power tower target and power lines; the preprocessing includes adjusting the image resolution, image grayscale, histogram equalization and denoising; wherein in the grayscale process, only the red and blue channels are retained to generate a grayscale image. 3. According to the power tower and power line detection method with associated correction of power towers and power lines as described in claim 1, it is characterized in that, in step 3, the two endpoints of the line segment in the edge map are detected using probabilistic Hough transform; then the coordinates of the center point of the line segment, the slope of the line segment and the intercept of the line segment are calculated, and the line segments are grouped according to their geometric distances. After grouping, the line segments in each group are fitted into straight lines using the least squares method and extended to pass through the image, and finally the straight lines with similar slopes are classified into parallel line groups; finally, the parallel line groups are clustered according to the slopes, the straight lines with the most counts in the clustering results are retained, and the messy straight lines with large differences in slope from the power lines are filtered out. 4. The method for detecting power towers and power lines with associated correction of power towers and power lines according to claim 3 is characterized in that, among the power lines represented by the obtained parallel line group, the width of the power tower is inferred from the distance D between the two outermost power lines, and then the area W where the power tower is located is estimated based on the aspect ratio of the power tower, and the edge map is rotated as a whole by α according to the power line detection result so that the power tower area in the map is horizontal, wherein α is the angle between the line group obtained after clustering and the vertical direction. 5. According to the power tower and power line detection method for power tower and power line association correction described in claim 4, it is characterized in that, in step 4, the edge map is detected by using a corner point detection method based on curvature scale space to obtain a corner point distribution map; then the corner point distribution map is integrally projected in the horizontal direction and the vertical direction by pixel statistical projection to obtain the projection histograms in the horizontal direction and the vertical direction; then each projection value in the horizontal direction is replaced by the median of the n adjacent projections on the left and right, and each projection value in the vertical direction is replaced by the median of the m adjacent projections on the upper and lower sides to obtain a smoothed projection histogram; the proportion of the power tower in the edge map is estimated by the width and height of the power line area, and the projection histograms in the horizontal direction and the vertical direction are divided into multiple small areas according to the proportion, and the height difference between each small area and the adjacent area is calculated, and the area with the largest height difference has the left and right boundaries and the upper and lower boundaries of the power tower. 6. According to the power tower and power line detection method with associated correction of power towers and power lines as described in claim 5, it is characterized in that the corner point detection method based on curvature scale space includes: first, using the image contour extracted by edge detection to fill the gaps in the binary edge contour; second, after filling, calculating the curvature of each pixel point on the contour at a large scale, and if it exceeds a threshold, it is determined to be a candidate corner point; finally, tracking each pixel point in the candidate corner point set at a small scale to accurately locate the position of the corner point. 7. The method for detecting power towers and power lines with correction of power tower and power line association according to claim 5, wherein the updating of power tower detection parameters is to update the number of divisions of the small area. 8. The method for detecting power towers and power lines with associated correction of power towers and power lines according to claim 1 or 6 is characterized in that, in step 5, the degree of change of the results in the iterative detection process of the power towers is evaluated using IoU as an indicator, and the number of iterations is determined based on IoU; the IoU is the result obtained by dividing the overlapping part of two areas by the collective part of the two areas, and is used to measure the degree of overlap of two power tower detection results; when IoU is greater than a set threshold t, the two detection results are considered to be consistent, the current result is output as the final result, and the task is terminated; when IoU is less than or equal to the threshold t, the next round of iterative detection is continued. If the number of iterations exceeds the given maximum number of iterations, the task is forced to end and the detection result of the last round is output. 9. A power tower and power line detection system for power tower and power line correlation correction, comprising: A pre-processing module performs grayscale pre-processing on the images of power towers and power lines taken from above; The edge detection module performs edge detection on the preprocessed image to obtain a binary edge map; A power line detection module, which uses the edge map to perform power line detection, including line segment detection, line segment merging and parallel line group clustering in sequence, to obtain power lines represented by parallel line groups; A power tower detection module, which uses the edge map to detect power towers and obtains a power tower area defined by a rectangular frame; The associated correction module maps the rectangular frame to the edge map, excludes the background areas on both sides of the rectangular frame of the power tower, and re-detects the power lines in the area inside the rectangular frame of the power tower; after obtaining the new power line detection results, the proportion of the power tower area in the edge map is estimated by the left and right widths of the power line distribution area, and then the power tower detection parameters are updated, and the power tower is re-detected until the requirements are met or the number of iterations is reached. 10. The power tower and power line detection system for power tower and power line correlation correction according to claim 9 is characterized in that the detection system is deployed on an unmanned aerial vehicle embedded platform to provide the unmanned aerial vehicle flight control program with detection and positioning functions for the positions of power towers and power lines; the images of the power towers and power lines are acquired by photographing the unmanned aerial vehicle.

Images