WO2024154504A1 - Information processing method, information processing system, and program - Google Patents

Abstract

In this information processing method for extracting straight lines corresponding to a plurality of multi-conductor type power lines from a captured image captured by an unmanned aerial vehicle, a computer executes: a step (S102) for employing an edge image generating unit to generate an edge image from the captured image; and a step (S103) for employing a specific straight line extracting unit to extract, from among a plurality of straight lines in the edge image, an expected number of straight lines meeting an extraction condition for straight lines forming the plurality of power lines, on the basis of straight line vote number information acquired by subjecting edges in the generated edge image to a Hough transform, and straight line extraction condition information indicating the extraction condition, which includes at least a condition relating to expected number information indicating the expected number of the plurality of power lines expected to be in the edge image.

Description

Information processing method, information processing system, and program

The present invention relates to an information processing method, an information processing system, and a program.

In recent years, drones, unmanned aerial vehicles (UAVs), and other flying objects (hereinafter collectively referred to as "flying objects") have begun to be used in industry. In this context, Patent Document 1 discloses a system that uses flying objects to photograph and inspect power lines.

JP 2020-196355 A

Here, for example, when identifying a target power line in order to track the power line under inspection, one possible method is to perform edge detection on the captured image to extract the edges of the power line and identify the position of the power line. However, since various power lines and linear structures such as buildings and roads are reflected in the background of the captured image, edge detection may extract a large number of edges that become noise, making it difficult to identify the edges of the target power line. In particular, if the edge of a linear object (especially a long and thin object) in the background that is not the subject of inspection is extracted, it may end up being tracked, causing the target power line to be lost and making it difficult to perform a proper inspection.

In addition, although Patent Document 1 does not specifically mention this, the power lines strung on each cross arm of a support such as a steel tower that supports the power lines may each consist of a single power line (single conductor system), but may also consist of a multi-conductor system in which each cross arm is composed of two or more power lines (conductors). When performing edge detection as described above on a multi-conductor power line, it is necessary to identify which of the edges of multiple linear objects (particularly long and thin objects) extracted in the captured image are the target power line, which is more difficult to identify than identifying a single target power line.

The present invention was made in light of this background, and one of its objectives is to provide an information processing method, information processing system, and program that can extract straight lines corresponding to power lines of the multi-conductor type, in addition to single-conductor type, from images captured by an aircraft.

According to one aspect of the present invention, there is provided an information processing method for extracting straight lines corresponding to a plurality of multi-conductor power lines from an image captured by an unmanned aerial vehicle, the information processing method including: a step of generating an edge image from the captured image by an edge image generating unit; and a step of extracting, by a specific straight line extracting unit, from the straight lines in the edge image, straight line extraction condition information indicating extraction conditions for the straight lines constituting the plurality of power lines, the straight line extraction condition information including at least conditions related to straight line voting number information obtained by a Hough transform for edges in the generated edge image and expected number information indicating the expected number of straight lines of the plurality of power lines expected in the edge image, based on the expected number of straight lines that fall under the extraction conditions.

The present invention provides an information processing method, information processing system, and program that can extract straight lines corresponding to multiple multi-conductor power lines from images captured by an aircraft.

1 is a diagram showing an overall configuration of an embodiment of the present invention; 1 is a diagram showing a system configuration of an information processing system according to an embodiment of the present invention; FIG. 3 is a block diagram showing a hardware configuration of the terminal of FIG. 2. FIG. 3 is a block diagram showing a hardware configuration of the server in FIG. 2 . A block diagram showing the hardware configuration of the unmanned aerial vehicle of Figure 2. FIG. 3 is a block diagram showing the functions of the terminal and the server in FIG. 2 . This is an example of an image acquired from the imaging unit of an unmanned aerial vehicle. 8 is an example of an edge image generated from the image of FIG. 7. 11 is another example of an edge image. 13 is an example of an image in which a straight line estimated using straight line voting number information is visualized. 13 is an example of an image in which an extracted specific straight line is visualized. 13 is an example of an image in which an extracted specific straight line is visualized. 13 is an example of an image in which an extracted specific straight line is visualized. 1 is an example of an image in which a tracking target position is visualized. 1 is a flowchart showing a process for implementing an edge extraction method by the information processing system according to the present embodiment.

The contents of the embodiments of the present invention will be listed and described below. An information processing method, an information processing system, and a program according to the embodiments of the present invention have the following configuration.
[Item 1]
An information processing method for extracting straight lines corresponding to multiple power lines of a multi-conductor system from an image captured by an unmanned aerial vehicle, comprising:
generating an edge image from the captured image by an edge image generating unit;
a step of extracting lines corresponding to the assumed number of lines that fall under extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines constituting the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by a Hough transform for edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image, from the multiple lines in the edge image by a specific line extraction unit;
An information processing method comprising:
[Item 2]
The number of assumed straight lines is twice the number of the plurality of power lines,
The extraction condition is that the number of lines parallel to the first line having the largest number of line votes is the number of lines assumed to be the number of lines including the first line.
2. The information processing method according to item 1,
[Item 3]
The method further includes a step of detecting, by the edge image generation, a portion in which an amount of change in pixel value increases and is greater than a reference value in each pixel row in a predetermined direction as rising edge information, and detecting a portion in which an amount of change in pixel value decreases and is greater than a reference value as falling edge information,
The number of assumed straight lines is the same as the number of the plurality of power lines,
The extraction condition is that a line parallel to a first line having the largest number of line votes in either one of the lines corresponding to the rising edge information or the lines corresponding to the falling edge information forms a plurality of pairs with the other of the lines corresponding to the rising edge information or the lines corresponding to the falling edge information located nearby the first line, and the number of pairs is the expected number of lines.
2. The information processing method according to item 1,
[Item 4]
The method further includes a step of determining, by a gradient determination unit, which of the gradients of the plurality of straight lines detected by the specific straight line extraction unit is a first gradient that is most prevalent, by gradient voting;
The number of assumed straight lines is twice the number of the plurality of power lines,
the extraction condition is that at least the number of assumed straight lines is a straight line having a slope within a predetermined range with respect to the first slope determined by the slope determination unit;
The specific line extraction unit further includes a step of extracting lines, the number of which is equal to the number of expected lines, from lines having a slope within a predetermined range with respect to the determined most common slope in descending order of the number of line votes.
2. The information processing method according to item 1,
[Item 5]
detecting, by the edge image generation, a portion in each pixel row in a predetermined direction where the amount of change in pixel value increases is greater than a reference value as rising edge information, and detecting, by the edge image generation, a portion in each pixel row in a predetermined direction where the amount of change in pixel value decreases is greater than a reference value as falling edge information;
and determining, by a gradient determining unit, which of the gradients is a first gradient having the most prevalent gradient in a straight line corresponding to either the rising edge information or the falling edge information, by gradient voting;
The assumed number of straight lines is the number of the plurality of power lines,
the extraction condition is that a plurality of pairs are formed by a straight line corresponding to either the rising edge information or the falling edge information, the straight line being determined to have a gradient within a predetermined range with respect to the first gradient determined by the determining unit, and a straight line corresponding to the other of the rising edge information or the falling edge information in the vicinity of the straight line, and the number of pairs is at least the number of the assumed straight lines;
extracting lines corresponding to the pairs of the expected number of lines from the pairs in descending order of the number of line votes by the specific line extraction unit,
2. The information processing method according to item 1,
[Item 6]
a step of detecting, by the edge image generation, a portion in each pixel row in a predetermined direction where a change amount in pixel value increases is greater than a reference value as rising edge information, and detecting a portion in each pixel row in a predetermined direction where a change amount in pixel value decreases is greater than a reference value as falling edge information;
and determining, by a gradient determining unit, which of a plurality of pairs of gradients formed by a straight line corresponding to either the rising edge information or the falling edge information and a straight line adjacent thereto corresponding to the other of the rising edge information or the falling edge information is a first gradient having the most number of gradients by gradient voting,
The assumed number of straight lines is the number of the plurality of power lines,
the extraction condition is that the number of pairs determined to have a gradient within a predetermined range with respect to the first gradient determined by the determining unit is at least the number of assumed straight lines;
extracting lines corresponding to the pairs of the expected number of lines from the pairs in descending order of the number of line votes by the specific line extraction unit,
2. The information processing method according to item 1,
[Item 7]
The method further includes a step of setting a target position to be tracked by the image capture unit of the unmanned aerial vehicle based on the straight line extracted by the specific straight line extraction unit, by a tracking target position setting unit.
7. The information processing method according to any one of items 1 to 6.
[Item 8]
An information processing system for extracting straight lines corresponding to multiple power lines of a multi-conductor type from an image captured by an unmanned aerial vehicle,
an edge image generating unit that generates an edge image from the captured image;
a specific line extraction unit that extracts lines that correspond to the assumed number of lines that fall under the extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines that constitute the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by performing a Hough transform on edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image; and
An information processing system comprising:
[Item 9]
A program for causing a computer to execute an information processing method for extracting straight lines corresponding to a plurality of multi-conductor power lines from an image captured by an unmanned aerial vehicle, comprising:
The information processing method includes:
generating an edge image from the captured image by an edge image generating unit;
a step of extracting lines corresponding to the assumed number of lines that fall under extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines constituting the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by a Hough transform for edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image, from the multiple lines in the edge image by a specific line extraction unit;
A program that executes the following.

<Details of the embodiment>
Hereinafter, an information processing system according to an embodiment of the present invention will be described. In the accompanying drawings, identical or similar elements are given identical or similar reference symbols and names, and duplicated descriptions of identical or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment may be applied to other embodiments as long as they are not mutually inconsistent.

<Outline of this embodiment>
As shown in Fig. 1, the information processing system in this embodiment extracts straight lines corresponding to a plurality of power lines (particularly multi-conductor power lines, for example, two, three, four, six, etc. power lines separated by span spacers) from an image captured along the plurality of power lines extending side by side, for example, on a support (for example, a steel tower, etc.). As an example, images of the plurality of power lines may be captured by remotely controlling a camera mounted on an unmanned aerial vehicle 4 as shown in Fig. 1, which flies autonomously or remotely, based on instructions from a terminal 1 owned by a user.

More specifically, the information processing system in this embodiment generates an edge image from an image captured by an aircraft, and makes it possible to extract from the edge image an estimated number of straight lines that fall under the straight line extraction conditions, based on straight line voting information obtained by a Hough transform of edges in the edge image and straight line extraction conditions indicating extraction conditions for straight lines that make up power lines based on estimated number information indicating the estimated number of straight lines related to power lines estimated in the captured image.

In this way, by using information about the expected number of straight lines associated with power lines expected in the captured image (especially the expected number of straight lines corresponding to twice the number of power lines in a multi-conductor system), it is possible to extract an appropriate number of straight lines from within the captured image.

<System Configuration>
As shown in FIG. 2, the information processing system in this embodiment has a terminal 1, a server 2, and an unmanned aerial vehicle 4. The terminal 1, the server 2, and the unmanned aerial vehicle 4 may be connected to each other so as to be able to communicate with each other via a network NW. The illustrated configuration is an example, and is not limited to this, and for example, the unmanned aerial vehicle 4 may not be connected to the network NW. In that case, the unmanned aerial vehicle 4 may be operated by a transmitter (so-called radio control) operated by a user, or image data acquired by a camera of the unmanned aerial vehicle 4 may be stored in an auxiliary storage device (for example, a memory card such as an SD card or a USB memory) connected to the unmanned aerial vehicle 4, and the image data may be read out from the auxiliary storage device to the terminal 1 or the server 2 and stored by the user after the fact, and the unmanned aerial vehicle 4 may be connected to the network NW only for the purpose of operation or for the purpose of storing image data.

<Hardware configuration of terminal 1>
3 is a diagram showing a hardware configuration of the terminal 1 in this embodiment. Note that the illustrated configuration is an example, and the terminal 1 may have other configurations.

The terminal 1 includes at least a processor 10, a memory 11, a storage 12, a transmission/reception unit 13, an input/output unit 14, etc., which are electrically connected to each other via a bus 15. The terminal 1 may be, for example, a general-purpose computer such as a workstation or a personal computer.

The processor 10 is a computing device that controls the operation of the entire terminal 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and deployed in the memory 11 to perform various information processing operations.

Memory 11 includes a main memory consisting of a volatile storage device such as DRAM (Dynamic Random Access Memory) and an auxiliary memory consisting of a non-volatile storage device such as a flash memory or HDD (Hard Disc Drive). Memory 11 is used as a work area for processor 10, and also stores the BIOS (Basic Input/Output System) that is executed when terminal 1 is started up, various setting information, etc.

Storage 12 stores various programs such as application programs. A database that stores data used for each process may be constructed in storage 12. In addition, a storage unit 130, which will be described later, may be provided in part of the storage area.

The transmission/reception unit 13 is a communication interface that enables the terminal 1 to communicate with an external device (not shown) or the unmanned aerial vehicle 4 via a communication network. The transmission/reception unit 13 may further include a Bluetooth (registered trademark) or BLE (Bluetooth Low Energy) short-range communication interface, a USB (Universal Serial Bus) terminal, etc.

The input/output unit 14 includes information input devices such as a keyboard and mouse, and output devices such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

<Server 2>
4 also includes a processor 20, a memory 21, a storage 22, a transmission/reception unit 23, an input/output unit 24, etc., which are electrically connected to each other via a bus 25. The server 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The functions of each element can be configured in the same way as the terminal 1 described above, so detailed description of each element will be omitted.

<Unmanned Aerial Vehicle 4>
5 is a block diagram showing a hardware configuration of the unmanned aerial vehicle 4. The flight controller 41 can have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)).

Flight controller 41 also has and can access memory 411. Memory 411 stores logic, code, and/or program instructions that the flight controller can execute to perform one or more steps. Flight controller 41 may also include sensors 412, such as inertial sensors (accelerometers, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

The memory 411 may include, for example, a separable medium such as an SD card or random access memory (RAM) or an external storage device. Data acquired from the camera/sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image/video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited to this, and may be recorded in at least one of the terminal 1 and the server 2 from the camera/sensor 42 or the internal memory via the network NW. The camera 42 is installed on the unmanned aerial vehicle 4 via a gimbal 43.

The flight controller 41 includes a control module (not shown) configured to control the state of the unmanned aerial vehicle 4. For example, the control module controls the propulsion mechanism (motor 45, etc.) of the unmanned aerial vehicle 4 via an ESC 44 (Electric Speed Controller) to adjust the spatial arrangement, speed, and/or acceleration of the unmanned aerial vehicle 4, which has six degrees of freedom (translational motion _x , _y, and _z , and rotational motion θ x , θ y, and θ z ). The propellers 46 rotate by the motor 45 powered by a battery 48, thereby generating lift for the unmanned aerial vehicle 4. The control module can control one or more of the states of the onboard units and sensors.

The flight controller 41 can communicate with a transceiver 47 configured to transmit and/or receive data from one or more external devices (e.g., a transceiver 49, a terminal 1, a display device, or other remote control). The transceiver 49 can use any suitable communication means, such as wired or wireless communication.

For example, the transceiver unit 47 can utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, etc.

The transmitter/receiver 47 can transmit and/or receive one or more of the following: data acquired by the sensors 42, processing results generated by the flight controller 41, specific control data, user commands from a terminal or a remote controller, etc.

The sensors 42 in this embodiment may include inertial sensors (accelerometers, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), or vision/image sensors (e.g., cameras).

<Functions of Terminal 1>
FIG. 6 is a block diagram illustrating functions implemented in the terminal 1 and the server 2. In this embodiment, the terminal 1 includes an image acquisition unit 115, a processing unit 120, and a storage unit 130. The processing unit 120 includes an edge image generation unit 121, a specific straight line extraction unit 122, a slope determination unit 123, and a tracking target position setting unit 124. The storage unit 130 includes an information/image storage unit 131. Note that the various functional units are illustrated as functional units in the processor 10 of the terminal 1, but some or all of the various functional units may be realized in any of the configurations of the processor 10 of the terminal 1 or the processor 20 of the server 2, and the controller 41 of the unmanned aerial vehicle 4, depending on the capabilities of the processor 10 of the terminal 1 or the processor 20 of the server 2, and the controller 41 of the unmanned aerial vehicle 4.

The communication unit 110 communicates with the server 2 and the unmanned aerial vehicle 4 via the network NW. The communication unit 110 also functions as a reception unit that receives various requests and data from the server 2, the unmanned aerial vehicle 4, etc.

The image acquisition unit 115 acquires images captured by a digital camera mounted on the unmanned aerial vehicle 4 or a digital camera used by a user from the digital camera, for example, by wireless communication via a communication interface or wired communication via a USB terminal or the like. The image acquisition unit 115 may be configured to acquire images via a storage medium such as a USB memory or SD memory, but it is more preferable to configure the image acquisition unit 115 to acquire images in real time, particularly by wireless communication from the unmanned aerial vehicle 4 or within the unmanned aerial vehicle 4.

The processing unit 120 has various functional units that execute a series of processes to extract straight lines corresponding to multiple power lines from an image acquired by the image acquisition unit 115 (for example, an image captured along multiple power lines extending alongside one another, or an image that has been grayscaled or binarized).

The edge image generating unit 121 detects edge information in the image acquired by the image acquiring unit 115, and generates an edge image based on the edge information (including the xy coordinates in the image where the detected edge exists). The edge detection method may be any known method (such as a differential filter) as long as it is possible to determine the outer edge (edge) of the power line in the image, but may also be a method in which a portion where the amount of change in pixel value (luminance) increases is greater than a reference value in each pixel row in a specified direction in the image is detected as rising edge information, and a portion where the amount of change in pixel value (luminance) decreases is greater than a reference value is detected as falling edge information. For example, when an image is binarized, the pixel value information is set to 1 for white and 0 for black, and when the image is grayscaled, the pixel value information is set to 255 for the whitest pixel and 0 for the blackest pixel, and adjacent pixels in each pixel row in a specified direction in the image are compared to calculate the difference value, and as described above, pixels with difference values equal to or greater than a preset reference value are set as white, and other pixels are set as black, and an edge image can be generated. For example, when generating the edge image of FIG. 8 from the captured image of FIG. 7, if rising edges and falling edges are to be determined as described above, each edge can be visualized in the edge image as shown in FIG. 8 by setting, for example, rising edges to green, falling edges to blue, and other edges to black.

In the example of FIG. 8, not only the edges of the power lines but also the edges of grass and the like in the background are detected in detail, making it difficult to extract only edge information related to multiple power lines from such noise. In particular, as in the example of FIG. 9, when not only the trees and grass in the background but also linear objects (particularly elongated objects) with a similar shape and structure to the power lines are reflected (for example, when four linear objects extending diagonally are power lines and two linear objects extending to the left and right are unrelated elongated objects), the detection of more linear edges makes it even more difficult to extract only edge information related to multiple power lines from the image. Therefore, the processing unit 120 of the present invention further has the following functional units.

The specific line extraction unit 122 extracts lines that correspond to the expected number of lines that fall under the extraction conditions from the detected lines, based on line extraction condition information indicating the extraction conditions for lines that constitute the power lines, which includes at least conditions related to line vote number information obtained by performing a Hough transform on edge information in the generated edge image and expected number information indicating the expected number of lines related to the power lines expected in the edge image.

More specifically, the line vote number information is obtained by first executing a Hough transform on the edge information in the edge image by the specific line extraction unit 122. The Hough transform is a method for detecting lines contained in an image by majority vote. For example, when a perpendicular line is drawn from the origin to a certain line, ρ is the length, and θ is the angle between the perpendicular line and the x-axis, the xy coordinate information of each point constituting an edge in the image is used to vote for the ρθ coordinate, line vote number information indicating the number of votes is obtained, and a combination of ρθ indicating a line is estimated by majority vote from the line vote number information. In particular, when distinguishing between rising edge information and falling edge information, for example, a rising edge image in which only the rising edges are extracted and a falling edge image in which only the falling edges are extracted are generated, and a Hough transform is performed on each image to obtain line vote number information in which the lines corresponding to the rising edges and the lines corresponding to the falling edges can be distinguished from each other.

As a result, when the specific straight line extraction unit 122 refers only to the straight line vote number information, for example, straight lines as shown in FIG. 10 (eight lines extending diagonally and four lines extending to the left and right) are estimated by majority vote from the edge image illustrated in FIG. 9. Note that since the Hough transform obtains "straight lines" that continue infinitely and not "line segments" of finite length, it is possible to extract (estimate) straight lines that make up power lines even when edges are interrupted, while even an unrelated long and thin object that is interrupted in the background as shown in FIG. 9 can be extracted as a straight line.

Next, the expected number information indicating the expected number of straight lines related to the power lines expected in the edge image is information that can be set by the user via the terminal 1 or the like. The expected number information (i.e., the expected number of straight lines) may be information indicating twice the number of lines constituting multiple power lines (e.g., the number of lines constituting a multi-conductor power line and the number of lines bundled together by a span spacer). This is because one power line is composed of two straight lines (outer edges). Alternatively, when distinguishing between rising edge information and falling edge information, the expected number information may be information indicating the same number as the number of lines constituting multiple power lines (e.g., the number of lines constituting a multi-conductor power line). This is because one power line is composed of a straight line extracted by one rising edge and a straight line extracted by a falling edge. More specifically, for example, if the background of the power line is a background with many areas of pixel values darker than the power line (e.g., forest trees, soil, grass, etc.), when the above-mentioned edge detection method example is used, in each pixel row in a predetermined direction in the image, the boundary portion where the pixel value of the dark background changes to the pixel value of the power line is detected as rising edge information because the amount of change in the pixel value (brightness) is greater than the reference value, and the boundary portion where the pixel value of the power line returns to the pixel value of the dark background is detected as falling edge information because the amount of change in the pixel value (brightness) is greater than the reference value, so it can be said that one power line is composed of both edges. Also, if the background is a background with many areas of pixel values brighter than the power line (e.g., when snow is piled up), the relationship between the rising edge and the falling edge is reversed, but as a result, it can be said that one power line is similarly composed of both edges.

The line extraction condition information indicating the extraction conditions of the lines constituting the multiple power lines is information that can be set by the user via the terminal 1 or the like. For example, the first extraction condition is that there are at least the expected number of lines (twice the number of lines constituting the multiple power lines) parallel to the first line with the highest number of line votes, including the first line, and when the specific line extraction unit 122 determines that this extraction condition is satisfied, it extracts the corresponding line (i.e., the line constituting the multiple power lines). That is, in FIG. 9, for example, if any of the lines extending diagonally is selected as the first line based on the line vote number information, there are a number of parallel lines corresponding to the expected number of lines (8 lines), so the lines constituting the power lines are extracted as shown in FIG. 11. Also, for example, if any of the lines extending to the left and right are selected as the first line based on the line vote number information, there are not eight lines, which is the expected number of lines, so they are not extracted. In this case, the line vote count information for the lines extending to the left and right that are determined to be parallel to each other may select the first line with the most votes as a non-target line, or alternatively or in addition, the Hough transform may be run again. When combining the two, for example, if the number of lines indicated by the expected number information (e.g., twice the number of lines that make up the multiple power lines) is not extracted even after multiple reselections, the Hough transform may be run again. Note that "parallel" here includes lines that are approximately parallel and can be approximated as parallel, and the same applies below.

Furthermore, instead of the first extraction condition, the extracted straight lines may be paired with each other if they are located close to each other (within a predetermined reference distance range, in particular with a width that is approximately the same as the outer diameter of the power line (and furthermore with a width that is approximately the same as the outer diameter of the power line taking into account the front and back sides of the multiple power lines)), and the number of pairs may be the expected number of straight lines (in this case, the same number as the number of lines that make up the multiple power lines).

Furthermore, when distinguishing between rising edge information and falling edge information, for example, a second extraction condition is to pair a straight line parallel to the first straight line that has the highest number of straight line votes for either the rising edge or the falling edge with a straight line (particularly a straight line parallel to the first straight line) based on the other of either the rising edge or the falling edge that is in its vicinity (within a specified reference distance range, particularly a width that approximately matches the outer diameter of the power line (and furthermore a width that approximately matches the outer diameter of the power line taking into account the front and back sides of the multiple power lines)), and the number of such pairs, including the pair that includes the first straight line, is the expected number of straight lines (the same number as the number of lines that make up the multiple power lines), and when the specific straight line extraction unit 122 determines that this extraction condition is satisfied, the corresponding straight line (i.e., the straight lines that make up the multiple power lines) is extracted. That is, in FIG. 9, for example, when any of the straight lines extending diagonally (straight lines shown by solid lines in FIG. 12) is selected as the first straight line based on the straight line vote number information corresponding to the rising edge, straight lines parallel to the first straight line are determined, and pairs are formed with the straight lines corresponding to the falling edge (straight lines shown by dotted lines in FIG. 12) in the vicinity of each determined straight line, and since the number of pairs (four pairs) corresponding to the expected number of straight lines (four) exists, the straight lines constituting the power line are extracted as shown in FIG. 11. Also, for example, when any of the straight lines extending to the left and right are selected as the first straight line based on the straight line vote number information corresponding to the rising edge, the straight line is not extracted because the number of pairs formed (two pairs) does not exist, which is the expected number of straight lines. In this case, the first straight line with the largest number of straight line votes for the pair of straight lines extending to the left and right determined to be parallel to each other may be selected again as a non-target, or alternatively or in addition, the Hough transform may be executed again. When combining the two, for example, if the number of pairs indicated by the expected number information (e.g., the same number as the number of power lines) is not extracted even after multiple reselections, the Hough transform may be executed again.

Furthermore, a slope determination unit 123 may be provided that determines which of the slopes of the detected multiple straight lines is the first slope with the most votes based on the number of slope votes. The determination method based on the number of slope votes may be, for example, based on information about the angle (θ) included in the straight line vote number information obtained by the Hough transform, and the slope determination unit 123 may determine the first slope with the most votes based on the angle (slope vote number).

In the case where the inclination determination unit 123 is provided, for example, a third extraction condition is that there are at least the expected number of straight lines (twice the number of lines constituting the multiple power lines) whose inclination is within a predetermined range with respect to the first inclination determined by the inclination determination unit 123, and when the specific straight line extraction unit 122 determines that this extraction condition is satisfied, the specific straight line extraction unit 122 extracts the expected number of straight lines from among the straight lines whose inclination is within a predetermined range with respect to the first inclination in order of the number of straight line votes indicated by the straight line vote number information. That is, in FIG. 9, for example, when the inclination of any of the straight lines extending diagonally is determined to be the first inclination, there are straight lines (other straight lines extending diagonally) whose inclination is within a predetermined range with respect to the first inclination in a number corresponding to the expected number of straight lines (8 lines), so that the straight lines constituting the power lines are extracted as shown in FIG. 11. In this case, since the inclination of the straight lines extending to the left and right is less likely to be determined to be the first inclination, the possibility of erroneous detection may be lower.

Furthermore, in the case where the above-mentioned slope determination unit 123 is provided and the rising edge information and the falling edge information are to be distinguished from each other, for example, a fourth extraction condition is to determine a line corresponding to either a rising edge or a falling edge, the slope of which is within a predetermined range with respect to the first slope determined by the slope determination unit 123, and pair the determined line with a line based on either the rising edge or the falling edge (particularly a line parallel to the first slope) that is in the vicinity of the determined line (within a predetermined reference distance range, particularly a width that is approximately the same as the outer diameter of the power line (and furthermore a width that is approximately the same as the outer diameter of the power line taking into account the front and rear sides of the multiple power lines)), and the pairs include at least the expected number of straight lines (the same number as the number of lines that make up the multiple power lines). If the specific straight line extraction unit 122 determines that this extraction condition is satisfied, only straight lines that correspond to the pairs of the expected number of straight lines are extracted from the pairs in order of the number of straight line votes indicated by the straight line vote number information. That is, in FIG. 9, for example, if the slope of any of the diagonally extending straight lines (straight lines shown by solid lines in FIG. 12) corresponding to the rising edge is determined to be the first slope, straight lines (other diagonally extending straight lines) with a slope within a predetermined range with respect to the first slope are determined, and a pair is formed with a diagonally extending straight line (straight line shown by dotted lines in FIG. 12) corresponding to the falling edge in the vicinity of the determined straight line, and since there are a number of such pairs corresponding to the expected number of straight lines (4), the straight lines constituting the power line are extracted as shown in FIG. 11. Even in this case, the slope of the straight lines extending to the left and right is unlikely to be determined to be the first slope, so the possibility of erroneous detection can be reduced.

Furthermore, in the case where the above-mentioned slope determination unit 123 is provided and the rising edge information and the falling edge information are to be distinguished from each other, for example, a fifth extraction condition is to pair a straight line corresponding to either a rising edge or a falling edge with a straight line corresponding to the other of either a rising edge or a falling edge in its vicinity (within a specified reference distance range), and the slope determination unit 123 determines which of the pairs has the most common first slope based on the number of slope votes (for example, the slope of the pair is the average slope of the slopes of both straight lines constituting the pair), and there are at least the expected number of straight lines (the same number as the number of lines constituting the multiple power lines) of pairs with slopes within a specified range relative to the first slope, and when the specific straight line extraction unit 122 determines that this extraction condition is satisfied, only straight lines corresponding to the pairs of the expected number of straight lines are extracted from the pairs in order of the most number of straight line votes indicated by the straight line vote number information. That is, in FIG. 9, for example, a straight line corresponding to a rising edge (a straight line shown by a solid line in FIG. 13) is paired with a straight line corresponding to a nearby falling edge (a straight line shown by a dotted line in FIG. 13), and the slope determination unit 123 determines which slope is the most common first slope based on the number of slope votes (in FIG. 13, the slope of the line extending diagonally is determined to be the first slope), and pairs of slopes within a predetermined range relative to the first slope are extracted in descending order of the number of line votes, which is the expected number of straight lines, that is, four pairs. Even in this case, the slope of a line extending to the left or right is unlikely to be determined to be the first slope, so the possibility of erroneous detection can be reduced.

In this way, even when detecting the edges of non-target linear objects (especially elongated objects) in the background, it is possible to extract only the straight lines that make up multiple power lines of a multi-conductor system from within the image.

The tracking target position setting unit 124 sets a target position to be tracked by the camera (photographing unit) of the unmanned aerial vehicle 4 based on the straight lines extracted by the specific straight line extraction unit 122 (i.e., the straight lines constituting the multiple power lines). As a more specific example, the tracking target position setting unit 124 may calculate an average straight line at the average position of the multiple straight lines extracted by the specific straight line extraction unit 122 (visualize and add to the image as necessary), and set the average straight line as the target position (target straight line) to be tracked. FIG. 14 illustrates an example of an image in which the average straight line is visualized and added by a dotted line. The tracking target position setting unit 124 may also set one of the multiple straight lines or pairs extracted by the specific straight line extraction unit 122 as the target position (target straight line) to be tracked. Then, the tracking target position setting unit 124 generates instruction information for controlling at least one of the photographing unit or the unmanned aerial vehicle itself so that the set target position is included in the image photographed by the photographing unit, and transmits the instruction information to the unmanned aerial vehicle 4 via the communication unit 110. As a more specific example, first, it is determined whether a predetermined position (e.g., a central position or an approximate central position, etc.) of the target straight line is included in a predetermined range in the image (particularly, a range near the center of the image and further based on the central position or the approximate central position of the image). Then, if it is determined that the target straight line is not included in the predetermined range, difference information between the predetermined position of the target straight line and the predetermined range is generated, and instruction information for controlling at least one of the state of the shooting unit (e.g., shooting direction information, angle of view information, etc.) or the state of the unmanned aerial vehicle itself (e.g., current position information (latitude, longitude, altitude coordinates), attitude information, rotor motor output information, etc.) is generated based on the difference information. The difference information may be any information that allows the amount of deviation and the direction of deviation from the predetermined range to be grasped, and may be, for example, information on the direction and distance (number of pixels) on the image from the center of the image to the predetermined position of the target straight line. The instruction information may be generated based on, for example, the difference information, the correspondence information between the direction and distance on the image and the control direction and control distance in the shooting unit or the unmanned aerial vehicle shooting unit, and at least one of the current state information of the unmanned aerial vehicle, but is not limited thereto. If it is determined that the specified position of the target line is within a specified range in the image, the state of the imaging unit or unmanned aerial vehicle is maintained without generating new instruction information.

Next, the information/image storage unit 131 of the storage unit 130 at least temporarily stores, in addition to the images acquired by the image acquisition unit 115, edge images and edge information generated by the edge image generation unit 121, line vote number information generated by the specific line extraction unit 122 and information on the lines that make up multiple power lines, information on the first line, slope vote number information and information on the first slope generated by the slope determination unit 123, an image in which the target line is visualized by the tracking target position setting unit 124, expected number information set by the user, reference value information for rising edges or falling edges, reference range information indicating the vicinity, and other information and data generated in the processing by each of the functional units 121 to 126 of the processing unit 120.

<An example of a line extraction method>
Next, a straight line extraction method by the information processing system according to the present embodiment will be described with reference to Fig. 15 etc. Fig. 15 is a flowchart showing a process for implementing the straight line extraction method by the information processing system according to the present embodiment.

First, the image acquisition unit 115 of the terminal 1 acquires images such as images captured by a camera mounted on the unmanned aerial vehicle 4 (S101).

Next, the edge image generating unit 121 of the terminal 1 detects edge information in the image acquired by the image acquiring unit 115, and generates an edge image based on the edge information (including the x and y coordinates in the image where the detected edge exists) (S102).

Next, the specific line extraction unit 122 of the terminal 1 extracts lines from the detected lines according to the line extraction condition information indicating the extraction conditions for the lines constituting the power lines, which includes at least conditions related to the line vote number information obtained by the Hough transform of the edge information in the generated edge image and the expected number information indicating the expected number of lines related to the power lines expected in the edge image (S103).

In this way, with the terminal 1 of this embodiment, even when detecting the edges of non-target linear objects (especially elongated objects) in the background, it is possible to extract only the straight lines that make up multiple power lines of the multi-conductor system from within the image.

The above-described embodiment is merely an example to facilitate understanding of the present invention, and is not intended to limit the interpretation of the present invention. The present invention can be modified and improved without departing from the spirit of the invention, and it goes without saying that the present invention includes equivalents.

1 Terminal 2 Server 4 Unmanned aerial vehicle

Claims (9)

1. An information processing method for extracting straight lines corresponding to a plurality of power lines from an image captured by an unmanned aerial vehicle by a computer, comprising:
   generating an edge image from the captured image by an edge image generating unit;
   a step of extracting lines corresponding to the assumed number of lines that fall under extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines constituting the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by a Hough transform for edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image, from the multiple lines in the edge image by a specific line extraction unit;
   The information processing method is executed by the computer.
2. The number of assumed straight lines is twice the number of the plurality of power lines,
   The extraction condition is that the number of lines parallel to the first line having the largest number of line votes is the number of lines assumed to be the number of lines including the first line.
   2. The information processing method according to claim 1,
3. The method further includes a step of detecting, by the edge image generating unit, a portion in which an amount of change in pixel value increases and is greater than a reference value in each pixel row in a predetermined direction as rising edge information, and detecting a portion in which an amount of change in pixel value decreases and is greater than a reference value as falling edge information,
   The number of assumed straight lines is the same as the number of the plurality of power lines,
   The extraction condition is that a line parallel to a first line having the largest number of line votes in either one of the lines corresponding to the rising edge information or the lines corresponding to the falling edge information forms a plurality of pairs with the other of the lines corresponding to the rising edge information or the lines corresponding to the falling edge information located nearby the first line, and the number of pairs is the expected number of lines.
   2. The information processing method according to claim 1,
4. The method further includes a step of determining, by a gradient determination unit, which of the gradients of the plurality of straight lines detected by the specific straight line extraction unit is a first gradient that is most prevalent, by gradient voting;
   The number of assumed straight lines is twice the number of the plurality of power lines,
   the extraction condition is that at least the number of assumed straight lines is a straight line having a slope within a predetermined range with respect to the first slope determined by the slope determination unit;
   The method further includes a step of extracting, by the specific line extraction unit, lines having a slope within a predetermined range from the determined most common slope, the number of lines corresponding to the number of expected lines in descending order of the number of line votes.
   2. The information processing method according to claim 1,
5. detecting, by the edge image generation, a portion in each pixel row in a predetermined direction where a change amount in which a pixel value increases is greater than a reference value as rising edge information, and detecting, by the edge image generation, a portion in which a change amount in which a pixel value decreases is greater than a reference value as falling edge information;
   and determining, by a gradient determining unit, which of the gradients is a first gradient having the most prevalent gradient in a straight line corresponding to either the rising edge information or the falling edge information, by gradient voting;
   The assumed number of straight lines is the number of the plurality of power lines,
   the extraction condition is that a plurality of pairs are formed by a straight line corresponding to either the rising edge information or the falling edge information, the straight line being determined to have a gradient within a predetermined range with respect to the first gradient determined by the determining unit, and a straight line corresponding to the other of the rising edge information or the falling edge information in the vicinity of the straight line, and the number of pairs is at least the number of the assumed straight lines;
   extracting lines corresponding to the pairs of the expected number of lines from the pairs in descending order of the number of line votes by the specific line extraction unit,
   2. The information processing method according to claim 1,
6. a step of detecting, by the edge image generation, a portion in each pixel row in a predetermined direction where a change amount in pixel value increases is greater than a reference value as rising edge information, and detecting a portion in each pixel row in a predetermined direction where a change amount in pixel value decreases is greater than a reference value as falling edge information;
   and determining, by a gradient determining unit, which of a plurality of pairs of gradients formed by a straight line corresponding to either the rising edge information or the falling edge information and a straight line adjacent thereto corresponding to the other of the rising edge information or the falling edge information is a first gradient having the most number of gradients by gradient voting,
   The assumed number of straight lines is the number of the plurality of power lines,
   the extraction condition is that the number of pairs determined to have a gradient within a predetermined range with respect to the first gradient determined by the determining unit is at least the number of assumed straight lines;
   extracting lines corresponding to the pairs of the expected number of lines from the pairs in descending order of the number of line votes by the specific line extraction unit,
   2. The information processing method according to claim 1,
7. The method further includes a step of setting a target position to be tracked by the image capture unit of the unmanned aerial vehicle based on the straight line extracted by the specific straight line extraction unit, by a tracking target position setting unit.
   7. The information processing method according to claim 1,
8. An information processing system for extracting straight lines corresponding to multiple power lines of a multi-conductor type from an image captured by an unmanned aerial vehicle,
   an edge image generating unit that generates an edge image from the captured image;
   a specific line extraction unit that extracts lines that correspond to the assumed number of lines that fall under the extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines that constitute the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by performing a Hough transform on edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image; and
   An information processing system comprising:
9. A program for causing a computer to execute an information processing method for extracting straight lines corresponding to a plurality of multi-conductor power lines from an image captured by an unmanned aerial vehicle, comprising:
   The information processing method includes:
   generating an edge image from the captured image by an edge image generating unit;
   a step of extracting lines corresponding to the assumed number of lines that fall under extraction conditions from the lines in the edge image based on line extraction condition information indicating extraction conditions for lines constituting the multiple power lines, the line extraction condition information including at least conditions related to line vote number information obtained by a Hough transform for edges in the generated edge image and assumed number information indicating the assumed number of lines of the multiple power lines assumed in the edge image, from the multiple lines in the edge image by a specific line extraction unit;
   A program that executes the following.