JP6912281B2 - Aircraft, flight control systems, flight control methods, programs and recording media - Google Patents

Description

The present disclosure relates to a flying object capable of tracking a predetermined object, a flight control system, a flight control method for controlling the flying object, a program, and a recording medium.

There is known an air vehicle (for example, UAV: Unmanned Aerial Vehicle) which is equipped with an image pickup device and can take an aerial image of a subject while flying in the air. Some of the flying objects that can be used for this type of aerial photography have a function of automatically tracking a predetermined target such as a person or a vehicle, which is the main subject.

When the flying object is performing a tracking operation, the tracking target is automatically lost and lost, such as when the direction, distance, position, etc. of the subject to be tracked change significantly, or when the tracking target is hidden by another subject and blocked. Tracking may fail. When the tracking operation is interrupted in this way, the tracking is restarted by the user operating the terminal for controlling the flying object and resetting the tracking target.

As an example of an air vehicle having an automatic tracking function, for example, Patent Document 1 discloses an autonomous flying mobile body capable of tracking a target even in a situation where a bird's-eye view cannot be obtained. In this conventional autonomous flying vehicle, when the target cannot be recognized, the route near the unrecognizable position is searched based on the map information, and the target is tracked based on the searched route. Is.

Japanese Unexamined Patent Publication No. 2017-068298

In a conventional flying object having a tracking function, if the tracking target is lost during the tracking operation, it is necessary for the user to perform an operation such as resetting the tracking target using a terminal for controlling the flying object. When resetting the tracking target, the user cannot confirm the state of the aircraft of the flying object at the same time as operating the terminal, so that it is difficult to set the tracking target and operate the flying object at the same time. As described above, in the conventional air vehicle, when the automatic tracking of the tracking target fails, there is a problem in the operability for continuing the tracking.

In one aspect, it is an air vehicle having an imaging device that images a predetermined tracking target, a sound collecting device that collects ambient sounds including the tracking target, and a control unit that executes processing related to tracking the tracking target. The control unit tracks the tracking target using the image captured by the imaging device, estimates the sound source direction corresponding to the tracking target from the sound information acquired by the sound collecting device, and based on the estimation result of the sound source direction. It is a flying object that adjusts the shooting direction by the image pickup device.

In tracking the tracking target, the control unit may calculate the feature value of the image related to the tracking target in the captured image, compare the feature values of the images, and search for the tracking target.

When the tracking of the tracking target fails, the control unit may acquire the sound information emitted by the tracking target, estimate the sound source direction, and adjust the shooting direction according to the sound source direction.

The control unit may calculate the characteristic value of the sound related to the tracking target in the sound information, estimate the sound source direction of the tracking target from the characteristic value of the sound, and track the tracking target according to the sound source direction.

In tracking the tracking target, the control unit may limit the area for searching the tracking target in the captured image according to the estimated sound source direction.

In estimating the sound source direction, the control unit may extract the sound component to be tracked from the sound information acquired by the sound collecting device and perform the sound source direction estimation process for the sound component to be tracked.

The control unit may control the movement of at least one of the rotor blade mechanism of the flying object and the support mechanism that supports the image pickup device in adjusting the photographing direction.

In one aspect, a flight control system comprising an air vehicle and a terminal that communicates with the air vehicle, wherein the air vehicle collects an imaging device that captures a predetermined tracking target and ambient sound including the tracking target. A sound collecting device that makes a sound, a communication unit that transmits candidate information of a tracking target to a terminal, and a process related to tracking the tracking target are executed, and the tracking target is tracked using an image captured by the imaging device. The terminal has a control unit that estimates the sound source direction corresponding to the tracking target from the sound information acquired by the sound collecting device and adjusts the shooting direction by the imaging device based on the estimation result of the sound source direction. It is a flight control system having a communication unit for receiving target candidate information, a display unit for displaying tracking target candidates, and an operation unit for inputting information regarding tracking target settings.

The display unit of the terminal may display the tracking failure when the tracking of the tracking target fails.

In one aspect, it is a flight control method for controlling an air vehicle that tracks a predetermined tracking target, in which a step of imaging the tracking target by an imaging device and a sound collecting device for surrounding sounds including the tracking target are picked up. A step of tracking the tracking target using the captured image, a step of estimating the sound source direction corresponding to the tracking target based on the collected sound information, and a shooting by the imaging device based on the estimation result of the sound source direction. It is a flight control method having a step of adjusting a direction.

The step of tracking the tracking target may include a step of calculating the feature value of the image related to the tracking target in the captured image, comparing the feature values of the images, and searching for the tracking target.

The step of estimating the sound source direction includes a step of acquiring the sound information emitted by the tracking target and estimating the sound source direction when the tracking of the tracking target fails, and the step of adjusting the shooting direction is in the estimated sound source direction. It may include a step of adjusting the shooting direction accordingly.

The step of calculating the characteristic value of the sound regarding the tracking target in the sound information, and the step of tracking the tracking target estimates the sound source direction of the tracking target from the sound characteristic value and tracks the tracking target according to the sound source direction. May include steps.

The step of tracking the tracking target may include a step of limiting the region for searching the tracking target in the captured image according to the estimated sound source direction.

The step of tracking the tracking target may include a step of extracting the sound component of the tracking target from the sound information acquired by the sound collecting device and performing an estimation process of the sound source direction for the sound component of the tracking target.

The step of adjusting the imaging direction may include a step of controlling the movement of at least one of the rotor blade mechanism of the flying object and the support mechanism supporting the image pickup device.

In the flight control method, a step of transmitting candidate information of a tracking target to a terminal communicating with an air vehicle, a step of receiving candidate information of a tracking target by the terminal, and a step of displaying the candidate of the tracking target by a display unit. It may have a step of making input regarding the setting of the tracking target.

In the flight control method, there may be a step of displaying the tracking failure by the display unit when the tracking of the tracking target fails.

In one aspect, the program includes a step of capturing the tracked object with an image pickup device and a step of picking up ambient sound including the tracked object by a sound picking device on a computer controlling an air vehicle that tracks a predetermined tracking target. , The step of tracking the tracking target using the captured image, the step of estimating the sound source direction corresponding to the tracking target from the collected sound information, and adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction. It is a program to execute the steps to be performed.

In one aspect, the recording medium is a step of imaging the tracked object by the imaging device on a computer controlling the flying object that tracks the predetermined tracked object, and a step of collecting the ambient sound including the tracked object by the sound collecting device. The step of tracking the tracking target using the captured image, the step of estimating the sound source direction corresponding to the tracking target from the collected sound information, and the shooting direction by the imaging device based on the estimation result of the sound source direction. A computer-readable, recording medium that records the steps to adjust and the program to run it.

The outline of the above invention does not list all the features of the present disclosure. Sub-combinations of these feature groups can also be inventions.

It is a schematic diagram which shows the structural example of the flight control system in embodiment. It is a figure which shows an example of the appearance of an unmanned flying object. It is a block diagram which shows an example of the hardware composition of an unmanned air vehicle. It is a block diagram which shows an example of the hardware configuration of a terminal. It is a figure which shows typically the state of the 1st Embodiment which tracks a tracking object by an unmanned flying object. It is a flowchart which shows an example of the processing procedure about the initial setting of the tracking operation in an embodiment. It is a figure which shows an example of the initial setting screen for performing the initial setting of the tracking operation in an embodiment. It is a flowchart which shows an example of the processing procedure of the automatic tracking operation in 1st Embodiment. It is a sequence diagram which shows an example of the processing procedure at the time of tracking failure in an embodiment. It is a figure which shows an example of the notification screen which notifies the tracking failure in an embodiment. It is a figure which shows an example of the confirmation screen at the time of resetting the tracking target in an embodiment. It is a figure which shows typically the state of the 2nd Embodiment which tracks a tracking object by an unmanned flying object. It is a flowchart which shows an example of the processing procedure of the automatic tracking operation in 2nd Embodiment.

Hereinafter, the present disclosure will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of features described in the embodiments are essential to the solution of the invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

The flight control method according to the present disclosure defines various processes (steps) in the information processing device for controlling the flight of the flying object. Aircraft include aircraft moving in the air (eg, drones, helicopters). The air vehicle may be an unmanned aerial vehicle (UAV). The flying object flies along a preset flight path for performing various operations involving flight such as aerial photography, transportation of objects, or spraying of pesticides, fertilizers, water, and the like. When performing the automatic tracking operation, the flying object flies while tracking the set tracking target.

The information processing device according to the present disclosure is a computer, and the information processing device may be included in the flying object itself, for example. The information processing device according to the present disclosure is a transmitter for instructing remote control of various processes including movement of an air vehicle, or a terminal device connected to the transmitter so that information and data can be input and output. Alternatively, it includes various terminals (platforms) such as terminal devices connected to the aircraft so that information and data can be input and output. The terminal may be, for example, a PC, a tablet terminal, a mobile terminal, or the like.

The program according to the present disclosure is a program for causing an information processing apparatus to execute various processes (steps).

The recording medium according to the present disclosure is one in which a program (that is, a program for causing an information processing apparatus to execute various processes (steps)) is recorded.

The flight control system according to the present disclosure includes a flying object as an example of a moving object and a terminal (platform) for remotely controlling the operation or processing of the flying object. The flight control system according to the present disclosure may be configured to track and fly a tracking target by itself.

In each embodiment according to the present disclosure shown below, an unmanned aerial vehicle (UAV) is exemplified as the flying object. In the drawings attached herein, the unmanned aerial vehicle is referred to as "UAV". In each of the following embodiments, the flying object sets a tracking target and flies while moving the position following the tracking target.

In each of the following embodiments, the information processing apparatus can generate, acquire, and change feature value information including image information, sound information, and the like as feature values to be tracked in the flying object. The information processing device can generate, acquire, and change flight position information including at least one of a flight start position, a flight end position, a predetermined position on a flight path, a flight altitude, and a flight speed in an air vehicle. Is. When the terminal includes an information processing device, the information processing device can communicate with the flying object, and can transmit the feature value information of the tracking target and the flight position information of the flying object to the flying object. The term "communication" as used herein is a broad concept including data communication in general, and includes not only the case of wired connection by a cable or the like but also the case of wireless communication. Further, it includes not only the case where the information processing device directly communicates with the flying object but also the case where the information processing device indirectly communicates via the transmitter or the storage medium.

FIG. 1 is a schematic diagram showing a configuration example of the flight control system 10 according to the embodiment. The flight control system 10 includes an unmanned aircraft 100 and a terminal 50. The unmanned aircraft 100 and the terminal 50 can communicate with each other using wired communication or wireless communication (for example, wireless LAN (Local Area Network) or Bluetooth (registered trademark)). The terminal 50 is used, for example, in a state of being held by both hands of a person (hereinafter, referred to as “user”) who uses the terminal 50. The terminal 50 may be, for example, a transmitter, a tablet terminal, a mobile terminal, a PC, or the like. The terminal 50 may have a configuration in which a tablet terminal or a mobile terminal is attached to the transmitter and is provided so as to be able to communicate with each other.

FIG. 2 is a diagram showing an example of the appearance of the unmanned aircraft 100. The unmanned flying object 100 is an example of a moving object provided with an imaging device 220 as an example of an imaging unit. The moving body is a concept including an unmanned flying object 100, other aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like. Here, as shown in FIG. 2, it is assumed that the roll axis (see the x-axis in FIG. 2) is defined in the direction parallel to the ground and along the moving direction STV0. In this case, the pitch axis (see the y-axis in FIG. 2) is defined in the direction parallel to the ground and perpendicular to the roll axis, and further, the yaw axis is perpendicular to the ground and perpendicular to the roll axis and the pitch axis. (Z-axis in FIG. 2) is defined.

The unmanned aerial vehicle 100 includes a UAV main body 102, a rotary wing mechanism 210, an imaging device 220, a gimbal 230, and a sound collecting device 180. The unmanned aircraft 100 can move based on, for example, a remote control instruction transmitted from the terminal 50. The movement of the unmanned vehicle 100 means flight, and includes at least ascending, descending, left-turning, right-turning, left-horizontal movement, and right-horizontal movement.

The unmanned aircraft 100 includes a plurality of rotary wing mechanisms (propellers) 210. The unmanned aircraft 100 includes, for example, four rotor mechanisms 210. The unmanned aviation body 100 moves the unmanned aviation body 100 by controlling the rotation of these rotor blade mechanisms 210. However, the number of rotor blades is not limited to four. Further, the unmanned aircraft 100 may be a fixed-wing aircraft having no rotary wings.

The image pickup apparatus 220 is a camera that captures a subject (for example, a person, a vehicle, etc.) included in a desired imaging range. The imaging device 220 is attached to a gimbal 230, which is an example of a support mechanism, and the imaging range is adjusted by the movement of the gimbal 230.

The sound collecting device 180 is a microphone that collects sound around the imaging range including the subject. The sound collecting device 180 may be a microphone array having a plurality of microphones.

Next, a configuration example of the unmanned aircraft 100 will be described.

FIG. 3 is a block diagram showing an example of the hardware configuration of the unmanned aircraft 100. The unmanned aircraft 100 includes a control unit 110, a communication interface 150, a memory 160, a storage 170, a sound collecting device 180, a battery 190, a rotary wing mechanism 210, an imaging device 220, a gimbal 230, and a GPS. The configuration includes a receiver 240, an inertial measurement unit 250, a magnetic compass 260, a barometric altimeter 270, and an ultrasonic sensor 280.

The control unit 110 is configured by using a processor, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The control unit 110 performs signal processing for controlling the operation of each part of the unmanned aircraft 100, data input / output processing with and from other parts, data calculation processing, and data storage processing. The control unit 110 has a function of executing a process related to flight control in the unmanned aircraft 100.

The control unit 110 controls the flight of the unmanned vehicle 100 according to a program stored in the memory 160 or the storage 170. Further, the control unit 110 controls the movement (that is, flight) of the unmanned vehicle 100 according to a command received from the remote terminal 50 via the communication interface 150.

The control unit 110 controls the flight of the unmanned vehicle 100 by controlling the rotary wing mechanism 210. That is, the control unit 110 controls the position and orientation of the unmanned aircraft 100 including the latitude, longitude, and altitude by controlling the rotary wing mechanism 210. The control unit 110 controls the rotor mechanism 210 based on the position information acquired by at least one of the GPS receiver 240, the inertial measurement unit 250, the magnetic compass 260, the barometric altimeter 270, and the ultrasonic sensor 280.

For example, the control unit 110 acquires posture information indicating the posture state of the image pickup device 220 from the gimbal 230 as information indicating the image pickup direction of the image pickup device 220. The information indicating the posture state of the image pickup apparatus 220 indicates the rotation angle of the gimbal 230 from the reference rotation angle of the pitch axis and the yaw axis. The control unit 110 acquires information indicating the direction of the unmanned aircraft 100.

The control unit 110 controls the image pickup device 220 and the gimbal 230. The control unit 110 controls the imaging range of the imaging device 220 by changing the imaging direction or the angle of view of the imaging device 220. The control unit 110 controls the imaging range of the imaging device 220 supported by the gimbal 230 by controlling the rotation mechanism of the gimbal 230.

The control unit 110 acquires image data (hereinafter, may be referred to as “captured image”) of the subject captured by the image pickup device 220.

The control unit 110 acquires sound information of the subject and its surroundings collected by the sound collecting device 180.

The communication interface 150 is an example of a communication unit and communicates with the terminal 50. The communication interface 150 receives various information about the tracking target, the flight path, and the like from the terminal 50. The communication interface 150 receives various commands from the terminal 50 to the control unit 110.

The memory 160 is an example of a storage unit. The memory 160 has a rotary wing mechanism 210, an image pickup device 220, a gimbal 230, a GPS receiver 240, an inertial measurement unit 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, a communication interface 150, and a storage 170. The program and the like necessary for controlling the sound collecting device 180 are stored. The memory 160 stores various information and data used at the time of processing by the control unit 110. The memory 160 may be a computer-readable recording medium, and may be a SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and an EEPROM. It may include at least one flash memory such as a USB memory. The memory 160 may be provided inside the unmanned vehicle 100 or may be detachably provided from the unmanned vehicle 100. Tracking information including feature value information to be tracked may be stored in the memory 160.

The storage 170 is an example of a storage unit. The storage 170 stores and holds various data and information. The storage 170 may be an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a USB memory, or the like. The storage 170 may be provided inside the unmanned vehicle 100 or may be detachably provided from the unmanned vehicle 100.

The sound collecting device 180 includes a microphone array in which a plurality of microphones are arranged in a row. The sound collecting device 180 may include, for example, two or more microphones and a signal processing circuit for processing the sound signal picked up by each microphone. The sound collecting device 180 collects the surrounding sound including the tracking target in order to estimate the sound source direction of the tracking target, and acquires the information of the sound signal of the subject including the tracking target collected by each of a plurality of microphones. do.

The battery 190 has a function as a drive source for each part of the unmanned vehicle 100, and supplies necessary power to each part of the unmanned vehicle 100.

The rotary blade mechanism 210 has a plurality of rotary blades and a plurality of drive motors for rotating the plurality of rotary blades. The rotary wing mechanism 210 generates an air flow in a specific direction by rotating the rotary wing, and controls the flight (up, down, horizontal movement, turning, inclination, etc.) of the unmanned vehicle 100.

The image pickup apparatus 220 captures a subject in a desired imaging range and generates data of the captured image. The captured image may be a moving image or a still image. The image data obtained by the imaging of the imaging device 220 is stored in a memory or memory 160 of the imaging device 220, a storage 170, or the like.

The gimbal 230 is an example of a support mechanism that supports the image pickup device 220, and supports the image pickup device 220 rotatably around at least one axis. The gimbal 230 may rotatably support the image pickup device 220 about the yaw axis, the pitch axis, and the roll axis. The gimbal 230 may change the imaging direction of the imaging device 220 by rotating the imaging device 220 around at least one of the yaw axis, the pitch axis, and the roll axis.

The GPS receiver 240 receives a plurality of signals indicating the time transmitted from the plurality of navigation satellites (that is, GPS satellites) and the position (coordinates) of each GPS satellite. The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aircraft 100) based on the plurality of received signals. The GPS receiver 240 outputs the position information of the unmanned aircraft 100 to the control unit 110. The position information of the GPS receiver 240 may be calculated by the control unit 110 instead of the GPS receiver 240. In this case, information indicating the time included in the plurality of signals received by the GPS receiver 240 and the position of each GPS satellite is input to the control unit 110.

The inertial measurement unit (IMU) 250 detects the attitude of the unmanned aircraft 100 and outputs the detection result to the control unit 110. The inertial measurement unit 250 determines the posture of the unmanned vehicle 100 as the acceleration in the front-back, left-right, and up-down directions of the unmanned vehicle 100 and the angular velocities in the three axes of the pitch axis, the roll axis, and the yaw axis. To detect.

The magnetic compass 260 detects the direction of the nose of the unmanned aircraft 100 and outputs the detection result to the control unit 110. The barometric altimeter 270 detects the altitude at which the unmanned vehicle 100 flies and outputs the detection result to the control unit 110. The ultrasonic sensor 280 irradiates ultrasonic waves, detects ultrasonic waves reflected by the ground or an object, and outputs the detection result to the control unit 110. The detection result indicates, for example, the distance (that is, altitude) from the unmanned aircraft 100 to the ground. The detection result may indicate, for example, the distance from the unmanned flying object 100 to the object.

Next, an example of the function of the control unit 110 of the unmanned aircraft 100 will be described.
The control unit 110 has a tracking unit 111 that performs processing related to tracking a predetermined tracking target in the image data of the captured image captured by the imaging device 220, and sound information data (sound information data acquired by the sound collecting device 180). It includes a sound source direction estimation unit 112 that performs processing related to estimation of a predetermined sound source direction based on (sound data).

The tracking unit 111 moves the unmanned aircraft 100 to track the set tracking target, and controls the rotary wing mechanism 210, the imaging device 220, and the gimbal 230 so that the tracking target can be imaged by the imaging device 220. .. The tracking unit 111 tracks the tracking target using the captured image acquired by the imaging device 220. The tracking unit 111 calculates, compares, and updates the feature values of the tracking target in the captured image captured by the imaging device 220, and searches for the tracking target. The tracking unit 111 compares the feature value of the tracking target of the previous frame with the feature value of the current frame in the captured image of the moving image, and searches for the tracking target having the same feature value.

The sound source direction estimation unit 112 separates the sound emitted by the tracking target and the environmental sound from the sound data acquired by the sound collecting device 180, acquires the phase information of the sound of the tracking target acquired by each microphone, and tracks the sound. Estimate the direction of the sound source corresponding to the target.

The tracking unit 111 controls the image pickup device 220 and the gimbal 230 based on the estimated sound source direction, and adjusts the imaging direction of the image pickup device 220 so as to move toward the tracking target according to, for example, the sound source direction.

Next, a configuration example of the terminal 50 will be described.

FIG. 4 is a block diagram showing an example of the hardware configuration of the terminal 50. The terminal 50 includes a processing unit 51, a memory 52, a wireless communication unit 53, a display unit 54, an operation unit 55, an interface unit 56, a storage 57, and a battery 58. The terminal 50 is an example of an information processing device having a function of an operation terminal for transmitting an instruction for remotely controlling the unmanned aircraft 100. The terminal 50 is an example of an information processing device having a function of a setting terminal that inputs / outputs various information and data related to the flight of the unmanned vehicle 100. The terminal 50 may have a separate configuration in which the transmitter and the tablet terminal or mobile terminal are connected to each other.

The processing unit 51 is configured by using a processor (for example, CPU, MPU or DSP). The processing unit 51 performs signal processing for controlling the operation of each unit of the terminal 50 in a centralized manner, data input / output processing with each other unit, data calculation processing, and data storage processing.

The processing unit 51 may acquire data and information from the unmanned aircraft 100 via the wireless communication unit 53. The processing unit 51 may acquire data or information from another device via the interface unit 56. The processing unit 51 may acquire data or information input via the operation unit 55. The processing unit 51 may acquire data and information held in the memory 52. The processing unit 51 may send data or information to the display unit 54 and display the display information based on the data or information on the display unit 54. The processing unit 51 may send data or information to the storage 57 and store the data or information. The processing unit 51 may acquire data and information stored in the storage 57.

Based on the operation input of the operation unit 55, the processing unit 51 may input settings related to the setting of the tracking target, such as the setting of the image area including the tracking target.

The processing unit 51 may generate an operation signal for remotely controlling the movement of the unmanned aircraft 100 based on the operation input of the operation unit 55. The processing unit 51 may transmit the generated operation signal as a movement control command to the unmanned vehicle 100 via the wireless communication unit 53 to remotely control the unmanned vehicle 100.

The processing unit 51 may generate at least one display screen of a setting screen, an operation screen, a notification screen, and a confirmation screen to be displayed on the display unit 54.

The memory 52 is an example of a storage unit. The memory 52 temporarily stores, for example, a ROM (Read Only Memory) in which data of a program or set value that defines the operation of the processing unit 51 is stored, and various information and data used during processing of the processing unit 51. It has a RAM (Random Access Memory) for storing. The program and set value data stored in the ROM of the memory 52 may be copied to a predetermined recording medium (for example, CD-ROM, DVD-ROM).

The wireless communication unit 53 is an example of a communication unit, and communicates with the unmanned aircraft 100 by various wireless communication methods via an antenna to transmit / receive information and data. The wireless communication method may include, for example, wireless LAN, Bluetooth®, short-range wireless communication, or communication via a public wireless line. The wireless communication unit 53 may communicate with other devices to send and receive information and data.

The display unit 54 is configured by using, for example, an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence) display, and displays various information and data output from the processing unit 51 on the display screen. The display unit 54 may display at least one display screen of the setting screen, the operation screen, the notification screen, and the confirmation screen. The display unit 54 may have, for example, an indicator lamp using an LED (Light Emission Diode). The indicator lamp displays, for example, at least one of the wireless connection state between the unmanned vehicle 100 and the terminal 50, the activation state of the unmanned vehicle 100, and the remaining capacity of the battery of the unmanned vehicle 100 or the terminal 50. You can do it.

The operation unit 55 receives operation instructions, data, or information input by the user holding the terminal 50. The operation unit 55 may include a joystick, buttons, keys, a touch panel, a microphone, and the like. The operation unit 55 is used, for example, in an operation for remotely controlling the movement of the unmanned flying object 100 by the user (for example, moving the unmanned flying object 100 back and forth, moving left and right, moving up and down, and changing the direction). The operation unit 55 is used, for example, in an operation of inputting a setting related to a tracking target. The operation unit 55 is used, for example, in operations such as confirmation of a tracking target and an instruction to start tracking.

The interface unit 56 inputs and outputs information and data between the terminal 50 and other devices. The interface unit 56 may be, for example, a USB port (not shown) provided in the terminal 50. The interface unit 56 may be an interface other than the USB port.

The storage 57 is an example of a storage unit. The storage 57 stores and holds various data and information. The storage 57 may be a flash memory, an SSD (Solid State Drive), a memory card, a USB memory, or the like. The storage 57 may be provided so as to be removable from the main body of the terminal 50.

The battery 58 has a function as a drive source for each part of the terminal 50, and supplies necessary power to each part of the terminal 50.

Next, the tracking operation by the unmanned aircraft 100 will be described.

(First Embodiment)
FIG. 5 is a diagram schematically showing a state of the first embodiment in which a tracking target is tracked by an unmanned flying object. The first embodiment shows an example in which a person is tracked and imaged as a tracking target. The unmanned flying object 100 images a person 410 as a subject by the image pickup device 220, and sets the person in the captured image as a tracking target. The unmanned aircraft 100 adjusts the imaging direction by rotating the imaging device 220 by the gimbal 230 so that the set tracking target is continuously imaged. At this time, the gimbal 230 rotates the image pickup device 220 in the horizontal direction Dh centered on the yaw axis and in the vertical direction Dv centered on the pitch axis, respectively. The unmanned vehicle 100 controls the rotary wing mechanism 210 according to the movement of the person 410 to be tracked, and makes the position of the unmanned vehicle 100 follow.

For example, if the tracking target cannot be searched for in the captured image data and the tracking fails, for example, the tracking target moves significantly and deviates from the angle of view of the image pickup device 220, the unmanned vehicle 100 is subjected to the sound collection device 180 to capture the image area. The surrounding sound is picked up, and the sound source direction SD corresponding to the tracking target is estimated. At this time, the person 410 to be tracked emits a voice toward the unmanned flying object 100, such as calling to the unmanned flying object 100 when the tracking failure is recognized. The unmanned aircraft 100 detects the direction of the tracking target by acquiring the sound emitted by the tracking target and estimating the sound source direction SD. Based on the estimated sound source direction SD, the unmanned aircraft 100 controls the gimbal 230 to adjust the shooting direction of the image pickup apparatus 220 toward the sound source direction SD, that is, toward the person 410 to be tracked.

FIG. 6 is a flowchart showing an example of a processing procedure related to the initial setting of the tracking operation in the embodiment. FIG. 7 is a diagram showing an example of an initial setting screen for initializing the tracking operation in the embodiment.

The unmanned aircraft 100 sets a tracking target in the initial setting of the tracking operation. In this example, the procedure for setting the tracking target using the terminal 50 is shown. The terminal 50 inputs an operation instruction of the tracking target range in the captured image by the user, and sets the tracking target range (S11). At this time, the terminal 50 displays the setting screen 300 having the captured image 310 divided into a plurality of regions on the display unit 54 as in the example shown in FIG. When the user specifies the area including the tracking target TG1 in the captured image 310 by a touch operation or the like, an image area frame indicating the tracking target range 321 is displayed, and when the user touches the setting button (SET) 322 after confirmation, the terminal 50 Sets the corresponding image area as the tracking target range. The terminal 50 transmits the tracking target range information to the unmanned aviation body 100, and the unmanned aviation body 100 receives and inputs the tracking target range information to set the tracking target range in the captured image.

The unmanned vehicle 100 calculates the feature value of the image to be tracked in the image data of the range to be tracked in the captured image (S12). As the feature value calculated from the image data to be tracked, for example, SIFT (Scale-Invariant Feature Transform), which is an example of the image feature amount, may be used. In addition to SIFT, SURF (Speeded Up Robust Features), GLOH (Gradient Location and Orientation Histogram), FAST (Features from Accelerated Segment Test), or the like may be used as the image feature amount to be tracked. As the feature value of the tracking target, the feature value of the image of only the tracking target may be calculated, or the feature value of the image of a predetermined region including the tracking target may be calculated. The feature value of the tracking target may be calculated by the terminal 50 and transmitted to the unmanned vehicle 100, and the feature value may be acquired by the unmanned vehicle 100.

FIG. 8 is a flowchart showing an example of the processing procedure of the automatic tracking operation in the first embodiment. The processing of the automatic tracking operation may be mainly executed by the tracking unit 111 and the sound source direction estimation unit 112 of the control unit 110 in the unmanned aircraft 100. The control unit 110 of the unmanned aircraft 100 captures a predetermined imaging region by the imaging device 220 and acquires an captured image (S21). The control unit 110 extracts an image corresponding to the tracking target from the captured image (S22). At this time, the control unit 110 extracts an image of the region including the tracking target based on the set tracking target range. Then, the control unit 110 searches for a tracking target in the extracted image of the predetermined region (S23). In the tracking target search process, the control unit 110 compares the feature values of the images, compares the feature values of the extracted image of the predetermined region with the previously calculated feature values of the tracking target, and determines whether or not they match. do. In the tracking target search process, the control unit 110 may update the feature value of the tracking target by using the image area of the tracking target in the captured image currently captured. The control unit 110 may extract and search for a region that matches the feature value of the tracking target from the entire region of the captured image.

The control unit 110 determines whether or not the tracking has failed depending on whether or not the feature values of the tracking targets match (S24). When the tracking is successful (S24: No), the control unit 110 repeatedly executes the search process of the tracking target in the captured image (S21 to S24).

When the tracking fails (S24: Yes), the control unit 110 receives the sound signal picked up by the sound collecting device 180 and acquires the sound data around the imaging region (S25). The control unit 110 separates the sound data of a plurality of sound sources from the acquired sound data (S26). In the sound source separation process, the control unit 110 may separate sound data from a plurality of sound sources by using, for example, BSS (Blind Source Separation). The control unit 110 separates environmental sounds such as noise from sounds emitted by a subject corresponding to the tracking target by sound source separation processing. In this example, the voice data emitted by the person 410 to be tracked is separated from the environmental sound. The control unit 110 extracts the separated sound data with the sound component to be tracked as the main component (S27). The control unit 110 calculates the sound feature value as the feature value to be tracked with respect to the sound data of the principal component. For the feature value calculated from the sound data to be tracked, for example, frequency characteristics, presence / absence of directivity, envelope, and the like may be used.

The control unit 110 estimates the sound source direction of the sound data of the principal component corresponding to the tracking target, for example, from the characteristic value of the sound (S28). In the sound source direction estimation process, the control unit 110 estimates the arrival direction of the sound from the sound source by using, for example, the phase information of the sound data acquired by each microphone by the microphone array of the sound collecting device 180. The sound source direction estimation process includes, for example, a method of estimating the sound source position by forming an acoustic beam by a delay sum array such as a beamforming method and spatially scanning the sound beam, a CSP (Cross-power Spectrum Phase analysis) method, and the like. A method of estimating sound source localization by calculating the delay time by obtaining the arrival time difference between a plurality of microphones may be used.

The control unit 110 detects the direction of the tracking target based on the estimation result of the sound source direction, and adjusts the shooting direction by the imaging device 220 (S29). At this time, the control unit 110 controls the movement of at least one of the image pickup device 220 and the gimbal 230, and adjusts the imaging direction of the image pickup device 220 so as to move toward the tracking target. The control unit 110 may control the rotary wing mechanism 210 to move and rotate the unmanned vehicle 100 to adjust the photographing direction. Then, the control unit 110 repeatedly executes the search process for the tracking target in the captured image (S21 to S24).

FIG. 9 is a sequence diagram showing an example of a processing procedure at the time of tracking failure in the embodiment. FIG. 10 is a diagram showing an example of a notification screen for notifying a tracking failure in the embodiment. FIG. 11 is a diagram showing an example of a confirmation screen at the time of resetting the tracking target in the embodiment.

When the tracking of the tracking target fails, the unmanned aircraft 100 and the terminal 50 may perform processing such as notifying the user and confirming the reset tracking target. When the unmanned aircraft 100 determines that the search for the tracking target has failed during the automatic tracking operation (S31), the unmanned aircraft 100 transmits information indicating the tracking failure, which is an example of the tracking status information, to the terminal 50 (S32). ). When the terminal 50 receives the tracking status information indicating the tracking failure, the terminal 50 performs the tracking failure display for notifying the user of the tracking failure status as an example of the tracking status display (S33). At this time, the terminal 50 displays the notification screen 330 for notifying the tracking failure on the display unit 54, as in the example shown in FIG. On the notification screen 330, the tracking failure display (MISSED) 323 is displayed on the captured image 310 and can be visually recognized by the user. In addition to the display of the notification screen 330, the terminal 50 may perform notification by sound such as an alarm sound, notification by light such as LED light emission, and the like.

The unmanned aircraft 100 executes the processing of estimating the sound source direction and adjusting the shooting direction as in the procedure (S25 to S29) shown in FIG. 8, and re-searches the tracking target (S34). In the search for the tracking target, the unmanned vehicle 100 may limit the area for searching the tracking target in the captured image according to the estimated sound source direction, and extract candidates for the tracking target. The unmanned aircraft 100 transmits the information indicating the tracking target candidate extracted by the re-search to the terminal 50 as the tracking target candidate information (S35). When the terminal 50 receives the tracking target candidate information, the terminal 50 displays the tracking target candidate indicating the tracking target candidate to the user in order to confirm the tracking target (S36). At this time, the terminal 50 displays a confirmation screen 350 on the display unit 54 for confirming the tracking target candidate as in the example shown in FIG. On the confirmation screen 350, the tracking target frame 324 surrounding the tracking target TG1 in the captured image 310 is displayed, and the tracking target confirmation display (TARGET?) 325 is displayed. When the user touches the confirmation button (OK) 326 after confirming the tracking target, the terminal 50 inputs the confirmation instruction by the user (S37) and transmits the confirmation response to the unmanned aircraft 100 (S38).

Upon receiving the confirmation response from the terminal 50, the unmanned aircraft 100 sets an area including the corresponding tracking target as a new tracking target range, calculates and updates the feature value of the tracking target (S39). The unmanned aircraft 100 then continues the automatic tracking operation (S40).

In the first embodiment described above, the unmanned aircraft estimates the sound source direction corresponding to the tracking target, and corrects the tracking operation by using the sound source direction estimation result. In the embodiment, when the tracking target fails to be tracked, the unmanned aircraft acquires the sound information emitted by the tracking target, estimates the sound source direction, and adjusts the shooting direction according to the sound source direction. As a result, the imaging device can be directed toward the sound source corresponding to the tracking target, and the tracking target can be continuously imaged so that the tracking target can be tracked again, and the tracking result can be corrected. In addition, robustness during tracking operation can be improved. In this case, the operation of resetting the tracking target becomes unnecessary, and the operability can be improved.

When the tracking target is set in the initial setting, or when the tracking target is set again after the tracking failure, the unmanned aircraft displays the tracking target candidate on the terminal and inputs the instruction operation after confirmation by the user. As a result, the user can accurately and easily confirm and set the tracking target, and the operability can be improved.

(Second Embodiment)
FIG. 12 is a diagram schematically showing a state of a second embodiment in which a tracking target is tracked by an unmanned flying object. The second embodiment shows an example in which a moving object such as a vehicle is tracked and imaged as a tracking target. The unmanned vehicle 100 takes an image of the vehicle 420 as a subject by the image pickup device 220, and sets the vehicle in the captured image as a tracking target. The unmanned aircraft 100 adjusts the imaging direction by rotating the imaging device 220 by the gimbal 230 so that the set tracking target is continuously imaged. At this time, the gimbal 230 rotates the image pickup device 220 in the horizontal direction Dh centered on the yaw axis and in the vertical direction Dv centered on the pitch axis, respectively. The unmanned vehicle 100 controls the rotary wing mechanism 210 according to the movement of the vehicle 420 to be tracked, and makes the position of the unmanned vehicle 100 follow.

When the unmanned aircraft 100 tracks the tracking target, the sound collecting device 180 collects the sound around the imaging region and estimates the sound source direction SD corresponding to the tracking target. A moving body such as a vehicle generates various sounds such as an engine sound, a motor sound, a tire sound, and a wind noise when moving. The unmanned flying object 100 detects the direction of the tracking target by acquiring the sound emitted by the moving body and estimating the sound source direction SD. Based on the estimated sound source direction SD, the unmanned vehicle 100 controls the gimbal 230 to adjust the shooting direction of the image pickup device 220 toward the sound source direction SD, that is, toward the vehicle 420 to be tracked.

When the unmanned aircraft 100 performs the initial setting of the tracking operation, the feature value of the image and the feature value of the sound related to the tracking target are calculated in the feature value calculation procedure (S12) of the tracking target shown in FIG. As the feature value of the image, for example, SIFT, which is an example of the image feature amount, may be used. For the characteristic value of the sound, for example, a frequency characteristic or the like may be used.

FIG. 13 is a flowchart showing an example of the processing procedure of the automatic tracking operation in the second embodiment. The processing of the automatic tracking operation may be mainly executed by the tracking unit 111 and the sound source direction estimation unit 112 of the control unit 110 in the unmanned aircraft 100. The control unit 110 of the unmanned aircraft 100 captures a predetermined imaging region by the imaging device 220 and acquires an captured image (S51). The control unit 110 extracts an image corresponding to the tracking target from the captured image (S52). At this time, the control unit 110 extracts an image of the region including the tracking target based on the set tracking target range.

Further, the control unit 110 receives the audio signal collected by the sound collecting device 180 and acquires the sound data around the imaging region (S61). The control unit 110 separates the sound data of a plurality of sound sources from the acquired sound data (S62). The control unit 110 separates environmental sounds such as noise from sounds emitted by a subject corresponding to the tracking target by sound source separation processing. In this example, sound data such as running sound emitted by the vehicle 420 to be tracked is separated from the environmental sound. The control unit 110 extracts the separated sound data with the sound component to be tracked as the main component (S63). The control unit 110 calculates the sound feature value as the feature value to be tracked with respect to the sound data of the principal component. The control unit 110 estimates the sound source direction of the sound data of the principal component corresponding to the tracking target, for example, from the characteristic value of the sound (S64). In the sound source direction estimation process, the control unit 110 estimates the arrival direction of the sound from the sound source by using, for example, the phase information of the sound data acquired by each microphone by the microphone array of the sound collecting device 180.

The control unit 110 searches for a tracking target in the extracted image of a predetermined area (S53). In the tracking target search process, the control unit 110 compares the feature values of the images, compares the feature values of the extracted image of the predetermined region with the previously calculated feature values of the tracking target, and determines whether or not they match. do. At this time, the control unit 110 uses the estimation result of the sound source direction to limit the region for searching the tracking target in the captured image based on the sound source direction. In the tracking target search process, the control unit 110 may update the feature value of the tracking target image by using the tracking target image area in the currently captured captured image. The control unit 110 may extract and search for a region that matches the feature value of the tracking target from the entire region of the captured image. In the tracking target search process, the control unit 110 may update the characteristic value of the tracking target sound by using the currently acquired tracking target sound data. The control unit 110 may search by extracting a predetermined region corresponding to the sound source direction from the entire region of the captured image.

The control unit 110 determines the identity of the tracking target depending on whether or not the feature values of the tracking target match. The control unit 110 determines whether or not adjustment of the shooting direction is necessary based on the determination result of the feature value of the tracking target and the estimation result of the sound source direction (S54). When it is necessary to adjust the shooting direction (S54: Yes), the control unit 110 detects the direction of the tracking target based on the estimation result of the sound source direction, and adjusts the shooting direction by the imaging device 220 (S55). At this time, the control unit 110 controls the movement of at least one of the image pickup device 220 and the gimbal 230, and adjusts the imaging direction of the image pickup device 220 so as to move toward the tracking target. The control unit 110 may control the rotary wing mechanism 210 to move and rotate the unmanned vehicle 100 to adjust the photographing direction. Then, the control unit 110 repeatedly executes the search process for the tracking target in the captured image (S51 to S54, S61 to S64).

In the second embodiment described above, the unmanned vehicle estimates the sound source direction corresponding to the tracking target, and uses the sound source direction estimation result to search for the tracking target. In the embodiment, the area for searching the tracking target in the captured image is limited according to the sound source direction, and image processing related to calculation and comparison of feature values of the tracking target is performed. As a result, the amount of processing related to tracking can be reduced. In addition, robustness during tracking operation can be improved. When there are a plurality of tracking target candidates in the captured image, the tracking target can be accurately specified depending on the sound source direction. In addition, the shooting direction can be adjusted using the sound source direction estimation result, and the tracking operation can be corrected. According to the embodiment, according to the direction of the sound source of the tracking target, the tracking target can be imaged without losing sight of the tracking target, the tracking target in the captured image can be accurately searched, and the tracking operation can be continued. In this case, the operation of resetting the tracking target becomes unnecessary, and the operability can be improved.

According to the present embodiment, when tracking a tracking target using a captured image in a flying object, the sound source direction corresponding to the tracking target can be estimated, and the shooting direction by the imaging device can be adjusted based on the estimated sound source direction. As a result, the imaging device can be directed to the tracking target, and the tracking result can be corrected. Therefore, the operation of resetting the tracking target becomes unnecessary, and the operability can be improved.

According to the present embodiment, when the tracking of the tracking target fails, the sound information emitted by the tracking target can be acquired, the sound source direction can be estimated, and the shooting direction can be adjusted according to the sound source direction. As a result, even if the tracking of the tracking target fails, the imaging direction can be adjusted so that the imaging device is directed to the tracking target, and the tracking result can be corrected.

According to the present embodiment, when tracking a tracking target using a captured image, an area in which sound information emitted by the tracking target is acquired, the sound source direction is estimated, and the tracking target in the captured image is searched according to the sound source direction. Can be limited. As a result, the tracking result can be corrected, the amount of processing related to tracking can be reduced, and the robustness during the tracking operation can be improved.

Although the present disclosure has been described above using the embodiments, the technical scope of the invention according to the present disclosure is not limited to the scope described in the above-described embodiments. It will be apparent to those skilled in the art to make various changes or improvements to the embodiments described above. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The execution order of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawing is particularly "before" and "prior to". As long as the output of the previous process is not used in the subsequent process, it can be realized in any order. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it does not mean that it is essential to carry out in this order. do not have.

10 Flight control system 50 Terminal 51 Processing unit 52 Memory 53 Wireless communication unit 54 Display unit 55 Operation unit 56 Interface unit 57 Storage 58 Battery 100 Unmanned aerial vehicle (UAV)
102 UAV main unit 110 Control unit 111 Tracking unit 112 Sound source direction estimation unit 150 Communication interface 160 Memory 170 Storage 180 Sound pickup device 190 Battery 210 Rotating wing mechanism 220 Imaging device 230 Gimbal 240 GPS receiver 250 Inertial measurement unit 260 Magnetic compass 270 Atmospheric altimeter 280 ultrasonic sensor

Claims (18)

An imaging device that captures a predetermined tracking target, and
A sound collecting device that collects ambient sound including the tracking target, and
An air vehicle having a control unit that executes processing related to tracking the tracking target.
The control unit
The tracking target is tracked using the captured image captured by the imaging device, and the tracking target is tracked.
The sound source direction corresponding to the tracking target is estimated from the sound information emitted by the tracking target acquired by the sound collecting device, and the sound source direction corresponding to the tracking target is estimated.
On the basis of the sound source direction estimation results by adjusting the imaging direction of the imaging device,
The control unit
When the tracking of the tracking target fails, the sound information emitted by the tracking target is acquired, the sound source direction is estimated, and the shooting direction is adjusted according to the sound source direction.
Aircraft. The control unit
In the tracking of the tracking target,
The feature value of the image related to the tracking target in the captured image is calculated, and the feature value of the image is compared to search for the tracking target.
The flying object according to claim 1. The control unit
Calculate the characteristic value of the sound related to the tracking target in the sound information,
The sound source direction of the tracking target is estimated from the characteristic value of the sound, and the tracking target is tracked according to the sound source direction.
The flying object according to claim 1 or 2. The control unit
In the tracking of the tracking target,
The region for searching the tracking target in the captured image is limited according to the estimated sound source direction.
The flying object according to any one of claims 1 to 3. The control unit
In the estimation of the sound source direction,
The sound component of the tracking target is extracted from the sound information acquired by the sound collecting device, and the sound component of the tracking target is estimated in the sound source direction.
The flying object according to any one of claims 1 to 4. The control unit
In the adjustment of the shooting direction,
Controlling the movement of at least one of the rotor blade mechanism of the flying object and the support mechanism supporting the image pickup device.
The flying object according to any one of claims 1 to 5. A flight control system having an air vehicle and a terminal that communicates with the air vehicle.
The flying object
An imaging device that captures a predetermined tracking target, and
A sound collecting device that collects ambient sound including the tracking target, and
A communication unit that transmits the candidate information of the tracking target to the terminal, and
The process related to the tracking of the tracking target is executed, the tracking target is tracked using the image captured by the imaging device , and when the tracking of the tracking target fails, the sound collecting device acquires the tracking target. It has a control unit that estimates the sound source direction corresponding to the tracking target from the sound information emitted by the tracking target, and adjusts the shooting direction by the imaging device based on the estimation result of the sound source direction.
The terminal
A communication unit that receives the candidate information to be tracked, and
A display unit that displays the candidates to be tracked, and
It has an operation unit for inputting the setting of the tracking target.
Flight control system. The display unit of the terminal is
When the tracking of the tracking target fails, the tracking failure display is displayed.
The flight control system according to claim 7. A flight control method for controlling an air vehicle that tracks a predetermined tracking target.
A step of imaging the tracking object with an imaging device,
A step of collecting the surrounding sound including the tracking target by the sound collecting device, and
A step of tracking the tracking target using the captured image, and
A step of estimating the sound source direction corresponding to the tracking target from the sound information emitted by the tracking target that has collected the sound, and a step of estimating the sound source direction corresponding to the tracking target.
Have a, and adjusting the imaging direction of the imaging device based on the sound source direction estimation result,
The step of estimating the sound source direction is
When the tracking of the tracking target fails, the sound information emitted by the tracking target is acquired to estimate the sound source direction, and the sound source direction is estimated.
The step of adjusting the shooting direction is
The shooting direction is adjusted according to the estimated sound source direction.
Flight control method. The step of tracking the tracking target is
The feature value of the image related to the tracking target in the captured image is calculated, and the feature value of the image is compared to search for the tracking target.
The flight control method according to claim 9. It has a step of calculating a characteristic value of a sound related to the tracking target in the sound information.
The step of tracking the tracking target is
The sound source direction of the tracking target is estimated from the characteristic value of the sound, and the tracking target is tracked according to the sound source direction.
The flight control method according to claim 9 or 10. The step of tracking the tracking target is
The region for searching the tracking target in the captured image is limited according to the estimated sound source direction.
The flight control method according to any one of claims 9 to 11. The step of tracking the tracking target is
The sound component of the tracking target is extracted from the sound information acquired by the sound collecting device, and the sound component of the tracking target is estimated in the sound source direction.
The flight control method according to any one of claims 9 to 12. The step of adjusting the shooting direction is
Controlling the movement of at least one of the rotor blade mechanism of the flying object and the support mechanism supporting the image pickup device.
The flight control method according to any one of claims 9 to 13. A step of transmitting the candidate information of the tracking target to a terminal communicating with the flying object, and
A step of receiving the candidate information of the tracking target by the terminal, and
A step of displaying the candidate to be tracked by the display unit, and
It has a step of inputting the setting of the tracking target.
The flight control method according to claim 9. When the tracking of the tracking target fails, the display unit has a step of displaying the tracking failure.
The flight control method according to claim 15. To a computer that controls an air vehicle that tracks a given tracking target,
A step of imaging the tracking object with an imaging device,
A step of collecting the surrounding sound including the tracking target by the sound collecting device, and
A step of tracking the tracking target using the captured image, and
When the tracking of the tracking target fails , the step of estimating the sound source direction corresponding to the tracking target from the sound information emitted by the tracking target that has collected the sound, and the step of estimating the sound source direction corresponding to the tracking target.
To execute the step of adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction.
program. To a computer that controls an air vehicle that tracks a given tracking target,
A step of imaging the tracking object with an imaging device,
A step of collecting the surrounding sound including the tracking target by the sound collecting device, and
A step of tracking the tracking target using the captured image, and
When the tracking of the tracking target fails , the step of estimating the sound source direction corresponding to the tracking target from the sound information emitted by the tracking target that has collected the sound, and the step of estimating the sound source direction corresponding to the tracking target.
A computer-readable computer that records a program for adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction and executing the step.
recoding media.

Images