JP2019006154A - Flight vehicle, flight control system, flight control method, program, and recording medium - Google Patents

Abstract

To enable tracking to be easily continued even when automatic tracking of a tracking object fails.SOLUTION: A flight vehicle includes: an imaging device for taking an image of a predetermined tracking object; a sound pickup device for picking up a sound of the circumference including the tracking object; and a control part for executing processing related to tracking of the tracking object. The control part tracks the tracking object by using the taken image taken by the imaging device. A sound source direction corresponding to the tracking object is estimated by sound information acquired by the sound pickup device. An imaging direction of the imaging device is adjusted on the basis of the result of estimation in the sound source direction.SELECTED DRAWING: Figure 8

Description

  The present disclosure relates to an aircraft capable of tracking a predetermined object, a flight control system, a flight control method for controlling the aircraft, a program, and a recording medium.

  2. Description of the Related Art A flying body (for example, UAV: Unmanned Aerial Vehicle) that is equipped with an imaging device and is capable of performing aerial shooting that captures an image of a subject while flying in the air is known. Among the flying bodies that can be used for this kind of aerial photography, there is one that has a function of automatically tracking a predetermined target such as a person or a vehicle as a main subject.

  When the flying object is tracking, if there is a large change in the direction, distance, position, etc. of the subject to be tracked, or if the subject is hidden behind another object Sometimes tracking fails. When the tracking operation is interrupted in this way, the tracking is resumed by the user operating the terminal for controlling the flying object to reset the tracking target again.

  As an example of a flying object having an automatic tracking function, for example, Patent Document 1 discloses an autonomous flying moving object capable of tracking a target even in a situation where the bird's-eye view cannot be seen. In this conventional example, when the target cannot be recognized, the autonomous flying vehicle searches for a route near the position where the target cannot be recognized based on the map information, and performs tracking of the target based on the searched route. It is.

JP 2017-068298 A

  In a flying object having a conventional tracking function, when the tracking object is lost during the tracking operation, an operation such as setting the tracking object again by using a terminal for controlling the flying object by the user is required. When performing the resetting operation of the tracking target, the user cannot confirm the state of the aircraft body at the same time as the terminal operation, and therefore it is difficult to simultaneously perform the tracking target setting and the flying vehicle maneuvering. As described above, the conventional aircraft has a problem in operability for continuing the tracking when the tracking of the tracking target fails.

  In one aspect, the vehicle includes an imaging device that captures an image of a predetermined tracking target, a sound collection device that collects surrounding sounds including the tracking target, and a control unit that executes processing related to tracking of the tracking target. The control unit tracks the tracking target using the captured image captured by the imaging device, estimates the sound source direction corresponding to the tracking target based on the sound information acquired by the sound collection device, and based on the estimation result of the sound source direction It is a flying body that adjusts the shooting direction of the imaging device.

  In tracking the tracking target, the control unit may calculate a feature value of the image related to the tracking target in the captured image and compare the feature values of the images to search for the tracking target.

  When the tracking of the tracking target fails, the control unit may acquire sound information emitted from the tracking target, estimate the sound source direction, and adjust the shooting direction according to the sound source direction.

  The control unit may calculate a sound feature value related to the tracking target in the sound information, estimate a sound source direction of the tracking target based on the sound feature value, and track the tracking target according to the sound source direction.

  In the tracking of the tracking target, the control unit may limit a region for searching for the tracking target in the captured image according to the estimated sound source direction.

  In the estimation of the sound source direction, the control unit may extract a sound component to be tracked from sound information acquired by the sound collection device, and perform a sound source direction estimation process for the sound component to be tracked.

  The control unit may control at least one movement of the rotating wing mechanism of the flying object and the support mechanism that supports the imaging device in the adjustment of the shooting direction.

  In one aspect, a flight control system comprising a flying object and a terminal that communicates with the flying object, wherein the flying object collects an imaging device that images a predetermined tracking target and ambient sounds including the tracking target. A sound collection device for sounding, a communication unit for transmitting tracking target candidate information to a terminal, and processing for tracking the tracking target, and tracking the tracking target using a captured image captured by the imaging device A control unit that estimates a sound source direction corresponding to a tracking target based on sound information acquired by the sound collection device, and adjusts a shooting direction by the imaging device based on an estimation result of the sound source direction. The flight control system includes a communication unit that receives target candidate information, a display unit that displays tracking target candidates, and an operation unit that performs input related to setting of the tracking target.

  The display unit of the terminal may perform tracking failure display when tracking of the tracking target fails.

  In one aspect, a flight control method for controlling a flying object that tracks a predetermined tracking target, the step of imaging the tracking target with an imaging device, and the collection of ambient sounds including the tracking target with a sound pickup device A step of tracking a tracking target using the picked-up captured image, a step of estimating a sound source direction corresponding to the tracking target based on collected sound information, and photographing by an imaging device based on the estimation result of the sound source direction Adjusting the direction, and a flight control method.

  The step of tracking the tracking target may include a step of calculating a feature value of the image related to the tracking target in the captured image and searching for the tracking target by comparing the feature values of the image.

  The step of estimating the sound source direction includes a step of estimating sound source direction by acquiring sound information emitted from the tracking target when tracking of the tracking target fails, and the step of adjusting the shooting direction is performed on the estimated sound source direction. Accordingly, a step of adjusting the photographing direction may be included.

  A step of calculating a sound feature value related to the tracking target in the sound information, and the step of tracking the tracking target estimates a sound source direction of the tracking target based on the sound feature value and tracks the tracking target according to the sound source direction. Steps may be included.

  The step of tracking the tracking target may include a step of limiting a region for searching the tracking target in the captured image according to the estimated sound source direction.

  The step of tracking the tracking target may include a step of extracting a sound component of the tracking target from the sound information acquired by the sound collecting device and performing a sound source direction estimation process on the tracking target sound component.

  The step of adjusting the shooting direction may include a step of controlling the movement of at least one of the rotating wing mechanism of the flying object and the support mechanism that supports the imaging device.

  In the flight control method, the step of transmitting the tracking target candidate information to a terminal that communicates with the flying object, the step of receiving the tracking target candidate information by the terminal, the step of displaying the tracking target candidate by a display unit, And making an input regarding the setting of the tracking target.

  The flight control method may include a step of displaying a tracking failure on the display unit when the tracking of the tracking target fails.

  In one aspect, the program captures the tracking object with an imaging device on a computer that controls a flying object that tracks the predetermined tracking object, and collects ambient sounds including the tracking object with a sound collection device. , Tracking the tracking target using the captured image, estimating the sound source direction corresponding to the tracking target from the collected sound information, and adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction And a step for executing the program.

  In one aspect, the recording medium includes a step of capturing an image of the tracking target by an imaging device and a step of collecting a surrounding sound including the tracking target by a sound collecting device on a computer that controls a flying object that tracks the predetermined tracking target. Tracking the tracking target using the captured image, estimating the sound source direction corresponding to the tracking target from the collected sound information, and determining the shooting direction by the imaging device based on the estimation result of the sound source direction. A computer-readable recording medium having recorded thereon a program for executing the adjusting step.

  Note that the summary of the invention described above does not enumerate all the features of the present disclosure. In addition, a sub-combination of these feature groups can also be an invention.

It is a mimetic diagram showing the example of composition of the flight control system in an embodiment. It is a figure which shows an example of the external appearance of an unmanned air vehicle. It is a block diagram which shows an example of the hardware constitutions of an unmanned air vehicle. It is a block diagram which shows an example of the hardware constitutions of a terminal. It is a figure which shows typically the mode of 1st Embodiment which tracks a tracking object with an unmanned air vehicle. It is a flowchart which shows an example of the process sequence regarding the initial setting of the tracking operation | movement in embodiment. It is a figure which shows an example of the initial setting screen for performing the initial setting of the tracking operation | movement in embodiment. It is a flowchart which shows an example of the process sequence of the automatic tracking operation | movement in 1st Embodiment. It is a sequence diagram which shows an example of the process sequence at the time of tracking failure in embodiment. It is a figure which shows an example of the notification screen which notifies tracking failure in embodiment. It is a figure which shows an example of the confirmation screen at the time of the reset of the tracking object in embodiment. It is a figure which shows typically the mode of 2nd Embodiment which tracks a tracking object with an unmanned air vehicle. It is a flowchart which shows an example of the process sequence of the automatic tracking operation | movement in 2nd Embodiment.

  Hereinafter, although this indication is explained through an embodiment of the invention, the following embodiment does not limit the invention concerning a claim. Not all combinations of features described in the embodiments are essential for the solution of the invention.

  The claims, the description, the drawings, and the abstract include matters subject to copyright protection. The copyright owner will not object to any number of copies of these documents as they appear in the JPO file or record. However, in other cases, all copyrights are reserved.

  In the flight control method according to the present disclosure, various processes (steps) in the information processing apparatus for controlling the flight of the flying object are defined. The flying object includes an aircraft (eg, drone, helicopter) that moves in the air. The flying vehicle may be an unmanned aerial vehicle (UAV). The flying object flies along a predetermined flight path in order to perform various operations involving flying such as aerial photography, transportation of objects, or spraying work of agricultural chemicals, fertilizers, water and the like. When performing the automatic tracking operation, the flying object flies while tracking the set tracking target.

  The information processing apparatus according to the present disclosure is a computer, and the information processing apparatus may be included in the flying object itself, for example. An information processing apparatus according to the present disclosure includes a transmitter for instructing remote control of various processes including movement of a flying object, or a terminal device connected to be able to input and output information and data with the transmitter, Alternatively, various terminals (platforms) such as a terminal device connected so as to be able to input and output information and data with the flying object are included. The terminal may be a PC, a tablet terminal, a portable terminal, or the like, for example.

  The program according to the present disclosure is a program for causing an information processing apparatus to execute various processes (steps).

  The recording medium according to the present disclosure stores a program (that is, a program for causing an information processing apparatus to execute various processes (steps)).

  The flight control system according to the present disclosure includes a flying object as an example of a moving object and a terminal (platform) for remotely controlling the operation or processing of the flying object. The flight control system according to the present disclosure may be configured to track and track a tracking target with a single flying object.

  In each embodiment according to the present disclosure described below, an unmanned aerial vehicle (UAV) is illustrated as a flying object. In the drawings attached to this specification, an unmanned air vehicle is denoted as “UAV”. In each embodiment described below, the flying object sets a tracking target and flies while moving the position following the tracking target.

  In each embodiment described below, the information processing apparatus can generate, acquire, and change feature value information including image information, sound information, and the like as a tracking target feature value in the flying object. The information processing device can generate, acquire, and change flight position information including at least one of the flight start position, flight end position, predetermined position on the flight path, flight altitude, flight speed, etc. It is. When the information processing device is included in the terminal, the information processing device can communicate with the flying object, and can transmit the characteristic value information of the tracking target and the flight position information of the flying object to the flying object. Here, “communication” is a broad concept including data communication in general, and includes not only a case where a wired connection is made by a cable or the like but also a case where a connection is made by wireless communication. In addition to the case where the information processing apparatus directly communicates with the flying object, the case where the information processing apparatus communicates indirectly via a transmitter or a storage medium is also included.

  Drawing 1 is a mimetic diagram showing the example of composition of flight control system 10 in an embodiment. The flight control system 10 includes an unmanned air vehicle 100 and a terminal 50. The unmanned air vehicle 100 and the terminal 50 can communicate with each other using wired communication or wireless communication (for example, a wireless local area network (LAN) or Bluetooth (registered trademark)). The terminal 50 is used in a state of being held by both hands of a person who uses the terminal 50 (hereinafter referred to as “user”), for example. The terminal 50 may be a transmitter, a tablet terminal, a portable terminal, a PC, or the like, for example. The terminal 50 may have a configuration in which a tablet terminal or a portable terminal is attached to a transmitter so that they can communicate with each other.

  FIG. 2 is a diagram illustrating an example of the appearance of the unmanned air vehicle 100. The unmanned aerial vehicle 100 is an example of a moving body that includes an imaging device 220 as an example of an imaging unit and moves. The moving body is a concept including, in addition to the unmanned air vehicle 100, other aircraft that moves in the air, vehicles that move on the ground, ships that move on the water, and the like. Here, as shown in FIG. 2, it is assumed that a roll axis (see the x-axis in FIG. 2) is defined in a direction parallel to the ground and along the moving direction STV0. In this case, the pitch axis (see the y-axis in FIG. 2) is defined in a direction parallel to the ground and perpendicular to the roll axis, and further, the yaw axis is perpendicular to the ground and perpendicular to the roll axis and the pitch axis. (Z-axis in FIG. 2) is defined.

  The unmanned air vehicle 100 includes a UAV main body 102, a rotary wing mechanism 210, an imaging device 220, a gimbal 230, and a sound collection device 180. The unmanned air vehicle 100 can move based on a remote control instruction transmitted from the terminal 50, for example. The movement of the unmanned air vehicle 100 means a flight, and includes at least ascending, descending, left turning, right turning, left horizontal movement, and right horizontal movement.

  The unmanned air vehicle 100 includes a plurality of rotary wing mechanisms (propellers) 210. The unmanned air vehicle 100 includes, for example, four rotary wing mechanisms 210. The unmanned aerial vehicle 100 moves the unmanned aerial vehicle 100 by controlling the rotation of the rotary wing mechanism 210. However, the number of rotor blades is not limited to four. The unmanned air vehicle 100 may be a fixed wing aircraft that does not have rotating wings.

  The imaging device 220 is a camera that images a subject (for example, a person, a vehicle, etc.) included in a desired imaging range. The imaging device 220 is attached to a gimbal 230 that is an example of a support mechanism, and the imaging range is adjusted by the movement of the gimbal 230.

  The sound collection device 180 is a microphone that collects sound around the imaging range including the subject. The sound collection device 180 may be a microphone array having a plurality of microphones.

  Next, a configuration example of the unmanned air vehicle 100 will be described.

  FIG. 3 is a block diagram illustrating an example of a hardware configuration of the unmanned air vehicle 100. The unmanned air vehicle 100 includes a control unit 110, a communication interface 150, a memory 160, a storage 170, a sound collection device 180, a battery 190, a rotary wing mechanism 210, an imaging device 220, a gimbal 230, and a GPS. The configuration includes a receiver 240, an inertial measurement device 250, a magnetic compass 260, a barometric altimeter 270, and an ultrasonic sensor 280.

  The control unit 110 is configured using a processor, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The control unit 110 performs signal processing for overall control of operations of each unit of the unmanned air vehicle 100, data input / output processing with other units, data calculation processing, and data storage processing. The control unit 110 has a function of executing processing related to flight control in the unmanned air vehicle 100.

  The control unit 110 controls the flight of the unmanned air vehicle 100 according to a program stored in the memory 160 or the storage 170. Further, the control unit 110 controls the movement (that is, the flight) of the unmanned air vehicle 100 in accordance with a command received from the remote terminal 50 via the communication interface 150.

  The control unit 110 controls the flight of the unmanned air vehicle 100 by controlling the rotary wing mechanism 210. That is, the control unit 110 controls the position and direction of the unmanned air vehicle 100 including the latitude, longitude, and altitude by controlling the rotary wing mechanism 210. The control unit 110 controls the rotary wing mechanism 210 based on position information acquired by at least one of the GPS receiver 240, the inertial measurement device 250, the magnetic compass 260, the barometric altimeter 270, and the ultrasonic sensor 280.

  The control unit 110 acquires posture information indicating the posture state of the imaging device 220 from the gimbal 230 as information indicating the imaging direction of the imaging device 220, for example. The information indicating the posture state of the imaging device 220 indicates a rotation angle from the reference rotation angle of the pitch axis and yaw axis of the gimbal 230. The control unit 110 acquires information indicating the direction of the unmanned air vehicle 100.

  The control unit 110 controls the imaging device 220 and the gimbal 230. The control unit 110 controls the imaging range of the imaging device 220 by changing the imaging direction or angle of view of the imaging device 220. The control unit 110 controls the imaging range of the imaging device 220 supported by the gimbal 230 by controlling the rotation mechanism of the gimbal 230.

  The control unit 110 acquires image data of a subject imaged by the imaging device 220 (hereinafter sometimes referred to as “captured image”).

  The control unit 110 acquires sound information of the subject and surroundings collected by the sound collection device 180.

  The communication interface 150 is an example of a communication unit and communicates with the terminal 50. The communication interface 150 receives various types of information regarding the tracking target, the flight path, and the like from the terminal 50. The communication interface 150 receives various commands from the terminal 50 to the control unit 110.

  The memory 160 is an example of a storage unit. In the memory 160, the control unit 110 includes a rotary blade mechanism 210, an imaging device 220, a gimbal 230, a GPS receiver 240, an inertial measurement device 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, a communication interface 150, a storage 170, And a program necessary for controlling the sound collecting device 180 is stored. The memory 160 stores various types of information and data used during processing by the control unit 110. The memory 160 may be a computer-readable recording medium, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and It may include at least one flash memory such as a USB memory. The memory 160 may be provided inside the unmanned aerial vehicle 100 or may be detachable from the unmanned aerial vehicle 100. The memory 160 may store tracking information including feature value information of the tracking target.

  The storage 170 is an example of a storage unit. The storage 170 accumulates and holds various data and information. The storage 170 may be an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a USB memory, or the like. The storage 170 may be provided inside the unmanned aerial vehicle 100 or may be detachable from the unmanned aerial vehicle 100.

  The sound collection device 180 includes a microphone array in which a plurality of microphones are arranged in a line. The sound collection device 180 may include, for example, two or more microphones and a signal processing circuit that processes a sound signal collected by each microphone. The sound collection device 180 collects surrounding sounds including the tracking target to estimate the sound source direction of the tracking target, and acquires information on the sound signal of the subject including the tracking target collected by a plurality of microphones. To do.

  The battery 190 has a function as a drive source for each part of the unmanned air vehicle 100 and supplies necessary power to each part of the unmanned air vehicle 100.

  The rotary blade mechanism 210 includes a plurality of rotary blades and a plurality of drive motors that rotate the plurality of rotary blades. The rotary wing mechanism 210 generates airflow in a specific direction by rotating the rotary wing, and controls the flight (ascending, descending, horizontally moving, turning, tilting, etc.) of the unmanned air vehicle 100.

  The imaging device 220 captures a subject within a desired imaging range and generates captured image data. The captured image may be a moving image or a still image. Image data obtained by imaging by the imaging device 220 is stored in the memory or the memory 160, the storage 170, or the like included in the imaging device 220.

  The gimbal 230 is an example of a support mechanism that supports the imaging device 220, and supports the imaging device 220 so as to be rotatable about at least one axis. The gimbal 230 may support the imaging device 220 rotatably about the yaw axis, pitch axis, and roll axis. The gimbal 230 may change the imaging direction of the imaging device 220 by rotating the imaging device 220 about at least one of the yaw axis, the pitch axis, and the roll axis.

  The GPS receiver 240 receives a plurality of signals indicating times and positions (coordinates) of each GPS satellite transmitted from a plurality of navigation satellites (that is, GPS satellites). The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned air vehicle 100) based on the received signals. The GPS receiver 240 outputs position information of the unmanned air vehicle 100 to the control unit 110. The calculation of the position information of the GPS receiver 240 may be performed by the control unit 110 instead of the GPS receiver 240. In this case, information indicating the time and the position of each GPS satellite included in the plurality of signals received by the GPS receiver 240 is input to the control unit 110.

  An inertial measurement unit (IMU) 250 detects the attitude of the unmanned air vehicle 100 and outputs the detection result to the control unit 110. The inertial measurement device 250 uses, as the attitude of the unmanned aerial vehicle 100, accelerations in the three axial directions of the front and rear, left and right, and up and down of the unmanned aerial vehicle 100, and angular velocities in the three axial directions of the pitch axis, roll axis, and yaw axis. To detect.

  The magnetic compass 260 detects the nose direction of the unmanned air vehicle 100 and outputs the detection result to the control unit 110. Barometric altimeter 270 detects the altitude at which unmanned air vehicle 100 flies, and outputs the detection result to control unit 110. The ultrasonic sensor 280 emits ultrasonic waves, detects ultrasonic waves reflected by the ground or an object, and outputs the detection result to the control unit 110. A detection result shows the distance (namely, altitude) from unmanned air vehicle 100 to the ground, for example. The detection result may indicate a distance from the unmanned air vehicle 100 to the object, for example.

Next, an example of the function of the control unit 110 of the unmanned air vehicle 100 will be described.
The control unit 110 includes a tracking unit 111 that performs processing related to tracking of a predetermined tracking target in a subject in the image data of the captured image captured by the imaging device 220, and sound information data ( A sound source direction estimation unit 112 that performs processing related to estimation of a predetermined sound source direction based on sound data.

  The tracking unit 111 moves the unmanned air vehicle 100 to track the set tracking target, and controls the rotary wing mechanism 210, the imaging device 220, and the gimbal 230 so that the imaging device 220 can capture the tracking target. . The tracking unit 111 tracks the tracking target using the captured image acquired by the imaging device 220. The tracking unit 111 calculates, compares, and updates the characteristic value of the tracking target in the captured image captured by the imaging device 220 and searches for the tracking target. The tracking unit 111 compares the feature value of the tracking target of the previous frame with the feature value of the current frame in the captured image of the moving image, and searches for a tracking target having the same feature value.

  The sound source direction estimating unit 112 separates the sound emitted from the tracking target and the environmental sound from the sound data acquired by the sound collecting device 180, acquires the phase information of the tracking target sound acquired by each microphone, and tracks it. Estimate the direction of the sound source corresponding to the target.

  The tracking unit 111 controls the imaging device 220 and the gimbal 230 based on the estimated sound source direction, and adjusts the imaging direction of the imaging device 220 so as to be directed to the tracking target according to the sound source direction, for example.

  Next, a configuration example of the terminal 50 will be described.

  FIG. 4 is a block diagram illustrating an example of a hardware configuration of the terminal 50. The terminal 50 includes a processing unit 51, a memory 52, a wireless communication unit 53, a display unit 54, an operation unit 55, an interface unit 56, a storage 57, and a battery 58. The terminal 50 is an example of an information processing apparatus having a function of an operation terminal that transmits an instruction for remotely controlling the unmanned air vehicle 100. The terminal 50 is an example of an information processing apparatus having a function of a setting terminal that inputs and outputs various information and data related to the flight of the unmanned air vehicle 100. Note that the terminal 50 may have a separate configuration in which a transmitter and a tablet terminal or a mobile terminal are connected to each other.

  The processing unit 51 is configured using a processor (for example, a CPU, MPU, or DSP). The processing unit 51 performs signal processing for overall control of operations of each unit of the terminal 50, data input / output processing with other units, data calculation processing, and data storage processing.

  The processing unit 51 may acquire data and information from the unmanned air vehicle 100 via the wireless communication unit 53. The processing unit 51 may acquire data and information from other devices via the interface unit 56. The processing unit 51 may acquire data and information input via the operation unit 55. The processing unit 51 may acquire data and information held in the memory 52. The processing unit 51 may send data and information to the display unit 54 and cause the display unit 54 to display display information based on the data and information. The processing unit 51 may send data and information to the storage 57 and store the data and information. The processing unit 51 may acquire data and information stored in the storage 57.

  Based on the operation input of the operation unit 55, the processing unit 51 may perform setting input related to setting of the tracking target such as setting of an image area including the tracking target.

  The processing unit 51 may generate an operation signal for remotely controlling the movement of the unmanned air vehicle 100 based on the operation input of the operation unit 55. The processing unit 51 may transmit the generated operation signal as a movement control command to the unmanned aerial vehicle 100 via the wireless communication unit 53 to remotely control the unmanned aerial vehicle 100.

  The processing unit 51 may generate at least one display screen among a setting screen, an operation screen, a notification screen, and a confirmation screen to be displayed on the display unit 54.

  The memory 52 is an example of a storage unit. The memory 52 temporarily stores, for example, a ROM (Read Only Memory) in which a program that defines the operation of the processing unit 51 and data of setting values are stored, and various types of information and data that are used when the processing unit 51 performs processing. RAM (Random Access Memory) to be stored. The program and setting value data stored in the ROM of the memory 52 may be copied to a predetermined recording medium (for example, CD-ROM, DVD-ROM).

  The wireless communication unit 53 is an example of a communication unit, and communicates with the unmanned air vehicle 100 and transmits / receives information and data via various antennas via an antenna. The wireless communication method may include, for example, wireless LAN, Bluetooth (registered trademark), short-range wireless communication, or communication via a public wireless line. The wireless communication unit 53 may transmit and receive information and data by communicating with other devices.

  The display unit 54 is configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence) display, and displays various information and data output from the processing unit 51 on the display screen. The display unit 54 may display at least one display screen among a setting screen, an operation screen, a notification screen, and a confirmation screen. The display unit 54 may include a display lamp using, for example, an LED (Light Emission Diode). The display lamp displays at least one of, for example, a wireless connection state between the unmanned air vehicle 100 and the terminal 50, an activated state of the unmanned air vehicle 100, and a remaining capacity of the battery of the unmanned air vehicle 100 or the terminal 50. You can do it.

  The operation unit 55 receives an operation instruction, data, or information input by a user holding the terminal 50. The operation unit 55 may include a joystick, a button, a key, a touch panel, a microphone, and the like. The operation unit 55 is used, for example, in an operation for remotely controlling the movement of the unmanned air vehicle 100 by the user (for example, moving the unmanned air vehicle 100 back and forth, moving left and right, moving up and down, and changing the direction). The operation unit 55 is used, for example, in an operation for inputting a setting regarding a tracking target. The operation unit 55 is used in operations such as confirmation of a tracking target and an instruction to start tracking, for example.

  The interface unit 56 inputs and outputs information and data between the terminal 50 and other devices. The interface unit 56 may be a USB port (not shown) provided in the terminal 50, for example. The interface unit 56 may be an interface other than the USB port.

  The storage 57 is an example of a storage unit. The storage 57 stores and holds various data and information. The storage 57 may be a flash memory, an SSD (Solid State Drive), a memory card, a USB memory, or the like. The storage 57 may be provided so as to be removable from the main body of the terminal 50.

  The battery 58 has a function as a drive source for each part of the terminal 50 and supplies necessary power to each part of the terminal 50.

  Next, the tracking operation by the unmanned air vehicle 100 will be described.

(First embodiment)
FIG. 5 is a diagram schematically illustrating a state of the first embodiment in which a tracking target is tracked by an unmanned air vehicle. In the first embodiment, an example in which a person is tracked as a tracking target and captured is shown. The unmanned aerial vehicle 100 images the person 410 as a subject by the imaging device 220, and sets the person in the captured image as a tracking target. The unmanned air vehicle 100 adjusts the shooting direction by rotating the imaging device 220 with the gimbal 230 so as to continuously capture the set tracking target. At this time, the gimbal 230 rotates the imaging device 220 in, for example, a horizontal direction Dh centered on the yaw axis and a vertical direction Dv centered on the pitch axis. The unmanned aerial vehicle 100 controls the rotary wing mechanism 210 in accordance with the movement of the person 410 to be tracked to follow the position of the unmanned aerial vehicle 100.

  For example, if the tracking target cannot be searched for in the captured image data because the tracking target has moved greatly and deviated from the angle of view of the imaging device 220 and the tracking has failed, the unmanned air vehicle 100 is captured by the sound collection device 180. The surrounding sound is collected, and the sound source direction SD corresponding to the tracking target is estimated. At this time, the person 410 to be tracked emits a sound toward the unmanned air vehicle 100, such as calling the unmanned air vehicle 100 when the tracking failure is recognized. The unmanned air vehicle 100 detects the direction of the tracking target by acquiring the sound emitted by the tracking target and estimating the sound source direction SD. The unmanned aerial vehicle 100 controls the gimbal 230 based on the estimated sound source direction SD so as to adjust the shooting direction of the imaging device 220 toward the sound source direction SD, that is, toward the person 410 to be tracked.

  FIG. 6 is a flowchart illustrating an example of a processing procedure related to the initial setting of the tracking operation according to the embodiment. FIG. 7 is a diagram illustrating an example of an initial setting screen for performing initial setting of the tracking operation according to the embodiment.

  The unmanned air vehicle 100 sets a tracking target in the initial setting of the tracking operation. In this example, a procedure for setting a tracking target using the terminal 50 is shown. The terminal 50 inputs an operation instruction for the tracking target range in the captured image by the user, and sets the tracking target range (S11). At this time, the terminal 50 displays the setting screen 300 having the captured image 310 divided into a plurality of areas on the display unit 54 as in the example illustrated in FIG. 7. When the user designates an area including the tracking target TG1 in the captured image 310 by a touch operation or the like, an image area frame indicating the tracking target range 321 is displayed. When the user touches the setting button (SET) 322 after confirmation, the terminal 50 Sets the corresponding image area as the tracking target range. The terminal 50 transmits the tracking target range information to the unmanned air vehicle 100, and the unmanned air vehicle 100 receives and inputs the tracking target range information to set the tracking target range in the captured image.

  The unmanned air vehicle 100 calculates the feature value of the tracking target image in the image data of the tracking target range in the captured image (S12). For example, SIFT (Scale-Invariant Feature Transform), which is an example of an image feature amount, may be used as the feature value calculated from the image data to be tracked. In addition to SIFT, SURF (Speeded Up Robust Features), GLOH (Gradient Location and Orientation Histogram), FAST (Features from Accelerated Segment Test), and the like may be used as the image feature amount to be tracked. As the feature value of the tracking target, the feature value of the image of only the tracking target may be calculated, or the feature value of the image of a predetermined region including the tracking target may be calculated. Note that the feature value of the tracking target may be calculated in the terminal 50 and transmitted to the unmanned air vehicle 100, and the unmanned air vehicle 100 may acquire the feature value.

  FIG. 8 is a flowchart illustrating an example of the processing procedure of the automatic tracking operation according to the first embodiment. The processing of the automatic tracking operation may be mainly executed by the tracking unit 111 and the sound source direction estimating unit 112 of the control unit 110 in the unmanned air vehicle 100. The control unit 110 of the unmanned air vehicle 100 captures a predetermined imaging area by the imaging device 220 and acquires a captured image (S21). The control unit 110 extracts an image corresponding to the tracking target from the captured image (S22). At this time, the control unit 110 extracts an image of an area including the tracking target based on the set tracking target range. Then, the control unit 110 searches for the tracking target in the extracted image of the predetermined area (S23). In the tracking target search process, the control unit 110 compares the feature values of the image, compares the extracted feature value of the image in the predetermined area with the previously calculated feature value of the tracking target, and determines whether or not they match. To do. In the tracking target search process, the control unit 110 may update the characteristic value of the tracking target using the image area of the tracking target in the currently captured image. Note that the control unit 110 may extract and search a region that matches the characteristic value of the tracking target from the entire region of the captured image.

  The control unit 110 determines whether or not the tracking has failed depending on whether or not the feature values of the tracking target match (S24). When the tracking is successful (S24: No), the control unit 110 repeatedly executes a tracking target search process in the captured image (S21 to S24).

  When tracking fails (S24: Yes), the control part 110 receives the audio | voice signal picked up by the sound collection apparatus 180, and acquires the sound data around an imaging area (S25). The controller 110 separates sound data of a plurality of sound sources from the acquired sound data (S26). In the sound source separation process, the control unit 110 may separate sound data from a plurality of sound sources using, for example, BSS (Blind Source Separation). The control unit 110 separates environmental sound such as noise and sound emitted from the subject corresponding to the tracking target by sound source separation processing. In this example, the audio data emitted by the tracking target person 410 is separated from the environmental sound. The control unit 110 extracts the tracked sound component as a main component from the plurality of separated sound data (S27). The control unit 110 calculates the feature value of the sound as the feature value to be tracked for the main component sound data. The characteristic value calculated from the sound data to be tracked may use, for example, frequency characteristics, presence / absence of directivity, envelope, and the like.

  For the main component sound data corresponding to the tracking target, the control unit 110 estimates the sound source direction based on, for example, a sound feature value (S28). In the sound source direction estimation process, the control unit 110 estimates the arrival direction of the sound from the sound source using, for example, the phase information of the sound data acquired by each microphone by the microphone array of the sound collection device 180. The sound source direction estimation processing includes, for example, a method of estimating a sound source position by spatially scanning an acoustic beam by forming a delay sum array such as a beam forming method, a CSP (Cross-power Spectrum Phase analysis) method, and the like. A method of estimating sound source localization by calculating a delay time by obtaining a difference in arrival time in a plurality of microphones may be used.

  The control unit 110 detects the direction of the tracking target based on the estimation result of the sound source direction, and adjusts the shooting direction by the imaging device 220 (S29). At this time, the control unit 110 controls the movement of at least one of the imaging device 220 and the gimbal 230 and adjusts the shooting direction of the imaging device 220 so as to be directed to the tracking target. The control unit 110 may adjust the shooting direction by controlling the rotating wing mechanism 210 to move and rotate the unmanned air vehicle 100. And the control part 110 repeatedly performs the search process of the tracking target in a captured image (S21-S24).

  FIG. 9 is a sequence diagram illustrating an example of a processing procedure when tracking fails in the embodiment. FIG. 10 is a diagram illustrating an example of a notification screen for notifying tracking failure in the embodiment. FIG. 11 is a diagram illustrating an example of a confirmation screen when the tracking target is reset in the embodiment.

  If tracking of the tracking target fails, the unmanned air vehicle 100 and the terminal 50 may perform processing such as notification to the user and confirmation of the reset tracking target. When the unmanned air vehicle 100 determines that the search for the tracking target has failed during the automatic tracking operation (S31), the unmanned air vehicle 100 transmits information indicating tracking failure, which is an example of tracking state information, to the terminal 50 (S32). ). When receiving the tracking state information indicating the tracking failure, the terminal 50 performs a tracking failure display for notifying the user of the tracking failure status as an example of the tracking state display (S33). At this time, the terminal 50 displays, on the display unit 54, a notification screen 330 that notifies the tracking failure as in the example illustrated in FIG. On the notification screen 330, a tracking failure display (MISSED) 323 in the captured image 310 is displayed and is visible to the user. In addition to the display of the notification screen 330, the terminal 50 may perform notification by sound such as an alarm sound, notification by light such as LED light emission, and the like.

  The unmanned air vehicle 100 performs sound source direction estimation and imaging direction adjustment processing as in the procedure (S25 to S29) shown in FIG. 8, and re-searches the tracking target (S34). In the search for the tracking target, the unmanned air vehicle 100 may limit a region for searching the tracking target in the captured image in accordance with the estimated sound source direction, and extract the tracking target candidates. The unmanned air vehicle 100 transmits information indicating the tracking target candidate extracted by the re-search to the terminal 50 as tracking target candidate information (S35). When the terminal 50 receives the tracking target candidate information, the terminal 50 displays a tracking target candidate indicating the tracking target candidate to the user for confirmation of the tracking target (S36). At this time, the terminal 50 displays a confirmation screen 350 for confirming the tracking target candidate on the display unit 54 as in the example shown in FIG. On the confirmation screen 350, a tracking target frame 324 surrounding the tracking target TG1 in the captured image 310 is displayed, and a tracking target confirmation display (TARGET?) 325 is displayed. When the user touches the confirmation button (OK) 326 after confirming the tracking target, the terminal 50 inputs a confirmation instruction by the user (S37) and transmits a confirmation response to the unmanned air vehicle 100 (S38).

  Upon receiving the confirmation response from the terminal 50, the unmanned air vehicle 100 sets a region including the corresponding tracking target as a new tracking target range, and calculates and updates the characteristic value of the tracking target (S39). The unmanned air vehicle 100 then continues the automatic tracking operation (S40).

  In the first embodiment described above, the unmanned air vehicle estimates the sound source direction corresponding to the tracking target, and corrects the tracking operation using the sound source direction estimation result. In the embodiment, when the tracking of the tracking target fails, the unmanned air vehicle obtains sound information emitted from the tracking target, estimates the sound source direction, and adjusts the shooting direction according to the sound source direction. Thereby, the imaging device can be directed toward the sound source direction corresponding to the tracking target, and the tracking target can be continuously captured so that the tracking target can be tracked again, and the tracking result can be corrected. In addition, the robustness during the tracking operation can be improved. In this case, an operation for resetting the tracking target becomes unnecessary, and operability can be improved.

  When setting the tracking target in the initial setting or setting the tracking target again after the tracking failure, the unmanned air vehicle displays the tracking target candidate on the terminal and inputs an instruction operation after confirmation by the user. Thereby, the user can confirm and set the tracking target accurately and easily, and the operability can be improved.

(Second Embodiment)
FIG. 12 is a diagram schematically illustrating a second embodiment in which a tracking target is tracked by an unmanned air vehicle. In the second embodiment, an example in which a moving body such as a vehicle is tracked as a tracking target and captured is shown. The unmanned air vehicle 100 images the vehicle 420 as a subject by the imaging device 220, and sets the vehicle in the captured image as a tracking target. The unmanned air vehicle 100 adjusts the shooting direction by rotating the imaging device 220 with the gimbal 230 so as to continuously capture the set tracking target. At this time, the gimbal 230 rotates the imaging device 220 in, for example, a horizontal direction Dh centered on the yaw axis and a vertical direction Dv centered on the pitch axis. The unmanned aerial vehicle 100 controls the rotary wing mechanism 210 in accordance with the movement of the tracked vehicle 420 to follow the position of the unmanned aerial vehicle 100.

  When the unmanned air vehicle 100 tracks the tracking target, the sound collecting device 180 collects sound around the imaging region and estimates the sound source direction SD corresponding to the tracking target. A moving body such as a vehicle generates various sounds such as an engine sound, a motor sound, a tire sound, and a wind noise when moving. The unmanned air vehicle 100 detects the direction of the tracking target by acquiring the sound emitted by the moving body and estimating the sound source direction SD. The unmanned aerial vehicle 100 controls the gimbal 230 based on the estimated sound source direction SD so as to adjust the shooting direction of the imaging device 220 toward the sound source direction SD, that is, toward the tracking target vehicle 420.

  When performing the initial setting of the tracking operation, the unmanned air vehicle 100 calculates the feature value of the image and the sound feature value regarding the tracking target in the tracking target feature value calculation procedure (S12) shown in FIG. For example, SIFT, which is an example of an image feature amount, may be used as the image feature value. For example, a frequency characteristic may be used as the sound characteristic value.

  FIG. 13 is a flowchart illustrating an example of the processing procedure of the automatic tracking operation according to the second embodiment. The processing of the automatic tracking operation may be mainly executed by the tracking unit 111 and the sound source direction estimating unit 112 of the control unit 110 in the unmanned air vehicle 100. The control unit 110 of the unmanned air vehicle 100 captures a predetermined imaging area by the imaging device 220 and acquires a captured image (S51). The control unit 110 extracts an image corresponding to the tracking target from the captured image (S52). At this time, the control unit 110 extracts an image of an area including the tracking target based on the set tracking target range.

  Further, the control unit 110 receives the audio signal collected by the sound collection device 180 and acquires sound data around the imaging region (S61). The controller 110 separates sound data of a plurality of sound sources from the acquired sound data (S62). The control unit 110 separates environmental sound such as noise and sound emitted from the subject corresponding to the tracking target by sound source separation processing. In this example, sound data such as running sound generated by the tracked vehicle 420 is separated from the environmental sound. The control unit 110 extracts the tracked sound component as the main component from the plurality of separated sound data (S63). The control unit 110 calculates the feature value of the sound as the feature value to be tracked for the main component sound data. For the main component sound data corresponding to the tracking target, the control unit 110 estimates the sound source direction based on, for example, a sound feature value (S64). In the sound source direction estimation process, the control unit 110 estimates the arrival direction of the sound from the sound source using, for example, the phase information of the sound data acquired by each microphone by the microphone array of the sound collection device 180.

  The control unit 110 searches for the tracking target in the extracted image of the predetermined area (S53). In the tracking target search process, the control unit 110 compares the feature values of the image, compares the extracted feature value of the image in the predetermined area with the previously calculated feature value of the tracking target, and determines whether or not they match. To do. At this time, the control part 110 limits the area | region which searches the tracking target in a captured image based on a sound source direction using the estimation result of a sound source direction. In the tracking target search process, the control unit 110 may update the feature value of the tracking target image using the tracking target image area in the currently captured image. Note that the control unit 110 may extract and search a region that matches the characteristic value of the tracking target from the entire region of the captured image. The control unit 110 may update the characteristic value of the tracking target sound using the currently acquired tracking target sound data in the tracking target search process. Note that the control unit 110 may extract and search a predetermined area corresponding to the sound source direction from the entire area of the captured image.

  The control unit 110 determines the identity of the tracking target based on whether the feature values of the tracking target match. The control unit 110 determines whether the adjustment of the shooting direction is necessary based on the determination result of the characteristic value to be tracked and the estimation result of the sound source direction (S54). When adjustment of the shooting direction is necessary (S54: Yes), the control unit 110 detects the direction of the tracking target based on the estimation result of the sound source direction, and adjusts the shooting direction by the imaging device 220 (S55). At this time, the control unit 110 controls the movement of at least one of the imaging device 220 and the gimbal 230 and adjusts the shooting direction of the imaging device 220 so as to be directed to the tracking target. The control unit 110 may adjust the shooting direction by controlling the rotating wing mechanism 210 to move and rotate the unmanned air vehicle 100. And the control part 110 repeatedly performs the search process of the tracking target in a captured image (S51-S54, S61-S64).

  In the second embodiment described above, the unmanned air vehicle estimates the sound source direction corresponding to the tracking target, and searches for the tracking target using the sound source direction estimation result. In the embodiment, an area for searching for a tracking target in a captured image is limited according to a sound source direction, and image processing related to calculation and comparison of a tracking target feature value is performed. Thereby, the processing amount regarding tracking can be reduced. In addition, the robustness during the tracking operation can be improved. When there are a plurality of tracking target candidates in the captured image, the tracking target can be accurately specified by the sound source direction. Further, the tracking direction can be corrected by adjusting the shooting direction using the sound source direction estimation result. According to the embodiment, according to the sound source direction of the tracking target, an image can be captured without losing sight of the tracking target, the tracking target in the captured image can be accurately searched, and the tracking operation can be continued. In this case, an operation for resetting the tracking target becomes unnecessary, and operability can be improved.

  According to the present embodiment, when tracking a tracking target using a captured image in a flying object, a sound source direction corresponding to the tracking target can be estimated, and a shooting direction by the imaging device can be adjusted based on the estimated sound source direction. Thereby, the imaging device can be directed to the tracking target, and the tracking result can be corrected. Therefore, an operation for resetting the tracking target is unnecessary, and operability can be improved.

  According to this embodiment, when tracking of a tracking target fails, sound information emitted from the tracking target can be acquired to estimate the sound source direction, and the shooting direction can be adjusted according to the sound source direction. As a result, even when tracking of the tracking target fails, the shooting direction can be adjusted so that the imaging device is directed toward the tracking target, and the tracking result can be corrected.

  According to this embodiment, when tracking a tracking target using a captured image, the sound information emitted from the tracking target is acquired to estimate the sound source direction, and the tracking target in the captured image is searched according to the sound source direction. Can be limited. As a result, the tracking result can be corrected, the amount of processing related to tracking can be reduced, and the robustness during the tracking operation can be improved.

  As mentioned above, although this indication was explained using an embodiment, the technical scope of the invention concerning this indication is not limited to the range given in the embodiment mentioned above. It will be apparent to those skilled in the art that various modifications and improvements can be made to the embodiment described above. It is also apparent from the scope of the claims that the embodiment added with such changes or improvements can be included in the technical scope of the present invention.

  The execution order of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. ”And the like, and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it does not mean that it is essential to carry out in this order. Absent.

DESCRIPTION OF SYMBOLS 10 Flight control system 50 Terminal 51 Processing part 52 Memory 53 Wireless communication part 54 Display part 55 Operation part 56 Interface part 57 Storage 58 Battery 100 Unmanned air vehicle (UAV)
102 UAV main body 110 Control unit 111 Tracking unit 112 Sound source direction estimation unit 150 Communication interface 160 Memory 170 Storage 180 Sound collection device 190 Battery 210 Rotor blade mechanism 220 Imaging device 230 Gimbal 240 GPS receiver 250 Inertial measurement device 260 Magnetic compass 270 Barometric altimeter 280 Ultrasonic sensor

Claims (20)

An imaging device for imaging a predetermined tracking target;
A sound collection device for collecting ambient sounds including the tracked object;
A control unit that executes processing related to tracking of the tracking target,
The controller is
Track the tracking target using a captured image captured by the imaging device,
Estimating the sound source direction corresponding to the tracking target from the sound information acquired by the sound collection device,
Adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction;
Flying body. The controller is
In tracking the tracking target,
Calculating a feature value of an image related to the tracking target in the captured image, and searching for the tracking target by comparing the feature value of the image;
The flying object according to claim 1. The controller is
When tracking of the tracking target fails, the sound information emitted by the tracking target is acquired to estimate the sound source direction, and the shooting direction is adjusted according to the sound source direction.
The flying object according to claim 1 or 2. The controller is
Calculating a characteristic value of a sound related to the tracking target in the sound information;
Estimating the sound source direction of the tracking target according to the feature value of the sound, and tracking the tracking target according to the sound source direction;
The flying object according to claim 1 or 2. The controller is
In tracking the tracking target,
Limiting the area to search for the tracking target in the captured image according to the estimated sound source direction;
The flying object according to any one of claims 1 to 4. The controller is
In the estimation of the sound source direction,
Extracting the tracking target sound component from the sound information acquired by the sound collection device, and performing a sound source direction estimation process for the tracking target sound component,
The flying object according to any one of claims 1 to 5. The controller is
In adjusting the shooting direction,
Controlling the movement of at least one of the rotary wing mechanism of the flying object and the support mechanism for supporting the imaging device;
The flying object according to any one of claims 1 to 6. A flight control system comprising a flying object and a terminal that communicates with the flying object,
The aircraft is
An imaging device for imaging a predetermined tracking target;
A sound collection device for collecting ambient sounds including the tracked object;
A communication unit for transmitting the tracking target candidate information to the terminal;
A sound source that performs processing related to tracking of the tracking target, tracks the tracking target using a captured image captured by the imaging device, and corresponds to the tracking target based on sound information acquired by the sound collecting device; A control unit that estimates a direction and adjusts a shooting direction by the imaging device based on an estimation result of the sound source direction;
The terminal
A communication unit for receiving the candidate information to be tracked;
A display unit for displaying the tracking target candidates;
An operation unit for performing input related to the setting of the tracking target,
Flight control system. The display part of the terminal
When tracking of the tracking target fails, a tracking failure display is performed.
The flight control system according to claim 8. A flight control method for controlling an aircraft that tracks a predetermined tracking object,
Imaging the tracking object with an imaging device;
Collecting ambient sounds including the tracking target by a sound collecting device;
Tracking the tracking object using the imaged captured image;
Estimating a sound source direction corresponding to the tracking target based on the collected sound information;
Adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction,
Flight control method. The step of tracking the tracking object includes:
Calculating a feature value of an image related to the tracking target in the captured image, and searching for the tracking target by comparing the feature value of the image;
The flight control method according to claim 10. Estimating the sound source direction comprises:
If tracking of the tracking target fails, the sound information emitted by the tracking target is acquired to estimate the sound source direction,
The step of adjusting the shooting direction includes:
Adjusting the shooting direction according to the estimated sound source direction;
The flight control method according to claim 10 or 11. Calculating a feature value of a sound related to the tracking target in the sound information,
The step of tracking the tracking object includes:
Estimating the sound source direction of the tracking target according to the feature value of the sound, and tracking the tracking target according to the sound source direction;
The flight control method according to claim 10 or 11. The step of tracking the tracking object includes:
Limiting the area to search for the tracking target in the captured image according to the estimated sound source direction;
The flight control method according to any one of claims 10 to 13. The step of tracking the tracking object includes:
Extracting the tracking target sound component from the sound information acquired by the sound collection device, and performing a sound source direction estimation process for the tracking target sound component,
The flight control method according to any one of claims 10 to 14. The step of adjusting the shooting direction includes:
Controlling the movement of at least one of the rotary wing mechanism of the flying object and the support mechanism for supporting the imaging device;
The flight control method according to any one of claims 10 to 15. Transmitting the tracking candidate information to a terminal that communicates with the aircraft;
Receiving the candidate information to be tracked by the terminal;
Displaying the tracking target candidates on a display unit;
Performing input relating to the setting of the tracking target,
The flight control method according to claim 10. When the tracking of the tracking target fails, the display unit displays a tracking failure display,
The flight control method according to claim 17. To a computer that controls a vehicle that tracks a predetermined tracking target,
Imaging the tracking object with an imaging device;
Collecting ambient sounds including the tracking target by a sound collecting device;
Tracking the tracking object using the imaged captured image;
Estimating a sound source direction corresponding to the tracking target based on the collected sound information;
Adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction,
program. To a computer that controls a vehicle that tracks a predetermined tracking target,
Imaging the tracking object with an imaging device;
Collecting ambient sounds including the tracking target by a sound collecting device;
Tracking the tracking object using the imaged captured image;
Estimating a sound source direction corresponding to the tracking target based on the collected sound information;
Adjusting the shooting direction by the imaging device based on the estimation result of the sound source direction, and a computer-readable recording program for executing the program,
recoding media.

Images