JP6621942B2 - Remote control system, remote control method, and program - Google Patents

Description

  The present invention relates to a remote control system, a remote control method, and a program.

  Various mechanisms have been developed to support human work. For example, as a mechanism for supporting farm work, Patent Document 1 discloses a plurality of agricultural machines 2 that are movable and perform farm work, a data collection device 3 that collects farm work data in each farm machine 2 and transmits the data to the outside, and data A system including a plurality of movable portable terminals 4 that receive farm work data transmitted from the collection device 3 and transmit it to the server 5 is disclosed. The farm work data accumulated in the server 5 is used by farm workers to create daily work reports, make work plans, graph farm work results in each field, and create management plans.

JP 2014-194601 A

  For example, in the technique described in Patent Document 1, it is left to the farm worker to determine what kind of farm work is performed. On the other hand, an object of the present invention is to make it possible to determine whether or not to perform various processes such as farm work without leaving it to a human.

Means to solve the problem

The present invention relates to an acquisition unit that acquires image data indicating an image captured by an imaging device that is a movable mobile device, position data indicating a position of the image, and a condition regarding an analysis result of the image data; Storage means for storing the processing performed by the device to be controlled in association with each other, determination means for analyzing the acquired image data and determining whether the analysis result matches the stored condition, Instructing means for instructing the device to be controlled to perform the processing associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition. , Hisage depending on the type of plants and animals included in the area where the processing is performed, the remote control system, characterized in that it comprises a movement control means for controlling the movement of the imaging device To.
In addition, the present invention performs an acquisition unit that acquires image data indicating an image captured by an imaging device, position data indicating the position of the image, a condition regarding an analysis result of the image data, and a control target device. Storage means for storing processing in association with each other, determination means for analyzing the acquired image data and determining whether an analysis result matches the stored condition, and the analysis result stored in the storage An instruction means for instructing the device to be controlled to perform the process associated with the condition in an area based on the acquired position data when the acquired condition data is met. The area based on the position data is an area that does not include the position indicated by the position data and has a predetermined positional relationship with the position indicated by the position data. Providing remote control system characterized and.

The present invention also relates to an acquisition step of acquiring image data indicating an image captured by an imaging device that is a movable mobile device, and position data indicating a position of the image, and an analysis result of the image data. A determination step of determining whether or not an analysis result of the acquired image data matches the stored condition in a storage unit that stores a condition and a process performed by the control target device in association with each other; When the analysis result matches the stored condition, an instruction step for instructing the control target device to perform the process associated with the condition in the area based on the acquired position data ; And a movement control step for controlling movement of the imaging device according to the type of animals and plants included in the area where the processing is performed. To provide the interval control method.
In addition, the present invention performs an acquisition step of acquiring image data indicating an image captured by an imaging device and position data indicating the position of the image, a condition regarding an analysis result of the image data, and a control target device. A storage step for storing the processing in association with each other, a determination step for analyzing the acquired image data and determining whether or not the analysis result matches the stored condition, and the analysis result is stored in the storage An instruction step for instructing the device to be controlled to perform the process associated with the condition in an area based on the acquired position data when the acquired condition data is met. The area based on the position data is an area that does not include the position indicated by the position data, and has a predetermined positional relationship with the position indicated by the position data. It provides a remote control method, which is a that area.

According to another aspect of the present invention, there is provided an acquisition step of acquiring, in one or more computers, image data indicating an image captured by an imaging device that is a movable mobile device, and position data indicating a position of the image, In a storage means for storing the condition relating to the analysis result of the image data and the process performed by the device to be controlled in association with each other, it is determined whether or not the acquired analysis result of the image data matches the stored condition A decision step to
An instruction step for instructing the device to be controlled to perform the process associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition; A program for executing a movement control step for controlling movement of the imaging device according to the type of flora and fauna included in the area where the processing is performed is provided.
In addition, the present invention provides an acquisition step of acquiring image data indicating an image captured by an imaging device and position data indicating the position of the image in one or more computers, and a condition regarding an analysis result of the image data, A storage step for storing the processing performed by the device to be controlled in association with each other, a determination step for analyzing the acquired image data and determining whether an analysis result matches the stored condition, If the analysis result matches the stored condition, the area does not include the position indicated by the acquired position data, and the area has a predetermined positional relationship with the position indicated by the position data. And providing an instruction step for instructing the device to be controlled to perform the process associated with the condition.

  According to the present invention, it is possible to determine whether or not to perform processing without leaving it to human judgment.

The figure which illustrates the outline | summary of the remote control system 1 which concerns on one Embodiment of this invention. 2 is a diagram illustrating a hardware configuration of a moving body imaging apparatus 20. FIG. 2 is a diagram illustrating a hardware configuration of a remote control device 10. FIG. 4 is a diagram illustrating information stored in an auxiliary storage device 104 of the remote control device 10. FIG. 4 is a diagram illustrating a condition processing table stored in the auxiliary storage device 104 of the remote control device 10. FIG. The figure which illustrates the function structure of the remote control system 1. 4 is a sequence chart illustrating the operation of the remote control system 1.

DESCRIPTION OF SYMBOLS 1 ... Remote control system, 10 ... Remote control apparatus, 11 ... Movement control part, 12 ... Acquisition part, 13 ... Memory | storage part, 14 ... Judgment part, 15 ... Instruction | indication part, 16 ... History storage part, 20 ... Moving body imaging device , 21 ... moving part, 22 ... imaging part, 23 ... transmitting part, 24 ... position detecting part, 30 ... controlled device, 31 ... processing part, 90 ... network, 101 ... CPU, 102 ... RAM, 103 ... ROM, 104 DESCRIPTION OF SYMBOLS ... Auxiliary storage device 105 ... Communication IF, 201 ... CPU, 202 ... RAM, 203 ... ROM, 204 ... Auxiliary storage device, 205 ... Communication IF, 206 ... Camera, 207 ... Sensor, 2081 ... Rotation mechanism.

1. Configuration FIG. 1 is a diagram illustrating an overview of a remote control system 1 according to an embodiment of the present invention. The remote control system 1 includes a remote control device 10, a moving body imaging device 20, a control target device 30, and a network 90 that connects these devices 10 to 30 so that they can communicate with each other. The network 90 is, for example, a local area network (LAN), a wide area network (WAN), or a combination thereof, and may include a wired section or a wireless section. The control target device 30 is an agricultural machine for performing various processes such as watering, additional fertilization, agricultural chemical application, and harvesting on plants cultivated in a field, for example. The moving body imaging device 30 is a device that is movable and has an imaging function. In the present embodiment, the moving body imaging device 30 is a flightable rotorcraft called a drone or a multicopter. The moving body imaging device 30 flies over the field in accordance with the instruction from the remote control device 10 and captures an image of the plant cultivated in the field. The remote control device 10 analyzes the image picked up by the moving body image pickup device 30, determines what processing should be performed on the plant according to the analysis result, and the control target device performs the determined processing. 30. The control target device 30 performs the instructed process automatically or according to the operation of the farmer.

  FIG. 2 is a diagram illustrating a hardware configuration of the moving body imaging apparatus 20. The moving body imaging device 20 includes a CPU 201 (Central Processing Unit), a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an auxiliary storage device 204, a communication IF 205, a camera 206, a sensor 207, and a rotation mechanism 208. Is a processor that performs various operations. The RAM 202 is a volatile memory that functions as a work area when the CPU 201 executes a program. The ROM 203 is a non-volatile memory that stores programs and data used for starting the mobile imaging device 20, for example. The auxiliary storage device 204 is a nonvolatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores programs and data used in the mobile imaging device 20. When the CPU 201 executes this program, the moving body imaging device 20 functions as a computer device, and a function shown in FIG. 6 to be described later is realized. The communication IF 205 is an interface for performing communication via the network 90 in accordance with a predetermined communication standard. This communication standard may be a wireless communication standard or a wired communication standard. For example, the camera 206 images a space below the moving body imaging device 20 and generates image data indicating the captured image. The positioning device 207 is a GPS (Global Positioning System) unit, for example, and detects the position of the moving body imaging device 20. The rotation mechanism 208 is a means for generating buoyancy in the moving body imaging apparatus 20, and includes a propeller, a shaft, a motor, and other driving mechanisms.

  FIG. 3 is a diagram illustrating a hardware configuration of the remote control device 10. The remote control device 10 is a computer device having a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, and a communication IF 105. The CPU 101 is a processor that performs various calculations. The RAM 102 is a volatile memory that functions as a work area when the CPU 101 executes a program. The ROM 103 is a non-volatile memory that stores programs and data used for starting the remote control device 10, for example. The auxiliary storage device 104 is a nonvolatile storage device such as an HDD or an SSD, and stores programs and data used in the remote control device 10. When the CPU 101 executes this program, the functions shown in FIG. 6 to be described later are realized. The communication IF 105 is an interface for performing communication via the network 90 in accordance with a predetermined communication standard.

  FIG. 4 is a diagram illustrating information stored in the auxiliary storage device 104 of the remote control device 10. The auxiliary storage device 104 includes the date and time when an image (moving image in this case) was captured by the moving body imaging device 10, position data indicating the position where the image was captured, image data indicating the captured image, and the captured image. As a result of the analysis, processing target position data indicating the position of the area where processing is required as a result of the analysis, an analysis result obtained by analyzing the captured image by the image recognition technology, and a content of the processing required Correspondingly stored.

  FIG. 5 is a diagram illustrating a condition processing table stored in the auxiliary storage device 104 of the remote control device 10. In the condition processing table, a condition of an analysis result obtained by analyzing a captured image that is assumed in advance and a process that is necessary at a position that matches the condition are associated with each other. For example, if the shape, color, size, etc. of the object (plant image object) included in the image matches the condition of the pest occurrence symptom, this condition processing table stipulates that processing such as pesticide spraying is executed. Has been.

  FIG. 6 is a diagram illustrating a functional configuration of the remote control system 1. The remote control system 1 includes a movement unit 21, an imaging unit 22, a position detection unit 23, a transmission unit 24, a movement control unit 11, an acquisition unit 12, a storage unit 13, a determination unit 14, an instruction unit 15, a history storage unit 16, and A processing unit 31 is included. In this example, the moving unit 21, the imaging unit 22, the position detection unit 23, and the transmission unit 24 are mounted on the moving body imaging device 20, and the movement control unit 11, the acquisition unit 12, the storage unit 13, the determination unit 14, The instruction unit 15 and the history storage unit 16 are mounted on the remote control device 10, and the processing unit 31 is mounted on the control target device 30.

  The moving unit 21 is realized by the rotation mechanism 208 of the moving body imaging device 20, the imaging unit 22 is realized by the camera 206 of the moving body imaging device 20, and the position detection unit 23 is realized by the positioning device 207 of the moving body imaging device 20. The transmission unit 24 is realized by the communication IF 205 of the moving body imaging apparatus 20. The movement control unit 11 is realized by the CPU 101 and the communication IF 105 of the remote control device 10, the acquisition unit 12 is realized by the communication IF 105 of the remote control device 10, and the storage unit 13 is realized by the auxiliary storage device 104 of the remote control device 10. The determination unit 14 is realized by the CPU 101 and the auxiliary storage device 104 of the remote control device 10, the instruction unit 15 is realized by the communication IF 105 of the remote control device 10, and the history storage unit 16 is realized by the auxiliary storage device 104 of the remote control device 10. Is done.

  The movement control unit 11 gives a movement instruction including a movement route, a movement speed, an altitude when moving, a timing of starting and ending imaging, and the like to the moving unit 21 in order to control movement of the moving body imaging device 20. The moving unit 21 moves the moving body imaging device 20 in accordance with a movement instruction from the movement control unit 11. The imaging unit 25 captures a moving image over a period from the start to the end of imaging as the mobile body imaging device 20 moves. The position detection unit 23 detects the position of the moving body imaging device 20, that is, the position of the image to be captured. The transmission unit 24 transmits image data indicating the image captured by the imaging unit 22 and position data indicating the position of the image to the remote control device 10.

  The acquisition unit 12 acquires the image data and position data transmitted from the transmission unit 24. The storage unit 13 stores the above-described condition processing table, that is, information in which the condition relating to the analysis result of the image data is associated with the process performed by the control target device 30. The determination unit 14 analyzes the image data acquired by the acquisition unit 12 and determines whether or not the analysis result matches the conditions stored in the storage unit 13. When the analysis result by the determination unit 14 matches the condition stored in the storage unit 13, the instruction unit 15 specifies the area to be processed based on the position data acquired by the acquisition unit 12, and The control target device 30 is instructed to perform processing associated with the above condition in the area. The history storage unit 16 stores the process instructed by the instruction unit 15 and the position of the area where the process is performed as a process history. The processing unit 31 performs processing in accordance with an instruction from the instruction unit 15.

2. Operation FIG. 7 is a sequence chart illustrating the operation of the remote control system 1. In FIG. 7, the movement control unit 11 of the remote control device 10 moves to the moving unit 21 via the network 90 including the movement route, the moving speed, the altitude when moving, the timing for starting and ending imaging, and the like. An instruction is given (step S11). This movement instruction is created in advance by an administrator of the remote control system 1 and is input to the remote control device 10.

  The moving unit 21 of the moving body imaging apparatus 20 drives the rotating mechanism 208 to move the moving body imaging apparatus 20 in accordance with a movement instruction from the movement control unit 11. The imaging unit 25 images a moving image over a period from the start to the end of imaging included in the movement instruction as the moving body imaging device 20 moves. The position detection unit 23 periodically detects the position of the moving body imaging device 20, that is, the position of the image to be captured (for example, every 10 seconds). The transmission unit 24 transmits image data indicating the image captured by the imaging unit 22 and position data indicating the position detected by the position detection unit 23 to the remote control device 10 via the network 90. This transmission process may be performed after the mobile imaging device 20 has landed at the base facility provided at the end point of the flight, or may be performed while the mobile imaging device 20 is flying.

  The acquisition unit 12 of the remote control device 10 acquires image data and position data transmitted from the transmission unit 24. The determination unit 14 analyzes the image data acquired by the acquisition unit 12. Specifically, the determination unit 14 analyzes the shape, color, size, and the like of an object included in each image indicated by the image data using an image recognition technique, and the analysis result is stored in the storage unit 13 ( It is determined whether or not any of the pest occurrence symptom, water deficiency symptom, fertilizer deficiency symptom, growth completion symptom is met.

  If there is an image that results in an analysis that matches the conditions stored in the storage unit 13, the instruction unit 15 of the remote control device 10 performs processing based on the position data corresponding to the image. Identify the location of the area of interest. More specifically, the instruction unit 15 has an analysis result that matches the conditions stored in the storage unit 13 (infestation symptom, water deficiency symptom, fertilizer deficiency symptom, growth completion symptom ...). An area with a predetermined radius centered on the position data corresponding to is processed. And the instruction | indication part 15 reads the process (any of agrochemical spraying, watering, additional fertilization, harvest ...) matched with the said conditions in a condition process table, and the process instruction | indication including process target position data and its process Is transmitted to the control target device 30 via the network 90. The history storage unit 16 stores the processing instructed by the instruction unit 15 and the processing target position data of the area where the processing is performed as a processing history.

  The processing unit 31 of the control target device 30 receives a processing instruction from the instruction unit 15. Then, in the control target device 30, the process included in the processing instruction is performed in the area indicated by the processing target position data included in the processing instruction. Specifically, when the control target device 30 can perform all the processing automatically, the control target device 30 itself moves to the area indicated by the processing target position data and executes the processing. When at least a part of processing is assisted or executed by a human, for example, the human refers to the processing instruction in the control target device 30 and operates the control target device 30 to move to the area indicated by the processing target position data. Then, the process is executed while operating the control target device 30 in the area.

  According to the present embodiment, it is possible to determine whether or not to perform various processes such as farm work without relying on human judgment.

3. Modifications The present invention is not limited to the above-described embodiments, and various modifications can be made. Hereinafter, some modifications will be described. Two or more of the following modifications may be used in combination.

3-1. Modification 1
The present invention can be applied to a case where treatment is performed on animals such as seaweed, fish and shellfish cultivated in the ocean and rivers, or animals such as livestock kept on the ranch, in addition to plants cultivated in the fields. It is.

3-2. Modification 1
The movement control unit 11 may control the movement of the moving body imaging device 20 according to the type of animals and plants included in the area to be processed. For example, depending on the type of animals and plants, when the moving body imaging device 20 does not fly at a low altitude and the image cannot be analyzed, the movement control unit 11 moves the moving body imaging device 20 so as to fly at a relatively low altitude. To control. In addition, for example, depending on the type of animals and plants, when the moving path per unit area of the moving body imaging device 20 cannot be analyzed unless the moving path is dense, the movement control unit 11 per unit area of the moving body imaging device 20 The movement of the moving body imaging device 20 is controlled so as to fly along a movement path with a dense movement path. Further, for example, depending on the type of animals and plants, if the image cannot be analyzed unless the moving speed of the moving body imaging apparatus 20 is slowed, the movement control unit 11 moves the moving body imaging apparatus 20 so as to fly at a relatively slow speed. To control.
In the embodiment, the determination unit 14 analyzes the image and selects a necessary process. Instead, for example, the process is first designated by a system administrator or the like, and the determination unit 14 selects the image. Analysis may be performed to determine whether or not the specified process should be performed. In this case, the movement control unit 11 may control the movement of the moving body imaging apparatus 20 according to the designated process. For example, when the process of watering is specified, the movement control unit 11 is not suitable because the movement path per unit area is sparse and it is sufficient for image analysis even if the movement path is relatively high and fast. The moving body imaging device 20 is controlled to move. On the other hand, for example, when the process of spraying agricultural chemicals is designated, the movement control unit 11 is not able to perform sufficient image analysis unless the movement route per unit area is dense and the movement is slow at a relatively low altitude. The moving body imaging device 20 is controlled to move.

3-3. Modification 3
The control target device 30 and the moving body imaging device 20 may be the same. That is, the moving body imaging device 20 may be used as the control target device 30. In this case, the transmission unit 24 of the mobile body imaging device 20 uses the image data indicating the image captured by the imaging unit 22 and the position data indicating the position detected by the position detection unit 23 as the mobile body imaging device 20. It may be transmitted during the flight. In response to this, the determination unit 14 immediately analyzes the image data acquired by the acquisition unit 12 and determines whether or not the analysis result matches the condition stored in the storage unit 13. Then, when the analysis result by the determination unit 14 matches the condition stored in the storage unit 13, the instruction unit 15 instructs the control target device 30 to perform processing associated with the condition. These processes are performed in real time during the flight of the mobile imaging device 20.

3-4. Modification 4
The area to be processed may be an area that does not include the position where the image is captured and that has a predetermined positional relationship with the position where the image is captured. For example, when seaweed is cultivated in the ocean, a sign that a red tide may occur may appear on the open ocean side of the seaweed culture area. Therefore, the moving body imaging device 20 images the open sea side from the seaweed culture area, and when the occurrence of red tide or the like is predicted by analysis of the captured image, the moving body imaging apparatus 20 Move and spray appropriate drugs. In this case, the area to be processed corresponds to an area that does not include the position where the image is captured and has a predetermined positional relationship with the position where the image is captured.

3-5. Modification 5
The instruction unit 15 determines processing to be instructed to the control target device 30 based on the processing history stored in the history storage unit 16. For example, when pesticide spraying is again performed on an area where a processing history of pesticide spraying in the past is stored, the instruction unit 15 instructs the processing to reduce the amount of the pesticide from the previous time. Conceivable.

3-6. Modification 6
The instructing unit 15 instructs the control target device 30 based on information (for example, temperature, humidity, precipitation) related to the natural environment at the position of the image before the time when the image is captured by the mobile imaging device 20. May be determined. For example, when the amount of precipitation is extremely small before the time when the image is picked up by the mobile body imaging device 20, the instruction unit 15 instructs the processing to increase the amount of water spraying in the area to be processed. An example is conceivable.

3-7. Modification 7
The imaging device in the present invention is not limited to the moving body imaging device 20 such as the rotorcraft exemplified in the embodiment, and may be a device mounted on a vehicle moving on the ground, or a smartphone or a digital camera. Thus, the apparatus carried by the user may be sufficient. In addition, the imaging device according to the present invention is not necessarily a movable device, and may be a device that is fixed in a state where the camera is directed in a known direction at a known position, for example.

3-8. Other Modifications Part of the functional configuration illustrated in FIG. 6 may be omitted. For example, the storage unit 13 may be provided by an external computer device different from the remote control system 1. Further, the sharing of functions in the remote control device 10 and the moving body imaging device 20 is not limited to that illustrated in FIG. In the embodiment, some of the functions implemented in the remote control device 10 may be implemented in the mobile device 20. For example, some or all of the functions of the determination unit 14 may be implemented in the mobile imaging device 20. In addition, a computer group consisting of a plurality of devices may function as the remote control device 10 in the remote control system 1.

  The program executed by the CPU 101 and the CPU 201 may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet. Further, these programs may not execute all the steps described in the embodiment.

Claims (6)

Acquisition means for acquiring image data indicating an image captured by an imaging device that is a movable mobile device, and position data indicating a position of the image;
Storage means for storing the conditions relating to the analysis result of the image data in association with the processing performed by the control target device;
A determination means for analyzing the acquired image data and determining whether an analysis result matches the stored condition;
Instructing means for instructing the device to be controlled to perform the processing associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition. ,
A remote control system comprising: a movement control unit that controls movement of the imaging device according to the type of animals and plants included in the area where the processing is performed . Acquisition means for acquiring image data indicating an image captured by the imaging device and position data indicating the position of the image;
Storage means for storing the conditions relating to the analysis result of the image data in association with the processing performed by the control target device;
A determination means for analyzing the acquired image data and determining whether an analysis result matches the stored condition;
Instructing means for instructing the device to be controlled to perform the processing associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition. With
The area based on the acquired position data is an area that does not include the position indicated by the position data and has a predetermined positional relationship with the position indicated by the position data.
A remote control system characterized by that. An acquisition step of acquiring image data indicating an image captured by an imaging device that is a movable mobile device, and position data indicating a position of the image;
Whether or not the acquired analysis result of the image data matches the stored condition in a storage unit that stores the condition relating to the analysis result of the image data and the process performed by the control target device in association with each other. A judgment step to judge;
An instruction step for instructing the device to be controlled to perform the process associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition;
A movement control step for controlling movement of the imaging device according to the type of animals and plants included in the area where the processing is performed;
A remote control method comprising: An acquisition step of acquiring image data indicating an image captured by the imaging device and position data indicating a position of the image;
A storage step of storing a condition relating to the analysis result of the image data and a process performed by the control target device in association with each other;
A determination step of analyzing the acquired image data and determining whether or not an analysis result matches the stored condition;
An instruction step for instructing the device to be controlled to perform the process associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition; With
The area based on the acquired position data is an area that does not include the position indicated by the position data and has a predetermined positional relationship with the position indicated by the position data.
A remote control method characterized by the above. On one or more computers,
An acquisition step of acquiring image data indicating an image captured by an imaging device that is a movable mobile device, and position data indicating a position of the image;
Whether or not the acquired analysis result of the image data matches the stored condition in a storage unit that stores the condition relating to the analysis result of the image data and the process performed by the control target device in association with each other. A judgment step to judge;
An instruction step for instructing the device to be controlled to perform the process associated with the condition in the area based on the acquired position data when the analysis result matches the stored condition;
A movement control step for controlling movement of the imaging device according to the type of animals and plants included in the area where the processing is performed;
A program for running On one or more computers,
An acquisition step of acquiring image data indicating an image captured by the imaging device and position data indicating a position of the image;
A storage step of storing a condition relating to the analysis result of the image data and a process performed by the control target device in association with each other;
A determination step of analyzing the acquired image data and determining whether or not an analysis result matches the stored condition;
If the analysis result matches the stored condition, the area does not include the position indicated by the acquired position data, and the area has a predetermined positional relationship with the position indicated by the position data. An instruction step for instructing the device to be controlled to perform the processing associated with the condition;
A program for running

Images