JP7437930B2 - Mobile objects and imaging systems - Google Patents

Description

The present invention relates to a mobile object and an imaging system.

BACKGROUND ART A plurality of cameras are used together in order to image and monitor a certain amount of space. For example, by installing a reference imaging device that can take wide-angle images and mounting a camera on a moving object, such as a drone, an operator (observer) who visually observes the image of the reference imaging device should take an image of an abnormal state. When an abnormal condition is detected, the mobile object is directed to the location where the abnormal condition is occurring and an enlarged image is taken.

Once the operator specifies the imaging target in the image of the reference imaging device, if the configuration is such that the drone is automatically flown and the on-board camera images the imaging target, the position of the imaging target can be grasped. There is no need for any sensors. In order to do this, even if the image information of the specified imaging target and the image information of the camera (drone camera) mounted on a moving object (hereinafter, the case of a drone will be explained as an example) are imaged at different positions. need to be associated.

In Non-Patent Document 1, as a method for associating image information of objects captured in images taken at different imaging positions, two images whose viewpoints have changed due to movement of the camera, etc., are characterized by comparing local image features. Performing point matching. Then, when the feature points extracted from the other image that match the feature points extracted from the object in one image are concentrated in local areas with several different sizes, the two images are combined. It is determined that the same object is shown in the image.

Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., Cheng, M.M.: GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence.In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE(2017) 4181-4190

If the target object is located far from the reference imaging device in the monitoring area, there will be a large difference in the size of the target object in the image of the drone camera and the image of the reference imaging device when the drone has just launched. However, when the drone approaches the target object, the size of the target object in the image taken by the drone camera and the image taken by the reference imaging device differs greatly. The target object appearing in the image of the reference imaging device appears small, partly due to the fact that the reference imaging device has a wide angle. In images taken by a drone camera, the object appears large because it is close to the object.

To determine whether or not different images are the same object, for example, as in Non-Patent Document 1, feature points are extracted from the object region, and image features are calculated from the surrounding small regions to determine similarity. Can be done. However, if there is too much difference in size in the image, the feature values obtained from the surrounding feature points of the same part of the same object will be different, making it difficult to treat the same target object as the same and causing the object to be lost. There was a problem that different objects could be determined to be the same object (false alarm).

Therefore, in the present invention, even if the size of the target object on the image changes as the moving body moves, it is possible to accurately determine whether the target object is included in the image captured by the imaging means of the moving body. The purpose of the present invention is to provide a mobile object and an imaging system that can perform the following tasks.

In order to achieve the above object, a moving object according to the present invention is a moving object that moves relative to a target object, and a reference image capturing device captures a reference image of a monitoring range in which the target object may exist. a communication means for receiving a region designated by an operator as a determination image in which the target object is captured in the reference image as a target object region, and sequentially outputting moving object images captured in the monitoring range. a moving object imaging means; a determining means for comparing the feature points of the determination image with the feature points of the moving object image to determine the presence or absence of a partial region corresponding to the target object region in the moving object image; a moving object storage means for storing the moving object image in which it has been determined that a partial region exists in association with the partial region; If the degree of difference exceeds a predetermined threshold, the latest moving object image stored in the moving object storage means is changed to the determination image, and the partial area associated with the moving object image is changed to the It is characterized by changing to the target object area.

According to the mobile object according to the present invention, the communication means receives, as a determination image, a reference image obtained by a reference imaging device capturing a monitoring range in which the target object may exist, and the operator selects the target object in the reference image. A region designated as an object is received as a target region. The moving object imaging means sequentially outputs moving object images captured in the monitoring range.

Then, the determining means compares the feature points of the determination image and the feature points of the moving body image to determine whether or not there is a partial area corresponding to the target object area in the moving body image. The determination means may be configured to select the latest moving object image stored in the moving object storage means when the degree of difference between the size of the object area and the size of the partial area exceeds a predetermined threshold. The image is changed to a determination image, and the partial area associated with the moving body image is changed to the target object area. Then, the moving body storage means stores the moving body image in which it is determined that the partial area exists in association with the partial area.

Further, the moving object further includes a focal length of the reference imaging device when capturing the reference image, a focal length of the moving object imaging means when capturing the moving object image, and a focal length of the moving object imaging means when capturing the reference image. The size of the target object area and the moving body memory are determined based on the distance from the reference imaging device to the target object and the distance from the moving body imaging means to the target object when capturing the moving body image. It is preferable to include a degree of difference estimating means for calculating the degree of difference as a ratio of the size of the partial area stored in the means.

The imaging system according to the present invention includes a reference imaging device that images a monitoring range in which a target object may exist, and a moving body image that moves relative to the target object and captures the monitoring range. An imaging system including a mobile body including a mobile body imaging means, and an information processing device communicatively connected to the mobile body and the reference imaging device, wherein the mobile body or the information processing device is connected to the reference imaging device. a communication means for receiving a reference image captured by a reference imaging device of the monitoring range as a determination image, and receiving a region specified by an operator as a target object region in the reference image as a target object region; determining means for comparing feature points of a determination image and feature points of the moving object image to determine whether or not a partial region corresponding to the object region exists in the moving object image; a moving object storage means for storing the moving object image in association with the partial area, and the determining means is configured such that the degree of difference between the size of the object area and the size of the partial area is a predetermined threshold. , the latest moving object image stored in the moving object storage means is changed to the determination image, and the partial area associated with the moving object image is changed to the target object area. It is characterized by

According to the imaging system according to the present invention, the communication means receives, as a determination image, a reference image captured by the reference imaging device of the monitoring range, and also specifies that the target object is captured in the reference image by the operator. The target area is received as the target area. The moving object imaging means sequentially outputs moving object images captured in the monitoring range.

Then, the determining means compares the feature points of the determination image and the feature points of the moving body image to determine whether or not there is a partial area corresponding to the target object area in the moving body image. The determination means may be configured to select the latest moving object image stored in the moving object storage means when the degree of difference between the size of the object area and the size of the partial area exceeds a predetermined threshold. The image is changed to a determination image, and the partial area associated with the moving body image is changed to the target object area. Then, the moving body storage means stores the moving body image in which it is determined that the partial area exists in association with the partial area.

As explained above, according to the moving object and the imaging system of the present invention, even if the size of the target object on the image changes as the moving object moves, the image captured by the imaging means of the moving object will not change. The effect is that it is possible to accurately determine whether or not the target object is captured.

1 is a schematic diagram showing the configuration of an imaging system according to an embodiment of the present invention. FIG. 1 is a schematic diagram showing the configuration of a center-side device according to an embodiment of the present invention. 1 is a schematic diagram showing the configuration of a drone according to an embodiment of the present invention. FIG. 6 is a diagram schematically illustrating the processing of a feature point selection means and a feature point search means. FIG. 6 is a diagram illustrating an example of a result of a projection process in which determined three-dimensional coordinates are projected onto a reference image. FIG. 6 is a diagram illustrating an example of a result of a process of matching feature points of a reference image and a moving object image and selecting feature points for obtaining three-dimensional coordinates. FIG. 6 is a diagram illustrating an example of a result of a process of matching feature points between moving object images and selecting feature points whose three-dimensional coordinates are to be determined. FIG. 3 is a diagram for explaining a method for determining whether a target object is included in a moving body image. 3 is a flowchart showing the operation of the main device according to the embodiment of the present invention. It is a flow chart showing operation of a drone concerning an embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings. In this embodiment, an example will be described in which the present invention is applied to an imaging system that monitors a monitoring area using a reference imaging device and a moving object imaging means mounted on a drone.

<System configuration>
Hereinafter, a configuration of an embodiment of the present invention will be described with reference to FIG. 1 showing a schematic configuration of an imaging system 1 to which the present invention is applied.

The imaging system 1 includes a center-side device 100, a drone 200, and a reference imaging device 300. The center device 100, the drone 200, and the reference imaging device 300 are connected via closed area wireless communication such as Wi-Fi or wide area wireless communication such as LTE. Note that the center-side device 100 and the reference imaging device 300 may communicate by wire.

FIG. 1 shows an overhead view of a monitoring area 10 where a target object may exist, and a wall 40 exists so as to surround a ground 50. The height of this wall 40 is, for example, 5 [m], and the reference imaging device 300 is installed on the corner of the wall 40.

It is assumed that the reference imaging device 300 has an angle of view (focal length) that includes the entire monitoring region 10 in its field of view, and the imaging direction is adjusted. A lighting device may be provided to brighten the imaging range at night.

The reference imaging device 300 is fixedly installed, images the entire monitoring area 10 as shown in FIG. 1, and transmits the obtained image (“reference image”) to the center-side device 100.

In this embodiment, it is assumed that a fixed focus lens having an angle of view of 90° is used.

Also, instead of a wide-angle fixed focus lens, a zoom lens may be used. However, in the case of a zoom lens, focal length information is required in order to determine the position of the target object 20 in the world coordinate representation, so a mechanism that can obtain the focal length every time zooming is required.

In this embodiment, the monitoring area 10 is assumed to have a size of several hundred meters square, like a parking lot of a large commercial facility. Therefore, a plurality of reference imaging devices 300 may be used depending on the angle of view.

A target object 20 exists in the monitoring area 10. In the present embodiment, for example, an object that the operator of the imaging system 1 (hereinafter simply referred to as the operator) has determined to be suspicious and desires to visually confirm, such as an abandoned bag, is the target object 20. Illustrated. Alternatively, examples of the target object 20 include a vehicle that has been left parked for a long period of time, a person who is unnaturally staying in the parking lot, and a person who has fallen down due to poor physical condition.

A standby base 30 for a drone 200, which is an example of a moving object in the imaging system 1, is also installed in the monitoring area 10. Until the operator inputs a command to start and take an image, the vehicle is stored inside and the battery is charged.

In this embodiment, a case will be explained in which the target object 20 is far from the reference imaging device 300 and the standby base 30.

As shown in FIG. 1, in this imaging system 1, a drone 200 is launched and flown from a standby base 30 toward the position of the target object 20 so that the operator can easily visually confirm the target object 20. The object 20 is approached and imaged. Thereafter, the drone 200 waits for input of a return command and returns to the standby base 30.

FIG. 2 shows a block diagram of a center-side device 100 installed on the center side of the present imaging system 1.

The center device 100 includes a display device 120, a designation input device 130, and a main device 140. Note that the main device 140 is an example of an information processing device.

The display device 120 is a monitor device that displays the reference image and the image transmitted from the drone 200 (“moving object image”) and configures the function of the UI when a designation is input by the designation input device 130. Further, as the display device 120, a size and a resolution may be appropriately selected in consideration of the resolution of the reference imaging device 300, etc., the width of the monitoring area 10 in which the operation is expected, and the like.

The designation input device 130 allows the operator to specify, in the reference image displayed on the display device 120, the target area in which the target object 20 is reflected, for example, simply by the upper left point and the lower right point. This is a means of specifying input using a rectangle. It is typically a mouse, and may also include a touch pen and keyboard. You may specify that the outline of the target object 20 be traced as it is. The specified input target object area is transmitted to the drone 200 through the main device communication unit 110, which will be described later, and is stored in the target object information 2223. Further, the target object area is an example of a partial area.

The image included in the designated object area is used as an "image of the object area" in each process described below. Further, the position (coordinate information) of the target object area in the reference image is used to find the actual position of the target object 20 in the monitoring area 10 expressed in world coordinates.

The main device 140 is realized by a computer equipped with a CPU, ROM, RAM, etc., and has an input/output I/F with each device including the main device communication section 110, a main device processing section 150, and a main device storage section 160. have

The main device communication unit 110 is a communication interface with each of the drone 200 and the reference imaging device 300. It consists of a closed area wireless communication device such as Wi-Fi or a wide area wireless communication device such as LTE.

The main device storage unit 160 stores programs and various parameters for each means realized by software, as well as standard imaging parameters 161 and a monitoring area map 162.

The reference imaging parameter 161 is an imaging parameter of the reference imaging device 300, and in addition to its focal length and resolution, values representing the imaging direction of the reference imaging device 300 in the monitoring area 10 in world coordinates are stored.

If focal length information can be obtained by using a zoom lens in the reference imaging device 300, the focal length is updated and stored each time zooming is performed.

The monitoring area map 162 is a collection of information on elements that determine the monitoring area 10, assuming that its structure is known. In addition to walls 40, a world coordinate system with the corner of the ground as the origin is defined for buildings (not shown) and materials stored outdoors, and geometric information such as the size and position of walls and buildings is stored in BIM (Building Information This is information expressed using a modeling method such as Model). Note that the world coordinate system is commonly used in the imaging system 1, in addition to the monitoring area map 162, the reference imaging parameters 161, and the position and orientation acquisition unit 223, which will be described later.

When the operation of this imaging system starts, the standard imaging parameters and the monitoring area map are sent to the drone 200 as an initial setting through the main device communication unit 110 and used for processing. Even when the orientation of the imaging device 300 changes or a change occurs in the building in the monitoring area 10, the reference imaging parameters and the monitoring area map are sent to the drone 200 again.

The object position calculation means 151 refers to the reference imaging parameters 161 and the monitoring area map 162, and calculates the actual target object 20 in the monitoring area with respect to the object area specified by the system operator by operating the specification input device 130. This is a means of determining the position of in world coordinate representation. A well-known perspective projection method may also be used. Lens distortion information may also be considered.

When the system operator visually inspects the reference image and specifies the target object area as a rectangle, the position of the target object 20 is determined by the vertices of the four corners, especially the lower part of the reference image as the side that is in contact with the ground. Select the corner vertices of and represent them in world coordinates. Alternatively, the center of gravity of the object region may be selected.

Note that the object position determined by the object position calculation means 151 cannot necessarily be determined with sufficient accuracy due to errors that may be included in the reference imaging parameters. As mentioned above, this embodiment assumes a large parking lot with one side of the lot measuring several hundreds of meters, so for example, if the reference image pickup device 300 is installed and the direction is shifted by one degree, the number of This is because a distance of several meters can make a difference of several meters.

Therefore, it should be noted that even if the drone 200 is directed using the position of the target object 20 determined by the object position calculation means 151 as it is, the target object 20 is not necessarily located there.

The object feature point extraction means 152 is a means for extracting object feature points from the object region using a well-known method and calculating image feature amounts from the surroundings thereof. Local feature points are extracted at edges, corners, etc. in an image.

A method called ORB (Oriented FAST and Rotated BRIEF) may be used to extract local feature points and calculate image feature amounts. In other words, a corner is detected as a local feature using the brightness difference in the image, and the magnitude of the brightness of multiple pixel pairs in a rectangular area of a specific size around the corner is expressed as a bit string as an image feature. .

Further, the object feature point extracting means 152 extracts not only several points at most but several thousand object feature points, such as only clear corners, from the object region, for example.

Note that the processing of the object feature point extracting means 152 may be directly performed by the moving object feature point extracting means 2211 of the drone 200, since the reference image is transmitted to the drone 200 as described later. Further, the image of the target object region, the target object feature points, and the position of the target object region in the reference image are transmitted to the drone 200 through the main device communication unit 110 and stored in the mobile object storage unit 222 described later.

FIG. 3 shows a block diagram of the drone 200. The drone 200 is, for example, a quadrotor-type small unmanned helicopter. Note that the present invention is not limited to quadrotor type small unmanned helicopters, but can be similarly applied to single rotor type small unmanned helicopters.

The drone 200 includes, in addition to a mobile body 220, a mobile body imaging means 210, for example, four rotors 231 that generate buoyancy, and a rotor drive unit 230 that controls the rotors 231 to achieve a desired flight.

The rotor drive unit 230 includes four motors whose rotating shafts are connected to corresponding rotors 231, an ESC (electronic speed control), and the like. The rotational speed of each motor is independently controlled by the ESC, and each rotor 231 rotates independently to generate acceleration in an arbitrary direction in the drone 200.

The mobile object imaging means 210 is a camera composed of a CCD or CMOS image sensor and a lens, and after taking off from the standby base 30 in accordance with the launch command transmitted from the main device 140, the mobile object imaging means 210 captures the area around the drone 200, that is, monitors the area around the drone 200. The region 10 is imaged and a moving object image is output to the moving object main body 220.

In this embodiment, an image sensor with a resolution of 1600 x 1200 pixels is used, assuming that the drone 200 moves at an altitude of 10 meters above the ground and can distinguish the type of object on the ground of 1 meter square at a horizontal distance of 10 meters. , a reference focus lens having an angle of view of 60° is used.

A zoom camera may be used as long as zooming processing is possible and focal length information can be obtained. It is also preferable to include an illumination device that illuminates the imaging range.

As shown in FIG. 3, the mobile body 220 includes a mobile body storage section 222, a mobile object processing section 221, a position/orientation acquisition section 223, a mobile communication section 224, and a flight control section 225. Of these, the moving object processing section 221 corresponds to the determination means of the present invention.

The monitoring area map 2221 in the mobile object storage unit 222 is the same as the monitoring area map 162 stored in the main device 140. The settings are sent from the main device 140 and stored as initial settings. When the shape of the monitoring area (how materials are placed, etc.) is changed, the monitoring area map 2221 is also updated by being retransmitted from the main device 140 accordingly.

The reason why the monitoring area map is also stored on the drone 200 side is that it is used to obtain information such as the position and attitude of the drone 200 itself in the monitoring area 10.

The reference captured image 2222 is an area for storing a reference image transmitted from the main device 140 as a determination image. As will be described later, the reference captured image is mainly used shortly after the drone 200 takes off to determine whether the target object 20 is included in the moving body image.

Further, in association with the reference captured image 2222, the imaging parameters of the reference imaging device 300 when the reference image was captured are also stored in a buffer area (not shown).

The object information 2223 stores the object region in the reference image, object feature points extracted from the object region, and the position of the object region in the reference image, which are transmitted from the main device 140.

The target object area is used to determine the block size of block division performed in evaluating the concentration of projection points by the evaluation determining means 2216, which will be described later.

The object feature points extracted from the object region are used in feature point selection means 2212, which will be described later, to select feature points in the moving object image.

The position of the object area in the reference image is used to determine the boundary between blocks when dividing into blocks.

The moving object imaging information 2224 includes moving object feature points extracted by the moving object feature point extraction means 2211 from the moving object image, image feature amounts calculated from images of small areas surrounding each of the moving object feature points, and position and orientation acquisition described later. The position and orientation information (posture information) of the drone 200 expressed in world coordinates in the monitoring area 10 obtained by the unit 223 are stored.

As will be described later, the moving object feature points and image feature amounts stored in the moving object imaging information 2224 are obtained by capturing images of the moving object 20 by the moving object imaging means 210 at different times, that is, from different viewpoints with respect to the target object 20. These are moving object feature points and image feature amounts extracted from past moving object images for determining the correspondence between the feature points extracted by the feature point extracting means 2211 and finding three-dimensional coordinates.

This method calculates the three-dimensional coordinates expressed in world coordinates in the monitoring area 10 for moving object feature points, but uses so-called stereo image technology (three-dimensional measurement), and for this purpose, it uses the moving object image at the current time and This is because one or more moving object images are required.

In addition, if the drone 200 itself hovers and hardly changes its position, 3D measurement becomes difficult, so it is difficult to perform 3D measurement, so it is difficult to perform 3D measurement. Mobile feature points and image feature amounts are preferred.

This information will be updated when a moving object image with a certain parallax is newly acquired.

The determined moving object image 2225 is a moving object image that has been determined in the past to include the target object 20 by the evaluation determining means 2216, which will be described later.

This is because after the drone 200 takes off, as it approaches the target object 20, the size of the target object 20 shown in the moving body image is too different from the size of the target object 20 shown in the reference image, and the feature points described later are Since it would be inappropriate for the selection unit 2212 to use the reference image as the determination image, the feature point selection unit 2212 uses the determined moving object image as the determination image instead of the reference image.

Further, for the same reason, in the 3D coordinate projection means 2215, which will be described later, the reference image is inappropriate as the projection destination image, so the determined moving object image is used as the determination image instead of the reference image.

The moving object imaging information 2224 further stores a target object region in which the target object 20 is shown in the determined moving object image and target object feature points extracted from the region.

The position and orientation acquisition unit 223 is a means for acquiring the position and orientation information of the drone 200 itself expressed in world coordinates in the monitoring area 10 at each processing time.

The position and orientation acquisition unit 223 internally includes a receiver that receives radio waves (navigation signals) transmitted from a navigation satellite (artificial satellite) such as GNSS (Global Navigation Satellite System), an acceleration sensor that measures acceleration, and an acceleration sensor that measures direction. It has an electronic compass, a gyro sensor to measure angular velocity, and a barometric pressure sensor to measure altitude. Other sensors may also be provided.

Then, the position and orientation acquisition unit 223 calculates the position of the drone 200 by integrating these sensors.

The moving object feature point extracting means 2211 is a means for extracting moving object feature points from the moving object image captured by the moving object imaging means 210 by the same process as the object feature point extracting means 152 of the main device 140.

The number of moving body feature points to be extracted is tens of thousands for a moving body image of 1600×1200 pixels. Rather than carefully processing a small number of highly accurate feature points, that is, feature points that have been accurately extracted from the target object, this method first extracts a large number of feature points and then adjusts them accordingly to prevent omissions. It is based on the idea that feature points that do not satisfy conditions are excluded from processing during the processing process.

The feature point selection means 2212 is a means for selecting moving object feature points to be used for calculating three-dimensional coordinates as described later from among the moving object feature points extracted by the moving object feature point extraction means 2211.

As described above, the number of feature points extracted by the moving object feature point extracting means 2211 is tens of thousands. These are not necessarily feature points extracted in accordance with the image of the target object 20 in the moving body image. Therefore, it is meaningless to mechanically calculate the three-dimensional coordinates of all feature points, so if a target object is captured by the operator, it will be associated with the target feature points extracted from the target object area specified by the operator. Select moving object feature points.

For this purpose, the feature point selection means 2212 reads image feature amounts obtained from surrounding small areas including each moving body feature point, and object feature points and object regions stored in the object information, and selects the object. The feature points are compared with the image feature values obtained from the surrounding small area including the feature points, and the feature points determined to be most similar regardless of degree are associated with each other, and the associated moving object feature points are These points are treated as "tracked feature points" and are subject to subsequent processing.

In other words, for each target object feature point, one of the moving body feature points is always selected as a tracking feature point. Considering that there is a difference in the imaging angle of the reference imaging device 300 and the mobile object imaging means 210 of the drone 200, the feature values will change and it will be difficult to correlate them, but among them, the images are as similar as possible. The intention is to select In addition, we will make a large number of correspondences forcibly once, and then remove inappropriate correspondences from the subsequent processing. On the other hand, moving object feature points that are not matched are not actively deleted.

That is, among the tens of thousands of moving object feature points extracted by the moving object feature point extracting means 2211, several thousand points, the same number as the target object feature points, are selected as tracking feature points.

In the above description, the feature point selection means 2212 uses the image of the reference imaging device 300 (as a judgment image) stored in the object information 2223 when selecting feature points used to calculate three-dimensional coordinates. The feature points (object feature points) of the moving object image (reference image) and the corresponding feature points (moving object feature points) are selected. Furthermore, as will be described later, if conditions such as the size of the target object in the moving object image and the size of the target object area in the reference image as the determination image are significantly different are met, the size of the target object in the moving object image is stored in the target object information 2223. Instead of the target object feature points, feature points whose three-dimensional coordinates are calculated are selected based on the correspondence (matching) with the target object feature points of the target object region in the determined moving object image as the determination image.

The feature point search means 2213 reads the image feature amount obtained from the moving object feature point and the small area around the moving object feature point for the past moving object image stored in the moving object imaging information 2224, and selects the feature point at the current time. In the means 2212, the degree of similarity between the selected tracking feature point and the image feature of a small area surrounding it is checked, a corresponding moving object feature point ("key frame feature point") is identified, and the tracking feature point and the key frame feature point are identified. Find pairs with frame feature points.

As mentioned above, since there are thousands of tracked feature points, thousands of pairs of tracked feature points and key frame feature points are also required. Also, in this case as well, it may be inappropriate to associate them based on the degree of similarity between image feature amounts, but from the perspective of placing emphasis on preventing missing feature points, pairs are determined for all tracked feature points. However, even if emphasis is placed on preventing missing feature points, performance may deteriorate if there are only pairs of completely unrelated feature points, so we will impose an epipolar constraint condition that is well known in stereo image technology. That is, for the tracked feature points, key frame feature points that satisfy the epipolar constraint are selected and paired.

FIG. 4 schematically shows the processing of the feature point selection means 2212 and the feature point search means 2213. Note that FIG. 4 shows a state in which the drone 200 has just taken off from the standby base 30 and started flying, and the reference image is used as the determination image.

FIG. 4A shows a reference image 401 acquired by the reference imaging device 300, and is an image stored in the reference captured image 2222 of the drone 200. A target object 20 is shown near the center.

FIG. 4B shows a moving object image 402 acquired by the moving object imaging means 210. This is assumed to have been obtained at the current time. A target object 20 is shown near the center.

FIG. 4C shows processing for object feature points extracted from the reference image 401 and moving object feature points extracted from the moving object image 402.

Reference numeral 4010 indicates the target object 20 (target region) captured in the reference image 401 and the extracted target object feature points (see the x marks). As mentioned above, several thousand points are actually extracted and stored in the object information 2223.

Reference numeral 4020 indicates the target object 20 captured in the moving object image 402 at the current time t, and the extracted moving object feature points (see the x marks).

Reference numeral 4021 indicates past time tn, and indicates the target object 20 captured in the moving body image 402 and the moving body feature point (see the cross mark) extracted from the target object region in which the target object 20 is captured. There is. As mentioned above, tens of thousands of points are actually extracted and stored in the moving object imaging information 2224.

The feature point selection means 2212 compares the extracted moving body feature points with the feature points of the determination image and selects tracking feature points corresponding to the feature points of the determination image.

That is, while the determination image is the reference image, the feature point selection means 2212 selects the extracted moving body feature points (marked with an x in 4020) and the information stored in the object information 2223, as shown by reference numeral 404. The correspondence relationship with the target object feature point (x mark 4010) is determined. The correspondence relationship depends on the degree of similarity between the image features obtained from the small area centered on the feature point.
Further, after the determined moving object image is updated to the determination image, the feature point selection means 2212 selects the extracted moving object feature points and the selected moving object image stored in the determined moving object image 2225. The correspondence between the target object area and the moving body feature points is determined.

As a result, several thousand "tracking feature points" are selected from the tens of thousands of moving object feature points extracted from the moving object image 402.

Then, the feature point search means 2213, as indicated by reference numeral 403, examines the correspondence between the tracked feature points and the tens of thousands of moving object feature points at time tn stored in the moving object imaging information 2224. , identify thousands of pairs of tracked feature points and keyframe feature points.

The parallax determining means 2214 compares the past moving object image and the current moving object image using the pairs obtained by the feature point searching means 2213, and determines the parallax sufficient to obtain the three-dimensional coordinates of the feature point. Determine whether or not.

For this purpose, the parallax determining means 2214 calculates the average value of the difference in the coordinate values of the feature points in the moving object image for the thousands of pairs, and if the average value exceeds a predetermined threshold, the It is determined that the parallax is sufficient to obtain the dimensional coordinates.

The 3D coordinate projection means 2215 is means for determining the three-dimensional coordinates expressed in the world coordinates of the pair of feature points found in the feature point searching means 2213 and projecting them onto the determination image (reference image or determined moving object image). be.

For this purpose, the 3D coordinate projection means 2215 uses information such as the position of the drone 200 in the monitoring area 10, the orientation/attitude of the drone 200, and the focal length of the mobile object imaging means 210 stored in the moving object imaging information 2224 to create a so-called stereo image. Three-dimensional coordinates in the world coordinate system are determined based on the image method.

Then, based on the reference imaging parameters or the imaging parameters related to the determined moving object image transmitted from the main device 140 and stored in a buffer area (not shown) of the moving object storage section 222, the determined three-dimensional coordinates are determined in the judgment image ( Projection processing is performed to determine which pixel in the reference image or determined moving object image is pointed to.

The evaluation determining means 2216 determines whether or not the target object is reflected in the moving body image based on the state of the projection point projected onto the determination image. The projection point is a projection of the tracking feature point, and the tracking feature point is a moving body feature point that is determined to correspond to the feature point of the determination image. In other words, the evaluation determining means 2216 determines whether or not the target object is included in the moving body image based on the moving body feature points determined to correspond to the feature points of the determination image.

This situation is schematically shown in FIG.

FIG. 5A shows how the three-dimensional coordinates of the pair of feature points found by the feature point searching means 2213 are projected onto the reference image 501. The x mark is the result of projection. This is what happened shortly after Drone 200 took off.

FIG. 5A is a schematic diagram showing an ideal projection when a reference image is used as a determination image. This is a case where the tracking feature point selected in association with the target object feature point by the feature point selection means 2212 is ideally located on the target object in the moving object image.

In this case, the projection destination onto the reference image 501 matches the target object area. As a result, the evaluation determining means 2216 determines that the target object 20 is included in the moving object image.

However, in reality, not all of the tracking feature points associated with target object feature points are located at the target object in the moving object image. This is because, as shown in FIG. 5B, the reference imaging device 300 and the moving body imaging means 210 have different imaging distances and imaging angles with respect to the target object 20 in the monitoring area 10, and accordingly, the target object 20 in the image This is because they look different.

FIG. 6A shows an example of a moving object image 601 when the moving object imaging means 210 images the target object 20 from the position of the drone 200 shown in FIG. 5B.

The size of the target object 20 shown in the moving object image 601 in FIG. 6(a) is significantly different from that of the target object shown in the reference image 501 shown in FIG. 5(a).

Due to the difference in visibility, when the feature point selection means 2212 selects a moving object feature point based on the target object feature point and sets it as a tracked feature point, the moving object feature point on the target object in the moving object image is not tracked. The possibility of being selected as a feature point decreases. In addition, an exhaustive search is performed for the moving object feature points in the moving object image for the target object feature points, and in some cases, moving object feature points at a completely different position from the target object may be selected. could be.

This is because when matching feature points with each other (feature point matching), a small region of approximately several dozen pixels square is considered around the feature point of interest, and image features within that region are compared. In other words, when the sizes of the target objects are completely different as shown in Figures 5(a) and 6(a), the difference in resolution is large, and even if feature points extracted from the same part of the same object are This is because the images contained therein will be completely different, resulting in failure to correspond (matching error).

The situation is shown in FIG. 6(b). Reference numeral 6010 indicates the target object 20 (target area) captured in the reference image 501 and the extracted target object feature points (see the cross marks). Reference numeral 6020 indicates the target object 20 captured in the moving object image at the current time t and the extracted moving object feature points (see the x mark). Reference numeral 6021 indicates past time tn, and indicates the target object 20 captured in the moving object image and the extracted moving object feature points (see the x mark).

As can be seen from the figure, the moving body feature points marked with an x surrounded by a dotted line are located at different positions from the target object 20, but they are associated with the target object feature points, and the feature point searching means 2213 They are selected as a pair for finding dimensional coordinates.

Even if the three-dimensional coordinates are obtained by pairing the feature points of completely different objects, the values become meaningless, and even if they are projected onto the reference image 501, as shown in FIG. 6(c), pixels far from the target object 20 are obtained. The projection point can only be located at .

Therefore, as shown in FIGS. 5(a) and 6(a), when the size of the target object 20 in the reference image 501 and the target object 20 in the moving object image of the moving object imaging means 210 are significantly different, The determination image used by the point selection means 2213 and the 3D coordinate projection means 2215 is replaced with the determined moving object image instead of the previous reference image.

That is, the 3D coordinate projection means 2215 calculates the difference between the size of the object area in the reference image stored in the object information 2223 and the size of the object area of the moving body image calculated by the image size determination means 2217. If it is above a certain level, the determined moving body image is used as the determination image to calculate the three-dimensional coordinates and perform a projection process onto the determination image.

The processing will be explained using FIG. 7 while comparing it with FIG. 6.

FIG. 7A shows a moving object image 701 and extracted moving object feature points (see cross marks) when the above-mentioned difference is a certain value or more. The moving object feature points are extracted approximately from the vicinity of the target object 20.

Similarly to FIG. 6(b), FIG. 7(b) schematically shows the process of matching feature points between moving object images and selecting feature points for which three-dimensional coordinates are to be determined by the feature point selection means 2212. ing.

Reference numeral 7010 indicates the target object 20 (target object region) captured in the determined moving body image and target object feature points (see the cross marks) extracted from the region. Reference numeral 7020 indicates the target object 20 captured in the moving object image at the current time t, and the extracted moving object feature points (see the x mark). Reference numeral 7021 indicates past time tn, and indicates the target object 20 captured in the moving object image and the extracted moving object feature points (see the x marks).

A comparison with FIG. 6B shows that the moving object feature points selected by the feature point selection means 2212 for determining three-dimensional coordinates are prevented from being specified far from the target object 20.

This is because the sizes of the images of the target object 20 in the determined moving body image and the moving body image are approximately the same.

Then, the 3D coordinate projection means 2215 obtains the three-dimensional coordinates of the moving object feature points selected by the feature point selection means 2212, and projects them onto the determined moving object image as the judgment image.

The evaluation determining means 2216 divides the determination image 801 shown in FIG. 8(a) into blocks. Note that FIG. 8 schematically shows a case where a reference image is used as the determination image. In other words, as shown in FIG. 8(b), the position and size of the object area in the reference image stored in the object information 2223 or the determined moving object image stored in the moving object storage section 222 Based on the position and size of the object area inside, one block containing the entire object area is secured and divided into blocks (see the dotted line in FIG. 8(b)). Note that a block is an example of a local area. A block containing the entire object region contains many projection points.

The evaluation determining means 2216 then counts the number of projection points included in each block and identifies the block with the largest number.

The number of projection points included in the largest number of blocks is set to Nmax, the total number of projection points projected by the 3D coordinate projection means 2215 is set to Nall, the area ratio of the local region to the area of the determination image is set to R, and the intensive adjustment is performed. Let the coefficient be α (for example, 10.0). At this time, if the following equation is satisfied, the evaluation determining means 2216 determines that the target object 20 is included in the moving body image.

Nmax > α×R×Nall (1)

Equation (1) is used to evaluate how many more projection points are actually projected compared to the number of projection points that could be included in the largest number of blocks when the projection points are completely evenly scattered throughout the judgment image. . Note that α×R×Nall is an example of the concentration determination threshold.

The explanation so far has been based on the assumption that the target object 20 is shown in the moving object image, but even if the orientation of the drone 200 is inappropriate and the target object 20 is not shown in the moving object image, the three-dimensional coordinates will be asked for.

If the target object 20 is not captured in the moving body image, even if the three-dimensional coordinates are projected onto the determination image, they will be scattered throughout the determination image 801 as shown in FIG. 8(c), so the above (1) Does not satisfy the formula.

Note that in the above description, the determination image is divided into blocks such that one block coincides with the target object area. Instead, as shown in FIG. 8(d), while the size of one block (width and height are W) remains unchanged, blocks are shifted vertically and horizontally relative to the object area. You can divide it. For example, by setting the shift adjustment coefficient β to 0<β<1.0, and shifting the largest number of divided blocks by β·W in at least one of the vertical and horizontal directions, the number of projection points is increased. , and the block with the largest number may be searched. In addition, if the shift adjustment coefficient β that defines the shift amount with respect to the target object area of the block with the largest number of projection points, which is searched by shifting, is less than a threshold value (for example, 1.0), and the projection points included in the block are If the number is larger than α×R×Nall, it is determined that the target object is included in the moving object image. Note that the threshold value for the shift adjustment coefficient β is set in advance as a setting parameter, but is determined by the accuracy of the position and orientation acquisition unit 223. For example, 3.0 can be set as the upper limit depending on the measurement error circle of the built-in GNSS.

This is because the parameters of the reference imaging device 300 stored in the reference imaging parameters 161 and the position and attitude of the drone 200 stored in the mobile object imaging information 2224 inevitably include errors.

For example, even if the imaging angle of the reference imaging device 300 is precisely measured at the time of installation and stored in the reference imaging parameter 161, an error of just 1 degree can become tens of centimeters to several meters at a distance of several tens of meters.

Additionally, multiple GNSS and sensors are used to determine the position and attitude of the drone 200, but these include certain errors.

These errors affect the projection onto the judgment image by the 3D coordinate projection means 2215, and as shown in FIG. 8(e), although the projection points are concentrated in a certain local area, It appears as a deviation from the area.

Therefore, as shown in FIG. 8(f), it is preferable to allow the block in which the projection points are concentrated to shift within a certain range (see the dashed-dotted line in FIG. 8(f)).

Then, the evaluation determining means 2216 updates and stores the moving object image determined to include the target object at the update storage timing in the moving object storage section 222 as a determined moving object image, as will be described later. Further, the object area of the moving object image is stored in the moving object imaging information 2224.

Furthermore, after a determined moving object image has been stored, the size of the target object included in the latest moving object image is referred to, and if there is a large difference in size from the size of the target object included in the stored determined moving object image, You can update it if it becomes too much.

The image size determination means 2217 uses the focal length (angle of view) of the mobile object imaging means 210 stored in the mobile object imaging information 2224, the position of the drone 200 in the monitoring area 10 acquired by the position and orientation acquisition unit 223, and the target object 20. The size of the target object area in which the target object 20 is photographed in the moving body image determined by the evaluation determining means 2216 as having the object 20 is calculated based on the posture relative to the target object 20 .

In this embodiment, imaging parameters such as the focal length and imaging direction of the reference imaging device 300 are stored in the object information 2223. Further, the object position calculation means 151 of the main device 140 calculates the position of the target object 20 in the real world.

Therefore, the image size determining means 2217 determines the actual size of the target object 20 and calculates how large it will be reflected in the moving object image. The size of the target object area in which the target object 20 is captured in the moving body image determined to be present is calculated.

Note that the target object area in which the target object 20 is photographed and its size are projected into the target object area in the determination image when the evaluation determination means 2216 determines that the target object 20 is photographed in the moving object image. The moving object feature point is determined from the circumscribed rectangle in the moving object image.
Then, the determined object area in which the target object 20 is captured and its size are stored in the moving object imaging information 2224.

The operation of the imaging system 1 according to the present invention will be described below using a flowchart.

FIG. 9 is a flow diagram showing the operation of the main device 140 installed on the center side.

First, the system operator starts from a point where he or she perceives the presence of the target object 20 in the monitoring area 10 by, for example, the sound of some kind of alarm device or visual confirmation of the display device 120, and wants to display it in an enlarged manner. do.

It is assumed that the focal length of the reference imaging device 300, the position expressed in world coordinates, and the imaging direction are stored in advance in the reference imaging parameter 161, and the shape information of the monitoring region 10 is stored in the monitoring region map 162. Further, the reference image is sequentially transmitted to the main device 140 and displayed on the display device 120.

First, in step S100, the operator of the system visually checks the display device 120 and operates the designation input device 130 to designate an area representing the target object 20 on the reference image. Typically, a region representing the target object 20 is specified by a circumscribing rectangle. The image inside the designated area becomes the image of the object area.

In step S105, the object feature point extracting means 152 extracts object feature points from the object region using a well-known method. The number of object feature points is, for example, approximately several thousand points.

In step S<b>110 , the object position calculation means 151 calculates the position of the target object 20 captured in the reference image expressed in world coordinates in the monitoring area 10 while referring to the reference imaging parameters 161 .

In step S120, the main device 140 transmits the reference image, the world coordinates of the position of the target object 20 obtained in step S110, and the image of the target object region in the reference image and its region information to the drone through the main device communication unit 110. Send to 200.

In step S130, a takeoff and start command for heading to the world coordinates of the position of the target object 20 is transmitted to the drone 200 via the main device communication unit 110 as a control command input by the operator by operating the designated input device 130. do.

In step S140, it is determined whether a new moving object image has been transmitted from the drone 200. When a new moving object image is appropriately transmitted from the drone 200 (Yes branch in step S140), the main device 140 updates and displays the image on the display device 120 in step S150. On the other hand, until a new moving object image is transmitted, the previous moving object image continues to be displayed (branch of the same No.).

In step S160, main device 140 determines whether a return command for drone 200 has been input by the operator who operated designation input device 130.

When a return command for the drone 200 is input by the operator who operated the designated input device 130 (Yes branch in step S160), a control signal indicating the return command is sent to the drone 200 via the main device communication unit 110 in step S170. Sent. If no input is made, the process returns to step S140 and waits for a new moving object image to be transmitted (No branch of step S160).

FIG. 10 is a flow diagram showing the operation of the drone 200. It is assumed that before step S310 shown in FIG. 10, the drone 200 itself takes off in response to the takeoff/start command shown in step S130 of FIG.

In step 310, the moving object processing unit 221 uses the reference image stored in the reference captured image 2222 as the determination image.

In step S320, the position and orientation acquisition unit 223 refers to the GNSS technology and outputs of sensors (not shown) (altitude sensor, direction sensor, angle sensor, etc.) to obtain a world defined with the corner of the monitoring area 10 as the origin, for example. The position and attitude (imaging direction and tilt) of the drone 200 in the coordinate system are calculated.

In step S330, the flight control unit 225 takes into consideration the world coordinates of the position of the target object 20 calculated by the object position calculation means 151 of the main device 140, and the flight control unit 225 moves the drone 200 closer to the target object 20 in the monitoring area 10. A control signal is output to rotor drive section 230.

In step S<b>340 , the mobile object imaging means 210 captures an image at the position of each drone 200 at each time as appropriate, and outputs the acquired mobile object image to the mobile object processing unit 221 of the mobile main body 220 .

The moving object feature point extracting means 2211 then extracts moving object feature points from the moving object image using a well-known method. Alternatively, an image of a small area around the extracted moving body feature point may be extracted.

In step S350, it is determined whether various information has already been stored in the moving object imaging information 2224 of the moving object storage section 222. If the process in step S340 is for the first moving object image (if the branch is "none" in step S350), the process proceeds to step S530, where the extracted moving object feature points, the moving object image, or the extracted small The image of the area and the position and orientation of the drone 200 obtained in step S320 are newly recorded in the mobile object imaging information 2224 of the mobile object storage section 222.

In step S360, the moving object processing section 221 checks whether the moving object storage section 222 stores a determined moving object image.

If the determined moving object image is not stored in the moving object storage unit 222 (if the branch is No in step S360), the drone 200 has just taken off and started, and for example, the target object 20 is out of the field of view of the moving object image. This means that it has never been determined that the image is visible. Therefore, the process proceeds to step S380 to perform processing by the feature point selection means 2212 using the object feature points of the image of the object region extracted in the flowchart of the main device 140 in FIG. 9 (step S105).

In step S360, if the determined moving object image is stored in the moving object storage unit 222 (in the case of a Yes branch in step S360), it is determined that the target object 20 is shown in the moving object image. This means that there is a proven track record.
Therefore, in step S370, the moving object feature point extracting means 2211 performs a process of extracting moving object feature points from the target object region in which the target object 20 is shown in the stored determined moving object image (or When storing the determined moving object image, the moving object feature points are extracted from the object region and stored in the moving object storage section 222 at the same time), and these points are used as the object feature points.

In step S380, the feature point selection means 2212 selects one corresponding to the object feature point from among the moving body feature points of the moving body image extracted in step S340, and sets it as a tracking feature point (FIG. 6 (b), see FIG. 7(b). Select the moving object feature point at time t).

In step S390, the feature point searching means 2213 selects the moving object feature point of the past moving object image stored in the moving object imaging information 2224, which corresponds to the tracking feature point selected in step S380, Set as a key frame feature point (see FIGS. 6(b) and 7(b). Select the moving object feature point of the moving object image at time tn).

In step S400, the parallax determining means 2214 determines the moving body image of pairs (for example, several thousand pairs) of the key frame feature point selected in step S390 and the corresponding tracking feature point selected in step S380. The average value of the distances above is determined as the parallax. In step S410, it is determined whether the parallax is above a certain level, and if the parallax is less than a certain level (if the branch is No in step S410), the amount of movement of the drone 200 is small and stereo image technology is used. Since it is inappropriate to obtain the three-dimensional coordinates of the feature points, the process moves to step S320.

If the average value of the distances is equal to or greater than a certain value (Yes in step S410), it is determined that the parallax is sufficiently accurate to obtain the three-dimensional coordinates of the feature point, and the process proceeds to step S420.

In step S420, the 3D coordinate projection unit 2215 determines the tracking feature point ( Refer to FIGS. 6(b) and 7(b). The three-dimensional coordinates (world coordinate representation) of the moving body feature point at time t are calculated. In this case, the information stored in the monitoring area map 2221, the moving object imaging information 2224, etc. may be appropriately referred to, and the information may be determined using a so-called stereo image method.

In step S430, the moving object processing unit 221 determines the size of the target object 20 in the moving object image stored in the moving object imaging information 2224 and the target object area in the reference image stored in the object information 2223. The sizes are compared and it is determined whether the degree of difference is greater than a certain level.

If the degree of difference is small (No branch in step S430), the reference image stored in the reference captured image 2222 is set as the determination image in step S440.

In step S430, if the degree of difference is large (in the case of a Yes branch in step S430), in step S450, the stored determined moving object image is set as the determination image.

In step S460, the 3D coordinate projection unit 2215 projects the three-dimensional coordinates of the tracking feature point obtained in step S420 onto the determination image (see FIG. 8(a)).

In step S470, the evaluation determining means 2216 divides the determination image onto which the tracking feature points are projected into blocks. At the initial stage, the reference image transmitted from the main device 140 is used as the determination image. In this case, the size of one block is made to match the size of the target object area for the target object 20 specified and input in step S100. When one block is made to match the target object area, other blocks are sequentially defined accordingly (see FIG. 8). When the determined moving body image is a determination image, the size of one block is made to match the size of the target object area of the determined moving body image. This target object area and its size are stored in the moving object imaging information 2224.

Then, for each block, the number of included projection points is counted to identify the block with the largest number and the number of projection points.

In step S480, it is determined that the number of projection points included in the largest number of blocks is equal to or greater than a certain value, that block overlaps with the object area in the determination image, and that the amount of deviation is within a predetermined range (FIGS. 8(d) to (f) )). If the number of projection points included in the largest number of blocks is a certain number or more, and the block overlaps with the target object area in the determination image, and the amount of deviation is within a predetermined range (in the case of a Yes branch in step S480), In step S490, it is determined that the target object 20 is included in the latest moving object image. It also sends a message to that effect to the main device 140.

In step S500, it is determined whether it is time to update the determined moving object image. For example, by referring to the size of the target object area included in the moving body image determined to include the target object 20, the object appearing in the determination image (reference image or stored determined moving body image) If the difference with the size of the object area exceeds the threshold, it is determined that it is time to update the determined moving object image (Yes branch in step S500), and in step S510, it is determined that the target object 20 is captured. The determined moving object image is newly stored or updated as a determined moving object image, and the object feature points extracted from the object region of the moving object image are updated and stored in the moving object imaging information 2224. Note that the size of the target object area included in the moving body image determined to include the target object 20 is the size of the target object area included in the target object image that is determined to include the target object 20 of the moving body feature point projected within the target object area of the determination image. It may be determined from the circumscribed rectangle in the moving object image that is determined to be the same.

On the other hand, by referring to the size of the target object area included in the moving body image determined to include the target object 20, the difference between the size of the target object area included in the stored and determined moving body image is determined. If it is less than or equal to the threshold, it is determined that it is not time to update the determined moving object image (No branch of step S500), and the process moves to step S520.

Note that it may be determined whether it is the update timing based on whether a certain period of time has elapsed or the position of the drone 200 in the monitoring area 10.

In step S520, the moving object processing unit 221 transmits the latest moving object image to the main device 140.
Note that if it is determined that the target object 20 is included in the moving body image and the size of the target object area is smaller than the size of the target object area in the determination image by a predetermined amount or more, the reference image is replaced with the determination image. You can also reset it as . This is to ensure determination accuracy even if the drone moves away from the target object 20 during the flight control process of the drone 200. Furthermore, even if it continues to be determined that the target object 20 is not included in the moving body image for a certain period of time or more, the reference image may be reset as the determination image.

In step S530, the moving object feature point of the moving object image extracted in step S340, the image information of the moving object image itself or a small area around the moving object feature point, the position and orientation of the drone 200 calculated in step S320. etc. are updated and stored as moving object imaging information 2224.

In step S540, it is determined whether a return command to the standby base 30 has been received from the main device 140. If a command to return to the standby base 30 has been received from the main device 140 (Yes branch in step S540), the flight control unit 225 returns to the standby base 30 while referring to the monitoring area map 2221 in step S550. A control signal is output to the rotor drive unit 230 as needed. If the return command has not been received (No branch in step S550), the process returns to step S320 and continues processing such as imaging.

As described above, in the imaging system 1 according to the embodiment of the present invention, the drone 200, which is a moving object, has a plurality of movements captured by the moving object imaging means at a plurality of imaging positions having a predetermined parallax. Extract the moving object feature points from each body image, perform 3D measurement to obtain the 3D coordinates of the moving object feature points, and project the 3D coordinates onto a determination image that is a reference image or a determined moving object image. Find the projection point. The drone 200 determines that the target object is reflected in the moving body image when the number of projection points included in the block corresponding to the target object area in the determination image is greater than the concentration determination threshold. Thereby, even if the size of the target object in the moving body image changes as the moving body moves, it is possible to accurately determine whether the target object is included in the moving body image.

<Modified example>
Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, the image size determination means can determine the focal length of the reference imaging device when capturing the reference image, the focal length of the moving object imaging means when capturing the moving object image, and the object size from the reference imaging device when capturing the reference image. Based on the distance to the object and the distance from the moving object imaging means to the target object when the moving object image is captured, the size of the target object area and the object shown in the moving object image stored in the moving object storage means are determined. Dissimilarity estimation means for calculating a ratio of the size of the object as the dissimilarity between the size of the target object area in which the target object is photographed in the reference image and the size of the target object photographed in the moving body image photographed by the photographing means. You may be prepared. In this case, instead of determining the size of the target object in the moving object image, the size ratio is determined from the focal length of each camera and the distance to the object in the real world, and if the size ratio is above a certain value, If there is, the determined moving object image may be used as the determination image.

Here, the focal length information indicates how many pixels (px) appear on the image when an object of 1 m size located 1 m ahead is imaged. The focal length of camera a is Pa [px], the focal length of camera b is Pb [px], the distance from camera a to the target object is Da [m], and the distance from camera b to the same target object When Db[m] is the actual size of the target object and S[m] is the actual size of the target object, the following equation is obtained.

Size of target object in image of camera a = (S×Pa)/Da
Size of target object in image of camera b = (S×Pb)/Db
Size ratio of the target object in the image of camera b to the image of camera a = (S × Pb / Db) ÷ (S × Pa / Da) = (Pb × Da) / (Pa × Db)

From the above equation, even if the actual size S of the target object is unknown, the size ratio on the image can be determined as the degree of dissimilarity.

In addition, there is a function to extract feature points from a moving object image, a function to obtain the three-dimensional coordinates of the feature point and project it onto a determination image, and a function to determine whether a target object is reflected in a moving object image based on the number of projected points in a local area. The main device 140 may be configured to have a function to do so. In this case, the drone 200 may sequentially transmit moving object images captured by the moving object imaging means to the main device 140, and the main device 140 may include each means of the moving object processing section 221.

Furthermore, although a case has been described in which a drone is used as an example of a moving body, the present invention is not limited to this, and any moving body other than a drone may be used as long as it is equipped with a moving body imaging means.

Further, in the above embodiment, a case has been described in which the reference imaging device is fixedly installed, but the present invention is not limited to this. For example, if a plurality of drones are used, the reference imaging device may be provided in a drone different from the drone provided with the moving object imaging means, and the reference imaging device may be moved. In that case, it is assumed that the position of each drone expressed in world coordinates in the imaging parameters and monitoring area can be determined as appropriate.

Furthermore, in the above embodiment, a drone, that is, a flying robot, is used as an example of a moving body, but the present invention is not limited to this. It can also be applied to floor-moving robots that have a travel control unit, wheels, etc. instead of a flight control unit, rotor, etc.

Furthermore, instead of being a flying robot or a floor-moving robot, a wearable imaging device worn by a security guard may be configured as a moving object, as long as it is miniaturized to the extent that it can be easily transported.

Further, in the embodiment described above, the reference image and the image of the target object area are transmitted to the drone as an example, but the present invention is not limited to this. For example, the object area specified by the operator who visually confirmed the reference image may be directly transmitted to the drone. The transmitted information (position, size) of the target object area may be stored in the target object information 2223, and the projection point may be projected onto the imaging surface of the reference imaging device.

As described above, those skilled in the art can make various changes within the scope of the present invention according to the embodiment.

1 Imaging system 10 Monitoring area 20 Target object 100 Center side device 110 Main device communication section 120 Display device 130 Specification input device 140 Main device 150 Main device processing section 151 Object position calculation means 152 Object feature point extraction means 160 Main device storage section 161 Reference imaging parameters 200 Drone 210 Mobile object imaging means 220 Mobile object body 221 Mobile object processing section 222 Mobile object storage section 223 Position and orientation acquisition section 224 Mobile communication section 225 Flight control section 230 Rotor drive section 231 Rotor 300 Reference imaging device 2211 Mobile feature point extraction means 2212 Feature point selection means 2213 Feature point search means 2214 Parallax determination means 2215 Coordinate projection means 2216 Evaluation determination means 2217 Image size determination means

Claims (4)

A moving body that moves relative to a target object,
The reference imaging device receives a reference image capturing a monitoring range in which the target object may exist as a determination image, and receives as a target object area an area designated by an operator as the target object in the reference image. a means of communication to
a mobile object imaging means for sequentially outputting mobile object images captured in the monitoring range;
a determination unit that compares the feature points of the determination image and the feature points of the moving object image to determine the presence or absence of a partial area corresponding to the target object area in the moving object image;
a moving body storage means for storing the moving body image in which it has been determined that the partial area exists in association with the partial area;
has
The determining means is configured to use the latest moving object image stored in the moving object storage means for the determination when the degree of difference between the size of the object area and the size of the partial area exceeds a predetermined threshold. A moving body, characterized in that the image is changed to an image, and a partial area associated with the moving body image is changed to the target object area. The mobile body further includes:
the focal length of the reference imaging device when capturing the reference image, the focal length of the moving object imaging means when capturing the moving object image, and the distance from the reference imaging device to the target object when capturing the reference image. and the distance from the moving object imaging means to the target object when the moving object image is captured, the size of the target object area and the partial area stored in the moving object storage means. The moving object according to claim 1, further comprising a dissimilarity estimation unit that calculates a ratio of the dissimilarity to a size as the dissimilarity. A mobile unit that includes a reference imaging device that images a monitoring range in which a target object may exist, and a mobile imaging unit that moves relative to the target object and outputs a mobile image that captures the monitoring range. An imaging system comprising a body,
The mobile body is
a communication means that receives a reference image captured by the reference imaging device of the monitoring range as a determination image, and receives a region designated by an operator as the target object in the reference image as a target object region;
a determination unit that compares the feature points of the determination image and the feature points of the moving object image to determine the presence or absence of a partial area corresponding to the target object area in the moving object image;
a moving body storage means for storing the moving body image in which it has been determined that the partial area exists in association with the partial area;
has
The determining means is configured to use the latest moving object image stored in the moving object storage means for the determination when the degree of difference between the size of the object area and the size of the partial area exceeds a predetermined threshold. An imaging system characterized by changing the partial region associated with the moving body image to the target object region. A mobile device comprising: a reference imaging device that images a monitoring range in which a target object may exist; and a mobile imaging unit that moves relative to the target object and outputs a moving object image that captures the monitoring range. An imaging system including a body, and an information processing device communicatively connected to the moving body and the reference imaging device,
The information processing device includes :
The reference imaging device receives a reference image capturing the monitoring range as a determination image, and receives a region designated by an operator as a target object in the reference image as a target object region , and communication means for receiving the mobile object image from the mobile object ;
a determination unit that compares the feature points of the determination image and the feature points of the moving object image to determine the presence or absence of a partial area corresponding to the target object area in the moving object image;
a moving body storage means for storing the moving body image in which it is determined that the partial area exists in association with the partial area;
has
The determining means is configured to use the latest moving object image stored in the moving object storage means for the determination when the degree of difference between the size of the object area and the size of the partial area exceeds a predetermined threshold. An imaging system characterized in that the partial region associated with the moving body image is changed to the target object region.