JP6997057B2 - Devices, programs and methods for estimating terminal position from non-animal body information - Google Patents

Description

The present invention relates to a technique for autonomously estimating the position of a terminal.

Currently, as seen in autonomous mobile bodies such as self-driving cars and autonomous mobile robots, the development of technology for autonomous movement is being actively promoted regardless of operations and controls from outside or outside the aircraft. In this technology, for example, an image taken by a camera or a point cloud (point cloud) by LiDAR is acquired to grasp the surrounding situation, but at that time, it is also possible to estimate its own position. It has become an important issue.

In addition, pedestrian dead reckoning (PDR) technology that estimates the current position and movement route of a user who owns a communication terminal using information acquired by the terminal, regardless of GPS (Global Positioning System), has also made great progress. Is showing.

As such an autonomous position estimation technique, for example, Patent Document 1 discloses a technique for visually identifying a position by applying information acquired from one camera to a three-dimensional LiDAR map. There is. In this technique, the rough position is first determined by GPS, and then a composite image for comparison with the image data obtained by the camera is generated. Here, since the standardized common image information is set to be acquired as much as possible, it is possible to sequentially align the image data on the generated map in real time.

Further, for example, in Non-Patent Document 1, as a position estimation technique in a wide range of indoors and buildings, a dense feature is extracted and a matching process is performed in a series of position verification steps that are sequentially tightened. The technology is disclosed. With this technology, the orientation (posture) of the camera is estimated by performing matching processing using dense features instead of local features in order to deal with poorly textured backgrounds such as walls, ceilings, and floors. To. Here, as the orientation of the camera that captured the target image, the orientation of 6DoF (6 Degrees of Freedom) in the three-dimensional map is also determined.

Further, for example, Non-Patent Document 2 discloses a position estimation technique in which SLAM (Simultaneously Localization and Mapping) and position estimation in a global range are combined. Of these, SLAM is implemented on mobile terminals and estimates 6DoF for terminals. On the other hand, the position estimation in a large area using the key frame is performed in the server, the position estimation result is notified to the mobile terminal, and is used together with the GPS positioning result. Here, by performing the matching process using continuous key frames, the estimation performance of the terminal orientation is improved, and the range of estimation information in the position estimation in a large region is expanded.

U.S. Patent Application Publication No. 2016/209846

H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii, “InLoc: Indoor Visual Localization with Dense Matching and View Synthesis” In proc. Computer Vision and Pattern Recognition, 2018 J. Ventura, C. Arth, G. Reitmayr, and Di. Schmalstieg, “Global Localization from Monocular SLAM on a Mobile Phone,” IEEE Transactions on Visualization and Computer Graphics, 20 (4), p531-539, 2014

However, in the prior art including the above-mentioned techniques, the problem that the position estimation error increases due to the dependence on GPS or the influence of a moving object has not been solved yet.

For example, in the technique described in Patent Document 1, the target image is aligned on the reference point cloud by performing the matching process with the composite image. Here, since the initial position is acquired by GPS, if the GPS positioning accuracy is low, there is a possibility that the composite image will be generated at an erroneous position. Further, after that, by comparing the target image and the composite image, errors in image alignment may be accumulated. Furthermore, dynamic obstacles may appear in a wide range in the target image, or deterioration of image information such as image blurring may occur, which may cause problems in the alignment process and the position estimation process. be.

Further, for example, in the technique described in Non-Patent Document 1, a series of camera orientation verification steps cannot cope with such a sudden change in a scene including a large number of moving objects or a scene with extremely high movement, and verification processing is performed. There is a concern that it will fail. In particular, when a moving object occupies most of the target image, the position estimation process and the camera orientation estimation process are extremely seriously affected by the moving object.

Further, for example, in the technique described in Non-Patent Document 2, the local model is constantly adjusted by adding a key frame. As a result, there may be a contradiction between the map on the mobile terminal side and the map on the server side. Since the feature amount detected from the key frame includes the features of all the objects in the field of view of the camera, the presence of the moving object when updating the map on the mobile terminal side and the position estimation result in the global range. Alignment errors can occur.

Therefore, an object of the present invention is to provide a device, a program, and a method capable of performing terminal position estimation processing in which the influence of the presence of a moving object is suppressed without depending on an external positioning system such as GPS. ..

According to the present invention, it is a terminal position estimation device that estimates the position of a terminal in a predetermined area.
Peripheral object information acquisition means for acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A peripheral object that uses a point group information classifier to determine whether the point group information, which is information related to an individual object included in the peripheral point group information, is point group information related to a preset movable object. Information judgment means and
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Peripheral model generation means to generate a model, and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model with each non-animal body as a node and the distance between the non-animal bodies as an edge is generated, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to this non-animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation device having a terminal position determining means for determining is provided.

Further, in the terminal position estimation device according to the present invention , it is also preferable that the terminal position determining means also determines the orientation of the terminal based on the point cloud translation amount and the point cloud rotation amount that maximize the degree of superposition.

Furthermore, in the terminal position estimation device according to the present invention , the terminal position estimation device converts the peripheral point cloud information into point cloud information for each object, which is a set of point data adjacent to each other, based on the distance between the point data. It also has peripheral object information classification means for sorting,
It is also preferable that the peripheral model generation means generates the peripheral non-animal body model by removing the point cloud information determined to be the point cloud information related to the movable object from the point cloud information for each object.

According to the present invention, it is also a terminal position estimation device that estimates the position of a terminal in a predetermined area.
Peripheral image information acquisition means for acquiring peripheral image information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
Peripheral determination using an image information classifier to determine whether the image partial information, which is information related to an individual object included in the peripheral image information, is the image partial information related to a preset movable object. Object information determination means and
Of the information included in the peripheral image information, by performing masking processing on the image partial information determined to be the image partial information related to the movable object, the peripheral image information related to the non-animal body existing in the periphery is applied. Peripheral model generation means to generate an inanimate body model,
Image partial information related to individual non-animal bodies is obtained from a reference non-animal body model, which is a pre- generated model and image information related to non-animal bodies existing in the area, or from a part that is a candidate for a terminal position in the model. Based on the degree of image superimposition between the extracted image partial information and the image partial information related to each inanimate object included in the peripheral inanimate body model, the information relating to the position of the terminal in the area is obtained. With the terminal position determination means to determine
A terminal position estimation device having the above is provided .

Further, in the terminal position estimation device according to the present invention that employs the above peripheral image information, the peripheral image information is obtained by the camera image information generated by the camera mounted on the terminal or the depth camera mounted on the terminal. It is also preferable that it is the generated distance image information.
Further, as an embodiment of the terminal position estimation device according to the present invention described above, the terminal position estimation device is
Based on the information of the observation result at each of the plurality of positions in the area and the information related to the objects existing in the vicinity of the position, the object information in the area which is the information related to the objects existing in the area is generated.
Of the information contained in the object information in the area, the inanimate object information in the area generated by removing the point group information determined to be the point group information related to the movable object, or the object information in the area. Of the information contained in the above, the reference inanimate model is generated or updated by the in-area inanimate information generated by masking the image partial information determined to be the image partial information related to the movable object.
It is also preferred to have additional reference model management means.
Furthermore, the point cloud information related to the non-animal body or the image partial information related to the non-animal body described above may include the object type information in the non-animal body and the information related to the distance between the non-animal bodies. preferable.

According to the present invention, it is also a program for operating a computer mounted on a device for estimating the position of a terminal in a predetermined area.
Peripheral object information acquisition means for acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A peripheral object that uses a point group information classifier to determine whether the point group information, which is information related to an individual object included in the peripheral point group information, is point group information related to a preset movable object. Information judgment means and
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Peripheral model generation means to generate a model, and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model is generated with each non-animal body as a node and the distance between the non-animal bodies as an edge, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to the non -animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation program for operating a computer is provided as a terminal position determining means for determining.

According to the present invention, there is a method for estimating a terminal position in a computer mounted on a device for estimating the position of a terminal in a predetermined area.
A step of acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A step of using a point cloud information classifier to determine whether the point cloud information, which is information related to an individual object included in the peripheral point cloud information, is point cloud information related to a preset movable object. ,
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Steps to generate a model and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model is generated with each non-animal body as a node and the distance between the non-animal bodies as an edge, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to the non -animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation method with a step to determine is provided.

According to the terminal position estimation device, the program, and the method of the present invention, it is possible to carry out the terminal position estimation process in which the influence of the presence of a moving object is suppressed without depending on an external positioning system such as GPS.

It is a schematic diagram which shows one Embodiment of the terminal position estimation system including the terminal position estimation apparatus by this invention. It is a schematic diagram which showed the Example of the peripheral model generation process in the peripheral model generation part schematically. It is a schematic diagram for demonstrating an embodiment of the terminal position candidate information determination process in a terminal position determination unit. It is a schematic diagram for demonstrating an embodiment of the terminal position candidate information determination process in a terminal position determination unit. It is a schematic diagram which shows the other embodiment of the terminal position estimation system including the terminal position estimation apparatus by this invention. It is a schematic diagram which showed the Example of the peripheral model generation process in the peripheral model generation part schematically. It is a flowchart which shows one embodiment of the construction / update process of the reference non-animal body model in the terminal position estimation method according to this invention. It is a flowchart which shows one embodiment of the terminal position estimation process using the reference non-animal body model in the terminal position estimation method by this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a schematic diagram showing an embodiment of a terminal position estimation system including a terminal position estimation device according to the present invention.

The terminal position estimation system of the present embodiment shown in FIG. 1 is
(A) LiDAR3 installed in an autonomous mobile robot as a terminal equipped with a communication function that moves within a predetermined area.
(B) Includes a terminal position estimation device 1 communicatively connected to LiDAR 3 via wireless (wireless LAN, short-range wireless communication, wireless operator access network, etc.) or wired (cable, etc.).

Of these, LiDAR3 in (a) is a distance measuring sensor that emits a laser pulse outside the visible spectrum, measures the time until the pulse is reflected by the target and returns, and measures the distance to the target. It is a LiDAR (Light Detection And Ranging) device as.

Here, the measurement target of LiDAR3 may exist in the vicinity of LiDAR3 in this area.
(A) People, animals such as pets, vehicles such as carts and automobiles, subway vehicles, and moving or movable "movable objects" such as trolleys and chairs, and (b) areas such as escalator. Normally, it does not move or moves from the installation location, such as internal equipment, area facilities such as stores, installations in the area such as flower beds, vases and signage, and structures in the area such as ceilings, walls, pillars and floors. "Animal body" that cannot be obtained
It has become.

Specifically, LiDAR3 rotates a head equipped with a sensor and outputs point data including reflection intensity (or a three-dimensional distance calculated from it) at each angular position (coordinates) around the entire circumference (0 to 359.9 °). It is a device that outputs one frame (one file) for each lap. In the vertical direction, the head radiates, for example, 10 to 20 lasers in the range of plus or minus 10 ° to 20 °, measures each reflected wave, and acquires point data with a predetermined angular resolution. ..

On the other hand, in the terminal position estimation device 1 of the above (b), first, "information related to an object" existing around the terminal generated by LiDAR3 installed in the terminal (autonomous mobile robot in FIG. 1), specifically. The point cloud (point cloud) information, which is a set of the above point data, is acquired. Next, by analyzing the acquired point cloud, the position of the terminal (autonomous mobile robot) in the area is estimated, and the estimated position information is notified to the terminal.

More specifically, the terminal position estimation device 1 is characterized by its characteristics.
(A) Peripheral point to acquire "peripheral object information" which is "information related to an object" existing around the terminal (autonomous mobile robot) and is an observation result at the position of the terminal, and a peripheral point cloud in this embodiment. Group acquisition unit (peripheral object information acquisition unit) 121 and
(B) Peripheral point cloud immobility determination unit (peripheral object) that determines whether the target point cloud, which is information related to individual objects included in the peripheral point cloud (peripheral object information), is information related to "movable objects". Information judgment unit) 123 and
(C) Of the information contained in the peripheral point cloud (peripheral object information), the target point cloud determined to be information related to the "movable object" is subjected to removal processing, invalidation processing, or information amount reduction processing (book). In the embodiment, the peripheral model generation unit 124 that generates the "peripheral non-animal body model" which is the information related to the "non-animal body" existing in the periphery by performing the removal process).
(D) The "non-animal body" in the "reference non-animal body model" which is a model generated in advance and is information related to the "non-animal body" existing in this area, and the "non-animal body" included in the "peripheral non-animal body model". It has a terminal position determining unit 125 that determines information relating to the position of the terminal (autonomous mobile robot) in this area based on the collation result with the above.

In this way, the terminal position estimation device 1 generates a "peripheral non-animal body model" which is information related to the "non-animal body" existing in the periphery based on the peripheral point cloud which is the measurement result of the terminal sensor (LiDAR3). , The information of the terminal position is estimated by collating this with the "reference object model". That is, the terminal position estimation process can be performed with the information that the terminal can acquire by itself without depending on the external positioning system such as GPS.

Therefore, according to the terminal position estimation device 1, of course, it is possible to estimate the terminal position outdoors and provide a navigation service such as a car or a cart, but it is difficult to perform accurate GPS positioning indoors, for example. It is also possible to provide terminal position grasping / notification services indoors such as inside huge shopping centers and factories, and even in underground areas such as underground passages / squares and basements. Further, it is also possible to apply the terminal position estimation method in the terminal position estimation device 1 to the user terminal position estimation process in the AR (Augmented Reality) providing service.

Furthermore, since the information related to "movable objects" is removed (or invalidated or reduced) in both models used, the terminal position estimation suppresses the influence of the presence of moving objects in the peripheral point cloud (peripheral object information). It is possible to carry out the processing. Actually, conventionally, when the image information obtained by the sensor of the terminal includes a moving object such as a pedestrian or a car, the position estimation error of the terminal is increased or an error occurs in the estimation process. Has become a big problem. On the other hand, since the terminal position estimation device 1 suppresses the influence of the "movable object" that can move as described above, it is possible to avoid an error in the position estimation process and realize higher position estimation accuracy. It will be possible.

Furthermore, since the terminal position estimation device 1 estimates the terminal position by the alignment of the non-animal body, the alignment becomes easier as compared with the position estimation method by the alignment of the background, which is caused by the terminal sensor. It is possible to obtain more robust estimation results for error factors.

As a matter of course, the destination where the terminal position estimation device 1 according to the present invention acquires the point cloud information is not limited to the LiDAR device. For example, it is possible to acquire and analyze point cloud information from a radar (Radio detecting and ranging) device or a scanning sonar (Sound navigation and ranging) device.

Further, as a matter of course, the terminal equipped with such a distance measuring sensor device is not limited to the autonomous mobile robot. For example, mobile / wearable terminals owned by pedestrians / runners, various automobiles, flying drones, various transportation facilities, etc., if they have communication functions and need to know their position, various things. However, it may correspond to the terminal as the position estimation target according to the present invention.

Further, the "information related to the object" according to the present invention is not limited to the point cloud information. For example, it may be (RGB) image information generated by a camera (image sensor) mounted on the terminal, or distance image information generated from a depth value acquired by a depth camera mounted on the terminal. You can also do it. An embodiment of terminal position estimation using these image information will be described in detail later with reference to FIG.

Incidentally, in the present embodiment using the point cloud information, the terminal position is estimated using the three-dimensional "reference non-animal body model" for the "peripheral object information" which is the three-dimensional point cloud information. On the other hand, in the embodiment using the image information described in detail later, the terminal position is estimated using the three-dimensional "reference non-animal body model" for the "peripheral object information" which is the two-dimensional image information. Become.

[One Embodiment of the terminal position estimation device]
Similarly, according to the functional block diagram of FIG. 1, the terminal position estimation device 1 of the present embodiment has a communication interface 101 capable of receiving measurement information from LiDAR 3, a point cloud storage unit 102, a position information storage unit 103, and a processor. -Has a memory.

Here, the processor memory stores one embodiment of the terminal position estimation program according to the present invention, and also has a computer function, and the terminal position estimation is performed by executing the terminal position estimation program. Carry out the process. For this reason, the terminal position estimation device 1 may be a terminal position estimation dedicated device or a unit, but is equipped with a terminal position estimation program according to the present invention, for example, a personal computer (PC), a notebook type or a tablet type computer. May be. Further, it is also possible to use the terminal itself (the target of position estimation) having improved calculation processing capacity, for example, a communicable robot, a smartphone, or the like.

Further, the processor memory includes a reference model management unit 111, a peripheral point group acquisition unit 121, a peripheral point group division unit 122, a peripheral point group immobility determination unit 123, a peripheral model generation unit 124, and a terminal position determination. It has a unit 125, a communication control unit 131, and an application program unit (AP) 132. It should be noted that these functional components can be regarded as the functions of the terminal position estimation program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the terminal position estimation device 1 in FIG. 1 with arrows is also understood as an embodiment of the terminal position estimation method according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the communication control unit 131 acquires a signal including a point cloud for one head circumference (360 °) in one frame from LiDAR3 via the communication interface 101, and points this point cloud. It is output to the reference model management unit 111 and the peripheral point cloud information acquisition unit 121 while appropriately buffering the cloud storage unit 102. At this time, it is also preferable that the point cloud is filtered by a known method in order to remove noise and sparsely existing outliers.

The reference model management unit 111
(A) A point cloud (observation result information) at each of a plurality of positions in a predetermined area where the terminal position should be estimated, and this area is based on the point cloud related to an object existing around the position. Generates a point cloud in the area (object information in the area), which is information related to the objects existing in the area.
(B) Of the information contained in the point cloud in this area, the target point cloud determined to be information related to a movable object is "referenced" by the inanimate object information in the area generated by performing the removal process. Generate or update the "animal body model".

Here, in the above process (a), the point cloud is generated for each frame of the input point cloud (for example, the terminal is sequentially generated at each position in the area by moving all over the area). , Based on the distance between points (point data), it is divided into target point clouds, which are a set of points (point data) adjacent to each other. Here, the entire divided target point cloud becomes the object information in the area. At this time, points in the point cloud may be grouped by applying a predetermined distance condition. It is also preferable that the Euclidean distance is adopted as the distance between the points.

More specifically, as the division process, first, a kd tree (kd-tree, k-dimensional tree) having a cluster whose list is empty is generated, and a set of points that satisfy a predetermined distance condition and are close to each other is obtained. It may be assigned to the same cluster, the point cloud may be classified into clusters, and the set of points belonging to each cluster may be each divided target point cloud. Here, the kd tree is a kind of known spatially divided data structure for classifying points in the k-dimensional Euclidean space.

Further, in the above process (b), for example, PointNet, which is a known technology using a deep learning classifier for identifying a point cloud, is used, and each frame of the point cloud is an object of the target point cloud included in the frame. Determine the type. By roughening the identification target in the form of an object type rather than the object itself in this way, the amount of calculation in this discrimination process can be reduced. Here, by setting in advance whether the type is a movable object or an inanimate object for each object type, it is possible to determine whether or not the object included in the frame is a movable object. For example, a person, a car and a chair may be set as movable objects. In addition, the non-animal body can be another object that is not set as a movable object. Regarding PointNet, for example, non-patent documents: Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, In Proc. Computer Vision and Pattern Recognition. (CVPR), 2017.

Further, in the above process (b), by removing the target point cloud determined to be a movable object, the inanimate object information in the area, which is a set of the target point clouds of the inanimate object, is generated. In this information, a label of the non-animal body type is given to each target point cloud related to the remaining non-animal body, and a list of the non-animal body types existing in the area is also generated.

Next, a three-dimensional reference non-animal body model is generated from the non-animal body information in this area by using, for example, Kinect fusion, which is a known three-dimensional model construction technique. It is also preferable that this model is updated for each frame based on the in-area inanimate information generated for each frame of the input point cloud. Regarding Kinectfusion, for example, non-patent literature: T. Whelan, M. Kaess, M. Fallon, H. Johannsson, John. J. Leonard and J. McDonald, “Kintinuous: Spatially Extended KinectFusion”, InRSS RGB-D Workshop , 2012.

In this way, by removing the traces of the movable object from the reference non-animal body model in advance, the point cloud is caused by the movable object changing its position in the terminal position estimation using this reference non-animal body model. It is possible to avoid the failure of the collation process. Further, by removing the trace of the possible object, the amount of data is reduced, and the amount of collation processing calculation can be reduced.

Similarly, in the functional block diagram of FIG. 1, the peripheral point cloud acquisition unit 121 is a point cloud related to an object existing around the terminal (autonomous mobile robot equipped with LiDAR3), and performs terminal position estimation processing with the point cloud. Enter the point cloud as the peripheral object information that will be different. For example, it is possible to input the time-series frame of the point cloud every moment and estimate the current position of the terminal continuously in almost real time.

The peripheral point cloud classification unit 122 converts the peripheral point cloud as peripheral object information acquired by the peripheral point cloud acquisition unit 121 into a target point cloud which is a set of point data adjacent to each other based on the distance between the point data. Divide. The classification process here can be carried out in the same manner as the classification process carried out by the reference model management unit 111 described above.

The peripheral point cloud immobility determination unit 123 determines whether or not each of the divided target point clouds is information related to a movable object. The determination process here can also be performed in the same manner as the determination process performed by the reference model management unit 111 described above.

The peripheral model generation unit 124 removes the target point cloud determined to be information related to a possible object from the peripheral point cloud as peripheral object information, and the peripheral non-animal body model which is information related to the non-animal body existing in the periphery. To generate. At this time, a label for each of the target point clouds related to the remaining non-animal bodies is given, and a list of non-animal bodies existing in the peripheral non-animal body model is also generated. Such a model generation process can also be performed in the same manner as the model generation process performed by the reference model management unit 111 described above.

By the way, by collating the peripheral inanimate model with the movable object removed as described above with the reference inanimate model with the movable object removed, the error caused by the moving object changing its position It is possible to suppress the occurrence and secure higher terminal position estimation accuracy.

FIG. 2 is a schematic diagram schematically showing an embodiment of the peripheral model generation process in the peripheral model generation unit 124.

As shown in FIG. 2A, in this embodiment, the area where the terminal position should be estimated is a predetermined range in the shopping center, and the point cloud as the peripheral object information initially acquired from LiDAR3 is a movable object. Includes target point clouds for multiple people. On the other hand, as shown in FIG. 2B, the peripheral model generation unit 124 removes the target point cloud determined to be a “person” (preliminarily set to be a movable object) and within the area. Generate inanimate body information.

Next, the peripheral model generation unit 124 uses the determination result by the determination device in the peripheral point cloud immobility determination unit 123 for each of the target point clouds related to the inanimate object included in the inanimate object information in the area, or Using a classifier for identifying an animal body, a label for the corresponding animal body type such as a ceiling, a store A, a plant pot, an escalator, a floor, etc. is attached.

Similarly, in the functional block diagram of FIG. 1, the terminal positioning unit 125 is divided into an animal body in the reference animal body model generated or updated by the reference model management unit 111 and a peripheral animal body model generated by the peripheral model generation unit 124. Based on the collation result with the included non-animal body, the position information in the area of the terminal (autonomous mobile robot equipped with LiDAR3) is determined.

Here, it is also possible to determine the terminal position while collating the three-dimensional peripheral inanimate model with the entire three-dimensional reference inanimate model. However, since it is a collation process between three-dimensional models, there is a concern that the amount of processing calculation will increase and the processing time will also increase. Therefore, in the present embodiment, after determining a plurality of candidate regions where the terminal position can exist in the area in advance, the collation process with the candidate region corresponding portion in the reference non-animal body model is performed to estimate the terminal position. This makes it possible to reduce the amount of processing calculation and shorten the processing time.

Specifically, the candidate position generation unit 125a of the terminal position determination unit 125 has a plurality of terminal position candidate information in the area of the terminal (autonomous mobile robot equipped with LiDAR3) as information related to the position of the terminal. Determine the candidate area. For example, the area is divided into a plurality of areas in advance, and a list of non-animal bodies existing in or observable from the area is determined for each area. Here, adjacent areas may have portions that overlap each other. Next, the list of non-animal bodies in each area is collated with the list of non-animal bodies included in the peripheral non-animal body model, and the candidate area is an area having a list in which the contents of both lists match or both lists satisfy a predetermined condition. Can be.

At this time, as the predetermined condition, for example, it is possible to set a condition that the presence or absence of a predetermined non-animal body (for example, an escalator) always matches between the two lists. Furthermore, in the collation of both lists, certain non-animal objects, such as ceilings and floors, are usually found in any area, so they are not included in the above list or are not subject to collation (whether or not they are present between the lists). (Do not compare) is also preferable.

3 and 4 are schematic views for explaining an embodiment of the terminal position candidate information determination process in the terminal position determination unit 125.

FIG. 3 shows a floor map of an area where the terminal position should be estimated, and it can be seen that the area is a predetermined range in the shopping center. Here, a case where the list of non-animal bodies included in the peripheral non-animal body model includes an escalator, a store, and a flower pot will be described. In this case, two candidate areas 1 and 2 are determined as terminal position candidate information having a list that matches this list.

Next, the terminal position determination unit 125 together with the non-animal body existing at the position corresponding to each of the terminal position candidate information (candidate regions 1 and 2 in the embodiment of FIG. 3) determined as described above in the reference non-animal body model. , The terminal position candidate information including the terminal position is selected based on the result of the "collation process" with the non-animal body in the peripheral non-animal body model. Specifically, in this embodiment, as shown in FIG. 4, the candidate region 1 is selected.

Here, an embodiment of the above-mentioned "verification process" will be described. First, as a premise, each of the target point clouds related to the non-animal body obtained by analyzing the point cloud acquired from LiDAR3 has a determined distance from the target point cloud related to other adjacent non-animal bodies. And. Since this distance is a set of point data whose position in the three-dimensional space is determined by the target point cloud itself, it is also determined by, for example, calculating the position of the center of gravity and calculating the distance between the centers of gravity. It is possible. Furthermore, each of the target point clouds related to the non-animal body is given a label as the object type information of the non-animal body.

Returning to the functional block diagram of FIG. 1, next, the graph matching processing unit 125b of the terminal position determining unit 125 may be used.
(A) From each "terminal position candidate information" part of the reference non-animal body model, a "reference model graph" which is a graph of the reference non-animal body model with each non-animal body as a node and the distance between the non-animal bodies as an edge is generated. ,
(B) From the peripheral non-animal body model, a "peripheral model graph" which is a graph of the peripheral non-animal body model in which each non-animal body is used as a node and the distance between the non-animal bodies is used as an edge is generated.
(C) Matching processing is performed between the "reference model graph" and the "peripheral model graph", and the terminal position candidate information (candidate areas 1 and 2 in FIG. 3) that can include the terminal position is selected.

Specifically, the reference model graph G _r (V _r , E _r ) and the peripheral model graph G _t (V _t , E _t ) are defined. Here, V _r and V _t are node groups corresponding to the "terminal position candidate information" part of the reference non-animal model and the peripheral non-animal model, each of which is given object type information. In addition, _Er and _Et are edge groups in the "terminal position candidate information" part of the reference non-animal body model and the peripheral non-animal body model, respectively, each given a distance between the non-animal bodies.

By the way, since the region of G _r (V _r , E _r ) is larger than the region of G _t (V _t , E _t ), sub-matching processing is performed for collation of both graphs below. Specifically, V _r ⊇ V _t and E _r ⊇ E _t , and the following equation (1) f (f (u), f (v)) ∈ E _r
If is satisfied, then the following equation (2) (u, v) ∈ E _t
Function f: V _t → V _r such that the relationship of
If is present, the peripheral model graph G _t (V _t , _Et ) is determined to be a subset of the reference model graph G _r (V _r , E _r ), and this reference model graph G _r (V _r , E r) is determined. The terminal position candidate information (candidate area) related to _Er ) is selected as being capable of including the terminal position (candidate area 1 is selected in FIG. 4).

The "collation process" of the non-animal body in both models is not limited to the above-mentioned method. For example, it is also possible to directly map the peripheral object model on the reference object model (each "terminal position candidate information" part). However, in reality, there is distortion in the model due to the generation / update of the model from the point cloud, and the position of the inanimate object is often drifting. Therefore, for example, direct mapping processing may make proper collation difficult. On the other hand, as described above, by performing the graph matching process, it is possible to suppress the influence of the object drift due to such distortion and perform more appropriate collation.

As described above, at this stage, the terminal position determining unit 125 determines a rough position range in which the terminal exists. Next, in the present embodiment, the ICP processing unit 125c of the terminal position determination unit 125 determines the detailed position of the terminal and the orientation of the terminal (LiDAR3 mounted on the terminal).

Specifically, the ICP processing unit 125c
(A) A target point cloud (hereinafter abbreviated as a reference target point cloud) related to an inanimate object existing at a position related to the selected terminal position candidate information (candidate area 1 in FIG. 4).
(B) Point cloud parallel movement amount and point cloud that maximizes the degree of superposition with the target point cloud (hereinafter abbreviated as peripheral target point cloud) related to the non-animal body corresponding to this non-animal body in the peripheral non-animal body model. The position and orientation of the terminal are determined based on the amount of rotation. Incidentally, the correspondence relationship (matching relationship) between the non-animal body existing in the selected terminal position candidate information (candidate area) and the non-animal body of the peripheral non-animal body model is determined by the above-mentioned collation process.

Here, it is also preferable to carry out such a terminal position / orientation determination process using an ICP (Iterative Closest Point) algorithm. The ICP algorithm is a known method for aligning between point clouds.
(A) For each point included in one point cloud, the closest point in the other point cloud is determined, and the points are set as the nearest points.
(B) Find a rigid body transformation that minimizes the distance between the determined nearest points.
By repeatedly carrying out the above (a) and (b), the conversion that completes the alignment of both point clouds is determined.

Specifically, the ICP processing unit 125c first specifically displays the reference target point cloud group.
(3) P _r = [p _r ¹ , p _r ² , ・ ・ ・, p _r ⁿ ]
And, the peripheral target point cloud group,
(4) P _t = [p _t ¹ , p _t ² , ・ ・ ・, p _t ^m ]
Further, R is a rotation transformation (operator), t is a translational transformation (operator), and the mapping error is the following equation (5) E (R, t) = Σ _{i = 1} ⁿ Σ _{j = 1} ^m . w _{i, j}・ || p _r ⁱ － (Rp _t ^j ＋ t) || ²
Is defined by. Here, Σ _{i = 1} ⁿ and Σ _{j = 1} ^m are the summation from i = 1 to n and the sum from j = 1 to m, respectively. In addition, w _{i, j} becomes 1 (w _{i, j} = 1) when p _r ⁱ and p _t ^j are the same non-animal body type, and 0 (w _{i, j} = 0) in other cases. Is a coefficient.

On this basis, the ICP processing unit 125c calculates the rotation conversion R and the translational conversion t that minimize the mapping error in the above equation (5), thereby, as a result, the target before conversion with respect to the reference target point cloud group _Pr . The deviation of the position and orientation of the point cloud group P _t is found. As a result, the suitable position of the peripheral inanimate model in the three-dimensional reference inanimate model can be determined, and finally, the position in the area of the terminal (LiDAR3) at the time of the peripheral inanimate model and the position in the area of the terminal (LiDAR3). It is possible to determine the information related to the orientation.

The terminal position determination unit 125 stores the information related to the position and orientation of the terminal determined as described above in the position information storage unit 103, and appropriately via the communication control unit 131 and the communication interface 101, the terminal (the terminal (the terminal) It transmits to an autonomous mobile robot equipped with LiDAR3). The terminal (autonomous mobile robot equipped with LiDAR3) that has acquired the information related to this position and orientation can know its own position and orientation in the area based on the information.

Further, the terminal position and orientation information determined by the terminal position determining unit 125 is taken into an application program (AP) 132 mounted on the device 1 (for example, for autonomous robot control), and its output (control signal) is output. , May be provided to a terminal (autonomous mobile robot equipped with LiDAR3) via the communication control unit 131 and the communication interface 101. Further, as a modification mode, the terminal position and orientation information determined by the terminal position determination unit 125 can be transmitted to another information processing device other than the terminal and used for information processing in the information processing device. be.

[Other Embodiments of Terminal Position Estimator]
FIG. 5 is a schematic diagram showing another embodiment of the terminal position estimation system including the terminal position estimation device according to the present invention.

The terminal position estimation system of the present embodiment shown in FIG. 5 is
(A) A camera 41 mounted on a smartphone 4 owned by a user who moves within a predetermined area, and
(B) Includes a terminal position estimation device 2 that is communication-connected to the smartphone 4 via wireless (wireless LAN, short-range wireless communication, wireless operator access network, etc.).

Here, the shooting target of the camera 41 of (a) is the same as the measurement target of LiDAR3 described with reference to FIG. 1 in the present embodiment, and exists in the vicinity of the user (camera 41) in the area. It is a "movable object" such as a person to obtain, and an "inanimate body" such as equipment in the area, installations and facilities, and structures in the area such as ceilings, walls, pillars, and floors.

On the other hand, the terminal position estimation device 2 of the above (b) first generates "information about an object" existing around the smartphone 4 generated by the camera 41 of the smartphone 4, specifically, a captured image. Acquire information (moving image information). Next, by analyzing the acquired image information, the position of the smartphone 4 in the area is estimated, and the estimated position information is notified to the terminal.

More specifically, the terminal position estimation device 2 is characterized by its characteristics.
(A) Peripheral image acquisition unit (periphery) that acquires "peripheral object information" that is "information related to an object" existing around the smartphone 4 and is an observation result at the position of the terminal, and peripheral image information in this embodiment. Object information acquisition unit) 221 and
(B) Peripheral image immobility determination unit (peripheral object information determination) for determining whether or not the image partial information related to each object included in the peripheral image information (peripheral object information) is the image partial information related to the "movable object". Part) 223 and
(C) Of the information contained in the peripheral image information (peripheral object information), the image partial information determined to be the image partial information related to the "movable object" is invalidated or the amount of information is reduced (this implementation). Peripheral model generation unit 224 that generates "peripheral non-animal body model" which is image information related to "non-animal body" existing in the periphery by applying mask processing with one color in the form.
(D) From the "reference non-animal body model" which is a pre-generated model and is the image information related to the "non-animal body" existing in this area, or from the part which is a candidate for the terminal position in this model, each "immobility". The image partial information related to the "object" is extracted, and the terminal is based on the degree of image superposition between the image partial information and the image partial information related to each inanimate object included in the "peripheral inanimate model". It has a terminal position determining unit 225 for determining a position.

In this way, the terminal position estimation device 2 also performs the terminal position estimation process with the information that the terminal can acquire by itself without depending on the external positioning system such as GPS, as in the terminal position estimation device 1 described above. Can be done. Therefore, according to the terminal position estimation device 2, of course, it is possible to estimate the terminal position outdoors and provide a navigation service such as a car or a cart, but it is difficult to perform accurate GPS positioning indoors. For example, it is possible to provide a terminal position grasping / notification service indoors such as the inside of a huge shopping center or a factory, or even in an underground area such as an underground passage / plaza or a basement. Further, it is also possible to apply the terminal position estimation method in the terminal position estimation device 2 to the user terminal position estimation process in the AR providing service.

Furthermore, since the information related to "movable objects" is invalidated or reduced in both models used, the terminal position estimation process that suppresses the influence of the presence of moving objects in the peripheral image information (peripheral object information) should be performed. Is possible. In fact, conventionally, when the image information obtained by the camera of the terminal includes a moving object such as a pedestrian or a car, the position estimation error of the terminal may increase or an error may occur in the estimation process. It has become a big problem. On the other hand, since the terminal position estimation device 2 suppresses the influence of the "movable object" that can move as described above, it is possible to avoid an error in the position estimation process and realize higher position estimation accuracy. It will be possible.

Incidentally, the peripheral image information handled in the present embodiment is not limited to the camera image information, and may be, for example, the distance image information generated by the depth camera mounted on the terminal.

According to the functional block diagram of FIG. 5, the terminal position estimation device 2 of the present embodiment has a communication interface 201 capable of receiving image information generated by a camera 41 from a smartphone 4, an image data storage unit 202, and a position. It has an information storage unit 203 and a processor memory.

Here, the processor memory stores one embodiment of the terminal position estimation program according to the present invention, and also has a computer function, and the terminal position estimation is performed by executing the terminal position estimation program. Carry out the process. For this reason, the terminal position estimation device 2 may be a terminal position estimation dedicated device or a unit, but is equipped with a terminal position estimation program according to the present invention, for example, a personal computer (PC), a notebook type or a tablet type computer. May be. Further, it is also possible to use the terminal itself (the target of position estimation) with improved calculation processing capacity, for example, a smartphone, a communicable robot, or the like.

Further, the processor memory includes a reference model management unit 211, a peripheral image acquisition unit 221, a peripheral image immobility determination unit 223, a peripheral model generation unit 224, a terminal position determination unit 225, and a communication control unit 231. It has an application program unit (AP) 232. It should be noted that these functional components can be regarded as the functions of the terminal position estimation program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the terminal position estimation device 2 in FIG. 5 with arrows is also understood as an embodiment of the terminal position estimation method according to the present invention.

Here, among the above-mentioned functional components, those that perform specific processing because they handle image information will be described in comparison with the functional components of the terminal position estimation device 1.

The reference model management unit 211
(A) This area is based on image information (observation result information) at each of a plurality of positions in a predetermined area where the terminal position should be estimated and is related to an object existing around the position. Generates in-area image partial information (in-area object information), which is information related to objects existing in
(B) Of the information included in the image partial information in the area, the image partial information determined to be information related to a movable object is subjected to mask processing to generate "in-area non-animal body information". Generate or update a "reference non-animal model".

Here, in the above processing (a), for example, using a known image identification technique using a high-speed R-CNN (Faster Regions with Convolutional Neural Networks) classifier, each frame of image information (moving image information) is used. In addition, the type of the object related to the image partial information included in the frame can be determined. Regarding high-speed R-CNN, for example, non-patent documents: Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, IEEE Transactions on Pattern Analysis and Machine It is described in Intelligence, Vol.39, Issue.6, p1137-1149, 2017.

Further, in the above process (b), whether the type is a movable object or an inanimate object is set in advance for each object type, and the image determined to be the image partial information related to the object type which is a movable object. A mask process for setting a black rectangular area including the image portion can be performed on the partial information. Further, in the in-area non-animal body information after the mask processing, a label for each non-animal body type is given to each image partial information related to the remaining non-animal body, and a list of non-animal bodies existing in the area is also generated. , This produces a three-dimensional reference inanimate model.

In this way, by removing the trace of the movable object from the reference non-animal body model in advance, the image collation is performed by changing the position of the movable object in the terminal position estimation using the reference non-animal body model. It is possible to avoid processing failures. Further, by removing the trace of the movable object, the amount of data is reduced, and the amount of collation processing calculation can be smaller.

Similarly, in the functional block diagram of FIG. 5, the peripheral image acquisition unit 221 is image information related to an object existing in the vicinity of the smartphone 4 (camera 41), and the peripheral object is used for estimating the terminal position. Enter peripheral image information as information. For example, it is possible to input the time-series frame of the image information every moment and estimate the current position of the terminal continuously in almost real time.

The peripheral image immobility determination unit 223 identifies the image partial information related to the object in the peripheral image information, and determines whether the object type of the identified image partial information is the object type related to the preset movable object. do. It should be noted that such identification / determination processing can be performed in the same manner as the identification / determination processing performed by the reference model management unit 211 described above.

Peripheral model generation unit 224 performs mask processing on the image partial information determined to be information related to a possible object, and generates a peripheral non-animal body model which is information related to non-animal bodies existing in the periphery. Further, at this time, a label of the non-animal body type is attached to each image partial information related to the remaining non-animal body, and a list of the non-animal body types existing in the peripheral non-animal body model is also generated. Such a model generation process can also be performed in the same manner as the model generation process performed by the reference model management unit 211 described above.

By the way, it is caused by the fact that the movable object changes its position by collating the peripheral inanimate model in which the movable object is invalidated and the reference inanimate model in which the movable object is invalidated as described above. It is possible to suppress the occurrence of errors and secure higher terminal position estimation accuracy.

FIG. 6 is a schematic diagram schematically showing an embodiment of the peripheral model generation process in the peripheral model generation unit 224.

As shown in FIG. 6A, in this embodiment, the area where the terminal position should be estimated is a predetermined range in the shopping center, and the image information as the peripheral object information initially acquired from the camera 41 is movable. It contains partial image information relating to a plurality of people who are objects. On the other hand, as shown in FIG. 2B, the peripheral model generation unit 224 performs mask processing on the image portion information identified as a "person" (preliminarily set to be a movable object). Then, the inanimate body information in the area is generated.

Next, the peripheral model generation unit 224 uses the object type result by the discriminator in the peripheral point group immobility determination unit 223 for each of the image partial information related to the animal body included in the in-area immovable body information. Labels for applicable object types such as ceilings, store A, escalator, floors, etc. are given.

Similarly, in the functional block diagram of FIG. 1, the terminal positioning unit 225 is divided into an animal body in the reference animal body model generated or updated by the reference model management unit 211 and a peripheral animal body model generated by the peripheral model generation unit 124. Based on the collation result with the included non-animal body, the position information in the area of the smartphone 4 (the user who possesses it) is determined.

Specifically, the terminal positioning unit 225 uses a known matching technique, for example, a preprint for the paper accepted for publication in ICCV 2011, “Fast Image-Based Localization using Direct 2D-to-3D Matching”, [ online], [Search on August 23, 2018], 2D-3D matching technology described in the Internet <URL: https://www.graphics.rwth-aachen.de/media/papers/sattler_iccv11_preprint_011.pdf> Is used to collate the peripheral inanimate model against the reference inanimate model.

As a result, the suitable position of the peripheral inanimate model in the reference inanimate model can be determined, and finally, the position in the area of the smartphone 4 (the user who possesses) at the time of the peripheral inanimate model. And it becomes possible to determine the information related to the direction.

By the way, similar to the collation process between the models related to the point cloud described above, in the reference non-animal body model, after predetermining a plurality of candidate regions in which the terminal position can exist, the part corresponding to the candidate region in the reference non-animal body model is used. It is also possible to perform collation processing and estimate the terminal position. This makes it possible to reduce the amount of processing calculation and shorten the processing time.

Specifically, the candidate position generation unit 225a of the terminal position determination unit 225 determines at least one candidate area which is a plurality of terminal position candidate information in the area. For example, the area is divided into a plurality of areas in advance, and a list of non-animal bodies existing in or observable from the area is determined for each area. Here, adjacent areas may have portions that overlap each other. Next, the list of non-animal bodies in each area is collated with the list of non-animal bodies included in the peripheral non-animal body model, and the candidate area is an area having a list in which the contents of both lists match or both lists satisfy a predetermined condition. Can be.

At this time, as the predetermined condition, for example, it is possible to set a condition that the presence or absence of a predetermined non-animal body (for example, an escalator) always matches between the two lists. Furthermore, in the collation of both lists, certain non-animal objects, such as ceilings and floors, are found in any area, so they are not included in the above list or are not subject to collation (comparison between the lists). (No) is also preferable.

Next, the candidate position generation unit 225a of the terminal position determination unit 225 determines the collation result between the animal body in the candidate region of the reference animal body model and the animal body included in the peripheral animal body model generated by the peripheral model generation unit 124. Based on this, the position information in the area of the smartphone 4 (the user who possesses the smartphone 4) is determined. As a result, it is not necessary to perform collation processing with all the non-animal bodies included in the reference non-animal body model, and as a result, it is possible to reduce the collation processing calculation amount and shorten the collation processing time.

[Terminal position estimation method]
FIG. 7 is a flowchart schematically showing an embodiment of a reference non-animal body model construction / update process in the terminal position estimation method according to the present invention. Here, steps S101 to S106 described below are processes for constructing a reference inanimate object model for the observation target area and updating each acquired frame.

(S101) First, time-series point cloud data or time-series image data is acquired.
(S102) In each frame of the acquired data, an object detection process for detecting an object is performed.
(S103) Among the detected objects, the movable object that is set to be movable in advance is subjected to the deletion process in the case of the point cloud, while the mask process is performed in the case of the image data.

(S104) Based on the data after the deletion / mask processing, a list of inanimate objects existing in the observation target area is generated, or if it has already been generated, it is updated.
(S105) A reference non-animal body model, which is non-animal body information in the observation target area, is constructed, or is updated if it has already been constructed.
(S106) For the processing up to the relevant time point, it is determined whether or not one of the processing time, the number of processing frames, and the amount of processing data exceeds a predetermined threshold value set in advance. Here, if it is determined that the excess is exceeded, this process is terminated. On the other hand, if it is determined that the excess is not exceeded, the process from step S101 is repeated again.

FIG. 8 is a flowchart schematically showing an embodiment of the terminal position estimation process using the reference non-animal body model in the terminal position estimation method according to the present invention. Here, steps S201 to S210 described below correspond to one cycle from the acquisition of predetermined time-series data to the determination of the terminal position and the notification to the terminal.

(S201) Acquire time-series point cloud data or time-series image data.
(S202) In each frame of the acquired data, an object detection process for detecting an object is performed.
(S203) Among the detected objects, the movable object that is set to be movable in advance is subjected to the deletion process in the case of the point cloud, while the mask process is performed in the case of the image data.

(S204) Based on the data after the deletion / mask processing, a list of inanimate objects existing around the terminal is generated, or if it has already been generated, it is updated.
(S205) A reference non-animal body model, which is non-animal body information existing around the terminal, is constructed, or is updated if it has already been constructed.
(S206) For the processing up to the relevant time point, it is determined whether or not one of the processing time, the number of processing frames, and the amount of processing data exceeds a predetermined threshold value set in advance. Here, if it is determined that the excess is exceeded, the process proceeds to step S207. On the other hand, if it is determined that the excess is not exceeded, the process from step S201 is repeated again.

(S207) Based on the generated / updated list of non-animal bodies, a plurality of candidate regions that can correspond to the peripheral non-animal body model in the reference non-animal body model are determined.
(S208, S209) Collation processing between the non-animal bodies in the reference non-animal body model and the peripheral non-animal body model is performed, a candidate region matching the peripheral non-animal body model in the reference non-animal body model is selected, and further, the selected candidate region is selected. Determines the position and orientation of the terminal in.
(S210) Information on the position and orientation of the determined terminal is transmitted to the terminal.

Here, by repeating one cycle of steps S201 to S210 described above, the position and orientation information can be notified to the terminal each time, and by performing this cycle at a sufficiently high speed, it is almost real-time. Therefore, it is possible to notify the terminal of the current position and orientation.

As described in detail above, according to the present invention, based on the peripheral object information which is the measurement result of the sensor of the terminal, the peripheral inanimate model which is the information related to the inanimate existing in the periphery is generated, and the peripheral inanimate model is generated. Information on the terminal position is estimated by collating with the reference non-animal body model. That is, the terminal position estimation process can be performed with the information that the terminal can acquire by itself without depending on the external positioning system such as GPS.

Therefore, according to the terminal position estimation device 1, of course, it is possible to estimate the terminal position outdoors and provide a navigation service such as a car or a cart, but it is difficult to perform accurate GPS positioning indoors, for example. It is also possible to provide terminal position grasping / notification services indoors such as inside huge shopping centers and factories, and even in underground areas such as underground passages / squares and basements. Further, it is also possible to apply the terminal position estimation method in the terminal position estimation device 1 to the user terminal position estimation process in the AR providing service.

In the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily carried out by those skilled in the art. The above explanation is just an example and does not attempt to limit anything. The present invention is limited only to the scope of claims and their equivalents.

1, 2 Terminal position estimation device 101, 201 Communication interface 102 Point cloud storage unit 103, 203 Location information storage unit 111, 211 Reference model management unit 121 Peripheral point cloud acquisition unit 122 Peripheral point cloud division unit 123 Peripheral point cloud immobility judgment Unit 124, 224 Peripheral model generation unit 125, 225 Terminal positioning unit 131, 231 Communication control unit 132, 232 Application program unit (AP)
202 Image data storage unit 221 Peripheral image acquisition unit 223 Peripheral image immobility determination unit 3 LiDAR
4 smartphone 41 camera

Claims (9)

A terminal position estimation device that estimates the position of a terminal in a predetermined area.
Peripheral object information acquisition means for acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A peripheral object that uses a point group information classifier to determine whether the point group information, which is information related to an individual object included in the peripheral point group information, is point group information related to a preset movable object. Information judgment means and
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Peripheral model generation means to generate a model, and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model is generated with each non-animal body as a node and the distance between the non-animal bodies as an edge, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to the non -animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation device comprising a terminal position determining means for determining. The terminal position estimation according to claim 1 , wherein the terminal position determining means also determines the orientation of the terminal based on the point cloud translation amount and the point cloud rotation amount that maximize the degree of superimposition. Device. The terminal position estimation device further has peripheral object information classification means for classifying the peripheral point cloud information into point cloud information for each object, which is a set of point data adjacent to each other, based on the distance between the point data. ,
The peripheral model generation means is characterized in that the peripheral inanimate model is generated by removing the point cloud information determined to be the point cloud information related to the movable object from the point cloud information for each object. The terminal position estimation device according to claim 1 or 2 . A terminal position estimation device that estimates the position of a terminal in a predetermined area.
Peripheral image information acquisition means for acquiring peripheral image information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal .
Peripheral determination using an image information classifier to determine whether the image partial information, which is information related to an individual object included in the peripheral image information, is the image partial information related to a preset movable object. Object information determination means and
Of the information included in the peripheral image information, by performing masking processing on the image partial information determined to be the image partial information related to the movable object, the peripheral image information related to the non-animal body existing in the periphery is applied. Peripheral model generation means to generate an inanimate body model,
Image partial information related to individual non-animal bodies is obtained from a reference non-animal body model, which is a pre- generated model and image information related to non-animal bodies existing in the area, or from a part that is a candidate for a terminal position in the model. Based on the degree of image superimposition between the extracted image partial information and the image partial information related to each inanimate object included in the peripheral inanimate body model, the information relating to the position of the terminal in the area is obtained. With the terminal position determination means to determine
A terminal position estimation device characterized by having . The fourth aspect of claim 4 is characterized in that the peripheral image information is camera image information generated by a camera mounted on the terminal or distance image information generated by a depth camera mounted on the terminal. Terminal position estimation device. Based on the information of the observation result at each of the plurality of positions in the area and the information related to the objects existing in the vicinity of the position, the object information in the area which is the information related to the objects existing in the area is generated.
Of the information contained in the object information in the area, the inanimate body information in the area generated by removing the point group information determined to be the point group information related to the movable object, or in the area. Of the information included in the object information, the reference inanimate model is generated or updated by the in-area inanimate information generated by masking the image partial information determined to be the image partial information related to the movable object. The terminal position estimation device according to any one of claims 1 to 5, further comprising a reference model management means. Any of claims 1 to 6, wherein the point group information related to the non- animal body or the image partial information related to the non-animal body includes the object type information in the non-animal body and the information related to the distance between the non-animal bodies. The terminal position estimation device according to item 1 . A program that activates a computer mounted on a device that estimates the position of a terminal in a predetermined area.
Peripheral object information acquisition means for acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A peripheral object that uses a point group information classifier to determine whether the point group information, which is information related to an individual object included in the peripheral point group information, is point group information related to a preset movable object. Information judgment means and
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Peripheral model generation means to generate a model, and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model is generated with each non-animal body as a node and the distance between the non-animal bodies as an edge, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to the non -animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation program characterized in that a computer functions as a terminal position determining means for determining. It is a terminal position estimation method in a computer mounted on a device that estimates the position of a terminal in a predetermined area.
A step of acquiring peripheral point cloud information which is information related to an object existing in the vicinity of the terminal and is an observation result at the position of the terminal.
A step of using a point cloud information classifier to determine whether the point cloud information, which is information related to an individual object included in the peripheral point cloud information, is point cloud information related to a preset movable object. ,
By removing the point cloud information determined to be the point cloud information related to the movable object from the information included in the peripheral point cloud information, the peripheral non-animal body which is the point cloud information related to the non-animal body existing in the vicinity. Steps to generate a model and
As the information relating to the position of the terminal in the area, the terminal position candidate information which is the information relating to the position candidate in the area of the terminal is determined, and the model generated in advance and existing in the area is immovable. From the reference non-animal body model which is the point group information related to the object, a graph related to the reference non-animal body model is generated with each non-animal body as a node and the distance between the non-animal bodies as an edge, and the graph related to the reference non-animal body model is generated. The matching process is performed between the graph and the graph related to the peripheral animal body model in which the individual animal bodies included in the peripheral animal body model are used as nodes and the distance between the animal bodies is used as an edge, and the terminal position candidate information is used. , The terminal position candidate information relating to the position of the terminal is selected, the point group information relating to the animal body existing at the position corresponding to the selected terminal position candidate information in the reference non-animal body model, and the peripheral non-animal body model . Based on the point group parallel movement amount and the point group rotation amount that maximize the degree of superposition with the point group information related to the non-animal body corresponding to the non -animal body, the information related to the position of the terminal in the area is obtained. A terminal position estimation method comprising: a step to determine.

Images