WO2024252507A1 - Information processing device, information processing system, location identification method, and non-transitory computer readable medium - Google Patents

Abstract

The purpose of the present invention is to provide an information processing device (10) for suppressing the degradation of estimation accuracy. An information processing device (10) as in the present disclosure is provided with: an estimation unit (11) that estimates at least one candidate location for the location at which captured was a first captured image captured from a mobile object, such estimation using the distance between an object of interest included in the first captured image and the location at which the first captured image was captured, as well as a location of the object of interest in reference data generated in advance as data indicating the location of an object in a predetermined space; and an identification unit (12) that identifies the location at which the first captured image was captured, by using the first captured image and an estimation image of when the predetermined space is imaged at the at least one candidate location.

Description

Information processing device, information processing system, location determination method, and non-transitory computer-readable medium

The present disclosure relates to an information processing device, an information processing system, a location identification method, and a non-transitory computer-readable medium.

Workers working in facilities such as substations need to know their own location accurately within the facility in order to perform their work safely and efficiently.

Patent Document 1 discloses the configuration of a vehicle position detection system that determines the position of the vehicle equipped with the on-board camera that captured the actual image by matching the actual image captured by the on-board camera with reference data that includes images of the scenery captured in advance from the vehicle. The vehicle position detection system of Patent Document 1 determines, based on the vehicle behavior, which of the images included in the reference data should be excluded from matching with the actual image.

JP 2011-215055 A

The locations where substations and other facilities are located are often deep in the mountains and do not include distinctive objects in the surrounding scenery. Furthermore, substation facilities contain many objects of the same shape. In such a situation, if position estimation as described in Patent Document 1 is performed, there is a problem that the estimation accuracy may deteriorate because there are no distinctive objects in the scenery within the substation.

In view of the above-mentioned problems, the objective of the present disclosure is to provide an information processing device, an information processing system, a position identification method, and a non-transitory computer-readable medium that suppress deterioration of estimation accuracy.

The information processing device according to the first aspect of the present disclosure includes an estimation unit that estimates at least one candidate position for the position at which the first photographed image was taken, using the distance between a target object included in a first photographed image taken from a moving body and the position at which the first photographed image was taken, and the position of the target object in reference data that is generated in advance as data indicating the position of an object in a specified space, and an identification unit that identifies the position at which the first photographed image was taken, using an estimated image when the specified space is photographed at the at least one candidate position and the first photographed image.

The information processing system according to the second aspect of the present disclosure includes an image capturing device that moves with a moving object and captures an image of the surroundings of the moving object to generate a first captured image, a first information processing device that generates reference data in advance as data indicating the position of an object in a specified space, and a second information processing device that estimates at least one candidate position for the position where the first captured image was captured using the distance between a target object included in the first captured image and the position where the first captured image was captured and the position of the target object in the reference data, and identifies the position where the first captured image was captured using an estimated image when the specified space is captured at the at least one candidate position and the first captured image.

The position identification method according to the third aspect of the present disclosure estimates at least one candidate position for the position at which the first photographed image was taken using the distance between a target object included in a first photographed image taken from a moving body and the position at which the first photographed image was taken, and the position of the target object in reference data that is generated in advance as data indicating the position of an object in a specified space, and identifies the position at which the first photographed image was taken using an estimated image of the specified space photographed at the at least one candidate position and the first photographed image.

The non-transitory computer-readable medium according to the fourth aspect of the present disclosure stores a program that causes a computer to estimate at least one candidate position for the position at which the first photographed image was taken, using the distance between a target object included in a first photographed image taken from a moving body and the position at which the first photographed image was taken, and the position of the target object in reference data that is generated in advance as data indicating the position of an object in a specified space, and to identify the position at which the first photographed image was taken, using an estimated image of the specified space photographed at the at least one candidate position and the first photographed image.

The present disclosure makes it possible to provide an information processing device, an information processing system, a location identification method, and a non-transitory computer-readable medium that suppress deterioration of estimation accuracy.

FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to the present disclosure. 1 is a diagram showing an example of a flow of a position identification process executed in an information processing device according to the present disclosure. 1 is a diagram showing an example of a flow of a position identification process executed in an information processing device according to the present disclosure. FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to the present disclosure. 1 is a diagram showing an example of a flow of a position identification process executed in an information processing device according to the present disclosure. 1 is a diagram showing an example of a flow of a position identification process executed in an information processing device according to the present disclosure. FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to the present disclosure. FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to the present disclosure.

(Embodiment 1)
Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. A configuration example of an information processing device 10 will be described with reference to FIG. 1. The information processing device 10 may be a computer device that operates when a processor executes a program stored in a memory. The information processing device 10 has an estimation unit 11 and an identification unit 12. The estimation unit 11 and the identification unit 12 may be software or a module that performs processing when a processor executes a program stored in a memory. Alternatively, the estimation unit 11 and the identification unit 12 may be hardware such as a circuit or a chip.

The estimation unit 11 estimates at least one candidate position for the position where the moving object captured the captured image. The estimation unit 11 estimates the candidate position using the distance between the target object included in the captured image and the position where the captured image was captured, and the position of the target object in reference data that is generated in advance as data indicating the position of an object in a specified space.

The moving body has a camera that can capture images of the surrounding environment while moving. The moving body may be a vehicle equipped with a camera. The vehicle may be a vehicle that can run autonomously. The vehicle may be a vehicle that runs by remote control. The vehicle may be a vehicle that runs by being directly operated by a driver. Or, the moving body may be a human being that moves while holding a camera.

When a moving body captures an image, it may mean that the image is captured by a camera mounted on the moving body. Alternatively, when a moving body captures an image, it may mean that the image is captured by a camera held by a human. The candidate positions and positions in the specified space may be positions in a so-called world coordinate system. Alternatively, the candidate positions and positions in the specified space may be positions in an arbitrarily defined unique coordinate system.

The target object may be an object that has a characteristic shape compared to other objects. Or, the target object may be an object that exists less in number than other objects in a specified space. The target object may be designated by a user who operates or manages the information processing device 10. Designated may be rephrased as selected, determined, etc. Or, the target object may be designated by a user who operates a computer device from a different computer device connected to the information processing device 10 via a network. Or, the target object may be an object that is identified by the information processing device 10 as an object that meets the criteria for a target object by the information processing device 10 analyzing a captured image. The criteria for a target object may be, for example, that the object has a characteristic shape or that the object exists less in number than other objects, etc.

The specified space may be a space through which a moving object moves. The specified space may be, for example, a facility such as a substation. The reference data may be three-dimensional data indicating the position or arrangement of an object in the specified space. The reference data may be generated using a photographed image taken within the specified space. Alternatively, the reference data may be three-dimensional data generated using design data or the like in the specified space. The reference data may be, for example, CAD (Computer Aided Design) data.

The candidate position may be at least one position identified as a position that is separated from the target object in a specified space by a distance between the target object included in the captured image and the position at which the captured image was captured. The candidate positions may be positions that are separated by a predetermined distance or more.

The identification unit 12 identifies the location where the captured image was taken by using an estimated image obtained when a specific space is captured at at least one candidate location and the captured image. The estimated image may be, for example, an image estimated as an image that is generated when an image is captured at a candidate location with a specific angle of view and in a specific direction from a specific height. The angle of view and height can be changed. Alternatively, the estimated image may be an image captured in advance at or near the candidate location.

The identification unit 12 using the estimated image and the captured image may mean comparing the estimated image and the captured image. Alternatively, the identification unit 12 using the estimated image and the captured image may mean analyzing the estimated image and the captured image. The identification unit 12 may, for example, identify a candidate position associated with an estimated image similar to the captured image as the position where the captured image was captured. The identification unit 12 identifying a position may mean the identification unit 12 selecting one candidate position from among multiple candidate positions. Alternatively, the identification unit 12 identifying a position may mean the identification unit 12 narrowing down the multiple candidate positions to one candidate position.

Next, the position identification method executed in the information processing device 10 will be described with reference to FIG. 2. First, the estimation unit 11 estimates at least one candidate position of the position where the captured image was taken by a moving object (S11). The estimation unit 11 estimates the candidate position using the distance between the target object included in the captured image and the position where the captured image was taken, and the position of the target object in reference data that is generated in advance as data indicating the position of an object in a specified space.

Next, the identification unit 12 identifies the position where the captured image was taken by using an estimated image obtained when a specific space is captured at at least one candidate position and the captured image (S12).

As described above, the information processing device 10 estimates the candidate positions using the distance between the target object and the position where the captured image was taken, and the position of the target object. Furthermore, the information processing device 10 narrows down the candidate positions using the estimated image and the captured image. This allows the information processing device 10 to estimate with high accuracy the position where the captured image was taken.

(Embodiment 2)
The process or operation of the identification unit 12 according to the second embodiment will be described. The process or operation of the estimation unit 11 is substantially the same as that of the first embodiment, and therefore detailed description thereof will be omitted. The identification unit 12 identifies the position where the captured image was captured based on the matching result between the estimated image and the captured image.

Matching an estimated image with a captured image may involve associating similar feature points by comparing feature points included in the estimated image with feature points included in the captured image. Associating similar feature points may be rephrased as matching or linking similar feature points. Similar feature points may mean that the feature amounts of the feature points indicate values that are the same or that the difference between the feature amounts is equal to or less than a predetermined threshold. The feature amount may be, for example, information indicating a color or information indicating the shape of an object. The feature amount may be represented as a vector. The feature points may be extracted or detected using, for example, SIFT, SURF, etc.

The matching result may be the number of associated feature points. The identification unit 12 may consider an estimated image to be more similar to the captured image if the estimated image has a larger number of feature points associated with feature points contained in the captured image.

The identification unit 12 may identify, as the location where the captured image was taken, a candidate location associated with an estimated image having the largest number of feature points associated with feature points included in the captured image. Alternatively, the identification unit 12 may identify, as the location where the captured image was taken, a candidate location associated with at least one estimated image having a number of feature points associated with feature points included in the captured image that exceeds a predetermined threshold.

The identification unit 12 can select or extract from the multiple estimated images the estimated image that is more similar to the captured image, thereby identifying a more accurate position from the multiple candidate positions as the position where the captured image was captured.

The identification unit 12 may also identify the position at which the captured image was taken based on a matching result between the target object included in the estimated image and the target object included in the captured image. In other words, the identification unit 12 may consider the estimated image to be more similar the greater the number of feature points indicating the target object included in the estimated image that are associated with feature points indicating the target object included in the captured image.

The identification unit 12 may identify, as the location where the captured image was taken, a candidate location associated with an estimated image having the largest number of feature points associated with feature points indicating a target object included in the captured image. Alternatively, the identification unit 12 may identify, as the location where the captured image was taken, a candidate location associated with at least one estimated image having a number of feature points associated with feature points indicating a target object included in the captured image that exceeds a predetermined threshold.

The identification unit 12 can select or extract from among multiple estimated images the estimated image that is most similar to the captured image, thereby being able to identify a more accurate position as the position where the captured image was captured from among multiple candidate positions, even in an environment with many objects of the same shape, etc.

Next, a case where the identification unit 12 identifies multiple positions as positions at which the captured image was captured will be described with reference to FIG. 3. The captured image in this case will be referred to as the first captured image.

Step S21 is similar to step S11 in FIG. 2, and therefore a detailed description thereof will be omitted. Next, the identification unit 12 identifies a plurality of positions as positions where the first photographed image was photographed (S22).

Next, the estimation unit 11 estimates at least one candidate position for the position where the moving object captured the second photographed image (S23). The second photographed image is an image captured at a timing different from the timing when the first photographed image was captured. For example, the second photographed image may be an image captured after the first photographed image was captured. The estimation of the candidate position for the position where the second photographed image was captured is performed in the same procedure as step S21.

The identification unit 12 identifies the location where the second captured image was taken by using an estimated image obtained when a specific space is captured at at least one candidate location and the second captured image (S24).

In step S23, the estimation unit 11 may select a candidate position from among at least one candidate position estimated using the second captured image, the candidate position having a difference in distance from the first position identified as the position where the first captured image was captured that is less than a predetermined value. In other words, the estimation unit 11 may delete candidate positions other than the candidate position having a difference in distance from the first position that is less than the predetermined value.

The estimation unit 11 narrows down the candidate positions to candidate positions near the first position identified as the position where the previous photographed image was taken, thereby reducing the processing load of the identification unit 12 in identifying the position where the second photographed image was taken. Furthermore, the estimation unit 11 narrows down the candidate positions, which also narrows down the first position identified as the position where the photographed image identified by the identification unit 12 was taken. As a result, by repeating processing using photographed images taken at different times, the positions identified by the identification unit 12 are narrowed down to one.

(Embodiment 3)
Next, a configuration example of the information processing device 20 will be described with reference to FIG. 4. The information processing device 20 may be a computer device that operates when a processor executes a program stored in a memory. The information processing device 20 has a candidate position estimation unit 21, a position identification unit 22, a reference data storage unit 23, a surrounding image data acquisition unit 24, a position information acquisition unit 25, and a setting unit 26. The components that configure the information processing device 20 may be software or modules that perform processing when a processor executes a program stored in a memory. Alternatively, the components that configure the information processing device 20 may be hardware such as a circuit or a chip.

The reference data storage unit 23 stores reference data that has been generated in advance as data indicating the position or arrangement of an object in a specified space. The reference data may be generated in the information processing device 20. Alternatively, the reference data may be generated in another computer device. The reference data storage unit 23 may acquire the reference data generated in the other computer device via a network. Alternatively, the reference data storage unit 23 may acquire the reference data offline.

The computer device that generates the reference data may generate the reference data using multiple captured images taken within a specified space. For example, the computer device may generate reference data that is three-dimensional data by executing SfM (Structure from Motion) using multiple captured images. SfM calculates all feature points of a series of already acquired two-dimensional images (or frames) and estimates matching feature points from multiple captured images that occur before and after in time. Furthermore, SfM accurately estimates the three-dimensional position or orientation of the camera that captured each frame based on the difference in position on a two-dimensional plane in the frames in which each feature point appears. The reference data storage unit 23 may store information indicating the position and orientation at which each captured image was captured in association with the captured image.

Alternatively, the computer device may generate reference data, which is three-dimensional data, by inputting design data in a specified space into software that generates three-dimensional data from the design data.

Because the reference data is three-dimensional data, the information processing device 20 can determine the distance from any point in the reference data to any object.

The surrounding image data acquisition unit 24 acquires surrounding image data. The surrounding image data is a photographed image taken by an imaging device capable of photographing the surrounding environment while moving. The photographed image may be a camera image taken by a camera and point cloud data measured or generated by a sensor or the like. The point cloud data may be data indicating the distance from the position of the sensor to an object and data having three-dimensional coordinates in a specified space. The point cloud data may be measured by a sensor device capable of measuring the distance to a target object. The sensor device may be, for example, a LiDAR (Light Detection And Ranging) device. The sensor device and the imaging device are substantially in the same position. In other words, the distance between the sensor device and an object included in the camera image is substantially equal to the distance between the imaging device and the object included in the camera image.

The surrounding image data acquisition unit 24 may acquire surrounding image data from a photographing device via a network. Alternatively, if the information processing device 20 is mounted on a moving object, the surrounding image data acquisition unit 24 may be a photographing device capable of photographing the surrounding environment.

The location information acquisition unit 25 acquires location information of the imaging device that captured the image as the surrounding image data. The location information may be, for example, location information measured by the imaging device using the Global Navigation Satellite System (GNSS). The location information acquisition unit 25 may acquire location information from the imaging device via a network. Alternatively, if the surrounding image data acquisition unit 24 is an imaging device, the location information acquisition unit 25 may acquire location information from the surrounding image data acquisition unit 24.

The setting unit 26 holds information indicating the target object. The information indicating the target object may be rephrased as information for identifying the target object or parameters indicating the target object. Alternatively, the information indicating the target object may be information indicating the name of the target object.

The candidate position estimation unit 21 and the position identification unit 22 correspond to the estimation unit 11 and the identification unit 12 in FIG. 1.

Next, the flow of the location identification process executed by the information processing device 20 will be described with reference to FIG. 5. It is assumed that the information processing device 20 generates reference data in advance before starting the process shown in FIG. 5. In other words, it is assumed that the reference data storage unit 23 stores the reference data.

First, the candidate position estimation unit 21 generates a position candidate group master as an empty set (S31). The position candidate group master is a recording area in which information indicating the position identified by the position identification unit 22 at which the photographed image was taken is recorded. Recording may be rephrased as storing, storing, etc. The position candidate group master may be a database. At the time of step S31, the position identification unit 22 has not identified the position at which the photographed image was taken, so the position candidate group master is an empty set. In addition, the time at which the candidate position estimation unit 21 generates the position candidate group master in step S31 is set as the initial time t.

Next, the surrounding image data acquisition unit 24 acquires the surrounding image captured by the imaging device (S32). The surrounding image may include the camera image captured by the imaging device and point cloud data.

Next, the surrounding image data acquisition unit 24 recognizes objects contained in the surrounding image (S33). Recognizing an object means determining what object each of the smallest elements that make up the image data (pixels for two-dimensional image data, points for three-dimensional point cloud data, and voxels for three-dimensional voxel data; hereafter referred to as "elements") is a part of (identifying the name and category of the object for each element). One common method for recognizing objects is a technology called semantic segmentation.

For example, the surrounding image data acquisition unit 24 may perform semantic segmentation on the camera image. The surrounding image data acquisition unit 24 recognizes what objects are included in the camera image by performing semantic segmentation. Specifically, the surrounding image data acquisition unit 24 identifies what object each pixel of the camera image represents. For example, the surrounding image data acquisition unit 24 may assign a label indicating the name of the object to each pixel. The surrounding image data acquisition unit 24 may also determine the category into which each pixel is classified. The surrounding image data acquisition unit 24 may also perform semantic segmentation on each point constituting the point cloud data in the same way as for the pixels, thereby identifying what object each point represents. When performing semantic segmentation, the surrounding image data acquisition unit 24 may use, for example, a learning model that identifies or classifies objects included in the image from the image.

Next, the surrounding image data acquisition unit 24 identifies the distance between the object associated with each pixel constituting the camera image and the captured image that captured the camera image (S34). The surrounding image data acquisition unit 24 uses the point cloud data to identify the distance related to the object associated with each pixel constituting the camera image. The imaging device that captures the camera image and the sensor device that acquires the point cloud data are substantially in the same position. Therefore, each pixel of the camera image is associated with a point included in the point cloud data. In other words, the distance to the object associated with each pixel is the distance indicated in the point cloud data associated with each pixel.

Next, the candidate position estimation unit 21 determines whether or not the peripheral image includes a target object (S35). The target object may be specified by, for example, information input by an operator who operates the information processing device 20. Alternatively, the candidate position estimation unit 21 may acquire information indicating the target object from the setting unit 26. Alternatively, the candidate position estimation unit 21 may identify an object included in reference data by analyzing reference data generated in advance. Furthermore, the candidate position estimation unit 21 may select a target object from among the multiple identified objects. The analysis of the reference data and the selection of the target object may be performed using a learning model generated by performing machine learning.

When the candidate position estimation unit 21 determines that the peripheral image includes the target object, it sets a position candidate group (t+1) based on the distance to the target object (S36). Specifically, the candidate position estimation unit 21 sets the position candidate group (t+1) based on distance information indicated by the point cloud data constituting the target object. The candidate position estimation unit 21 includes at least one area or position in the reference data that is separated from the target object included in the reference data by the distance between the image capture device and the target object in the position candidate group (t+1). Including at least one area or position in the position candidate group (t+1) may be rephrased as setting at least one area or position in the position candidate group (t+1). At least one area or position in the reference data that is separated from the target object included in the reference data by the distance between the image capture device and the target object may be a position on a circumference with the target object at the center and the distance between the image capture device and the target object as a radius.

The location candidate group (t+1) is a recording area in which information indicating location information associated with time t+1 is recorded. The location candidate group (t+1) may be a database. At least one area or location included in the location candidate group (t+1) is a candidate for the location where the captured image included in the peripheral image acquired by the peripheral image data acquisition unit 24 was captured.

If the candidate position estimation unit 21 determines that the target object is not included in the peripheral image, it sets a group of position candidates (t+1) based on the distance to the object other than the target object (S37). The candidate position estimation unit 21 includes in the group of position candidates (t+1) at least one area in the reference data that is separated from the object other than the target object included in the reference data by the distance between the image capture device and the object.

Next, the candidate position estimation unit 21 leaves in the position candidate group (t+1) only the position candidates that are in the vicinity of the position stored in the position candidate group master among the position candidates included in the position candidate group (t+1) (S38). The vicinity of the position stored in the position candidate group master may be, for example, within a predetermined distance from the position stored in the position candidate group master. If the position candidate group master is an empty set and no information indicating a position is recorded in the position candidate group master, the candidate position estimation unit 21 leaves all the position candidates included in the position candidate group (t+1). Leaving a position candidate may be rephrased as maintaining or not deleting a position candidate. Furthermore, position candidates other than the remaining position candidates are deleted.

Next, the position identification unit 22 generates a viewpoint image at a viewpoint from a position candidate included in the position candidate group (t+1) (S39). The viewpoint image corresponds to the estimated image described in other embodiments. The viewpoint is determined by the position of the position candidate and the viewpoint direction. The position identification unit 22 identifies the position of the position candidate in the reference data, and further generates a viewpoint image in the reference data in the determined direction from the identified position. The viewpoint image may be an image equivalent to a captured image captured in the determined direction from the identified position. The position identification unit 22 may, for example, predetermine the angle of view when capturing an image in the determined direction from the identified position. The position identification unit 22 may generate the viewpoint image using the predetermine angle of view.

Alternatively, the position identification unit 22 may use, as the viewpoint image, a two-dimensional image used when generating reference data, which is three-dimensional data, by executing SfM. The position identification unit 22 may use, as the viewpoint image, a two-dimensional image used when generating reference data, in which the photographed position is the position of the candidate position or a position within a predetermined range from the position of the candidate position.

Next, the position identification unit 22 compares the viewpoint image with the surrounding image and calculates the similarity (S40). For example, the position identification unit 22 extracts feature points in each of the viewpoint image and the camera image included in the surrounding image. The position identification unit 22 compares the extracted feature points with each other. The comparison result between the feature points may be indicated as a score. For example, the higher the score, the higher the similarity between the viewpoint image and the surrounding image. Furthermore, the comparison between the viewpoint image and the surrounding image may be performed on a pixel-by-pixel basis. For the pixel-by-pixel comparison, for example, a score may be calculated on a pixel-by-pixel basis, and the calculated score may be statistically processed to generate a score indicating the comparison result between the viewpoint image and the surrounding image.

Next, the position identification unit 22 stores the positions and directions of viewpoint images with high similarity in the position candidate group master (S41). A viewpoint image with high similarity may be, for example, a viewpoint image associated with a score higher than a predetermined score. The position identification unit 22 may store the positions and directions of viewpoint images with similarity exceeding a threshold in the position candidate group master.

Next, the position identification unit 22 determines whether the position candidate group master contains only one element (S42). An element is information that indicates a position and direction contained in the position candidate group master. If the position candidate group master contains only one element, the position identification unit 22 identifies the position and direction contained in the position candidate group master as the position and direction at which the surrounding image was captured (S43). If the position candidate group master contains multiple elements, the position identification unit 22 advances the time by one unit, setting t = t + 1 (S44). After step S44, the processing from step S32 onwards is repeated.

In step S38, the information processing device 20 narrows down the location candidates to be left in the location candidate group (t+1), thereby narrowing down the location information to be stored in the location candidate group master in step S41. As a result, the information processing device 20 can identify a single location where the captured image was captured by repeating the processes shown in Figures 5 and 6.

The candidate position estimation unit 21 may also narrow down the candidate positions in steps S36 and S37 based on the position information of the imaging device acquired by the position information acquisition unit 25. For example, since there is a steel structure above a substation or a transmission tower, accurate position information may not be acquired by GNSS. In other words, the position information acquired by the position information acquisition unit 25 may not accurately indicate the position of the imaging device. However, the position information acquired by the position information acquisition unit 25 indicates the approximate position of the imaging device. Therefore, the candidate position estimation unit 21 may include only candidate positions that are in the vicinity of the position information indicating the approximate position of the imaging device in the position candidate group (t+1). In this way, by narrowing down the candidate positions to be included in the position candidate group (t+1) by the candidate position estimation unit 21, the position identification unit 22 can quickly identify the position where the image was taken.

Furthermore, in steps S36 and S37, the distance information used in setting the position candidate group (t+1) may take into account an arbitrary set error. That is, for example, in step S36, the candidate position estimation unit 21 may include in the position candidate group (t+1) at least one position in the reference data that is away from the target object included in the reference data by the distance between the shooting device and the target object ±A (A is an arbitrary error). Furthermore, the value A indicating the error may be set to be smaller as time progresses. By decreasing the value A as time progresses, the number of positions to be included in the position candidate group (t+1) can be reduced, and the position identification unit 22 can quickly identify the position where the captured image was captured.

In addition, in step S41, the position identification unit 22 may increase the threshold value used when extracting or selecting the positions and directions of the viewpoint images to be stored in the position candidate group master as time advances. By increasing the threshold value as time advances, the number of positions and directions of the viewpoint images to be stored in the position candidate group master can be reduced, and the position identification unit 22 can quickly identify the position where the captured image was taken.

Furthermore, a case will be described in which either the peripheral image or the viewpoint image acquired by the peripheral image data acquisition unit 24 is only point cloud data. In this case, the position identification unit 22 calculates edges using the point cloud data. An edge is an area where the orientation of a surface changes. The position identification unit 22 may set the color of the edge to a color different from other areas. The position identification unit 22 calculates the similarity using the edge on the point cloud data and the position of the edge of the peripheral image or the viewpoint image acquired by the peripheral image data acquisition unit 24.

(Embodiment 4)
Next, a configuration example of an information processing system will be described with reference to Fig. 7. The information processing system in Fig. 7 includes an image capturing device 40, an information processing device 50, and an information processing device 60. The image capturing device 40 moves together with a moving object 30, captures an image of the surroundings of the moving object 30, and generates a captured image.

The moving body 30 may be a vehicle equipped with the imaging device 40. The vehicle may be a vehicle capable of running autonomously. The vehicle may be a vehicle that runs by remote control. The vehicle may be a vehicle that runs by being directly operated by a driver. Or, the moving body 30 may be a human being that moves while holding the imaging device 40.

The information processing device 50 generates reference data in advance as data indicating the position of an object in a specified space. The information processing device 50 may generate reference data using a captured image captured within the specified space. Alternatively, the information processing device 50 may generate reference data using design data or the like in the specified space. The reference data may be three-dimensional data.

The information processing device 60 estimates at least one candidate position for the position where the photographed image was taken, using the distance between the target object included in the photographed image and the position where the photographed image was taken, and the position of the target object in the reference data. Furthermore, the information processing device 60 identifies the position where the photographed image was taken, using the photographed image and an estimated image obtained when a specific space is photographed at the at least one candidate position.

As described above, the generation of reference data may be performed in an information processing device 50 that is different from the information processing device 60 that estimates and identifies the location where the captured image was taken. By separating the information processing devices in this way, the load is distributed compared to when the generation of reference data and the estimation and identification of the location where the captured image was taken are performed in a single information processing device.

Furthermore, there are various examples of functional arrangements other than the functional arrangement shown in FIG. 7. For example, the estimation unit 11 and the identification unit 12 included in the information processing device 10 of FIG. 1 may each be executed in a different information processing device. When the functions are distributed to multiple information processing devices in this manner, each information processing device may be connected via a network.

FIG. 8 is a block diagram showing an example configuration of the information processing devices 10, 20, 50, and 60 (hereinafter referred to as information processing devices 10, etc.) described in the above-mentioned embodiments. Referring to FIG. 7, the information processing device 10, etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 may be used to communicate with a network node. The network interface 1201 may include, for example, a network interface card (NIC) that complies with the IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers.

The processor 1202 reads out and executes software (computer programs) from the memory 1203 to perform the processing of the measuring device 20 described using the flowcharts in the above-mentioned embodiment. The processor 1202 may be, for example, a microprocessor, an MPU, or a CPU. The processor 1202 may include multiple processors.

Memory 1203 is composed of a combination of volatile memory and non-volatile memory. Memory 1203 may include storage located away from processor 1202. In this case, processor 1202 may access memory 1203 via an I/O (Input/Output) interface (not shown).

In the example of FIG. 7, the memory 1203 is used to store a group of software modules. The processor 1202 reads out and executes these groups of software modules from the memory 1203, thereby performing the processing of the information processing device 10 and the like described in the above embodiment.

As explained using FIG. 7, each of the processors of the information processing device 10 in the above-mentioned embodiment executes one or more programs including a set of instructions for causing a computer to execute the algorithm explained using the drawings.

In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium may include electrical, optical, acoustic, or other forms of propagated signals.

The present disclosure has been described above with reference to the embodiments, but the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

Some or all of the above embodiments can be described as follows, but are not limited to the following:

(Appendix 1)
an estimation unit that estimates at least one candidate position of the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating the position of an object in a predetermined space;
An information processing device comprising: an identification unit that identifies a position where the first captured image was captured using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image.
(Appendix 2)
The identification unit is
The information processing device according to claim 1, further comprising: a processor configured to detect a position where the first captured image was captured based on a matching result between the estimated image and the first captured image.
(Appendix 3)
The identification unit is
The information processing device according to claim 1 or 2, further comprising: determining a position at which the first captured image was captured based on a matching result between the target object included in the estimated image and the target object included in the first captured image.
(Appendix 4)
The identification unit is
An information processing device as described in any one of appendices 1 to 3, wherein when a plurality of first positions are identified as positions at which the first captured image was captured, the position at which the second captured image was captured is identified using an estimated image of the specified space captured at the candidate position estimated using a second captured image captured at a time different from the time at which the first captured image was captured, and the second captured image.
(Appendix 5)
The estimation unit is
The information processing device according to claim 4, further comprising: selecting, from among at least one of the candidate positions estimated using the second captured image, a candidate position whose difference in distance from the plurality of first positions is less than a predetermined value.
(Appendix 6)
The information processing device according to any one of appendixes 1 to 5, wherein the first captured image includes point cloud data, and the distance between the target object and the position at which the first captured image was captured is determined using point cloud data indicating the target object included in the first captured image.
(Appendix 7)
The estimation unit is
The information processing device according to any one of claims 1 to 6, further comprising: estimating the at least one candidate position based on position information using GNSS.
(Appendix 8)
The identification unit is
The information processing device according to any one of claims 1 to 7, wherein a captured image for reference data captured at the candidate position when generating the reference data is used as the estimated image.
(Appendix 9)
an image capturing device that moves together with the moving object and captures an image of the surroundings of the moving object to generate a first captured image;
A first information processing device that generates reference data in advance as data indicating a position of an object in a predetermined space;
and a second information processing device that estimates at least one candidate position for the position at which the first captured image was captured using a distance between a target object included in the first captured image and a position at which the first captured image was captured and the position of the target object in the reference data, and identifies the position at which the first captured image was captured using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image.
(Appendix 10)
The second information processing device is
The information processing system according to claim 9, further comprising: identifying a position where the first captured image was captured based on a matching result between the estimated image and the first captured image.
(Appendix 11)
estimating at least one candidate position for the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating a position of an object in a predetermined space;
A position identification method, comprising: identifying a position at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image.
(Appendix 12)
When identifying the location where the captured image was captured,
The position identification method according to claim 11, further comprising identifying a position at which the first captured image was captured based on a matching result between the estimated image and the first captured image.
(Appendix 13)
When identifying the location where the captured image was captured,
The position identification method according to claim 11 or 12, further comprising identifying a position at which the first captured image was captured based on a matching result between the target object included in the estimated image and the target object included in the first captured image.
(Appendix 14)
When identifying the location where the captured image was captured,
The position identification method according to any one of appendices 11 to 13, in which, when a plurality of first positions are identified as positions at which the first photographed image was taken, the position at which the second photographed image was taken is identified using an estimated image of the specified space photographed at the candidate position estimated using a second photographed image taken at a timing different from the timing at which the first photographed image was taken, and the second photographed image.
(Appendix 15)
When estimating the candidate positions,
The position identification method according to claim 14, further comprising selecting, from among at least one of the candidate positions estimated using the second captured image, a candidate position whose difference in distance from the plurality of first positions is less than a predetermined value.
(Appendix 16)
The position identification method according to any one of appendixes 11 to 15, wherein the first captured image includes point cloud data, and the distance between the target object and a position at which the first captured image was captured is identified using the point cloud data indicating the target object included in the first captured image.
(Appendix 17)
When estimating the candidate positions,
17. The method of claim 11, further comprising estimating the at least one candidate position based on location information using GNSS.
(Appendix 18)
When identifying the location where the captured image was captured,
18. The position identification method according to any one of appendices 11 to 17, wherein a captured image for reference data captured at the candidate position when generating the reference data is used as the estimated image.
(Appendix 19)
generating reference data as data indicating a position of an object in a predetermined space;
Moving together with the moving object and capturing an image of the surroundings of the moving object to generate a first captured image;
estimating at least one candidate position of the position at which the first photographed image was photographed, using a distance between a target object included in the first photographed image and a position at which the first photographed image was photographed, and a position of the target object in the reference data;
A position identification method, comprising: identifying a position at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image.
(Appendix 20)
estimating at least one candidate position for the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating a position of an object in a predetermined space;
A non-transitory computer-readable medium having stored thereon a program for causing a computer to execute the following: identify the location at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate location and the first captured image.

REFERENCE SIGNS LIST 10 Information processing device 11 Estimation unit 12 Identification unit 20 Information processing device 21 Candidate position estimation unit 22 Position identification unit 23 Reference data storage unit 24 Peripheral image data acquisition unit 25 Position information acquisition unit 26 Setting unit 30 Mobile object 40 Photographing device 50 Information processing device 60 Information processing device

Claims (20)

an estimation unit that estimates at least one candidate position of the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating the position of an object in a predetermined space;
An information processing device comprising: an identification unit that identifies a position where the first captured image was captured using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image. The identification unit is
The information processing apparatus according to claim 1 , further comprising: identifying a position where the first captured image was captured based on a matching result between the estimated image and the first captured image. The identification unit is
The information processing apparatus according to claim 1 , further comprising: determining a position where the first captured image was captured based on a matching result between the target object included in the estimated image and the target object included in the first captured image. The identification unit is
4. The information processing device according to claim 1, wherein, when a plurality of first positions are identified as the positions at which the first captured image was captured, the position at which the second captured image was captured is identified using an estimated image of the specified space captured at the candidate position estimated using a second captured image captured at a time different from the time at which the first captured image was captured, and the second captured image. The estimation unit is
The information processing apparatus according to claim 4 , further comprising: selecting, from among the at least one candidate position estimated using the second captured image, a candidate position whose difference in distance from the plurality of first positions is less than a predetermined value. The information processing device according to any one of claims 1 to 5, wherein the first captured image includes point cloud data, and the distance between the target object and the position where the first captured image was captured is determined using point cloud data indicating the target object included in the first captured image. The estimation unit is
The information processing device according to claim 1 , further comprising: a step of estimating the at least one candidate position based on position information using a GNSS. The identification unit is
The information processing apparatus according to claim 1 , wherein a captured image for reference data captured at the candidate position when generating the reference data is used as the estimated image. an image capturing device that moves together with the moving object and captures an image of the surroundings of the moving object to generate a first captured image;
A first information processing device that generates reference data in advance as data indicating a position of an object in a predetermined space;
and a second information processing device that estimates at least one candidate position for the position at which the first captured image was captured using a distance between a target object included in the first captured image and a position at which the first captured image was captured and the position of the target object in the reference data, and identifies the position at which the first captured image was captured using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image. The second information processing device is
The information processing system according to claim 9 , further comprising: identifying a position where the first captured image was captured based on a matching result between the estimated image and the first captured image. estimating at least one candidate position for the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating a position of an object in a predetermined space;
A position identification method, comprising: identifying a position at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image. When identifying the location where the captured image was captured,
The position specifying method according to claim 11 , further comprising the step of specifying a position at which the first photographed image was taken based on a matching result between the estimated image and the first photographed image. When identifying the location where the captured image was captured,
The position specifying method according to claim 11 or 12, further comprising specifying a position at which the first captured image was captured based on a matching result between the target object included in the estimated image and the target object included in the first captured image. When identifying the location where the captured image was captured,
14. The method for identifying a position according to claim 11, wherein, when a plurality of first positions are identified as positions at which the first photographed image was taken, the position at which the second photographed image was taken is identified using an estimated image of the specified space photographed at the candidate position estimated using a second photographed image photographed at a timing different from the timing at which the first photographed image was taken, and the second photographed image. When estimating the candidate positions,
The position identification method according to claim 14 , further comprising selecting, from among the at least one candidate position estimated using the second captured image, a candidate position whose difference in distance from the plurality of first positions is less than a predetermined value. The position determination method according to any one of claims 11 to 15, wherein the first captured image includes point cloud data, and the distance between the target object and the position where the first captured image was captured is determined using point cloud data indicating the target object included in the first captured image. When estimating the candidate positions,
The method of claim 11 , further comprising estimating the at least one candidate position based on location information using GNSS. When identifying the location where the captured image was captured,
The position specifying method according to claim 11 , further comprising the step of: using, as the estimated image, a reference data captured image taken at the candidate position when generating the reference data. generating reference data as data indicating a position of an object in a predetermined space;
Moving together with the moving object and capturing an image of the surroundings of the moving object to generate a first captured image;
estimating at least one candidate position of the position at which the first photographed image was photographed, using a distance between a target object included in the first photographed image and a position at which the first photographed image was photographed, and a position of the target object in the reference data;
A position identification method, comprising: identifying a position at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate position and the first captured image. estimating at least one candidate position for the position at which the first photographed image was photographed, using a distance between a target object included in a first photographed image photographed from a moving body and a position at which the first photographed image was photographed, and a position of the target object in reference data that is generated in advance as data indicating a position of an object in a predetermined space;
A non-transitory computer-readable medium having stored thereon a program for causing a computer to execute the following: identify the location at which the first captured image was captured by using an estimated image obtained when the specified space is captured at the at least one candidate location and the first captured image.

Images