JP7183058B2 - Three-dimensional measuring device and three-dimensional measuring program - Google Patents

Description

The present invention relates to three-dimensional measurement.

Three-dimensional data of features are used to estimate the position of the vehicle with high accuracy.
Patent Literature 1 discloses a method of estimating the position of a vehicle with high accuracy using three-dimensional data of features.

Patent Documents 2 to 7 will be mentioned in the embodiment.

JP 2018-116014 A Japanese Patent No. 5582691 Japanese Patent No. 4344869 Japanese Patent No. 4132068 Japanese Patent No. 4619962 Japanese Patent No. 4568845 Japanese Patent No. 3966419

An object of the present invention is to enable measurement of the three-dimensional position of a feature.

A three-dimensional measuring device of the present invention includes a camera, a measurement sensor, and a display.
A first image of the object to be measured is obtained by first photographing at a first location using the camera, and a second image of the object to be measured is obtained by second photographing at a second location using the camera. an imaging unit for obtaining a second image;
The three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the first imaging, and the three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the second imaging. a position and orientation measurement unit that
a measurement location reception unit that displays the first image on the display and receives a measurement location that is a location where the measurement target is shown in the first image;
Based on the three-dimensional position and three-dimensional orientation of the camera at the time of the first imaging and the three-dimensional position and three-dimensional orientation of the camera at the time of the second imaging, at the measurement location in the second image a corresponding portion identification unit that identifies a corresponding portion that is a corresponding portion;
a target position calculation unit that calculates a three-dimensional position of the measurement target based on the measurement position in the first image and the corresponding position in the second image.

According to the present invention, the three-dimensional position of a feature can be measured.

1 is a configuration diagram of a three-dimensional measurement system 100 according to Embodiment 1. FIG. 1 is a configuration diagram of a three-dimensional measuring device 200 according to Embodiment 1. FIG. FIG. 2 is a configuration diagram of an information management device 300 according to Embodiment 1; FIG. 4 is a flowchart of a three-dimensional measurement method according to Embodiment 1; 4 is a flowchart of first point processing (S110) according to Embodiment 1; 4 is a flow chart of measurement point reception processing (S120) according to the first embodiment; 4 is a flowchart of second point processing (S130) according to Embodiment 1; FIG. 10 is a flowchart of corresponding part identification processing (S140) according to the first embodiment; FIG. 4 is a flowchart of target position calculation processing (S150) according to the first embodiment; 4 is a diagram showing a first specific example of a three-dimensional measurement method according to Embodiment 1; FIG. FIG. 7 shows a second specific example of the three-dimensional measurement method according to Embodiment 1; FIG. 11 is a flowchart of another example of the corresponding location identification process (S140) in the first embodiment; FIG. 4 is a flowchart of another example of the target position calculation process (S150) according to Embodiment 1;

The same or corresponding elements are denoted by the same reference numerals in the embodiments and drawings. Descriptions of elements having the same reference numerals as those described will be omitted or simplified as appropriate. Arrows in the figure mainly indicate the flow of data or the flow of processing.

Embodiment 1.
A three-dimensional measurement system 100 will be described with reference to FIGS. 1 to 13. FIG.

*** Configuration description ***
The configuration of the three-dimensional measurement system 100 will be described based on FIG.
A three-dimensional measurement system 100 is a system for measuring the three-dimensional position of an object.

A three-dimensional measurement system 100 includes an information management sensor 101 and a three-dimensional measurement device 200 . The information management sensor 101 includes an information management device 300 .
The three-dimensional measuring device 200 and the information management device 300 communicate with each other via the network 102 .

The configuration of the three-dimensional measuring device 200 will be described based on FIG.
The three-dimensional measuring device 200 is a computer having hardware such as a processor 201 , a memory 202 , an auxiliary storage device 203 , a communication device 204 , a touch panel 205 , a camera 206 and a measurement sensor 207 . These pieces of hardware are connected to each other via signal lines.
A specific example of a computer that implements the three-dimensional measuring device 200 is a smart phone or a tablet PC. PC is an abbreviation for personal computer. However, other types of computers may be used as the three-dimensional measuring device 200 .

A processor 201 is an IC (Integrated Circuit) that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit).
Memory 202 is a volatile storage device. Memory 202 is also referred to as main storage or main memory. For example, the memory 202 is RAM (Random Access Memory). The data stored in memory 202 is saved in auxiliary storage device 203 as needed.
Auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed.
Communication device 204 is a receiver and transmitter. For example, communication device 204 is a communication chip or NIC. NIC is an abbreviation for Network Interface Card.
A touch panel 205 is an input/output device having a display. The three-dimensional measuring device 200 may include an input/output device of a different type than the touch panel 205 . For example, the three-dimensional measuring device 200 may have a display and numeric keys instead of the touch panel 205 .
Camera 206 is a digital camera.
The measurement sensor 207 is one or more sensors for measuring the position and orientation of the three-dimensional measurement device 200 . For example, the measurement sensor 207 is a GPS sensor, a gyro sensor, an acceleration sensor, and the like. GPS sensors are also called GPS receivers or GPS modules. GPS is an abbreviation for Global Positioning System.

The three-dimensional measurement apparatus 200 includes an imaging unit 211, a position/orientation measurement unit 212, a point cloud acquisition unit 213, a position/orientation correction unit 222, a point cloud projection unit 221, a measurement location reception unit 231, a corresponding location identification unit 232, and a target location calculation unit. It comprises an element such as part 233 . These elements are implemented in software.

The auxiliary storage device 203 includes an imaging unit 211, a position/orientation measurement unit 212, a point cloud acquisition unit 213, a position/orientation correction unit 222, a point cloud projection unit 221, a measurement location reception unit 231, a corresponding location identification unit 232, and a target location calculation unit. A three-dimensional measurement program for functioning a computer is stored as part 233 . A three-dimensional measurement program is loaded into the memory 202 and executed by the processor 201 .
Furthermore, the auxiliary storage device 203 stores an OS (Operating System). At least part of the OS is loaded into memory 202 and executed by processor 201 .
That is, the processor 201 executes the three-dimensional measurement program while executing the OS.
Data obtained by executing the three-dimensional measurement program is stored in a storage device such as memory 202 , auxiliary storage device 203 , registers within processor 201 , or cache memory within processor 201 .

Memory 202 functions as storage unit 290 . However, another storage device may function as storage unit 290 instead of memory 202 or together with memory 202 .

The three-dimensional measuring device 200 may include multiple processors that substitute for the processor 201 . A plurality of processors share the role of processor 201 .

The three-dimensional measurement program can be computer-readable (stored) in a non-volatile recording medium such as an optical disc or flash memory.

The configuration of the information management device 300 will be described based on FIG.
The information management device 300 is a computer having hardware such as a processor 301 , a memory 302 , an auxiliary storage device 303 , a communication device 304 and an input/output interface 305 . These pieces of hardware are connected to each other via signal lines.
A specific example of a computer that implements the information management device 300 is a database server. However, other types of computers may be used as the information management device 300 .

A processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, processor 301 is a CPU, DSP or GPU.
Memory 302 is a volatile storage device. Memory 302 is also referred to as main storage or main memory. For example, memory 302 is RAM. The data stored in the memory 302 is saved in the auxiliary storage device 303 as required.
Auxiliary storage device 303 is a non-volatile storage device. For example, the auxiliary storage device 303 is ROM, HDD or flash memory. Data stored in the auxiliary storage device 303 is loaded into the memory 302 as required.
Communication device 304 is a receiver and transmitter. For example, communication device 304 is a communication chip or NIC.
The input/output interface 305 is a port to which an input/output device is connected. For example, the input/output interface 305 is a USB terminal, the input device is a keyboard and mouse, and the output device is a display. USB is an abbreviation for Universal Serial Bus.

The information management device 300 includes elements such as a request reception unit 311 , a point cloud extraction unit 312 and a point cloud response unit 313 . These elements are implemented in software.

The auxiliary storage device 303 stores an information management program for causing the computer to function as a request reception unit 311 , a point cloud extraction unit 312 and a point cloud response unit 313 . The information management program is loaded into memory 302 and executed by processor 301 .
Furthermore, an OS is stored in the auxiliary storage device 303 . At least part of the OS is loaded into memory 302 and executed by processor 301 .
That is, the processor 301 executes the information management program while executing the OS.
Data obtained by executing the information management program is stored in a storage device such as memory 302 , auxiliary storage device 303 , registers within processor 301 , or cache memory within processor 301 .

Memory 302 and auxiliary storage device 303 function as point cloud database 390 .
The point cloud database 390 is a database for managing three-dimensional point cloud data.
The 3D point cloud data represents a 3D point cloud. The format of the 3D point cloud data is, for example, a LAS file. A LAS file refers to an industry standard binary format for storing LIDAR data measured by aircraft or vehicles. LIDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging.
A 3D point cloud is one or more 3D points.
One 3D point corresponds to one point. Each 3D point has a 3D coordinate value.

The information management device 300 may include multiple processors that substitute for the processor 301 . A plurality of processors share the role of processor 301 .

The information management program can be recorded (stored) in a non-volatile recording medium such as an optical disc or flash memory in a computer-readable manner.

***Description of operation***
The operation of the three-dimensional measurement system 100 corresponds to a three-dimensional measurement method. Also, the procedure of the three-dimensional measurement method corresponds to the procedure of the three-dimensional measurement program.

A three-dimensional measurement method will be described based on FIG.
In step S110, the photographing unit 211 obtains a first image by first photographing using the camera 206 at a first point.
The first photography is photography performed using the camera 206 at the first location.
The first image is an image of the object to be measured, and is obtained by the first imaging.

Also, the position and orientation measurement unit 212 measures the three-dimensional position and three-dimensional orientation of the camera 206 using the measurement sensor 207 during the first image capturing.
A three-dimensional position is represented by three-dimensional coordinate values.
A three-dimensional posture is represented by a roll angle, a pitch angle, and a yaw angle.

Based on FIG. 5, the procedure of the first point processing (S110) will be described.
In the description of FIG. 5, the three-dimensional measuring device 200 is called a "smartphone".

In step S111, the user carries the smartphone and moves to a first point, points the camera 206 of the smartphone toward the measurement target, and performs a photographing operation.
Then, the imaging unit 211 performs the first imaging by operating the camera 206 . Thereby, the first image is obtained.

In step S112, the position and orientation measurement unit 212 causes the measurement sensor 207 to operate. As a result, the three-dimensional position of the three-dimensional measuring device 200 during the first imaging and the three-dimensional orientation of the three-dimensional measuring device 200 during the first imaging are measured.
Then, the position/orientation measuring unit 212 adds the position offset of the camera 206 to the three-dimensional position of the three-dimensional measuring device 200 at the time of the first imaging. Thereby, the three-dimensional position of the camera 206 at the time of the first imaging is calculated. The position offset of camera 206 is registered in advance in storage unit 290 .
Furthermore, the position and orientation measurement unit 212 adds the orientation offset of the camera 206 to the three-dimensional orientation of the three-dimensional measuring device 200 at the time of the first imaging. Thereby, the three-dimensional posture of the camera 206 at the time of the first photographing is calculated. The attitude offset of camera 206 is registered in advance in storage unit 290 .

In step S113, the point cloud acquisition unit 213 acquires first point cloud data based on the three-dimensional position of the camera 206 during the first imaging and the three-dimensional posture of the camera 206 during the first imaging.
The first point cloud data represents the first point cloud.
The first point group is a three-dimensional point group within the range shown in the first image.

The point cloud acquisition unit 213 acquires the first point cloud data as follows.
First, the point cloud acquisition unit 213 uses the communication device 204 to transmit the first point cloud request data to the information management device 300 . The first point cloud request data includes the 3D position of the camera 206 at the time of the first shooting, the 3D attitude of the camera 206 at the time of the first shooting, and the camera parameters at the time of the first shooting. For example, camera parameters are focal length, image size or pixel size.
Next, the request receiving unit 311 uses the communication device 304 to receive the first point cloud request data.
Next, the point cloud extraction unit 312 extracts first point cloud data from the point cloud database 390 based on the first point cloud request data from the request reception unit 311 .
Next, the point cloud response unit 313 uses the communication device 304 to transmit the first point cloud data from the point cloud extraction unit 312 to the three-dimensional measuring device 200 . By restricting the depth direction from the position of the camera, the amount of 3D point cloud data can be suppressed. An example limit is within 10 meters depth from the camera center.
Then, the point cloud acquisition unit 213 uses the communication device 204 to receive the first point cloud data.

Extraction of the first point cloud data will be described.
The point cloud extraction unit 312 extracts from the point cloud database 390 the point cloud data of the three-dimensional point cloud projected onto the first image plane. The extracted point cloud data is the first point cloud data.
The projection of the 3D point cloud onto the first image plane is described in step S114.

The first imaging may be performed when acquiring the first point cloud data. Specific examples are described below.
When the user operates the touch panel 205 of the smartphone to start the application, camera images can be obtained periodically. From this state, the first point cloud request data is issued periodically (for example, once every few seconds) at the timing when the approximate position and the approximate orientation of the camera 206 are sent. The first imaging request data is sent, and the first point cloud is displayed on the display of the touch panel 205 . After fine-tuning the pose, the message "target selection" appears on the display, and a target selection gate appears on the display. The user puts the target in the gate and clicks the message "First photography OK?" displayed on the display. As a result, the camera 206 performs the first photographing, obtains the first image, and determines the time of the first photographing. The first photographing time and the first image are temporarily stored in the memory 202 . After that, a second photographing operation, which will be described later, is performed.
The user may press the point cloud acquisition button on the touch panel 205 . When the point cloud acquisition button is pressed, the camera 206 of the smart phone is activated and shooting is started. In that state, the user inputs a photographing instruction through the photographing button on the touch panel 205 . Then, the camera 206 performs the first photographing, obtains the first image, and determines the time of the first photographing. The first photographing time and the first image are temporarily stored in the memory 202 . Then, the first point cloud request data is generated.

In step S114, the point cloud projection unit 221 projects the first point cloud data acquired from the communication device 204, the three-dimensional position of the camera 206 at the time of the first imaging, and the three-dimensional posture of the camera 206 at the time of the first imaging. Project the first point cloud onto the first image based on .

The projection of the first point cloud onto the first image will be described.
Projecting the first point cloud onto the first image corresponds to projecting the first point cloud onto the first image plane.
The first image plane is the image plane corresponding to the first image.
An image plane is a three-dimensional plane corresponding to an image. The image plane includes a point at a focal distance in the direction of the camera's line of sight from the position of the camera and is orthogonal to the direction of the camera's line of sight. The size of the image plane is determined by the parameters of the camera.
Each 3D point is projected to the intersection of the line-of-sight vector from the camera to the 3D point and the image plane. A line-of-sight vector is also called an LOS vector. LOS is an abbreviation for Line Of Sight.

In step S114, the technology disclosed in Patent Document 4 or Patent Document 5 can be used.
Patent Document 4 discloses a projection point of an object point on a digital image plane (see FIG. 3).
Japanese Patent Laid-Open No. 2002-200000 discloses a projection conversion process for projecting three-dimensional point cloud data onto an image plane of a camera.

In step S115, the position/posture correction unit 222 detects a first feature point group corresponding to the first feature portion in the first image from the first point group.
The first characteristic portion is a characteristic portion included in the first image. For example, the feature portion is an edge portion. A specific example of a feature is the edge of a road shoulder.
The first feature point group is a three-dimensional point group representing the first feature portion in the first point group data.

In step S115, techniques disclosed in Patent Documents 2 and 3 can be used.
Patent Literature 2 discloses identifying edge portions represented by a point cloud, and that each edge is a road shoulder, sidewalk, wall surface, or pole (see FIG. 21). A similar disclosure is also made in Patent Document 3.

In step S116, the position/posture correction unit 222 determines whether the detected first feature point group is projected onto the first feature portion.
For example, the position/posture correction unit 222 calculates the amount of positional deviation between the first characteristic portion in the first image and the first feature point group projected on the first image, and calculates the calculated amount of deviation as an error. Compare with threshold. If the deviation amount is equal to or less than the error threshold, the position/orientation correction unit 222 determines that the first feature point group is projected onto the first feature portion.
If the first feature point group is projected onto the first feature portion, the first point processing (S110) ends.
If the first feature point group is not projected onto the first feature portion, the process proceeds to step S117.

In step S116, the technology disclosed in Patent Document 5 can be used.
Patent Literature 5 discloses matching between a step formed by a point group projected on an image and a step reflected in the image (see FIG. 13).

In step S117, the position/orientation correction unit 222 corrects at least one of the three-dimensional position of the camera 206 during the first imaging and the three-dimensional orientation of the camera 206 during the first imaging.
Specifically, the position/posture correction unit 222 adds the position correction amount to the three-dimensional position of the camera 206 at the time of the first imaging. Also, the position/posture correction unit 222 adds the posture correction amount to the three-dimensional posture of the camera 206 at the time of the first imaging. Each of the position correction amount and the posture correction amount may be a specified value, or may be a value calculated based on the amount of positional deviation between the first feature portion and the first feature point group. Alternatively, the user may correct the position and orientation of the camera 206 by viewing the amount of positional deviation between the first feature portion and the first feature point group on the display.
After step S117, the process proceeds to step S114.

Returning to FIG. 4, the description continues from step S120.
In step S120, the measurement point accepting unit 231 displays the first image on the display of the touch panel 205 and accepts the measurement point.
The measurement location is a location where the measurement target appears in the first image, and is designated by the user.

Based on FIG. 6, the procedure of the measurement point acceptance process (S120) will be described.
In the description of FIG. 6, the three-dimensional measuring device 200 is called a "smartphone".

In step S<b>121 , the measurement point reception unit 231 causes the display of the touch panel 205 to display the first image.

In step S122, the measurement point reception unit 231 superimposes the first point group on the displayed first image.
Specifically, the point cloud projection unit 221 projects the first point cloud onto the first image (see step S114 in FIG. 5). Then, the measurement location reception unit 231 draws each three-dimensional point of the first point group at the projection location in the first image.

The measurement point reception unit 231 can use the technology disclosed in Patent Document 5. Patent Document 5 discloses three-dimensional point cloud data superimposed on an optical image (see FIG. 13).

In step S123, the user specifies a measurement point in the first image by operating touch panel 205 of the smartphone.
Then, the measurement point reception unit 231 receives designation of the measurement point via the touch panel 205 .

Returning to FIG. 4, the description continues from step S130.
In step S130, the imaging unit 211 obtains a second image by second imaging performed using the camera 206 at a second location.
The second point is a point different from the first point. For example, the second point is a point where the user has moved several meters in the lateral direction on foot while holding the smartphone.
The second photography is photography performed using the camera 206 at the second location.
The second image is an image showing the measurement target in the first image, and is obtained by the second imaging.

Also, the position and orientation measurement unit 212 measures the three-dimensional position and three-dimensional orientation of the camera 206 using the measurement sensor 207 during the second image capturing.

Based on FIG. 7, the procedure of the second point processing (S130) will be described.
In the description of FIG. 7, the three-dimensional measuring device 200 is called a "smartphone".

In step S131, the user carries the smartphone and moves to the second location, points the camera 206 of the smartphone toward the measurement target, and performs a photographing operation.
Then, the imaging unit 211 performs the second imaging by operating the camera 206 . A second image is thus obtained.

In step S132, the position and orientation measurement unit 212 causes the measurement sensor 207 to operate. As a result, the three-dimensional position of the three-dimensional measuring device 200 during the second imaging and the three-dimensional orientation of the three-dimensional measuring device 200 during the second imaging are measured.
Then, the position/orientation measuring unit 212 adds the position offset of the camera 206 to the three-dimensional position of the three-dimensional measuring device 200 at the time of the second imaging. Thereby, the three-dimensional position of the camera 206 at the time of the second photographing is calculated. The position offset of camera 206 is registered in advance in storage unit 290 .
Furthermore, the position and orientation measurement unit 212 adds the orientation offset of the camera 206 to the three-dimensional orientation of the three-dimensional measuring device 200 during the second imaging. Thereby, the three-dimensional posture of the camera 206 at the time of the second photographing is calculated. The attitude offset of camera 206 is registered in advance in storage unit 290 .

In step S133, the point cloud acquisition unit 213 acquires second point cloud data based on the three-dimensional position of the camera 206 during the second imaging and the three-dimensional posture of the camera 206 during the second imaging.
The second point cloud data represents the second point cloud.
The second point group is a three-dimensional point group within the range shown in the second image.

The method for acquiring the second point cloud data is the same as the method for acquiring the first point cloud data (see step S113 in FIG. 5). That is, the point cloud acquisition unit 213 acquires the second point cloud data from the information management device 300 using the communication device 204 .

A second imaging may be performed when acquiring the second point cloud data. Specific examples are described below.
Due to the mobile stereo system, the user moves to a second location and determines the position and orientation of the camera 206 . After that, the second point cloud request data is output in the same manner as the first point cloud request data. Then, the movement of the user to the second point is completed, and the second point group is displayed on the touch panel 205 display. When the fine adjustment of the position and orientation and the selection of the target are finished, the user clicks the message "2nd shooting OK?" displayed on the display. As a result, the camera 206 performs the second photographing, obtains the second image, and determines the time of the second photographing. The second photographing time and the second image are temporarily stored in the memory 202 .
The user may press the point cloud acquisition button on the touch panel 205 . When the point cloud acquisition button is pressed, the camera 206 of the smart phone is activated and shooting is started. In that state, the user inputs a photographing instruction through the photographing button on the touch panel 205 . Then, the camera 206 performs the second photographing to obtain the second image, and the time of the second photographing is determined. The second photographing time and the second image are temporarily stored in the memory 202 . Then, the second point cloud request data is generated.

In step S134, the point cloud projection unit 221, based on the obtained second point cloud data, the three-dimensional position of the camera 206 during the second imaging, and the three-dimensional posture of the camera 206 during the second imaging, Project the second point cloud onto a second image.

The method of projecting the second point cloud onto the second image is the same as the method of projecting the first point cloud onto the first image (see step S114 in FIG. 5).

In step S135, the position/posture correction unit 222 detects a second feature point group corresponding to the second feature portion in the second image from the second point group.
The second characteristic portion is a characteristic portion included in the second image. For example, the feature portion is an edge portion. A specific example of a feature is the edge of a road shoulder.
The second feature point group is a three-dimensional point group representing the second feature portion in the second point group data.

The method for detecting the second feature point group is the same as the method for detecting the first feature point group (see step S115 in FIG. 5). In other words, the position/orientation correction unit 222 can detect the second feature point group by using the techniques disclosed in Patent Document 2 and Patent Document 3.

In step S136, the position/posture correction unit 222 determines whether the detected second feature point group is projected onto the second feature portion. The determination method is the same as the method in step S116 (see FIG. 5).
If the second feature point group is projected onto the second feature portion, the second point processing (S130) ends.
If the second feature point group is not projected onto the second feature portion, the process proceeds to step S137.

In step S137, the position/orientation correction unit 222 corrects at least one of the three-dimensional position of the camera 206 during the second imaging and the three-dimensional orientation of the camera 206 during the second imaging. The correction method is the same as the method in step S117 (see FIG. 5).
After step S137, the process proceeds to step S134.

Returning to FIG. 4, the description continues from step S140.
In step S140, the corresponding part identification unit 232 determines the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging, and the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging. A corresponding portion in the second image is identified based on the posture.
The corresponding location is a location corresponding to the measurement location in the second image. In other words, the corresponding point is a point in the second image where the same point as the point shown in the measurement point in the first image appears. That is, the measurement target is shown in each of the measurement location in the first image and the corresponding location in the second image.

Based on FIG. 8, the procedure of the corresponding part identification process (S140) will be described.
In the description of FIG. 8, the three-dimensional measuring device 200 is called a "smartphone".

In step S141, the corresponding part identifying unit 232 determines the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging, and the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging. An epipolar line in the second image is calculated based on the orientation and the measurement location in the first image.

In step S141, the technology disclosed in Patent Document 2, Patent Document 3, or Patent Document 4 can be used.
Patent Document 2 discloses a first epipolar line calculation process for calculating an epipolar line based on the camera position and orientation (see FIG. 16). A similar disclosure is also made in Patent Document 3.
Patent Document 4 discloses a method of calculating an epipolar constraint line (see FIG. 3).

In step S142, the corresponding portion identification unit 232 displays the second image on the display of the touch panel 205 with the calculated epipolar line marked.

In step S143, the corresponding part identification unit 232 superimposes the second point group on the displayed second image. The superimposition method is the same as the method in step S122 (see FIG. 6).

In step S<b>144 , the user operates touch panel 205 of the smartphone to specify a corresponding portion on the epipolar line marked on the displayed second image.
Then, the corresponding portion identification unit 232 receives designation of the corresponding portion via the touch panel 205 .

Returning to FIG. 4, step S150 will be described.
In step S150, the target position calculation unit 233 calculates the three-dimensional position of the measurement target based on the measurement points in the first image and the corresponding points in the second image.

Based on FIG. 9, the procedure of the target position calculation process (S150) will be described.
In step S151, the target position calculation unit 233 calculates the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging, and the three-dimensional position of the camera 206 at the time of the first imaging and the time of the second imaging. The parallax between the measurement point in the first image and the corresponding point in the second image is calculated based on the posture.

In step S152, the target position calculator 233 calculates the three-dimensional position of the measurement target based on the calculated parallax.

In the target position calculation process (S150), the technology disclosed in Patent Document 4, Patent Document 6, or Patent Document 7 can be used.
Patent Document 4 discloses a three-dimensional restoration means for three-dimensionally restoring the shape of an object by using parallax (see FIG. 2).
Patent Literature 6 discloses a GIS model creation unit that creates a three-dimensional model by using parallax (see FIG. 1). A similar disclosure is also made in Patent Document 7.

A first specific example of the three-dimensional measurement method will be described with reference to FIG.
In FIG. 10, the three-dimensional measuring device 200 is called "smartphone 112".
The user aims the smartphone 112 at the measurement target 111 at the first point 113 and performs a photographing operation. A measurement target 111 is an electric wire hanging from a utility pole.
The photographing unit 211 obtains the first image 114 by photographing the photographing range 110 . A measurement target 111 is shown in the first image 114 .
Measurement point reception unit 231 displays first image 114 on touch panel 205 . A three-dimensional point group is superimposed on the displayed first image 114 (not shown). The user confirms that the three-dimensional point cloud is superimposed on the first image 114 .
By operating the touch panel 205, the user sets the marker 115 for designating the measurement point in the first image 114 at the measurement point. The marker 115 is set by double-clicking or the like. The size of the marker 115 is variable, and the measurement point reception unit 231 appropriately adjusts the size of the marker 115 .
Although the marker 115 shown in FIG. 10 has a cross shape, the marker 115 may have a circular shape, or may have a polygonal shape such as a rectangular shape or an arrowhead shape.
The measurement point reception unit 231 receives the position of the marker 115 in the first image 114 as the measurement point.
The position of the marker 115 here means the position of a specific designated point on the marker 115 . A specific designated point on the marker 115 may be appropriately set in advance. For example, if the marker 115 has a cross shape or a circular shape, the center of the marker 115 may be set as a specific designated point. Also, if the marker 115 has a rectangular shape, either the center of the marker 115 or the four corners of the marker 115 may be set as the specific designated point. Also, if the marker 115 has an arrowhead shape, the tip of the marker 115 may be set as a specific designated point.
The user carries the smartphone 112 and moves to the second location 116 .
The user aims the smartphone 112 at the measurement target 111 at the second point 116 and performs a photographing operation.
The photographing unit 211 obtains the second image 117 by photographing the photographing range 110 . The measurement target 111 is shown in the second image 117 .
Corresponding part identification unit 232 displays second image 117 on touch panel 205 . An epipolar line 118 is marked on the displayed second image 117 . Furthermore, a three-dimensional point group is superimposed on the displayed second image 117 (not shown).
The user sets the marker 119 at the corresponding location on the epipolar line 118 by operating the touch panel 205 . The marker 119 is set by double-clicking or the like. The size of the marker 119 is variable, and the corresponding portion identification unit 232 appropriately adjusts the size of the marker 119 .
The corresponding portion identification unit 232 receives the position of the marker 119 on the epipolar line 118 as the corresponding portion.
The target position calculation unit 233 calculates the three-dimensional position of the measurement target 111 by using the parallax between the measurement position where the marker 115 is set and the corresponding position where the marker 119 is set. As a result, the three-dimensional coordinate values of the lowest portion of the electric wire hanging from the utility pole are obtained.

A second specific example of the three-dimensional measurement method will be described with reference to FIG.
In FIG. 11, the three-dimensional measuring device 200 is called "smartphone 112".
The user aims the smartphone 112 at the measurement target 111 at the first point 113 and performs a photographing operation. A measurement target 111 is a road surface marker (road marking) newly marked on the road.
The photographing unit 211 obtains the first image 114 by photographing the photographing range 110 . A measurement target 111 is shown in the first image 114 .
Measurement point reception unit 231 displays first image 114 on touch panel 205 . A three-dimensional point group is superimposed on the displayed first image 114 (not shown). The user confirms that the three-dimensional point cloud is superimposed on the first image 114 .
The user sets the marker 115 at the measurement point in the first image 114 by operating the touch panel 205 . The marker 115 is set by double-clicking or the like. The size of the marker 115 is variable, and the measurement point reception unit 231 appropriately adjusts the size of the marker 115 .
The measurement point reception unit 231 receives the position of the marker 115 in the first image 114 as the measurement point.
The user carries the smartphone 112 and moves to the second location 116 .
The user aims the smartphone 112 at the measurement target 111 at the second point 116 and performs a photographing operation.
The photographing unit 211 obtains the second image 117 by photographing the photographing range 110 . The measurement target 111 is shown in the second image 117 .
Corresponding part identification unit 232 displays second image 117 on touch panel 205 . An epipolar line 118 is marked on the displayed second image 117 . Furthermore, a three-dimensional point group is superimposed on the displayed second image 117 (not shown).
The user sets the marker 119 at the corresponding location on the epipolar line 118 by operating the touch panel 205 . The marker 119 is set by double-clicking or the like. The size of the marker 119 is variable, and the corresponding portion identification unit 232 appropriately adjusts the size of the marker 119 .
The corresponding portion identification unit 232 receives the position of the marker 119 on the epipolar line 118 as the corresponding portion.
The target position calculation unit 233 calculates the three-dimensional position of the measurement target 111 by using the parallax between the measurement position where the marker 115 is set and the corresponding position where the marker 119 is set. As a result, the three-dimensional coordinate values of the road surface marker (road marking) newly marked on the road are obtained.

***Description of Examples***
The three-dimensional measuring device 200 may have the point cloud database 390 .
In that case, the point cloud acquisition unit 213 extracts the first point cloud data and the second point cloud data from the point cloud database 390 instead of the point cloud extraction unit 312 of the information management device 300 . The information management device 300 becomes unnecessary.

The three-dimensional measuring device 200 does not have to include the position/orientation correction section 222 .
In that case, the target position calculation unit 233 uses the three-dimensional position and three-dimensional orientation of the camera 206 measured by the position and orientation measurement unit 212 to calculate the three-dimensional position of the measurement target.

The measurement point reception unit 231 does not have to superimpose the three-dimensional point cloud on the image displayed on the touch panel 205 .

The corresponding part identification unit 232 may automatically identify the corresponding part without being specified by the user.
Based on FIG. 12, the procedure of the corresponding point specifying process (S140) for automatically specifying the corresponding point will be described.
In step S141, the corresponding part identification unit 232 calculates epipolar lines in the second image. The calculation method is as described with reference to FIG.

In step S142, the corresponding portion identifying unit 232 identifies, from a plurality of pixels located on the epipolar line in the second image, pixels that match the pixels located at the measurement locations in the first image. The location where the identified pixel is located is the corresponding location.

The technology disclosed in Patent Document 2, Patent Document 3, Patent Document 4, Patent Document 6, or Patent Document 7 can be used in the corresponding location identification process (S140).
Patent Literature 2 discloses a first corresponding point search process for finding corresponding points on image plane B corresponding to feature points on image plane A by searching on epipolar lines (see FIG. 16). A similar disclosure is also made in Patent Document 3.
Patent Document 4 discloses image matching means for identifying the same portion in both images by comparing an image on the epipolar constraint line with a first image at a first point (see FIG. 4).
Patent Literature 6 discloses a GIS model creation unit that specifies points of gaze corresponding to each other on images of two points (see FIG. 1). A similar disclosure is also made in Patent Document 7.

The target position calculator 233 may calculate the three-dimensional position of the measurement target based on the line-of-sight vector.
Based on FIG. 13, the procedure of the three-dimensional position calculation process (S150) for calculating the three-dimensional position of the measurement target based on the line-of-sight vector will be described.
In step S151, the target position calculation unit 233, based on the three-dimensional position of the camera 206 at the time of the first imaging, the three-dimensional posture of the camera 206 at the time of the first imaging, and the measurement points in the first image, A first line-of-sight vector is calculated.
The first line-of-sight vector is the line-of-sight vector of the camera 206 at the time of the first image capturing, and passes through the measurement point on the first image plane.
The first image plane is a three-dimensional plane corresponding to the first image.

In step S152, the target position calculation unit 233, based on the three-dimensional position of the camera 206 during the second imaging, the three-dimensional orientation of the camera 206 during the second imaging, and the measurement points in the second image, A second line-of-sight vector is calculated.
The second line-of-sight vector is the line-of-sight vector of the camera 206 at the time of the second image capturing, and passes through the corresponding location on the second image plane.
The second image plane is a three-dimensional plane corresponding to the second image.

In step S153, the target position calculation unit 233 calculates the three-dimensional coordinate values of the intersection of the first line-of-sight vector and the second line-of-sight vector.
The calculated three-dimensional coordinate value is the three-dimensional position of the measurement target.

The technique disclosed in Patent Document 2 can be used in the three-dimensional position calculation process (S150).
Patent Literature 2 discloses image three-dimensional restoration processing for calculating the three-dimensional coordinates indicated by the intersection of two LOS vectors as the three-dimensional coordinates of the measurement target point (see FIG. 15).

The three-dimensional measuring device 200 may have the following configuration.
A three-dimensional measurement device 200 includes a camera 206 , a measurement sensor 207 , a display, and a processor 201 . For example, the display is the touch panel 205 .
A measurement sensor 207 measures a three-dimensional position and a three-dimensional orientation.
The camera 206 obtains a first image showing the object to be measured during the first photographing, and obtains a second image showing the object to be measured at a position different from the position during the first photographing during the second photographing.
The display displays a marker designating a measurement point on each of the first image and the second image.
The processor 201 obtains 3D point group information whose positions are known by communicating with an external information management device. The processor 201 projects the three-dimensional point group information onto the first image and the second image based on the measurement information of the measurement sensor. A processor 201 aligns the three-dimensional point cloud information. The processor 201 calculates the three-dimensional position of the measurement point based on the designated point of the measurement point by the marker in the first image and the designated point of the measurement point by the marker in the second image. Calculate

*** Effect of Embodiment 1 ***
Indoors and outdoors where an existing 3D point cloud exists, it is possible to measure the 3D position of an object with the same accuracy as the existing 3D point cloud, using approximate position information, approximate orientation information, and a camera image. can.
Existing 3D point clouds are 3D point clouds that have been pre-measured using lasers, images, and the like. An existing 3D point cloud corresponds to the point cloud database 390 .
The rough position information corresponds to the three-dimensional position of the camera 206 measured using the measurement sensor 207 .
The general orientation information corresponds to the three-dimensional orientation of the camera 206 measured using the measurement sensor 207 .
The camera images correspond to the first image and the second image.

*** Supplement to the embodiment ***
The embodiments are examples of preferred modes and are not intended to limit the technical scope of the present invention. Embodiments may be implemented partially or in combination with other embodiments. The procedures described using flowcharts and the like may be changed as appropriate.

Each device described in the embodiments may be realized by multiple devices. That is, each device described in the embodiments may be implemented as a system.
Each element of the devices described in the embodiments may be implemented in software, hardware, firmware, or any combination thereof.
"Part" in the name of each element may be read as "processing" or "process".

100 three-dimensional measurement system, 101 information management sensor, 102 network, 110 imaging range, 111 measurement target, 112 smartphone, 113 first point, 114 first image, 115 marker, 116 second point, 117 second image, 118 epipolar line, 119 marker, 200 three-dimensional measuring device, 201 processor, 202 memory, 203 auxiliary storage device, 204 communication device, 205 touch panel, 206 camera, 207 measurement sensor, 211 imaging unit, 212 position and orientation measurement unit, 213 point group acquisition 221 point cloud projection unit 222 position and orientation correction unit 231 measurement location reception unit 232 corresponding location identification unit 233 target position calculation unit 290 storage unit 300 information management device 301 processor 302 memory 303 auxiliary storage Device, 304 communication device, 305 input/output interface, 311 request reception unit, 312 point cloud extraction unit, 313 point cloud response unit, 390 point cloud database.

Claims (6)

A three-dimensional measuring device comprising a camera, a measurement sensor, and a display,
A first image of the object to be measured is obtained by first photographing at a first location using the camera, and a second image of the object to be measured is obtained by second photographing at a second location using the camera. an imaging unit for obtaining a second image;
The three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the first imaging, and the three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the second imaging. a position and orientation measurement unit that
a measurement location reception unit that displays the first image on the display and receives a measurement location that is a location where the measurement target is shown in the first image;
Based on the three-dimensional position and three-dimensional orientation of the camera at the time of the first imaging and the three-dimensional position and three-dimensional orientation of the camera at the time of the second imaging, at the measurement location in the second image a corresponding portion identification unit that identifies a corresponding portion that is a corresponding portion;
a target position calculation unit that calculates the three-dimensional position of the measurement target based on the measurement location in the first image and the corresponding location in the second image ;
The corresponding part identifying unit
Based on the three-dimensional position and three-dimensional orientation of the camera during the first imaging, the three-dimensional position and three-dimensional orientation of the camera during the second imaging, and the measurement locations in the first image , calculating epipolar lines in the second image;
displaying the second image on the display with the calculated epipolar line marked;
Receive a location specified on the epipolar line marked on the displayed second image as the corresponding location.
Three-dimensional measuring device. The target position calculation unit
The three-dimensional position and three-dimensional orientation of the camera during the first imaging, the three-dimensional position and three-dimensional orientation of the camera during the second imaging, the measurement location in the first image, and the second 2. The three-dimensional measuring apparatus according to claim 1 , wherein the three-dimensional position of the object to be measured is calculated based on parallax with the corresponding portion in the image. The target position calculation unit
calculating a first line-of-sight vector based on the three-dimensional position and three-dimensional posture of the camera at the time of the first imaging and the measurement location in the first image; calculating a second line-of-sight vector based on the position, the three-dimensional orientation, and the corresponding location in the second image, and measuring the three-dimensional coordinate values of the intersection of the first line-of-sight vector and the second line-of-sight vector; The three-dimensional measuring device according to claim 1, wherein the three-dimensional position of the object is calculated. A three-dimensional measuring device comprising a camera, a measurement sensor, and a display,
A first image of the object to be measured is obtained by first photographing at a first location using the camera, and a second image of the object to be measured is obtained by second photographing at a second location using the camera. an imaging unit for obtaining a second image;
The three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the first imaging, and the three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the second imaging. a position and orientation measurement unit that
a measurement location reception unit that displays the first image on the display and receives a measurement location that is a location where the measurement target is shown in the first image;
Based on the three-dimensional position and three-dimensional orientation of the camera at the time of the first imaging and the three-dimensional position and three-dimensional orientation of the camera at the time of the second imaging, at the measurement location in the second image a corresponding portion identification unit that identifies a corresponding portion that is a corresponding portion;
a target position calculation unit that calculates a three-dimensional position of the measurement target based on the measurement position in the first image and the corresponding position in the second image ;
Acquiring first point cloud data representing a first point cloud, which is a three-dimensional point cloud of the range shown in the first image, based on the three-dimensional position and the three-dimensional posture of the camera at the time of the first imaging. and second point cloud data representing a second point cloud, which is a three-dimensional point cloud of the range shown in the second image, based on the three-dimensional position and three-dimensional posture of the camera at the time of the second imaging. a point cloud acquisition unit that acquires
projecting the first point cloud onto the first image based on the obtained first point cloud data and the three-dimensional position and three-dimensional orientation of the camera at the time of the first imaging, and obtaining a second point cloud; a point cloud projection unit that projects the second point cloud onto the second image based on the data and the three-dimensional position and three-dimensional posture of the camera at the time of the second imaging;
A first feature point group corresponding to a first feature portion in the first image is detected from the first point group, and the detected first feature point group is projected onto the first feature portion. correcting the three-dimensional position and three-dimensional posture of the camera at the time of the first photographing, and detecting a second feature point group corresponding to a second feature portion in the second image from the second point group; a position/orientation correction unit that corrects the three-dimensional position and three-dimensional orientation of the camera at the time of the second photographing such that the second feature point group thus obtained is projected onto the second feature portion;
with
The point cloud acquisition unit is
transmitting first point cloud request data including the three-dimensional position and three-dimensional attitude of the camera at the time of the first imaging to an information management device, receiving the first point cloud data from the information management device;
transmitting second point cloud request data including the three-dimensional position and three-dimensional attitude of the camera at the time of the second imaging to the information management device, and receiving the second point cloud data from the information management device measuring device. a measurement sensor that measures a three-dimensional position and three-dimensional orientation;
a camera that obtains a first image showing an object to be measured at the time of first photography, and obtains a second image showing the object to be measured at a position different from the position at the time of the first photography at the time of second photography;
a display that displays a marker designating a measurement point on each of the first image and the second image;
By communicating with an external information management device, three-dimensional point group information whose position is known is obtained, and based on the measurement information of the measurement sensor, the three-dimensional point group information is converted into the first image and the second image. , and aligning the three-dimensional point group information, the designated point of the measurement location by the marker in the first image, and the designated point of the measurement location by the marker in the second image A processor that calculates the three-dimensional position of the measurement point based on
A three-dimensional measuring device. A first image in which the measurement target is captured is obtained by first photography performed using a camera at a first point, and a second image in which the measurement target is captured is obtained by second photography performed using the camera at a second point. a photographing process for obtaining two images;
The three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the first imaging, and the three-dimensional position and three-dimensional orientation of the camera are measured using the measurement sensor during the second imaging. position and orientation measurement processing;
a measurement point acceptance process for displaying the first image on a display and accepting a measurement point, which is a point where the measurement target is shown in the first image;
Based on the three-dimensional position and three-dimensional orientation of the camera at the time of the first imaging and the three-dimensional position and three-dimensional orientation of the camera at the time of the second imaging, at the measurement location in the second image Corresponding part identification processing for identifying a corresponding part, which is a corresponding part;
and a target position calculation process for calculating a three-dimensional position of the measurement target based on the measurement position in the first image and the corresponding position in the second image. A measurement program ,
The corresponding location identifying process includes:
Based on the three-dimensional position and three-dimensional orientation of the camera during the first imaging, the three-dimensional position and three-dimensional orientation of the camera during the second imaging, and the measurement locations in the first image , calculating epipolar lines in the second image;
displaying the second image on the display with the calculated epipolar line marked;
Receive a location specified on the epipolar line marked on the displayed second image as the corresponding location.
3D measurement program.

Images