JP2023158825A - Three-dimensional shape measurement system, three-dimensional shape measurement method and three-dimensional shape measurement program - Google Patents

Abstract

To provide an image processing system that can accurately obtain a three-dimensional image of an object even if the number of picked-up images is small, and can also suppress the effects of light reflection.SOLUTION: A three-dimensional shape measurement system for measuring a three-dimensional shape of an object performing known movement comprises a projection unit 1 that projects one projection image 101 having a code pattern onto the object performing known movement, an imaging unit 2 that continuously captures the object onto which the one projection image 101 is projected to generate a plurality of camera images 201, and a restoration unit 44 that sets a plurality of hypothesis points m1 to m4 on the camera line of sight for an arbitrary point p0 in the camera image and determines a hypothesis point among the plurality of hypothetical points m1 to m4 that satisfies motion constraints based on the known movement as a restoration point for the arbitrary point in question.SELECTED DRAWING: Figure 3

Description

Application for application of Article 30, Paragraph 2 of the Patent Act has been filed [Publication date] September 24, 2021 [Publication address] https://eprints. lib. hokudai. ac. jp/dspace/handle/2115/83276

The present invention provides a three-dimensional shape measuring system, a three-dimensional shape measuring method, and a three-dimensional shape measuring system that projects a coded pattern image onto an object and measures the three-dimensional shape of the object by photographing the object onto which the coded pattern image is projected. Concerning a three-dimensional shape measurement program.

Many techniques have been devised to measure the three-dimensional shape of an object by projecting a pattern image onto the object. As types of pattern images, spot light projection method, slit light projection method, etc. have been used. The spot light projection method projects a laser beam onto the object, and the slit light projection method, also called the light sectioning method, is a method of determining the three-dimensional position of the object by projecting a slit light onto the object. The slit light projection method is widely used in industrial measurement and the like.

Brenner, C., Boehm, J. and Guehring, J.: Photogrammetric calibration and accuracy evaluation of a crosspattern stripe projector, Videometrics VI, Vol. 3641, International Society for Optics and Photonics, pp. 164-173 (1998). Sato, K. and Inokuchi, S.: Three-dimensional surface measurement by space encoding range imaging, Journal of Robotic Systems, Vol. 2, pp. 27-39 (1985). Sagawa, R., Furukawa, R. and Kawasaki, H.: Dense 3D reconstruction from high frame-rate video using a static grid pattern, IEEE transactions on pattern analysis and machine intelligence, Vol. 36, No. 9, pp. 1733- 1747 (2014). T. Weise, B. Leibe, and L. V. Gool. : Fast 3D scanning with automatic motion compensation, In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, 2007. Shiba, Y., Ono, S., Furukawa, R., Hiura, S., & Kawasaki, H.: Temporal Shape Super-Resolution by Intra-frame Motion Encoding Using High-fpsStructured Light, In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (pp. 115-123).

However, with conventional projection methods, a large number of captured images are required to restore the image of the object, and the restoration of the image is often affected by light reflection from the object and surrounding objects. .

SUMMARY OF THE INVENTION Therefore, the present invention provides an image processing system, an image processing device, and a program that can accurately obtain a three-dimensional image of an object even with a small number of captured images, and can suppress the effects of light reflection. The purpose is to provide

A three-dimensional shape measuring system according to one aspect of the present invention is a three-dimensional shape measuring system that measures the three-dimensional shape of an object that is moving in a known manner, and the system is a three-dimensional shape measuring system that measures the three-dimensional shape of an object that is moving in a known manner. a projection unit that projects one projection image having a plurality of projection images; a photographing unit that generates a plurality of camera images by continuously photographing the object having the known movement and onto which the one projection image is projected; For any point in the camera image, a plurality of hypothetical points are set on the camera's line of sight, and among the plurality of hypothetical points, a hypothetical point that satisfies the movement constraint based on the known movement is selected as the restoration point of the arbitrary point. It has a configuration including a restoration unit that determines as follows.

With this configuration, compared to a type of measurement system that projects only a one-dimensional line with a single projection image, such as the slit light projection method, it is possible to use a single projection image with a coding pattern as described above. In the case of projection, the three-dimensional shape of the object can be precisely measured even if the number of captured images required for measurement is relatively small. Furthermore, the orthogonality characteristic of the random pattern makes it possible to measure three-dimensional shapes that are robust against mutual reflection. In other words, in the case of a measurement system that projects only one-dimensional lines, such as the slit light projection method, it cannot be made robust against optical phenomena such as interreflection and halation. Using images can improve robustness to optical phenomena such as interreflection and halation.

In the above three-dimensional shape measurement system, the encoding pattern may be a random slit pattern in which a plurality of slits are randomly arranged.

Since the random slit pattern has few boundaries between lines and is less susceptible to the influence of adjacent lines, this configuration allows the three-dimensional shape of an object to be measured particularly well in scenes where strong optical phenomena are not occurring.

In the above three-dimensional shape measurement system, the direction in which the slit extends may be perpendicular to the known movement.

With this configuration, a sufficient brightness change pattern can be obtained in a plurality of images taken while the object is moving in a known manner.

In the three-dimensional shape measurement system described above, the random pattern may be a random dot pattern in which a plurality of dots are randomly arranged in a two-dimensional manner.

This configuration has the advantage that, in particular, robustness against interreflection is high and noise points are less likely to be reconstructed.

In the above three-dimensional shape measurement system, the restoring unit may move the plurality of hypothesis points using a motion parameter representing the known movement, and transfer the plurality of hypothesis points that have been moved to the camera image and the projection image. It may be determined that the hypothetical point at which the correlation between the set of luminance change patterns obtained by perspectively projecting the images satisfies the motion constraint is determined as the restoration point.

With this configuration, when searching for a depth point on a moving object from among multiple hypothetical points, the projection obtained at the hypothetical point with the correct depth value is used as a constraint that the object moves with known motion parameters. The shape can be restored by utilizing the fact that the set of luminance change patterns of coordinates and photographing coordinates has the maximum correlation.

The above-mentioned three-dimensional shape measurement system converts the hypothetical point to the restoring point when the initial hypothetical point set by the restoring unit and the moved hypothetical point do not satisfy an epipolar constraint and/or a depth constraint. It may further include a geometric constraint processing unit for excluding candidates.

With this configuration, if multiple hypothesis points are set on the camera's line of sight, the number of hypotheses will explode, resulting in extremely poor computational efficiency and the probability of obtaining a correct restoration point will decrease. It is possible to search for restoration points by limiting the number of hypothesis points using depth constraints.

The three-dimensional measurement system described above uses a calibration pattern instead of the object to determine a movement flow of restoration points of points on the calibration pattern obtained by the projection unit, the imaging unit, and the restoration unit. The image forming apparatus may further include a motion parameter acquisition section that obtains the motion parameter.

With this configuration, motion parameters can be estimated using the same known movement as when measuring the three-dimensional shape of an actually moving object.

A three-dimensional shape measuring method according to one aspect of the present invention is a three-dimensional shape measuring method for measuring a three-dimensional shape of an object that is moving in a known manner, and in which a coding pattern is applied to the object that is moving in a known manner. a projection step of projecting a projection image having a projection image; a photographing step of generating a plurality of camera images by continuously photographing the object having the known movement on which the one projection image is projected; and a photographing step of generating a plurality of camera images. For an arbitrary point, a plurality of hypothetical points are set on the camera's line of sight, and among the plurality of hypothetical points, a hypothetical point that satisfies the movement constraint based on the known movement is determined as the restoration point of the arbitrary point. The configuration includes a restoring step.

With this configuration as well, the three-dimensional shape of an object can be precisely measured even if the number of captured images is relatively small, and the three-dimensional shape can be measured robustly against optical phenomena such as mutual reflection and halation.

A three-dimensional shape measurement program according to one aspect of the present invention is a three-dimensional shape measurement program that measures the three-dimensional shape of an object that moves in a known manner, and is executed by an information processing device connected to a projector and a camera. This causes the projector to project one projection image having a coded pattern on the object that is moving in the known manner, and causes the camera to project one projection image having the encoded pattern on the object that is moving in the known manner. A plurality of camera images are generated by continuously photographing the object, and a plurality of hypothetical points are set on the camera's line of sight for any point in the camera image, and among the plurality of hypothetical points, the It has a configuration in which a hypothetical point that satisfies a motion constraint based on known motion is determined as a restoration point for the arbitrary point.

With this configuration as well, the three-dimensional shape of an object can be precisely measured even if the number of captured images is relatively small, and the three-dimensional shape can be measured robustly against optical phenomena such as mutual reflection and halation.

FIG. 1 is a diagram showing an application scene of a three-dimensional shape measuring system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the principle of three-dimensional shape measurement by the three-dimensional shape measurement system according to the embodiment of the present invention. FIG. 3 is a diagram for explaining processing steps of an algorithm for decoding and hypothesis point determination using motion constraints according to an embodiment of the present invention. FIG. 4 is a block diagram showing the functional configuration of the three-dimensional shape measuring system according to the embodiment of the present invention. FIG. 5 is a flowchart of a three-dimensional shape measuring method according to an embodiment of the present invention. FIG. 6 is a flowchart of motion parameter estimation according to the embodiment of the present invention. FIG. 7 is a diagram illustrating an apparatus for estimating motion parameters according to an embodiment of the present invention. FIG. 8A is a diagram showing an example of a projected image according to the embodiment of the present invention. FIG. 8B is a diagram showing an example of a projected image according to the embodiment of the present invention. FIG. 8C is a diagram showing a single slit projection image used in the conventional method. FIG. 9 is a flowchart showing the encoding pattern generation process according to the embodiment of the present invention. FIG. 10 is a flowchart of restoration according to an embodiment of the present invention. FIG. 11 is a flowchart of geometric constraint processing according to the embodiment of the present invention. FIG. 12 is a diagram showing the results of three-dimensional shape measurement in the example. FIG. 13 is a graph showing the relationship between the number of captured images and the restoration rate in the example. FIG. 14A is a diagram showing an experiment in a situation where overexposure occurs and its results. FIG. 14B is a diagram showing an experiment in a situation where mutual reflection occurs and its results. FIG. 14C is a diagram illustrating experiments on objects made of a plurality of materials including a specular reflection object and the results thereof.

Embodiments of the present invention will be described below with reference to the drawings. Note that the embodiment described below shows an example of the case where the present invention is implemented, and the present invention is not limited to the specific configuration described below. In implementing the present invention, specific configurations depending on the embodiments may be adopted as appropriate.

FIG. 1 is a diagram showing an application scene of a three-dimensional shape measuring system according to an embodiment of the present invention. The three-dimensional shape measurement system 100 includes a projector 10, a camera 20, and a three-dimensional shape measurement device 30. The projector 10 projects a projection image having a coding pattern (hereinafter also referred to as "encoding pattern image"). The camera 20 is installed so as to photograph the workpiece W, which is the object of three-dimensional shape measurement. The camera 20 generates a camera image by capturing the encoded pattern image projected by the projector 10 and reflected by the moving work W at a predetermined frame rate.

The geometrical relationship between the projector 10 and the camera 20, that is, the relationship in position and orientation, is fixed during operation, and this geometrical relationship is known in the three-dimensional shape measuring device 30. Further, the projector 10 and the camera 20 are both connected to a three-dimensional shape measuring device 30 so as to be able to communicate by wire or wirelessly. Although FIG. 1 shows an example in which the projector 10 and camera 20 are integrally configured, the projector 10 and camera 20 may be configured separately.

The three-dimensional shape measuring device 30 measures the three-dimensional shape including positional information of the workpiece W based on the encoded pattern image projected by the projector 10 and the camera image taken by the camera 20. The three-dimensional shape measuring device 30 may be realized, for example, by a general-purpose computer executing the three-dimensional shape measuring program of this embodiment.

In the example of FIG. 1, a plurality of workpieces W are placed on a belt conveyor C, and the rotation of the belt conveyor C causes the workpieces W to move in a constant direction at a constant speed. The three-dimensional shape measuring device 30 is connected to an arm robot 40. The arm robot 40 includes an arm having a plurality of joints, and performs the job of gripping a moving workpiece W. The three-dimensional shape measurement result of the workpiece W obtained by the three-dimensional shape measuring device 30 is given to the arm robot 40. The arm robot 40 operates based on the position information and three-dimensional shape information of the workpiece W. Thereby, the arm robot 40 can appropriately work on the target workpiece W, for example, it becomes possible to appropriately pick the workpiece W. Such a system is called a robot picking system.

Below, the principle of three-dimensional shape measurement by the three-dimensional shape measurement system 100 of this embodiment will be explained first, and then the configuration for performing such three-dimensional shape measurement will be explained.

FIG. 2 is a diagram illustrating the principle of three-dimensional shape measurement by the three-dimensional shape measurement system according to the embodiment of the present invention. The three-dimensional shape measuring system 100 of the present embodiment measures the three-dimensional shape of an object by using the known movement of the object as it is carried by the belt conveyor C as a geometric constraint called a movement constraint. It is assumed that the belt conveyor C moves at a constant speed in one direction, and that the object placed on it and the point P on the object also move in the same way. The motion vector of this point P is called an object flow. A camera image 210 is obtained by photographing a scene in which an object moves on a belt conveyor C while projecting a projection image 101 by a projector 10 with a camera 20.

In an environment where the movement of an object or object flow can be set strictly as in this embodiment, in the camera image 201 taken by the camera 20 and the projection image 101 projected by the projector 10, the point P on the object is epipolar. You will be moving outside the line. In other words, a new constraint axis other than the epipolar constraint is provided. The three-dimensional shape measurement system 100 uses this constraint as a geometric constraint. This geometric constraint is called a motion constraint.

FIG. 3 is a diagram for explaining processing steps of an algorithm for decoding and hypothesis point determination using motion constraints according to an embodiment of the present invention. The outline of the measurement algorithm using motion constraints is as follows. First, for an arbitrary point p0 in the camera image 201, a plurality of hypothetical points m1 to m4 that satisfy the geometric constraint condition are set in the depth direction on the camera line of sight L. Here, the camera line of sight L is a line connecting the focal point of the camera 20 and the arbitrary point P0 on the imaging surface of the camera.

Then, a plurality of hypothetical points m1 to m4 are perspectively projected onto the camera image 201 and the projection image 101. This perspective projection is performed while moving each hypothetical point m1 to m4 in the object flow direction F. At this time, the encoding pattern of the project image to be projected is changed in synchronization with the movement. Also, at this stage, hypothetical points that do not satisfy the geometric constraints are excluded. Then, a brightness change pattern (hereinafter also referred to as "decoded signal") 202 perspectively projected onto the camera image and a brightness change pattern (hereinafter referred to as "coded signal") perspectively projected onto the projector image of the plurality of hypothesis points m1 to m4. 102 is obtained, the correlation values between the encoded signal and the decoded signal of all these pairs are calculated, and the hypothetical point at which the correlation value is maximum is determined as the restoration point. shall be. In the example of FIG. 3, since the correlation of the hypothesis point m2 is the largest, the hypothesis point m2 is set as the restoration point.

FIG. 4 is a block diagram showing the functional configuration of the three-dimensional shape measuring system according to the embodiment of the present invention. The three-dimensional shape measurement system 100 includes a projection section 1, an imaging section 2, an information processing section 4, and a three-dimensional information output section 5.

The projection unit 1 is composed of a projector 10. The photographing section 2 includes a camera 20. The information processing section 4 and the three-dimensional information output section 5 may be provided in the three-dimensional shape measuring device 30, or only a part thereof may be provided in the three-dimensional shape measuring device 30, and the other part may be provided in the projector. 10 or the camera 20, or another device connected to the three-dimensional shape measuring device 30.

The projection unit 1 projects a coding pattern image based on the coding pattern. The projection unit 1 projects a coded pattern image onto an object moving by a belt conveyor C. Note that the projection unit 1 may be a device capable of projecting an arbitrary projection image, such as the projector 10, or a device such as an OHP, which projects a projection image corresponding to a prepared plate. good.

The photographing unit 2 performs photographing and obtains a camera image. The photographing unit 2 obtains a camera image sequence consisting of a plurality of consecutive camera images. In this embodiment, the photographing unit 2 detects only the luminance value of incident light and generates a monochrome image as a camera image. Further, in this embodiment, the photographing unit 2 obtains a camera image using an image sensor with a resolution of 1024×768 pixels.

The information processing section 4 includes a geometric calibration parameter acquisition section 41, a motion parameter acquisition section 42, a restoration section 44, and a geometric constraint processing section 45. The geometric calibration parameter acquisition unit 41 performs geometric calibration between the projection unit 1, the imaging unit 2, and the object flow. When the relationship between the projection section 1, the photographing section 2, and the object flow is fixed, the geometric calibration parameter is constant. The motion parameter acquisition unit 42 acquires the object flow of an object that moves in a predetermined manner by estimation. The restoring unit 44 estimates the depth (restored point) by decoding a point that satisfies the motion constraint from among the plurality of hypothetical depth points using the camera image sequence, the geometric calibration parameter, and the motion parameter. The geometric constraint processing unit 45 excludes hypothetical points that do not satisfy the geometric constraints from candidates when the restoration unit 44 estimates the depth.

The three-dimensional information output unit 5 outputs the set of restoration points obtained by the restoration unit 44 as a measurement result of the three-dimensional shape of the object.

FIG. 5 is a flowchart of a three-dimensional shape measuring method according to an embodiment of the present invention. First, the geometric calibration parameter acquisition unit 41 acquires previously obtained geometric calibration parameters (step S41). Furthermore, the motion parameter acquisition section 42 acquires relative motion parameters between the projection section 1 and the photographing section 2 and the moving object, that is, object flow (step S42).

The projection unit 1 projects a projection image onto the moving object according to the motion parameter acquired by the motion parameter acquisition unit 42 (step S43). The photographing unit 2 photographs N images of the object onto which the projection image is projected, and inputs a sequence of these camera images to the information processing unit 4 as input data (step S44).

The restoring unit 44 uses the input data from the imaging unit 2 to generate M depth hypothetical points that satisfy the geometric constraint conditions defined by the geometric constraint processing unit 45, and calculates the difference between the encoded signal 102 and the decoded signal 202. The depth hypothesis point with the maximum set score is adopted as the restoration point (step S45). Finally, the 3D information output unit 5 outputs the finally obtained three-dimensional restoration result (step S46).

FIG. 6 is a flowchart of motion parameter estimation according to the embodiment of the present invention. FIG. 7 is a diagram illustrating an apparatus for estimating motion parameters according to an embodiment of the present invention. First, a calibration board 61 having a chess pattern is installed in a device that moves in the same way as the object to be measured (step S61). The process of photographing the calibration board at the same position as the object to be measured in photographing frame n, moving to the next position, and photographing is repeated (step S62).

The motion parameter acquisition unit 42 estimates motion parameters (R, T)n from frame n-1 to frame n using the camera image sequence of the calibration board 61 (step S63). At this time, the motion parameter acquisition unit 42 estimates motion parameters from the three-dimensional average motion flow of the corners of the chess pattern on the calibration board 61.

Specifically, for example, if a calibration board with a specific inclination is moved 10 mm at a time and photographed 7 times, if you focus on a certain corner, you will notice that the 3D point at the corner of the n frame and the 3D point at the corner of the n-1 frame. Six vectors are obtained from the difference between , and by calculating the average of these vectors, the average motion flow of that corner can be calculated. By performing this process on all corners, finding the average motion flow for each corner, and further averaging them, the overall average motion flow can be found. The motion parameter acquisition unit 42 uses this entire average motion flow as an object flow, that is, a motion parameter.

Note that even for a device such as a robot arm that makes more complicated movements, [Rt] of each corner can be determined using the same method. A plurality of calibrators may be used to draw corners more closely, and in areas where there are no corners, the motion flow of surrounding corners may be interpolated by linear interpolation or the like.

8A and 8B are diagrams showing examples of projected images according to the embodiment of the present invention. FIG. 8A shows an example in which the projected image has a random dot pattern in which a plurality of dots are randomly arranged in a two-dimensional manner, and FIG. 8B shows an example in which the projected image has a plurality of slits extending in the vertical direction in a random pattern. An example is shown with a random slit pattern arranged in . In the example of FIG. 8A, the projected image has a pixel count of 1280×800 pixels, and the dot size is 10×10 pixels. In the example of FIG. 8B, the projected image has a pixel count of 1280×800 pixels, and the slit width is 10 pixels.

FIG. 8C is a diagram showing a single slit projection image used in the conventional method. The random dot pattern (FIG. 8A) and the random slit pattern (FIG. 8B), which are encoded patterns, have independent encoded information embedded in each dot or slit, so that measurements that are robust to optical phenomena can be expected. On the other hand, in the case of a single slit pattern (FIG. 8C), if the projected light is reflected multiple times due to mutual reflection, etc., it is expected that it will be difficult to identify the direct reflection, and it will be difficult to measure the correct three-dimensional point.

A random dot pattern and a random slit pattern contain encoded information compared to a single slit, and thus a random dot pattern and a random slit pattern can be said to be encoded patterns in comparison with a single slit. The random dot pattern is a pattern that has encoded information two-dimensionally in the vertical and horizontal directions, and the random slit pattern has encoded information only in either the vertical or horizontal direction (in the case of FIG. 8B, it has encoded information only in the horizontal direction). ) is a pattern with encoded information.

FIG. 9 is a flowchart showing the process of generating the encoding patterns shown in FIGS. 8A and 8B. First, for each dot (in the case of random dots) or slit (in the case of random slits) of the encoding pattern, a random number from 0 to 255 is generated based on an appropriate seed (step S81). In this embodiment, random numbers are generated using Mersenne twister or M sequence. For each dot/slit, if the generated random number is equal to or greater than the threshold value of 128, the pixel value of the dot/slit is set to 255, and if the generated random number is less than 128, the pixel value of the dot/slit is set to 255. The value is set to 0 (step S82).

FIG. 10 is a flowchart of restoration according to an embodiment of the present invention. The restoring unit 44 first captures a camera image by photographing a moving object onto which a coded pattern image is projected (step S101). Next, the restoring unit 44 sets M_p depth hypothesis points that satisfy the epipolar constraint and the depth constraint, which are the geometric constraint conditions, at the pixel p of the camera image (step S102).

The restoring unit 44 performs three-dimensional coordinate transformation of the hypothetical point m using the motion parameter (R, T)_n of each frame (step S103). The process of perspectively projecting the moved hypothetical point m_p onto the camera image and the projection image is repeated for N frames to obtain a pair of the encoded signal 102 and the decoded signal 202 (step S104) (see FIG. 3). Next, the restoring unit 44 calculates the correlation between the encoded signal 102 and the decoded signal 202 for the hypothesis point m_p using zero-mean normalized cross-correlation, and stores the obtained correlation value (step S105).

The restoring unit 44 determines whether the process of obtaining correlations with M_p hypothesis points has been completed (step S106), and if it has not been completed (NO in step S106), the process returns to step S103 and repeats the process. . If correlation values are obtained for the M_p hypothesis points (YES in step S106), the hypothesis point m_p that has obtained the maximum among the M_p correlation values is adopted as the restoration point for the pixel p (step S106). S107). The restoring unit 44 performs the above processing on all pixels of the camera image (step S108).

FIG. 11 is a flowchart of geometric constraint processing according to the embodiment of the present invention. The geometric constraint processing unit 45 acquires the hypothetical depth point m_p at the time of frame n, which has been moved from the initial position (step S111). The geometric constraint processing unit 45 determines whether the depth value of the hypothetical point m_p is within the restoration range (depth constraint) (step S112). Here, it is common practice to heuristically set the restoration range in advance based on the positional relationship between the measurement target and the camera 20 and projector 10, but the depth constraint is a constraint that focuses on the depth direction. .

If the depth value of the hypothetical point m_p is within the restoration range (YES in step S112), the geometric constraint processing unit 45 perspectively projects the hypothetical point m_p onto the camera image and the projection image (step S113). It is determined whether the point is on the epipolar line (epipolar constraint) (step S114). The epipolar constraint is determined from parameters (F matrix) obtained in advance through geometric calibration. The geometric constraint processing unit 45 determines whether the perspective projection point of the hypothetical point lies on the epipolar line of both the camera image and the projection image.

A threshold that allows some distance from the epipolar line may be used to reduce the effect of calibration errors. For example, if the number of pixels of the camera 20 is 1024 x 768 pixels and the number of pixels of the projector 10 is 1280 x 800 pixels, the threshold value th_C may be set to about 10 to 20 pixels on the camera 20 side, and An error based on the pixel size ratio (threshold th_P=th_C×(1280/1024)) may be allowed.

If the perspective-projected point is on the epipolar line (YES in step S114), that is, if both the depth constraint and the epipolar constraint are satisfied, the depth hypothetical point m_p is a hypothetical point that satisfies the geometric constraint. It is determined that (step S115). If either the depth constraint or the epipolar constraint is not satisfied (NO in step S112 or NO in step S114), the geometric constraint processing unit 45 determines that the hypothetical depth point m_p is a hypothetical point that does not satisfy the geometric constraint. (Step S116).

(Example)
An example of three-dimensional shape measurement will be described below. Using the 3D measurement system 100 shown in FIG. 2, N images are taken while moving a Lambertian model of the Stanford bunny (about 10 x 10 x 10 cm) as the object to be measured, and the ^3D hair shape is measured. We conducted an experiment to measure the results. Using a camera image and a projection image, a three-dimensional shape measurement (shape restoration) of a bunny moving with the above-mentioned known motion (object flow) was performed using a measurement algorithm using movement constraints (see FIG. 10).

FIG. 12 is a diagram showing the results of three-dimensional shape measurement in the example. In Figure 12, the upper row shows the results when a random dot pattern is used, the middle row shows the results when a random slit is used, and the lower row shows the results when a single slit pattern is used. . In addition, in Fig. 12, the left side of each row shows the results when the number of shots is N = 9 (taken every 6 mm movement), and the right side of each row shows the result when the number of shots is N = 54 (taken every 1 mm move). The results are shown for the case where the images were taken separately.

As shown in FIG. 12, it can be seen that for a Lambertian object, if a sufficient number of images are taken (on the far right side), the correct shape can be obtained even in the case of a single slit. However, in the case of a single slit, there is more noise compared to a random dot pattern or a random slit pattern.

FIG. 13 is a graph showing the relationship between the number of captured images and the restoration rate in the example. FIG. 13 shows the restoration rates for the random dot pattern, the random slit pattern, and the single slit when the number of captured images N=27, 18, and 9, respectively. Note that a large number of captured images N means that the moving speed of the object is low and/or a low frame rate of imaging by the camera 20, and a small number of captured images N means the opposite. Therefore, the fact that the restoration rate can be maintained high even if the number of captured images N is small means that the moving speed of the object can be increased and/or that the camera 20 with a low frame rate is sufficient.

As shown in Figure 13, in the case of the random dot pattern and the random slit pattern, both achieved a higher restoration rate (higher density) than the single slit pattern, and even when the number of captured images N was reduced. , the reduction in recovery rate (density) is suppressed to a small level compared to the case of a single slit. This is clear even when comparing the left column and right column in FIG. 12.

Furthermore, we conducted experiments to examine robustness against optical phenomena by deploying objects with various reflective properties. FIG. 14A shows an experiment and its results in a situation where overexposure occurs, FIG. 14B shows an experiment and its results in a situation where mutual reflection occurs, and FIG. 14C shows a plurality of objects including specularly reflecting objects. Experiments on objects made of materials and their results are shown.

As shown in FIG. 14A, it was confirmed that in a situation where halation (overexposure) occurs in a camera image, the effects of missing measurements due to overexposure can be suppressed in the case of random dot patterns and random slit patterns. Furthermore, as shown in FIG. 14B, in a situation where mutual reflection occurs, in the case of a random slit pattern, an incorrect shape is obtained at the arrow portion. Similarly, the single slit also has an incorrect shape at the arrow part. On the other hand, the overall shape of the random dot pattern, which is thought to be robust to mutual reflection, was successfully restored.

In FIG. 14C, the upper row shows the restoration results seen from the front, and the lower row shows the restoration results seen from above. In the example shown in FIG. 14, stainless steel, styrofoam, and wooden balls were placed side by side in this order and the three-dimensional shape was measured. As shown in Figure 14C, when a random slit pattern is used, an incorrect shape is measured due to mutual reflection at the arrow part, and even in the case of a single slit, due to the influence of mutual reflection between stainless steel and Styrofoam. Shape measurement could not be performed correctly. As for the random dot pattern, the shape could be restored by suppressing the influence of mutual reflection on the whole.

From the above experiments, the following findings were obtained. In other words, the random slit pattern has fewer boundaries between lines and is less susceptible to the influence of adjacent lines, and tends to restore the shape particularly well in scenes where strong optical phenomena do not occur. However, in scenes where interreflection occurs, which is susceptible to effects outside the epipolar line due to interreflection, there was a tendency for erroneous measurements and somewhat large noise lumps to be generated.

In addition, although random dot patterns tend to be more susceptible to the influence of adjacent dots during coordinate decoding, making it difficult to restore points, it was found to have the advantage of being more robust against interreflection, making noise points less likely to be restored. . The high robustness of both the random slit pattern and random dot pattern encoding patterns against optical phenomena is due to the fact that the random codes are arranged as slits or dots, which suppresses the influence of the inside and outside of the epipolar line. It is thought that it was obtained in

As described above, in this embodiment, an object moving along a known object flow is irradiated with a projection image having a random pattern from the projector 10 and photographed by the camera 20, and an arbitrary point in the camera image is By setting multiple hypothetical depth points on the camera line of sight and perspectively projecting the hypothetical points onto the camera image and projection image while moving the hypothetical points along the object flow, it is possible to calculate the brightness changes of the hypothetical points in the camera image and the projected image. The hypothetical point with the highest correlation is determined as the restoration point. As a result, the three-dimensional shape of a moving object can be measured without stopping the movement of the moving object, without using high-speed (high frame rate) projector 10 and camera 20.

Note that in the above embodiment, it is assumed that the object to be measured has reflection characteristics close to Lambertian, but this embodiment may be applied when the object is not a Lambertian reflector. When measuring in a relatively stationary state, there are areas where restoration is missing due to the influence of specularly reflected light, but this embodiment can reduce the loss by obtaining information about the object while it is moving.

Further, in the above embodiment, the encoding pattern in the projection image is constant without changing in order to obtain multiple camera images, but instead of this, different encoding patterns are used for the multiple camera images. You may project the projection image which has. In this case, the three-dimensional shape measurement system 100 may include a synchronization control unit that synchronizes the change rate of the encoding pattern of the projected image by the projection unit 1 and the frame rate of the imaging by the imaging unit 2. Furthermore, in this case, the three-dimensional shape measurement system 100 may include a coding pattern generation unit for generating a changing coding pattern.

Further, it is preferable that the dots of the monochrome encoding pattern generated by the encoding pattern generation unit 43 described above be larger because errors in geometric calibration and motion estimation can be absorbed. Furthermore, when the restoring unit 44 sets a hypothesis point and moves the hypothesis point along the object flow, the hypothesis point may correspond to a boundary between dots (boundary pixel) in the encoding pattern.

In this case, such a hypothetical point will have a pixel value between white and black. Therefore, hypothetical points that are often boundary pixels may be excluded from restoration point candidates. For example, when moving a hypothetical point over 50 frames, if 20 frames (40%) or more of them are boundary pixels, such a hypothetical point may be excluded from restoration point candidates.

Further, in the above embodiment, one projector 10 and one camera 20 are used, but a plurality of these may be used. For example, even if one projector and two cameras, or three projectors and three cameras are combined, each projector will generate its own encoding pattern image sequence, and the ZNCC of the encoded signal generated by each projector will be relative. If the ratio is low, it is possible to separate the encoded information.

Further, in the above embodiment, a known movement is achieved using a belt conveyor, that is, a horizontally moving slide stage, but in addition to or in place of this, a device that performs a known operation, for example, Known movement may be realized using a robot arm, an AGV, or the like. Since the robot arm and AGV are parameter-controlled by humans, by converting these control signals into rotational and translational information, it is possible to perform calculations equivalent to the translational movement of a hypothetical point using a slide stage. Even in a device such as a robot arm that makes complicated movements, it is possible to obtain [Rt] of each corner during calibration. It may be possible to use a plurality of calibrators to draw corners more densely, or to interpolate areas without corners by linear interpolation of the flow of surrounding corners.

1 Projection section 2 Photography section 4 Information processing section 41 Geometric calibration parameter acquisition section 42 Movement parameter acquisition section 44 Restoration section 45 Geometric constraint processing section 5 Three-dimensional information output section 10 Projector 20 Camera 30 Three-dimensional shape measuring device 40 Arm robot 100 Three-dimensional shape measurement system 101 Projected image 102 Brightness change pattern (encoded signal) of a hypothetical point perspectively projected onto the projected image
201 Camera image 202 Brightness change pattern (decoded signal) of a hypothetical point perspectively projected onto the camera image
W work

Claims (9)

A three-dimensional shape measurement system that measures the three-dimensional shape of an object with known movement,
a projection unit that projects one projection image having a coding pattern onto the object that is moving in a known manner;
a photographing unit that generates a plurality of camera images by continuously photographing the known moving object onto which the one projection image is projected;
For any point in the camera image, a plurality of hypothetical points are set on the camera's line of sight, and among the plurality of hypothetical points, a hypothetical point that satisfies the movement constraint based on the known movement is restored to the arbitrary point. a restoration part determined as a point;
A 3D shape measurement system equipped with The three-dimensional shape measurement system according to claim 1, wherein the encoding pattern is a random slit pattern in which a plurality of slits are randomly arranged. The three-dimensional shape measuring system according to claim 2, wherein the direction in which the slit extends is a direction perpendicular to the known movement. The three-dimensional shape measuring system according to claim 1, wherein the random pattern is a random dot pattern in which a plurality of dots are randomly arranged in a two-dimensional shape. The restoring unit moves the plurality of hypothesis points using a motion parameter representing the known movement,
determining that the hypothetical point at which the correlation of a set of brightness change patterns obtained by perspectively projecting the plurality of moved hypothetical points onto the camera image and the projection image satisfies the movement constraint; The three-dimensional shape measurement system according to claim 1, wherein the hypothetical point is determined as the restoration point. a geometric constraint processing unit that excludes the initial hypothesis point set by the restoration unit and the moved hypothesis point from the restoration point candidates when the hypothesis point does not satisfy an epipolar constraint and/or a depth constraint; The three-dimensional shape measurement system according to claim 1 or 2, further comprising: The motion parameter is determined by using a calibration pattern instead of the object and determining a movement flow of restoration points of points on the calibration pattern obtained by the projection unit, the imaging unit, and the restoration unit. The three-dimensional measurement system according to claim 5, further comprising a parameter acquisition section. A three-dimensional shape measurement method for measuring a three-dimensional shape of an object with known movement, the method comprising:
a projection step of projecting a projection image having a coding pattern onto the object having the known movement;
a photographing step of continuously photographing the known moving object onto which the one projection image is projected to generate a plurality of camera images;
For any point in the camera image, a plurality of hypothetical points are set on the camera's line of sight, and among the plurality of hypothetical points, a hypothetical point that satisfies the movement constraint based on the known movement is restored to the arbitrary point. a restoration step determined as a point;
A three-dimensional shape measurement method, including A three-dimensional shape measurement program that measures the three-dimensional shape of an object with known movement, and is executed by an information processing device connected to a projector and a camera.
causing the projector to project one projection image having a coding pattern onto the object having the known movement;
causing the camera to continuously photograph the known moving object onto which the one projection image is projected to generate a plurality of camera images;
For any point in the camera image, a plurality of hypothetical points are set on the camera's line of sight, and among the plurality of hypothetical points, a hypothetical point that satisfies the movement constraint based on the known movement is restored to the arbitrary point. Determine as a point,
3D shape measurement program.

Images