JP6626338B2 - Information processing apparatus, control method for information processing apparatus, and program - Google Patents

Description

  The present invention relates to an information processing device, a control method for the information processing device, and a program.

  With the development of robot technology in recent years, robots are replacing complex tasks that have been performed by humans, such as assembly of industrial products. Such a robot grasps and assembles a component with an end effector such as a hand. In order for the robot to grip a component, it is necessary to measure the relative position and orientation between the component to be gripped and the robot (hand). Such measurement of the position and orientation of the object is not only for grasping parts by the robot, but also for estimating its own position for the robot to move autonomously, and for aligning the real space (real object) with the virtual object in augmented reality. It is applied to various purposes.

  As a method for measuring the position and orientation of an object, there is a method using a model feature of a three-dimensional shape model and a measurement data feature in a two-dimensional image obtained from an imaging device such as a camera. In Non-Patent Document 1, the position and orientation of an object are measured by applying a projection image of a three-dimensional shape model of the object represented by a set of line segments to an edge which is a measurement data feature on a two-dimensional image. A method for doing so is disclosed. According to this method, a line segment in a three-dimensional shape model is projected on a two-dimensional image based on a rough position and orientation given as known information, and a control discretely arranged on the projected line segment is performed. Edges corresponding to each of the points are detected on the two-dimensional image. Based on the correspondence between the control point (model feature) and the edge (measurement data feature) obtained in this way, the sum of squares of the distance between the projected image of the line to which the control point belongs and the corresponding edge on the image The final position and orientation can be obtained by correcting the approximate position and orientation so that is minimized.

T. Drummond and R. Cipolla, “Real-time visual tracking of complex structures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.7, pp.932-946, 2002. RY Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE Journal of Robotics and Automation, vol.RA-3, no.4, 1987. (Referred to in the form) Baba, Asada, Amano, "A Trial of Image Synthesis by Calibrated Computer Graphics: Generation and Verification of Arbitrary Focus Image Based on Camera Calibration", Transactions of Information Processing Society of Japan 39 (7), 2180-2188, 1998 (refer to the embodiment) Be done) Y. Hel-Or and M. Werman, “Pose estimation by fusing noisy data of different dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.17, no.2, pp.195-201, 1995. Referenced by)

  However, in the above-described method, when an erroneous correspondence occurs between a model feature (control point) and a measurement data feature (edge), the estimation process of the position and orientation of the object fails, or the estimation of the position and orientation is performed. There is a problem that accuracy is reduced. The present invention has been made in view of the above problems, and has as its object to realize more accurate association between a model feature of a shape model and a measurement data feature on a two-dimensional image.

An information processing apparatus according to one embodiment of the present invention for achieving the above object has the following configuration. That is,
Based on a shape model representing the shape of the measured object, a prediction unit that predicts image degradation of the measured object that occurs in an image of the measured object captured by an imaging unit,
Searching means for searching for a measurement data feature corresponding to a model feature of the shape model, for a two-dimensional image obtained by imaging the object to be measured by the imaging means;
Evaluation means for evaluating image deterioration using the two-dimensional image for the measurement data feature searched by the search means;
Calculation means for calculating the accuracy of the correspondence between the model feature and the measurement data feature based on the image deterioration estimated by the prediction means and the image deterioration evaluated by the evaluation means,
A matching unit that matches the shape model with a measured object in the two-dimensional image based on the accuracy of the correspondence between the model feature and the measurement data feature.

  According to the present invention, it is possible to realize a more accurate association between a model feature of a shape model of a measured object and a measurement data feature on a two-dimensional image. Therefore, for example, the position and orientation of the measured object can be estimated with higher accuracy.

FIG. 2 is a diagram illustrating a configuration of an information processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating a three-dimensional shape model of a measurement target according to the embodiment. 9 is a flowchart illustrating a processing procedure for measuring a position and orientation according to the first embodiment. FIG. 4 is a view for explaining edge detection from an image. 5 is a flowchart illustrating a procedure for calculating a predicted image deterioration value according to the first embodiment. 5 is a flowchart illustrating a procedure for calculating an image deterioration evaluation value of a two-dimensional image according to the first embodiment. 7 is a flowchart illustrating a processing procedure for calculating an association weight according to the first embodiment. 5 is a flowchart illustrating a processing procedure for calculating a position and orientation according to the first embodiment. FIG. 4 is a diagram illustrating a relationship between a projected image of a line segment and a detected edge. FIG. 13 is a diagram illustrating a functional configuration example of an information processing apparatus according to a third embodiment. The figure which shows the example of a structure of the robot system in 4th Embodiment.

  Hereinafter, some preferred embodiments of the present invention will be described with reference to the accompanying drawings.

<First embodiment>
In the first embodiment, a local line feature is associated with a measurement data feature by using the magnitude of blur and blur of an image obtained as a two-dimensional image. That is, for each local line feature, the magnitude of blur and blur obtained in the two-dimensional image is predicted, compared with the magnitude of blur and blur actually measured from the two-dimensional image, and the values of features having similar values are shown. The degree of contribution is made higher and associated. In the first embodiment, the size of blur and blur is determined by simulation based on the shape represented by the shape model imitating the measured object and the relative moving direction and speed of the imaging device and the measured object determined in advance. By calculating, the magnitude of blur or blur is predicted.

  FIG. 1A is a block diagram illustrating a hardware configuration example of an information processing apparatus 1 according to the present embodiment. 1A, the CPU 101 implements various controls in the information processing apparatus 1 by executing a program stored in the ROM 102 or the RAM 103. The ROM 102 stores various programs and various data including programs executed when the CPU 101 is started. The RAM 103 functions as a main memory of the CPU 101. For example, an application program read from the external storage device 105 is loaded on the RAM 103 for execution by the CPU 101. The display unit 104 performs various displays under the control of the CPU 101. The external storage device 105 is configured by, for example, a hard disk or the like, and stores application programs, data of shape models, and two-dimensional images to be processed. The interface 106 communicates with an external device. For example, the imaging device 20 is connected to the interface 106, and receives a two-dimensional image from the imaging device 20. The bus 107 enables the components described above to communicate with each other.

  FIG. 1B is a block diagram illustrating a functional configuration example of the information processing apparatus 1 according to the present embodiment. Each function illustrated in FIG. 1B is realized by the CPU 101 executing a program stored in the ROM 102 and / or a program expanded from the external storage device 105 to the RAM 103. It is apparent that some or all of the functional units may be realized by dedicated hardware. 1B, the information processing apparatus 1 includes a model input unit 110, a predicted value calculation unit 120, an image input unit 130, a search unit 140, an evaluation value calculation unit 150, an accuracy calculation unit 160, and a collation unit 170. . Further, the configuration shown in FIG. 1 is a configuration that is an application example of the information processing apparatus of the present invention. Hereinafter, each functional unit of the information processing apparatus 1 will be described.

  The model input unit 110 inputs a shape model (in the present embodiment, a three-dimensional shape model stored in the external storage device 105) representing the shape of the measured object. The shape model according to the present embodiment includes local three-dimensional line segment information (hereinafter referred to as a local line feature and a local line feature) on an object contour composed of three-dimensional positions and three-dimensional line segment directions as shown in FIG. ). The shape model is provided to the search unit 140 via the model input unit 110.

  The predicted value calculation unit 120 calculates a predicted image deterioration value of a two-dimensional image of the measured object. In the present embodiment, the size D of the blur due to defocus and the image D are calculated using a three-dimensional shape model of the measured object, approximate position / posture values, and internal parameters of the imaging device that has captured the two-dimensional image to be processed. The magnitude B of the blur caused by the parallel movement on the plane is defined as the predicted image deterioration value. The details of the process of acquiring the blur size D, the blur size B, and the predicted image deterioration value will be described later.

  The image input unit 130 acquires a two-dimensional image of the measured object. The acquired two-dimensional image may be a gray image or a color image. In the present embodiment, a two-dimensional image is acquired from an external storage device (not shown) that stores a two-dimensional image captured in advance. The unit 130 may directly acquire from the imaging device 20. In any case, the two-dimensional image is held in association with the internal parameters of the imaging device 20 that has captured the image.

  The search unit 140 searches the two-dimensional image input from the image input unit 130 for a candidate for the local line feature of the shape model of the measured object input from the model input unit 110. This search is performed using the approximate values of the positions and orientations of all the local line features constituting the shape model and the internal parameters of the imaging device that has captured the two-dimensional image. In the present embodiment, an edge feature in a two-dimensional image is detected as a corresponding candidate (measurement data feature). The edge feature is a point at which the density gradient has an extreme value detected by the differential filter. FIG. 4 is a diagram illustrating edge detection in the present embodiment. Using the approximate position and orientation of the measured object and the calibrated internal parameters of the imaging device 20 that has captured the two-dimensional image, a projected image of each line constituting the shape model on the two-dimensional image is calculated.

  As shown in FIG. 4A, the projected image of the line segment is also a line segment on the two-dimensional image (projected line segment 401). Next, control points 402 are set on the projected line segments 401 so as to be at equal intervals on the two-dimensional image, and one for each set control point 402 in the normal direction of the projected line segments 401. Dimensional edge detection is performed (FIG. 4A). That is, edge detection is performed along the search line 403 having the normal direction of the projected line segment 401 through the control point 402 set on the projected line segment 401. Since the edge 404 of the two-dimensional image is detected as an extreme value of the density gradient of the pixel value, as shown in FIG. May be detected. In the present embodiment, all detected edges are held as hypotheses.

  The evaluation value calculation unit 150 estimates the size of blur and the size of blur of the two-dimensional image obtained by the image input unit 130. In the present embodiment, by applying a function representing a change in luminance of an edge when blurring and blurring occurs to a change in luminance of a pixel along a direction orthogonal to an edge in a two-dimensional image, each of the edge features is obtained. An image deterioration evaluation value is calculated. The process of detecting an edge in a two-dimensional image is the same process as the edge detection by the search unit 140. Details of the processing will be described later.

  The accuracy calculation unit 160 calculates the accuracy of each of the edge features held by the search unit 140 as a hypothesis of the association with the local line feature of the shape model of the measured object. The calculation of the accuracy is performed based on the image deterioration prediction value obtained by the prediction value calculation unit 120 and the image deterioration evaluation value obtained by the evaluation value calculation unit 150. The details of the processing will be described later. . The matching unit 170 determines the position and orientation of the measured object based on the edge features on the two-dimensional image corresponding to each local line feature in the shape model of the measured object and the accuracy calculated by the accuracy calculating unit 160. calculate. Details of the process of calculating the position and orientation by the matching unit 170 will be described later.

  FIG. 3 is a flowchart showing a processing procedure for calculating the position and orientation of the measured object on the two-dimensional image in association with the edge features and the local line features in the first embodiment. First, the model input unit 110 inputs a shape model of a measured object and takes it into the information processing device 1 (step S301). Next, the prediction value calculation unit 120 calculates an image deterioration prediction value of the two-dimensional image (Step S302). In calculating the predicted image deterioration value, the approximate values of the positions and orientations of all the local line features constituting the shape model of the measured object, the relative moving direction and speed between the imaging device and the measured object, and the two-dimensional image The internal parameters of the imaging device at the time of acquisition are used. In the present embodiment, the approximate position and posture values of the object are used as approximate values of the position and posture of the local line feature. Details of the process of calculating the predicted image deterioration value of the two-dimensional image will be described later with reference to FIG.

  Next, the image input unit 130 inputs a two-dimensional image and takes it into the information processing device 1 (step S303). The search unit 140 searches for a candidate corresponding to the local line feature of the shape model from the two-dimensional image input by the image input unit 130 (step S304). The method of searching for a correspondence candidate is as described above with reference to FIG. 4. The approximate values of the positions and orientations of all the local line features of the shape model input by the model input unit 110 and the imaging apparatus for capturing a two-dimensional image Is made based on the internal parameters of

  The evaluation value calculation unit 150 estimates the magnitude of blur and blur of the two-dimensional image obtained by the image input unit 130, and calculates an image deterioration evaluation value (step S305). More specifically, the evaluation value calculation unit 150 performs edge detection from the two-dimensional image obtained by the image input unit 130. Then, the evaluation value calculation unit 150 applies a function representing the luminance change of the edge when blurring / blurring occurs to the luminance change of the pixel along the direction orthogonal to the detected edge feature, and The blur / blur width is estimated to obtain an image deterioration evaluation value. Details of this processing will be described later. Next, the accuracy calculation section 160 uses the image deterioration prediction value obtained by the prediction value calculation section 120 and the image deterioration evaluation value obtained by the evaluation value calculation section 150 to obtain the local line feature obtained by the search section 140. Then, the accuracy of the correspondence between the edge feature and the edge feature is calculated (step S306). Details of the accuracy calculation process will be described later with reference to FIG. Then, the matching unit 170 calculates the position and orientation of the measurement target object by applying the calculated edge feature having the maximum accuracy to the local line feature of the shape model (step S307).

  FIG. 5 is a flowchart illustrating the process of calculating the predicted image deterioration value in step S302. In the present embodiment, image deterioration due to image spreading due to blurring and / or blurring of a two-dimensional image of a model feature is predicted. Hereinafter, the image deterioration prediction value calculation processing by the prediction value calculation unit 120 will be described with reference to the flowchart in FIG. First, the predicted value calculation unit 120 was used to capture the approximate values of the position and orientation of all the local line features of the shape model, the relative movement direction and speed between the imaging device and the measured object, and the capturing of a two-dimensional image. An internal parameter is obtained (step S501). The relative movement direction and speed between the imaging apparatus and the object to be measured in the present embodiment are based on, for example, design data when the object to be measured is arranged on a device (for example, a belt conveyor) that performs a translational motion in one axis direction. Then, the moving amount and moving direction of the translational motion are obtained.

Next, the predicted value calculation unit 120 selects a local line feature from the shape model to calculate the predicted image deterioration value σ ₀ (step S502). Then, the predicted value calculation unit 120 calculates the size D of the blur predicted in the selected one local line feature by using the following equation (Equation 1) (step S503).

In the formula [Number 1] above, f is the focal length of the taking lens, L _o is the distance focus position of the virtual viewpoint, L _n from the virtual viewpoint to the model points, F is the lens aperture, [Delta] d is the pixel size Represents The focal length f and the aperture value F of the taking lens are obtained by referring to the specifications of the imaging device to be used. Ln is calculated from the approximate position and orientation input in step S501. The internal parameters such as the focal length, principal point position, and lens distortion parameters of the camera lens used for shooting are calibrated in advance by referring to the specifications of the equipment to be used or by the method described in Non-Patent Document 2. Obtained by doing.

  Next, the predicted value calculation unit 120 calculates the magnitude of the blur predicted in the same local line feature as the magnitude of the blur predicted in step S503 (step S504). In the present embodiment, the movement amount of each local line feature on the two-dimensional image during the exposure time is used as the blur amount. Specifically, the amount of movement of the projected local line feature on the image plane during the exposure time, which is detected from an image obtained by projecting the shape model into a two-dimensional image, is used. In step S504, the Jacobian of the projected local line feature obtained in the model projection image is calculated, and based on the Jacobian of the projected local line feature and the relative moving direction and speed between the imaging device and the object to be measured input in step S502. , The magnitude of the blurring of the projected local line feature is calculated.

The Jacobian of the projected local line feature is a value representing the rate of change of the projected local line feature on the image when the parameter of the position and orientation 6 degrees of freedom changes minutely. Based on the approximate position and orientation s of the object to be measured, a blur of the corresponding edge feature in the projection local line feature (du, dv), the normal direction of projection local line feature _{(n u,} n v) (unit vector ), The signed distance err _2D between the correspondences can be calculated from the following equation [Equation 2]. Note that du = u'-u and dv = v'-v.

Here, the position and orientation s of the measured object are six-dimensional vectors, and three elements (s ₁ , s ₂ , s ₃ ) representing the position of the target object and three elements (s ₄ , s) representing the orientation _5, consisting of _{s 6).} The three elements representing the posture are represented by, for example, an Euler angle or a three-dimensional vector whose direction represents a rotation axis passing through the origin and whose norm represents the rotation angle. The Jacobian of the projected local line feature is calculated as shown in the following Expression [3] by partially differentiating the distance err _2D between correspondences with each parameter of the position and posture s.

Ｂはスカラーであり、露光時間中に画像平面上で、投影局所線特徴の２次元位置が移動する量を表す。以上の処理を全ての局所線特徴に対して行い、全ての局所線特徴に対してブレ量を算出する。 As described above, the Jacobian of the projected local line feature selected in step S502 is calculated. Using Jacobian of the projection local line feature, the object during the exposure time t _i of the 2-dimensional image is generated by moving a relative position and orientation velocity V, between the edge feature in the local line feature and the two-dimensional image Can be calculated by the following equation [Equation 4].

B is a scalar and represents the amount by which the two-dimensional position of the projected local line feature moves on the image plane during the exposure time. The above processing is performed for all the local line features, and the blur amount is calculated for all the local line features.

Next, the prediction value calculation unit 120 calculates an image deterioration prediction value σ ₀ of the local line feature selected in step S502 from the blur size D calculated in step S503 and the blur size B calculated in step S504, It is calculated using the following equation [Equation 5] (step S505).

For all the local line feature to calculate the image deterioration prediction value sigma ₀ is completed, it repeats the processing of steps S502～S505 (step S506). When the calculation of the predicted image deterioration value σ ₀ has been completed for all the local line features in this manner, the present process is terminated.

  Next, the process of evaluating the magnitude of blur and blur from the two-dimensional image in step S305 will be described. In the present embodiment, image deterioration due to the spread of the image due to blur and / or blur in the two-dimensional image of the measurement data feature is evaluated. FIG. 6 is a flowchart illustrating a process of calculating an image deterioration evaluation value performed by the evaluation value calculation unit 150 in step S305. The evaluation value calculation unit 150 selects one of the edge features that are the corresponding candidates detected in step S304 (step S601), and calculates an image degradation evaluation value for the selected edge feature (step S602).

数６において、ｘ_０、ｙ_０はエッジのある着目画素位置、ｒは着目画素位置からの距離、θはエッジの法線方向、ｔは着目画素からの探索範囲、σはボケの大きさとブレの大きさを統合した値（画像劣化評価値）を示す。なお、ｔは任意の正の値である。 The calculation of the image deterioration evaluation value by the evaluation value calculation unit 150 will be described. The evaluation value calculation unit 150 calculates the magnitude of blur and blur of the edge feature from the position of the edge feature and the normal direction of the edge selected in step S601. As an edge model used in the present embodiment, fitting is performed using an error function erf as represented by the following equation [Equation 6], and σ is determined as the magnitude of blur and blur.

In Equation 6, x ₀ and y ₀ are the pixel position of interest having an edge, r is the distance from the pixel position of interest, θ is the normal direction of the edge, t is the search range from the pixel of interest, and σ is the magnitude and blur of blur. Are shown as values (image deterioration evaluation values) obtained by integrating the sizes of the images. Note that t is an arbitrary positive value.

数７において、Ｉ（ｘ，ｙ）は座標（ｘ，ｙ）における撮影画像の輝度値を示す。 The image deterioration evaluation value σ is estimated by minimizing an evaluation function E represented by the following equation [Equation 7] by a steepest descent method, a repetitive operation by the Levenberg-Marquardt method, or the like.

In Expression 7, I (x, y) indicates the luminance value of the captured image at the coordinates (x, y).

  Steps S601 and S602 are repeated for all the edge features detected in step S304 until the above-described evaluation of the magnitude of blur or blur is completed (step S603). When the evaluation of the magnitude of blur / blur is completed for all the edge features detected in step S304, this processing ends.

  FIG. 7 is a flowchart showing the accuracy calculation processing in step S306. In the present embodiment, the closer the image degradation prediction value that is the degree of image degradation calculated by the prediction value calculation unit 120 and the image deterioration evaluation value that is the degree of image degradation evaluated by the evaluation value calculation unit 150, the higher the value. The accuracy is calculated as follows. First, the accuracy calculation unit 160 selects one local line feature in the shape model of the measured object (step S701). Next, the accuracy calculation unit 160 selects one edge feature (candidate for correspondence) searched by the search unit 140 in step S304 as a candidate for association with the local line feature selected in step S701 (step S702). ), And calculate the accuracy (step S703). Hereinafter, the calculation of the accuracy will be described.

The accuracy calculation unit 160 acquires an image deterioration evaluation value σ corresponding to the edge feature selected in step S702 with reference to the calculation result of the evaluation value calculation unit 150. In addition, the accuracy calculation unit 160 acquires the predicted image deterioration value σ ₀ of the local line feature selected in step S701. The accuracy calculation unit 160 calculates the accuracy T from the predicted image deterioration value σ ₀ and the image deterioration evaluation value σ according to the following equation [Equation 8].

  Steps S702 to S703 are repeated until the accuracy T has been calculated for all the corresponding candidates (edge features) searched by the search unit 140 for the local line feature selected in step S701. When the calculation of the accuracy T is completed for all the corresponding candidates (edge features), the process proceeds to step S705. The accuracy calculation unit 160 repeats the above processing (steps S701 to S704) until all the local line features in the shape model of the measured object have been completed. When the processing has been completed for all the local line features, this processing ends.

  FIG. 8 is a flowchart illustrating a processing procedure of calculating the position and orientation by the matching unit 170 in step S307. The collation unit 170 uses the Gauss-Newton method, which is one of the nonlinear optimization techniques, to outline the position and orientation (hereinafter, represented by a 6-dimensional vector s) of the measurement target object so that the shape model is applied to the measurement data. The value is repeatedly corrected by an iterative operation.

  First, the matching unit 170 performs an initialization process (step S801). In the present embodiment, the approximate position and posture values of the object to be measured are input to the matching unit 170 as approximate values. This approximate value is the same as that used in step S302. Next, the matching unit 170 selects an edge feature to be matched with the local line feature of the model (Step S802). For the selection, the accuracy T of the correspondence between the edge feature and the local line feature calculated by the accuracy calculation unit 160 is used. For example, when a plurality of edge features are detected as a correspondence hypothesis with respect to one local line feature, the correspondence candidate (edge feature) having the highest accuracy T is associated with the local line feature and the subsequent processing is performed. I do.

  Subsequently, the matching unit 170 calculates a coefficient matrix and an error vector for calculating the position and orientation (step S803). Here, each element of the coefficient matrix is a first-order partial differential coefficient relating to each element of the position and the attitude when the distance between the point and the straight line on the image is a function of the position and the attitude. The error vector is a signed distance on the image between the projected line segment and the detected edge for the edge, and a signed distance in the three-dimensional space between the point and the model surface for the point cloud data.

Next, derivation of a coefficient matrix will be described. FIG. 9 is a diagram illustrating a relationship between a line segment 901 which is a projected image and a detected edge 902. In FIG. 9, the horizontal direction and the vertical direction of the image are defined as u axis and v axis, respectively. A position (u ₀ , v ₀ ) of a certain control point 903 (a point obtained by dividing each projected line segment at equal intervals on the image) on the image is represented on the image of the line segment 901 to which the control point 903 belongs. Is represented as a gradient θ with respect to the u-axis. The inclination θ is calculated by projecting the three-dimensional coordinates of both ends of the line segment on an image based on the position and orientation s of the measured object, and calculating the inclination of a straight line connecting the coordinates of both ends on the image. The normal vector on the image of the line segment 901 is (sin θ, −cos θ). The coordinates of the corresponding point 904 of the control point 903 on the two-dimensional image are (u ′, v ′). Here, a point (u, v) on a straight line (the edge 902 indicated by a broken line in FIG. 9) that passes through the point (u ′, v ′) and has an inclination θ is represented by the following equation [Equation 9]. (Θ is a constant). Here, d = u′sin θ−v′cos θ (constant).

The position of the control point on the image changes depending on the position and orientation of the measurement target object. The degree of freedom of the position and orientation of the measurement target object is six degrees of freedom. That is, the position and orientation s is a six-dimensional vector, and includes three elements representing the position of the measurement target object and three elements representing the orientation. The three elements representing the posture are represented by, for example, an Euler angle or a three-dimensional vector whose direction represents a rotation axis passing through the origin and whose norm represents the rotation angle. The image coordinates (u, v) of the control point, which changes depending on the position and orientation, can be approximated by the following Taylor expansion in the vicinity of (u ₀ , v ₀ ) by the following Taylor expansion. However, in Expression 10, Δs _i (i = 1, 2,..., 6) represents a minute change of each component of s.

位置及び姿勢の概略値によって算出された各制御点の奥行き（視点から制御点までの距離＝ｚ）を、画像上の誤差に乗じることによって３次元空間中の誤差に変換する。 Assuming that the difference between the approximate value of the position and orientation and the actual position and orientation is not so large, it can be assumed that the position on the image of the control point obtained by the correct s is on the straight line represented by Expression [9]. By substituting u and v approximated by Equation [Equation 10] into Equation [Equation 9], Equation [Equation 11] is obtained.

The depth of each control point (distance from the viewpoint to the control point = z) calculated based on the approximate values of the position and orientation is converted into an error in a three-dimensional space by multiplying the error on the image.

The simultaneous equations to be solved are as shown in Equation [12].

線形連立方程式の係数行列Ｊを算出するための偏微分係数の算出は、例えば非特許文献２に開示されている方法によって行うことができる。 In the equation [Equation 12], z ₁ , z ₂ ... Represent the depth of each edge. Here, equation [12] is represented as equation [13]. In [Equation 13], J is a coefficient matrix, E is an error vector, and Δs is a correction value.

The calculation of the partial differential coefficient for calculating the coefficient matrix J of the linear system of equations can be performed by, for example, a method disclosed in Non-Patent Document 2.

Next, the matching unit 170 calculates the position and orientation correction value Δs using the generalized inverse matrix (J ^T · J) ^-1 · J ^T of the matrix J based on the equation [Equation 13]. (Step S804). However, the edge or point cloud data has many outliers due to erroneous detection or the like, so the following robust estimation method is used. In general, the value of the error vector on the right side of Expression [12] is large for edge or point cloud data that is an outlier. Therefore, a small weight is given to data with a large absolute value of the error, and a large weight is given to data with a small error. The weight is given by, for example, a Tukey function as shown in Expression [Equation 14].

上記式［数１４］において、ｃ_１、ｃ_２は定数である。なお、重みを与える関数はＴｕｋｅｙの関数である必要はなく、例えばＨｕｂｅｒの関数など、誤差が大きいデータには小さな重みを与え、誤差が小さいデータには大きな重みを与える関数であればなんでもよい。

In the above expression [14], c ₁ and c ₂ are constants. The weighting function does not need to be a Tukey function, but may be any function that gives a small weight to data with a large error and a large weight to data with a small error, such as a Huber function.

The weights corresponding to each measurement data (edge or point group data) and w _i. Here, a weight matrix W is defined as in Expression [15].

式［数１６］を式［数１７］のように解くことにより補正値Δｓを求める。

Weight matrix W is all but diagonal elements are square matrices of 0, the diagonal components enters the weight w _i calculated in accordance with Equation [Expression 14]. Using this weight matrix W, equation [13] is transformed as equation [16].

The correction value Δs is obtained by solving Expression [16] as in Expression [17].

Returning to FIG. 8, the collation unit 170 corrects the approximate value of the position and orientation using the position and orientation correction value Δs calculated in step S804 (step S805).

  Subsequently, the matching unit 170 performs a convergence determination, and ends if it has converged, and returns the process to step S802 if it has not converged (step S806). In the convergence determination, it is assumed that the convergence has occurred when the correction value Δs is almost 0, or when the sum of squares of the error vector hardly changes before and after the correction (when the difference before and after the correction becomes equal to or less than a predetermined threshold). judge. In the above, an example using the Gauss-Newton method as the nonlinear optimization method has been described. However, the nonlinear optimization technique is not limited to this, and other nonlinear optimization techniques such as the Newton-Raphson method, the Levenberg-Marquardt method, the steepest descent method, and the conjugate gradient method may be used.

  The shape information held as the model of the measured object in the present embodiment may be two-dimensional or three-dimensional geometric information representing the target shape, and there is no particular limitation on the expression format. For example, when expressing two-dimensional geometric information, it may be represented by a set of simple two-dimensional points or a set of two-dimensional lines. In the case of representing three-dimensional geometric information, a polygonal shape represented by a set of simple three-dimensional points, a set of three-dimensional lines representing edges, and a set of three-dimensional points It may be represented by information.

  Further, as the relative movement direction / speed between the imaging device and the measured object input in step S501, information on the moving direction and the movement amount assuming that the measured object performs a translational movement in a certain direction is input. Although an example has been described, the present embodiment is not limited to this. For example, the above-described sensor may be attached to the measured object, and the speed of the relative position and orientation between the imaging device and the measured object may be calculated from the information. In a situation where the imaging device moves, the movement of the imaging device may be acquired. For example, when an imaging device is attached to a robot, the robot motion information may be used as an input of the relative movement direction / speed between the imaging device and the measured object. Alternatively, a physical sensor, such as a magnetic sensor or an ultrasonic sensor, that measures the position and orientation with six degrees of freedom may be attached to the imaging device, and the information may be input. In the present embodiment, any method may be used as long as the relative position and orientation speed of the imaging device and the measured object can be calculated, and there is no particular limitation on the selection of means and devices.

  Further, the amount estimated as the predicted image deterioration value by the predicted value calculation unit 120 is not limited to the above information. For example, any value may be used as long as it can represent the amount of deterioration of the two-dimensional image due to relative movement between the imaging device and the measured object, and there is no particular limitation on the calculation method and expression. For example, the magnitude of the blurring of the edge feature of the two-dimensional image of the object to be measured may be represented as a movement amount in the three-dimensional space calculated by back-projecting the two-dimensional image into the three-dimensional space. Further, a point spread function (PSF) created based on the size D of the blur and the size B of the blur may be used as the image deterioration evaluation value. Further, in the above-described embodiment, the predicted value is obtained in consideration of the influence of both blur and blur, but the predicted value may be obtained in consideration of the influence of only one of them.

  In the present embodiment, an edge feature is used as a measurement data feature that is a candidate for correspondence in the search unit 140. However, the present invention is not limited to this. For example, any feature that can specify a position on a two-dimensional image such as a point feature may be used. Instead of using only a specific type of feature, a correspondence between a plurality of types of features (for example, points and edges) and a model feature may be detected.

  In addition, the method of estimating the magnitude of blur or blur from a single image in the evaluation value calculation unit 150 is not limited to the above-described method. For example, assuming that the spread of blur and blur is spread in a Gaussian distribution, the blur distribution is obtained from the standard deviation obtained when fitting the luminance distribution of the edge portion where the blur and blur occurs to the Gaussian function using the least squares method. -The magnitude of blur may be obtained. In this embodiment, the image deterioration evaluation value is calculated for each edge feature. However, the present invention is not limited to this. For example, a two-dimensional image may be divided into rectangular partial areas, and an image deterioration evaluation value may be calculated for each partial area.

  In addition, although the accuracy calculation unit 160 calculates the accuracy using Expression [8], the method of calculating the accuracy is not limited to this. The function for calculating the accuracy is such that when the difference between the image deterioration evaluation value and the image deterioration prediction value is large, the accuracy is small, and when the difference between the image deterioration evaluation value and the image deterioration prediction value is small, the accuracy is calculated to be large. Function. For example, a Gaussian function, a Tukey function, a Huber function, or the like can be used.

  In addition, the matching unit 170 matches the pair of the local line feature and the edge feature with the highest accuracy, but the present invention is not limited to this. It is sufficient that a pair with the calculated accuracy is selected with priority.For example, even if the pair with the highest accuracy is selected, if the accuracy is below a certain threshold, it is not used for matching. Is also good. Similarly, even when only one pair of the local line feature and the edge feature is detected, if the accuracy is equal to or less than the threshold value, the pair may not be used for matching. Furthermore, in the present embodiment, the collation unit 170 acquires the position and orientation of the measured object, but the present invention is not limited to this. For example, the positions of local line features on a two-dimensional image are held in advance in a plurality of positions and orientations, and the local line features and edge features are collated by pattern matching to roughly identify the position and orientation, or May be specified.

  In steps S302, S501, and S801, the approximate position and orientation of the object are given as approximate values of the position and orientation of the model of the measured object. However, the method of inputting the approximate position and posture is not limited to this. For example, the measurement value obtained by the previous (previous time) measurement may be used as the approximate position and orientation. In addition, the speed and angular velocity of the object are estimated by a time-series filter based on the measurement of the past position and orientation, and the current position and orientation are estimated from the past position and orientation and the estimated speed / acceleration. May be used as the position and orientation of the image. If the position and orientation of the object can be measured using another sensor, the output values of the sensor may be used as approximate values of the position and orientation. For example, a magnetic sensor that measures the position and orientation by detecting the magnetic field generated by the transmitter with a receiver attached to the object, and the position and orientation by shooting a marker placed on the object with a camera fixed to the scene There is an optical sensor for measuring. In addition, any sensor may be used as long as it measures the position and orientation with six degrees of freedom. If the approximate position and orientation of the object is known in advance, the values may be used as approximate values.

  As described above, in the first embodiment, the accuracy of association is calculated from the magnitude of blur and blur of an image obtained from a two-dimensional image of the measured object, and the model feature is compared with the measurement data feature. Has been described. This makes it possible to correctly collate the model features with the measurement data features.

<Modification 1>
In the first embodiment, the matching unit 170 selects and matches a pair of a local line feature and an edge feature with high accuracy. However, the present invention is not limited to this. The calculated high-precision pair may be preferentially used for the matching process.For example, the degree of accuracy of the correspondence between each local line feature and the edge feature is determined for each edge feature when performing position and orientation estimation. May be set as the weight, and the matching may be performed based on the weight. The weighting method will be described. First, the accuracy T of the correspondence between the local line feature and the edge feature is given to the diagonal components of the weight matrix W calculated by Expression [15]. Using the weight matrix W to solve the equation [Equation 17], the position and orientation correction value Δs is obtained, and the approximate values of the position and orientation are corrected. According to the above-described method, it is possible to preferentially collate the calculated high-accuracy pair.

  Although the accuracy T itself is given as a value to be given to the diagonal component of the weight matrix W, the present invention is not limited to this. Any pair that has a high degree of accuracy gives a large weight, and a pair that has a low degree of accuracy has a small weight.A ratio obtained by dividing all the degrees of accuracy by the maximum value of the degrees of accuracy is given as a weight. Is also good.

  As described above, in the first modification, the accuracy of the association is set as the weight of each data feature from the magnitude of blur and blur of the image obtained from the two-dimensional image of the measured object, and the local line feature is set. The method of matching the edge feature with the edge feature has been described. Thereby, it is possible to perform the matching with high accuracy.

<Second embodiment>
In the first embodiment, the magnitude of blur and blur occurring on a two-dimensional image is predicted from a model, and the accuracy is calculated by comparing the magnitude with the magnitude of blur and blur of an image obtained in the two-dimensional image. The method of using for the association with the local line feature has been described. In the second embodiment, an image deterioration prediction value of each local line feature is calculated from a two-dimensional image in which the magnitude of blur and blur is calculated in advance, and the calculated value is compared with the magnitude of blur and blur of an actually measured image to obtain a corresponding accurate value. The degree is calculated, and the features having the highest accuracy are associated with each other. Hereinafter, the second embodiment will be described. The configuration of the information processing apparatus and the processing procedure for measuring the position and orientation in the second embodiment are the same as those in the first embodiment (FIGS. 1 and 3).

In the second embodiment, in step S302, the predicted value calculation unit 120 estimates the size of blur and the size of blur from a two-dimensional image obtained before (previous time). The method of estimating the magnitude of blur and blur from the two-dimensional image is the same as the method of estimating the magnitude of blur and blur of the edge feature portion by the evaluation value calculation unit 150 of the first embodiment. However, the magnitude of blur / blur is measured at the projection position of the model feature instead of the edge feature portion. For example, the predicted value calculation unit 120 projects a shape model (for example, a local line feature) of the measured object on the two-dimensional image obtained last time using the approximate position and orientation of the measured object. Then, an image deterioration evaluation value is calculated for the projection position by the method described in steps S601 and S602, and this is set as an image deterioration prediction value. Therefore, the predicted image degradation value σ ₁ of the local line feature obtained by the predicted value calculation unit 120 is determined by blurring or blurring at the coordinate position of the model feature (for example, the local line feature) when the shape model is projected on the two-dimensional image. Is the size of

The accuracy calculating section 160 is searched by the searching section 140 from the image deterioration predicted value obtained by the predicted value calculating section 120 in step S302 and the image deterioration evaluation value obtained by the evaluation value calculating section 150 in step S305. The accuracy is calculated for each of the edge features. The edge feature searched by the search unit 140 is a candidate for the correspondence of the local line feature of the model of the measured object, and is stored in the memory as a hypothesis (Step S304). The accuracy T is calculated using the image degradation evaluation value σ of the edge feature on the two-dimensional image held as a hypothesis corresponding to the local line feature in the model and the predicted image degradation value σ ₁ of the local line feature in the model as follows. Is calculated according to the equation [Equation 19]. When a plurality of hypotheses for associating the edge features in the two-dimensional image with the local line features in the model are detected, the following equation [Equation 19] is applied to all the edges listed as candidates. Is used to calculate the accuracy T.

  As described above, in the second embodiment, the magnitude of blur or blur is calculated using a two-dimensional image measured in advance, and the accuracy of correspondence between features with similar magnitudes of blur or blur is increased. By doing so, the association is performed. As a result, it is possible to correctly perform matching between the features.

  Note that the predicted value calculation unit 120 in the second embodiment estimates the magnitude of blur and the magnitude of blurring from the two-dimensional image obtained last time (previous time), but the present invention is not limited to this. Instead of the measured object, a two-dimensional image obtained by photographing an object capable of simulating the object (hereinafter, a simulated object) may be used as the “two-dimensional image obtained last time”. In addition, the size of blur and the size of blur detected on a two-dimensional image obtained by photographing the measured object or the simulated object at a plurality of positions / postures may be tabulated in advance, and this table may be referred to. Good. That is, the predicted value calculating unit 120 may obtain the blur / blur magnitude corresponding to the approximate position and orientation of the measured object as the image deterioration predicted value by referring to this table. In addition, an image deterioration prediction value may be obtained using a CG image created by projecting a shape model two-dimensionally, instead of a two-dimensional image of a real photograph. In simulating the blurring and blurring phenomenon on such a CG image, a known method, for example, a method disclosed in Non-Patent Document 3 can be used.

  As described above, by using the method of the second embodiment, the accuracy of association is calculated from the magnitude of blur or blur using a two-dimensional image measured in advance or a created CG image. The feature and the measurement data feature can be correctly collated.

<Third embodiment>
In the first and second embodiments, the magnitude of blurring and blurring of an image that may occur in a two-dimensional image is estimated as an image deterioration prediction value (σ ₀ , σ ₁ ), and an edge obtained in the two-dimensional image is obtained. It was used to select the correspondence between the features and the local line features of the model. In the third embodiment, the predicted image deterioration value is used for associating the actually measured three-dimensional measurement points (hereinafter, referred to as three-dimensional points) with the surface features of the model. In the third embodiment, the size of blur or blur on the two-dimensional image of the local surface feature and the blur or blur on the two-dimensional image when the measured three-dimensional position of the three-dimensional point is projected on the two-dimensional image. A high degree of accuracy is given to the correspondence of a feature having a size close to the size of, and the correspondence is selected.

  FIG. 10 is a diagram illustrating an example of a functional configuration of the information processing device 3 according to the third embodiment. Note that the hardware configuration of the information processing device 3 is the same as that in FIG. The information processing device 3 includes a three-dimensional point input unit 210 and a position calculation unit 220 in addition to the configuration of the information processing device 1. The configuration illustrated in FIG. 10 is a configuration that is an application example of the information processing apparatus of the present invention. Each function shown in FIG. 10 is realized by CPU 101 executing a program stored in ROM 102 and / or a program loaded from external storage device 105 to RAM 103. It is apparent that some or all of the functional units may be realized by dedicated hardware.

  The three-dimensional point input unit 210 acquires three-dimensional coordinates of a point group on the surface of the measured object. In the third embodiment, the three-dimensional coordinates of the three-dimensional point group previously obtained using the distance sensor are obtained from a storage device stored externally, but the present invention is not limited to this. For example, a three-dimensional point group acquired using a distance sensor (for example, a three-dimensional measuring device) may be input.

  The position calculation unit 220 calculates a two-dimensional image position when the three-dimensional coordinates of the point group input by the three-dimensional point input unit 210 are projected on a two-dimensional image. The two-dimensional image position of the three-dimensional coordinates is calculated from the internal parameters of the imaging device that captured the two-dimensional image and the relative position and orientation of the imaging device and the distance sensor used to measure the three-dimensional point. Is done. Note that the distance sensor and the imaging device are fixed to each other, and it is assumed that the relative position and posture of the two do not change, and calibration is performed in advance. For example, a calibration object having a known three-dimensional shape is observed from various directions, and the difference between the position and orientation of the calibration object based on the two-dimensional image and the position and orientation of the calibration object based on the distance image is also determined. At this time, the relative position and orientation of the distance sensor and the imaging device are obtained.

Note that the model input by the model input unit 110 in the third embodiment is a local three-dimensional object on the object surface composed of a three-dimensional position and a three-dimensional normal direction, as shown in FIG. It is assumed that the information is constituted by plane information (hereinafter, referred to as a local plane feature). The predicted value calculation unit 120 calculates the predicted deterioration value σ ₀ of the local surface feature by the same method as in the first embodiment.

  The search unit 140 detects a correspondence between the two-dimensional image input by the image input unit 130, the three-dimensional point input by the three-dimensional point input unit 210, and the local surface feature of the shape model input by the model input unit 110. In associating the local surface features with the three-dimensional points, the local surface features are approximated based on the approximate values of the positions and orientations of all the local surface features constituting the model of the measured object input by the model input unit 110. Are detected. When there are a plurality of three-dimensional points in the vicinity, all detected three-dimensional points are held as hypotheses.

The evaluation value calculation unit 150 calculates, as the image deterioration evaluation value σ, the magnitude of blur or blur at the position on each two-dimensional image of the three-dimensional point calculated by the position calculation unit 220. The method of calculating the image deterioration evaluation value σ is the same as in the first embodiment. The accuracy calculation unit 160 calculates a hypothesis of association with a local surface feature of the model from the predicted image deterioration value σ ₀ obtained by the predicted value calculation unit 120 and the image deterioration evaluation value σ obtained by the evaluation value calculation unit 150. The corresponding accuracy T is calculated for each of the three-dimensional point features held as. The method of calculating the accuracy T is the same as in the first embodiment.

  The matching unit 170 according to the third embodiment uses the three-dimensional points used for matching among the three-dimensional points corresponding to each local surface feature in the model of the measured object based on the accuracy T calculated by the accuracy calculating unit 160. Select a point. Here, it is assumed that the pair of the local surface feature and the three-dimensional point having the highest accuracy T is selected, but the present invention is not limited to this. For example, even if the pair with the highest accuracy T is selected, if the accuracy T is equal to or smaller than a certain threshold, the pair may not be used for comparison. Similarly, even when only one pair of the local surface feature and the three-dimensional point is detected, if the accuracy is equal to or less than the threshold, the pair may not be used for matching. Then, the matching unit 170 calculates the position and orientation of the measured object using the selected three-dimensional points. As the method of calculating the position and orientation by the matching unit 170, the same method as in the first embodiment can be used.

  Note that as a distance sensor used for measuring a three-dimensional point in the present embodiment, an active sensor that captures reflected light of laser light applied to an object with a camera and measures the distance by triangulation is used. However, the distance sensor is not limited to this, and may be a time-of-flight system that uses the time of flight of light. These active distance sensors are suitable when an object having a small surface texture is targeted. In addition, a passive type that calculates the depth of each pixel by triangulation from an image captured by a stereo camera may be used. The passive distance sensor is suitable for an object having a sufficient surface texture. In addition, any measurement of three-dimensional points does not impair the essence of the present invention.

  As described above, in the third embodiment, the size of blur or blur on the two-dimensional image of the local surface feature, and the two-dimensional image obtained by projecting the three-dimensional position of the actually measured three-dimensional point on the two-dimensional image A method has been described in which a high degree of accuracy is given to the correspondence of a feature close to the size of the above-described blur or blur, and the model feature is compared with a three-dimensional measurement point. This makes it possible to correctly collate the model feature with the three-dimensional measurement point.

<Modification 2>
In the third embodiment, the matching unit 170 selects and matches a pair of a local surface feature and a three-dimensional point with high accuracy. However, the present invention is not limited to this. The accuracy of the correspondence between each local surface feature and the ternary point may be set as a weight for each three-dimensional point when estimating the position and orientation. For example, similarly to the first modification, the degree of accuracy may be set as a weight for each three-dimensional point, and matching may be performed based on the weight. As for the weighting method, the accuracy T of the correspondence between the local surface features and the three-dimensional points is given to the diagonal components of the weight matrix W calculated by the equation [Equation 15] as in the first modification. Using the weight matrix W to solve the equation [Equation 17], the position and orientation correction value Δs is obtained, and the approximate values of the position and orientation are corrected. According to the above-described method, it is possible to preferentially collate the calculated high-accuracy pair.

  Although the accuracy T itself is given as a value given to the diagonal component of the weight matrix W, the value is not limited to this as in the first modification. Any pair may be used as long as a high accuracy pair gives a large weight and a low accuracy pair gives a small weight.

  As described above, in the second modification, the accuracy of the association is set as the weight of each data feature based on the magnitude of blur and blur of the image obtained from the two-dimensional image of the measured object, and the local surface feature is set. The method of performing the matching of the three-dimensional points with the above has been described. Thereby, it is possible to perform the matching with high accuracy.

<Modification 3>
In the third embodiment, the method of calculating the magnitude of blur performed in step S504 may be the following method. The amount of movement of each local surface feature in the three-dimensional space during the exposure time is used as the magnitude of the blur. Specifically, the Jacobian of each local surface feature is calculated based on the approximate position and orientation s of the shape model, and the relative movement direction of the Jacobian of each local surface feature and the input of the imaging device and the object to be measured input in step S502. The magnitude of the blur of each local surface feature is calculated from the speed.

The Jacobian of the local surface feature is a value representing the rate of change of the local surface feature in the three-dimensional space when the parameter of the position and orientation 6 degrees of freedom changes minutely. Based on the approximate position and orientation s of the object to be measured, a blur amount of the three-dimensional point corresponding to the local surface characteristics (dx, dy, dz), the normal direction of the local surface characteristics _{(n x, n y, n} z ) (Unit vector), the signed distance err _3D between the correspondences can be calculated from the following equation [Equation 20].

Then, similarly to the Jacobian of the local line feature, the Jacobian matrix of the local surface feature is calculated by partial differentiation of the correspondence distance err _3D with each parameter of the position and the posture s as in the following [Equation 21]. .

Ｂはスカラーであり、露光時間中に三次元空間中の局所面特徴の３次元位置が移動する量を表す。以上の処理を全ての局所面特徴に対して行い、全ての局所面特徴に対してブレ量を算出する。 As described above, the Jacobian of the projected local line feature selected in step S502 is calculated. Using Jacobian of the projection local line feature, the object during the exposure time t _i of the 2-dimensional image is generated by moving a relative position and orientation velocity V, the distance change B between the three-dimensional point local areal feature , Can be calculated by the following equation [Equation 22].

B is a scalar and represents an amount by which the three-dimensional position of the local surface feature in the three-dimensional space moves during the exposure time. The above processing is performed for all local surface features, and the blur amount is calculated for all local surface features.

  As described above, in the third modification, the method of calculating the magnitude of the blur from the moving amount of the local surface feature in the three-dimensional space has been described. By using this method, it is possible to accurately obtain the predicted value of the blur amount of the model feature.

<Modification 4>
The method of calculating the coefficient matrix and the error vector for calculating the position and orientation performed in step S803 in the third embodiment may be the following method.

The three-dimensional coordinates of the point group represented by the camera coordinates are converted into three-dimensional coordinates (x, y, z) in the coordinate system of the measurement target object using the position and orientation s of the measurement target object. It is assumed that a certain point in the point cloud data is converted into the coordinates of the object to be measured (x ₀ , y ₀ , z ₀ ) based on the approximate position and orientation. (X, y, z) varies depending on the position and orientation of the measurement target object, and is approximated by Equation (23) by a first-order Taylor expansion near (x ₀ , y ₀ , z ₀ ). it can.

The equation of the local surface feature associated with a certain point in the point cloud data in the coordinate system of the object to be measured is expressed as ax + by + cz = e (a ² + b ² + c ² = 1, a, b, c, e is a constant). Assume that (x, y, z) transformed by the correct s satisfies the plane equation ax + by + cz = e. By substituting equation [23] into the plane equation, equation [24] is obtained.

式［数２４］はsの各成分の微小変化Δsi（i=1,2,・・・,6）についての方程式であるため、式［数２５］のようなΔsiに関する線形連立方程式を立てることができる。

Since Equation [24] is an equation for a small change Δsi (i = 1, 2,..., 6) of each component of s, it is necessary to formulate a linear simultaneous equation for Δsi as Equation [25]. Can be.

  Here, Expression [Expression 25] is represented as Expression [Expression 13]. Using the coefficient matrix J and the error vector E, the position and orientation are calculated in the same manner as in the first and second embodiments. By using the method of the third modification, the position and orientation of the measurement object can be calculated more accurately.

<Fourth embodiment>
In the first and second embodiments, the model feature and the measurement data feature are compared only from the correspondence between the local line feature and the edge feature, and in the third embodiment, only from the correspondence between the local surface feature and the three-dimensional point. It is not limited. The matching may be performed using both the correspondence between the local line feature and the edge feature and the correspondence between the local surface feature and the three-dimensional point. According to the method of the present embodiment, the accuracy of the association is calculated from the magnitude of blur and blur of the image obtained from the two-dimensional image obtained by capturing the measured object, so that the actually measured edge features and local line features, and It is possible to correctly match both the three-dimensional point and the local surface feature.

The configuration of the information processing apparatus according to the fourth embodiment is the same as the configuration according to the first, second, and third embodiments. The matching unit 170 in the fourth embodiment calculates the accuracy of the correspondence between the edge feature and the local line feature and the accuracy of the correspondence between the three-dimensional point and the local surface feature, which are calculated by the accuracy calculation unit 160.
An edge feature corresponding to each local line feature and local surface feature in the model of the measured object;
Select an edge feature and a three-dimensional point to be used for matching from among the three-dimensional points. Subsequently, based on the selected correspondence, a coefficient matrix and an error vector for calculating the position and orientation are calculated. In the present embodiment, in order to determine the position and orientation in combination an edge feature and a three-dimensional point, the equation [Expression 12] The combined equation [Equation 25], relates Delta] s _i of Equation [Expression 26] A linear simultaneous equation can be established.

  Here, Equation [26] is represented as Equation [13]. Using the coefficient matrix J and the error vector E, the position and orientation are calculated in the same manner as in the first, second, and third embodiments.

  Alternatively, as described in Modifications 1 and 2, the degree of accuracy may be set as a weight for each three-dimensional point, and matching may be performed based on the weight. Any value may be used as long as a value that gives not only the accuracy T itself but also a large weight to a pair with high accuracy and a small weight to a pair with low accuracy as in the first and second modifications.

  By using the method of the fourth embodiment described above, it is possible to perform matching using both the correspondence between local line features and edge features and the correspondence between local surface features and three-dimensional points. According to the method of the present embodiment, it is possible to perform more accurate matching between the model feature and the measurement data than performing matching in each correspondence.

<Fifth embodiment>
One preferred application example of the information processing apparatuses 1 and 3 will be described below. That is, there is a use example in which the position and orientation of the measured object are estimated based on the two-dimensional image obtained from the imaging device, and the industrial robot arm grips the object and the like. Hereinafter, an example in which the information processing apparatuses 1 and 3 are applied to a robot system will be described with reference to FIG. FIG. 11 is a diagram illustrating a configuration example of a robot system that grips the measured object 40 using the robot 60 based on the estimation result of the position and orientation by the information processing apparatuses 1 and 3.

  The robot 60 has a movable axis composed of, for example, a rotation and / or translation axis, and each movable axis is driven and controlled by a robot controller 61. For example, the robot 60 moves a hand to a position instructed by the robot controller 61 to grip an object or the like. Since the position of the measured object 40 on the workbench changes, it is necessary to estimate the current position and orientation of the measured object 40 and perform grip control of the robot. The imaging device 20 is a camera that captures a normal two-dimensional image, and is installed at a position where an object to be measured 40 such as a hand of an industrial robot arm can be imaged. The information processing devices 1 and 3 estimate the position and orientation of the measured object 40 based on the two-dimensional image obtained from the imaging device 20. The position and orientation of the measured object 40 estimated by the information processing devices 1 and 3 are input to the robot controller 61, and the robot controller 61 performs gripping of the measured object 40 based on the input estimation result of the position and orientation. To control the robot arm.

  As described above, according to the robot system of the fifth embodiment, the estimation result of the position and orientation of the measured object 40 can be obtained from the information processing devices 1 and 3 that more accurately estimate the position and orientation. Therefore, the robot 60 can grip the measured object 40 more reliably.

<Effect>
As described above, according to the first embodiment, the accuracy of the association is calculated from the magnitude of blur and blur of an image obtained from a two-dimensional image of the object to be measured. Measurement data features can be correctly collated. Further, according to the third embodiment, the accuracy of the association is calculated from the magnitude of blur and blur of the two-dimensional image obtained by capturing the measured object, so that the actually measured three-dimensional point and the three-dimensional point are calculated. It is possible to correctly collate with the surface feature of the dimensional shape model. According to the fourth embodiment, the accuracy of the association is calculated from the magnitude of blur and blur of the image obtained from the two-dimensional image obtained by capturing the object to be measured. The line features of the three-dimensional shape model and the actually measured three-dimensional points and the surface features of the three-dimensional shape model can be simultaneously and correctly collated. Further, according to the second embodiment, the accuracy of association is calculated from the magnitude of blur or blur acquired in advance using a two-dimensional image or a CG image that has been measured in the past. Thus, it is possible to correctly collate the model feature and the measurement data feature. Further, according to the fifth embodiment, the position and orientation of the measured object are estimated, and the robot system can grip and move the measured object based on the estimation result.

<Definition>
The shape information input as the shape model of the object to be measured includes a set of simple two-dimensional points, a set of two-dimensional lines, a set of simple three-dimensional points, and a set of three-dimensional lines representing edges. Alternatively, it may be represented by shape information of a polygon composed of three three-dimensional points.

  In the prediction value calculation unit, as the image deterioration prediction value, the magnitude of blur due to defocus and the size of blur caused by parallel movement on the image plane are set as the image deterioration prediction value. However, the present invention is not limited to this. Absent. Any value may be used as long as it can represent the amount of deterioration of the two-dimensional image due to relative movement between the imaging device and the measured object, and there is no particular limitation on the calculation method and expression. For example, the magnitude of the blurring of the edge feature of the two-dimensional image of the object to be measured may be represented as a movement amount in the three-dimensional space calculated by back-projecting the two-dimensional image into the three-dimensional space. Further, a point spread function (PSF) created based on the magnitude of blur or magnitude of blur may be used as the image deterioration evaluation value. Further, in addition to a predicted value that takes into account the effects of both blurring and blurring, a predicted value that takes into account only one of the effects may be used.

  The two-dimensional image of the object to be measured captured by the image input unit may be of any type as long as a target image is obtained. For example, the image may be a gray image or a color image. In the present invention, a two-dimensional image captured in advance is input, but the present invention is not limited to this. You may input the result image | photographed using the imaging device. The measurement data feature that is a candidate for correspondence detected by the search unit 140 may be an edge feature or any feature such as a point feature that can specify a position on a two-dimensional image. Instead of using only a specific type of feature, a correspondence between a plurality of types of features (for example, points and edges) and a model feature may be detected.

  The evaluation value calculation unit calculates the edge feature by applying a function representing the brightness change of the edge when blurring and blurring occurs to the brightness change of the pixel along the direction orthogonal to the edge in the two-dimensional image. An image deterioration evaluation value is calculated for each of them. Here, as a function representing the luminance change, a Gaussian function or the like may be used instead of the error function as shown in Expression [6]. Further, in the above-described embodiment, the deterioration evaluation value is calculated for each feature such as the corresponding point corresponding to the control point and the projection position of the three-dimensional point. However, the present invention is not limited to this. For example, a two-dimensional image may be divided into rectangular partial areas, and a deterioration evaluation value may be calculated for each partial area.

When the difference between the image deterioration evaluation value (σ) and the image deterioration prediction value (σ ₀ , σ ₁ ) is large, the accuracy is small. When the difference is small, any method may be used as long as the accuracy is calculated to be large. For example, the calculation may be performed using a Gaussian function, a Tukey function, a Huber function, or the like, instead of the calculation method as shown in Expression [8].

  The matching unit selects a pair of a model feature and an image feature whose accuracy is equal to or greater than a certain threshold and performs matching. At this time, even when only one pair is detected, if the number is equal to or smaller than the threshold, the pair need not be used for the matching. However, the present invention is not limited to this, and weighting may be performed such that a pair having higher accuracy is preferentially used in the matching process. At this time, any value may be used as a value given by weighting as long as a pair with high accuracy is given a large weight and a pair with low accuracy is given a small weight. The matching unit may calculate the position and orientation of the measured object, or hold the positions of the model features in a plurality of positions and orientations in advance, and match the model features with the measurement data features by pattern matching. The approximate position and orientation may be identified, or the type of the object may be identified.

<Other embodiments>
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

1,3: Information processing device, 110: Model input unit, 120: Prediction value calculation unit, 130: Image input unit, 140: Search unit, 150: Evaluation value calculation unit, 160: Accuracy calculation unit, 170: Collation unit , 210: three-dimensional point input unit, 220: position calculation unit

Claims (18)

Based on a shape model representing the shape of the measured object, a prediction unit that predicts image degradation of the measured object that occurs in an image of the measured object captured by an imaging unit,
Searching means for searching for a measurement data feature corresponding to a model feature of the shape model, for a two-dimensional image obtained by imaging the object to be measured by the imaging means;
Evaluation means for evaluating image deterioration using the two-dimensional image for the measurement data feature searched by the search means;
Calculation means for calculating the accuracy of the correspondence between the model feature and the measurement data feature based on the image deterioration estimated by the prediction means and the image deterioration evaluated by the evaluation means,
An information processing apparatus comprising: a matching unit configured to match the shape model with a measured object in the two-dimensional image based on the accuracy of the correspondence between the model feature and the measurement data feature.   The information processing apparatus according to claim 1, wherein the image deterioration is image deterioration caused by an image spread due to blurring and / or blurring of a two-dimensional image.   The information processing apparatus according to claim 1, wherein the evaluation unit evaluates the image deterioration by applying an error function to the measurement data feature of the two-dimensional image.   The prediction unit is configured to determine the image based on the shape model, a given position and orientation of the measured object, a change in a relative position and orientation between the measured object and the imaging unit, and internal parameters of the imaging unit. The information processing apparatus according to any one of claims 1 to 3, wherein deterioration is predicted.   The predicting unit is configured to project the shape model onto an image obtained in advance by photographing the object to be measured or an object simulating the object to be measured, and the coordinate position of a model feature of the previously obtained image. The information processing apparatus according to any one of claims 1 to 4, wherein the image deterioration is obtained.   The information according to any one of claims 1 to 5, wherein the prediction unit obtains image degradation based on a CG image created by projecting the shape model of the measured object two-dimensionally. Processing equipment.   4. The apparatus according to claim 1, wherein the calculating unit calculates the accuracy such that the closer the degree of image deterioration estimated by the predicting unit and the degree of image deterioration evaluated by the evaluating unit, the higher the value. 7. The information processing device according to any one of 6. The matching means selects a correspondence between the model feature and the measurement data feature based on the accuracy,
The information processing apparatus according to claim 1, wherein:   The information processing apparatus according to claim 8, wherein the matching unit preferentially selects a correspondence with a high degree of accuracy calculated by the calculation unit.   The information processing apparatus according to claim 1, wherein the matching unit sets a weight for the measurement data feature based on the accuracy.   The information processing apparatus according to claim 1, wherein the matching unit is a unit that calculates a position and / or a posture of the measured object in a three-dimensional space. Said imaging means;
The information processing apparatus according to claim 1, further comprising: an acquisition unit configured to acquire the two-dimensional image by imaging the object to be measured by the imaging unit. Based on a shape model representing the shape of the measured object, a prediction unit that predicts image degradation of the measured object that occurs in an image of the measured object captured by an imaging unit,
Acquiring means for acquiring three-dimensional coordinates of a three-dimensional measurement point on the surface of the measured object;
Evaluation means for evaluating, in the two-dimensional image captured by the imaging means, image degradation at a position of a point cloud projected on the two-dimensional image based on the three-dimensional coordinates acquired by the acquisition means;
Searching means for searching for a correspondence between the three-dimensional measurement points and model features of the shape model;
Calculating means for calculating the accuracy of the correspondence between the model feature and the three-dimensional measurement point based on the image deterioration predicted by the predicting means and the image deterioration evaluated by the evaluating means;
Information comprising: matching means for comparing the shape model with a three-dimensional measurement point indicating the shape of the measured object based on the accuracy of the correspondence between the model feature and the three-dimensional measurement point. Processing equipment. A measuring device for measuring three-dimensional coordinates of the three-dimensional measuring point,
14. The information processing apparatus according to claim 13, wherein the obtaining unit obtains dimensional coordinates from the measuring device.   15. The control device according to claim 1, further comprising: a control unit configured to control a robot arm based on an estimation result of the position and orientation of the measured object estimated by the matching unit. Information processing device. Based on a shape model representing the shape of the measured object, a prediction step of predicting image degradation of the measured object that occurs in an image of the measured object captured by an imaging unit,
A search step of searching for a measurement data feature corresponding to a model feature of the shape model, for a two-dimensional image obtained by imaging the object to be measured by the imaging unit;
An evaluation step of evaluating image deterioration using the two-dimensional image with respect to the measurement data feature searched in the search step;
A calculation step of calculating the accuracy of the correspondence between the model feature and the measurement data feature based on the image deterioration estimated in the prediction step and the image deterioration evaluated in the evaluation step;
A collation step of collating the shape model with an object to be measured in the two-dimensional image based on the accuracy of the correspondence between the model feature and the measurement data feature. Method. Based on a shape model representing the shape of the measured object, a prediction step of predicting image degradation of the measured object that occurs in an image of the measured object captured by an imaging unit,
An acquisition step of acquiring three-dimensional coordinates of a three-dimensional measurement point on the surface of the measured object;
An evaluation step of evaluating image deterioration at a position of a point cloud projected on the two-dimensional image based on the three-dimensional coordinates obtained in the obtaining step, using a two-dimensional image captured by the imaging unit;
A search step of searching for a correspondence between the three-dimensional measurement point and a model feature of the shape model;
A calculating step of calculating the accuracy of the correspondence between the model features and the three-dimensional measurement points based on the image deterioration estimated in the prediction step and the image deterioration evaluated in the evaluation step;
A collation step of collating the shape model with a three-dimensional measurement point indicating the shape of the measured object based on the accuracy of the correspondence between the model feature and the three-dimensional measurement point. A method for controlling a processing device.   A program for causing a computer to execute each step of the information processing apparatus control method according to claim 16.

Images