CN111415391B - A Calibration Method for External Orientation Parameters of Multi-eye Cameras Using Mutual Shooting Method - Google Patents

Abstract

A method for calibrating external azimuth parameters of a multi-camera by adopting a mutual shooting method. The method comprises the steps of establishing a calibration system of the multi-camera space measurement system; calibrating parameters in a monocular camera; the auxiliary camera is combined with the measurement camera to calibrate external parameters of the binocular vision system; calculating the pose of the target under an auxiliary camera coordinate system; calculating the pose of the target under the self-measuring camera coordinate system; calibrating external parameters of mutual shooting; and rotating the measuring camera to the working posture and measuring. The external azimuth parameter calibration method of the multi-camera by adopting the mutual shooting method can quickly and accurately calibrate the external parameters between each measuring camera in the multi-camera measuring system, and can flexibly adapt to various working environments, so that the measuring work can be efficiently carried out. In addition, the external parameters between cameras can be calibrated rapidly, accurate parameters are provided for subsequent measurement work, the working efficiency and the precision can be improved, and the measurement cost is reduced.

Description

A method for calibrating external orientation parameters of multi-cameras using mutual shooting method

Technical Field

The invention belongs to the technical field of machine vision three-dimensional space coordinate measurement, and in particular relates to a method for calibrating external orientation parameters of a multi-eye camera using a mutual shooting method.

Background Art

With the continuous development of my country's machinery manufacturing, aerospace, robotics and other fields, especially the development of domestic large aircraft production technology, the requirements for spatial measurement accuracy in the manufacturing and assembly process are also increasing. The accuracy of measurement technology directly determines the accuracy of industrial manufacturing and assembly. Traditional three-dimensional coordinate measuring machines are too large and have low calculation efficiency. In addition, traditional measurement methods often require contact with the object being measured, so it is easy to cause damage to the object being measured. It has long been unable to meet the accuracy and efficiency requirements of modern industry for measurement systems.

With the improvement of hardware level, the computing speed of computers has also been rapidly improved, making computer vision and photogrammetry technology widely used in various industries, greatly improving the working efficiency and accuracy of the measurement system, and reducing labor costs and time costs. In the field of computer vision spatial three-dimensional coordinate measurement, multi-camera vision measurement systems have been widely used due to their high measurement accuracy, strong adaptability, high measurement efficiency and low cost. Compared with traditional three-dimensional coordinate measurement methods, it has the advantages of high accuracy and zero loss of the measured object. Therefore, the study of multi-camera vision measurement systems has important practical significance.

In multi-camera measurement systems, the main measurement method used is the triangulation algorithm. In this algorithm, the calibration accuracy of the camera's internal and external parameters will directly affect the measurement accuracy of the spatial coordinates. The camera's internal parameters can be accurately calibrated using mature calibration tools such as Zhang's calibration method. However, as the working scene changes, the external parameters between cameras in the system are also changing, and due to the limitations of the working environment, the traditional binocular vision external parameter calibration method with the help of chessboard calibration is often not possible. Therefore, the development of a fast, accurate, and flexible camera external parameter calibration method is of utmost importance in the development of a multi-camera spatial three-dimensional measurement system.

Summary of the invention

In order to solve the above problems, the object of the present invention is to provide a method for calibrating the external orientation parameters of a multi-eye camera using a mutual shooting method.

In order to achieve the above object, the method for calibrating the external orientation parameters of a multi-camera using the mutual shooting method provided by the present invention comprises the following steps performed in sequence:

Step 1) Establishing a multi-camera spatial measurement system calibration system: the system comprises a first measuring camera, a second measuring camera, a target, an auxiliary camera and a precision rotating stage; wherein a precision rotating stage is respectively arranged at the lower end of the first measuring camera, the second measuring camera and the auxiliary camera, and the three precision rotating stages are arranged in a triangle; a target is respectively installed on the first measuring camera and the second measuring camera;

Step 2) Calibration of the internal parameters of the monocular camera: According to the pinhole imaging model, the mathematical models of the first measurement camera, the second measurement camera and the auxiliary camera are respectively established; according to the homography matrix mapping principle and the nonlinear optimization principle, the internal parameter matrix and distortion parameter of the first measurement camera, the second measurement camera and the auxiliary camera are calibrated with the help of a checkerboard calibration plate;

Step 3) The auxiliary camera and the measuring camera are combined to calibrate the external parameters of the binocular vision system: the first measuring camera, the second measuring camera and the auxiliary camera respectively form a binocular vision system, and then the external parameter matrices of the two binocular vision systems are calibrated according to the binocular vision external parameter calibration principle;

Step 4) Calculate the position and pose of the target in the auxiliary camera coordinate system: rotate the auxiliary camera, and use the extrinsic parameter matrix of the binocular vision system calibrated above to obtain the extrinsic parameter matrix between the auxiliary camera after rotation and the first or second measurement camera: then use the auxiliary camera to shoot the target, and calculate the precise position and pose of the target in the auxiliary camera coordinate system;

Step 5) Calculate the pose of the target in the coordinate system of its own measurement camera: Utilize the external parameter matrix between the rotated auxiliary camera and the first measurement camera or the second measurement camera obtained in step 4) and the calculated pose of the target in the coordinate system of the auxiliary camera, and calculate the pose of the target in the coordinate system of its own measurement camera through simple coordinate transformation;

Step 6) Mutual shooting extrinsic parameter calibration: Use two measuring cameras to shoot the target on each other's measuring camera, calculate the pose of the other target, and calculate the current extrinsic parameters of the two measuring cameras by conversion;

Step 7) Use a precision rotation stage to rotate the measurement camera to the working posture and perform measurement.

In step 2), the monocular camera internal parameter calibration is implemented using the calibration toolbox in Matlab or the calibration function in OpenCV.

In step 3), the method of calibrating the external parameters of the binocular vision system by combining the auxiliary camera with the measuring camera is: first, a binocular vision system is formed by the first measuring camera, the second measuring camera and the auxiliary camera respectively, and then the checkerboard calibration plate is photographed simultaneously by the first measuring camera and the auxiliary camera as well as the second measuring camera and the auxiliary camera, and spatial feature points are matched after the photographing, and the essential matrix is calculated after the paired spatial feature points are obtained, and the rotation matrix R and the translation vector T in the extrinsic parameter matrix can be decomposed through the essential matrix.

In step 4), the method for calculating the position and pose of the target in the auxiliary camera coordinate system is: first, using a precision rotating table to rotate the auxiliary camera to face the target on the first measuring camera or the second measuring camera, and then using the auxiliary camera to image the target. After obtaining the image, the pixel coordinates of the target can be obtained, and the world coordinates of the target are known. Therefore, the position and pose of the target in the auxiliary camera coordinate system can be calculated using the relationship between the world coordinates and the pixel coordinates of the target.

In step 5), the method for calculating the position and posture of the target in the coordinate system of its own measurement camera is: multiplying the external parameter matrix between the rotated auxiliary camera and the first measurement camera or the second measurement camera obtained in step 4) and the calculated position and posture of the target in the auxiliary camera coordinate system to obtain the coordinates of the target in the coordinate system of its own measurement camera.

In step 6), the method for mutual shooting extrinsic parameter calibration is: the pose of the target in the coordinate system of its own measuring camera is calculated by step 5), the target on the second measuring camera is photographed by the first measuring camera, or the target on the first measuring camera is photographed by the second measuring camera, and then the pose of the other target is calculated by the PnP algorithm, and the current extrinsic parameters of the two measuring cameras are obtained by conversion.

In step 7), the method of using a precision rotating table to rotate the measuring camera to a working posture and perform measurement is: before performing measurement work, the first measuring camera and the second measuring camera are rotated to an angle facing the object to be measured; the precision rotating table is used to record the rotation angle of the first measuring camera and the second measuring camera from the calibration posture to the working posture, and the external parameters of the two measuring cameras in the working posture are obtained by calculation, and then the measurement work can be performed.

The multi-camera external orientation parameter calibration method using the mutual shooting method provided by the present invention can quickly and accurately calibrate the external parameters between each measuring camera in the multi-camera measurement system, and can flexibly adapt to various working environments, so that the measurement work can be carried out efficiently. In addition, the external parameters between cameras can be quickly calibrated, providing accurate parameters for subsequent measurement work, which can not only improve work efficiency and accuracy, but also reduce measurement costs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for calibrating external orientation parameters of a multi-camera using a mutual shooting method provided by the present invention.

FIG. 2 is a schematic diagram of the external parameter calibration process of the binocular vision system in the present invention.

FIG. 3 is a schematic diagram of coordinate transformation in the present invention.

DETAILED DESCRIPTION

The following is a detailed description of the method for calibrating the external orientation parameters of a multi-camera using the mutual shooting method provided by the present invention in conjunction with the accompanying drawings and specific embodiments. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention.

As shown in FIGS. 1, 2 and 3, the method for calibrating the external orientation parameters of a multi-camera using the mutual shooting method provided by the present invention comprises performing the following steps in sequence:

Step 1) Establish a multi-camera space measurement system calibration system: the system includes a first measuring camera 1, a second measuring camera 2, a target L, an auxiliary camera 3 and a precision rotating table 4; wherein, a precision rotating table 4 is respectively arranged at the lower end of the first measuring camera 1, the second measuring camera 2 and the auxiliary camera 3, and the three precision rotating tables 4 are arranged in a triangle; a target L is respectively installed on the first measuring camera 1 and the second measuring camera 2; the first measuring camera 1 and the second measuring camera 2 are image acquisition devices; the target L is a mutual shooting calibration auxiliary tool for realizing rapid external parameter calibration before measurement; the auxiliary camera 3 is used to calculate the coordinates of each target L in the measuring camera coordinate system; the precision rotating table 4 is used to rotate the camera thereon to a suitable measurement angle;

Step 2) Calibration of the internal parameters of the monocular camera: According to the pinhole imaging model, the mathematical models of the first measuring camera 1, the second measuring camera 2 and the auxiliary camera 3 are respectively established; according to the homography matrix mapping principle and the nonlinear optimization principle, the internal parameter matrix and distortion parameter of the first measuring camera 1, the second measuring camera 2 and the auxiliary camera 3 are calibrated with the help of a checkerboard calibration plate;

The monocular camera internal parameter calibration is implemented using the calibration toolbox in Matlab or the calibration function in OpenCV.

Step 3) The auxiliary camera and the measuring camera are combined to calibrate the external parameters of the binocular vision system: the first measuring camera 1, the second measuring camera 2 and the auxiliary camera 3 respectively form a binocular vision system, and then the external parameter matrices of the two binocular vision systems are calibrated according to the binocular vision external parameter calibration principle;

FIG2 is a schematic diagram of the external parameter calibration process of the binocular vision system in the present invention. As shown in FIG2, O _L X _L Y _L Z _L is the measurement camera coordinate system, the imaging plane coordinate system is o _l x _l y _l , and the optical axis direction is Z _L ; similarly, O _R X _R Y _R Z _R is the auxiliary camera coordinate system, the imaging plane coordinate system is o _R x _R y _R , and the optical axis direction is Z _R ; _PL and _PR are the pixel coordinates of the spatial feature point on the image plane of the measurement camera and the auxiliary camera, respectively. The intersection point P _W of the two rays in the figure is the coordinate of the spatial feature point in the world coordinate system X _W Y _W Z _W. The coordinate conversion relationship between any two coordinate systems is:

表示坐标系O_RX_RY_R到坐标系O_LX_LY_L的旋转矩阵；in,

Represents _the rotation matrix from coordinate _{system ORXRYR} to _coordinate system _{OLXLYL ;}

T = (t ₁ t ₂ t ₃ ) ^T , represents the translation vector corresponding to the rotation matrix R above.

Taking the measurement camera coordinate system O ₁ X ₁ Y ₁ Z ₁ and the auxiliary camera coordinate system O ₃ X ₃ Y ₃ Z ₃ in Figure 3 as an example, their coordinate transformation relationship is:

表示坐标系O₁X₁Y₁到坐标系O₃X₃Y₃的旋转矩阵；in,

Represents the rotation matrix from coordinate system O ₁ X ₁ Y ₁ to coordinate system O ₃ X ₃ Y ₃ ;

T = (t ₁ t ₂ t ₃ ) ^T , represents the translation vector corresponding to the rotation matrix R above.

The above rotation matrix is calibrated. The external parameter calibration of the binocular vision system is the process of solving the rotation matrix R and the translation vector T. The two calibrated external parameter matrices of the binocular vision system are R ₁₃ T ₁₃ and R ₂₃ T ₂₃ in Figure 3.

Step 4) Calculate the position and pose of the target in the auxiliary camera coordinate system: rotate the auxiliary camera, and use the extrinsic parameter matrix of the binocular vision system calibrated above to obtain the extrinsic parameter matrix between the auxiliary camera after rotation and the first or second measurement camera: then use the auxiliary camera to shoot the target, and calculate the precise position and pose of the target in the auxiliary camera coordinate system;

After the rotation matrix R and the translation vector T are obtained in step 3), the auxiliary camera 3 needs to be rotated by a certain angle using a precision rotating stage 4 so that it can capture the target L fixed on the first measuring camera 1 or the second measuring camera 2. The rotation of the auxiliary camera 3 is around the Z axis of the measuring camera coordinate system. The conversion relationship between the rotation angle θ and the rotation matrix R _z (θ) is:

The external parameter matrix between the current auxiliary camera 3 and the first measuring camera 1 or the second measuring camera 2 after the rotation is obtained by multiplying the rotation matrix R calibrated in step 3) with the translation vector T and the rotation matrix R _z (θ) obtained by rotating the precision rotating stage 4 .

The precision rotating table 4 under the auxiliary camera 3 is rotated until the auxiliary camera 3 can capture the target L on the first measuring camera 1 or the second measuring camera 2, and then the target L is captured. After that, the pose of the target L in the auxiliary camera coordinate system is calculated according to the PnP algorithm, and then the reprojection error of the feature points on the target is optimized using a nonlinear optimization algorithm, so as to obtain the precise pose of the target L in the auxiliary camera coordinate system;

Because the world coordinates of the feature points on the target L are known, the pixel coordinates on the imaging plane can be obtained through the image. The auxiliary camera 3 is used to shoot the target L in order to prepare for solving the position and posture of the target L in its own measurement camera coordinate system;

The principle of the PnP algorithm is as follows:

PNP (Perspective-n-Points) is an algorithm for solving camera pose based on 3D to 2D point pairs. It describes how to solve the camera pose when there are N pairs of space and image matching points. The present invention defines this PnP problem as a nonlinear least squares problem solved on Lie algebra. Since it is more convenient to differentiate Lie algebra in nonlinear optimization, Lie algebra is used to represent the pose of the target L in the auxiliary camera coordinate system. The pose is used as the optimization variable and a nonlinear optimization method is used to minimize the reprojection error for optimization and solution.

There are N three-dimensional spatial feature points P on the target L and N projection points p on the imaging plane. What needs to be calculated is the position R and T of the target L, and its Lie algebra is represented as ξ. Assume that the coordinates of the spatial coordinate point on the target L are _Pi = [ _{Xi Yi Zi} ] ^T , and the corresponding pixel coordinates on the imaging plane are _Ui = [ _uiv ] ^T. _The relationship between the pixel position and the spatial feature point is as follows:

Where K is the intrinsic parameter matrix of the camera obtained by calibration in step 1), ξ is the pose of the target L represented by Lie algebra, which is written in matrix form as: s _i U _i =Kexp(ξ^)P _i . Due to factors such as the pose of the target L and imaging noise, this equation has errors. Therefore, the present invention adds up all the errors to construct a nonlinear least squares problem, and iterative solution can obtain the accurate pose:

The error term in the PnP problem is the error obtained by comparing the observed pixel coordinates with the position of the 3D point projected onto the imaging plane according to the current estimated pose, so it is called the reprojection error. There are many methods for solving nonlinear least squares problems, such as the first-order steepest descent method, the second-order Gauss-Newton method, and the Levenberg-Marquardt method, which are nonlinear optimization algorithms.

Step 5) Calculate the pose of the target in the coordinate system of its own measurement camera: Utilize the external parameter matrix between the rotated auxiliary camera and the first measurement camera or the second measurement camera obtained in step 4) and the calculated pose of the target in the coordinate system of the auxiliary camera, and calculate the pose of the target in the coordinate system of its own measurement camera through simple coordinate transformation;

Because the target L is fixedly mounted on the first measuring camera 1 and the second measuring camera 2, that is, the position and posture of the target L in the coordinate system of its own measuring camera will not change, the schematic diagram of the calculated position and posture of the target in the coordinate system of its own measuring camera is shown in Figure 3. Through the external parameter calibration of step 3), it can be known that the external parameter matrices R ₁₃ T ₁₃ and R ₂₃ T ₂₃ of the two binocular vision systems are known, and the external parameter matrix between the auxiliary camera 3 and the first measuring camera 1 or the second measuring camera 2 after rotation and the accurate position and posture of the target L in the auxiliary camera coordinate system O ₃ X ₃ Y ₃ Z ₃ have been obtained in step 4), so the position and posture of the target L on the first measuring camera 1 in the measuring camera coordinate system O ₁ X ₁ Y ₁ Z ₁ and the target L on the second measuring camera 2 in the measuring camera coordinate system O ₂ X ₂ Y ₂ Z ₂ can be obtained by simple coordinate transformation.

Step 6) Mutual shooting extrinsic parameter calibration: Use two measuring cameras to shoot the target on each other's measuring camera, calculate the pose of the other target, and calculate the current extrinsic parameters of the two measuring cameras by conversion;

Through step 5), the position and posture of the target L in the coordinate system of its own measuring camera has been calculated. The first measuring camera 1 is used to photograph the target L on the second measuring camera 2, or the second measuring camera 2 is used to photograph the target L on the first measuring camera 1. Then, the position and posture of the other target L is calculated using the PnP algorithm. The current external parameters of the two measuring cameras can be obtained by conversion.

Step 7) Use the precision rotation stage to rotate the measurement camera to the working posture and perform the measurement:

In step 6), the first measuring camera 1 and the second measuring camera 2 are opposite to each other, but when performing measurement work, the two measuring cameras need to face the object to be measured, so the two measuring cameras need to be rotated to an angle facing the object to be measured before performing measurement work. The rotation angle of the first measuring camera 1 and the second measuring camera 2 from the calibration posture to the working posture is recorded using a precision rotating table 3. The external parameters of the two measuring cameras in the working posture can be obtained by calculation, and then the measurement work can be performed.

After the rotation, the rotation matrix in the external parameters of the two measurement cameras changes again. It is necessary to convert the rotation angle of the precision rotating stage 3 into a rotation matrix and update the external parameter matrix. The relationship between the rotation angle and the rotation matrix is:

If the rotation angle is θ, and the rotation axes are X, Y, and Z, then the rotation matrices are:

The specific implementation modes of the present invention are described above in conjunction with the accompanying drawings, but these descriptions cannot be understood as limiting the scope of the present invention. The protection scope of the present invention is defined by the attached claims, and any changes based on the claims of the present invention are within the protection scope of the present invention.

Claims (7)

1. A method for calibrating the external orientation parameters of a multi-camera using a mutual shooting method, characterized in that: the method for calibrating the external orientation parameters of a multi-camera using a mutual shooting method comprises the following steps performed in sequence: Step 1) establishing a multi-camera spatial measurement system calibration system: the system comprises a first measurement camera (1), a second measurement camera (2), a target (L), an auxiliary camera (3) and a precision rotating table (4); wherein a precision rotating table (4) is respectively arranged at the lower end of the first measurement camera (1), the second measurement camera (2) and the auxiliary camera (3), and the three precision rotating tables (4) are arranged in a triangle; a target (L) is respectively installed on the first measurement camera (1) and the second measurement camera (2); Step 2) Calibration of the internal parameters of the monocular camera: mathematical models of the first measurement camera (1), the second measurement camera (2) and the auxiliary camera (3) are respectively established according to the pinhole imaging model; the internal parameter matrix and distortion parameter calibration of the first measurement camera (1), the second measurement camera (2) and the auxiliary camera (3) are performed with the aid of a checkerboard calibration plate according to the homography matrix mapping principle and the nonlinear optimization principle; Step 3) The auxiliary camera and the measuring camera are combined to calibrate the external parameters of the binocular vision system: the first measuring camera (1), the second measuring camera (2) and the auxiliary camera (3) respectively form a binocular vision system, and then the external parameter matrices of the two binocular vision systems are calibrated according to the binocular vision external parameter calibration principle; Step 4) Calculate the position and pose of the target in the auxiliary camera coordinate system: rotate the auxiliary camera, and use the extrinsic parameter matrix of the binocular vision system calibrated above to obtain the extrinsic parameter matrix between the auxiliary camera after rotation and the first or second measurement camera: then use the auxiliary camera to shoot the target, and calculate the precise position and pose of the target in the auxiliary camera coordinate system; Step 5) Calculate the pose of the target in the coordinate system of its own measurement camera: Utilize the external parameter matrix between the rotated auxiliary camera and the first measurement camera or the second measurement camera obtained in step 4) and the calculated pose of the target in the coordinate system of the auxiliary camera, and calculate the pose of the target in the coordinate system of its own measurement camera through simple coordinate transformation; Step 6) Mutual shooting extrinsic parameter calibration: Use two measuring cameras to shoot the target on each other's measuring camera, calculate the pose of the other target, and calculate the current extrinsic parameters of the two measuring cameras by conversion; Step 7) Use a precision rotation stage to rotate the measurement camera to the working posture and perform measurement. 2. According to the method for calibrating the external orientation parameters of a multi-eye camera using the mutual shooting method in claim 1, it is characterized in that: in step 2), the calibration of the internal parameters of the monocular camera is implemented using the calibration toolbox in Matlab or the calibration function in OpenCV. 3. The method for calibrating the external orientation parameters of a multi-camera using the mutual shooting method according to claim 1 is characterized in that: in step 3), the method for calibrating the external parameters of the binocular vision system by the auxiliary camera and the measuring camera is: first, a binocular vision system is formed by the first measuring camera (1), the second measuring camera (2) and the auxiliary camera (3), respectively; then, the checkerboard calibration plate is photographed simultaneously by the first measuring camera (1) and the auxiliary camera (3) and the second measuring camera (2) and the auxiliary camera (3); after the photographing, spatial feature points are matched; after obtaining the paired spatial feature points, the essential matrix is calculated; and the rotation matrix R and the translation vector T in the external parameter matrix can be decomposed by the essential matrix. 4. The method for calibrating external orientation parameters of a multi-camera using a mutual shooting method according to claim 1 is characterized in that: in step 4), the method for calculating the position and posture of the target in the auxiliary camera coordinate system is: first, using a precision rotating table (4), the auxiliary camera (3) is rotated to face the target (L) on the first measuring camera (1) or the second measuring camera (2), and then the auxiliary camera (3) is used to image the target (L). After obtaining the image, the pixel coordinates of the target (L) can be obtained, and the world coordinates of the target (L) are known. Therefore, the position and posture of the target (L) in the auxiliary camera coordinate system can be calculated using the relationship between the world coordinates and the pixel coordinates of the target (L). 5. The method for calibrating external orientation parameters of a multi-camera using a mutual shooting method according to claim 1 is characterized in that: in step 5), the method for calculating the position and posture of the target in the coordinate system of its own measurement camera is: the coordinates of the target (L) in the coordinate system of its own measurement camera can be obtained by multiplying the external parameter matrix between the rotated auxiliary camera (3) and the first measurement camera (1) or the second measurement camera (2) obtained in step 4) and the calculated position and posture of the target (L) in the coordinate system of the auxiliary camera. 6. The method for calibrating external orientation parameters of a multi-camera using a mutual shooting method according to claim 1 is characterized in that: in step 6), the method for calibrating external parameters of mutual shooting is: the pose of the target (L) in the coordinate system of the own measurement camera has been calculated through step 5), the target (L) on the second measurement camera (2) is photographed by the first measurement camera (1), or the target (L) on the first measurement camera (1) is photographed by the second measurement camera (2), and then the pose of the other target (L) is calculated using the PnP algorithm, and the current external parameters of the two measurement cameras are obtained by conversion. 7. The method for calibrating the external orientation parameters of a multi-camera using a mutual shooting method according to claim 1 is characterized in that: in step 7), the method of using a precision rotating table to rotate the measuring camera to a working posture and perform measurement is: before performing measurement work, the first measuring camera (1) and the second measuring camera (2) are rotated to an angle facing the object to be measured; the precision rotating table (3) is used to record the rotation angle of the first measuring camera (1) and the second measuring camera (2) from the calibration posture to the working posture, and the external parameters of the two measuring cameras in the working posture are obtained by calculation, and then the measurement work can be performed.

Images