JP5432835B2 - How to calibrate the camera - Google Patents

Description

  The present invention relates to a camera calibration method for calculating camera calibration parameters such as the camera position and orientation, the image center and scale of the camera.

  Conventionally, a stereo camera has been used as a camera for measuring a three-dimensional shape of a measurement object based on two captured images. In this stereo camera, the displacement of the attachment position directly affects the displacement between the imaged data, and therefore high-precision position adjustment is required at the time of production or attachment to the site.

  Thus, in three-dimensional measurement using a stereo camera, it is essential to obtain camera calibration parameters such as the position and orientation of the camera, the image center and scale of the camera. Obtaining this camera calibration parameter is called calibration. A calibration parameter obtained by this calibration is called a calibration parameter.

[Calibration parameters]
FIG. 19 shows an outline of three-dimensional measurement using a stereo camera. In this figure, G1 is an image of the left camera, and G2 is an image of the right camera. Now, the point in space

, The point on the left camera image G1

Suppose that the point M in the space is reflected at the position of the point m on the image G1 of the left camera.
In this case, M and m have the following relationship.
sm = A [Rt] M = PM (1)

  Here, s represents a scale parameter. R is a 3 × 3 rotation matrix representing the camera posture, and t is a 3 × 1 translation vector representing the camera position. The combined 3 × 4 matrix [Rt] is called an external parameter. A is a 3 × 3 matrix expressed by the following equation (2).

Here, [u ₀ , v ₀ ] represents the image center, α _u and α _v represent the u-axis and v-axis scale factors, respectively, and b represents the twist of the two axes. The u axis represents the horizontal direction of the image, the v axis represents the vertical direction of the image, and the upper left of the image is the origin. The matrix A is composed only of camera internal variables and is called camera internal parameters.

  The matrix P = A [Rt] is a 3 × 4 matrix that projects points in three dimensions (three-dimensional data) onto points on the image (two-dimensional data), and is called a P matrix. In the calibration, this P matrix, that is, a P matrix composed of internal parameters and external parameters is obtained as a calibration pattern.

  In addition, the relationship of said Formula (1) assumes a pinhole camera model (refer the nonpatent literature 1 for details).

[Principle of three-dimensional measurement]
Next, a method for measuring the shape of the original three-dimensional object from images obtained by imaging the three-dimensional object with two cameras will be described. The three-dimensional point M = [X, Y, Z, 1] ^T is imaged by two cameras, the point projected on the left camera is m = [u, v, 1] ^T , and the point projected on the right camera is When m ′ = [u ′, v ′, 1] ^T , there are relationships such as sm = PM and sm ′ = PM ′. However,

And represents the P matrix of each camera. The above relationship can be transformed into the following equation (3).

  In this simultaneous equation, the number of unknowns is 3 and the number of equations is 4, so the point M in space can be calculated.

From the above, if the P matrix of each stereo camera is known, the three-dimensional points from the points m = [u, v, 1] ^T and m ′ = [u ′, v ′, 1] ^T observed by the two cameras M = [X, Y, Z, 1] ^T can be obtained. Here, it is assumed that the correspondence between the points m and m ′ on the two images is known.

[Calculation of internal parameters]
When the coordinates of the three-dimensional space are set so that the three-dimensional plane is Z = 0, the above equation (1) can be transformed into the following equation (4).
sm = A [Rt] [X, Y, 0, 1] ^T = A [r ₁ r ₂ t] [X, Y, 1] ^T (4)

Here, the rotation matrix R is R = [r ₁ r ₂ r ₃ ]. Also, the meaning that the three-dimensional plane is set to be Z = 0 means that a direction perpendicular to a calibration pattern to be described later is considered as the Z axis, and the position of the calibration pattern is the origin of the Z axis, that is, Z = 0. Is to take the coordinate system.

Here, if H = A [r ₁ r ₂ t] is defined as the projective transformation matrix, the above equation (4) becomes the following equation (5).
s [u, v, 1] ^T = H [X, Y, 1] ^T (5)

The projective transformation matrix is also called a “Homography matrix” and represents a transformation from planar coordinates to planar coordinates. Hereinafter, the projective transformation matrix is referred to as an H matrix. This H matrix can be calculated if there are four or more corresponding points on the three-dimensional plane and on the image. Here, [h ₁ h ₂ h ₃ ] = λA [r ₁ r ₂ t] is set, and by utilizing the fact that r ₁ and r ₂ are orthogonal unit vectors, the following equation is obtained.

From one image, the constraint equations regarding the two internal camera variables are obtained. When solving five camera internal variables, at least three images are required. For this reason, as described later, it is necessary to image the calibration parameter three times or more from different viewpoints. If a matrix B = A ^−T A ⁻¹ is defined and a vector in which its elements are arranged is defined as b = [B ₁₁ , B ₁₂ , B ₂₂ , B ₁₃ , B ₂₃ , B ₃₃ ] ^T ,
h _i ^T Bh _j = v _ij ^T b (7)
It becomes.
here,

It is. Furthermore, the above formulas (6) and (7) become the following formula (8).

  The above equation (8) is an equation obtained from one H matrix. If more than two H matrices are obtained, b can be calculated by stacking them. If b is obtained, the internal parameter can be calculated from the following equation.

[Calculation of external parameters]
The external parameter R = [r ₁ r ₂ r ₃ ] and t can be calculated by the following formula once the internal parameter A is obtained.

  In this way, internal parameters and external parameters can be calculated, and a P matrix that is a calibration parameter can be obtained from the internal parameters and external parameters. In obtaining this P matrix, as can be seen from the above description, it is important to obtain an H matrix that is a correspondence between the captured image and the planar object to be imaged.

[Calibration pattern]
FIG. 20 shows an example of a conventional calibration pattern used for calibration of a stereo camera. The calibration pattern 1 is a grid-like plane pattern that is arranged so that the colors in the grids are alternately inverted. In this example, the checkerboard pattern in which white and black grids are alternately arranged in a grid pattern is used. It is a pattern of patterns.

  FIG. 21 shows a flow chart of calibration using this calibration pattern 1. FIG. 22 shows an outline of a system for performing calibration according to this flowchart. In this system, the calibration pattern 1 is photographed (captured) by the stereo camera 2, and an image captured by the stereo camera 2 is sent to the image processing device 3. The image processing apparatus 3 performs calibration of the stereo camera 2 semi-automatically.

  Here, in order to simplify the description, the manner in which the left camera of the stereo camera 2 is calibrated will be described. The right camera calibration is performed in the same manner.

  First, the operator captures (captures) the calibration pattern 1 from an arbitrary direction with the stereo camera 2 (step S101). FIG. 23 shows an example image (left camera image) of the calibration pattern 1 captured by the stereo camera 2. This image is sent to the image processing apparatus 3.

  The operator manually designates four intersections of the squares in the captured calibration pattern 1 (step S102). Here, as shown in FIG. 23, intersections X1, X2, X3, and X4 of squares at the four corners are designated.

  When the intersection points X1, X2, X3, and X4 are designated, the image processing device 3 detects (automatically detects) all the coordinates of the intersection points within the designated four intersection points (step S103).

  Next, the operator changes the direction of the camera (step S105), shoots (captures) the calibration pattern 1 (step S101), and, in the same manner as described above, Four intersection points in the pattern are manually designated (step S102).

  Then, in the same manner as described above, the image processing apparatus 3 detects (automatically detects) all the coordinates of the intersections inside the designated four intersections (step S103).

  In this way, the operator images the calibration pattern 1 from at least three directions, and repeats manual designation of the intersections of arbitrary squares in the captured calibration pattern 1.

  When the automatic detection of the intersection coordinates of the calibration pattern 1 that has been imaged N or more (in this example, three or more per eye) is completed (YES in step S104), the image processing device 3 detects that. A calibration parameter is calculated based on the intersection coordinates (step S106).

  In this case, the image processing device 3 calculates the H matrix for each of the calibration patterns 1 taken from the N direction in association with the intersection coordinates with the actual calibration pattern 1 (see FIG. 24). ) The left camera calibration parameter is calculated from the calculated H matrix.

JP 2003-307466 A

3D CG from photographs, Xugang (Author), Modern Scientific Society, 2001. “OpenCV, camera calibration”, [May 13, 2010 search], Internet <URL: http://opencv.jp/sample/camera_calibration.html#calibration> C. Harris and M. Stephens: “A Combined Corner And Edge Detector”, The Plessey Company plc. 1988.

  However, according to the calibration method described above (hereinafter referred to as the conventional method), it is necessary to manually specify the intersection of arbitrary squares with respect to the captured calibration pattern, and the calibration processing time increases. There was a problem to do. Further, since the intersection point is manually specified, there is a risk that a manual error may occur. In addition, it is necessary to perform shooting so that the four corners of the calibration pattern are reflected, and adjustment is required during shooting. In addition, since the calibration pattern cannot be displayed on the entire screen (background appears on the screen), there is a problem in that the accuracy of calibration is reduced.

  In Patent Document 1, two types of marks are used as calibration patterns, the position of one pattern is manually specified, and the other pattern is automatically recognized based on the result, and precise association is performed. Like to do. However, the method disclosed in Patent Document 1 has features such as being able to recognize a pattern with high accuracy and being able to cope with lens distortion of a camera. Therefore, it is difficult to automate the process, and there is a risk that manual mistakes may occur.

  In Non-Patent Document 2, the recognition of the calibration pattern is automated, but with the method shown in Non-Patent Document 2, the number of checkered patterns reflected in the target pattern is known. It is assumed that all the patterns are reflected. For this reason, as in the conventional method described above, adjustment is required at the time of shooting, and the calibration pattern cannot be displayed on the entire screen (background appears on the screen), resulting in a decrease in calibration accuracy. There are problems such as.

  SUMMARY OF THE INVENTION The present invention has been made to solve such problems, and the object of the present invention is to provide a camera calibration method capable of eliminating manual mistakes, reducing processing time, and improving accuracy. Is to provide.

  In order to achieve such an object, the present invention provides a grid-like plane pattern in which colors in the grids are alternately inverted, and a reference mark provided at a predetermined position in the plane pattern. A calibration pattern having the following is captured by a camera from at least three directions, and a main matrix of a projective transformation matrix (H matrix) between each captured calibration pattern and an actual calibration pattern is calculated for each. A camera calibration method for calculating camera calibration parameters from the main matrix of the calculated H matrix, wherein a pattern imaging step of imaging a calibration pattern with a camera so that a reference mark is inserted, and the pattern imaging step An arbitrary square is formed from the captured calibration pattern. The intersection coordinate detection step for detecting the intersection coordinates of any square in the calibration pattern, and the calibration pattern imaged by the pattern imaging step based on the intersection coordinates detected by the intersection coordinate detection step and the actual coordinates A temporary matrix calculation step of calculating an H matrix between the calibration pattern and a temporary H matrix, and a calibration imaged by the pattern imaging step based on the temporary H matrix calculated by the temporary matrix calculation step Calculate the intersection candidate position of each square in the pattern, check whether each of the intersection candidate positions of each square is actually an intersection, and based on this confirmation result, the intersection coordinates of each square Is captured by the intersection coordinate estimation step and the pattern imaging step. The reference mark detection step for detecting the reference mark in the calibration pattern, and the position of the reference mark detected by the reference mark detection step as the coordinate reference, and the intersection coordinates of each square estimated by the intersection coordinate estimation step And a main matrix calculating step for calculating the main matrix of the H matrix between the calibration pattern imaged in the pattern imaging step and the actual calibration pattern.

  In the present invention, the calibration pattern includes a grid-like plane pattern (hereinafter, this plane pattern is referred to as a check pattern) arranged so that the colors in the grids are alternately inverted, and a predetermined pattern in the check pattern. And a reference mark provided at the position. In this calibration pattern, the reference mark only needs to be distinguished from the check pattern, and may be a dot mark, a triangle, or a cross. Further, for example, one of the check pattern squares may be changed in color, and the changed square may be used as a reference mark.

  In the present invention, the calibration pattern is imaged by the camera from at least three directions so that the reference mark is inserted. Then, the main matrix of the H matrix between each captured calibration pattern and the actual calibration pattern is calculated for each. The calculation of the main matrix of the H matrix is performed as follows.

  First, the intersection coordinates of arbitrary squares in the calibration pattern are detected from the captured calibration pattern so as to form an arbitrary quadrangle. For example, in the calibration pattern so as to form an arbitrary square, one intersection point is found near the center of the captured calibration pattern, and three or more intersection points are found near the found intersection point. The coordinates of the intersection of any square of are detected. Then, based on the detected intersection coordinates, an H matrix between the captured calibration pattern and the actual calibration pattern is calculated as a temporary H matrix, and an image is captured based on the calculated temporary H matrix. The intersection candidate position of each square in the calibration pattern is calculated.

  In this case, the intersection candidate position of each square is not always the intersection of each square. Therefore, it is confirmed whether or not each intersection candidate position of each square is actually an intersection, and the intersection coordinates of each square are estimated based on the confirmation result. For example, for each intersection candidate position of each square, a small area centered on the intersection candidate position is extracted, the image data of the small area is binarized, and based on the binarized image data of the small area It is determined whether or not it is actually an intersection, and the intersection coordinates are estimated only for the intersection candidate positions determined to be actually intersections. Then, a reference mark in the imaged calibration pattern is detected, and the position of the detected reference mark is used as a coordinate reference, and based on the estimated intersection coordinates of each square, an actual calibration pattern is detected. The main matrix of the H matrix is calculated.

  In this way, in the present invention, the main matrix of the H matrix between each captured calibration pattern and the actual calibration pattern is calculated for each, and the camera calibration is performed from the calculated main matrix of the H matrix. A parameter (calibration parameter) is calculated.

  According to the present invention, from the calibration pattern imaged so that the reference mark is inserted, the intersection coordinates of an arbitrary square in the calibration pattern are detected so as to form an arbitrary square, and the detected intersection point is detected. The intersection candidate position of each square is calculated based on the coordinates, the intersection coordinates of each square are estimated from the calculated intersection candidate positions, and the reference mark of the reference mark is calculated based on the estimated intersection coordinates of each square. The main matrix of the H matrix between the captured calibration pattern and the actual calibration pattern is calculated with the position as the coordinate reference, and the camera coordinates are automatically set without manually specifying the intersection coordinates. Calibration parameters can be obtained, so that manual errors can be eliminated and processing time can be shortened.

  Further, according to the present invention, it is only necessary to image the calibration pattern so that the reference mark is inserted, and it is not necessary to image the entire calibration pattern (the check pattern may protrude from the screen). Can be greatly reduced. Further, the calibration pattern can be projected on the entire screen (the background is not included in the screen), and the calibration accuracy can be improved.

It is a figure which shows an example of the calibration pattern used with the calibration method of the camera which concerns on this invention. It is a flowchart of the calibration performed using this calibration pattern. It is a figure which shows the outline of the system which performs calibration according to this flowchart. It is a figure which shows the example of an image of the calibration pattern imaged with the stereo camera. It is a flowchart of the calculation process of this matrix of the H matrix between the imaged calibration pattern and an actual calibration pattern. It is a figure which shows a mode that one intersection is detected in the center vicinity of the image of the imaged calibration pattern. It is a figure which shows a mode that three or more intersections are detected so that a quadrangle may be formed in the vicinity of one detected intersection. It is a figure which shows the other example which detects an intersection so that a quadrangle may be formed. It is a figure which shows a response | compatibility of the intersection coordinate at the time of calculating the temporary H matrix between the imaged calibration pattern and an actual calibration pattern. It is a figure which shows the intersection candidate position in the imaged calibration pattern calculated based on the calculated temporary H matrix. It is a flowchart of the intersection determination process which determines whether it is actually an intersection about the calculated intersection candidate position. It is a figure which shows a mode that the image block of the small area | region centering on the intersection candidate position is cut out from the imaged calibration pattern image. It is a figure which shows a mode that the binarization process linked with a luminance value is performed with respect to the image data of the image block of the cut out small area | region. It is a figure which shows a mode that the determination process of whether an intersection is contained in the image block of the binarized small area is performed. It is a figure which shows a mode that an intersection position is estimated with high precision in the image block of the small area | region determined that an intersection is actually included. It is a figure which shows the estimated intersection coordinate in the imaged calibration pattern. It is a figure which shows a correspondence of the intersection coordinate at the time of calculating the main matrix of H matrix between the imaged calibration pattern and an actual calibration pattern. It is a functional block diagram of the principal part of an image processing apparatus. It is a figure which shows the outline | summary of the three-dimensional measurement using a stereo camera. It is a figure which shows an example of the conventional calibration pattern used for the calibration of a stereo camera. It is a flowchart (conventional system) of the calibration using this calibration pattern. It is a figure which shows the outline of the system which performs calibration according to this flowchart. It is a figure which shows the example of an image of the calibration pattern imaged with the stereo camera. It is a figure which shows a response | compatibility of the intersection coordinate at the time of calculating H matrix between the imaged calibration pattern and an actual calibration pattern.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Calibration pattern]
FIG. 1 is a diagram showing an example of a calibration pattern used in the camera calibration method according to the present invention. The calibration pattern 4 includes a grid-like plane pattern (check pattern) 4A arranged so that the colors in the grids are alternately inverted, and a reference mark provided at a predetermined position in the check pattern 4A. 4B.

  In this embodiment, the check pattern 4A is a checkered pattern in which white and black cells are alternately arranged in a grid pattern. In addition, the reference mark 4B is written as a dot mark in the grid at the center of the check pattern 4A.

  The reference mark 4B only needs to be distinguishable from the check pattern 4A, and may be a triangle or a cross. Alternatively, the square of the check pattern 4A may be changed (gray, red, etc.) and the changed square may be used as a reference mark.

  FIG. 2 shows a flow chart of calibration performed using this calibration pattern 4. FIG. 3 shows an outline of a system for performing calibration according to this flowchart. In this system, the calibration pattern 4 is photographed (captured) by the stereo camera 2, and the image captured by the stereo camera 2 is sent to the image processing device 5. The image processing apparatus 5 performs calibration of the stereo camera 2 fully automatically.

  The image processing device 5 is realized by hardware including a processor and a storage device, and a program that realizes various functions in cooperation with these hardware, and has an automatic calibration function as a function unique to the present embodiment. doing.

  Here, in order to simplify the description, a state in which the right camera of the stereo camera 2 is calibrated will be described. The left camera calibration is performed in the same manner.

  First, the operator captures (captures) the calibration pattern 4 with the stereo camera 2 so that the reference mark 4B enters from at least three directions (step S201). FIG. 4 shows an example of the calibration pattern 4 captured by the stereo camera 2 (image captured by the right camera). This image is sent to the image processing device 5.

  Then, the image processing apparatus 5 calculates the main matrix of the H matrix between each captured calibration pattern 4 and the actual calibration pattern 4 (step S202), and calculates the calculated H matrix. The right camera calibration parameter is calculated from the matrix (step S203).

[Calculation of the main matrix of the H matrix]
FIG. 5 shows a flowchart of the main matrix calculation process in step S202. According to this flowchart, the image processing apparatus 5 first calculates a temporary matrix of the H matrix, calculates an intersection candidate position in the calibration pattern imaged from the calculated temporary matrix of the H matrix, and calculates the intersection point from the intersection candidate position. The coordinates are estimated, and the main matrix of the H matrix is obtained using the position of the reference mark as the coordinate reference based on the estimated intersection coordinates.

[Intersection detection]
The image processing apparatus 5 finds one intersection of the squares of the calibration pattern 4 included in the image near the center of the captured image of the calibration pattern 4 (step S301). P1 shown in FIG. 6 is the detected intersection. In FIG. 6, the x mark is determined not to be an intersection. The search range is near the center of the image, and the step size for search is about 1/3 of the cut-out block size.

  Next, the image processing apparatus 5 finds three or more intersections in the vicinity of the intersection found in step S301 (step S302). At this time, intersections are found so that at least four of the obtained points form a quadrangle.

  In this case, for example, the points should not be placed so that the intersections are arranged in a straight line. Moreover, it is necessary to know the relative positional relationship of the found intersection. These restrictions are indispensable conditions for calculating the H matrix.

  FIG. 7 shows an example in which intersection points are found so as to form a square. In this example, intersections P2, P3, P4, and P5 are found so as to form an arbitrary quadrangle near the intersection P1 that is found first. As shown in FIG. 8A, the four corners of one square may be found as intersections P1, P2, P3, and P4. As shown in FIG. The intersection points P2 to P9 may be found so as to form a quadrangle composed of squares.

[Calculation of temporary H matrix]
Based on the coordinates (intersection coordinates) of the intersections of the intersection points P1, P2, P3, P4, and P5 detected in steps S301 and S302, the image processing apparatus 5 captures the captured calibration pattern 4 and the actual calibration pattern 4. Is calculated as a temporary H matrix (step S303: see FIG. 9).

  In this case, the H matrix can be calculated from Equation (5) described above. Here, the reason why the H matrix is assumed is that the positions of the corresponding intersections of the actual calibration pattern 4 and the captured calibration pattern 4 are shifted. Ultimately, the coordinates of both intersection points are associated with the position of the reference mark 4B in the captured calibration pattern 4 as a coordinate reference. This process is performed in steps S307 and S308 described later.

[Calculation of intersection candidate position]
Next, the image processing apparatus 5 calculates the intersection candidate position of each square in the captured calibration pattern 4 based on the temporary H matrix calculated in step S303 (step S304). FIG. 10 shows the calculated intersection candidate positions (x marks in the figure).

  If the H matrix is known, the intersection coordinates on the calibration pattern 4 captured from the intersection coordinates of the above-described equation (5) and the actual calibration pattern 4 can be estimated. For the intersection coordinates of the actual calibration pattern 4, a predetermined grid size may be used as it is.

  As can be seen from the results shown in FIG. 10, all the positions that are considered to be intersection candidates are marked with x. Here, since all the intersection candidate positions included in the image are displayed, there are portions where the pattern is interrupted, such as an image end. Further, although it is difficult to see from the figure, there is a slight deviation between the intersection candidate position and the actual intersection position. This problem is solved by the subsequent steps S305 and S306.

[Judgment of intersection]
The image processing apparatus 5 determines whether or not each intersection candidate position of each square calculated in step S304 is actually an intersection (step S305). FIG. 11 shows a flowchart of the intersection determination process in this case.

  The image processing apparatus 5 cuts out an image block BL of a small area centered on the intersection candidate position for each intersection candidate position of each square from the captured image of the calibration pattern 4 (step S401: FIG. 12). reference).

  In this example, the size of the image block BL is 11 × 11 pixels or more. The smaller block size allows detection even if the grid size on the image is smaller. However, since the detection accuracy deteriorates when the block size is smaller than 11 × 11 pixels, this value is used. The calibration pattern 4 desirably has a square size of 20 × 20 or more on the image. Note that the grid size of the calibration pattern 4 can be freely determined.

  Next, the image processing apparatus 5 performs binarization processing in conjunction with the luminance value on each of the small area image blocks BL cut out in step S401 (see step S402, FIG. 13). Specifically, binarization is performed using the average luminance value in the image block BL. Since the target pattern is not always photographed at a constant brightness, this processing can correctly divide the pattern into a white pattern and a black pattern.

  Then, the image processing apparatus 5 looks at the four corner areas of the binarized image clockwise, and extracts the extracted image depending on whether black → white → black → white or white → black → white → black. It is determined whether or not an intersection is included in the block BL (see step S403, FIG. 14). That is, if black → white → black → white, or white → black → white → black, the intersection candidate position from which the image block BL of the small area is extracted is determined to be an actual intersection.

  Here, in order to stabilize the process, not only the values at the four corners but also 2 × 2 small blocks are cut out from the four corners, and the result is voted for determination. As a whole, a total of 16 points of data of 4 2 × 2 small blocks are obtained, and if 14 or more of them match the black and white judgment condition, it is judged as an intersection. This process has an effect of preventing the data at the four corners from being erroneously determined in accordance with the conditions.

  It should be noted that the intersection determination cannot be made if the target grid pattern is rotated by a large angle such as 45 degrees with respect to the axis (Z-axis) in the line of sight of the camera. However, since the target pattern is not rotated and calibrated, it is not considered.

[Estimation of intersection coordinates]
Next, the image processing apparatus 5 estimates the intersection position in the image block BL with high accuracy only for the intersection candidate positions determined to be actually intersections in step S305 (see FIG. 15). The intersection coordinates thus updated are updated as intersection positions (step S306).

  Note that the high-precision estimation of the intersection position in step S306 is performed by a Harris corner detector (see Non-Patent Document 3). The basic idea of the Harris corner detector is to detect the most rapidly changing portion by looking at the gradient of the luminance values in the vertical and horizontal directions. If there is an intersection in the image, the point is detected because the luminance change is large both in the vertical and horizontal directions.

  FIG. 16 shows the intersection coordinates (marked with ● in the figure) estimated in step S306. As a result of the intersection determination in step S305, it can be seen that the end point of the image that has been the intersection candidate position in FIG.

[Detection of fiducial mark]
Next, the image processing apparatus 5 detects the reference mark 4B in the captured calibration pattern 4 (step S307). In this process, since the position of each square is known, the reference mark 4B is found based on whether or not there is a dot mark in the white square. Since the reference mark 4B is found by this process, the correct correspondence of the intersection between the actual calibration pattern 4 and the captured calibration pattern 4 can be known.

[Calculation of this matrix of H matrix]
Next, the image processing apparatus 5 uses the position of the reference mark 4B as a coordinate reference, and based on the intersection coordinates of each square estimated in step S306, the captured calibration pattern 4 and the actual calibration pattern 4 The main matrix of the H matrix is calculated (step S308).

  As a result, the main matrix of the H matrix between the captured calibration pattern 4 and the actual calibration pattern 4 is calculated using the intersection coordinates of the square where the reference mark 4B is provided as a reference point (see FIG. 17)

  As can be seen from the above description, in the present embodiment, an arbitrary quadrangle in the calibration pattern 4 is automatically formed so as to automatically form an arbitrary quadrilateral from the calibration pattern 4 imaged so that the reference mark 4B enters. The intersection coordinates of each square are detected, the intersection candidate positions of each square are calculated based on the detected intersection coordinates, and the intersection coordinates of each square are estimated from the calculated intersection candidate positions. The main matrix of the H matrix between the captured calibration pattern 4 and the actual calibration pattern 4 is calculated using the position of the reference mark 4B as a coordinate reference on the basis of the intersection coordinates of each square. It becomes. As a result, the calibration parameters can be obtained fully automatically without manually specifying the intersection coordinates, and it is possible to eliminate manual errors and shorten the processing time.

  Further, according to the present embodiment, it is only necessary to photograph the calibration pattern 4 so that the reference mark 4B is inserted, and it is not necessary to photograph the entire calibration pattern 4 (the check pattern 4A may protrude from the screen). ), It is possible to greatly reduce the trouble of adjustment at the time of shooting. In other words, in the conventional method, when the calibration pattern is projected on the entire screen, it is necessary to perform adjustment so that the four manually designated points are almost at the edge of the image. It is sufficient that only the mark 4B is shown, and the labor of adjustment for projecting the calibration pattern 4 can be greatly reduced. In addition, since the calibration pattern 4 can be projected on the entire screen (the background is not included in the screen), the accuracy of calibration can be improved.

  FIG. 18 shows a functional block diagram of the main part of the image processing apparatus 5. The image processing apparatus 5 is imaged by the stereo camera 2 and the actual pattern storage unit 51 that stores data such as the intersection coordinates of each square of the actual calibration pattern 4 and the position of the reference mark 4B as actual pattern data. An image capturing unit 52 that captures an image of the calibration pattern 4, and the intersection coordinates of arbitrary squares of the calibration pattern 4 that are captured so as to form an arbitrary quadrilateral from the image captured by the image capturing unit 52 And an intersection coordinate detection unit 53 for detecting. When the calibration pattern 4 is photographed by the stereo camera 2, the calibration pattern 4 is photographed so that the reference mark 4B is inserted, and the captured calibration pattern 4 always includes the reference mark 4B. And

  Further, the image processing device 5 calculates the H matrix between the captured calibration pattern 4 and the actual calibration pattern 4 as a temporary H matrix based on the intersection coordinates detected by the intersection coordinate detection unit 53. And calculating the intersection candidate position of each square in the captured calibration pattern 4 based on the temporary H matrix calculated by the temporary matrix calculating section 54 and the temporary matrix calculating section 54. It is confirmed whether or not each of the intersection candidate positions is actually an intersection, and based on the confirmation result, an intersection coordinate estimation unit 55 that estimates the intersection coordinates of each square, and the captured calibration pattern 4 And a reference mark detection unit 56 for detecting the reference mark 4B.

  Further, the image processing device 5 uses the position of the reference mark 4B detected by the reference mark detection unit 56 as a coordinate reference, and is based on the intersection coordinates estimated by the intersection coordinate estimation unit 55 and captured calibration. The main matrix calculation unit 57 for calculating the main matrix of the H matrix between the calibration pattern 4 and the actual calibration pattern 4, and the stereo camera 2 from the main matrix of at least three H matrices calculated by the main matrix calculation unit 57. And a calibration parameter calculation unit 58 for calculating the calibration parameters.

  In the image processing device 5, the intersection coordinate estimation unit 55 is actually configured by the intersection determination unit 55A that determines whether or not each intersection candidate position of each square is actually an intersection, and the intersection determination unit 55A. An intersection estimation unit 55B that estimates the intersection coordinates only for the intersection candidate positions determined to be intersections is provided. For each of the intersection candidate positions of each square, the intersection determination unit 55A extracts a small area centering on the intersection candidate position and a small area extracted by the small area extraction unit 55A1. , The binarization processing unit 55A2 for binarizing the image data of the small area, and the intersection of the extraction source of the small area based on the image data of the small area binarized by the binarization processing unit 55A2 A determination unit 55A3 that determines whether the candidate position is actually an intersection.

  The camera calibration method of the present invention can be applied to various cameras such as a stereo camera as a camera calibration method for calculating camera calibration parameters such as the position and orientation of the camera, the image center and scale of the camera. It is possible to expect various advantages such as high-speed processing, elimination of manual mistakes, and performance improvement. In addition, it is possible to cope with mass production of stereo cameras and to realize simple calibration on site.

  2 ... Stereo camera, 4 ... Calibration pattern, 4A ... Planar pattern (check pattern), 4B ... Reference mark, 5 ... Image processing device, 51 ... Actual pattern storage unit, 52 ... Image capture unit, 53 ... Intersection coordinate detection 54: provisional matrix calculation unit 55 ... intersection coordinate estimation unit 55A ... intersection determination unit 55A1 ... small region extraction unit 55A2 ... binarization processing unit 55A3 ... determination unit 55B ... intersection estimation unit 56 ... Reference mark detection unit, 57... Matrix calculation unit, 58 .. calibration parameter calculation unit, BL (BL1, BL2)... Image block, G1... Left camera image, G2.

Claims (3)

A calibration pattern having a grid-like plane pattern arranged so that the colors in the grids are alternately reversed and a reference mark provided at a predetermined position in the plane pattern is displayed by a camera from at least three directions. The main matrix of the projection transformation matrix between each captured calibration pattern and the actual calibration pattern is calculated for each individual, and the calibration parameters of the camera are calculated from the main matrix of the calculated projection transformation matrix. A camera calibration method to calculate,
A pattern imaging step of imaging the calibration pattern by the camera so that the reference mark enters;
An intersection coordinate detection step for detecting an intersection coordinate of an arbitrary square in the calibration pattern so as to form an arbitrary quadrilateral from the calibration pattern imaged by the pattern imaging step;
Based on the intersection coordinates detected by the intersection coordinate detection step, a temporary transformation matrix between the calibration pattern imaged by the pattern imaging step and the actual calibration pattern is calculated as a temporary projection transformation matrix. Matrix calculation step;
Based on the temporary projective transformation matrix calculated by the temporary matrix calculation step, the intersection candidate position of each square in the calibration pattern imaged by the pattern imaging step is calculated, and the intersection candidate position of each square An intersection coordinate estimation step for confirming whether or not each is actually an intersection, and estimating the intersection coordinates of each square based on the confirmation result;
A reference mark detection step of detecting the reference mark in the calibration pattern imaged by the pattern imaging step;
The position of the reference mark detected by the reference mark detection step is used as a coordinate reference, and the calibration pattern imaged by the pattern imaging step and the calibration pattern imaged by the pattern imaging step based on the intersection coordinates of each square estimated by the intersection coordinate estimation step And a matrix calculation step of calculating a matrix of a projection transformation matrix between the actual calibration pattern and a calibration pattern. The camera calibration method according to claim 1,
The intersection coordinate estimation step includes:
An intersection determination step for determining whether or not each intersection candidate position of each square is actually an intersection; and
A camera calibration method comprising: an intersection estimation step for estimating an intersection coordinate only for an intersection candidate position determined to be an actual intersection by the intersection determination step. The camera calibration method according to claim 2,
The intersection determination step includes
A small region extraction step for extracting a small region centered on the intersection candidate position for each intersection candidate position of each square;
For each of the small areas extracted by the small area extraction step, a binarization processing step for binarizing the image data of the small area;
A determination step of determining whether the intersection candidate position from which the small area is extracted is actually an intersection based on the image data of the small area binarized by the binarization processing step. A camera calibration method characterized by the above.

Images