JP2005351883A - Index calibration device and information processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire simply and accurately the position or the position attitude of an index mounted on an imaging device relative to the imaging device. <P>SOLUTION: This index calibration device for calculating arrangement information of a calibration object index mounted on the imaging device relative to the imaging device as calibration information is characterized by including an objective viewpoint imaging means for imaging the imaging device from an objective viewpoint position, the first image input means for inputting the first image from the imaging device, the second image input means for inputting the second image from the objective viewpoint imaging means, the first detection means for detecting information on an image coordinate of a standard index arranged in a scene from the first image, the second detection means for detecting information on an image coordinate of the calibration object index from the second image, and a calibration information calculation means for calculating the calibration information by using the information on the image coordinates of respective indexes detected by the the first detection means and the second detection means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention calibrates arrangement information of an index attached to an imaging apparatus with respect to the imaging apparatus.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents a mixed reality is drawn on a virtual space (for example, computer graphics) generated in accordance with the position and orientation of an imaging device on an image of a real space that is captured by an imaging device such as a video camera. This is realized by a video see-through method in which an image of a virtual object, character information, etc.) is overlaid and displayed.

  Applications of such image display devices include virtual reality, such as surgical support that superimposes the state of the body on the patient's body surface, and mixed reality games that fight against virtual enemies floating in real space. Different new fields are expected.

  A common requirement for these applications is how to accurately align the real space and the virtual space, and many efforts have been made in the past. The alignment problem in mixed reality results in the problem of determining the position and orientation of the imaging device in the scene (ie, in the world coordinate system).

  As a method of solving this problem, by arranging or setting a plurality of indicators in the scene, using the coordinates in the world coordinate system of the indicators and the coordinates of the projected image of the indicators in the image captured by the imaging device, Generally, the position and orientation of the imaging device in a scene are obtained.

  A method for calculating the position and orientation of an imaging device from a set of coordinates in a world coordinate system of an index in a scene and image coordinates of a projected image has been proposed for a long time in the field of photogrammetry (for example, non-patent literature). 1, see Non-Patent Document 2).

  In addition, by setting a plurality of indices on the measurement target object, photographing the target object with an objective viewpoint camera installed outside, and detecting the image coordinates of the projected image of the index in the captured objective viewpoint image, The position and orientation are also obtained (see, for example, Non-Patent Document 2 and Patent Document 1).

Further, the inventors of the present invention also calculate the position and orientation of the imaging device by detecting the projected image of the index in the scene from the image captured by the imaging device itself that is the measurement target object in Japanese Patent Application No. 2003-323101. The method and the method for calculating the position and orientation of the imaging device by detecting the projected image of the index set on the imaging device itself from the objective viewpoint image obtained by imaging the imaging device that is the measurement target object from the objective viewpoint position Thus, a method for measuring the position and orientation of the imaging apparatus has been proposed.
JP 09-323280 A R. M.M. Haralick, C.I. Lee, K.M. Ottenberg, and M.M. Nole: Review and analysis of solutions of the three point perspective positive estimation lab, International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994. D. G. Low: Fitting parametrized three-dimensional models to images, IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-450, 1991. Uchiyama, Yamamoto, Tamura, Hybrid Registration Method for Mixed Reality -Combination of 6 DOF Sensor and Vision Method-, Transactions of the Virtual Reality Society of Japan, vol. 8, no. 1, pp. 119-125, 2003. Fujii, Kanbara, Iwasa, Takemura, Yokoya, Positioning with stereo camera combined with gyro sensor for augmented reality, IEICE Technical Report PRMU99-192 (Science Tech. Vol.99, no. 574, pp. .1-8), 1999. A. I. Comfort, E.M. Marchand, F.M. Chaumette, A real-time tracker for markerless augmented reality, Proc. Int'l Symp. on Mixed and Augmented Reality 2004, pp. 36-45, 2004.

  By the way, in the method of calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the objective viewpoint image, the relative of the plurality of indices set to the measurement target object and the measurement target object The correct positional relationship must be known. When the relative positional relationship of each index is unknown, for example, when using the colored spherical index or the circular index and using the center of gravity of the projected image of the index on the image as a feature In addition, in the case where an index whose feature is described by one point is individually installed on the measurement target object, the three-dimensional position of each index in the measurement target object coordinate system is not measured in advance. Must not. On the other hand, when the relative positional relationship between the indices is known, for example, even when the indices similar to those described above are used, these indices are mounted on a base jig in advance. The position of each index has already been measured in this coordinate system (the coordinate system for describing the relative position of the index is referred to as the index coordinate system in the following), and this jig is attached to the object to be measured In such a case, the position and orientation of the jig in the measurement target object coordinate system need only be known (in addition, an index whose feature is described by a plurality of points, for example, a known shape If each vertex of the projected image of the index on the image is used as a feature using a square or triangle index, the index itself is interpreted as a collection of a plurality of indices, "When the positional relationship is known" To be considered as). However, a method for accurately calibrating them is not generally known, and until now there has been no measure other than using an inaccurate position or position and orientation obtained by manual measurement as a known value. As a result, in the method for calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the objective viewpoint image, the position and orientation of the target object can be measured only with practically low accuracy. There was room for improvement.

  The present invention has been made in view of the above problems, and an object of the present invention is to enable easy and accurate acquisition of index placement information with respect to an imaging apparatus.

  In order to achieve the above object, the invention according to claim 1 is an index calibration device that calculates, as calibration information, arrangement information of a calibration target index mounted on an imaging device with respect to the imaging device, and the imaging device is an objective viewpoint position. Objective viewpoint imaging means for photographing from, first image input means for inputting a first image from the imaging device, second image input means for inputting a second image from the objective viewpoint imaging means, and a scene First detection means for detecting information about the image coordinates of the reference index arranged in the first image, and second detection means for detecting information about the image coordinates of the calibration target index from the second image And calibration information calculation means for calculating the calibration information using information on the image coordinates of the respective indices detected by the first detection means and the second detection means.

  According to the present invention, it is possible to easily and accurately acquire index placement information with respect to an imaging apparatus.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
The index calibration apparatus according to the present embodiment acquires the position of the index attached to the imaging apparatus with respect to the imaging apparatus for each index. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

  FIG. 1 is a diagram illustrating a schematic configuration of an index calibration apparatus 100 according to the present embodiment. As shown in FIG. 1, the index calibration apparatus 100 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, data management. The unit 170 and the calibration information calculation unit 180 are connected to the imaging device 130 that is a calibration target.

On the imaging device 130, an index P ^k (k = 1,, K ₂ ) (hereinafter referred to as an objective viewpoint) to be calibrated by a user who uses the index calibration apparatus according to the present embodiment. It is assumed that at least one (referred to as an index) is set. The position of the objective viewpoint index in the subjective viewpoint camera coordinate system (a coordinate system in which one point on the imaging device 130 is defined as the origin and three orthogonal axes are defined as the X axis, the Y axis, and the Z axis, respectively) is unknown. The position of the objective viewpoint index in the unknown subjective viewpoint camera coordinate system is information that is calibrated by the index calibration apparatus according to the present embodiment, that is, the output of the index calibration apparatus according to the present embodiment.

At a plurality of positions in the real space, the world coordinate system (one point in the real space is defined as the origin as an index for photographing with the imaging device 130 (hereinafter referred to as a subjective viewpoint index (reference index)) and orthogonal to each other. A plurality of subjective viewpoint indices Q ^k (k = 1,, K ₁ ) whose positions in the three axes are defined in the coordinate system defined as the X axis, the Y axis, and the Z axis, respectively, are arranged. It is desirable that the subjective viewpoint index Q ^k is installed so that at least three or more indices are observed by the imaging device 130 when acquiring the data for index calibration. In the example of FIG. 1, four subjective viewpoint indices Q ¹ , Q ² , Q ³ , and Q ⁴ are arranged, and three of them, Q ¹ , Q ³ , and Q ^4, are included in the field of view of the imaging device 130. Shows the situation.

The subjective viewpoint index Q ^k may be constituted by, for example, circular markers each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a square area having a certain area. Any form may be used as long as the image coordinates of the projected image on the photographed image can be detected and the index can be identified.

  An image output from the imaging device 130 (hereinafter referred to as a subjective viewpoint image) is input to the subjective viewpoint index detection unit 110.

Person view marker detecting unit 110 inputs the subjective-view image from the imaging device 130, detects the image coordinates of the subjective-view-indices Q ^k has been taken in the input image. For example, when each of the subjective viewpoint indices Q ^k is composed of markers having different colors, an area corresponding to each marker color is detected from the subjective viewpoint image, and the center of gravity position is detected as the detected coordinate of the index. To do. Further, when each of the subjective viewpoint indices Q ^k is composed of feature points having different texture characteristics, template matching based on the template image of each index held in advance as known information is performed on the subjective viewpoint image. By applying, the position of the index is detected. When a quadratic index is used as the reference index, labeling is performed after binarizing the image, and an area formed by four straight lines is detected as an index candidate from a region having a certain area or more. Further, by detecting whether or not there is a specific pattern in the candidate area, false detection is eliminated, and the direction and identifier of the index are acquired. In the present specification, the quadrangular indices detected in this way are considered to be four indices formed by four vertices.

Furthermore, subjective-view-index detection unit 110 outputs the image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} each detected and the identifier _{k n} to the position and orientation calculation unit 120. Here, n (n = 1,..., N) is an index for each detected index, and N represents the total number of detected indices. For example, in the case of FIG. 1, N = 3, identifiers k ₁ = 1, k ₂ = 3, k ₃ = 4 and the corresponding image coordinates u ^Qk1 , u ^Qk2 , u ^Qk3 are output. The subjective viewpoint index detection unit 110 according to the present embodiment sequentially executes the index detection process triggered by the input of an image from the imaging device 130, but in response to a request from the position / orientation calculation unit 120 (at that time) The process may be executed using the subjective viewpoint image input in (1).

The position / orientation calculation unit 120 is based on the correspondence relationship between the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn and the world coordinates x _W ^Qkn of the indices previously stored as known information. 130 positions and orientations are calculated. A method for calculating the position and orientation of an imaging device from a set of world coordinates and image coordinates of a subjective viewpoint index has been proposed for a long time in the field of photogrammetry and the like (for example, see Non-Patent Document 1 and Non-Patent Document 2). ). For example, when the subjective viewpoint indices are arranged on the same plane, the position and orientation of the imaging device can be obtained based on the calculation of two-dimensional homography by detecting four or more indices. In addition, a method of using the position and orientation of an imaging device using six or more indices that are not on the same plane is also well known. In addition, the error between the theoretical value of the index image coordinates calculated from the estimated position and orientation of the imaging device and the actual measurement value of the index image coordinates is calculated by repeated calculation such as the Gauss-Newton method using the image Jacobian. By minimizing, a method for optimizing the estimated values of the position and orientation can be used. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 170 in accordance with a request from the data management unit 170. Note that the position / orientation calculation unit 120 in this embodiment sequentially executes the above-described position and orientation calculation processing triggered by the input of data from the subjective viewpoint index detection unit 110, but the request from the data management unit 170 It is also possible to adopt a configuration in which processing is executed in accordance with (using input data at that time). In the following, the position and orientation of the imaging device 130 are held by a 4 × 4 homogeneous coordinate matrix M _WC (modeling conversion matrix; a matrix that converts coordinates in the subjective viewpoint camera coordinate system into coordinates in the world coordinate system). Shall.

  The objective viewpoint camera 140 is fixedly arranged at a position where the imaging device 130 for acquiring the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 180 as known values.

The objective viewpoint index detection unit 150 inputs an image (objective viewpoint image) captured by the objective viewpoint camera 140, and an image of the objective viewpoint index ^Pk captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the coordinates, and outputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 170 and the identifier _{k m} to the data management unit 170. Here, m (m = 1,..., M) is an index attached to each detected index, and M represents the total number of detected indices. For example, in the case of FIG. 1, M = 2, identifiers k ₁ = 1, k ₂ = 2 and image coordinates u ^Pk1 and u ^Pk2 corresponding to these are output. The objective viewpoint index detection unit 150 in the present embodiment sequentially executes the above-described index detection processing triggered by the input of the objective viewpoint image, but in response to a request from the data management unit 170 (input at that time) The processing may be performed using an objective viewpoint image.

  The instruction unit 160 sends a “data acquisition” instruction to the data management unit 170 when a data acquisition command is input from an operator (not shown), and a “calibration information calculation” instruction when the calibration information calculation command is input. Transmit to the calculation unit 180. The command input to the instruction unit 160 can be performed by pressing a key to which a specific command is assigned, for example, using a keyboard. In addition, the command may be input by any method such as a GUI displayed on the display.

  When receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 170 inputs the position and orientation of the imaging apparatus 130 in the world coordinate system from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and the set of [position and orientation of the imaging device 130-image coordinate of the objective viewpoint index] is added to the data list prepared for each identifier of the objective viewpoint index and is retained. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. Further, the data management unit 170 outputs a data list for each identifier of the generated objective viewpoint index to the calibration information calculation unit 180 in accordance with a request from the calibration information calculation unit 180.

  Upon receiving an instruction “calculation information calculation” from the instruction unit 160, the calibration information calculation unit 180 inputs a data list from the data management unit 170, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position of each objective viewpoint index in the subjective viewpoint camera coordinate system) is output.

  FIG. 2 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S2010, the instruction unit 160 determines whether or not a data acquisition command has been input from the operator. The operator inputs a data acquisition command when the imaging device 130 is placed at a position where data for index calibration is acquired. If the data acquisition command is input, the instruction unit 160 shifts the process to step S2020.

In step S2020, the data management unit 170 inputs the position and orientation M _WC of the imaging apparatus 130 in the world coordinate system from the position / orientation calculation unit 120.

In step S2030, the data management unit 170, the objective-view-index detection unit 150, and inputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected by an objective-view-index detection unit 150 and the identifier _{k m.} By the above processing of steps S2020, S2030, the position and orientation of the imaging device can obtain an image coordinate of the objective-view-index when an _{M WC.} Note that the information input from the objective viewpoint index detection unit 150 is not necessarily related to all objective viewpoint indices, and may be information regarding the index detected on the objective viewpoint image at that time.

Next, in step S2040, the data management unit 170 adds the input data set to the data list L ^Pk for each detected objective viewpoint index P ^km . Specifically, the M _WC input from the position / orientation calculation unit 120 is M _WCi , the u ^Pkm input from the objective viewpoint index detection unit 150 is u _i ^Pk , and the set of [M _WCi , u _i ^Pk ] is objective. while sorting by referring to the identifiers k _m input from the view-index detection unit 150, and registers in the data list L ^Pk on objective view-index P ^k as i-th data about P ^k. Here, i (i = 1,..., I ^Pk ) is an index for each set of data registered in the data list L ^Pk , and I ^Pk represents the total number of sets of data registered for the objective viewpoint index P ^k. ing.

  Data is acquired by the above processing.

In step S2050, the data list related to all objective viewpoint indexes or the data list related to at least one objective viewpoint index acquired so far by the data management unit 170 is sufficient to calculate calibration information. A determination is made as to whether it has. If at least one data list or all data lists do not satisfy the condition, the process returns to step S2010 and waits for the input of a data acquisition command. On the other hand, if all the data lists or at least one data list satisfies the condition for calculating the calibration information, the process proceeds to step S2060. As a condition that a data list related to an objective viewpoint index P ^k has enough information to calculate calibration information, for example, two or more sets of data [M _WCi , u _i ^Pk ] having different data lists L ^Pk are obtained. It can be mentioned. As will be described later, two equations are obtained from one set of data (see Equation 3). If two or more sets of data are obtained, the subjective viewpoint camera coordinates of the objective viewpoint index, which is three parameters from the four equations, are obtained. The position in the system can be determined. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S2070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 180 handles the calibration information to be obtained, that is, the position of the objective viewpoint index in the subjective viewpoint camera coordinate system as a ternary vector [x _C y _C z _C ] ^T. Hereinafter, this unknown parameter is described as a state vector s ^Pk = [x _C ^Pky y _C ^Pk z _C ^Pk ] ^T. Hereinafter, the description will be continued assuming a process for a certain objective viewpoint index ^Pk, but each process is performed in common for all objective viewpoint indices corresponding to the data list that satisfies the conditions for calculating the calibration information. .

In step S2070, the calibration information calculation unit 180 gives an appropriate initial value (for example, [000] ^T ) to the state vector s ^Pk .

In step S2080, the calibration information calculation unit 180 calculates all _{i values from} each data [M _WCi , u _i ^Pk ] (i = 1, 2, _... , I ^Pk ) and the state vector s ^{Pk in} the data list L ^Pk. against an objective-view-index ^{P k} of the objective viewpoint image coordinates of theory ^{_{u i Pk '= [u xi}} Pk', u yi Pk '] is calculated. Here, the theoretical value of the objective viewpoint image coordinates of the objective viewpoint index refers to data of the position (coordinates) that should be visible in the objective viewpoint image of the objective viewpoint index P ^k given the position in the subjective viewpoint camera coordinate system. . The calculation of u _i ^Pk ′ is a function of the state vector s ^Pk representing the position of the objective viewpoint index P ^{k in} the subjective viewpoint camera coordinate system.

に基づいて行われる。

Based on.

Specifically, the function F _i () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^k when the i-th data is acquired (that is, when the position and orientation of the imaging device 130 are M _WCi ). The following equation for ^{obtaining the} position vector x _Bi ^Pk from s ^Pk :

And the following equation for ^obtaining the coordinates u _i ^Pk ′ on the objective viewpoint image of the objective viewpoint index P ^k from x _Bi ^Pk ,

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. _MWB is a transformation matrix for converting coordinates in the objective viewpoint camera coordinate system into world coordinates, and is calculated in advance based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that

In step S2090, the calibration information calculating unit 180, for all i, the actual image coordinates _u ^{i Pk} objective-view-index ^{P k} included in each data in the data list ^{L Pk,} the image coordinates of the corresponding an error △ _u ^{i Pk} of the theoretical value _u ^{i Pk} 'is calculated by the following equation.

In step S2100, the calibration information calculating unit 180, for all i, the image Jacobian about the state vector ^{s Pk} (i.e., solutions obtained by partially differentiating the function of Equation 1 _F i () for each element of the state vector ^{s Pk} Jacobian matrix of 2 rows × 3 columns having, in each _{^{element) J uis Pk (= ∂u i}} Pk / ∂s Pk) is calculated. Specifically, a 2-row × 3-column Jacobian matrix J _uxBi ^Pk (= ∂u) having a solution obtained by partial differentiation of the right side of Expression 3 with each element of the position vector x _Bi ^Pk on the objective viewpoint camera coordinate system. _i ^Pk / ∂x _Bi ^Pk ) and a 3 × 3 Jacobian matrix J _xBis ^Pk (= ∂x _Bi ^Pk /) having a solution obtained by partial differentiation of the right side of Equation 2 with each element of the state vector s ^Pk. ∂s ^Pk), and calculates the _J ^{uis Pk} by the following equation.

In step S2110, the calibration information calculating unit 180 is calculated in the above step, based on the error △ _u ^{i Pk} and the Jacobian matrix _J ^{uis Pk} for all ^i, and calculates the correction value △ ^{s Pk} of ^{s Pk.} Specifically, an error vector of a 2I ^Pk dimensional vector in which errors Δu _i ^Pk for all i are arranged vertically

And a Jacobian matrix _Juis ^Pk vertically arranged in 2I ^Pk rows × 3 columns matrix

And ^{Δs Pk} is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs ^Pk is a three-dimensional vector, if 2I ^Pk is 3 or more, that is, I ^Pk is 2 or more, ^{Δs Pk} can be obtained. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

In step S2120, the calibration information calculation unit 180 uses the correction value Δs ^Pk calculated in step S2110 to correct the position vector s ^Pk in the subjective viewpoint camera coordinate system of the objective viewpoint index P _k according to Equation 11 to obtain The new value is set as a new ^sPk .

In step S2130, the calibration information calculation unit 180 uses some criteria such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs ^Pk is smaller than a predetermined threshold. It is determined whether or not it has converged. If not converged, the processing after step S2080 is performed again using the corrected state vector s ^Pk .

The calculation is determined to have converged in step S2130, in step S2140, the calibration information calculating unit 180, the state vector s ^Pk obtained, as a parameter indicating the position in the subjective-view camera coordinate system of the objective-view-index P ^k Output.

  Finally, in step S2150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration apparatus 100 to end the calibration process, the process ends. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S2010 again, and the data Wait for input of get command.

  Through the above processing, the position of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 1-1>
In the present embodiment, the steepest descent method expressed by Equation 10 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. In this embodiment, in the processing from step S2070 to step S2130, an initial value is given to the position of the index to be obtained, and the optimum value is obtained by iterative calculation using the image Jacobian. It is also possible to obtain the position of. For example, if Equation 3 is expanded,

The relationship is obtained. Using this relationship,

Thus, s ^Pk can be obtained directly from the data list L ^Pk = [M _WCi , u _i ^Pk ] (i = 1,, I ^Pk ). here,

Represents.

<Modification 1-2>
Further, in the present embodiment, the position / orientation calculation unit 120 includes the image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn detected from the captured image of the imaging device 130 and the world of indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the correspondence with the coordinates x _W ^Qkn . However, the image features used to obtain the position and orientation of the imaging device need to be represented by dots. There is no. For example, a method for obtaining the position and orientation of an imaging apparatus using line features as shown in Non-Patent Document 2, and an imaging apparatus using geometric features such as an ellipse as shown in Non-Patent Document 5 The subjective viewpoint index detection unit 110 and the position / orientation calculation unit 120 may be realized by a method for obtaining the position and orientation of the camera, or may be a combination of these, or a camera using any other image feature A position and orientation estimation method may be used. In addition, a 6-DOF position and orientation sensor such as a magnetic sensor and a 3-DOF orientation sensor such as a gyro sensor are attached to the imaging device 130, and image characteristics detected from the captured image of the imaging device 130 and known information The position and orientation of the imaging device 130 may be calculated by a hybrid method based on both the correspondence relationship between the index held in advance and the world coordinates and the measurement value of the sensor attached to the imaging value 130. (For specific calculation methods, see, for example, Non-Patent Document 3 and Non-Patent Document 4).

[Second Embodiment]
In the index calibration apparatus according to the present embodiment, the index coordinates for the imaging apparatus when a plurality of indices mounted on the imaging apparatus have an index coordinate system and the coordinates of each index in the index coordinate system are known. The position and orientation of the system, or the position of each index with respect to the imaging device is acquired. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

  FIG. 3 is a diagram illustrating a schematic configuration of the index calibration apparatus according to the present embodiment. The same parts as those in FIG. 1 are given the same numbers and symbols, and detailed description thereof will be omitted. As shown in FIG. 3, the index calibration apparatus 300 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, and data management. The unit 370 and the calibration information calculation unit 380 are connected to the imaging device 130 that is a calibration target.

On the imaging device 130, a plurality of indices P ^k (k = 1,, K ₂ ) (first implementation) to be calibrated by a user who uses the index calibration apparatus according to this embodiment. In the following, it is assumed that, similarly to the form, this is referred to as an objective viewpoint index). As in the first embodiment, the position of the objective viewpoint index in the subjective viewpoint camera coordinate system is unknown, but in this embodiment, the relative positional relationship between the indices is known. For example, a case where a set of four objective viewpoint indices P ^k (k = 1, 2, 3, 4) forms one square index R ¹ having a known size is equivalent. FIG. 3 shows the situation. At this time, one point on the square indices R ¹ is defined as an origin, further triaxial respective X-axis orthogonal to each other, Y-axis and a coordinate system that is defined as Z-axis called an index coordinate system, index coordinate system The coordinates of each objective viewpoint index P ^k in are known. Therefore, if the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system (= conversion matrix for converting the coordinates in the index coordinate system to the subjective viewpoint camera coordinates) is found, the subjective viewpoint of each objective viewpoint index P ^k that is unknown is known. The position in the viewpoint camera coordinate system can be calculated. From this, it is assumed that the information calibrated by the index calibration device according to the present embodiment is the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system.

The plurality of positions in real space, as in the first embodiment, as an index for taking by the imaging apparatus 130, the position is the subjective-view-indices Q ^k is known is set in the world coordinate system.

Person view marker detecting unit 110, like the first embodiment, enter the subjective-view image capturing apparatus 130 is captured, detects the image coordinates of the subjective-view-indices Q ^k has been taken in an image, detected image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} to output the identifier _{k n} to the position and orientation calculation unit 120.

Similar to the first embodiment, the position / orientation calculation unit 120 includes the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn , and the world coordinates x _W ^{Qkn of the} indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the corresponding relationship. This calculation method is as described in the portion explaining the first embodiment. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 370 in accordance with a request from the data management unit 370.

  Similar to the first embodiment, the objective viewpoint camera 140 is fixedly disposed at a position where the image capturing apparatus 130 when capturing the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 380 as known values.

The objective viewpoint index detection unit 150 receives the objective viewpoint image captured by the objective viewpoint camera 140, as in the first embodiment, and is objectively captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the image coordinates of the viewpoint index ^{P k,} and outputs image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 370 and the identifier _{k m} to the data management unit 370. In the example of FIG. 3, since a square index is used as the index, the square index detection processing described in the first embodiment is executed.

  In the same way as in the first embodiment, the instruction unit 160 gives a “data acquisition” instruction to the data management unit 370 when a data acquisition command is input from an operator (not shown), and “ An instruction “calibration information calculation” is transmitted to the calibration information calculation unit 380.

  Upon receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 370 inputs the position and orientation in the world coordinate system of the imaging device 130 from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and a set of [position and orientation of the imaging device 130 -image coordinate of the objective viewpoint index -identifier of the objective viewpoint index] is added to one data list and held. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. In addition, the data management unit 370 outputs the generated data list to the calibration information calculation unit 380 in accordance with a request from the calibration information calculation unit 380.

  When the calibration information calculation unit 380 receives an instruction of “calibration information calculation” from the instruction unit 160, the calibration information calculation unit 380 inputs a data list from the data management unit 370, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system) is output.

  FIG. 4 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The same parts as those in FIG. 2 are given the same numbers and symbols, and detailed description thereof is omitted. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  From step S2010 to step S2030, processing similar to that of the first embodiment is performed. When step S2030 is completed, the process proceeds to step S4040.

Next, in step S4040, the data management unit 370 adds the input data set as data D _j to the data list L for all the detected objective viewpoint indices P ^km . Specifically, the _{M WC} input from the position and orientation calculation unit 120 as _{M WCJ,} the identifier _{k m} input from the objective-view-index detection unit 150 and _{k j,} the ^{u Pkm} similarly input from an objective-view-index detection unit 150 As u _j ^Pkj , a set of D _j = [M _WCj , u _j ^Pkj , k _j ] is registered in the data list L as the j-th data. Here, j (j = 1,..., J) is an index for each data set registered in the data list L, and J represents the total number of registered data sets.

  Data is acquired by the above processing.

In step S4050, the data management unit 370 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S2010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S2060. The condition that the data list has sufficient information to calculate the calibration information includes, for example, that the data list L includes data relating to at least three different objective viewpoint indices P ^k. . However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S4070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 380 handles the calibration information to be obtained, that is, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system as a 6-value vector [xyzξψζ] ^T. Hereinafter, this unknown parameter is described as a state vector s = [xyzξψζ] ^T.

In step S4070, the calibration information calculation unit 380 gives an appropriate initial value to the state vector s. As an initial value, an operator may manually input an approximate value via the instruction unit 160, or a plurality of objective viewpoint indices input at a certain same time (that is, detected from the same objective viewpoint image). Are extracted from the list L, and using this data, the position and orientation of the index coordinate system in the objective viewpoint coordinate system at the same time are calculated by a known method, and the obtained position and orientation are used in the index coordinate system. A transformation matrix M _BM for transforming the coordinates into the objective viewpoint camera coordinate system is obtained, and further, using the position and orientation M _WC of the imaging device input at the same time held in the list L, to obtain a conversion matrix M _CM representing the position and orientation of the index-based coordinates in person view camera coordinate system, may be using the position and orientation represented by this matrix as an initial value of s.

Here, _MWB is a transformation matrix that converts coordinates in the objective viewpoint camera coordinate system into world coordinates, and is based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that it has been calculated in advance.

  As a known method for calculating the position and orientation of the index coordinate system in the objective viewpoint coordinate system, for example, when the indices are arranged on the same plane, two-dimensional homology is used by using four or more indices. It is possible to use a technique for obtaining the position and orientation of the index coordinate system based on the calculation of the graphic. It is also possible to use a technique that uses six or more indices that are not on the same plane, or a technique that uses these solutions as initial values and obtains an optimal solution by iterative calculations such as the Gauss-Newton method.

In step S4080, the calibration information calculation unit 380 determines that each data D _j = [M _WCj , u _j ^Pkj , k _j ] (j = 1, 2, ^... J) in the data list L and the state vector s are The theoretical value u _j ^Pkj ′ = [u _xj ^Pkj ′, u _yj ^Pkj ′] of the objective viewpoint image coordinates of each objective viewpoint index P ^{kj is} calculated for all j. Here, the theoretical value of the objective viewpoint image coordinate of the objective viewpoint index ^means that the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system of the objective viewpoint index P ^kj whose position vector x _M ^Pkj in the index coordinate system is known is s. Is the data of the position (coordinates) that should be visible in the objective viewpoint image. The calculation of u _j ^Pkj ′ is a function of the state vector s representing the position of the index coordinate system in the subjective viewpoint camera coordinate system.

Based on.

Specifically, the function F _j () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^kj when the j-th data is acquired (that is, when the position and orientation of the imaging device 130 are M _WCj ). A position vector x _Bj ^Pkj is obtained from s as follows:

And the following equation for ^obtaining the coordinates u _j ^Pkj ′ on the objective viewpoint image of the objective viewpoint index P ^kj from x _Bj ^Pkj :

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. M _CM (s) is a modeling conversion matrix (matrix for converting coordinates in the index coordinate system to coordinates in the subjective viewpoint camera coordinate system) determined by s, and is defined by the following equation.

However,

It is.

In step S4090, the calibration information calculation unit 380 calculates the actual image coordinates u _j ^{Pkj of the} objective viewpoint index P ^kj included in each data in the data list L and the corresponding image coordinate theory for all j. error △ _u ^{j PKJ} the value _u ^{j PKJ} 'is calculated by the following equation.

In step S4100, the calibration information calculation unit 380 performs, for each _j, a solution obtained by partially differentiating the image Jacobian relating to the state vector s (that is, the function F _j () of Expression 17 by each element of the state vector s). 2 rows × 6 columns Jacobian matrix) J _uj ^Pkj _s (= ∂u _j ^Pkj / ∂s). Specifically, a 2-row × 3-column Jacobian matrix J _ujxBj ^Pkj (= 持_つ ^u) having a solution obtained by partial differentiation of the right side of Expression 19 with each element of the position vector x _Bj ^Pkj on the objective viewpoint camera coordinate system. _j ^Pkj / ∂x _Bj ^Pkj ) and a 3 × 6 Jacobian matrix J _xBj ^Pkj _s (= ∂x _Bj ^Pkj /) having a solution obtained by partial differentiation of the right side of Equation 18 with each element of the state vector s. ∂s) is calculated, and J _uj ^Pkj _s is calculated by the following equation.

In step S4110, the calibration information calculation unit 380 calculates the correction value ^{Δs of} _s based on the error ^Δu _j ^Pkj and the Jacobian matrix J _uj ^Pkj _s for all j calculated in the above steps. Specifically, an error vector of a 2J-dimensional vector in which errors ^Δu _j ^Pkj for all j are arranged vertically

And Jacobi matrix J _uj ^Pkj _s vertically arranged in 2J rows × 6 columns matrix

And Δs is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs is a six-dimensional vector, Δs can be obtained when 2J is 6 or more, that is, J is 3 or more. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

  In step S4120, the calibration information calculation unit 380 uses the correction value Δs calculated in step S4110 to correct the state vector s representing the position and orientation of the index system coordinate in the subjective viewpoint camera coordinate system according to Equation 27, and obtain Let the obtained value be a new s.

  In step S4130, calibration information calculation section 380 converges the calculation using some criterion such as whether error vector U is smaller than a predetermined threshold or whether correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S4080 is performed again using the corrected state vector s.

If it is determined in step S4130 that the calculation has converged, in step S4140, the calibration information calculation unit 380 uses the obtained state vector s as calibration information, that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Output as. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. then, it may be a coordinate transformation matrix M _CM generated from s.

  Finally, in step S2150, it is determined whether or not to end the calibration process. If the operator instructs the index calibration device 300 to end the calibration process, the process is terminated. If the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S2010 again to acquire data. Wait for command input.

  Through the above processing, the position or position / orientation of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 2-1>
In this embodiment, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system are output as the calibration information, but the objective viewpoint index P in the subjective viewpoint camera coordinate system is obtained from the obtained state vector s. The positions of ^k may be calculated individually and output as calibration information. In this case, the position of the objective viewpoint index P ^k in the subjective viewpoint camera coordinate system is obtained by the product of the coordinate transformation matrix M _CM obtained from s and the known position x _M ^Pk of the objective viewpoint index P ^k in the index coordinate system. be able to.

<Modification 2-2>
Further, in the present embodiment, the position / orientation calculation unit 120 includes the image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn detected from the captured image of the imaging device 130 and the world of indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the correspondence relationship with the coordinate x _W ^Qkn , but the position and orientation using other image information is the same as <Modification 1-2> of the first embodiment. Calculation and position / orientation calculation using other sensors may also be performed.

<Modification 2-3>
In the present embodiment, the steepest descent method expressed by Expression 26 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. Further, in the present embodiment, a method for obtaining an optimal value of s for all input data by giving appropriate initial values to the position and orientation s of the index coordinate system in the subjective viewpoint camera coordinate system, and by iterative calculation using the image Jacobian. However, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system can also be obtained by using another simple calculation method. For example, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system are determined by using only the detection coordinates of a plurality of objective viewpoint indices on one objective viewpoint image by the processing procedure described in the description of step S4070. It may be obtained and output as calibration information.

<Modification 2-4>
In the present embodiment, a square index is used as the objective viewpoint index, but any index type may be used as long as the index group has a known position of each index in the index coordinate system. For example, it may be a set of a plurality of circular indicators as used in the first embodiment, or a plurality of types of indicators may be mixed. Further, even when a plurality of index coordinate systems are provided, the above processing is performed individually for each index coordinate system, or the above processing is performed in parallel for each index coordinate system. Can be calibrated as well.

[Third Embodiment]
In the index calibration device according to the present embodiment, as in the second embodiment, a plurality of indexes mounted on the imaging apparatus have an index coordinate system, and the coordinates of each index in the index coordinate system are known. In this case, the position and orientation of the index coordinate system with respect to the imaging apparatus or the position of each index with respect to the imaging apparatus is acquired. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

FIG. 5 is a diagram showing a schematic configuration of the index calibration device according to the present embodiment. The same parts as those in FIG. 3 are given the same numbers and symbols, and detailed description thereof is omitted. As shown in FIG. 5, the index calibration apparatus 500 according to the present embodiment includes a subjective viewpoint index detection unit 110, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, a position and orientation calculation unit 520, and data management. 570 and a calibration information calculation unit 580, which are connected to the imaging device 130 to be calibrated. It is assumed that a plurality of indices P ^k (k = 1,, K ₂ ) to be calibrated are set on the imaging device 130 as in the second embodiment.
The subjective viewpoint index Q ^k is set at a plurality of positions in the real space, as in the second embodiment, and the subjective viewpoint index detection unit 110, the objective viewpoint camera 140, the objective viewpoint index detection unit 150, Since the operation of the instruction unit 160 is the same as that of the second embodiment, detailed description thereof is omitted. However, the subjective viewpoint index detection unit 110 outputs the detection result to the data management unit 570 in response to a request from the data management unit 570, and the objective viewpoint index detection unit 150 outputs the detection result to the position / orientation calculation unit 520. However, this is different from the second embodiment. In the second embodiment, since the position and orientation of the imaging device 130 are obtained using the detection result of the subjective viewpoint index, the number of indexes necessary for calculating the position and orientation of the imaging device 130 is 1. Although it has been necessary to detect from one subjective viewpoint image, in the present embodiment, the number of indices detected from one subjective viewpoint image may be one or more.

The position / orientation calculation unit 520 receives the image coordinates ^uPkm of the objective viewpoint index P ^km and its identifier from the objective viewpoint index detection unit 150, and the three-dimensional coordinates of each index in the index coordinate system held in advance as known information. Based on the correspondence with x _M ^Pkm , the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system are calculated. Since this calculation method is the same as the processing described in step S4070 of the second embodiment, detailed description thereof is omitted. The calculated position and orientation are output to the data management unit 370 in accordance with a request from the data management unit 370. In the following, the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system are held by a 4 × 4 homogeneous coordinate matrix M _BM (matrix that converts coordinates in the index coordinate system into coordinates in the objective viewpoint camera coordinate system). It shall be.

  Upon receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 570 inputs the image coordinates of the subjective viewpoint index and its identifier from the subjective viewpoint index detection unit 110, and the objective viewpoint camera coordinates from the position / orientation calculation unit 520. The position and orientation of the index coordinate system in the system is input, and the pair of [position and orientation of the index coordinate system in the objective viewpoint camera coordinate system−image coordinates of the subjective viewpoint index−identifier of the subjective viewpoint index] is input. Created for each viewpoint index, added to one data list and retained. Further, the data management unit 570 outputs the generated data list to the calibration information calculation unit 580 in accordance with a request from the calibration information calculation unit 580.

  When the calibration information calculation unit 580 receives an instruction of “calibration information calculation” from the instruction unit 160, the calibration information calculation unit 580 inputs a data list from the data management unit 570, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system) is output.

  FIG. 6 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S6010, the instruction unit 160 determines whether a data acquisition command has been input from the operator. The operator inputs a data acquisition command when the imaging device 130 is placed at a position where data for index calibration is acquired. If the data acquisition command is input, the instruction unit 160 shifts the process to step S6020.

In step S6020, the data management unit 570, from the position and orientation calculation unit 520 inputs the position and orientation _{M BM} index coordinate system in the objective view camera coordinate system.

In step S6030, the data management unit 570, from the subjective-view-index detection unit 110, and inputs the image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} of people each detected by the subjective-view-index detection unit 110 and the identifier _{k n.}

In step S6040, the data management unit 570 adds each input subjective viewpoint index Q ^kn to the data list L as data D _j . Specifically, the objective viewpoint camera coordinate position and orientation M _BM index coordinate system in system input from the position and orientation calculating section 520 and M _BMj, the identifier k _n input from the subjective-view-index detection unit 110 and k _j, Similarly, a group of D _j = [M _BMj , u _j ^Qkj , k _j ] is registered in the data list L as j-th data, where u ^Qkn inputted from the subjective viewpoint index detection unit 110 is u _j ^Qkj . Here, j (j = 1,..., J) is an index for each data set registered in the data list L, and J represents the total number of registered data sets.

  Data is acquired by the above processing.

In step S6050, the data management unit 570 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S6010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the condition for calculating the calibration information, the process proceeds to step S6060. The condition that the data list has sufficient information to calculate the calibration information includes, for example, that the data list L includes data relating to at least three different subjective viewpoint indices Q ^k. . However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S6060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S6070. If not, the process returns to step S6010 again to wait for the input of the data acquisition command.

As in the second embodiment, the calibration information calculation unit 580 handles the calibration information to be obtained, that is, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system as a six-value vector [xyzξψζ] ^T. Hereinafter, this unknown parameter is described as a state vector s = [xyzξψζ] ^T.

  In step S6070, the calibration information calculation unit 580 gives an appropriate initial value to the state vector s. As the initial value, for example, the operator manually inputs an approximate value via the instruction unit 160.

In step S6080, the calibration information calculation unit 580 determines each subjective viewpoint index Q for all j from the data D _j (j = 1, 2,..., J) and the state vector s in the data list L. The theoretical value u _j ^Qkj ′ = [u _xj ^Qkj ′, u _yj ^Qkj ′] of the subjective viewpoint image coordinates of ^kj is calculated. Here, the theoretical values of the subjective viewpoint image coordinates of the subjective viewpoint index are M _{BMj for} the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system, and s for the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Sometimes, the position x _W ^Qkj in the world coordinate system refers to the data of the position (coordinates) that should be visible in the subjective viewpoint image of the subjective viewpoint index Q ^kj . The calculation of u _j ^Qkj 'is a function of s

Based on.

Specifically, the function F _j () is the subjective viewpoint index Q ^kj when the j-th data is acquired (that is, when the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system is M _BMj ). The position x _Cj ^Qkj in the subjective viewpoint camera coordinates of

And the following equation for ^obtaining the coordinates u _j ^Qkj ′ on the subjective viewpoint image of the subjective viewpoint index Q ^kj from x _Cj ^Qkj :

It is constituted by. Here, f ^C _x and f ^C _y are the focal lengths of the imaging device 130 in the x-axis direction and the y-axis direction, respectively, and M _WB is a transformation matrix that converts coordinates in the objective viewpoint camera coordinate system into world coordinates, respectively. It is assumed that this value is held in advance. M _CM (s) is a modeling conversion matrix (matrix for converting coordinates in the index coordinate system to coordinates in the subjective viewpoint camera coordinate system) determined by s, and is defined by Expression 20. Note that the product (M _CM (s) · M _BMj ⁻¹ · M _WB ⁻¹ ) of the right-hand side matrix in Expression 29 is the position and orientation (M _WC ⁻ ) of the imaging device 130 in the world coordinate system determined by s and M _BMj . ¹ ).

In step S6090, the calibration information calculation unit 580 calculates the actual image coordinates u _j ^{Qkj of the} subjective viewpoint index Q ^kj included in each data in the data list L and the corresponding image coordinate theory for all j. error △ _u ^{j Qkj} the value _u ^{j Qkj} 'is calculated by the following equation.

In step S6100, the calibration information calculation unit 580 performs, for each j, an element obtained by partially differentiating the image Jacobian relating to the state vector s (that is, the function F _j () of Equation 28 by each element of the state vector s. (2 rows × 6 columns Jacobian matrix) J _uj ^Qkj _s (= ∂u _j ^Qkj / ∂s). Specifically, a 2-row × 3-column Jacobian matrix J _ujxCj ^Qkj (= ∂u) having a solution obtained by partial differentiation of the right side of Expression 30 with each element of the position vector x _Cj ^Qkj on the subjective viewpoint camera coordinate system. _j ^Qkj / ∂x _Cj ^Qkj ) and a 3 × 6 Jacobian matrix J _xCj ^Qkj _s (= ∂x _Cj ^Qkj /) each having a solution obtained by partial differentiation of the right side of 29 with each element of the state vector s s) and J _uj ^Qkj _s is calculated by the following equation.

In step S6110, the calibration information calculation unit 580 calculates the correction value ^{Δs of} _s based on the error ^Δu _j ^Qkj and the Jacobian matrix J _uj ^Qkj _s for all j calculated in the above steps. Specifically, an error vector U of 2J-dimensional vectors in which errors ^Δu _j ^Qkj for all j are arranged vertically and a Jacobian matrix J _uj ^Qkj _s are arranged in a matrix of 2J rows × 6 columns, Δs is calculated from Equation 26 using the pseudo inverse matrix Φ ⁺ of Φ. Here, since Δs is a six-dimensional vector, Δs can be obtained when 2J is 6 or more, that is, J is 3 or more. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

  In step S6120, calibration information calculation section 580 corrects state vector s according to equation 27 using correction value Δs calculated in step S6110, and sets the obtained value as new s.

  In step S6130, calibration information calculation section 580 converges the calculation using some criterion such as whether error vector U is smaller than a predetermined threshold or whether correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S6080 is performed again using the corrected state vector s.

If it is determined in step S6130 that the calculation has converged, in step S6140, the calibration information calculation unit 580 uses the obtained state vector s as calibration information, that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Output as. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. then, it may be a coordinate transformation matrix M _CM generated from s.

  Finally, in step S6150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration device 500 to end the calibration process, the process is ended. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S6010 again to acquire data. Wait for command input.

  Through the above processing, the position or position / orientation of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 3-1>
In this embodiment, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system are output as the calibration information, but the objective viewpoint index P in the subjective viewpoint camera coordinate system is obtained from the obtained state vector s. The positions of ^k may be calculated individually and output as calibration information.

<Modification 3-2>
In the present embodiment, the steepest descent method expressed by Expression 26 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Regardless of which numerical calculation method is applied, the essence of the present invention is not impaired.

<Modification 3-3>
In the present embodiment, a square index is used as the objective viewpoint index, but any index type may be used as long as the index group has a known position of each index in the index coordinate system. For example, it may be a set of a plurality of circular indicators as used in the first embodiment, or a plurality of types of indicators may be mixed. Further, even when a plurality of index coordinate systems are provided, the above processing is performed individually for each index coordinate system, or the above processing is performed in parallel for each index coordinate system. Can be calibrated as well.

<Modification 3-4>
In this embodiment, an index that gives a two-dimensional coordinate on an image is used as a subjective viewpoint index. However, line features and geometric features as disclosed in Non-Patent Document 2 and Non-Patent Document 5 are used. May be used as a reference index for evaluation. For example, when a line feature is used, the error vector U is constituted by an error Δd calculated from the detected value d from the image and the estimated value d ′ from the state vector s using the distance from the origin of the straight line as an error evaluation criterion. Then, the matrix Φ is constituted by a 1 row × 4 column Jacobian matrix J _ds (= ∂d / ∂s) having a solution obtained by partial differentiation of the calculation formula of d ′ with each element of the state vector s. The correction value can be calculated by the same framework as in the above embodiment. Note that the calculation formula of d ′ is disclosed as a function of the position and orientation of the imaging apparatus in Non-Patent Document 2, Non-Patent Document 5, and the like, and the position and orientation of the imaging apparatus is obtained as a function of s (Formula 29) Therefore, it can be easily obtained from these. In addition, it is possible to use these features together by stacking line features and point features, errors obtained from other indices, and image Jacobians.

[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above (shown in FIGS. 2, 4, and / or 6).

It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 1st Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 1st Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 2nd Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 3rd Embodiment.

Claims (16)

An index calibration device that calculates, as calibration information, arrangement information of a calibration target index mounted on an imaging device with respect to the imaging device,
Objective viewpoint imaging means for imaging the imaging device from an objective viewpoint position;
First image input means for inputting a first image from the imaging device;
Second image input means for inputting a second image from the objective viewpoint imaging means;
First detection means for detecting information on the image coordinates of the reference index arranged in the scene from the first image;
Second detection means for detecting information on the image coordinates of the calibration target index from the second image;
An index calibration apparatus comprising calibration information calculation means for calculating the calibration information using information on image coordinates of each index detected by the first detection means and the second detection means.   The index calibration apparatus according to claim 1, wherein the objective viewpoint imaging unit is fixed in the scene. The calibration information calculation means includes
The image pickup apparatus position and orientation calculation means for calculating the position and orientation of the image pickup apparatus using information on the image coordinates of the reference index detected by the first detection means,
The calibration information is calculated based on the position / orientation of the imaging device calculated by the imaging device position / orientation calculation unit and information on the image coordinates of the calibration target index detected by the second detection unit. The index calibration apparatus according to claim 1 or 2. The calibration information calculation means includes
In the second image of the calibration target index detected by the second detection means based on the position and orientation of the imaging apparatus calculated by the imaging apparatus position and orientation calculation means and the estimated value of the calibration information Estimating means for estimating information about image coordinates in
In order to reduce an error between the information about the image coordinates of the calibration target index detected by the second detection unit and the information about the image coordinates of the calibration target index estimated by the estimation unit, the calibration information The index calibration apparatus according to claim 3, further comprising a correction unit that corrects the estimated value. The calibration information is a position and orientation of the calibration target index attached to the imaging apparatus with respect to the imaging apparatus,
The calibration information calculation means includes
There is further provided an imaging device position / orientation calculating unit that calculates the position / orientation of the imaging device based on the information on the image coordinates of the calibration target index detected by the second detection unit and the estimated value of the current calibration information. And
The calibration information is calculated based on the position / orientation of the imaging apparatus calculated by the imaging apparatus position / orientation calculation means and information on the image coordinates of the reference index detected by the first detection means. The index calibration apparatus according to claim 1 or 2. The calibration information is a position and orientation of the calibration target index attached to the imaging apparatus with respect to the imaging apparatus,
The calibration information calculation means includes
The first of the reference indices detected by the first detection means based on the information about the image coordinates of the calibration target index detected by the second detection means and the estimated value of the current calibration information. Estimating means for estimating information related to image coordinates in the image of
The estimated value of the calibration information so as to reduce an error between the information about the image coordinates of the reference index detected by the first detection unit and the information about the image coordinates of the reference index estimated by the estimation unit. The index calibration apparatus according to claim 1, further comprising a correction unit that corrects the error.   The index calibration apparatus according to claim 1, wherein the information related to the image coordinates of the calibration target index is one or a plurality of image coordinates determined by the calibration target index. An index calibration method for obtaining, as calibration information, arrangement information of a calibration target index mounted on an imaging apparatus with respect to an imaging apparatus,
Obtaining a first image taken by the imaging device;
Obtaining a second image obtained by photographing the imaging device from an objective viewpoint position;
Detecting information about the image coordinates of the reference index arranged in the scene from the first image,
Detecting information about the image coordinates of the calibration target index from the second image;
An information processing method characterized in that the calibration information is obtained by using information on image coordinates of a reference index arranged in the detected scene and information on image coordinates of the calibration target index.   The information processing method according to claim 8, wherein the objective viewpoint position is a fixed position in the scene. Using the information about the image coordinates of the reference index, calculate the position and orientation of the imaging device,
The information processing method according to claim 8 or 9, wherein the calibration information is calculated based on information on a position and orientation of the imaging apparatus and image coordinates of the calibration target index. Based on the position and orientation of the imaging device and the estimated value of the calibration information, estimate information related to image coordinates in the second image of the calibration target index,
Correcting the estimated value of the calibration information so as to reduce an error between the information about the image coordinates of the calibration target index and the information about the image coordinates of the calibration target index estimated by the estimation unit. The information processing method according to claim 10. The calibration information is a position and orientation of the calibration target index attached to the imaging apparatus with respect to the imaging apparatus,
Based on the information about the image coordinates of the calibration target index and the estimated value of the current calibration information, calculate the position and orientation of the imaging device,
10. The information processing method according to claim 8, wherein the calibration information is calculated based on a position and orientation of the imaging apparatus and information on image coordinates of the reference index. The calibration information is a position and orientation of the calibration target index attached to the imaging apparatus with respect to the imaging apparatus,
Based on the information on the image coordinates of the calibration target index and the estimated value of the current calibration information, the information on the image coordinates in the first image of the reference index is estimated,
The correction value of the calibration information is corrected so as to reduce an error between the information about the image coordinates of the reference index and the information about the image coordinates of the reference index estimated by the estimation unit. The information processing method according to claim 8 or 9.   The information processing method according to claim 8, wherein the information regarding the image coordinates of the calibration target index is one or a plurality of image coordinates determined by the calibration target index.   The program code for making a computer perform the information processing method in any one of Claims 8 thru | or 14. A storage medium for storing the program code according to claim 15.

Images