JP5693691B2 - Information processing apparatus, processing method thereof, and program - Google Patents

Description

  The present invention relates to an information processing apparatus that calculates information on the position and orientation of an imaging apparatus with respect to an object imaged by the imaging apparatus, a processing method thereof, and a program.

  In recent years, research on AR (Augmented Reality) technology that superimposes and displays information in a virtual space on a real space has been actively conducted. For example, a video see-through type HMD (Head Mounted Display) is known as a representative example of an information presentation apparatus adopting the AR technology. A video see-through HMD has a built-in camera that captures a real space. In the video see-through HMD, a virtual object is drawn by CG (Computer Graphics) according to the position and orientation of the camera or the like. Then, a composite image obtained by superimposing the drawn virtual object on the image in the real space is displayed on an HMD display device such as a liquid crystal panel. Thereby, the user can feel as if the virtual object actually exists in the real space.

  One of the major problems that must be solved in realizing the AR technology is the “alignment” problem. The alignment in AR is to take geometrical consistency between the virtual object and the real space. In order for the user to feel that the virtual object actually exists in the real space, the registration must be performed correctly, and the virtual object must be presented to the user so that the virtual object always exists in the real space. Don't be.

  In an AR using a video see-through HMD, the position and orientation at the time of image capturing of a camera in a real space are generally measured every time an image is input from a camera built in the HMD. Then, a CG is drawn based on the camera position and orientation and camera-specific parameters such as the focal length, and is superimposed on the real space image. Therefore, in the alignment in AR, it is necessary to accurately measure the position and orientation of the camera built in the HMD. In general, the position and orientation of a camera are measured using a physical sensor such as a magnetic sensor, an ultrasonic sensor, or an optical sensor that can measure the position and orientation of the camera with six degrees of freedom.

  In the video see-through HMD, image information from a built-in camera can be used for alignment. When alignment is performed using image information, it is easier and less expensive than methods using physical sensors. In this alignment method, generally, an index with a known three-dimensional position in a real space is imaged by a camera, and the position and orientation of the camera are determined based on the correspondence between the index position on the captured image and the three-dimensional position. calculate. For the index, for example, markers that are artificially arranged in the real space and natural features such as corner points and edges that originally exist in the real space are used. In practice, artificial markers that are easy to detect and identify from image information are widely used from the viewpoint of stability and computational load.

  In relation to such a technique, Non-Patent Document 1 discloses a method of performing alignment using a square marker in which a unique two-dimensional pattern is drawn as an index. Artificial markers such as square markers are widely used because they can be easily used. However, the marker cannot be used when it is physically impossible or difficult to arrange the marker, or when it is not desired to arrange the marker for reasons such as deteriorating aesthetics.

On the other hand, along with the recent improvement in calculation capability, research on techniques for performing registration using natural features that originally exist in real scenes has been actively conducted. Examples of natural features used for alignment include point shape features such as corner points (hereinafter point features) and line features such as edges. Non-Patent Documents 2, 3, and 4 disclose alignment methods using edges. Since the edge is invariant with respect to the scale and the observation direction, the alignment using the edge is characterized by high accuracy. The alignment using the edge is premised on having a three-dimensional model data of a real space or a real object described by a set of line segments. The alignment using the edges disclosed in Non-Patent Documents 2, 3, and 4 is realized by the following processes 1 to 4.
(1). The aforementioned three-dimensional model data (line segment model) is projected on the image based on the camera position and orientation in the previous frame and the intrinsic parameters of the camera that have been calibrated in advance.
(2). Each projected line segment is divided at a constant interval on the image, and division points are set. Then, for each division point, an edge search is performed on a line segment (search line) that is a normal direction of the line segment that passes through the division point and the direction is projected, and the gradient of the luminance value on the search line is maximal and A point closest to the dividing point is detected as a corresponding edge.
(3). Calculate a correction value for the position and orientation of the camera that minimizes the sum of the distances on the image between the corresponding edge detected at each division point and the projected line segment. Based on this, the position and orientation of the camera are calculated.
(4). The optimization calculation is repeated until the processing of (3) is converged.

  Unlike point features, edges are less distinguishable on images. At the time of edge search, since only the information that the gradient of the luminance value on the search line is maximum is used, an erroneous edge is often detected. For this reason, in Non-Patent Documents 2, 3, and 4, a technique called M estimation is used to reduce the weight of edge data that seems to be erroneously detected so that the erroneously detected edge does not adversely affect the optimization calculation. To perform optimization calculations.

  Further, as described above, in the alignment using the edge, the three-dimensional model data of the line segment constituting the real space or the real object to be aligned is required. Conventionally, measurement of three-dimensional model data of a line segment has been performed manually or using an image. For manual measurement, a measure, a ruler, a protractor, etc. are used. In addition, photogrammetry software that takes images of scenes and objects to be measured and calculates 3D data based on the result of manual measurement by the measurer on the image is also used. Yes. The measurer searches the real space / real object for a line segment that is likely to be detected as an edge when performing alignment, such as a line between planes or a line whose brightness changes greatly on both sides of the line. And measurement using software. In addition to this, as shown in Non-Patent Document 5, a method for measuring three-dimensional data of a line segment using an image is also known. In Non-Patent Document 5, the direction of a straight line and the passing position in a three-dimensional space are estimated based on the correspondence of line segments on a plurality of images. In Non-Patent Document 5, the two-dimensional edge detection method shown in Non-Patent Document 6 is used as an edge detection method on an actual image.

Kato, M. Billinghurst, Asano, Tachibana, “Augmented reality system based on marker tracking and its calibration”, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-617, 1999. T. Drummond and R. Cipolla, “Real-time visual tracking of complex structures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.7, pp.932-946, 2002. A. I. Comport, E. Marchand, and F. Chaumette, “A real-time tracker for markerless augmented reality,” Proc. The 2nd IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR03), pp.36-45, 2003. L. Vacchetti, V. Lepetit, and P. Fua, “Combining edge and texture information for real-time accurate 3D camera tracking,” Proc. The 3rd IEEE / ACM International Symposium on Mixed and Augmented Reality (ISMAR04), pp.48 -57, 2004. C. J. Taylor and D. J. Kriegman, “Structure from motion from line segments in multiple images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.17, no.11, pp.1021-1032, 1995. J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.8, no.6, pp.679-698, 1986. RY Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV camera and lenses,” IEEE Journal of Robotics and Automation, vol.3, no.4, pp.323-344, 1987. H. Wuest, F. Vial, and D. Stricker, “Adaptive line tracking with multiple hypotheses for augmented reality,” Proc. The Fourth Int'l Symp. On Mixed and Augmented Reality (ISMAR05), pp.62-69, 2005 .

  In the alignment using the edge described above, three-dimensional model data is projected, and the edge is detected by a one-dimensional search based on the projection image. For this reason, since the 3D model data does not exist at the time of measuring the 3D model data, it is not possible to perform edge detection by a 1D search based on the projection of the 3D model data. That is, it is not known in advance whether an edge estimated by the measurer or an edge detected by a two-dimensional edge detection method is detected when performing alignment. This is not limited to the above-described edge detection method for detecting the extreme value of the density gradient, and other methods for performing a one-dimensional search by projecting three-dimensional model data (for example, using image information around edges). This is also a common problem in methods such as a one-dimensional search for edges.

  For this reason, when measuring manually, it is necessary for the measurer to determine whether or not the edge is used for alignment with his / her eyes, which requires skill in the work. Furthermore, even a skilled measurer may make a mistake in determination, so after measuring the three-dimensional model data, it is necessary to actually align and repeat the determination of which line segment is unnecessary. Therefore, it takes time to measure.

  In addition, in the conventional alignment using edges, even if information on line segments that are not actually detected as edges is included in the three-dimensional model, an attempt is made to detect and associate edges with respect to those line segments. . For this reason, misdetection of an edge on an image and miscorrespondence with a line segment of a model occur, so that the accuracy and stability of alignment are lowered.

  Therefore, the present invention has been made in view of the above problems, and reduces the calculation load using the three-dimensional information during the position / orientation calculation process, and suppresses a decrease in alignment accuracy and stability. The purpose is to provide the technology.

In order to achieve the above object, one embodiment of the present invention is an information processing apparatus that calculates position and orientation information of an imaging device with respect to an object imaged by the imaging device, and holds three-dimensional model data of the object Holding means, acquiring means for acquiring a plurality of images of the object imaged by the imaging device, and projecting the geometric features of the three-dimensional model data to each of the plurality of images, and corresponding to the projected geometric features a detecting means for geometric features detected from each of the plurality of images, detected by each of the plurality of images, based on the detection frequency of the corresponding geometric features geometric features included in the three-dimensional model data, the 3 take advantage of geometric features of the dimensional model data determining means for determining to use the position and orientation calculation process of the imaging device, the position and orientation calculation process by said determining means Based on the determined the three-dimensional model data of the geometric features and comprises an updating means for updating the 3-dimensional model data.

  ADVANTAGE OF THE INVENTION According to this invention, while reducing the calculation load using the three-dimensional information at the time of a position and orientation calculation process, the fall of the alignment precision and stability can be suppressed.

It is a figure which shows an example of the outline | summary of the video see-through type HMD incorporating the information processing apparatus concerning one embodiment of this invention. It is a figure which shows an example of a structure of the information processing apparatus 1 shown in FIG. An example of a three-dimensional model definition method according to the first embodiment will be described. 3 is a flowchart illustrating an example of a process flow in the information processing apparatus 1 illustrated in FIG. 1. It is a flowchart which shows an example of the flow of the edge detection process in FIG.4 S1030. It is a figure which shows an example which projected the three-dimensional model on the image. It is a figure which shows an example of the division | segmentation point on the image of a three-dimensional model. It is a figure which shows an example of the outline | summary of the edge detection method concerning Embodiment 1. FIG. FIG. 10 is a diagram illustrating an example of an outline of processing for calculating the position and orientation of the imaging apparatus 100 using edge information. It is a flowchart which shows an example of the flow of the model data verification process in step S1090 of FIG. 10 is a flowchart illustrating an example of a processing flow in the information processing apparatus 1 according to the second embodiment. It is a flowchart which shows an example of the flow of the model data verification process in step S2090 of FIG. 6 is a diagram illustrating an example of a configuration of an information processing apparatus 1 according to Embodiment 3. FIG. 14 is a flowchart illustrating an example of a processing flow in the information processing apparatus 3 according to the third embodiment.

  Hereinafter, an information processing apparatus, a processing method thereof, and an embodiment of a program according to the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a diagram showing an example of an outline of a video see-through type HMD incorporating an information processing apparatus according to an embodiment of the present invention. In the present embodiment, a case where the information processing apparatus 1 is used for AR (Augmented Reality) alignment will be described as an example. However, the information processing apparatus 1 is of course used for other purposes. May be used.

  The observer 40 is wearing a video see-through type HMD 30. The observer 40 observes the observation target object 10 through the video see-through HMD 30. As a result, the viewer 40 displays an image in which the virtual object 20 is superimposed on the observation target object 10. In this case, for example, an annotation for the observation target object 10 is displayed as the virtual object 20.

  The video see-through HMD 30 incorporates imaging devices (100, 101) corresponding to the left eye and the right eye, respectively. The imaging device 100 images the observation target object 10. An image captured by the imaging apparatus 100 is input to the information processing apparatus 1. The information processing apparatus 1 calculates the position and orientation of the imaging device (100, 101) based on the image captured by the imaging device (100, 101) and the three-dimensional model data of the observation target object 10. Note that camera-specific parameters such as the focal length of the imaging device (100, 101) may be obtained in advance by a known camera calibration method (for example, Non-Patent Document 7).

  The video see-through HMD 30 draws an image of the virtual object 20 based on the position and orientation of the imaging device (100, 101) calculated by the information processing device 1, and the image is captured by each of the imaging devices (100, 101). Superimpose on real space image. Thereby, the synthesized image is displayed on the display device (not shown) of the HMD 30.

  The above is the description of an example of the outline of the video see-through type HMD 30. The information processing apparatus 1 described above incorporates a computer. The computer includes main control means such as a CPU, and storage means such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). In addition, the computer may further include input / output means such as buttons, a display, or a touch panel, and communication means such as a network card. These components are connected by a bus or the like, and are controlled by the main control unit executing a program stored in the storage unit.

  In FIG. 1, the information processing apparatus 1 is incorporated in the video see-through HMD 30, but the information processing apparatus 1 may be realized as a separate apparatus outside the video see-through HMD 30.

  Further, in the present embodiment, in order to simplify the description, a case where verification of 3D model data is performed using an image captured by the imaging apparatus 101 will be described as an example. Of course, this processing may be performed using images captured by both the imaging device 100 and the imaging device 101.

  FIG. 2 is a diagram illustrating an example of the configuration of the information processing apparatus 1 illustrated in FIG.

  The information processing apparatus 1 includes an image acquisition unit 110, an image feature detection unit 120, a model data storage unit 130, a position / orientation calculation unit 140, and a model data verification unit 150.

  The image acquisition unit 110 inputs an image captured by the imaging apparatus 100 to the information processing apparatus 1. If the output of the imaging device 100 is an analog output such as NTSC, the image acquisition unit 110 is realized by an analog video capture board, for example. Further, if the output of the imaging apparatus is a digital output such as IEEE 1394, it is realized by, for example, an IEEE 1394 interface board. Note that the image acquisition by the image acquisition unit 110 may be performed by reading digital data of a still image or a moving image stored in advance in the storage device.

  The image feature detection unit 120 detects an image feature representing a line segment in the three-dimensional space from the image input by the image acquisition unit 110. The image feature representing a line segment in the three-dimensional space is, for example, a point taking an extreme value of a density gradient, that is, an edge. The image feature detection unit 120 performs edge detection based on model data composed of three-dimensional information stored in a model data storage unit 130 described later, and predicted values of the position and orientation of the imaging device 100.

  The model data storage unit 130 stores three-dimensional model data (hereinafter, may be abbreviated as a three-dimensional model) of the observation target object 10 serving as a reference object for alignment. Here, an example of a method for defining a three-dimensional model according to the present embodiment will be described with reference to FIG. A three-dimensional model is defined by a set of points, information on a surface formed by connecting each point, and information on a line segment constituting the surface. As shown to the left figure of Fig.3 (a), the three-dimensional model concerning this embodiment is a rectangular parallelepiped comprised from 8 points | pieces of the points P1-P8. Here, the X axis of the rectangular parallelepiped coordinate system is taken in the direction from the point P1 to the point P4, the Y axis of the rectangular parallelepiped coordinate system is taken in the direction from the point P5 to the point P1, and the Z axis of the rectangular parallelepiped coordinate system. Take in the direction from the point P1 to the point P2, and further take the origin at P5.

  As shown in the middle figure and the right figure of Fig.3 (a), the rectangular parallelepiped is comprised by the surfaces F1-F6. Similarly, the rectangular parallelepiped is composed of line segments L1 to L12. Furthermore, the three-dimensional model according to the present embodiment has a line segment L13 on the surface F1 and a line segment L14 on the surface F2. The line segment L13 connects the point P9 and the point P10, and the line segment L14 connects the point P10 and the point P11.

As shown in FIG. 3B, the points P1 to P11 are represented by three-dimensional coordinate values. Moreover, as shown in FIG.3 (c), surface F1-F6 is represented by ID (identifier) of the point which comprises a surface, and the order which connects each point. Further, as shown in FIG. 3D, corresponding to each of the line segments L <b> 1 to L <b> 14, an availability flag F _u indicating whether the information can be used for position / orientation calculation (available if TRUE). Provided.

The model data storage unit 130 stores model data verification data. Model data verification data is configured to include the accumulated data D _S and frame data D _1. Model D _1, D _S of data included in the verification data is stored in correspondence with the line segments each constituting the model data. The frame data D ₁ stores verification data for one frame and is composed of “the number of edges N _E1 to be detected” and “the number of edges actually detected N _D1 ”. The accumulated data D _S is composed of “accumulated data number N _S ”, “total number of edges N _ES to be detected”, and “total number of detected edges N _DS ”.

  The position / orientation calculation unit 140 calculates the position and orientation of the imaging device 100 in a coordinate system (hereinafter referred to as a reference coordinate system) with the observation target object 10 as a reference. Note that the position and orientation of the imaging apparatus 100 are calculated based on the edges in the captured image detected by the image feature detection unit 120 and the three-dimensional model data stored in the model data storage unit 130.

  The model data verification unit 150 verifies whether or not each line segment constituting the three-dimensional model data is used for position and orientation calculation. This verification is performed based on statistical data obtained from the edge detection result by the image feature detection unit 120.

Further, the information processing apparatus 1 has a model data verification flag _FMD . The model data verification flag _FMD is a flag indicating whether or not to verify model data. When the flag F _MD = TRUE, the model data is verified, and when the flag F _MD = FALSE, the verification is not performed. In the initial state, the flag _{F MD} is set to FALSE.

The flag _FMD is switched by the user using an input means (not shown) such as a keyboard or a mouse, for example. When the flag _{F MD} is set to TRUE, the accumulated data _{D S} stored in the model data storage unit 130 is reset to 0. In addition, the number of frames N _max for how many frames of data are stored is set in advance for the stored data. That is, after setting the flag _FMD to TRUE, the user determines in advance how many frames of data are to be accumulated and used for verification of the model data. Validation of the model data is the accumulated number of data N _S is equal to N _max. If the verification of the model data is completed, the flag _{F MD} is set to FALSE, the accumulated data _{D S} is reset to all zeros.

  Next, an example of the flow of processing in the information processing apparatus 1 shown in FIG. 1 will be described with reference to FIG.

(Step S1010)
In step S1010, the information processing apparatus 1 performs an initialization process. In the initialization process, a prediction value that predicts the position and orientation of the imaging apparatus 100 in the reference coordinate system is set, a model data verification flag, an availability flag, and accumulated data are reset.

  Here, the initialization process will be described in detail. First, setting of predicted values of the position and orientation of the imaging apparatus 100 will be described. In the present embodiment, position and orientation measurement is performed by a method in which the approximate position and orientation of the imaging apparatus are sequentially updated using line segment information. Therefore, it is necessary to give an approximate position and orientation of the imaging device in advance before starting position and orientation measurement. Therefore, for example, a predetermined position and orientation are set, and the imaging apparatus 100 is moved to the position and orientation. Alternatively, as shown in Non-Patent Document 1, an artificial index that can be recognized only by detection in an image is arranged, and the coordinates of each vertex of the artificial index and the three-dimensional position of each vertex in the reference coordinate system The position and orientation of the imaging apparatus may be obtained from the correspondence with the above, and the approximate position and orientation of the imaging apparatus may be obtained. Furthermore, a six-degree-of-freedom position and orientation sensor such as a magnetic type, an optical type, and an ultrasonic type may be provided in the imaging apparatus, and the position and orientation obtained from the sensor may be used as the approximate position and orientation of the imaging apparatus. The position and orientation of the imaging device measured by using the artificial index and the 6-degree-of-freedom position / orientation sensor, the 3-degree-of-freedom orientation sensor, and the 3-degree-of-freedom position sensor described above as the approximate position and orientation of the imaging device. Also good.

In step S1010, it sets the model data verification flag _{F MD} to FALSE. Then, for each line segment are stored in the model data storage unit 130, is set to TRUE the usability flag F _u indicating whether the information available to the position and orientation calculation. Furthermore, resetting the accumulated data _{D S} to all zeros. When this initialization is completed, the process proceeds to step S1020.

(Step S1015)
In step S1015, it is determined whether or not the position / orientation measurement is finished. If the user inputs an end instruction via an input unit such as a mouse or a keyboard, the determination in step S1015 is YES, and this process ends. In other cases, NO is determined in the step S1015, and the process proceeds to the step S1020.

(Step S1020)
In step S1020, the information processing apparatus 1 causes the image acquisition unit 110 to capture an image captured by the imaging apparatus 100 into the apparatus (in the information processing apparatus 1).

(Step S1030)
In step S1030, the information processing apparatus 1 causes the image feature detection unit 120 to detect an edge on the image input in step S1020. At the start of this process, frame data N _E1 and N _D1 for each line segment are set to zero. Details of the edge detection process will be described later.

(Step S1040)
In step S <b> 1040, the information processing apparatus 1 uses the position / orientation calculation unit 140 to calculate the position and orientation of the imaging apparatus 100. As described above, this calculation is performed based on the edge on the image detected in step S1040 and the three-dimensional model data stored in the model data storage unit 130. Details of the position / orientation calculation process will be described later.

(Step S1050)
In step S1050, the information processing apparatus 1 determines whether the model data verification unit 150 is in the verification mode. This determination is made based on the value of the model data verification flag _FMD . If the model data is to be verified (F _MD = TRUE), the process proceeds to step S1060. If not verified, the process returns to step S1015.

(Step S1060)
In step S1060, the information processing apparatus 1 starts model data verification processing. In this process, first, the accumulated data N _ES and N _DS for each line segment constituting the three-dimensional model data stored in the model data storage unit 130 are updated. Specifically, the frame data N _E1 and N _D1 for each line segment updated in step S1030 are added to N _ES and N _DS , respectively.

(Step S1070)
In step S1070, the information processing apparatus 1, increased by one stored data number _{N S} (accumulated data count ++).

(Step S1080)
In step S1080, the information processing apparatus 1, it is determined whether or not the stored data number _{N S} is equal to the accumulated number of data _{N max} that is set in advance. If they are equal, the process proceeds to step S1090. If they are not equal, the process returns to step S1015, and the next image, that is, the image of the next frame is acquired in step S1020.

(Step S1090)
In step S1090, the information processing apparatus 1 verifies whether or not each line segment constituting the three-dimensional model data is used for position and orientation calculation in the model data verification unit 150. Although details of this processing will be described later, the model data stored in the model data storage unit 130 is verified based on the accumulated data N _ES and N _DS for each line segment. When this process ends, the process returns to step S1015.

  Next, an example of the flow of edge detection processing in step S1030 in FIG. 4 will be described with reference to FIG.

(Step S1110)
First, the information processing apparatus 1 performs model projection in step S1110. Here, the model projection is to project the three-dimensional model data of the observation target object on the image based on the predicted position and orientation values of the imaging apparatus 100. More specifically, an equation of a straight line on the image when each line segment constituting the three-dimensional model data is projected on the image is obtained. The straight line equation is a straight line equation in which both ends of a line segment are projected on an image and the coordinates of both ends on the image are connected. As the predicted value of the position and orientation of the imaging apparatus 100, the position and orientation of the imaging apparatus 100 obtained by the initialization are used immediately after the initialization in step S1010 shown in FIG. If not immediately after initialization, the position and orientation of the imaging apparatus 100 calculated in the previous frame are used. In order to perform model projection, it is necessary to know camera specific parameters such as the focal length and principal point position as well as the position and orientation of the imaging apparatus 100. As described above, in this embodiment, it is assumed that the intrinsic parameters of the camera are measured and known in advance. Moreover, not performed model projection and edge detection whether the availability flag F _u line segment of FALSE segment is available for the position and orientation calculation.

  FIG. 6 is a diagram illustrating an example of the projection of a three-dimensional model on an image. FIG. 6A shows a captured image itself, and FIG. 6B shows a mode in which a three-dimensional model is projected on the image. When the predicted position and orientation are different from the actual position and orientation, as shown in FIG. 6 (b), between the actually imaged object and the projected image of the three-dimensional model indicated by the bold line. A relative shift occurs. In FIG. 6B, a line segment indicated by a broken line represents a line segment that is actually hidden and invisible among the line segments constituting the three-dimensional model.

(Step S1120, Step S1130)
Next, in step S1120, the information processing apparatus 1 sets division points such that the line segments projected using the straight line equation calculated in step S1110 are divided at equal intervals on the image. This division point is set for all the line segments constituting the three-dimensional model data.

  FIG. 7 is a diagram illustrating an example of division points on an image of a three-dimensional model. Here, the total number of division points is N, and each division point is DPj (j = 1, 2,..., N). The number N of division points can be controlled by changing the interval between the division points on the image. In addition, the interval between division points on the image may be sequentially changed so that the number of division points is constant.

  Hereinafter, edge detection is sequentially performed for DPj (j = 1, 2,..., N). First, j is set to 1 in step S1130.

(Step S1140)
Next, in step S1140, the information processing apparatus 1 determines whether or not the division point DPj is concealed. Concealment refers to a state in which the dividing point DPj is hidden on the other surface of the three-dimensional model. For example, the division point DPj on the line segment indicated by the broken line in FIG. 7 is not visible but is hidden. For example, as shown in Non-Patent Document 8, the determination of whether or not the division point DPj is visible is performed by drawing the three-dimensional model using graphics hardware, and then drawing the division point. This can be done by checking whether the depth buffer has been updated. As a result, if the division point DPj is concealed, the process proceeds to step S1160, otherwise, after adding 1 to the value of the corresponding line segment frame data N _E1, the process of step S1150 Proceed to

(Step S1150)
In step S1150, the information processing apparatus 1 detects an edge corresponding to the dividing point DPj. FIG. 8 is a diagram showing an example of an outline of the edge detection method according to the present embodiment. As shown in FIG. 8, at each division point, an edge is one-dimensionally searched on a line segment (hereinafter referred to as a search line) parallel to the normal direction of the projected line segment and passing through the division point. The edge exists at a position where the density gradient takes an extreme value on the search line. A threshold is provided for the absolute value of the extreme value of the concentration gradient so that an edge having a small absolute value of the extreme value is not detected. In this embodiment, when there are a plurality of edges on the search line, the edge closest to the dividing point is set as the corresponding point, and the image coordinates and the three-dimensional coordinates of the dividing points are held. In the present embodiment, the edge closest to the dividing point is used as the corresponding point. However, the present invention is not limited to this, and the edge having the largest absolute value of the density gradient on the search line may be used as the corresponding point. If an edge is detected in step S 1150, 1 is added to the value of the corresponding segment of the frame data N _D1.

(Step S1160, Step S1170)
In step S1160, the information processing apparatus 1 increments j by 1, and proceeds to step S1170. If the process has been completed for all the division points DPj, the determination in step S1170 is YES, and the edge detection process is terminated. If not completed, NO is returned in step S1170, and the process returns to step S1140.

  Next, an example of the position / orientation calculation process in step S1040 of FIG. 4 will be described in detail. In the position / orientation calculation process, after the end of edge detection, the predicted value of the position and orientation (hereinafter referred to as s) between the object and the imaging device is corrected by iterative calculation by nonlinear optimization calculation. Thereby, the position and orientation of the imaging apparatus are calculated.

Here, among the above-described division points DPj, the total number of division points that are not concealed and whose corresponding points are obtained in the edge detection processing in step S1030 is N _D1 . FIG. 9 is a diagram illustrating an example of an outline of processing for calculating the position and orientation of the imaging apparatus 100 using edge information. In this case, the horizontal and vertical directions of the image are the x-axis and the y-axis, respectively.

The projected image coordinates of a certain dividing point are represented by (u ₀ , v ₀ ), and the inclination of the line segment L to which the dividing point belongs on the image is represented as an inclination θ with respect to the x axis. The inclination θ is calculated as the inclination of a straight line obtained by projecting the three-dimensional coordinates at both ends of the line segment onto the image based on s and connecting the coordinates at both ends on the image. The normal vector on the image of the line segment L is (sin θ, −cos θ). Further, the image coordinates of the corresponding points of the dividing points are (u ′, v ′).

  Here, a point (x, y) on a straight line passing through the point (u, v) and having an inclination of θ is

  (U, v, and θ are constants).

The coordinates at which the division points are projected on the image vary depending on the position and orientation of the imaging apparatus 100. Further, the degree of freedom of the position and orientation of the imaging apparatus 100 is 6 degrees of freedom. That is, s is a six-dimensional vector, and includes three elements that represent the position of the imaging apparatus 100 and three elements that represent the posture. The three elements representing the posture are represented by, for example, an expression by Euler angles, a three-dimensional vector in which the direction represents the rotation axis, and the magnitude represents the rotation angle. The image coordinates (u, v) of the dividing point can be approximated as “Equation 3” by first-order Taylor expansion in the vicinity of (u ₀ , v ₀ ).

By substituting the approximate expression by “Equation 2” into Equation 1, “Equation 3” is obtained.

  Here, the correction value Δs of the position and orientation s of the imaging apparatus 100 is calculated so that the straight line represented by “Equation 3” passes through the image coordinates (u ′, v ′) of the corresponding point of the dividing point.

Is obtained. Since “Equation 4” is established for N _D1 division points, a linear simultaneous equation for Δs as in “Equation 5” is established.

  Here, “Equation 5” is simply expressed as “Equation 6”.

Based on “Equation 6”, Δs is obtained using the generalized inverse matrix (J ^T · J) ⁻¹ · J ^T of the matrix J by the Gauss-Newton method or the like. However, since there are many false detections in edge detection, the following robust estimation method is used. Generally, the error dr becomes large at the division point corresponding to the erroneously detected edge. For this reason, the degree of contribution to the simultaneous equations of “Equation 5” and “Equation 6” increases, and the accuracy of Δs obtained as a result decreases. Therefore, a small weight is given to data at a dividing point having a large error dr, and a large weight is given to data at a dividing point having a small error dr. The weight is given by a Tukey function as shown in “Expression 7”, for example.

  c is a constant. It should be noted that the function that gives the weight does not have to be a Tukey function, and may be a Huber function as represented by “Equation 8”, for example. That is, any function can be used as long as it is a function that gives a small weight to a dividing point having a large error dr, and gives a large weight to a dividing point having a small error dr.

  The weight corresponding to the dividing point DPi is set to wi. Here, the weight matrix W is defined as in “Equation 9”.

Weight matrix W is all but diagonal elements are N _c × N _c square matrix 0, the weights w _i enters the diagonal components. Using this weight matrix W, “Formula 6” is transformed into “Formula 10”.

  The correction value Δs is obtained by solving “Equation 10” as in “Equation 11”.

The position and orientation of the imaging device 100 are updated using Δs obtained in this way. Next, it is determined whether or not the iterative calculation of the position and orientation of the imaging apparatus 100 has converged. When the correction value Δs is sufficiently small, the sum of the errors dr is sufficiently small, or the sum of the errors dr does not change, it is determined that the calculation of the position and orientation of the imaging apparatus 100 has converged. If it is determined that it has not converged, the partial differential of the slopes θ, r, d and u, v of the line segment is calculated again for N _c points using the updated position and orientation of the imaging device 100. Then, the correction value Δs is obtained again from “Equation 11”. Here, the Gauss-Newton method is used as the nonlinear optimization method. However, the nonlinear optimization method is not limited to this, and other nonlinear optimization methods such as a Newton-Raphson method, a Levenberg-Marquardt method, a steepest descent method, and a conjugate gradient method may be used. The above is the description of the position / orientation calculation method of the imaging apparatus in step S1040.

  Next, an example of the model data verification process in step S1090 of FIG. 4 will be described with reference to FIG.

(Step S1210)
In step S1210, the information processing apparatus 1 verifies the model data based on “total number of edges N _ES to be detected” and “total number of edges N _DS actually detected”. _{Specifically,} to calculate the ratio _{R ED} of _{N DS} for _{N ES.}

(Step S1220, Step S1230)
In step S1220, the information processing apparatus 1 compares the percentage _{R ED} calculated in step S1210, the threshold _{T ED.} As a result of the comparison, if R _ED exceeds the threshold value T _ED , NO is determined in step S1230, and this process ends. On the other hand, if R _ED is less than threshold value T _ED , YES is determined in step S1230, and the process proceeds to step S1240.

(Step S1240)
In step S1240, the information processing apparatus 1, and the line segment of the detection rate is low, sets the usability flag _{F u} of the corresponding line segment to FALSE. As a result, the line segment is not used in the position / orientation calculation process thereafter. For example, in the model data shown in FIG. 3, if the edge corresponding to L14 by lighting condition is not detected, the flag F _u of L14 is set to FALSE, so not used for position and orientation calculation process.

(Step S1250)
In step S1250, the information processing apparatus 1 sets the model data verification flag _{F MD} to FALSE, the resets the accumulated data _{D S} to all 0, the processing ends.

  As described above, according to the first embodiment, for each line segment, the ratio of “the number of edges actually detected” to “the number of edges to be detected” is calculated, and the corresponding edge belongs based on the ratio. It is determined whether or not the line segment is used for position and orientation calculation. Thereby, it is possible to prevent a line segment that is not actually detected from being used for position and orientation calculation. Thereby, the measurement of a three-dimensional model can be simplified, and the fall of the precision and stability of alignment can be suppressed.

  In the above description, it is determined whether or not the corresponding line segment is used for position / orientation calculation based on the ratio of the division points where the edge (corresponding point) cannot be detected. However, the edge can be detected. The determination may be made based on information other than the ratio of the division points that have not been present. For example, the same line segment may be detected from a plurality of images, and it may be determined whether or not to use the line segment for position and orientation calculation based on the ratio of the line segment not being detected. .

(Embodiment 2)
In the first embodiment described above, the ratio of the “number of edges actually detected” to the “number of edges to be detected” is calculated, and whether or not each line segment is used for position and orientation calculation based on the ratio. I was doing. On the other hand, in the second embodiment, a case will be described in which it is determined whether or not each line segment is used for position and orientation calculation based on an error in performing position and orientation calculation. The configuration in the second embodiment is the same as that in FIG. 1 and FIG. 2 in which the first embodiment is described. Therefore, the description thereof will be omitted, and differences will be mainly described here.

First, model data verification data according to the second embodiment will be described. Constituted Here, as in Embodiment 1, the model data storage unit 130, model data verification data is stored, the model data verification data, and a storage data D _S and frame data D ₁ . Also, D _1, D _S included in the model data verification data is stored in correspondence with the line segments each constituting the model data.

The frame data D ₁ stores verification data for one frame, and is composed of “number of edges N _E1 to be detected” and “number of edges N _C1 used as effective information in position and orientation calculation”. The Further, the accumulated data D _S is obtained from “accumulated data number N _S ”, “total number of edges N _ES to be detected”, and “number of edges N _CS used as effective information in position and orientation calculation”. Composed. N _ES is the total sum of N _E1 for N _max frames, and N _CS is the total sum of N _C1 for N _max frames. That is, the second embodiment differs from the first embodiment in that the accumulated data stores not the “number of edges actually detected” but “the number of edges used as effective information in position and orientation calculation”. .

  An example of a processing flow in the information processing apparatus 1 according to the second embodiment will be described with reference to FIG.

(Step S2010)
In step S2010, the information processing apparatus 1 performs an initialization process. In the initialization process, setting of predicted values of the position and orientation of the imaging apparatus 100 in the reference coordinate system, model data verification flag, availability flag, and accumulated data are reset. It should be noted that the setting of the predicted value of the position and orientation of the imaging apparatus 100 and the resetting of the model data verification flag and the availability flag are the same as step S1010, and thus the description thereof is omitted. In the second embodiment, the “accumulated data number N _S ”, the “total number of edges N _ES to be detected”, and the “number of edges N _CS used as effective information in position and orientation calculation” are all set to 0 and accumulated. to reset the data _{D S.}

(Step S2015)
In step S2015, the position / orientation measurement end determination is performed. If the user inputs an end instruction via an input means such as a mouse or a keyboard, YES is returned in step S2015, and this process ends. In other cases, NO is determined in step S2015, and the process proceeds to step S2020.

(Step S2020)
In step S2020, the information processing apparatus 1 causes the image acquisition unit 110 to capture an image captured by the imaging apparatus 100 into the apparatus (in the information processing apparatus 1).

(Step S2030)
In step S2030, the information processing apparatus 1 causes the image feature detection unit 120 to detect an edge on the image input in step S2020. At the start of this process, the frame data N _E1 and N _C1 for each line segment are set to zero. The edge detection process is the same as that in FIG. 5 of the first embodiment, and a detailed description thereof is omitted. The difference from the first embodiment is that, in the first embodiment, N _E1 and N _D1 that are frame data are updated, but only N _E1 is updated in step S2030.

(Step S2040)
In step S2040, the information processing apparatus 1 uses the position / orientation calculation unit 140 to calculate the position and orientation of the imaging apparatus 100. The position / orientation calculation process is the same as that in the first embodiment, and a detailed description thereof will be omitted. As a difference from the first embodiment, in step S2030, after the position / orientation calculation process described in step S1030 is completed, the frame data N _C1 is obtained based on the weight matrix W (see Equation 9) obtained in the calculation process. Update. The weight matrix W is defined by a weight corresponding to each division point, and the weight is larger as the value of the error dr is smaller. Therefore, it is considered that an edge having a weight less than a certain value is erroneously detected or mishandled, and information on an edge having a weight greater than a certain value is effectively used. Add 1 to the value of N _C1.

(Step S2050)
In step S2050, the information processing apparatus 1 determines whether the model data verification unit 150 is in the verification mode. This determination is made based on the value of the model data verification flag _FMD . When the model data is verified (F _MD = TRUE), the process proceeds to step S2060. When the model data is not verified, the process returns to step S2015.

(Step S2060)
In step S2060, the information processing apparatus 1 starts model data verification processing. In this process, first, the accumulated data N _ES and N _CS are updated for each line segment constituting the three-dimensional model data stored in the model data storage unit 130. That is, the frame data N _E1 and N _C1 for each line segment updated in the processes of steps S2030 and S2040 are added to N _ES and N _CS , respectively.

(Step S2070)
In step S2070, the information processing apparatus 1, increased by one stored data number _{N S} (accumulated data count ++).

(Step S2080)
In step S2080, the information processing apparatus 1, it is determined whether or not the stored data number _{N S} is equal to the accumulated number of data _{N max} that is set in advance. If they are equal, the process proceeds to step S2090. If they are not equal, the process returns to step S2015, and the next image, that is, the image of the next frame is acquired by the process of step S2020.

(Step S2090)
In step S2090, the information processing apparatus 1 verifies in the model data verification unit 150 whether or not each line segment constituting the three-dimensional model data is used for position and orientation calculation. Although details of this process will be described later, in the second embodiment, the model data stored in the model data storage unit 130 is verified based on the accumulated data N _ES and N _CS for each line segment. When this process ends, the process returns to step S2015.

  Next, an example of the model data verification process in step S2090 in FIG. 11 will be described with reference to FIG.

(Step S2110)
In step S2110, the information processing apparatus 1 verifies the model data based on the “total number of edges N _ES to be detected” and the “number of edges N _CS used as effective information in position and orientation calculation”. Do. _{Specifically,} to calculate the ratio _{R EC} of _{N CS} for _{N ES.}

(Step S2120, Step S2130)
In step S2120, the information processing apparatus 1 compares the percentage _{R EC} calculated in step S2110, the threshold _{T EC.} Result of the _comparison, if exceeded _{R EC} is the threshold _{T EC,} is NO in step S2130, the process ends. On the other _hand, if _{R EC} is less than the threshold _{T EC,} the process proceeds at step S2130 YES, and the processing of step S2140.

(Step S2140)
In step S2140, the information processing apparatus 1, and that there is a high possibility of false detection or false corresponding sets the usability flag F _u of the corresponding line segment to FALSE. As a result, the line segment is not used in the position / orientation calculation process thereafter.

(Step S2150)
In step S2150, the information processing apparatus 1 sets the model data verification flag _{F MD} to FALSE, the resets the accumulated data _{D S} to all 0, the processing ends.

  As described above, according to the second embodiment, for each line segment, whether or not to use each line segment for position and orientation calculation based on the weight value determined by the magnitude of the error when performing position and orientation calculation. Judgment is made. As a result, it is possible to prevent a line segment with many erroneous detections and erroneous responses from being used for position and orientation calculation.

(Embodiment 3)
In the first and second embodiments described above, the case where the information processing apparatus 1 according to the embodiment of the present invention is applied to the alignment of the AR has been described. On the other hand, in the third embodiment, a case will be described as an example where line segment data is applied to measurement of line segment data using an image and suitable for alignment using an edge.

  FIG. 13 is a diagram illustrating an example of the configuration of the information processing apparatus 1 according to the third embodiment.

  The information processing apparatus 3 includes an image acquisition unit 310, an image storage unit 320, a model data calculation unit 330, a model data storage unit 340, and a model data verification unit 350. The information processing device 3 calculates a three-dimensional model of the line segment based on a plurality of images of the real space or real object imaged by the imaging device 300.

  The image acquisition unit 310 inputs an image captured by the imaging device 300 to the information processing device 3. The image acquisition unit 310 performs the same function as the image acquisition unit 110 of FIG. 2 described in the first embodiment.

  The image storage unit 320 stores the image input from the image acquisition unit 310. The image storage unit 320 stores the image based on an instruction from the user via the input unit.

  The model data calculation unit 330 calculates a three-dimensional model of the line segment using a plurality of images stored in the image storage unit 320. In the third embodiment, a three-dimensional model of a line segment is calculated using the method disclosed in Non-Patent Document 5.

  The model data storage unit 340 stores the three-dimensional model of the line segment calculated by the model data calculation unit 330.

  The model data verification unit 350 verifies the three-dimensional model of the line segment stored in the model data storage unit 340 by performing position and orientation calculation based on the image input from the image acquisition unit 310.

  Next, an example of a processing flow in the information processing apparatus 3 according to the third embodiment will be described with reference to FIG.

(Step S3010)
In step S3010, the information processing apparatus 3 stores an image necessary for calculation of a three-dimensional model of the line segment in the image storage unit 320. Non-Patent Document 5 states that “six correspondences in three images are necessary to restore a three-dimensional model of a line segment”, and therefore at least three or more images are stored in step S3010. . The image is stored by, for example, inputting an image storage command by an input unit such as a keyboard or a mouse while the user freely moves the imaging apparatus 300. When the image storage command is input, the image input from the image acquisition unit 310 is stored in the image storage unit 320 at the time when the command is input. Here, the image is stored by the user's input, but the image may be automatically stored based on the change of the screen.

(Step S3020)
In step S3020, the information processing apparatus 3 detects a line from the image stored in step S3010. The line is detected based on a user instruction. Specifically, it is performed manually by the user specifying the start and end points of the line on the image. Note that the line detection method is not limited to this, and any method can be used as long as it can detect a line on an image. For example, after performing edge detection by the method shown in Non-Patent Document 7, lines on the image may be detected by Hough transform.

(Step S3030)
In step S3030, the information processing apparatus 3 performs association between the images of the lines detected in step S3020. This association is performed based on a user instruction. That is, it is performed manually by the user. Note that the line association method is not limited to this, and for example, similar lines may be automatically associated using information on pixels around the line.

(Step S3040)
In step S3040, the information processing apparatus 3 causes the model data calculation unit 330 to calculate a three-dimensional model of line segments (lines) based on the line information associated in step S3030. Then, the three-dimensional position of the end point is calculated from the calculated three-dimensional model of the line segment and the position of the end point on the image, and stored in the model data storage unit 340 as a three-dimensional model of the line segment in the manner shown in FIG. To do. The three-dimensional model of the line segment is represented by a straight direction vector and a passing position in a three-dimensional space. The calculation of the three-dimensional model of the line segment is performed by a method disclosed in Non-Patent Document 5, for example. The method for calculating the three-dimensional model of the line segment is not limited to this, and any method may be used as long as it is calculated from the correspondence of lines between images.

Here, the outline of the calculation method of the three-dimensional model of the line segment in Non-Patent Document 5 will be briefly described.
(1). An initial value is randomly given for the posture of the imaging apparatus when each image is captured.
(2). The line direction vector is calculated based on the equation of the line on the image, the direction vector of the line in the three-dimensional space, and the constraint conditions satisfied by the attitude of the imaging device.
(3). Based on the equation of the line on the image, the passage position of the line, and the constraint conditions satisfied by the position of the imaging device, the passage position of the line and the position of the imaging device are calculated.
(4). The unknown parameters obtained in (1), (2), and (3) are optimized.
(5). The accuracy of the unknown parameter obtained in (4) is evaluated. If the accuracy is poor, the procedure returns to (1). Finish if accuracy is good.

(Step S3050)
In step S3050, the information processing apparatus 3 uses the model data verification unit 350 to verify the three-dimensional model of the line segment calculated in step S3040. Specifically, based on the image input from the image acquisition unit 310 and the three-dimensional model stored in the model data storage unit 340, each line constituting the model data by the method described in the first embodiment or the second embodiment. Evaluate minutes. Then, a line segment with a small number of actual detections relative to the number of times to be detected or a line segment with a small number of times used as effective information is deleted from the model data. Thereby, the three-dimensional model stored in the model data storage unit 340 is updated.

(Step S3060)
In step S3060, the information processing apparatus 3 presents the three-dimensional model calculation result in step S3040 and the model verification result in step S3050 to the user, for example, by displaying them. Thereby, the user determines whether to accept the three-dimensional model calculation result or the model verification result. If the user accepts the result, the fact is input to the information processing apparatus 3, YES is obtained in step S3060, and this process ends. If not accepted, information to that effect is input to the information processing apparatus 3, NO is determined in step S3060, and the process returns to step S3010.

  As described above, according to the third embodiment, model data suitable for alignment using an edge can be measured even when the measurement result of line segment data using an image is evaluated.

  The above is an example of a typical embodiment of the present invention, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be appropriately modified and implemented without departing from the scope of the present invention. .

(Modified Embodiment 1)
In the above description, using the first and second embodiments, two patterns have been described for determining whether to use each line segment for position and orientation calculation. These two patterns of processing do not necessarily have to be performed separately, and may be performed in combination with two determination methods. That is, the present invention can be implemented using at least one of the determination processes.

For example, as the frame data, “number of edges N _E1 to be detected”, “number of edges N _C1 used as effective information in position and orientation calculation”, “actually detected for each line segment in each frame Three pieces of information “edge number N _D1 ” are obtained. As the accumulated data, for each line segment, “number of edges N _ES to be detected”, “number of edges N _CS used as effective information in position and orientation calculation”, “number of edges actually detected N Three of “ _DS ” are accumulated. N _ES , N _CS , and N _DS are sums of frame data N _E1 , N _C1 , and N _ES for N _max frames. Based on the accumulated data N _ES , N _CS , N _DS , it is determined whether or not the line segment is used for position and orientation calculation. For example, the determination is performed using both N _ES and N _CS ratios R _ED and R _{CD to} N _DS . That is, threshold values T _ED and T _CD are provided for R _ED and R _CD , respectively, and when both R _ED and R _CD are less than the threshold values T _ED and T _CD , the corresponding line segment has a low detection rate. In addition, it is determined that the line segment is highly likely to be erroneously detected or handled. Then, the flag _Fu used for position / orientation calculation is set to FALSE, and thereafter, the line segment is not used for position / orientation calculation. Further, when both R _ED and R _CD are not less than the threshold values T _ED and T _CD and one of them is less than the threshold value, it may be determined that the line segment is not used for position and orientation calculation. Further, the ratio of (N _ES , + N _CS ) to N _DS may be used.

(Modified Embodiment 2)
In the above description, the model data is verified using a user input as a trigger. However, the model data may be verified automatically instead of being triggered by user input.

For example, the model data verification flag _FMD may be set to TRUE for a certain time after the start of position / orientation measurement, and the model data may be automatically verified. Further, when the accuracy of the position and orientation calculation has been reduced (if error summation is large), automatically model data verification flag F _MD is set to TRUE, the the validation of the model data to automatically perform May be. In this case, once after setting the usability flag F _u corresponding to all of the line segments to TRUE, the verification of the model data. As a result, it is possible to cope with a case where past verification is incorrect or a change in lighting conditions.

(Modified Embodiment 3)
In the above description, whether the line segment is used for position / orientation calculation is automatically determined based on the threshold value T on the information processing apparatus side, but may be determined by the user.

  For example, in the first and second embodiments, when a three-dimensional model of a line segment is drawn, the display method is changed between a line segment used for position and orientation calculation and a line segment not used such as changing the color and thickness. Present to the user. The user refers to the three-dimensional model of the presented line segment and determines whether to approve the determination result. The user's approval is performed through input means such as a mouse or a keyboard.

  Further, for example, in the third embodiment, based on the calculated three-dimensional model of the line segment and the position and orientation of the imaging device when each image is captured, the three-dimensional model of the line segment is displayed on each image. draw. At this time, the display method is changed between the line segment used for position and orientation calculation and the line segment not used, such as changing the color and thickness, and presented to the user. The user determines whether to approve the determination result based on the presentation result.

(Modified Embodiment 4)
In the above description, it is determined whether or not each line segment constituting the three-dimensional model data is used for position and orientation calculation. However, this determination is not necessarily limited to the unit of line segment.

  For example, it may be determined whether or not a set of a plurality of line segments is used for position and orientation calculation as one unit. On the contrary, one line segment may be divided into a plurality of segments, and it may be determined whether or not each divided line segment is used for position and orientation calculation. Non-Patent Document 5 shows a method for calculating a three-dimensional model of an infinitely long straight line. The three-dimensional coordinates of the end point are calculated from the position of the end point designated on the image and the three-dimensional model of the straight line. However, if a straight line is divided into a plurality of line segments and each line segment is used for position / orientation calculation, only the existing line segments will be used. This calculation process is unnecessary.

(Modified Embodiment 5)
In the above description, the three-dimensional model of the line segment is explicitly verified by setting a flag (model data verification flag) indicating whether to perform verification. However, instead of explicitly performing verification, model verification may always be performed while calculating the position and orientation.

For example, position / orientation calculation is performed using line segments used for position / orientation calculation, model projection is performed based on the calculation result, and edge detection is performed in the vicinity of the division points of the projected line segments. At this time, each line segment to project is all the line segments in the model data. The number of times that an edge should be detected and the number of times of actual detection are stored as accumulated data, and when N _max frames are accumulated, it is determined whether or not the line segment is used for position and orientation calculation. After the determination, the accumulated data is cleared, N _max frames of data are accumulated again, and the determination of whether or not the line segment is used for position and orientation calculation is repeated. Thereby, even if it is a case where illumination conditions and the position and attitude | position of an imaging device change significantly, the line segment suitable for each can be utilized. Since the calculation load for edge detection is large, it is not necessary to store the accumulated data every frame, and it may be accumulated every several frames.

(Modified Embodiment 6)
In the above description, the case where the line segment projected on the image is equally divided and the corresponding edge is detected on the search line passing through the dividing point and perpendicular to the projection line segment has been described as an example. However, the edge detection method is not limited to this, and any method may be used as long as the corresponding edge is detected based on the line segment projected on the image.

  For example, the corresponding edge may be detected using image information around the edge. Further, as disclosed in Non-Patent Document 8, after detecting a plurality of edges on a search line by the above-described method, corresponding edges may be determined using image information around the edges. Further, a one-dimensional search may be performed in the normal direction of the line segment without dividing the projected line segment. The above is the description of the modified embodiment.

  It should be noted that the present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  Further, the present invention achieves the functions of the embodiments by supplying a software program directly or remotely to a system or apparatus, and reading and executing the supplied program code by a computer incorporated in the system or apparatus. This includes cases where In this case, the supplied program is a computer program corresponding to the flowchart shown in the drawings in the embodiment.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention. In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to an OS (Operating System), or the like.

  Examples of the computer-readable recording medium for supplying the computer program include the following. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As another program supply method, a client computer browser is used to connect to a homepage on the Internet, and the computer program of the present invention is downloaded from the homepage to a recording medium such as a hard disk. In this case, the downloaded program may be a compressed file including an automatic installation function. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  Further, the program of the present invention may be encrypted, stored in a recording medium such as a CD-ROM, and distributed to users. In this case, a user who has cleared a predetermined condition is allowed to download key information for decryption from a homepage via the Internet, execute an encrypted program using the key information, and install the program on the computer. You can also.

  In addition to the functions of the above-described embodiment being realized by the computer executing the read program, the embodiment of the embodiment is implemented in cooperation with an OS or the like running on the computer based on an instruction of the program. A function may be realized. In this case, the OS or the like performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, so that part or all of the functions of the above-described embodiments are realized. May be. In this case, after a program is written to the function expansion board or function expansion unit, the CPU (Central Processing Unit) provided in the function expansion board or function expansion unit is a part of the actual processing or based on the instructions of the program. Do everything.

Claims (13)

An information processing device that calculates information on the position and orientation of an imaging device with respect to an object imaged by the imaging device,
Holding means for holding three-dimensional model data of the object;
Obtaining means for obtaining a plurality of images of the object imaged by the imaging device;
Detecting means for projecting a geometric feature of the three-dimensional model data onto each of the plurality of images, and detecting a geometric feature corresponding to the projected geometric feature from each of the plurality of images;
Based on the number of detections of the geometric feature corresponding to the geometric feature included in the three-dimensional model data detected in each of the plurality of images, the geometric feature of the three-dimensional model data is used for the position and orientation calculation processing of the imaging device. A determination means for determining whether to use;
An information processing apparatus comprising: an update unit configured to update the three-dimensional model data based on a geometric feature of the three-dimensional model data determined to be used for position and orientation calculation processing by the determination unit. The update unit sets first information to the geometric feature of the three-dimensional model data determined to be used for the position / orientation calculation process by the determination unit, and is determined to be used for the position / orientation calculation process by the determination unit. The information processing apparatus according to claim 1, wherein second information is set in a geometric feature of the three-dimensional model data that has not been set. The information processing apparatus according to claim 2, wherein the first information is a flag used for position and orientation calculation processing, and the second information is a flag that is not used for position and orientation calculation processing. . The determination means includes
If the detection count is smaller than a predetermined threshold, the corresponding geometric features included in the three-dimensional model data, according to claim 3, characterized in that determines not to use the position and orientation calculation process Information processing device. The determination means includes
The determination is performed based on a magnitude of an error indicating a deviation of a geometric feature between the geometric feature detected by the detecting unit and a line segment included in the three-dimensional model data corresponding to the detected geometric feature. the information processing apparatus according to claim 1 or 2, characterized in that. Presenting means for presenting the result of determination by the determining means to the user;
The information processing apparatus according to claim 1 or 2, characterized in that it further comprises an input means for inputting whether the user approves the basis of the determination result presented by the presenting means. The geometric features, the information processing apparatus according to any one of claims 1 to 6, characterized in that a line segment or point features. The geometric feature is a line segment;
The detection means includes
A plurality of division points are provided at predetermined intervals for each line segment included in the three-dimensional model data, and the corresponding line segments are detected from the image by searching for corresponding points corresponding to the division points;
The determination means includes
Wherein the detecting means, wherein said corresponding division points on the basis of the detection frequency of the corresponding point, the information processing apparatus according to claim 1 or 2, characterized by performing the determination. The information processing according to any one of claims 1 to 8 , further comprising deriving means for deriving a position and orientation of the imaging apparatus using the three-dimensional model data updated by the updating means. apparatus. An imaging device and an information processing device for deriving the position and orientation of the imaging device,
The imaging device
Imaging means for imaging an object;
Transmitting means for transmitting an image obtained by imaging the object;
Have
The information processing apparatus includes:
Holding means for holding three-dimensional model data of the object;
Acquisition means for acquiring a plurality of images of the object transmitted via the transmission means;
Projected onto each of the plurality of images, and detecting means for detecting a geometric feature corresponding to the projected geometric features from each of the plurality of images,
Based on the number of detections of the geometric feature corresponding to the geometric feature included in the three-dimensional model data detected in each of the plurality of images, the geometric feature of the three-dimensional model data is used for the position and orientation calculation processing of the imaging device. A determination means for determining whether to use;
Updating system for updating the three-dimensional model data based on the geometric feature of the three-dimensional model data determined to be used for the position and orientation calculation processing by the determination means. The system according to claim 10 , wherein the imaging device is provided in a head-mounted display. A processing method of an information processing device for calculating information on a position and orientation of an imaging device with respect to an object imaged by the imaging device,
An acquisition step of acquiring a plurality of images of the object imaged by the imaging device;
A detecting step of projecting onto each of the plurality of images, and detecting a geometric feature corresponding to the projected geometric feature from each of the plurality of images;
Based on the number of detections of the geometric feature corresponding to the geometric feature included in the three-dimensional model data detected by each of the plurality of images , the determination unit determines the geometric feature of the three-dimensional model data as the position of the imaging device. A determination step of determining whether to use for posture calculation processing;
Updating means for updating the three-dimensional model data based on the geometric features of the three-dimensional model data determined to be used in the position and orientation calculation processing by the determination step. Processing method of the processing device. A computer built in an information processing device for calculating information on the position and orientation of the imaging device relative to an object imaged by the imaging device;
Holding means for holding three-dimensional model data of the object;
Obtaining means for obtaining a plurality of images of the object imaged by the imaging device;
Detecting means for projecting a geometric feature of the three-dimensional model data onto each of the plurality of images, and detecting a geometric feature corresponding to the projected geometric feature from each of the plurality of images;
Based on the number of detections of the geometric feature corresponding to the geometric feature included in the three-dimensional model data detected in each of the plurality of images, the geometric feature of the three-dimensional model data is used for the position and orientation calculation processing of the imaging device. A determination means for determining whether to use;
A program for functioning as update means for updating the three-dimensional model data based on the geometric feature of the three-dimensional model data determined to be used for position and orientation calculation processing by the determination means.

Images