JP2016021097A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve recognition robustness of an object and reduce processing load in an AR technology.SOLUTION: An image processing device 1 which superposes virtual information on a preview image comprises an image acquisition unit 10, an image recognition unit 20, an object relation estimation unit 30, and a virtual information display unit 70. The image acquisition unit 10 acquires a preview image. The image recognition unit 20 recognizes an object in the preview image. The object relation estimation unit 30 estimates a relative posture between the objects recognized by the image recognition unit 20, classifies the objects, and recognizes objects other than the main objects classified into the same group on the basis of the recognition results of the main objects by the image precognition unit 20. The virtual information display unit 70 superposes virtual information on the preview image on the basis of the recognition results by the image recognition unit 20 and the recognition results by the object relation estimation unit 30.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an image processing method, and a program.

  In recent years, AR (augmented reality) technology for processing virtual space information (video) by a computer and superimposing virtual information has attracted attention. By using the AR technology, it becomes possible to support the user's action or present information intuitively to the user. For example, by using AR technology for signs and advertisements around the user, detailed information that cannot be conveyed in a limited space, video, 3D content, etc. can be presented, location, time, and viewer attributes The information to be presented can be changed as appropriate.

  Mobile terminals are expected as a major platform for AR technology. As this portable terminal, for example, there are terminals such as a smartphone and an HMD (Head Mounted Display) equipped with an imaging device (camera) and a display and having sufficient processing performance for image processing.

  In the AR technology, in order to superimpose virtual information at a correct position, it is necessary to estimate the relative posture (position and orientation) between the imaging device and the real space in real time.

  As a technique for estimating the posture described above, for example, a technique using a reference marker to be recognized has been proposed (for example, see Non-Patent Documents 1 and 2). As a reference marker, an AR marker is applied in Non-Patent Document 1, and an arbitrary image is applied in Non-Patent Document 2. However, in the methods shown in Non-Patent Documents 1 and 2, it is necessary to register a reference marker in advance in the apparatus that performs the posture estimation described above.

  Therefore, as a method for estimating the posture described above, a method for modeling the real space in the process of the previous stage of superimposing virtual information and treating the entire restored (modeled) space as a reference marker has been proposed (for example, Non-Patent Document 3). According to this method, since the reference marker is appropriately created, it is not necessary to register the reference marker in advance in the above-described posture estimation apparatus.

  There are trade-offs in convenience and processing load between the method using the AR marker, the method using an arbitrary image, and the method of appropriately creating a reference marker. For this reason, it is necessary to select an appropriate method according to the situation.

  In addition, a high-speed (efficient) recognition algorithm is being studied so that a terminal with low processing performance can cope with the above-described various methods. For example, Patent Document 1 proposes a method of tracking feature points in a preview image that is continuously input in posture tracking processing by combining initial posture estimation processing and posture tracking processing. . According to this method, a posture can be estimated in real time even with a terminal having low processing performance.

  By the way, in the AR technology, there are two methods for arranging virtual information. The first method is a method in which the virtual information is fixedly arranged in the AR space by registering the relative positional relationship of the virtual information with respect to the reference marker. The second method is a method of arranging virtual information in the AR space by registering a relative positional relationship of virtual information with respect to an object different from the reference marker.

  In the second method, the display position of the virtual information changes according to the movement of the camera and the movement of the reference object. This second method is used, for example, when displaying virtual information such as a 3D model on a trading card.

  When virtual information such as a 3D model is displayed on a trading card, since the posture between the object (trading card) and the real space is not fixed, in order to present an AR space (for example, a card game field) A fixedly arranged reference marker is required. Moreover, the same processing as that for recognizing the reference marker is required for the recognition of the object. Furthermore, when displaying virtual information on a plurality of objects, it is necessary to recognize each object (estimate the posture), and thus high processing capability is required of the terminal.

JP 2013-508844 A

H. Kato and M. Billinghurst, “Marker tracking and hmd calibration for a video-based augmented reality conferencing system,” in Proc. Of IEEE and ACM International Workshop on Augmented Reality, 1999. D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmalstieg, “Real-time detection and tracking for augmented reality on mobile phones,” IEEE Trans. On Visualization and Computer Graphics, 2010. G. Klein and D. Murray.Parallel tracking and mapping for small ar workspaces.In Proc.Of International Symposium on Mixed and Augmented Reality, 2007. S. Benhimane and E. Malis, “Homography-based 2d visual tracking and servoing,” International Journal of Robotics Research, 2007.

  In the methods of Patent Document 1 and Non-Patent Documents 1 and 2, when virtual information is registered in an object, each terminal needs to recognize the object independently. For this reason, as the number of objects increases, the processing load on each terminal increases, making it difficult to realize real-time processing on each terminal. As a result, the number of virtual information that can be displayed on each terminal is limited, and usability There was a risk that it would fall.

  Further, the estimation of the posture of the object is performed independently for each preview image. For this reason, the object recognition may be temporarily failed due to occlusion (shielding) or reflection of light (out-of-white), and display of virtual information may be interrupted.

  Furthermore, the process of estimating the initial posture may lack robustness with respect to the shooting angle and shooting distance compared to the tracking processing of the posture, so depending on the shooting position, if the virtual information display is interrupted, the virtual information display may be Sometimes it could not be resumed.

  Therefore, the present invention has been made in view of the above-described problems, and it is an object of the AR technology to reduce processing load and improve robustness of object recognition.

The present invention proposes the following matters in order to solve the above problems.
(1) The present invention is an image processing apparatus (for example, equivalent to the image processing apparatus 1 in FIG. 1) that superimposes virtual information on a preview image, and an image acquisition unit (for example, in FIG. 1) that acquires the preview image. Image recognition unit 10), an image recognition unit that recognizes an object in the preview image acquired by the image acquisition unit (for example, equivalent to the image recognition unit 20 in FIG. 1), and the image recognition unit. The relationship between objects (e.g., corresponding to the relative posture between objects described later) is estimated, the objects are classified based on the estimation result, and the main object that is one of the objects classified into the same group An object that recognizes an object other than the main object classified into the group based on the recognition result by the image recognition means. Acquired by the image acquisition means based on a recognition result by the image relation estimation means (for example, corresponding to the object relation estimation unit 30 in FIG. 1), a recognition result by the image recognition means, and a recognition result by the object relation estimation means. Proposed is an image processing apparatus including virtual information display means (for example, corresponding to the virtual information display unit 70 in FIG. 1) for superimposing virtual information on the preview image.

  According to the present invention, an image processing device that superimposes virtual information on a preview image is provided with image acquisition means, image recognition means, object relation estimation means, and virtual information display means, and the preview image is acquired by the image acquisition means. It was decided. The image recognition means recognizes the object in the preview image acquired by the image acquisition means. In addition, the object relationship estimation means estimates the relationship between the objects recognized by the image recognition means, classifies the objects based on the estimation results, and is a main object that is one of the objects classified into the same group Based on the recognition result by the image recognition means, objects other than the main objects classified into this group are recognized. The virtual information display means superimposes the virtual information on the preview image acquired by the image acquisition means based on the recognition result by the image recognition means and the recognition result by the object relationship estimation means. For this reason, the object relation estimating means can recognize the object based on the recognition result of the object having a high relation with the object. Therefore, by recognizing the object by the object relationship estimating means, it is possible to reduce the number of objects recognized by the image recognizing means or to recognize an object that could not be recognized by the image recognizing means. Therefore, in the AR technology, it is possible to reduce the processing load and improve the robustness of object recognition.

  (2) In the image processing apparatus according to (1), the object relationship estimation unit applies the object recognized by the image recognition unit as the main object, and classifies the main object into the same group as the main object. As an object other than the main object, an image processing apparatus is proposed in which an object that is classified into the same group as the main object and that cannot be recognized by the image recognition unit is applied.

  According to the present invention, in the image processing apparatus of (1), the object relation estimation means applies the object recognized by the image recognition means as the main object, and other than the main objects classified into the same group as the main object. As the object, an object classified into the same group as the main object and not recognized by the image recognition means is applied. For this reason, the object which could not be recognized by the image recognition means can be recognized based on the recognition result of the main object.

  (3) In the image processing apparatus according to (1) or (2), the object relation estimation unit selects one object from each group as a main object and classifies the same into the same group as the main object. Proposed is an image processing apparatus comprising a recognition processing control unit (for example, corresponding to the recognition processing control unit 40 in FIG. 11) that pauses recognition by the image recognition unit for objects other than the main object that has been performed. ing.

  According to the present invention, in the image processing apparatus according to (1) or (2), the object relation estimation unit selects one object from each group as a main object. In addition, the image processing apparatus of (1) or (2) is provided with a recognition processing control unit that pauses recognition by the image recognition unit for objects other than the main object classified into the same group as the main object. For this reason, objects other than the main object classified into the same group as the main object can be recognized by the object relationship estimation unit even when the recognition by the image recognition unit is suspended. The number can be reduced.

  (4) In the image processing apparatus according to (3), when the recognition processing control unit fails to recognize the main object by the image recognition unit, the main object classified into the same group as the main object. An image processing apparatus is proposed in which recognition by the image recognition means is resumed for objects other than the above.

  According to the present invention, in the image processing apparatus of (3), when the recognition processing control unit fails to recognize the main object by the image recognition unit, the objects other than the main object classified into the same group as the main object are The recognition by the image recognition means is resumed. For this reason, when recognition by an object relationship estimation means becomes impossible, recognition by an image recognition means can be restarted and the robustness of object recognition can be further improved.

  (5) In the image processing apparatus according to (3) or (4), the recognition processing control unit may recognize a recognition result by the object relationship estimation unit for an object whose recognition by the image recognition unit is suspended. The image processing apparatus has been proposed in which the preview image obtained by the image obtaining unit is collated and if the collation fails, the recognition of the object by the image recognizing unit is resumed.

  According to the present invention, in the image processing apparatus according to (3) or (4), the recognition processing control means recognizes the recognition result by the object relation estimation means for the object whose recognition by the image recognition means is suspended, as the image acquisition means. When the collation fails, the recognition by the image recognition means is resumed. For this reason, it can be determined whether the recognition result by the object relationship estimation means is correct.

  (6) In the image processing apparatus according to any one of (3) to (5), the recognition processing control unit is previously acquired by the image acquisition unit when the recognition by the image recognition unit is resumed. An image processing apparatus is proposed in which the image recognition unit is made to track the posture with the recognition result of the object relation estimation unit in the preview image as an initial value.

  Here, if the object recognition failure due to occlusion or light reflection is temporary, the posture tracking unit 23 succeeds in the posture tracking processing when the object recognition failure is resolved. It is assumed that In general, rather than the estimation of the initial value of the posture by the initial posture estimation unit 22, the posture tracking processing by the image recognition means using the accurate initial value of the posture is the processing load, the accuracy of the posture estimation, Excellent recognition robustness. Therefore, according to the present invention, in the image processing apparatus according to any one of (3) to (5), when the recognition processing control unit restarts the recognition by the image recognition unit, the preview acquired by the image acquisition unit last time. The recognition result of the object relation estimation unit in the image is used as an initial value, and the posture of the image recognition unit is tracked. For this reason, the processing load can be reduced, and the accuracy of posture estimation and the robustness of recognition can be improved.

  (7) In the image processing apparatus according to any one of (3) to (6), the recognition processing control unit includes the object relation estimation unit for an object whose recognition by the image recognition unit is suspended. Based on the recognition result, the object is projected onto the preview image acquired by the image acquisition unit to create a projection image, and the similarity between the projection image and the preview image acquired by the image acquisition unit If it is less than the threshold value, an image processing apparatus is proposed in which it is determined that the verification has failed.

  According to the present invention, in any one of the image processing apparatuses according to (3) to (6), the recognition processing control means is based on the recognition result by the object relation estimation means for the object whose recognition by the image recognition means is suspended. Then, an object is projected onto the preview image acquired by the image acquisition unit to create a projection image, and if the similarity between the projection image and the preview image acquired by the image acquisition unit is less than a threshold, collation is performed. It was decided that it failed. For this reason, it can be determined whether the recognition result by the object relationship estimation means is correct.

  (8) In the image processing apparatus according to (7), the recognition processing control unit estimates the posture that maximizes the similarity by iterative calculation, and corrects the recognition result by the object relationship estimation unit. An image processing apparatus characterized by this is proposed.

  According to the present invention, in the image processing apparatus of (7), the recognition processing control unit estimates the posture with which the degree of similarity is maximized by iterative calculation, and corrects the recognition result by the object relationship estimation unit. For this reason, in the AR technique, the processing load can be further reduced, and the robustness of object recognition can be further improved.

  (9) In the image processing apparatus according to any one of (3) to (8), the recognition processing control unit includes the object relationship estimation unit for an object whose recognition by the image recognition unit is suspended. Based on the recognition result, the object is projected onto the preview image acquired by the image acquisition unit to create a projection image, and template matching between the projection image and the preview image acquired by the image acquisition unit The image processing apparatus is characterized in that the matching part is estimated by the above, and if the response value at the matching part is less than the threshold, it is determined that the matching has failed.

  According to this invention, in any one of the image processing apparatuses according to (3) to (8), the recognition processing control unit is based on the recognition result of the object relation estimation unit for the object whose recognition by the image recognition unit is suspended. Then, the object is projected onto the preview image acquired by the image acquisition unit to create a projection image, and the matching portion is estimated by template matching between the projection image and the preview image acquired by the image acquisition unit. If the response value at the location is less than the threshold, it is determined that the verification has failed. For this reason, it can be determined whether the recognition result by the object relationship estimation means is correct.

  (10) According to the present invention, for any one of (1) to (9), the recognition result of an object recognized by a first image processing device different from the image processing device is used as the image processing device. And a virtual recognition display unit (e.g., corresponding to the cooperative recognition processing unit 60 in FIG. 16), and the virtual information display unit includes the recognition result by the image recognition unit and the object relation estimation. An image processing apparatus is proposed in which virtual information is superimposed on the preview image acquired by the image acquisition unit based on the recognition result by the unit and the recognition result by the cooperative recognition processing unit.

  According to the present invention, in any one of the image processing devices (1) to (9), the recognition result of the object recognized by the first image processing device different from the image processing device is used as a reference. It was decided to provide cooperative recognition processing means for converting to the recognized result. Further, the virtual information display means adds virtual information to the preview image acquired by the image acquisition means based on the recognition result by the image recognition means, the recognition result by the object relationship estimation means, and the recognition result by the cooperative recognition processing means. It was decided to superimpose. For this reason, the recognition result can also be used by another image processing apparatus for superimposing virtual information on the preview image. Therefore, in AR technology, the processing load can be further reduced and the robustness of object recognition can be further improved. Can be.

  (11) In the image processing apparatus according to any one of (1) to (10), the object relationship estimation unit may determine the relationship between the objects as the relationship between the objects recognized by the image recognition unit. Proposed is an image processing apparatus characterized by obtaining a relative posture indicating a general positional relationship.

  According to this invention, in the image processing apparatus according to any one of (1) to (10), the relative positional relationship between the objects as the relationship between the objects recognized by the image recognition unit by the object relationship estimation unit. The relative posture indicating For this reason, it is possible to search for highly related objects such as objects that are moving in the same manner using the relative posture between the objects.

  (12) In the image processing apparatus according to any one of (1) to (11), the object relationship estimation unit may cause the object in the preview image to be acquired each time the preview image is acquired by the image acquisition unit. Relative posture between the images is obtained, and the objects whose change amount between the preview images of the relative posture is less than the threshold value are continuously classified over a predetermined number of preview images, and are classified into the same group. An image processing apparatus is proposed.

  According to the present invention, in any one of the image processing apparatuses according to (1) to (11), each time the preview image is acquired by the image acquisition unit by the object relationship estimation unit, the relative posture between the objects in the preview image is acquired. The objects whose change amount between the preview images of the relative posture is less than the threshold value are continuously classified into the same group over a predetermined number of preview images. For this reason, it is possible to classify objects in consideration of the relationship between objects in a plurality of continuous preview images.

  (13) In the image processing apparatus according to (12), the object relationship estimation unit may determine the relative posture obtained in the latest preview image acquired by the image acquisition unit and the previous preview image. An image processing apparatus is proposed in which a difference between the average of the relative postures obtained in the preview image is obtained as the amount of change.

  According to the present invention, in the image processing apparatus of (12), the relative posture determined in the latest preview image acquired by the image acquisition unit by the object relationship estimation unit and the preview image before the latest preview image. The difference between the average of the calculated relative postures and the amount of change was determined. For this reason, it is possible to classify the objects more appropriately in consideration of the relationship between the objects.

  (14) In the image processing apparatus according to any one of (1) to (13), the image recognition unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object. The object relationship estimation means proposes an image processing apparatus that determines that the relative posture between objects having a recognition accuracy index added by the image recognition means equal to or greater than a threshold value is stable. ing.

  According to the present invention, in the image processing apparatus according to any one of (1) to (13), the image recognition unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object, and the object relationship The estimation means determines that the relative posture between objects whose recognition accuracy index added by the image recognition means is equal to or greater than a threshold value is stable. For this reason, the objects can be classified in consideration of the recognition accuracy of the recognition result of the objects.

  (15) In the image processing apparatus according to any one of (1) to (14), the image recognition unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object. The object relation estimation means proposes an image processing apparatus that applies the object with the highest recognition accuracy index added by the image recognition means to the main object.

  According to the present invention, in the image processing apparatus according to any one of (1) to (14), the image recognition unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object, and the object relationship The estimation means applies the object having the highest recognition accuracy index added by the image recognition means to the main object. For this reason, since the object can be recognized by the object relationship estimation means using the relationship with the object having the highest recognition accuracy, the robustness of the object recognition can be further improved.

  (16) In the image processing device according to (14) or (15), the image recognition unit may use at least one of a shooting distance to the object and a shooting angle to the object as an index of the recognition accuracy. Has proposed an image processing device characterized by the use of.

  According to this invention, in the image processing apparatus of (14) or (15), the image recognition means uses at least one of the shooting distance to the object and the shooting angle to the object as an index of recognition accuracy. It was. For this reason, the recognition accuracy index can be set using the shooting distance to the object and the shooting angle to the object.

  (17) In the image processing device according to any one of (14) to (16), the image recognition unit may include a local feature amount matching number and a local feature amount matching as an index of the recognition accuracy. An image processing apparatus that uses at least one of a score and a score is proposed.

  According to the present invention, in the image processing device according to any one of (14) to (16), the number of local feature values matching score, the local feature value matching score, as an index of recognition accuracy by the image recognition unit, At least one of them was used. For this reason, an index of recognition accuracy can be set using the matching number of local feature quantities and the matching score of local feature quantities.

  (18) In the image processing apparatus according to any one of (14) to (17), the image recognition unit may use an SSD (Sum of Squared Difference) response value, an NCC ( There has been proposed an image processing apparatus using at least one of a response value of (Normalized Cross Correlation).

  According to this invention, in the image processing apparatus according to any one of (14) to (17), at least one of an SSD response value and an NCC response value is used as an index of recognition accuracy by the image recognition means. It was decided to use. For this reason, the index of recognition accuracy can be set using the response value of SSD or the response value of NCC.

  (19) In the image processing apparatus according to any one of (1) to (18), the image recognition unit may add information serving as an index of a processing load required for recognition of the object to the recognition result for each object. In addition, the object relationship estimation means proposes an image processing apparatus characterized in that an object having a processing load index added by the image recognition means is less than a threshold is applied to the main object.

  According to the present invention, in the image processing device according to any one of (1) to (18), the image recognition unit adds information serving as an index of the processing load required for object recognition to the recognition result for each object. The object relation estimation means applies an object whose processing load index added by the image recognition means is less than a threshold value to the main object. For this reason, since the object can be recognized by the object relationship estimating means using the relationship with the object having a low processing load, the processing load can be further reduced.

  (20) The present invention proposes the image processing apparatus according to (19), wherein the image recognition means uses a time required for recognition as an index of the processing load.

  According to the present invention, in the image processing apparatus of (19), the time required for recognition is used as an index of the processing load by the image recognition means. For this reason, it is possible to set a processing load index using the time required for recognition.

  (21) The present invention proposes an image processing apparatus according to (19), wherein the image recognition means sets a value corresponding to the type of object as an index of the processing load. ing.

  According to the present invention, in the image processing apparatus of (19), the image recognition means uses a value corresponding to the type of object as the processing load index. Therefore, it is possible to set a processing load index using a value according to the type of object.

  (22) The present invention provides image acquisition means (for example, equivalent to the image acquisition unit 10 in FIG. 1), image recognition means (for example, equivalent to the image recognition unit 20 in FIG. 1), object relationship estimation means (for example, FIG. 1). Image processing apparatus (for example, the image of FIG. 1), and virtual information display means (for example, the virtual information display unit 70 of FIG. 1), which superimposes virtual information on the preview image. Image processing method) in which the image acquisition unit acquires the preview image, and the image recognition unit includes the preview image acquired in the first step. A second step of recognizing the object, and the object relationship estimating means includes a relationship (for example, an object described later) between the objects recognized in the second step. Corresponding to the relative posture between the objects), classifying the object based on the estimation result, and based on the recognition result of the main object that is one of the objects classified into the same group based on the second step A third step of recognizing an object other than the main object classified into the group, and the virtual information display means based on the recognition result of the second step and the recognition result of the third step. And a fourth step of superimposing virtual information on the preview image acquired in the first step.

  According to the present invention, the same effects as described above can be obtained.

  (23) The present invention provides image acquisition means (for example, equivalent to the image acquisition unit 10 in FIG. 1), image recognition means (for example, equivalent to the image recognition unit 20 in FIG. 1), object relationship estimation means (for example, FIG. 1). Image processing apparatus (for example, the image of FIG. 1), and virtual information display means (for example, the virtual information display unit 70 of FIG. 1), which superimposes virtual information on the preview image. 1 is a program for causing a computer to execute an image processing method (corresponding to the processing apparatus 1), wherein the image acquisition unit acquires the preview image, and the image recognition unit includes the first step. A second step of recognizing an object in the preview image acquired in the step, and the object relationship estimating means is recognized in the second step. The relationship between objects (e.g., corresponding to a relative posture between objects described later) is estimated, the objects are classified based on the estimation result, and the main object that is one of the objects classified into the same group A third step of recognizing an object other than the main object classified into the group based on a recognition result of the second step; and the virtual information display means includes a recognition result of the second step; For causing the computer to execute a program for causing the computer to execute a fourth step of superimposing virtual information on the preview image acquired in the first step based on the recognition result of step 3 Propose a program.

  According to the present invention, the same effect as described above can be obtained by executing the program using a computer.

  According to the present invention, in the AR technology, it is possible to reduce the processing load and improve the robustness of object recognition.

1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a schematic diagram which shows the usage example of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram which shows the usage example of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram which shows the usage example of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram which shows the usage example of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a block diagram of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram of the image processing apparatus which concerns on 3rd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements, and various variations including combinations with other existing constituent elements are possible. Accordingly, the description of the following embodiments does not limit the contents of the invention described in the claims.

<First Embodiment>
[Outline of Image Processing Apparatus 1]
FIG. 1 is a block diagram of an image processing apparatus 1 according to the first embodiment of the present invention. The image processing apparatus 1 is compatible with AR technology. An outline of the image processing apparatus 1 will be described below with reference to FIGS.

  FIG. 2 is a schematic diagram illustrating an example in which the terminal 100 in which the image processing apparatus 1 is mounted is capturing from the first viewpoint. In FIG. 2, three objects M1, M2, and M3 are linearly arranged on the table AA. The terminal 100 is photographing the table AA from the object M1 side with a built-in camera.

  FIG. 3 is a diagram showing the display screen 110a of the terminal 100 in FIG. On the display screen 110a, objects M1 to M3 are displayed in the order of objects M1, M2, and M3 from the lower side (lower side in FIG. 3) to the upper side (upper side in FIG. 3). Also, virtual information C1 associated with the object M1 is superimposed on the right side of the object M1 (right side in FIG. 3). Also, virtual information C2 associated with the object M2 is superimposed on the right side of the object M2 (right side in FIG. 3). Also, virtual information C3 associated with the object M3 is superimposed on the right side of the object M3 (right side in FIG. 3). For this reason, the owner of the terminal 100 can recognize the virtual information C1 to C3 existing in the AR space through the display screen 110a.

  FIG. 4 is a schematic diagram illustrating an example in which the terminal 100 on which the image processing apparatus 1 is mounted is shooting from the second viewpoint. The terminal 100 is photographing the table AA from the object M3 side with a built-in camera.

  FIG. 5 is a diagram showing a display screen 110b of the terminal 100 in FIG. On the display screen 110b, objects M1 to M3 are displayed in the order of the objects M1, M2, and M3 from the upper side (upper side in FIG. 5) to the lower side (lower side in FIG. 5). Also, virtual information C1 associated with the object M1 is superimposed on the left side of the object M1 (left side in FIG. 5). Also, virtual information C2 associated with the object M2 is superimposed on the left side of the object M2 (left side in FIG. 5). Also, virtual information C3 associated with the object M3 is superimposed on the left side of the object M3 (left side in FIG. 5). Therefore, the owner of the terminal 100 can recognize the virtual information C1 to C3 existing in the AR space through the display screen 110b.

  On the display screen 110b, each of the virtual information C1 to C3 is displayed with the virtual information C1 to C3 displayed on the display screen 110a rotated by 180 degrees. This is because the terminal 100 displaying the display screen 110b captures each of the objects M1 to M3 from a direction 180 degrees opposite to the terminal 100 displaying the display screen 110a. is there. Therefore, when the owner of the terminal 100 rotates the viewpoint for the virtual information C1 to C3 by 180 degrees from the viewpoint in FIG. 2 through the display screen 110b, the virtual information C1 to C3 in the AR space follows the rotation of the viewpoint. Can also be recognized as being rotated 180 degrees.

  2 to 5 show the case where the terminal is moved, but virtual information also moves on the display screen of the terminal 100 following the movement of the object even when the object is moved. Note that when the terminal or the object is moved, the virtual information moves while maintaining a relative positional relationship with the object. Further, each of the virtual information C1 to C3 does not exist in the real space, but is stored in the terminal 100 in association with each of the objects M1 to M3.

  Here, the recognition accuracy of the object in the image recognition process decreases as the distance between the object and the camera increases. Further, due to occlusion or light reflection, the object may not be temporarily displayed in the preview image, and recognition of this object may temporarily fail. For this reason, for example, a situation may occur in which the object M1 can be recognized from the first viewpoint in FIG. 2, but cannot be recognized from the second viewpoint in FIG.

  Therefore, first, the case where the AR space described above with reference to FIGS. 2 to 5 is realized by the technique of Patent Document 1 described above will be described below. In this case, if the above situation occurs, the object cannot be recognized even if the posture tracking process for the object M1 is performed. Therefore, the object M1 recognition process needs to be repeated from the initial posture estimation process. However, the robustness of the initial posture estimation process with respect to the distance and angle between the object and the camera is low. For this reason, there is a possibility that the recognition of the object M1 cannot be resumed even if the initial posture estimation process is repeated.

  Next, a case where the AR space described above with reference to FIGS. 2 to 5 is realized by the image processing apparatus 1 according to the present embodiment will be described below. In this case, in order to recognize the object M1, the image processing apparatus 1 preliminarily estimates a relative posture indicating a relative positional relationship between the object M1 and another object. If the recognition of the object M1 fails, the recognition result of another object is converted into the recognition result of the object M1 using the estimated relative posture. According to this, even if the terminal 100 cannot directly recognize the object M1, the recognition result of other objects can be converted and the object M1 can be recognized. For this reason, the robustness of object recognition can be improved, and the virtual information C1 can be displayed on the display screen 110 of the terminal 100.

[Configuration of Image Processing Apparatus 1]
The above image processing apparatus 1 will be described in detail below. Returning to FIG. 1, the image processing apparatus 1 can be mounted on a stationary computer such as a desktop PC, or a portable information terminal such as a laptop PC, a mobile phone, a portable game machine, or an HMD. The image processing apparatus 1 includes an image acquisition unit 10, an image recognition unit 20, an object relationship estimation unit 30, and a virtual information display unit 70.

[Configuration and Operation of Image Acquisition Unit 10]
The image acquisition unit 10 continuously acquires images taken by an imaging device such as a WEB camera or a camera module. In the present embodiment, the image acquisition unit 10 acquires an image at a frame rate of 60 fps. Note that the imaging device that continuously captures images may be provided inside the image processing device 1 or may be provided outside the image processing device 1.

[Configuration and Operation of Image Recognition Unit 20]
The image recognition unit 20 receives an image acquired by the image acquisition unit 10 (hereinafter referred to as a preview image). The image recognition unit 20 identifies an object in the input preview image, estimates the posture of each identified object, and recognizes each identified object. The image recognition unit 20 includes an object identification unit 21, an initial posture estimation unit 22, and a posture tracking unit 23.

  The object identification unit 21 receives the preview image acquired by the image acquisition unit 10 as an input. The object identifying unit 21 performs an object identifying process in the input preview image. In the identification processing, a local feature amount is detected from the preview image, and an object is identified by comparing with a local feature amount for each object registered in advance in a feature amount database (dictionary).

  The object identification process may be performed by, for example, an external server. In this case, the object identification unit 21 transmits the preview image to the external server and receives the result of the identification process from the external server. According to this, since the identification process can be outsourced, it is suitable for handling a large-scale object or a large number of objects.

  On the other hand, when the number of objects is small, the object identification unit 21 can be omitted from the image recognition unit 20.

  The initial posture estimation unit 22 receives the preview image acquired by the image acquisition unit 10 as an input. The initial posture estimation unit 22 estimates the posture of the object identified by the object identification unit 21 included in the input preview image, and sets the estimation result as the initial value of the posture. The initial posture estimation unit 22 estimates the posture described above when the posture tracking unit 23 described later starts tracking the posture of the object and when the posture tracking unit 23 stops tracking the posture of the object. .

  In the present embodiment, the posture of the object is expressed by a posture matrix of 6 degrees of freedom (4 rows and 4 columns). The posture matrix has information indicating the relative positional relationship between the imaging device that captures the preview image acquired by the image acquisition unit 10 and the object, and belongs to the three-dimensional special Euclidean group SE (3). Both are represented by a three-dimensional rotation matrix with three degrees of freedom and a three-dimensional translation vector. When the posture matrix is used, the relationship between the pixel coordinates of the object in the preview image and the coordinates on the object registered in advance in the initial posture estimation unit 22 can be expressed by the following formula (1).

  In Equation (1), A indicates an internal parameter of the imaging device. It is preferable that the internal parameters of the imaging apparatus are obtained in advance by camera calibration. However, even if the internal parameters of the imaging apparatus deviate from the actual values, they cancel each other out with the estimated posture matrix, so that the position where the virtual information is superimposed is not affected. For this reason, a general camera internal parameter can be substituted for the internal parameter of the imaging apparatus.

  In Expression (1), R represents a parameter representing rotation in the three-dimensional space. Each parameter in R can be expressed by three parameters by expression such as Euler angle.

  In Equation (1), t represents a parameter representing the parallel movement in the three-dimensional space. Each of X, Y, and Z represents an X coordinate, a Y coordinate, and a Z coordinate on the object registered in advance in the initial posture estimation unit 22. U and v represent the u coordinate and the v coordinate in the preview image.

  In the present embodiment, the posture matrix is estimated using natural features in the image. A natural feature is a feature that is calculated from a local region of an image in order to obtain or match a point correspondence between images. From a local region that can be easily matched, such as an edge or a corner in the image. Extracted. Typical examples of natural features include local feature quantities that can be associated with high accuracy, such as SIFT (Scale Invariant Feature Transform) and SURF (Speed Up Robust Features), and a method of calculating a posture matrix using these features Is generally known.

  The posture of the object changes every moment in the preview image continuously acquired by the image acquisition unit 10 as the object and the imaging apparatus move. For this reason, the initial posture estimation unit 22 is required to have a processing speed as compared with the object identification unit 21 described above. Therefore, the image acquisition unit 10 needs to be provided inside the image processing apparatus 1, and it is desirable to use an algorithm with a small processing load as disclosed in Non-Patent Document 2.

  The posture tracking unit 23 receives the preview image acquired by the image acquisition unit 10 and the initial value of the object posture estimated by the initial posture estimation unit 22 as inputs. The attitude tracking unit 23 performs object attitude tracking processing based on the input preview image and the initial value of the object attitude, estimates the object attitude, and recognizes the object.

  When the posture tracking unit 23 succeeds in tracking the posture of the object, that is, when the recognition of the object is successful, the identifier (ID) of the object that has been successfully recognized, the estimated value of the posture of the object that has been successfully recognized, Are output as recognition results. Further, the recognition result is used as an initial value when the tracking process is performed on the preview image of the next frame acquired by the image acquisition unit 10. For this reason, while the tracking of the posture of the object is successful, it is not necessary to perform processing by the initial posture estimation unit 22 on the object.

  Further, while the tracking of the posture of the object is successful, it is necessary to perform tracking processing for the object every time the preview image is acquired by the image acquisition unit 10. For this reason, the posture tracking unit 23 is required to have a processing speed as compared with the above-described initial posture estimation unit 22. Therefore, the posture tracking unit 23 needs to be provided inside the image processing apparatus 1 and must be able to perform tracking processing of the posture of the object in real time at least, and is disclosed in Non-Patent Document 2. Thus, it is desirable to use a posture tracking algorithm with a small processing load.

  In the method disclosed in Non-Patent Document 2, a predicted value of posture is estimated by linear prediction from an initial value of posture, and a movement amount of a feature point in an object is estimated by searching from the predicted value. The search is performed by obtaining a point having the highest correlation value of the local region from around the feature point. The estimated value of the posture is updated from the predicted value so as to satisfy the above formula (1) from the movement amount of each feature point. When posture tracking processing is repeatedly performed in real time, the amount of movement of feature points between frames is small. Therefore, the processing load can be reduced by limiting the search range of feature points.

  Further, when the movement amount of the feature point exceeds the search width due to the rapid movement of the camera or the object, the posture tracking fails in the above-described method. In this case, the above-described method can detect that the tracking has failed based on the feature point search status. For example, if the number of feature points that succeeded in the search (where the correlation value is greater than or equal to the threshold value) is less than or equal to the threshold value, the update of the estimated posture value is interrupted, and the tracking failure It is possible to output the indicated signal. When a signal indicating that the tracking has failed is output, the tracking processing by the posture tracking unit 23 may be interrupted, and processing for estimating the initial value of the posture by the initial posture estimation unit 22 may be executed in and after the next frame.

  In order to perform tracking processing of the posture of an object in real time, an upper limit of the number of objects that can be tracked may be set in advance, and tracking processing may not be performed for an object that exceeds the upper limit. According to this, when the number of objects to be tracked has reached the upper limit, the identification processing by the object identification unit 21 and the processing for estimating the initial value of the posture by the initial posture estimation unit 22 are suspended. . Note that the above upper limit is preferably set according to the processing capability of the image processing apparatus 1.

  The above image recognition unit 20 performs the above-described estimation of the posture of the object for each object. Since the posture estimation processing for each object is independent of each other, it may be performed in parallel or sequentially.

  When virtual information is fixedly arranged and superimposed in the AR space, the image recognition unit 20 recognizes a reference marker in addition to recognizing an object. When the reference marker can be recognized by the same processing as that for recognizing the object, the image recognition unit 20 performs recognition without distinguishing between the object and the reference marker. On the other hand, when the reference marker is an AR marker that can be recognized by the method of Non-Patent Document 1 or when it is a restored space that can be recognized by the method of Non-Patent Document 3, the reference marker is distinguished from an object. Only the reference marker is recognized by the corresponding recognition method. When there is no virtual information that is fixedly arranged and superimposed in the AR space, or when there is no reference marker in the first place, the image recognition unit 20 performs only object recognition.

  In any case, what the image recognition unit 20 performs is the estimation of the posture of the object (and the reference marker if it exists). In addition, the recognition method used for the presence / absence of the reference marker, the type of the reference marker, and the posture is not limited to the above-described method.

[Configuration and Operation of Object Relationship Estimation Unit 30]
The object relationship estimation unit 30 receives the recognition result from the image recognition unit 20 as an input. The object relationship estimation unit 30 estimates the relationship between objects recognized by the image recognition unit 20, and classifies the objects based on the estimation result. Further, an object that has failed to be recognized is recognized based on the recognition result by the image recognition unit 20 of the main object that is one of the objects classified into the same group as this object. The object relationship estimation unit 30 includes an object relationship estimation processing unit 31 and a posture conversion processing unit 32.

  The object relationship estimation processing unit 31 receives the recognition result from the image recognition unit 20 as an input. The object relationship estimation processing unit 31 estimates and classifies the relationship between objects recognized by the image recognition unit 20. Specifically, a relative posture between the objects recognized by the image recognition unit 20 is calculated, and each object is classified according to a change in the relative posture. In the present embodiment, the relative posture is also expressed by the above-described posture matrix. The object relationship estimation processing unit 31 will be described in detail below.

The object relationship estimation processing unit 31 operates when the number of objects recognized by the image recognition unit 20 is two or more. Here, for example, the posture matrix of the object A at the first viewpoint is the posture matrix W _A1, and the posture matrix of the object B at the first viewpoint is the posture matrix W _B1 . Then, relative orientation W _AB between objects A and B can be obtained by the following equation (2).

Here, if both the objects A and B are fixedly arranged and do not move while the camera moves and moves from the first viewpoint to the second viewpoint, the following formula (3) relative orientation as shown in W _AB does not change. Also, the object A, even if both the B is fixedly disposed on the same rigid object, even moving rigid object, relative orientation W _AB does not change.

  Therefore, the object relationship estimation processing unit 31 first estimates the relative posture between the objects in which the preview image acquired by the image acquisition unit 10 is recognized by the image recognition unit 20 for each frame. Next, for each object, a change amount of the relative posture between frames is obtained as a relative posture change amount, and it is determined that the relative posture is stable for one frame between objects whose relative posture change amount is less than the threshold value β. Next, these objects are classified into the same group between objects whose relative postures are continuously stable over α frame or more.

  The above-described relative posture change amount is obtained as follows. When it is determined that the relative posture is not stable in the previous frame, a change amount from the relative posture estimated in the previous frame to the relative posture estimated in the current frame is obtained as the above-described relative posture change amount. On the other hand, when it is determined that the relative posture is continuously stable over n frames (where n is an arbitrary integer satisfying n> 1), the interval between n frames before and the previous frame is determined. The average of the estimated relative postures (average relative posture) is obtained, and the difference between the average relative posture and the relative posture estimated in the current frame is obtained as the above-described relative posture change amount.

  The relative posture change amount described above may be divided into translation and rotation, and may be compared with independent threshold values.

Further, at the stage when the object relationship estimation processing unit 31 starts operation, that is, in the initial state of grouping, all objects recognized by the image recognition unit 20 belong to different groups (number of objects = groups). Number). In addition, it is possible to classify three or more objects into the same group. For example, the relative posture W _AB between the object A and the object B and the relative posture W between the object A and the object C can be classified. _{When AC} is continuously stable over α frames or more, the three objects A, B, and C can be classified into the same group.

  Further, when it is determined that the relative posture between the objects classified into the same group is not stable, the grouping of these objects is canceled and these objects are divided into different groups. As a case where the grouping of objects is canceled, for example, a case where at least one of these objects moves is assumed.

  In addition, recognition of this object may temporarily fail due to occlusion or light reflection. In such a case, it is impossible to estimate the relative posture between objects, but it is assumed that the relative posture between objects classified in the same group has not changed. Do not do.

  However, if frames in which the relative posture between objects cannot be estimated continue to occur, it cannot be guaranteed that the relative posture has not changed (relative posture invariance). In addition, when the reason why the relative posture between objects cannot be estimated is a temporary cause such as occlusion or light reflection, it is considered that this cause can be resolved and the estimation of the relative posture between objects can be resumed. Therefore, when the relative posture between the objects cannot be estimated continuously over the θ frame or more, the grouping is canceled.

  Here, the relative posture between the objects that cannot be recognized cannot be estimated. That is, for an object that cannot be recognized, the relative posture between this object and another object cannot be estimated. For this reason, for objects that cannot be recognized continuously over the θ frame or more, the relative posture with other objects cannot be continuously estimated over the θ frame or more, and the grouping is canceled. Will be.

  The posture conversion processing unit 32 receives the recognition result by the image recognition unit 20 and the classification result and the relative posture estimation result by the object relationship estimation processing unit 31 as inputs. The posture conversion processing unit 32 recognizes the recognition result by the image recognition unit 20 for other objects classified into the same group with respect to the object that has failed to be recognized by the image recognition unit 20, between the other objects. Recognize using relative posture. The posture conversion processing unit 32 will be described in detail below.

The posture conversion processing unit 32 operates when there are one or more sets of objects classified into the same group, that is, when two or more objects are classified into the same group. Here, for example, the image recognition unit 20 has failed to recognize the object E, and the image recognition unit 20 has succeeded in recognizing the object F classified into the same group as the object E. It is assumed that the relative posture W _EF between E and the object F is estimated. Further, the recognition has succeeded by the image recognition unit 20, it is assumed that the orientation matrix W _F1 objects F in a first aspect is obtained. Then, with the object F as the main object described above, the posture matrix W _E1 of the object E at the first viewpoint can be obtained by the following equation (4).

[Configuration and Operation of Virtual Information Display Unit 70]
The virtual information display unit 70 receives as input the preview image acquired by the image acquisition unit 10, the recognition result by the image recognition unit 20, and the recognition result by the object relationship estimation unit 30. The virtual information display unit 70 superimposes virtual information on the preview image based on the recognition results by the image recognition unit 20 and the object relationship estimation unit 30. When superimposing virtual information, the virtual information display unit 70 includes an internal parameter matrix (including information such as an angle of view) of the imaging device and an attitude matrix of an object associated with the virtual information to be superimposed. Used to superimpose this virtual information at the corresponding position by 3D rendering. Moreover, when superimposing virtual information, the virtual information display part 70 correct | amends the position and direction of virtual information based on an integrated recognition result.

  The virtual information display unit 70 displays images from an external monitor connected to the terminal via a wired cable or a wireless network, a display (including a retina projection type) mounted on the terminal, a projector, or the like. A display device for posting to a user is controlled. If this display device is an optical see-through type HMD or a projector that superimposes additional information directly on the field of view, only the virtual information may be displayed without displaying the preview image. .

[Operation of Image Processing Apparatus 1]
The operation of the image processing apparatus 1 having the above configuration will be described below with reference to FIGS.

  FIG. 6 is a flowchart of the image processing apparatus 1.

  In step S100, the image processing apparatus 1 acquires a preview image by the image acquisition unit 10, and proceeds to step S101.

  In step S101, the image processing apparatus 1 performs a first image recognition process using the image recognition unit 20 and the object relationship estimation processing unit 31, recognizes each object in the preview image acquired in step S100, and proceeds to step S102. Move processing. Details of the first image recognition process will be described later with reference to FIG.

  In step S102, the image processing apparatus 1 causes the object relationship estimation processing unit 31 to select two objects from all the objects recognized in the current frame in step S101, and moves the process to step S103.

  In step S103, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to determine whether or not the counter value of the relative posture continuous estimation number counter for the two objects selected in step S102 or step S107 is zero. If it is determined that the value is zero, the process proceeds to step S104. If it is determined that the value is not zero, the process proceeds to step S105. Note that the relative posture continuous estimation number counter is provided for each of the two objects selected in step S102 or step S107, and counts how many frames the relative posture is continuously obtained for each of the two objects. Is.

  In step S104, the image processing apparatus 1 performs a relative orientation calculation process using the object relationship estimation processing unit 31, and the process proceeds to step S106. The details of the relative posture calculation process will be described later with reference to FIG.

  In step S105, the image processing apparatus 1 performs a classification process by the object relationship estimation processing unit 31, and moves the process to step S106. Details of the classification process will be described later with reference to FIG.

  In step S106, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to determine whether all the combinations recognized in the current frame in step S101 are selected as one set of two in the current frame. Is determined. If it is determined that it has been selected, the process proceeds to step S108. If it is determined that it has not been selected, the process proceeds to step S107.

  In step S107, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to select two objects constituting a combination that has not been selected among all objects recognized in the current frame in step S101, and then proceeds to step S103. Return processing.

  In step S108, the image processing apparatus 1 performs the first superimposed display process using the orientation conversion processing unit 32 and the virtual information display unit 70, and ends the process illustrated in FIG. Details of the first superimposed display process will be described later with reference to FIG.

  FIG. 7 is a flowchart of the first image recognition process described above performed by the image processing apparatus 1.

  In step S110, the image processing apparatus 1 uses the posture tracking unit 23 to determine whether or not the preview image acquired in step S100 includes the object recognized in the previous frame. If it is determined that it is included, the process proceeds to step S111. If it is determined that it is not included, the process proceeds to step S116.

  In step S111, the image processing apparatus 1 recognizes each object determined by the posture tracking unit 23 as having been recognized in the previous frame in step S110 by performing posture tracking processing using the posture in the previous frame as an initial value, The process moves to step S112.

  In step S112, the image processing apparatus 1 causes the object relationship estimation processing unit 31 to reset the corresponding posture tracking consecutive failure number counter for each object whose posture has been tracked in the current frame in step S111, and performs the process in step S113. Move. According to this, the posture tracking consecutive failure number counter for the object recognized in the current frame among all the objects included in the preview image acquired by the image acquisition unit 10 is reset. The posture tracking continuous failure count counter is provided for each object, and counts the number of frames for which posture tracking has failed continuously for each object.

  In step S113, the image processing apparatus 1 causes the object relationship estimation processing unit 31 to increment the corresponding posture tracking consecutive failure number counter for each object whose posture could not be tracked in the current frame in step S111, and the process proceeds to step S114. Move. According to this, among all the objects included in the preview image acquired by the image acquisition unit 10, the posture tracking continuous failure number counter for the object that could not be recognized in the current frame is incremented by one. Become.

  In step S114, the image processing apparatus 1 causes the object relationship estimation processing unit 31 to reset the posture tracking continuous failure count counter and cancel grouping for objects whose posture tracking continuous failure count counter is equal to or greater than the threshold θ. Then, the process proceeds to step S115. According to this, among the all objects included in the preview image acquired by the image acquisition unit 10, the posture tracking consecutive failure counter is reset for objects that could not be recognized continuously over the θ frame or more. At the same time, the grouping is canceled.

  In step S115, the image processing apparatus 1 determines whether or not the number of objects being tracked has reached a predetermined upper limit value by the posture tracking unit 23. If it has reached, the process shown in FIG. 7 is terminated. If it has not reached, the process proceeds to step S116.

  In step S116, the image processing apparatus 1 uses the object identification unit 21 to identify the object in the preview image acquired in step S100, and the process proceeds to step S117.

  In step S117, the image processing apparatus 1 uses the initial posture estimation unit 22 to recognize each of the objects not recognized in the previous frame among the objects identified in step S116 included in the preview image acquired in step S110, and the current frame. The initial value of the posture is obtained and recognized for each of the objects for which tracking of the posture has failed, and the process proceeds to step S118.

  In step S118, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to newly provide a posture tracking continuous failure counter for each object for which the initial value of the posture has been obtained in step S117. The continuous failure frequency counter is reset, and the process shown in FIG.

  FIG. 8 is a flowchart of the above-described first relative posture calculation process performed by the image processing apparatus 1.

  In step S120, the image processing apparatus 1 obtains the relative posture in the current frame between the two objects selected in step S102 or step S107 by the object relationship estimation processing unit 31, and ends the process shown in FIG.

  FIG. 9 is a flowchart of the above-described classification process performed by the image processing apparatus 1.

  In step S130, the image processing apparatus 1 obtains the relative posture in the current frame between the two objects selected in step S102 or step S107 by the object relationship estimation processing unit 31, and moves the process to step S131.

  In step S131, the image processing apparatus 1 causes the object relationship estimation processing unit 31 to increment the counter value of the relative posture continuous estimation number counter to “1” for the relative posture between the two objects selected in step S102 or step S107. The average value of those estimated in each frame from the obtained frame (hereinafter referred to as the average relative posture estimation start frame) to the previous frame is obtained as the average relative posture, and the process proceeds to step S132.

The average relative posture is calculated from the average relative posture calculated in the previous frame (average relative posture from the average relative posture estimation start frame to the frame two frames before) and the relative posture W _N−1 calculated in the previous frame. It can be obtained as a weighted average. The average relative attitude calculated in the previous frame can be obtained by the following equation (5), and the weighted average can be obtained by the following equation (6). In the above-described relative posture W _N-1 and equations (5) and (6), N represents the counter value of the relative posture continuous estimation number counter when estimating the average relative posture.

  In step S132, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to calculate the absolute difference between the average relative posture and the relative posture estimated in the current frame between the two objects selected in step S102 or step S107. The value is obtained as a relative posture change amount, and the process proceeds to step S133.

  In step S133, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to determine whether or not the relative posture change amount obtained in step S132 is less than the threshold value β. If it is determined that the relative posture change amount is less than the threshold value β, the process proceeds to step S134. If the relative posture change amount is equal to or greater than the threshold value β, the process proceeds to step S137.

  In step S134, the image processing apparatus 1 increments the relative posture continuous estimation number counter for the two objects that have been determined in step S133 that the relative posture change amount is less than the threshold β by the object relationship estimation processing unit 31, and step S135. Move processing to.

  In step S135, the image processing apparatus 1 determines that the counter value of the relative orientation continuous estimation number counter for the two objects determined by the object relationship estimation processing unit 31 in step S133 that the relative orientation change amount is less than the threshold value β is the threshold value α. It is determined whether or not this is the case. If it is determined that the counter value is greater than or equal to the threshold value α, the process proceeds to step S136. If it is determined that the counter value is less than the threshold value α, the process illustrated in FIG. 9 ends.

  In step S136, the image processing apparatus 1 classifies the two objects whose counter value is determined to be greater than or equal to the threshold value α in step S135 by the object relationship estimation processing unit 31 into the same group, and performs the processing illustrated in FIG. Exit.

  In step S137, the image processing apparatus 1 uses the object relationship estimation processing unit 31 to reset the relative posture continuous estimation number counter for the two objects that have been determined in step S133 that the relative posture change amount is equal to or greater than the threshold value β, step S138. Move processing to.

  In step S138, the image processing apparatus 1 determines whether or not the two objects determined by the object relationship estimation processing unit 31 that the relative posture change amount is equal to or larger than the threshold β in step S133 are classified into the same group. Determine. If it is determined that they are classified into the same group, the process proceeds to step S139. If it is determined that they are not classified into the same group, the process shown in FIG. 9 is terminated.

  In step S139, the image processing apparatus 1 cancels that the two objects determined in step S133 that the relative posture change amount is equal to or larger than the threshold value β are classified into the same group, and performs the processing illustrated in FIG. Exit.

  FIG. 10 is a flowchart of the above-described first superimposed display process performed by the image processing apparatus 1.

  In step S <b> 140, the image processing apparatus 1 determines whether or not the object recognized in the previous frame includes an object (disappeared object) not recognized in the current frame by the posture conversion processing unit 32. If it is determined that it exists, the process proceeds to step S141. If it is determined that it does not exist, the process proceeds to step S143.

  In step S <b> 141, the image processing apparatus 1 determines whether or not the posture conversion processing unit 32 includes an object that has been classified into the same group in the previous frame as the lost object. If it is determined that it exists, the process proceeds to step S142. If it is determined that it does not exist, the process proceeds to step S143.

  In step S142, the image processing apparatus 1 uses the posture conversion processing unit 32 to determine the posture in the current frame of the object (the main object described above) that has been classified in the same group in the previous frame as the lost object in step S141. Using the relative posture estimated in the previous frame between the lost object and the main object, the lost object is converted into the posture in the current frame, and the process proceeds to step S143.

  In step S143, the image processing apparatus 1 uses the virtual information display unit 70 to add virtual information to the preview image acquired in step S100 using the recognition result in step S111 or step S117 and the posture estimated in step S142. The process shown in FIG.

  According to the image processing apparatus 1 described above, the following effects can be obtained.

  When there is an object that cannot be recognized by the image recognition unit 20, the image processing apparatus 1 is classified into the same group as the object by the object relationship estimation unit 30 and can be recognized by the image recognition unit 20. Is applied as a main object, and an object that cannot be recognized by the image recognition unit 20 is recognized based on the recognition result of the main object. For this reason, an object that could not be recognized by the image recognition unit 20 can be recognized based on the recognition result of an object that is highly related to the object. Therefore, since the object that could not be recognized by the image recognition unit 20 can be recognized, the robustness of object recognition can be improved in the AR technology.

  In the image processing apparatus 1, the object relationship estimation unit 30 obtains the relative posture between the objects as the relationship between the objects recognized by the image recognition unit 20. For this reason, it is possible to search for highly related objects such as objects that are moving in the same manner using the relative posture between the objects.

  In addition, the image processing apparatus 1 obtains the relative posture between objects in the preview image every time the preview image is acquired by the image acquisition unit 10 by the object relationship estimation unit 30, and continuously over the α frame, Objects whose relative posture changes between preview images are less than the threshold are classified into the same group. For this reason, it is possible to classify objects in consideration of the relationship between objects in a plurality of continuous preview images.

  In addition, the image processing apparatus 1 uses the object relationship estimation unit 30 to calculate the relative posture obtained in the latest preview image acquired by the image acquisition unit 10 and the relative posture obtained in the preview image before the latest preview image. The difference from the average is obtained as the amount of change. For this reason, it is possible to classify the objects more appropriately in consideration of the relationship between the objects.

Further, the image processing apparatus 1 obtains the average relative posture as a weighted average from the average relative posture calculated in the previous frame and the relative posture W _N−1 calculated in the previous frame. Therefore, the average relative attitude of the current frame is updated using the average relative attitude calculated in the previous frame. Therefore, each time a frame is switched, the processing load is reduced compared to the case where the average relative posture is calculated by recalculating the average value of the relative posture estimated from each frame from the average relative posture estimation start frame to the previous frame. In addition, since it is not necessary to store the average attitude calculated in each frame, the required memory capacity can be reduced.

Second Embodiment
[Outline of Image Processing Apparatus 1A]
FIG. 11 is a block diagram of an image processing apparatus 1A according to the second embodiment of the present invention. The image processing apparatus 1A is different from the image processing apparatus 1 according to the first embodiment of the present invention shown in FIG. In the image processing apparatus 1A, the same components as those of the image processing apparatus 1 are denoted by the same reference numerals, and the description thereof is omitted.

  Here, first, a case where the AR space described above with reference to FIGS. 2 to 5 is realized by the technique of Patent Document 1 described above will be described below. In this case, the terminal 100 needs to continue to recognize each of the objects M1 to M3 in real time, which increases the processing load.

  Next, the case where the AR space described above with reference to FIGS. 2 to 5 is realized by the image processing apparatus 1A according to the present embodiment will be described below. In this case, if the terminal 100 classifies the objects M1 to M3 into the same group, for example, if only one of the objects M1 to M3 is recognized by the tracking process, the relative attitude is set for the other two. Can be recognized. According to this, since the number of objects that the terminal 100 has to perform the tracking process by the posture tracking unit 23 decreases, the processing load on the terminal 100 can be reduced.

[Configuration of Image Processing Apparatus 1A]
The above image processing apparatus 1A will be described in detail below.

[Configuration and Operation of Recognition Processing Control Unit 40]
Returning to FIG. 11, the recognition processing control unit 40 provided in the image processing apparatus 1 </ b> A receives the object classification result by the object relationship estimation processing unit 31 and the recognition result by the image recognition unit 20. The recognition processing control unit 40 includes a recognition processing control processing unit 41 and a posture matching processing unit 42.

  The recognition processing control processing unit 41 receives the classification result of the object by the object relationship estimation processing unit 31 and the recognition result by the image recognition unit 20 as inputs. When there are two or more objects that have been successfully recognized among the objects classified into the same group, the recognition process control processing unit 41 selects one of these objects that have been successfully recognized. Register as a main object and register the rest as recognition pause objects. In addition, the posture tracking process by the posture tracking unit 23 is paused for the recognition pause object.

  Here, if the object recognition failure due to occlusion or light reflection is temporary, the posture tracking unit 23 succeeds in the posture tracking processing when the object recognition failure is resolved. It is assumed that In general, the posture tracking process by the posture tracking unit 23 using the accurate initial value of the posture is more accurate than the estimation of the initial value of the posture by the initial posture estimation unit 22. Excellent recognition robustness. Therefore, the object that has failed to be recognized by the image recognizing unit 20 is redoed from the initial posture estimation by the initial posture estimating unit 22, but the object that has failed to be recognized by the image recognizing unit 20 is the main object. If they are classified into the same group, the recognition processing control processing unit 41 obtains the posture of the object that has failed to be recognized by the image recognition unit 20 by the posture conversion processing unit 32, and uses the result as the initial value of the posture. The tracking process by the attitude tracking unit 23 is performed.

  The posture collation processing unit 42 receives the preview image acquired by the image acquisition unit 10 and the recognition result of the recognition pause object obtained by the conversion by the posture conversion processing unit 32 as inputs. Here, for the recognition pause object, the posture tracking process by the posture tracking unit 23 is paused, so that the relative posture with respect to the main object cannot be estimated. For this reason, when the recognition pause object moves, the relative posture cannot be appropriately updated. As a result, the recognition result of the recognition pause object obtained by the conversion by the posture conversion processing unit 32 is not appropriate, and the recognition pause object is recognized. It becomes impossible to superimpose virtual information at the correct position with respect to the pause object. Therefore, the posture matching processing unit 42 checks whether the recognition result obtained by the conversion of the posture conversion processing unit 32 is correct, that is, whether the relative posture is fluctuated, for the recognition pause object.

  Specifically, the posture collation processing unit 42 collates postures at high speed using a technique for evaluating the similarity of images such as SSD (Sum of Squared Difference) and NCC (Normalized Cross Correlation). At this time, the reference model of the object used by the image recognition unit 20 (an image in the case of a planar object, a 3D model in the case of a three-dimensional object) is projected with a recognition result obtained by conversion by the posture conversion processing unit 32. A template image is generated, and the similarity is evaluated by matching with the preview image. If the recognition result obtained by the conversion by the posture conversion processing unit 32 is correct, it is assumed that the response value (similarity) of the SSD or NCC is high. If it falls below γ, this object is excluded from the recognition pause object.

  In addition, when the error in the relative posture or the posture of the main object is large, the recognition result obtained by the conversion by the posture conversion processing unit 32 may include an error. Therefore, the posture collation processing unit 42 estimates the location with the highest similarity by sliding the template image vertically and horizontally within a certain range when evaluating the similarity of images such as SSD and NCC. The response value at the location with the highest similarity may be used as the similarity.

  In addition, in order to correct an error included in the recognition result obtained by the conversion by the posture conversion processing unit 32, the posture matching processing unit 42 performs the SSD in the posture when evaluating the similarity of images such as SSD and NCC. Alternatively, the posture that maximizes the similarity may be estimated by calculating the Jacobian of NCC and iterative calculation. For example, Non-Patent Document 4 discloses a method (ESM: Efficient Second Order Minimization) for estimating an attitude for minimizing SSD with respect to a planar object by efficient iterative calculation. In addition, by using the posture obtained by the posture conversion processing unit 32, the posture can be minimized with a small number of iterations, and the AR technology can further reduce the processing load and further improve the robustness of object recognition. can do.

[Operation of Image Processing Apparatus 1A]
The operation of the image processing apparatus 1A having the above configuration will be described below with reference to FIGS.

  FIG. 12 is a flowchart of the image processing apparatus 1A.

  In step S200, the image processing apparatus 1A acquires a preview image by the image acquisition unit 10, and moves the process to step S201.

  In step S201, the image processing apparatus 1A performs a second image recognition process using the image recognition unit 20, the object relationship estimation processing unit 31, and the recognition process control unit 40, and each object in the preview image acquired in step S200. Is recognized, and the process proceeds to step S202. Details of the second image recognition process will be described later with reference to FIG.

  In step S202, the image processing apparatus 1A causes the object relationship estimation processing unit 31 to select one object registered in the main object in the previous frame from all the objects recognized in the current frame in step S201, and In step S203, one object that is not classified into the same group as other objects is selected, and the process proceeds to step S203.

  In step S203, the image processing apparatus 1A causes the object relationship estimation processing unit 31 to determine whether or not the counter value of the relative posture continuous estimation number counter for the two objects selected in step S202 or step S207 is zero. If it is determined that it is zero, the process proceeds to step S204, and if it is determined that it is not zero, the process proceeds to step S205.

  In step S <b> 204, the image processing apparatus 1 </ b> A performs the relative orientation calculation process illustrated in FIG. 8 by the object relationship estimation processing unit 31, and moves the process to step S <b> 206.

  In step S <b> 205, the image processing apparatus 1 </ b> A performs the classification process illustrated in FIG. 9 using the object relationship estimation processing unit 31, and moves the process to step S <b> 206.

  In step S206, the image processing apparatus 1A causes the object relationship estimation processing unit 31 to recognize among the objects recognized in the current frame in step S201, the object registered in the main object in the previous frame, and other objects in the previous frame. It is determined whether or not all combinations of objects not classified in the same group have been selected in the current frame. If it is determined that it has been selected, the process proceeds to step S208. If it is determined that it has not been selected, the process proceeds to step S207.

  In step S207, the image processing apparatus 1A causes the object relationship estimation processing unit 31 to recognize among the objects recognized in the current frame in step S201, the object registered in the main object in the previous frame, and other objects in the previous frame. For objects that are not classified into the same group, two objects that constitute a combination that has not been selected are selected, and the process returns to step S203.

  In step S208, the image processing apparatus 1A performs the second superimposed display process using the orientation conversion processing unit 32 and the virtual information display unit 70, and ends the process illustrated in FIG. Details of the second superimposed display process will be described later with reference to FIG.

  13 and 14 are flowcharts of the above-described second image recognition process performed by the image processing apparatus 1A.

  In step S210, the image processing apparatus 1A determines whether or not the preview image acquired in step S200 includes the object recognized in the previous frame by the posture tracking unit 23. If it is determined that it is included, the process proceeds to step S211. If it is determined that it is not included, the process proceeds to step S225.

  In step S211, the image processing apparatus 1A includes two or more objects classified into the same group and successfully recognized in the previous frame in the preview image acquired in step S200 by the recognition processing control processing unit 41. It is determined whether or not. If it is determined that it exists, the process proceeds to step S212. If it is determined that it does not exist, the process proceeds to step S220.

  In step S212, the image processing apparatus 1A registers one of the two or more objects determined to have been successfully recognized in step S211 by the recognition processing control processing unit 41 as a main object, and the rest as a recognition pause object. And the process proceeds to step S213.

  In step S213, the image processing apparatus 1A causes the posture tracking unit 23 to recognize and recognize each object registered in the main object in step S212 by performing posture tracking processing using the posture in the previous frame as an initial value, and then proceeds to step S214. Move processing.

  In step S214, the image processing apparatus 1A uses the recognition process control processing unit 41 to determine whether there is a main object that has failed to be tracked in step S213. If it is determined that there is, the process proceeds to step S215. If it is determined that there is not, the process proceeds to step S220.

  In step S215, the image processing apparatus 1A causes the recognition process control processing unit 41 to exclude all objects classified into the same group as the main object that failed to be tracked in step S213 from the recognition pause object, and then proceeds to step S216. Move processing.

  In step S216, the image processing apparatus 1A determines whether or not there is a main object that has been successfully tracked in step S213 by the posture matching processing unit 42. If it is determined that there is, the process proceeds to step S217, and if it is determined that there is no, the process proceeds to step S220.

  In step S217, the image processing apparatus 1A causes the posture conversion processing unit 32 to convert the posture of the main object that has been successfully tracked in step S213 into the posture of the recognition paused object classified in the same group, and the posture matching process. The converted posture is collated by the unit 42, and the process proceeds to step S218.

  In step S218, the image processing apparatus 1A causes the posture matching processing unit 42 to determine whether there is an object for which the matching in step S217 has failed, that is, whether there is an object whose similarity obtained by the matching is lower than the threshold γ. Is determined. If it is determined that there is, the process proceeds to step S219. If it is determined that there is not, the process proceeds to step S220.

  In step S219, the image processing apparatus 1A causes the posture matching processing unit 42 to exclude the object that failed to be matched in step S218 from the recognition pause object, and moves the process to step S220.

  In step S220, the image processing apparatus 1A uses the posture tracking unit 23 to recognize an object that has been recognized in the previous frame but is not classified into another group in the same group, and an object that has been excluded from the recognition pause object in step S219. The posture is recognized by performing a tracking process with the posture in the previous frame as an initial value, and the process proceeds to step S221.

  In each of steps S221 to S227, the image processing apparatus 1A performs the same process as the process performed by the image processing apparatus 1 in each of steps S112 to S118 of FIG.

  FIG. 15 is a flowchart of the above-described second superimposed display process performed by the image processing apparatus 1A.

  In step S230, the image processing apparatus 1A uses the virtual information display unit 70 to add virtual information to the preview image acquired in step S200 using the recognition result in step S213 or step S220 and the posture estimated in step S142. The process shown in FIG. 15 is terminated.

  According to the above image processing apparatus 1A, in addition to the above-described effects that the image processing apparatus 1 can exhibit, the following effects can be achieved.

  In the image processing apparatus 1A, the object relationship estimation unit 30 selects one object from each group as a main object, and the recognition processing control unit 40 selects objects other than the main object classified into the same group as the main object. The recognition by the image recognition unit 20 is suspended. Further, the object relation estimation unit 30 recognizes the object whose recognition by the image recognition unit 20 is suspended based on the recognition result of the main object. For this reason, objects other than the main object classified into the same group as the main object can be recognized by the object relationship estimation unit 30 even if the recognition by the image recognition unit 20 is suspended. The number of objects to be reduced can be reduced. Therefore, the processing load can be reduced in the AR technology.

  In addition, the image processing apparatus 1 </ b> A uses the recognition processing control unit 40 to recognize the recognition result by the object relationship estimation unit 30 for the object for which the recognition by the image recognition unit 20 is suspended, as the preview image acquired by the image acquisition unit 10. If the collation fails and the collation fails, the recognition of the object by the image recognition unit 20 is resumed. For this reason, it can be determined whether the recognition result by the object relationship estimation part 30 is correct.

  In addition, when the recognition processing control unit 40 restarts the recognition by the image recognition unit 20, the image processing apparatus 1A uses the recognition result by the object relationship estimation unit 30 in the previous frame as an initial value by the image acquisition unit 10 to perform image recognition. The unit 20 is made to track the posture. For this reason, the processing load can be reduced, and the accuracy of posture estimation and the robustness of recognition can be improved.

  In the image processing apparatus 1 </ b> A, the object is acquired by the image acquisition unit 10 by the recognition processing control unit 40 based on the recognition result by the object relationship estimation unit 30 for the object whose recognition by the image recognition unit 20 is suspended. The projected image is projected to create a projected image, and if the similarity between the projected image and the preview image acquired by the image acquisition unit 10 is less than the threshold γ, it is determined that the matching has failed. For this reason, it can be determined whether the recognition result by the object relationship estimation part 30 is correct.

<Third Embodiment>
[Outline of Image Processing Apparatus 1B]
FIG. 16 is a block diagram of an image processing apparatus 1B according to the third embodiment of the present invention. The image processing apparatus 1B is different from the image processing apparatus 1A according to the second embodiment of the present invention illustrated in FIG. 11 in that it includes a recognition result sharing processing unit 50 and a cooperative recognition processing unit 60. In the image processing apparatus 1B, the same components as those in the image processing apparatus 1A are denoted by the same reference numerals, and the description thereof is omitted.

  In the present embodiment, it is assumed that the terminal and other terminals are assumed as terminals equipped with the image processing apparatus 1B, and that these two terminals share the same AR space. In this case, since the distance from the object to the camera and the position and orientation of the camera with respect to the object are different between the own terminal and the other terminal, an object that can be recognized by one terminal may not be recognized by the other terminal. Therefore, the object recognition result is shared between the own terminal and other terminals.

[Configuration and Operation of Recognition Result Sharing Processing Unit 50]
The recognition result sharing processing unit 50 receives as input the recognition result by the image recognition unit 20 of the own terminal and the recognition result by the image recognition unit 20 of the other terminal, and the recognition by the image recognition unit 20 of the input own terminal. The result is transmitted to the image recognition unit 20 of another terminal. According to this, the recognition result by the image recognition part 20 can be shared between the own terminal and another terminal.

  Transmission / reception of the recognition result with the image recognition unit 20 of another terminal is realized by ad hoc communication. According to this, it is possible to communicate with other terminals in the same LAN. Further, even when there is no access point, it is possible to perform communication between adjacent terminals using Wi-Fi Direct or Bluetooth (registered trademark). Software (libraries) having a pairing function, a discovery function, and the like necessary for ad hoc communication are publicly available, and this function can be easily realized by using such a library. However, this function can also be realized using a general communication protocol via a wireless network or a wired cable.

  In addition, since the process by the recognition result sharing process part 50 does not need to synchronize with an own terminal and another terminal, the process which transmits the recognition result by the image recognition part 20 of an own terminal to the image recognition part 20 of an other terminal, It can be executed independently of the process of receiving the recognition result by the image recognition unit 20 of the other terminal by the image recognition unit 20 of the own terminal. In addition, since a delay generally occurs in the communication processing for transmitting and receiving the recognition result, the transmission processing and the reception processing of the recognition result with the image recognition unit 20 of the other terminal are independent of other processing (program It can be executed in another thread above.

[Configuration and Operation of Cooperative Recognition Processing Unit 60]
The cooperative recognition processing unit 60 receives the recognition result by the image recognition unit 20 of the own terminal and the recognition result by the image recognition unit 20 of another terminal. The cooperative recognition processing unit 60 converts the recognition result at the other terminal into the recognition result based on the own terminal, and integrates the recognition result at the own terminal. The cooperative recognition processing unit 60 includes a relative posture estimation unit 61 and a posture conversion unit 62.

  The relative posture estimation unit 61 receives the recognition result by the image recognition unit 20 of the own terminal and the recognition result by the image recognition unit 20 of another terminal. The relative attitude estimation unit 61 estimates an attitude (relative attitude) indicating a relative positional relationship between the own terminal and the other terminal based on the recognition result at the own terminal and the recognition result at the other terminal. . Hereinafter, the own terminal on which the image processing apparatus 1B is mounted is referred to as the own terminal S, and the other terminal on which the image processing apparatus 1B is mounted is referred to as the other terminal T.

  The estimation of the relative posture can be executed when the recognition result for the same object is included in both the recognition result at the own terminal S and the recognition result at the other terminal T. Note that the same object may be a reference marker.

Hereafter, the same object is referred to as object a. In addition, the posture matrix of the object a estimated by the posture tracking unit 23 of the own terminal S is defined as a posture matrix W _Sa, and the posture matrix of the object a estimated by the posture tracking unit 23 of the other terminal T is represented as a posture matrix. _Let W _Ta . Then, the relative attitude W _ST between the terminal S and the other terminal T can be obtained by the following mathematical formula (7).

  In addition, when a reference marker exists as the above-mentioned same object, it is preferable to use a reference marker as the above-mentioned object a. This is because the reference marker is generally designed to be easily recognized, and the recognition accuracy by the image recognition unit 20 is higher than that of other objects.

  On the other hand, when the reference marker does not exist as the same object, an object that can be recognized by both the own terminal and the other terminal may be used as the object a. When the reference marker does not exist as the same object as described above, the reference marker does not exist in the preview image acquired by the image acquisition unit 10 in the first place, or the reference marker does not exist in the preview image acquired by the image acquisition unit 10. May exist but cannot be recognized by at least one of its own terminal and other terminals.

In addition, the estimation of the relative attitude described above using Equation (7) is a case where there are two terminals, that is, the own terminal S and the other terminal T. When there are three or more terminals, the relative posture can be estimated as follows. Here, for example, the three terminals are the own terminal S, the other terminal T, and the other terminal U, the relative attitude W _ST between the own terminal S and the other terminal T, and the relative attitude between the other terminal T and the other terminal U. It is assumed that _WTU can be obtained. In this case, the relative posture W _SU between the terminal S and the other terminal U can be obtained by the following mathematical formula (8).

Therefore, even when there is no object that can be recognized by both the own terminal S and the other terminal U, by using the equation (8) instead of the equation (7), the own terminal S and the other terminal U Relative posture W _SU can be obtained. However, in this case, the relative attitude W _TU between the other terminal T and the other terminal U needs to be input to the cooperative recognition processing unit 60 from at least one of the other terminal T and the other terminal U.

Posture changing unit 62 has an input and recognition result of the image recognition unit 20 of another terminal, the relative orientation W _ST estimated by the relative posture estimation unit 61, a. This posture conversion unit 62 converts the recognition result at the other terminal into a recognition result based on the own terminal using the relative posture _WST .

Here, the recognition result of the object b that the terminal S cannot recognize is included in the recognition result of the other terminal T, and the posture matrix of the object b estimated by the posture tracking unit 23 of the other terminal T is It is assumed that it is represented by the attitude matrix W _Tb . Then, the posture matrix W _Tb of the object b estimated by the posture tracking unit 23 of the other terminal T is converted into the posture matrix W _Sb of the object b in the own terminal S by the following mathematical formula (9). This can be the recognition result of the object b.

  According to this, the orientation conversion unit 62 of the own terminal S also recognizes the recognition result by the image recognition unit 20 of the other terminal T, the own terminal S, and the object b that is not recognized by the image recognition unit 20 of the own terminal S. It can be recognized based on the relative posture with the other terminal T.

  Further, the posture conversion unit 62 integrates the recognition result of the object b in the own terminal S and the recognition result (recognition result of the object a in the own terminal S) by the image recognition unit 20 of the own terminal S, and performs integrated recognition. As a result. According to this, the attitude | position conversion part 62 can obtain the recognition result in the own terminal S about the object a and the object b.

  In addition, by using a relative posture as described above, it is possible to convert all objects included in the recognition result at the other terminal from the recognition result at the other terminal into a recognition result based on the own terminal. However, among all the objects included in the recognition results at other terminals, the object used when obtaining the relative posture is converted to the recognition result at the own terminal using this relative posture, and this object It matches the recognition result. For this reason, it is meaningless to convert the object used when obtaining the relative posture among all the objects included in the recognition result at the other terminal using the relative posture.

  For objects that can be recognized by both the own terminal and the other terminal, either the recognition result of the own terminal or the result of converting the recognition result of the other terminal using a relative posture may be used. it can. However, in this embodiment, the recognition result at the own terminal is preferentially used, and only the object that is not recognized at the own terminal is obtained by converting the recognition result at the other terminal using the relative posture. . Note that the objects that are not recognized by the own terminal are objects that have been recognized by the own terminal but failed to be recognized, and objects that have not been recognized by the own terminal in the first place. .

  The virtual information display unit 70 superimposes virtual information on the preview image based on the recognition result by the posture conversion unit 62 in addition to the recognition result by the image recognition unit 20 and the object relationship estimation unit 30.

  According to the image processing apparatus 1B described above, in addition to the above-described effects that can be achieved by the image processing apparatus 1A, the following effects can be achieved.

  In the image processing apparatus 1B, the cooperative recognition processing unit 60 converts the recognition result of the object recognized by the other image processing apparatus into a recognition result based on the image processing apparatus 1B. Further, the virtual information display unit 70 acquired by the image acquisition unit 10 based on the recognition result by the image recognition unit 20, the recognition result by the object relationship estimation unit 30, and the recognition result by the cooperative recognition processing unit 60. Virtual information is superimposed on the preview image. For this reason, the recognition result can also be used by another image processing apparatus for superimposing virtual information on the preview image. Therefore, in AR technology, the processing load can be further reduced and the robustness of object recognition can be further improved. Can be.

  The processing of the image processing apparatus 1, 1A, 1B of the present invention is recorded on a computer-readable non-transitory recording medium, and the program recorded on this recording medium is read into the image processing apparatus 1, 1A, 1B. The present invention can be realized by executing.

  Here, for example, a nonvolatile memory such as an EPROM or a flash memory, a magnetic disk such as a hard disk, a CD-ROM, or the like can be applied to the above-described recording medium. Further, reading and execution of the program recorded on the recording medium is performed by a processor provided in the image processing apparatus 1, 1A, 1B.

  Further, the above-described program may be transmitted from the image processing apparatuses 1, 1A, 1B storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Good. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  Further, the above-described program may be for realizing a part of the above-described function. Furthermore, what can implement | achieve the above-mentioned function in combination with the program already recorded on the image processing apparatus 1, 1A, 1B, what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design that does not depart from the gist of the present invention.

  For example, in each of the above-described embodiments, the two-dimensional barcode is described as an object in FIGS. 2 to 4, but the present invention is not limited to this, and any figure, character, object, or the like may be used.

  Further, in the second embodiment described above, when the recognition processing control unit 40 fails to recognize the main object by the image recognition unit 20, the image recognition unit performs an object other than the main object classified into the same group as the main object. The recognition by 20 may be resumed. According to this, when recognition by the object relationship estimation part 30 becomes impossible, recognition by the image recognition part 20 can be restarted, and the robustness of object recognition can be improved further.

  In each of the above-described embodiments, the image recognition unit 20 adds information serving as a recognition accuracy index of the recognition result to the recognition result for each object, and the object relationship estimation unit 30 adds the information to the recognition result. It is also possible to determine that the relative posture between objects whose recognition accuracy index is equal to or greater than a threshold value is stable. Furthermore, the object with the highest recognition accuracy may be used as the main object. According to these, the objects can be classified in consideration of the recognition accuracy of the recognition result of the objects. As the above-described recognition accuracy index, for example, a shooting distance and a shooting angle with respect to an object, a matching number of local features and a matching score, SSD (Sum of Squared Difference), NCC ( When a template matching method such as Normalized Cross Correlation is used, the response value of SSD or NCC can be adopted as it is.

  In each of the above-described embodiments, the image recognition unit 20 adds information serving as an index of the processing load required for object recognition to the recognition result for each object, and the object relationship estimation unit 30 causes the image recognition unit 20 to An object for which the added processing load index is less than the threshold may be set as the main object. According to this, since the object relationship estimation unit 30 can recognize the object using the relationship with the object having a low processing load, the processing load can be further reduced. As the above-described processing load index, for example, the time required for recognition can be adopted, or a value corresponding to the type of object can be set.

DESCRIPTION OF SYMBOLS 1, 1A, 1B; Image processing apparatus 10; Image acquisition part 20; Image recognition part 30; Object relationship estimation part 40; Recognition process control part 50; Recognition result sharing process part 60; C1, C2, C3; virtual information M1, M2, M3; object

Claims (23)

An image processing device for superimposing virtual information on a preview image,
Image acquisition means for acquiring the preview image;
Image recognition means for recognizing an object in the preview image acquired by the image acquisition means;
The relationship between objects recognized by the image recognition means is estimated, the objects are classified based on the estimation results, and the main object that is one of the objects classified into the same group is recognized by the image recognition means. Based on the result, object relationship estimation means for recognizing objects other than the main object classified into the group,
Virtual information display means for superimposing virtual information on the preview image acquired by the image acquisition means based on the recognition result by the image recognition means and the recognition result by the object relationship estimation means. An image processing apparatus. The object relationship estimation means includes:
Applying the object recognized by the image recognition means as the main object,
The object that is classified into the same group as the main object and that cannot be recognized by the image recognition unit is applied as an object other than the main object classified into the same group as the main object. Item 8. The image processing apparatus according to Item 1. The object relationship estimation means includes:
Select one object from each group as the main object,
The image processing apparatus according to claim 1, further comprising a recognition processing control unit that pauses recognition by the image recognition unit for an object other than the main object classified into the same group as the main object. .   When the recognition of the main object by the image recognition unit fails, the recognition processing control unit restarts the recognition by the image recognition unit for objects other than the main object classified into the same group as the main object. The image processing apparatus according to claim 3.   The recognition processing control unit collates the recognition result by the object relation estimation unit for the object whose recognition by the image recognition unit is suspended with the preview image acquired by the image acquisition unit, and if the verification fails 5. The image processing apparatus according to claim 3, wherein recognition of the object is resumed by the image recognition means.   When the recognition processing control unit restarts the recognition by the image recognition unit, the recognition result by the object relationship estimation unit in the preview image acquired last time by the image acquisition unit is set as an initial value to the image recognition unit. The image processing apparatus according to claim 3, wherein the image processing apparatus is tracked. The recognition processing control means includes
Based on the recognition result by the object relation estimation unit for the object whose recognition by the image recognition unit is suspended, the object is projected onto the preview image acquired by the image acquisition unit, and a projection image is created.
The collation is determined to have failed if the similarity between the projection image and the preview image acquired by the image acquisition unit is less than a threshold value. Image processing device.   The image processing apparatus according to claim 7, wherein the recognition processing control unit estimates an attitude that maximizes the similarity by iterative calculation and corrects a recognition result by the object relationship estimation unit. The recognition processing control means includes
Based on the recognition result by the object relation estimation unit for the object whose recognition by the image recognition unit is suspended, the object is projected onto the preview image acquired by the image acquisition unit, and a projection image is created.
A matching part is estimated by template matching between the projection image and the preview image acquired by the image acquisition unit, and if a response value at the matching part is less than a threshold, it is determined that the matching has failed. The image processing apparatus according to claim 3. Coordinate recognition processing means for converting a recognition result of an object recognized by a first image processing device different from the image processing device into a recognition result based on the image processing device,
The virtual information display means is a preview image acquired by the image acquisition means based on a recognition result by the image recognition means, a recognition result by the object relationship estimation means, and a recognition result by the cooperative recognition processing means. The image processing apparatus according to claim 1, wherein virtual information is superimposed on the image processing apparatus.   11. The object relationship estimation unit obtains a relative posture indicating a relative positional relationship between the objects as the relationship between the objects recognized by the image recognition unit. An image processing apparatus according to 1. The object relationship estimation means includes:
Each time a preview image is acquired by the image acquisition means, a relative posture between objects in the preview image is obtained,
12. The objects of which the amount of change between the preview images of the relative posture is less than a threshold value continuously over a predetermined number of preview images are classified into the same group. The image processing apparatus according to any one of the above.   The object relationship estimation means calculates the difference between the relative attitude obtained in the latest preview image acquired by the image acquisition means and the average of the relative attitude obtained in the preview image before the latest preview image. The image processing apparatus according to claim 12, wherein the image processing apparatus calculates the amount of change. The image recognizing unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object,
14. The object relation estimation unit determines that the relative posture between objects having a recognition accuracy index added by the image recognition unit equal to or greater than a threshold value is stable. An image processing apparatus according to claim 1. The image recognizing unit adds information serving as an index of recognition accuracy of the recognition result to the recognition result for each object,
15. The image processing apparatus according to claim 1, wherein the object relation estimation unit applies the object having the highest recognition accuracy index added by the image recognition unit to the main object. .   The image processing apparatus according to claim 14, wherein the image recognition unit uses at least one of a shooting distance with respect to an object and a shooting angle with respect to the object as an index of the recognition accuracy.   The image recognition means uses at least one of a matching number of local feature values and a matching score of local feature values as an index of the recognition accuracy. An image processing apparatus according to 1.   The image recognition means uses at least one of a response value of SSD (Sum of Squared Difference) and a response value of NCC (Normalized Cross Correlation) as an index of the recognition accuracy. The image processing device according to any one of 14 to 17. The image recognizing unit adds information serving as an index of a processing load required for recognition of the object to the recognition result for each object,
The image according to any one of claims 1 to 18, wherein the object relation estimation unit applies an object having a processing load index added by the image recognition unit less than a threshold to the main object. Processing equipment.   The image processing apparatus according to claim 19, wherein the image recognition unit uses a time required for recognition as an index of the processing load.   The image processing apparatus according to claim 19, wherein the image recognition unit sets a value corresponding to a type of an object as the processing load index. An image processing method in an image processing apparatus comprising image acquisition means, image recognition means, object relationship estimation means, and virtual information display means, and superimposing virtual information on a preview image,
A first step in which the image acquisition means acquires the preview image;
A second step in which the image recognition means recognizes an object in the preview image acquired in the first step;
The object relationship estimation means estimates the relationship between the objects recognized in the second step, classifies the objects based on the estimation result, and is one of the objects classified into the same group A third step of recognizing an object other than the main object classified into the group based on the recognition result of the object in the second step;
The virtual information display means superimposes virtual information on the preview image acquired in the first step based on the recognition result in the second step and the recognition result in the third step. An image processing method comprising the steps of: A program for causing a computer to execute an image processing method in an image processing apparatus that includes image acquisition means, image recognition means, object relationship estimation means, and virtual information display means, and superimposes virtual information on a preview image,
A first step in which the image acquisition means acquires the preview image;
A second step in which the image recognition means recognizes an object in the preview image acquired in the first step;
The object relationship estimation means estimates the relationship between the objects recognized in the second step, classifies the objects based on the estimation result, and is one of the objects classified into the same group A third step of recognizing an object other than the main object classified into the group based on the recognition result of the object in the second step;
The virtual information display means superimposes virtual information on the preview image acquired in the first step based on the recognition result in the second step and the recognition result in the third step. Program for causing a computer to execute the above steps.

Images