JP3530772B2 - Mixed reality device and mixed reality space image generation method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mixed reality apparatus using a marker as a position index, without the player observing the mixed reality space being aware of the presence of the marker.
The present invention relates to a mixed reality apparatus that can concentrate on observation of a mixed reality space and a method for generating a mixed reality space image.

[0002]

In recent years, mixed reality for the purpose of seamless coupling of real space and virtual space (Mixed Reality, M
Research on R) has become popular. MR has been attracting attention as a technique for enhancing VR for the purpose of coexistence of the world of virtual reality (VR), which was conventionally available only in a situation separated from the physical space, and the physical space.

A typical device for realizing mixed reality is a head mounted display (HMD). That is, the mixed reality is realized by synthesizing and displaying the real space and the virtual space on the HMD. As an MR method using the HMD, a semi-transmissive (see-through type) head mounted display (H
There are an optical see-through method of superimposing an image of CG or the like on the MD) and a video see-through method of displaying an image on the HMD after the image of the CG or the like is combined with image data taken by a video camera mounted on the HMD.

[0004] MR is used as a medical assistance to present to a doctor as if the patient's internal condition is seen through, and
It is expected that new fields will be qualitatively completely different from the VRs used up to now, such as the use of work assistance in which the assembly procedure of products is displayed on the actual product in a factory.

What is commonly required for these applications is a technique for removing the "deviation" between the real space and the virtual space. “Misalignment” can be classified into misalignment, time misalignment, and qualitative misalignment. Among these, many efforts have been made to solve misalignment, which is the most basic requirement.

Particularly in the case of the video see-through type MR, an image processing method can be applied relatively easily as a method for correcting the positional deviation. Therefore, a conventional proposal has been made for the alignment using the image processing.

Specifically, a marker marked with a color that can be easily detected by image processing is arranged at a predetermined position in the real space, and the viewpoint position is determined by the marker position detected from the image taken by the camera worn by the player. There are a calculation method and a method of correcting the output signal of the position and orientation sensor according to the marker position in the image.

[0008]

When a marker in image data is detected and the position and orientation of the player is estimated based on the result, the marker must appear in the image at an appropriate size and at substantially uniform intervals. There is. Further, since it is necessary to detect a sufficient number of markers in the image at the same time to calculate the viewpoint position, it is necessary to arrange the markers so that they can be observed at a certain narrow interval in the image.

On the other hand, in order to improve the tracking accuracy and identification accuracy of the markers, it is necessary to arrange the markers so that they can be observed at a somewhat wide interval in the image.

When there is only one player, it is not difficult to arrange the markers so as to satisfy such a condition, but in an application where a plurality of players share a common mixed reality space, it is possible. It is possible that markers that are observable at equal intervals from the player are not observed at equal intervals from other players.

Therefore, in Japanese Unexamined Patent Publication No. 11-84307, in an air hockey game in which two players strike a virtual puck on a table, which is a real object, markers of different colors are provided for each player.
The markers can be observed by each player in a suitable arrangement and size.

However, as the number of players sharing the same mixed reality space increases, it becomes difficult to arrange markers by color coding. That is, in order to detect a color by image processing, the color of the marker and the background object and the color of the marker between different users must be colors that can be easily detected and separated by the image processing. If the number of colors increases, it becomes difficult to meet this condition, which causes an extraction error or an identification error between markers.

FIG. 10 is a diagram for explaining the cause of erroneous recognition when markers of a plurality of colors are used. In FIG.
The horizontal axis represents red and the vertical axis represents green, and the blue axis is omitted in order to simplify the description. In the figure, a region A shows a region defining the color distribution of the marker type A (red marker), and a region B shows a region defining the color distribution of the marker type B (orange marker). Thus, when the number of players increases and similar colors are used as markers,
Despite observing a red marker, for example, depending on the position of the light source or the angle of the image, the area B (that is, the orange marker)
There is a possibility that it will be detected as a color, which will cause erroneous detection.

Further, by arranging a plurality of markers of different colors for each player, a large number of markers exist in the real space, which complicates the field of view of the player and gives a sense of the mixed reality space. It becomes a cause to impair the sense of presence.

An object of the present invention is to solve the above-mentioned problems and enable a player to experience mixed reality without being aware of the presence of the marker even when the marker is arranged in the physical space. An object is to provide a possible mixed reality apparatus and a method for generating a mixed reality space image.

[0016]

That is, the gist of the present invention is to generate a virtual image according to the viewpoint position and orientation of the player.
Image generation means and shooting from the approximate viewpoint position and orientation of the player
Image input to input the image data obtained by
By combining the image data with the virtual image.
Mixed reality with synthesis means for generating an augmented reality space
And the absolute position input by the image input means.
Take a picture of the physical space where markers as placement indicators are placed
The marker area is detected from the image data obtained by
Marker detection means and the markers in the real space are arranged.
Image data obtained by shooting the unloaded state
Holding means for holding the
For the marker area detected by the output means,
A mixed reality apparatus having a means for replacing or superimposing a cha image .

Another gist of the present invention is the player's view.
An image generation step that generates a virtual image according to the point position and orientation.
And shooting from the player's approximate viewpoint position and orientation.
Image input step of inputting the obtained image data,
By combining image data and virtual images
Mixed reality space with a compositing step for generating inter-images
A method of generating an image, which is input by image input means.
Real sky where markers as absolute position indicators are placed
From the image data obtained by shooting the
Marker detection step to detect the mosquito region, and in the real space,
Obtained by taking a picture of the state where no marker is placed
Hold the stored image data as a texture image
Step and the marker detected by the marker detection step.
Replace or superimpose the texture image on the marker area
The method for generating a mixed reality space image is characterized by including the steps of:

[0018]

[0019]

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the marker placement method of the present invention will be described below with reference to the drawings. Although the present embodiment is a mixed reality game in which three players share the same virtual space, the marker placement method in the mixed reality space according to the present invention can be applied to any other application.

FIG. 1 is a diagram showing the configuration of a mixed reality apparatus for executing a mixed reality game to which the marker placement method according to the present invention is applied. FIG. 1 shows the configuration when there are three players. The mixed reality apparatus includes player processing units 100 to 300 provided for each player and a control unit 400 connected to each player processing unit, and is connected to the control unit 400 depending on the number of players. The number of player processing units also increases or decreases.

The player processing units 100 to 300 have the same structure, and input / output units (102, 103, 1) such as sensors and display devices mounted on the player's body.
05, 1011, 1021, 1022), a circuit part (101) for generating an image to be displayed on the display device based on the signal processing collected from the sensor and the result of this signal processing and the information of another player supplied from the control unit. , 104, 10
6, 107).

The configuration of the player processing unit will be described below with reference to FIG. 1 and FIG. 2 showing the types of input / output devices mounted by each player and the mounting locations in this embodiment. As shown in FIG. 2, in the present embodiment, each player has a head-mounted image display device (hereinafter, referred to as an HMD) 1 which is a display device for combining and displaying a real space and a virtual space on the head.
Wear 05. In the present invention, the HMD may be either a video see-through type or an optical see-through type, but in the following description, a case where a video see-through HMD is used will be described as an example.

Two small video cameras 103 are provided near the eyes of the HMD 105. An image captured by the video camera 103 from a viewpoint substantially the same as the viewpoint of the player is supplied to an image combining unit 106, which will be described later, via the image input unit 104, and is superimposed on the virtual space image and displayed on the HMD 105 of the player. To be done. Further, the image input unit 104 supplies the input image to the head position detection unit 101 in addition to the image synthesis unit 106.

Further, a head position / orientation sensor 1011 including, for example, a magnetic sensor is mounted on the player's head. The head position / orientation sensor 1011 is, for example, H
It can be attached using the MD 105. The output signal of the head position / orientation sensor 1011 is the head position / orientation measurement unit 1
01 is input. The head position / orientation measurement unit 101 detects a marker for position correction using the image supplied from the image input unit 104, and corrects the signal supplied from the head position / orientation sensor 1011 to obtain the viewpoint of the player. Estimate position and orientation.

On the other hand, the interactive operation input device 102 is mounted on the player's arm. Interactive operation input device 102
Has a position / orientation sensor 1021 for detecting the position and posture of the mounted part, and a switch (trigger) 1022 for turning on / off according to the movement of the mounted part. This is a device for inputting.

In the following description, a plurality of players (three people in this embodiment) share the same virtual space, destroy the enemy while avoiding the attack from the enemy appearing in the virtual space, and A case in which the following commands can be input using the interactive operation input device 102 in a game of competing for the number or points of the enemies destroyed until the damage caused by the enemy's attack reaches a predetermined amount will be described as an example. .

Command 1 (aiming command) A command for displaying an aiming line indicating the aiming position in the virtual space. With the back of the hand up, enter by placing the wrist above the elbow.・ Command 2 (shooting command) Command to shoot at the aiming position indicated by the line of sight. When the aiming line is displayed by the aiming command, the arm (from the elbow to the palm) is reciprocally moved back and forth with a certain acceleration or more to input. -Command 3 (defense command) A command that defends the opponent's attack. With the back of the hand facing the other party, point by pointing your finger upwards. -Command 4 (reset command) A command to be input when the aiming command is input again after the shooting command is input. Input by making the arm hang down.

That is, as a player's operation in the game of the present embodiment, normally, command input is repeated in a cycle of aiming command → shooting command → reset command → aiming command, and a defense command is issued as necessary in this cycle. Will be input.

The command input by the interactive operation input device 102 is supplied to the image generator 107.

The image generator 107 transfers the player's head position / posture information supplied from the head position / posture measuring unit 101 and the command information supplied from the interactive operation input device 102 to the control unit 400. Further, the head position / posture information of the player and the command information, the head position / posture information of the other player received from the control unit 400, the command information and the model information, the position, the moving direction, the state information of the enemy character, the space A virtual space image to be displayed on the HMD 105 of the corresponding player is created by using the information such as the position and shape of the obstacle arranged at, and is output to the image synthesizing unit 106.

The image synthesizing unit 106 synthesizes the image (real space image) of the video camera 103 attached at a position close to the player's viewpoint and the virtual space image created by the image creating unit 107, and the HMD 105 of the player. Supply.

The control section 400 is a mixed reality space management section 1 to which the player processing sections 100 to 300 as described above are connected.
And the physical object position measuring unit 2. The mixed reality space management unit 1 uses the player processing units 100 to 300.
The information regarding the position and posture of the player's head received from the player and the position, posture and command information of the interactive operation input device 102 are distributed, and the generation, disappearance, control and shooting commands of the enemy character displayed in the virtual space are used. Performs game processing such as shooting hit determination. Enemy character's model, position, movement direction, and state (whether it was destroyed)
Information is also distributed to all users who are connected together with the information of each player.

Further, as shown in FIG. 3, in order to adjust the difficulty level of the game, a real object 31 that may be an obstacle to shooting is provided.
~ 33 are arranged, the physical objects 31 to 3
The information about the shape and position of 3 also includes the mixed reality space management unit 1
Managed by.

FIG. 3A is a perspective view of a game field (mixed reality space) viewed from a certain player, and FIG. 3B is a top view thereof. In FIG. 3, three physical space objects 31 to 3 are placed on a table 30 (in physical space).
The case where 3 is arranged as an obstacle is shown. In the present embodiment, since the aiming command input operation using the interactive operation input device 102 is an operation of raising the wrist higher than the elbow as described above, the height is about the same as the waist in a general figure. Although the physical space object is arranged on the table 30 that it has, it goes without saying that the table is unnecessary depending on the command input operation by the interactive operation input device.

In FIG. 3, physical space objects 31 and 32 are fixed, and 33 is movable. If there is a movable physical space object, the difficulty level of the game can be dynamically changed. For example, if the movable physical object 33 moves to the left and right at random speed, the shooting of the enemy character that is the target becomes more difficult than when only the static physical objects 31 and 32 are arranged. The movement control of the movable reality object may be performed by the mixed reality space management unit 1 or by another control circuit. In this case, the model of the movable object is managed by the mixed reality space management unit 1, and the position thereof is measured by the physical object position measuring unit 2 using the object position sensor 331 provided on the movable object 33.

In the mixed reality apparatus described above, the control unit 400 is the server, and the player processing units 100 to 300.
It can be realized by a client = server system in which (the circuit portion of) is a client. It is possible to flexibly deal with the increase and decrease of the players by distributing the processing related to each player among the clients. More specifically, the player processing unit is a general-purpose computer device having a video input / output function and a signal receiving function from various sensors, and the control unit 400 can also communicate with each player processing unit and an object position measuring unit 2. It can be realized by a general-purpose computer device having a function of receiving a measurement signal from.

However, since it is necessary to perform the calculation relating to the three-dimensional image display in real time, it is preferable to use a relatively high speed computer device having an accelerator (so-called 3D accelerator) specialized for such calculation. . Further, the control unit 400 and the player processing units 100 to 3
The communication with 00 is also preferably connected by a line with a large capacity such as 100BASE-T. If the capacity of the communication line is small, the processing speed will decrease as the number of players increases.

(Marker Arrangement Method) FIG. 4 is a perspective view showing an example of marker arrangement in this embodiment. In the game of the present embodiment, an obstacle due to a physical object is arranged, but by arranging the marker using the obstacle, the marker which is in the visual field within the moving range of each player is limited, and It is possible to meet the conditions.

FIGS. 5A to 5C are diagrams showing the markers that can be seen from the players A, B, and C in FIG. 4 extracted. In this way, the markers seen by each player are observed at substantially equal intervals and the number necessary for the calculation of the viewpoint position, but the markers for other players are not visually recognized, so that it is not necessary to change the color. Also, FIG.
It is also possible to share the same marker among a plurality of players as shown in FIG.

6 (a) to 6 (c) show markers observed by each player when no obstacle is used, as shown in FIG. 5 (a).
It is a figure shown corresponding to- (c). It will be clearly understood from the comparison between FIG. 5 and FIG. 6 that the marker arrangement method according to the present invention significantly reduces the number of markers observed by each player and satisfies the above condition.

When the number of players is increased, it is possible to deal with the situation by changing the shape (cross-sectional shape, height, etc.) of the physical object on which the marker is provided, or by adding a color.
Even when the number of colors is increased, one color is not assigned to one player as in the conventional case, so that it is possible to arrange markers for many players with a small number of colors.

The placement position of the marker may be determined manually, but a model of the physical object and a model of the movable range of the viewpoint position of each player are generated in advance, and the model can be seen by the target player. It can be determined by obtaining the range in which the line of sight of another player is blocked. Further, the position may be calculated using the number of markers to be provided and the arrangement rule. On the contrary, the obstacle shape and / or the layout satisfying such a condition may be calculated.

Even if there is no object such as an obstacle that can be used for arranging the marker, the real object is arranged at a position where there is no problem in the application, and the existence of the real object is hidden by the virtual space image. Makes it possible to experience mixed reality space without being aware of the physical object for the marker. Concealment of markers and the like by the virtual space image will be described in detail later.

(Detection of Marker) Next, a method of detecting a marker will be described. FIG. 8 is a flowchart showing a flow of processing for detecting a type A marker (red marker) having a color included in the area A shown in FIG. The head position / orientation measurement unit 101 performs both the marker detection process and the use of the detected marker information.

First, the image captured by the video camera 103 is acquired via the image input unit 104 (step S).
701). Then, binarization processing is performed (step S70).
2). Specifically, the pixels included in the region A shown in FIG. 7 (the blue axis is omitted) are set to 1, and the other pixels are set to 0. Ii: i-th pixel Ri, Gi, Bi constituting the input color image I: R, G, B color values ITHi constituting Ii: i-th pixel value RminA, GminA, BminA of the binary image: Minimum values RmaxA, GmaxA, BmaxA of R, G, B defining the area A: If R, G, B maximum values of the area A are defined, RminA <Ri <Rmax for each Ii.
A and GminA <Gi <GmaxA and BminA <
A binary image ITH is formed by setting ITHi corresponding to Ii that satisfies Bi <BmaxA to 1 and ITHi corresponding to other Ii to 0.

Then, a labeling process is performed on the binary image ITH to extract a marker area (cluster) (step S703). And the center of gravity of each cluster (Xn, Yn)
And the area an are calculated (step S704) and output to a viewpoint position / posture estimation module (not shown) inside the head position / posture measurement unit 101 (step S705). The viewpoint position / posture estimation module uses the head position / posture sensor 10 based on the absolute coordinates of the markers registered in advance and the marker positions (cluster centroids) detected from the image.
The output signal of 11 is corrected, and the viewpoint position and orientation of the player are calculated.

In FIG. 8, the processing for detecting a certain color (red) marker has been described. However, when the number of players increases and there are markers of a plurality of colors, a binarized image is generated. The detection process for each color is repeated by changing the threshold value when performing.

(Erase of Marker) As described above, the marker is originally unnecessary if the output of the head position / orientation sensor 1011 is accurate. Further, it is desirable that the presence of the marker is not recognized in the mixed reality space, such that the player may lose the sense of presence by recognizing the marker.

Therefore, the present invention is characterized in that the marker is apparently erased so that the player is not aware of the presence of the marker. Various methods are conceivable for apparently erasing the marker, but a method of superimposing a virtual image on the marker and displaying it on the player is preferable because the processing load and discomfort are small.

FIG. 9 is a diagram for explaining a marker erasing method. FIG. 9A shows the markers for the player A arranged by the marker arrangement method according to the present embodiment, and corresponds to FIG. 5A. In order to erase the markers placed on the physical object such as the table and the obstacle in this way, first, before placing the markers, shoot the planned location with a video, digital camera, still camera, etc., and save the image data. Get it as a texture. Then, after the marker is arranged, of the image data displayed on the HMD of the player, the image data corresponding to the marker portion is replaced with the texture image acquired in advance, or the image data is displayed in a superimposed manner (FIG. 9 ( b)).

By replacing / superimposing the image data in this way, the player can concentrate on the game without recognizing the presence of the marker. It is preferable to acquire the image data prepared as a texture under the conditions (light source position and brightness) that are actually used, in order to reduce the visual discomfort, but it is arranged on a real object of the same material. The texture image from which the marker is deleted may be one common texture.

Further, in an application that does not use an obstacle, when a real object is arranged only for marker arrangement, or when it is difficult to replace / superimpose image data on individual markers, As shown in FIG. 9C, it is also possible to use an image 91 that covers all real objects. In this case, what kind of image data is used may be appropriately determined by the application.
For example, in a game application, on the contrary, the image 91 may be used to form a stage-like object in the virtual space, or in an actual space if the application does not want to feel a difference from the real space. It may be data of a photographed object. Of course, it is also possible to use the data of a three-dimensional object instead of the two-dimensional image data.

[0054]

[Other Embodiments] In the above embodiment, the information obtained from the marker is used for the error correction of the head position / orientation sensor 1011. However, the information can be obtained from the marker without using the head / orientation sensor 1011. It is also possible to obtain the viewpoint position and orientation of the player from only the information, and the present invention can be applied to such an application. In addition, although the above-described embodiment is a video see-through mixed reality apparatus, it goes without saying that the present invention is also applicable to an optical see-through mixed reality apparatus.

In the above embodiment, the trigger 1022 is not used, and the command is input by detecting the movement of the interactive operation input device. However, the player uses the trigger 1022 to input various commands. Alternatively, motion detection and trigger on / off may be combined. Further, the interactive operation input device may be incorporated with a switch that is turned on / off according to the player's operation and used as a trigger.

An object of the present invention is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply a computer (or CPU) of the system or apparatus. It is needless to say that it is achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted in the computer or the function expansion unit connected to the computer, based on the instruction of the program code. Needless to say, this also includes a case where a CPU or the like included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

[0058]

As described above, according to the present invention,
By apparently erasing the marker used as the position pointer in the mixed reality device, the player does not have to be aware of the presence of the marker and can experience the mixed reality without impairing the realism.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of a mixed reality apparatus to which a marker placement method according to the present invention can be applied.

FIG. 2 is a diagram illustrating a type of a device worn by a player and a mounting site.

FIG. 3 is a diagram illustrating a game according to the embodiment of the present invention.

FIG. 4 is a diagram illustrating a marker placement method according to the present invention.

5 is a diagram showing markers observed by each player in the arrangement of FIG.

FIG. 6 is a diagram showing markers observed by each player when the same markers as those in FIG. 4 are arranged without using obstacles.

FIG. 7 is a diagram showing a color region of a red marker according to the embodiment of the present invention.

8 is a flowchart showing a procedure of processing for detecting a marker in the color area shown in FIG. 7 from color image data.

FIG. 9 is a diagram illustrating a marker erasing mode.

FIG. 10 is a diagram illustrating erroneous detection when the number of colors of markers is increased.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁷ identification code FI H04N 7/18 H04N 7/18 V (56) Reference JP-A-10-326357 (JP, A) JP-A-10-210473 ( (58) Fields surveyed (Int.Cl. ⁷ , DB name) A63F 13/00-13/12 G06F 3/00 G06T 17/40 H04N 5/262 H04N 7/18

Claims (6)

(57) [Claims] 1. An image generation unit for generating a virtual image according to a player's viewpoint position / posture, an image input unit for inputting image data obtained by shooting from the player's approximate viewpoint position / posture, and the image data. A mixed reality apparatus having a synthesizing means for generating a mixed reality space by synthesizing the virtual image with the virtual image, wherein a real space in which a marker as an absolute position index, which is input by the image input means, is placed A marker detecting unit that detects a marker area from image data obtained by shooting, and image data obtained by shooting a state in which the marker is not arranged in the real space is held as a texture image. Holding means and replacing the texture image for the marker area detected by the marker detecting means Mixed reality apparatus characterized by having a means for superimposing the. 2. The texture image used for replacing or superimposing a plurality of marker regions corresponding to a plurality of markers arranged on a physical object of the same material is common. Mixed reality device. 3. A marker area detected by the marker detecting means, absolute coordinates of the marker, and the player.
The mixed reality apparatus according to claim 1, further comprising means for estimating a viewpoint position / orientation of the player from an output signal of the head position / orientation sensor . 4. An image generating step of generating a virtual image according to a player's viewpoint position / posture, an image input step of inputting image data obtained by shooting from the player's approximate viewpoint position / posture, and the image data And a synthesizing step of synthesizing the virtual image to generate a mixed reality space image, wherein a marker as an absolute position index input by the image input means is arranged. A marker detection step of detecting a marker area from image data obtained by photographing the real space; and texture image data obtained by photographing the state of the real space in which the markers are not arranged. A holding step of holding as an image, and the marker region detected by the marker detecting step. Respect, the method of generating the mixed reality space image, characterized by a step of replacing or superimposes the texture image. 5. The texture image used for replacing or superimposing a plurality of marker regions corresponding to a plurality of markers arranged on a physical object of the same material is common. A method for generating a mixed reality space image. 6. A marker area detected by the marker detection step, the absolute coordinates of the marker, the pre
The method for generating a mixed reality space image according to claim 4, further comprising the step of estimating the viewpoint position and orientation of the player from the output signal of the head position and orientation sensor of the player.