JP2012128779A - Virtual object display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem such that, in a virtual object display device since a display position of a virtual object is uniquely determined depending on the position and posture of a viewer, the viewer has to position him/herself and to make his/her own posture at a point where the virtual object is presented to obtain a desired virtual object.SOLUTION: The virtual object display device is characterized in that a virtual object is generated in a three-dimensional virtual space and having two or more image display manners. A coordinate system as a reference of a presentation position of a virtual object to be presented in the three-dimensional virtual space is determined based on the viewer's position and posture and a display manner selected from plural image display manners. Coordinate conversion of the virtual object as the display object in the three-dimensional virtual space is made based on the reference coordinate system and the viewer's position and posture. The virtual object as a result calculated by a coordinate calculation part 306 is presented to the viewer on the coordinate system determined by the reference coordinate determination part 305.

Description

  The present invention relates to a virtual object display device that generates a virtual object in a three-dimensional virtual space and displays it to an observer. More specifically, the present invention relates to a virtual object display device that determines and displays a display position of a virtual object from the position and orientation of an observer.

  2. Description of the Related Art In recent years, services using augmented reality that augments and displays virtual objects such as CG (Computer Graphics) and character information on a real space are actively performed. Augmented reality includes a video see-through method in which a virtual object is superimposed on an image of a real space taken by a video camera or the like, and an optical see-through method in which the real space is directly viewed through a half mirror and superimposed on the virtual object.

  As a device for presenting visual information, a small information terminal such as a mobile phone, a head mounted display (HMD), or the like is used.

  One of the services using such augmented reality is an application provided by Tonchi Dot Co., Ltd., “Sekai Camera”, and “Rayar” provided by SPRXmobile in the Netherlands (“Sekai Camera”, “Layer”). Are trademarks or registered trademarks of the respective companies). In these applications, when location information obtained by GPS (Global Positioning System) or the like is used and the mobile phone is held over a real space, the facility information is displayed superimposed on the scenery photographed by the mobile camera.

  There are also many techniques for recognizing a specific pattern (marker) using an image recognition technique and displaying a virtual object superimposed thereon.

  In the augmented reality world represented by the above, it is important to determine where the virtual object is to be superimposed on the three-dimensional space.

  A method for determining the display position of a virtual object based on position information and the posture of an observer is known (see, for example, Patent Document 1). In Patent Document 1, a technique for facilitating alignment between the real world and a virtual object is devised.

  FIG. 1 shows a simple configuration of a conventional virtual object display device.

  In FIG. 1, a posture sensor 101 measures the posture of an observer. When the orientation information output unit 102 converts the coordinate system of the virtual object from the sensor coordinate system to the world coordinate system, the correction information stored in the memory 103 is acquired as necessary, and the image generation unit 105 generates the output image. I do.

Japanese Patent No. 3840171

  However, in the conventional configuration, it is possible to accurately link the virtual object to the observer, but the display position of the virtual object is uniquely determined by the position and posture position of the observer, and the user can determine the position of the virtual object. Cannot be moved as the user likes. For this reason, in order to acquire the virtual object that the observer wants, the observer has to go to the position where the virtual object is displayed and express his / her posture.

  In order to solve the conventional problems, a virtual object display device of the present invention includes a position and orientation acquisition unit that acquires the position and orientation of an observer, and a display method determination unit that determines one display method from a plurality of image display methods. The display method switching unit that switches the display method to the display method determined by the display method determining unit, and the display of the virtual object to be displayed in the three-dimensional virtual space based on the display method determined by the display method determining unit Acquired by a reference coordinate system determination unit that determines a coordinate system that is a reference for a position, a data acquisition unit that acquires data of a virtual object, and a reference coordinate system and a position and orientation acquisition unit that are determined by the reference coordinate system determination unit The coordinate calculation unit that performs coordinate transformation of the virtual object to be displayed in the three-dimensional virtual space and the virtual object data acquired by the data acquisition unit are determined by the reference coordinate system determination unit from the position and orientation of the observer Coordinates And a display screen combining unit for displaying on the display unit is superimposed on a predetermined position in the three-dimensional virtual space coordinate transformation by the coordinate calculation section by.

  With such a configuration, the display position of the virtual object can be flexibly switched according to the application.

  According to the present invention, it is possible to easily change the display position of a virtual object according to the preference of an observer, and it is possible to easily acquire a desired virtual object. For example, by displaying a virtual object related to a landmark that can be acquired at a tourist destination on a trip linked to a photograph of the landmark, the viewer can feel as if he / she carried the virtual object or bookmarked it. can do.

It is a figure showing the structure of the conventional virtual object display apparatus. It is a figure which shows the usage condition of the virtual object display apparatus concerning this invention. It is a block diagram which shows the internal structure of a display apparatus. It is a figure explaining how a reference coordinate is determined by each display method. It is a figure showing an example of the display screen when a position fixed display system is selected. It is a figure showing an example of the display screen when an object fixed display system is selected. It is a figure showing an example of the display screen when a constant display system is selected. It is a figure showing an example of the display screen when a posture fixed display system is selected. 5 is a flowchart for explaining a processing procedure until reference coordinates are set in the first embodiment. It is a figure showing the state transition of a virtual object. It is a figure showing an example of the display screen at the time of an observer selecting a display system. It is a flowchart explaining the process sequence until it acquires virtual object data and displays it on a display part.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 2 is a diagram for explaining a virtual object display device which is an example of an embodiment according to the present invention.

  The display device 200 is formed by, for example, the imaging unit 201, the display unit 202, the sensor group 203, and the system control unit 204, and communicates with the server 205. The observer can observe the image displayed on the display unit 202 by wearing the display device 200 like glasses. The sensor group 203 includes, for example, a position sensor, a posture sensor, and the like.

  In this embodiment, the display device is assumed to be a head-mounted display (HMD). However, as a modification of this embodiment, if the imaging unit 201 and the sensor group 203 are provided, the display device can be observed by an observer. Any device may be used (for example, a mobile phone terminal).

  In the present embodiment, description will be made on the assumption that two imaging units 201 and two display units 202 are provided on each of the left and right sides, and that the images can be viewed in binocular stereoscopic vision by displaying separate images on the left and right sides. The modification of the present embodiment is not limited to a binocular stereoscopic image.

  Next, details of the display device 200 according to the present embodiment will be described.

  FIG. 3 is a block diagram illustrating an internal configuration of the display device 200. A system using the virtual object display device of the present invention includes a position sensor 301, a posture sensor 302, an imaging unit 201, a position and posture acquisition unit 303, a display method switching unit 304, a reference coordinate determination unit 305, a coordinate calculation unit 306, and event detection. A unit 307, a display method determination unit 308, a display image composition unit 309, a data acquisition unit 310, a display unit 202, and a server 205.

  The position and orientation acquisition unit 303, the display method switching unit 304, the reference coordinate determination unit 305, the coordinate calculation unit 306, the event detection unit 307, the display method determination unit 308, the display image composition unit 309, and the data acquisition unit 310 are illustrated in FIG. It is good also as a structure incorporated in the system control part 204. FIG. In this case, a system using the virtual object display device includes a display device 200 and a server 205.

  Alternatively, the position / orientation acquisition unit 303, the display method switching unit 304, the reference coordinate determination unit 305, the coordinate calculation unit 306, the event detection unit 307, the display method determination unit 308, the display image composition unit 309, and the data acquisition unit 310 may be, for example, a display It is good also as a structure included in the apparatus (for example, mobile phone terminal) which an observer different from the apparatus 200 can observe. In this case, a system using the virtual object display device includes a display device 200, a device (not shown) that can be observed by an observer different from the display device 200, and a server 205.

  The position sensor 301 is a sensor that measures the absolute value of where the observer is on the earth, such as GPS. The posture sensor 302 is, for example, a combination of a gyro sensor and an accelerator sensor, and is a sensor for measuring the posture of the observer with at least three degrees of freedom. Instead of providing the posture sensor 302, the user's posture may be obtained by photographing the user's field of view with a stereo camera.

  The data acquisition unit 310 communicates with the server 205. The server 205 stores virtual object data to be displayed.

  In the present embodiment, the data acquisition unit 310 acquires virtual object data by communicating with the server 205. However, as a modification of the present embodiment, the communication unit is not provided, but the display data recording unit is provided. Object data may be acquired.

  The display unit 202 displays the synthesized image for the observer. The display unit 202 displays the composite image and shows the calculated three-dimensional image to the observer. Further, in the following description, the description will be made on the assumption that a virtual object is superimposed on an image captured in the real world. However, as a modification of the present embodiment, an optical see-through method in which a display element is transparent and transparent is visible. Alternatively, it may be used in a VR (Virtual Reality) space in which a virtual object is superimposed on an image of the virtual world.

  The imaging unit 201 captures an image that enters the observer's field of view. When the head mounted display (HMD) is used as the display device 200, the imaging unit 201 captures an image that falls within the field of view of the observer when the display device 200 is not attached.

  When the display device 200 adopts an optical see-through method in which the display elements constituting the display unit 202 are transparent and transparent, the imaging unit 201 captures an image of the real world that falls within the observer's field of view. Become.

  The event detection unit 307 recognizes an observer's gesture using, for example, an image captured by the imaging unit 201 and monitors whether an event is input. When an event input occurs, the display method determination unit 308 determines the display method of the virtual object.

  The display method determination unit 308 holds information regarding a display method associated with an event, for example.

  When an event is detected in the event detection unit 307, the display method determination unit 308 examines a display method corresponding to the detected event from the stored information, and determines the display method.

  In this embodiment, the presence / absence of an event input is determined from the captured image, but as a modification of this embodiment, an observer's line-of-sight input, a physical device attached to a display device, and other remote controllers The presence or absence of an event input may be determined using an input device such as.

  The display method switching unit 304 has a storage unit (not shown) that stores information related to the switched display method while switching the display method according to the determination of the display method determining unit 308. By having the storage unit, the information related to the display method sent from the display method determining unit 308 is compared with the information stored in the storage unit, so that the information related to the display method newly sent from the display method determining unit 308 It can be determined whether or not is the same as the previous display method.

  The reference coordinate determination unit 305 determines a coordinate system serving as a reference for the presentation position of the virtual object to be presented in the three-dimensional virtual space based on the selected display method.

  The data acquisition unit 310 determines a virtual object to be displayed, and acquires virtual object data from the server 205.

  The coordinate calculation unit 306 performs coordinate conversion of the virtual object to be displayed from the measured position and orientation of the observer and the captured video. The UI image generation unit 311 determines the result of the event detection unit and generates various graphic user interfaces. The display image synthesis unit 309 superimposes and synthesizes the graphic user interface generated by the UI image generation unit 311, the coordinate-converted virtual object, and the captured real-world video, and outputs the resultant to the display unit 202.

  The virtual object display device according to the present invention has a plurality of display methods. For example, there are the following four types of display methods.

1. Position fixed display method for displaying a virtual object at a physical position (for example, information obtained by GPS) 2. Displaying a virtual object at a relative position from a specific object (object) such as a marker Object fixed display Method 3, continuous display method that always displays in the viewer's field of view 4. Physical position or display position that displays a virtual object when the observer takes a specific posture without linking to a specific object Fixed display method Here, the specific posture is, for example, up, down, left, right, and rear, and the display method displays the virtual object only when the observer is determined to face up. is there.

  In this embodiment, the description will be made on the assumption that the above four display methods are provided, but of course, it is not necessary to provide all of these four formats. Further, a display method other than the above four display methods may be provided.

  How to determine the reference coordinates for each display method will be described with reference to FIG.

In the figure, the orientation of the sensor coordinate system 401 in the fixed world coordinate system 400 is R _BW , the orientation of the orientation sensor 302 in the sensor coordinate system 401 is R _SB , and the observer's viewpoint 403 viewed from the orientation sensor 302 is relative. the attitude Rvs, the observer's position in the world coordinate system 400 and _{R VW.} Here, the world coordinate system 400 is a coordinate system having the observer position as the origin.

Here, R is a 4 × 4 matrix, and R _BA describes the posture of the object B in the coordinate system A, that is, a transformation matrix from the coordinate system A to B.

The matrix R is defined by a rotation matrix R _X defined by a rotation angle θ around the X axis, R _Y defined by a rotation angle φ around the Y axis, a rotation angle ψ around the Z axis, R _Z , and parallel. It is constituted by the product of _{R T} representing the _movement, the relation _{R = R Z R X R Y} R T is established.

At this time, the viewpoint R _VW of the observer in the world coordinate system 400 can be expressed by R _VW = R _VS · R _SB · R _BW .

In the case of the fixed position display method in which the virtual object is displayed at the physical position, the reference coordinates are placed in the coordinate system 404 with the physical position B as the origin. The observer's line of sight in the coordinate system 404 is expressed as R _VB = R _BW using a transformation matrix R _BW from the coordinate system 401 to the coordinate system 404 obtained using the value of the position A and the value of the position B that can be acquired from the position sensor. _Rvw = R _BW , R _VS , R _SB , R _BW .

  In the case of an object fixed display method in which a virtual object is displayed using a coordinate system with a specific object as an origin, such as a marker, as a reference coordinate, the reference coordinate system is placed in a coordinate system 405 with the center C of the upper surface of the object as the origin. The position C viewed from the viewpoint of the observer can be obtained from an image obtained by the imaging unit 201 using an image recognition technique or the like.

  In the case of the constant display method that always displays in the observer's field of view and the posture fixed display method that displays in accordance with the observer's posture, the reference coordinates are fixed to the line-of-sight coordinate system.

  FIG. 5 shows the field of view of the observer when displayed by the fixed position display method.

  A reference coordinate system is provided with a certain point in the real world as the origin, and a virtual object drawn in this coordinate system is displayed.

  FIG. 6 shows the field of view of the observer when displayed by the object fixed display method.

  A reference coordinate system is provided with an object in the real world as the origin, and a virtual object drawn in this coordinate system is displayed.

  FIG. 7 shows the field of view of the observer when the display is always performed.

  Regardless of the position and orientation of the observer, a virtual object is displayed at a specific position in the observer's field of view.

  FIG. 8 shows the field of view of the observer when displayed by the fixed posture display method. Only when the observer has a specific posture, a virtual object is drawn at a specific position in the field of view of the observer.

  Next, specific processing when drawing a virtual object will be described.

  FIG. 9 is a flowchart for explaining a processing procedure until setting reference coordinates in the first embodiment.

  First, in step S900, the event detection unit 307 monitors whether or not a display method switching event has occurred. When it is detected that an event input has occurred, the display method determination unit 308 determines a display method corresponding to the event that has occurred, and sends information related to the determined display method to the display method switching unit 304. The display method switching unit 304 determines whether the display method stored in the storage unit (not shown) is the same as the current display method (S901). If they are not the same, the display method is changed to the display method determined by the display method determination unit 308, information on the changed display method is stored in the storage unit (not shown), and the reference coordinate determination unit 305 is notified (S902). . When the switched display method is the fixed position display method (Yes in S903), position information for displaying the virtual object is written in the attribute information of the virtual object (S904). The attribute information of the virtual object is described by, for example, XML (extensible Markup Language). When the switched display method is the object fixed display method (Yes in S905), the feature point of the object serving as the reference of the display position in the three-dimensional virtual space is extracted from the imaging data (S906), and the attribute of the virtual object is extracted. The feature point data of the reference object is added to the information (S907). When the switched display method is the fixed posture display method (Yes in S908), the posture of the observer is acquired from the posture sensor 302 (S909), and the acquired posture is added to the virtual object attribute information as a virtual object display posture. (S910). According to the reference coordinate determination flow, a coordinate system for displaying a virtual object is set as a reference coordinate (S911).

  In addition, when all of step S903, step S905, and step S908 are No, the switched display system is a constant display system, and in this case, the process of step S911 is performed.

  As described above, the display method can be controlled based on the gesture of the observer.

  In the above description, the display method corresponding to the gesture of the observer is determined. However, the screen for selecting the display method is displayed on the UI screen by one or more gestures of the observer, and the UI screen is displayed. A configuration may be adopted in which a display mode switching command is input.

  The operation of this apparatus when a display mode switching command is generated via the UI screen will be described below.

  FIG. 10 shows the state transition of the virtual object. The virtual object is in a non-selected state in the initial state. When the event detection unit 307 detects that the user performs an operation of pinching the virtual object by hand, the virtual object transitions to the selected state. When the event detection unit 307 detects an operation of moving the object while maintaining the state in which the user is pinching the virtual object, for example, the same processing as the drag operation on the computer is performed, and the virtual object transitions to the moving state. . Next, when the event detection unit 307 detects an operation in which the user finishes pinching the virtual object and releases the hand, it is determined that the drag process has ended, and the virtual object transitions to the display method selection state. When transitioning to the display method selection state, the event detection unit 307 outputs a command for creating a UI screen to the UI image generation unit 311.

  The UI image generation unit 311 receives the output from the event detection unit 307 and creates a graphic user interface screen (that is, a UI screen) for the observer to select a display method. The UI screen created by the UI image generation unit 311 is synthesized by the display image synthesis unit 309. The synthesized image is displayed on the display unit 202.

  FIG. 11 shows an example of a screen that appears on the display unit in the display method selection state. The graphic user interface displays an icon representing each display method. The user selects a display method from the displayed options. When the display method is selected, the event detection unit 307 detects a display method switching event, and the virtual object transitions to the non-selected state again.

  At this time, the display method determination unit 308 outputs the selected display method to the display method switching unit 308.

  Next, processing from when the apparatus acquires virtual object data until it is displayed on the display unit 202 will be described with reference to FIG. The data acquisition unit 310 acquires the current display method from the display method determination unit (S1200).

  When the acquired display method is the fixed position display method (Yes in S1201), the position / posture acquisition unit 303 acquires the current position Pc of the observer on the earth (S1202). Next, the position information Po of the reference coordinates stored in step S703 is acquired (S1203), and it is determined whether Po is sufficiently close to Pc. Here, the sufficiently close range may be a predetermined fixed value or may be determined for each reference coordinate. If it is determined that Po and Pc are close enough, it is determined that Po is included in the display range (S1204), and virtual object data is acquired from the server 205 (S1213).

  When the acquired display method is the object fixed display method (Yes in S1205), the feature points of the reference coordinates stored in step S906 are acquired. Whether there is a feature point that matches the feature point in the visual field image of the observer acquired by the imaging unit 201 is collated (S1207). If it is determined that the feature points match (Yes in S1208), virtual object data is acquired from the server 205 (S1213).

  When the acquired display method is the fixed posture display method (Yes in S1209), the current posture of the observer is acquired from the position / orientation acquisition unit 303 (S1210). Next, the virtual object display posture Ao stored in the virtual object attribute information in step S909 is acquired (S1211). It is determined whether Ac and Ao are facing the same direction (that is, whether Ac is the display range) (S1212). When facing the same direction (that is, when Ac is the display range). The virtual object data is acquired from the server 205 (S1213). The coordinate calculation unit 306 performs coordinate conversion in a three-dimensional virtual space of a virtual object generated from the acquired data from the reference coordinate system and the position and orientation of the observer acquired by the position and orientation acquisition unit 303. (S1214) The coordinate-converted virtual object is rendered on the display unit (S1215).

(Modification 1)
As a modification of the present embodiment, an augmented reality service using location information may be applied. In an augmented reality service using position information, current position information is transmitted to a server and a virtual object to be displayed is acquired. When a different display method is selected from the fixed position display methods for the virtual object displayed at the position P in the fixed position display method, the position P is simulated to the server and a desired virtual object is acquired in step S1213. You may take the method of doing.

  With this configuration, an augmented reality service using position information can be used even in a display method that is not a fixed position display method.

(Modification 2)
In the present embodiment, processing for displaying one virtual object has been described, but the number of virtual objects is not limited to one, and there may be a plurality of virtual objects. In this case, although the virtual object A is in the position fixing mode, the display method may be changed depending on the virtual object so that the virtual object B is in the posture fixing mode, and the reference coordinates may be determined for each virtual object. .

  With such a configuration, a plurality of virtual objects can be displayed in a superimposed manner by a plurality of display methods, and thus a wider variety of augmented reality can be provided to the observer.

(Modification 3)
Although the present embodiment has been described on the assumption that the display method is switched by an observer input, the switching determination may be automatically determined from the attribute of the virtual object instead of the observer input. For example, if it is a virtual object such as a TODO list, it will always be displayed, if it is a virtual object such as a calendar or clock, it will be displayed when it is turned to the left. If the virtual object is related to a fixed display method or real estate advertisement, the position fixed display method is automatically determined.

  An example of processing for automatically determining the display method will be described below.

  In the attribute information of the virtual object, there are as many flags regarding the display method as the number of display methods, and each of the display methods corresponds. The display method flag is exclusive. When a flag of a certain display method is set to 1 (TRUE), the other flags are set to 0 (FALSE). This flag can be set, for example, when adding or editing a virtual object. The display method determination unit 308 acquires a display method flag group from the virtual object attribute information, and determines the display format in which the flag is set to 1 as the display format of the virtual object. After the display method is determined, the same processing as that after step S901 is performed.

  With this configuration, it is possible to automatically display the most appropriate display format corresponding to the characteristics of the virtual object while reducing the troublesomeness of the observer.

(Modification 4)
In this embodiment, the description has been made on the premise that there are four display methods: a fixed position display method, an object fixed display method, a constant display method, and a fixed posture display method. I do not care. Further, a display method other than the above four display methods may be provided.

  For example, there are other person's line-of-sight display methods that receive a third party's view image, posture position, position information, etc. and display a virtual object displayed in the third party's view also in their own view. May be.

  With such a configuration, it is possible to obtain the field of view information of another person in another place and perform a simulated experience or the like.

  According to the virtual object display device of the present invention, it is possible to operate a virtual object easily and with a high degree of freedom. Expected to be applied to coupon services using augmented reality, communication services, games in a virtual space where items are collected, and the like.

DESCRIPTION OF SYMBOLS 101 Posture sensor 102 Posture information output part 103 Memory 105 Image generation part 106 Display image 200 Display apparatus 201 Imaging part 202 Display part 203 Sensor group 204 System control part 205 Server 301 Position sensor 302 Posture sensor 303 Position and posture acquisition part 304 Display system switching Unit 305 Reference coordinate determination unit 306 Coordinate calculation unit 307 Event detection unit 308 Display method determination unit 309 Display image synthesis unit 310 Data acquisition unit 311 UI image generation unit

Claims (9)

A position and orientation acquisition unit for acquiring the position and orientation of the observer;
A display method determination unit for determining one display method from a plurality of image display methods;
A display method switching unit that switches the display method to the display method determined by the display method determining unit;
Based on the display method determined by the display method determination unit, a reference coordinate system determination unit that determines a coordinate system serving as a reference of a display position in a three-dimensional virtual space of a virtual object to be displayed;
A data acquisition unit for acquiring data of the virtual object;
From the reference coordinate system determined by the reference coordinate system determination unit and the position and orientation of the observer acquired by the position and orientation acquisition unit, coordinate conversion of the virtual object to be displayed in a three-dimensional virtual space is performed. A coordinate calculator to perform,
The virtual object data acquired by the data acquisition unit is superimposed on a predetermined position in the three-dimensional virtual space coordinate-converted by the coordinate calculation unit by the coordinate system determined by the reference coordinate system determination unit. Display screen composition unit to be displayed on the display unit,
A virtual object display device comprising:   The virtual object display device according to claim 1, wherein the display method determination unit determines a display method according to an event input from the observer.   The virtual object display device according to claim 2, wherein the display method determination unit displays a graphic user interface for the observer to select an image display method.   The virtual object display device according to claim 1, wherein the display method determination unit determines a display method according to characteristics of the virtual object to be displayed.   5. The virtual object display device according to claim 1, wherein one of the plurality of image display methods displays the virtual object at a position of an absolute value such as latitude, longitude, and altitude on the earth. .   The one of the plurality of image display methods displays the virtual object at a relative position from the specific object when the real or virtual specific object is in the field of view of the observer. The virtual object display device according to any one of the above.   The one of the plurality of image display methods always displays the virtual object at a specific position in the field of view of the observer regardless of the position and orientation of the observer. Virtual object display device.   One of the plurality of image display methods displays the virtual object when the posture of the observer matches a specific posture regardless of the position of the observer. The virtual object display device according to Item. The data of the virtual object acquired by the data acquisition unit includes attribute information for determining a display method corresponding to the virtual object,
The virtual object display device according to claim 1, wherein the display method determination unit automatically determines the display method using the attribute information.

Images