JP2010282497A - Different-world state reflection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for comparatively easily achieving environmental construction while guaranteeing a sense of presence as much as possible. <P>SOLUTION: A different-world state reflection device includes: an imaging information acquisition part 111 for acquiring image information of a second object as a source of reflecting motion of a first object in a closed space on a real world imaged by an imaging device from the imaging device; a positional information acquisition part 112 for acquiring the positional information of the first object in the closed space; a three-dimensional shape information preparation part 113 for preparing three-dimensional shape information corresponding to the second object from the acquired image information and positional information by using coordinate axes of a virtual world corresponding to the coordinate axes of the closed space; a texture preparation part 114 for preparing a texture corresponding to the second object from the acquired image information and positional information by using coordinate axes in the virtual world; and a transmission part 115 for transmitting the positional information, the three-dimensional shape information and the texture to a virtual space control device. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention is capable of communicating with a virtual space control device for displaying computer graphics including an avatar expressed from three-dimensional shape information and texture on a user terminal and a first object in the real world. The present invention relates to a different world state reflection device that reflects a state in a virtual world state.

  Conventionally, when one state is transmitted to the other as in the case of communication between remote locations, a telephone or videophone dedicated to a call has been used. In particular, when using a videophone, the state of the other party can be grasped visually, so that a more realistic state can be transmitted.

  On the other hand, humans in different real spaces as remote places wear special virtual object stereoscopic glasses and communicate while watching images of the other party's stereoscopic virtual objects (such as avatars). The system which can perform is disclosed (for example, refer to patent documents 1). In the method of Patent Document 1, information on the position and change of an object (such as a person or a table) obtained from an optical position sensor installed in each real space, and an image of a virtual object (such as an avatar) prepared in advance. Based on the above, the video of the virtual object corresponding to the position and change of the object on the own side is sent to the virtual object stereoscopic glasses on the other side. Thereby, the virtual object stereoscopic vision glasses on the other side can be displayed in a state where the virtual object that is an image corresponding to the object in the real space on the other side in addition to the real space on the other side is superimposed.

  However, in a system such as that disclosed in Patent Document 1, information on the position and change of an object in each real space at a remote location is acquired from an optical position sensor provided in each real space. For this reason, the virtual object stereoscopic glasses on the other side only display the image of the virtual object prepared in advance, not the real image of the own side. In this way, the virtual object corresponding to the real space object on the other side is superimposed and displayed on the other side's real space, but the presence is increased compared to the video phone, but the other side is There is a problem that a sense of reality is reduced as compared with a video phone in that a real video of the object is not displayed.

  Further, in the system as shown in Patent Document 1, since it is necessary to construct an environment such as a room where optical position sensors are installed at remote locations, it is easy to construct an environment for transmitting the state. The problem is not.

  The present invention has been made in view of the above circumstances, and provides a system that can realize environment construction relatively easily while ensuring the sense of reality as much as possible in transmitting the state between remote locations to each other. Objective.

  In order to solve the above-described problems and achieve the object, the present invention provides a virtual space control device for displaying computer graphics including an avatar represented from three-dimensional shape information and texture on a user terminal and in the real world. A different world state reflecting device that is communicable with a first object and reflects a state of a predetermined closed space in the real world in a state of a virtual world, the first object being photographed by a photographing device Imaging information acquisition means for acquiring image information of a second object that is a reflection source of movement from the imaging device, and first object position information indicating the position of the first object in the closed space is the first object. Using the position information acquisition means acquired from the object and the coordinate axes in the virtual world corresponding to the coordinate axes set in the closed space, Using 3D shape information creating means for creating 3D shape information corresponding to the second object from the image information of the second object and the first object position information, and the coordinate axes in the virtual world, Texture creating means for creating a texture corresponding to the second object from the image information of the second object and the first object position information, the first object position information, the three-dimensional shape information, Transmission means for transmitting the texture to the virtual space control device.

  According to the present invention, there is an effect that it is possible to construct an environment relatively easily while ensuring a sense of reality as much as possible when transmitting states between remote locations to each other.

FIG. 1 is a block diagram illustrating a configuration example of a state reflection system. FIG. 2 is a schematic diagram for explaining the relationship between the real space and the virtual space related to the state reflection system. FIG. 3 is a hardware configuration diagram of an example of the information providing server. FIG. 4 is a schematic diagram illustrating an example of correspondence between augmented reality and virtual reality.

  Exemplary embodiments of a different world state reflecting apparatus according to the present invention will be explained below in detail with reference to the accompanying drawings.

  Hereinafter, the state reflection server 110 to which the different world state reflection apparatus of the present invention is applied will be described. However, for easy understanding of the overall outline, the different world state reflection system including the state reflection server 110 is illustrated in FIG. This will be described with reference to FIG.

  FIG. 1 is a block diagram illustrating a configuration example of a state reflection system. FIG. 2 is a schematic diagram for explaining the relationship between the real space and the virtual space related to the state reflection system.

  As shown in FIG. 1, the different world state reflection system includes a client office 200 as a predetermined closed space in the real world (real space), an information providing server 100, and a PC (Personal Computer) 300 as a user terminal, They are connected in a state where they can communicate with each other via a network 400 such as the Internet or an intranet.

  First, the configuration of each device provided in the client office 200 will be described. The client office 200 includes a camera 210 as an imaging device, a base station 220, a reference position signal transmitter 230, and an HMD (head mounted display) 240 as a first object. As shown in FIG. 2, the client office 200 includes a user 22 as a second object. This user 22 is equipped with the HMD 240.

  The second object is not limited to a human being, and various objects such as other life forms such as dogs and cats, furniture such as chairs and desks, and furniture can be applied. Further, the first object is not limited to the HMD 240, and other communication devices such as RFID that can acquire the position information of the object can be applied.

  A plurality of cameras 210 are provided in the client office 200 (see FIG. 2). The camera 210 associates imaging information input by imaging the client office 200 with the time (imaging time) when the imaging information was captured, and sequentially transmits the information to the information providing server 100 via the network 400. Furthermore, the camera 210 also transmits a camera ID as image capturing apparatus specifying information for specifying the camera 210 to the information providing server 100 via the network 400.

  The base station 220 receives the position information and direction information (position / direction information) of the HMD 240 in the client office 200 from the HMD 240 and transmits the received position / direction information to the information providing server 100 via the network 400.

  The position information is information representing the position of the HMD 240 on a predetermined coordinate axis (coordinate axis 21 in FIG. 2) with a predetermined position in the client office 200 as the origin. The direction information is information representing the direction in which the HMD 240 faces in the coordinate system represented by the coordinate axis 21. For example, the forward direction of the user 22 equipped with the HMD 240 can be used as the direction information.

  The reference position signal transmitter 230 transmits a reference position signal for enabling the HMD 240 to detect the position direction information of the HMD 240 in the client office 200. For example, the reference position signal transmitter 230 is configured to transmit a reference position signal similar to a signal transmitted from a GPS satellite. As a result, a position / direction information detection unit 242 described later can detect the position and direction of the HMD 240 by the same method as that of the GPS function.

  The HMD 240 includes a display unit 241, a position / direction information detection unit 242, and a position / direction information transmission unit 243.

  The display unit 241 displays the image information received from the information providing server 100.

  The position / direction information detection unit 242 detects the position information and direction information of the HMD 240 in the client office 200 using the reference position signal received from the reference position signal transmitter 230. For example, the position / direction information detection unit 242 detects position information and direction information based on the phase of a carrier wave representing a reference position signal transmitted from a plurality of reference position signal transmitters 230.

  As described above, the position / direction information detection unit 242 can detect the position information and the direction information by a function similar to the GPS function. That is, in the case of normal GPS, position information and direction information are detected using a signal transmitted from a satellite. The position / direction information detection unit 242 according to the present embodiment transmits a reference position signal instead of a satellite. By cooperating with the machine 230, position information and direction information of the HMD 240 are detected.

  Note that the method of detecting the position / direction information is not limited to the above, and any conventionally used method can be applied as long as the method can detect the position and direction of the HMD 240. For example, the position / direction information may be detected by analyzing image information captured by a plurality of cameras 210.

  The position / direction information transmission unit 243 transmits the position information and direction information of the HMD 240 in the client office 200 detected by the position / direction information detection unit 242 to the base station 220. In addition, what is necessary is just to transmit direction information as needed.

  Here, the HMD 240 sequentially detects the position and direction information, and sequentially transmits the detected position and direction information to the base station 220, whereby the position and direction information of the HMD 240 is transmitted to the information providing server 100 in real time via the base station 220. input.

  Since the HMD 240 is attached to the user 22, the position / direction information of the HMD 240 can be restated as the position information of the user 22. Further, since the HMD 240 also operates in conjunction with the user 22, the user 22 can be said to be a reflection source of movement in the HMD 240.

  Next, the configuration of the information providing server 100 will be described. The information providing server 100 includes a state reflection server 110 as a different space state reflection device and a virtual space control server 120 as a virtual space control device.

  The state reflection server 110 includes an imaging information acquisition unit 111, a position information acquisition unit 112, a three-dimensional shape information generation unit 113, a texture generation unit 114, a transmission unit 115, a direction information acquisition unit 116, and a request unit 117. And.

  The imaging information acquisition unit 111 acquires imaging information and imaging time taken by the camera 210 via the network 400. Furthermore, the imaging information acquisition unit 111 also acquires a camera ID. Since a plurality of cameras 210 are provided, the imaging information acquisition unit 111 acquires imaging information and imaging time from each camera 210. Therefore, the shooting information acquisition unit 111 acquires captured images obtained by shooting the user 22 from a plurality of directions from the camera 210 via the network 400.

  The photographing information acquisition unit 111 is connected to an initial image storage unit (not shown). The initial image storage unit stores a captured image (hereinafter referred to as “initial image”) of the client office 200 in a state where the user 22 exists for each camera 210.

  Then, for each acquired camera 210, by taking the difference between the initial image stored in the initial image storage unit and the imaging information acquired from the camera 210, the imaging information acquisition unit 111 causes the image information of the second object to be acquired. The image information of the user 22 is extracted. Therefore, it can be said that the photographing information acquisition unit 111 is executing processing for acquiring image information of the second object from the camera 210.

  Note that the HMD 240 may execute the process of detecting the difference between the initial image and the captured image and extracting the image information of the user 22. Further, a separate image information extraction unit may be provided between the HMD 240 and the photographing information acquisition unit 111, and the image information extraction unit may execute the image information extraction unit.

  The position information acquisition unit 112 acquires the position information of the HMD 240 from the HMD 240 via the base station 220.

  The three-dimensional shape information creation unit 113 creates the three-dimensional shape information of the user 22 by the method described in Patent Document 2, for example. Here, an outline of processing in the three-dimensional shape information creation unit 113 will be described. First, the three-dimensional shape information creation unit 113 acquires the image information of the user 22 acquired by the shooting information acquisition unit 111 from the shooting information acquisition unit 111 for each camera 210. Then, the three-dimensional shape information creating unit 113 creates the three-dimensional shape information of the user 22 by extracting feature points from each image information. The three-dimensional shape information creating unit 113 may create a polygon as the three-dimensional shape information, or may be configured to create other three-dimensional shape information.

  The texture creation unit 114 creates a texture to be pasted on the three-dimensional shape information of the user 22 by, for example, the method described in Patent Document 2. Here, an outline of texture creation will be described. The texture creation unit 114 sets the image information of the user 22 acquired by the imaging information acquisition unit 111 as a texture corresponding to the direction of the 3D shape information created by the 3D shape information creation unit 113. That is, when the three-dimensional shape information facing the direction A is created, the texture to be pasted on the three-dimensional shape information is a captured image taken of the user 22 facing the direction of A-180 °.

  The transmission unit 115 transmits the 3D shape information created by the 3D shape information creation unit 113, the texture created by the texture creation unit 114, and the location information of the HMD 240 obtained by the location information acquisition unit 112 to a virtual space described later. It transmits to the control server 120. Here, as described above, since the position information of the HMD 240 can be rephrased as the position information of the user 22, the transmission unit 115 transmits the three-dimensional shape information, the texture, and the position information of the user 22 to the virtual space control server 120. I can say that.

  Through the above processing, the virtual space control server 120 described later can display an object expressed from the three-dimensional shape information and the texture in the three-dimensional virtual space according to the transmitted position information. That is, it is possible to place an object expressed from the transmitted three-dimensional shape information and texture at a position determined by the transmitted position information and the coordinate axis of the virtual world (virtual space). Therefore, the real world state can be reflected in the virtual world.

  In addition, when a living object is used as the second object, a texture is created using imaging information sequentially transmitted from the camera 210, so that the expression of the living object is sequentially displayed in the three-dimensional virtual space. , More advanced communication is possible.

  Note that the coordinate axes of the virtual space are determined to correspond to the coordinate axes of the client office 200 in the real space. In FIG. 2, the coordinate axis 31 of the virtual space 30 corresponds to the coordinate axis 21 of the client office 200 in the real space. A virtual space 30 in FIG. 2 schematically shows a virtual space generated by a virtual space generation unit 122 (details will be described later) of the virtual space control server 120 that generates the virtual space.

  FIG. 2 shows an example in which objects such as chairs, desks, and room partitions in the real space are displayed in the virtual space 30 as they are. For this reason, in the example of FIG. 2, the user 32 of the virtual world reflecting the state of the user 22 of the real world can be displayed in the virtual world (virtual space 30).

  Next, a configuration for reflecting the state of the virtual space in the real space will be described.

  The direction information acquisition unit 116 acquires the direction information of the HMD 240 from the HMD 240 via the base station 220.

  The request unit 117 transmits the acquired direction information of the HMD 240 and the position information of the HMD 240 acquired by the position information acquisition unit 112 to the virtual space control server 120, thereby causing the computer to the virtual space control server 120. Request graphic data.

  Here, computer graphic data expressed in a virtual space can be roughly divided into two types. One is objects such as backgrounds and buildings in a virtual space, and avatars such as people and animals. The avatar is a character that can be operated by the user via the PC 300 in the virtual space. FIG. 2 shows an example in which an avatar 33 that imitates an animal is displayed in the virtual space 30.

  The request unit 117 makes a transmission request for computer graphics of avatars among computer graphics.

  Next, details of the configuration of the virtual space control server 120 will be described. The virtual space control server 120 is a server for providing a three-dimensional virtual space, and is connected to the PC 300 via the network 400. The virtual space control server 120 includes a request reception unit 121, a virtual space generation unit 122, and an image transmission unit 123.

  The request receiving unit 121 receives visual field information and position information in the virtual space from the PC 300. The visual field information is information that represents the line-of-sight direction of the user of the PC 300 in the virtual space. For example, the visual field information is changed by a method called mouse look that moves the visual field in the virtual space by operating the mouse. The request reception unit 121 that has received the visual field information and the position information inputs the visual field information and the position information to the virtual space generation unit 122.

  The virtual space generation unit 122 generates computer graphics of a virtual space including avatars according to the visual field information and the position information. The virtual space generation unit 122 generates computer graphics that should be expressed at a position separated from the input position information by a predetermined distance, and that should exist in an area corresponding to the input visual field information. .

  The image transmission unit 123 displays the three-dimensional virtual space on the display of the PC 300 by transmitting the generated computer fix to the PC 300.

  Each process in the virtual space control server 120 can be realized by the same method as the provision of a three-dimensional virtual space in SecondLife (registered trademark) or the like.

  The request reception unit 121 further receives the position information and direction information of the HMD 240 from the state reflection server 110. Then, the request reception unit 121 that has received the position information and direction information of the HMD 240 inputs the position information and direction information to the virtual space generation unit 122.

  The virtual space generation unit 122 determines whether to generate computer graphics of a virtual space to be displayed on the PC 300 or to generate computer graphics of an avatar to be displayed on the display unit 241 of the HMD 240 based on the input information or the like. To do. The virtual space generation unit 122 receives, for example, request source information for determining the request source from the request reception unit 121, and the computer graphic of the avatar displayed on the display unit 241 when the request source information represents the state reflection server 110 It is determined that the When it is determined that the computer graphics of the avatar to be displayed on the display unit 241 is to be generated, the virtual space generation unit 122 is a computer graphic to be expressed at a position separated from the input position information by a predetermined distance. The computer graphics of the avatar which should exist in the area | region according to the direction information made is produced | generated.

  The virtual space generation unit 122 transmits the generated computer graphics of the avatar to the state reflection server 110 that is the request source. Then, the transmission unit 115 of the state reflection server 110 transmits the generated avatar image to the HMD 240 via the network 400.

  With the above processing, the avatar present in the three-dimensional virtual space can be displayed on the display unit 241 of the HMD 240 according to the direction information of the HMD 240. Then, the user 22 existing in the real world through the HMD 240 can visually recognize as if the avatar existing in the three-dimensional virtual space exists in the real world. In the example of FIG. 2, the avatar 23 corresponding to the avatar 33 in the virtual space 30 is displayed on the display unit 241 of the HMD 240 as if it exists in the client office 200.

  Next, the hardware configuration of the information providing server 100 will be described with reference to FIG. FIG. 3 is a hardware configuration diagram of an example of the information providing server 100.

  As shown in FIG. 3, the information providing server 100 includes an input device 11, a display device 12, a drive device 13, a storage medium 14, and a ROM (Read Only Memory) 15, which are mutually connected by a bus. And a RAM (Random Access Memory) 16, a CPU (Central Processing Unit) 17, an interface device 18, and an HDD (Hard Disk Drive) 19.

  The input device 11 includes a keyboard and a mouse operated by a user of the information providing server 100 and is used to input various operation signals to the information providing server 100.

  The display device 12 includes a display used by the user of the information providing server 100 and displays various types of information.

  The interface device 18 is an interface that connects the information providing server 100 to a network or the like.

  A program or the like of the information providing server 100 that performs various processes executed on the information providing server 100 is provided to the information providing server 100 by the storage medium 14 such as a CD-ROM or downloaded through the network. The storage medium 14 is set in the drive device 13, and the program and the like are installed in the ROM 15 from the storage medium 14 via the drive device 13.

  The ROM 15 stores data, the program, and the like. The RAM 16 reads and stores the program and the like from the ROM 15 when the information providing server 100 is activated. The CPU 17 executes processing in accordance with the program read out and stored in the RAM 16.

  The HDD 19 stores data and files.

  Note that the hardware configuration of the PC 300 is the same as the hardware configuration of the information providing server 100 shown in FIG. In the above description, the example in which the state reflection server 110 and the virtual space control server 120 are provided in the information providing server 100 has been described. However, the state reflection server 110 and the virtual space control server 120 are realized as physically different devices. May be. In this case, the state reflection server 110 and the virtual space control server 120 can be configured to have a hardware configuration as shown in FIG.

  Next, a specific example of the relationship between the real world (augmented reality) and the virtual world (virtual reality) according to the present embodiment will be described with reference to FIG. FIG. 4 is a schematic diagram illustrating an example of correspondence between augmented reality and virtual reality. Avatars appear in augmented reality and people can work in the metaverse, and events that happen in either the real world or the metaverse will always happen on the other side. An example is shown in FIG. The real world space and the virtual reality space created by copying it have common space coordinates, and the real world person 401 can see the avatar 402 in the real world by the HMD 240. Also, the user of the avatar can see the video of the person 403 in the real world in the virtual reality space on the display of the PC 300.

11 Input device 12 Display device 13 Drive device 14 Storage medium 15 ROM
16 RAM
17 CPU
18 Interface device 19 HDD
21, 31 Coordinate axes 22, 32 User 23, 33 Avatar 30 Virtual space 100 Information providing server 110 State reflection server 111 Imaging information acquisition unit 112 Position information acquisition unit 113 Three-dimensional shape information generation unit 114 Texture generation unit 115 Transmission unit 116 Direction information Acquisition unit 117 Request unit 120 Virtual space control server 121 Request reception unit 122 Virtual space generation unit 123 Image transmission unit 200 Client office 210 Camera 220 Base station 230 Reference position signal transmitter 241 Display unit 242 Position / direction information detection unit 243 Position / direction information Transmitter 400 network

JP 2002-305758 A JP 2001-022936 A

Claims (4)

A virtual space control device for displaying computer graphics including avatars expressed from three-dimensional shape information and textures on a user terminal and a first object in the real world can communicate with each other, and a predetermined closed space in the real world A different world state reflection device that reflects the state of the world in the state of the virtual world,
Shooting information acquisition means for acquiring image information of the second object, which is a reflection source of the movement of the first object, shot by the shooting device;
Position information acquisition means for acquiring first object position information indicating the position of the first object in the closed space from the first object;
Using the coordinate axes in the virtual world corresponding to the coordinate axes set in the closed space, the three-dimensional shape information corresponding to the second object is obtained from the image information of the second object and the first object position information. 3D shape information creation means to create,
Texture creating means for creating a texture corresponding to the second object from the image information of the second object and the first object position information using coordinate axes in the virtual world;
Transmitting means for transmitting the first object position information, the three-dimensional shape information, and the texture to the virtual space control device;
A different world state reflecting device characterized by comprising: First object method information acquisition means for acquiring, from the first object, first object direction information indicating the direction of the first object on the coordinate axis set in the closed space;
Requesting means for transmitting the first object position information and the first object direction information to the virtual space control device and requesting the virtual space control device for computer graphics data of the avatar. And
2. The difference according to claim 1, wherein the transmission unit transmits the computer graphics data of the avatar obtained from the virtual space control device to the first object by requesting the request unit. World state reflection device.   3. The different world state reflecting apparatus according to claim 1, wherein the second object is a life form, and the first object is a head mounted display that can be attached to the life form.   4. The different world state reflecting apparatus according to claim 1, wherein the second object is a fixture, and the first object is a communication device that can be attached to the fixture. 5.

Images