JP6794390B2 - Simulation system and program - Google Patents

Description

The present invention relates to a simulation system, a program, and the like.

Conventionally, a system that allows a user to experience the world of so-called virtual reality (VR) by wearing an HMD (head-mounted display device) on the head and viewing the image displayed on the HMD screen has been known. Has been done. As a conventional technique of a system using such an HMD, for example, there is a technique disclosed in Patent Documents 1 and 2.

Patent Document 1 discloses a method of opening the liquid crystal shutter of the HMD to switch the visual information displayed to the user to the surrounding environment visual information when the fluctuation state of the user's head becomes a predetermined fluctuation state. Has been done. In Patent Document 2, the motion detection unit identifies the user's head motion, executes the process desired by the user corresponding to the angular velocity, and does not reflect the return motion of the user's head in the process. A method for accurately performing an operation by head movement is disclosed.

Japanese Unexamined Patent Publication No. 7-261112 Japanese Unexamined Patent Publication No. 2012-123812

In the system using the HMD, the image seen from the virtual camera in the virtual space is displayed on the HMD. By displaying such an image on the HMD, a vast VR space spreads over the entire circumference of the user's field of view, so that the user's virtual reality can be significantly improved.

On the other hand, in a system using an HMD, it is desirable to display various information such as explanatory information and status information to the user.

However, if such information is not properly displayed to the user, it causes various problems. For example, in a system using an HMD, the line-of-sight direction of the user changes in various ways, and a large number of objects exist in the VR space. Therefore, if the information display to the user is inappropriate, the screen becomes difficult to see and the user's virtual reality is impaired.

According to some aspects of the present invention, it is possible to provide a simulation system, a program, or the like that can realize an appropriate display process of an information display object in a system using a head-mounted display device.

One aspect of the present invention is a virtual space setting unit that performs setting processing of a virtual space in which a plurality of objects are arranged, and a virtual that performs control processing of a virtual camera corresponding to the viewpoint of a user wearing a head-mounted display device. The virtual space setting unit includes a camera control unit and a display processing unit that generates an image that can be seen from the virtual camera in the virtual space as a display image of the head-mounted display device, and the virtual space setting unit changes the viewpoint of the user. At least one information display object controlled to follow the above is arranged in the virtual space, and the display processing unit receives when the information display object and the object in the virtual space have a given positional relationship. , The information display object or the simulation system for changing the display mode of the object. The present invention also relates to a program that causes a computer to function as each of the above parts, or a computer-readable information storage medium that stores the program.

According to one aspect of the present invention, a virtual space setting process in which a plurality of objects are arranged is performed, and an image seen from a virtual camera corresponding to the user's viewpoint is generated as a display image of the head-mounted display device. To. Then, an information display object controlled to follow the user's viewpoint change (change in at least one of the line-of-sight direction and the viewpoint position) is arranged in the virtual space, and the information display object and the object in the virtual space have a given positional relationship. When becomes, the process of changing the display mode of the information display object or the object is performed. In this way, even when the information display object controlled to follow the change in the viewpoint of the user and the object have a given positional relationship, the display mode of the information display object or the object is changed. Therefore, it is possible to prevent an inappropriate image from being displayed on the head-mounted display device. Therefore, it is possible to provide a simulation system or the like that can realize appropriate display processing of information display objects in a system using a head-mounted display device.

Further, in one aspect of the present invention, the display processing unit may change the display mode according to the type, priority or importance of the information display object.

In this way, it is possible to realize an appropriate display mode change process according to the type, priority or importance of the information display object.

Further, in one aspect of the present invention, when the information display object is the first type, the display processing unit changes the display mode of the object, and the information display object is the second type. In some cases, the display mode of the information display object may be changed.

By doing so, it is possible to realize an appropriate display process of the information display object by changing one of the display modes of the information display object and the object according to the type of the information display object.

Further, in one aspect of the present invention, the display processing unit may perform the change processing of the display mode by using the priority or the importance associated with the information display object.

In this way, the priority or importance can be associated with the information display object in advance, and the appropriate display mode change process can be executed using this priority or importance.

Further, in one aspect of the present invention, the display processing unit may change the display mode according to the elapsed time from the display of the information display object.

By doing so, it becomes possible to realize the process of changing the display mode that reflects the elapsed time since the information display object is displayed.

Further, in one aspect of the present invention, the display processing unit performs processing for erasing the information display object or the object, processing for making the information display object or the object translucent, and the line of sight of the user as processing for changing the display mode. The process of canceling the tracking of the information display object in the direction, the process of moving the information display object or the object, the process of resizing the information display object or the object, or the process of changing the color information of the information display object or the object. You may go.

In this way, even when the information display object and the object have a given positional relationship, the information display object or the object can be erased or made translucent, or the tracking of the information display object can be canceled. By performing, moving or resizing the information display object or object, or changing the color information of the information display object or object, it becomes possible to realize an appropriate display process of the information display object.

Further, in one aspect of the present invention, the information display object is a display object for explanatory display, a display object for status display, a display object for conversation display, a display object for command selection, or a menu display. It may be a display object for.

In this way, the explanation display, the status display, the conversation display, the command selection, the menu display, and the like can be realized by using the information display object.

Further, in one aspect of the present invention, the object may be a moving object, a fixed object, an item, or a background object appearing in the game.

In this way, even when an object such as a moving object, a fixed object, an item, or a background appearing in the game and the information display object have a given positional relationship, appropriate display processing of the information display object can be performed. It will be possible.

Further, in one aspect of the present invention, a determination unit for determining the positional relationship between the information display object and the object is included (a computer functions as a determination unit), and the display processing unit is combined with the information display object by the determination unit. When it is determined that the object and the object have the given positional relationship, the display mode may be changed.

In this way, the determination unit can determine the positional relationship between the information display object and the object, and can execute the process of changing the display mode of the information display object or the object based on the determination result.

Further, in one aspect of the present invention, the display processing unit changes the display mode when the determination unit determines that the information display object and the object have a collision positional relationship or an approaching positional relationship. Processing may be performed.

By doing so, even when the information display object and the object collide or approach each other and a problem occurs in the display image, the display mode of the information display object or the object can be changed. , Such a problem can be prevented.

Further, in one aspect of the present invention, the determination unit projects the information display object and the three-dimensional coordinate value of the object, the information display object and the depth value of the object, or the information display object and the object on a plane. The positional relationship may be determined based on the two-dimensional coordinate values of the case.

In this way, the positional relationship between the information display object and the object is used by using the three-dimensional coordinate values and depth values of the information display object and the object, or the two-dimensional coordinate values when the information display object and the object are projected on a plane. Is determined, and the display mode change process can be executed.

Further, in one aspect of the present invention, the virtual space setting unit may arrange the information display object at a given depth position in the line-of-sight direction of the user so as to follow a change in the viewpoint of the user.

In this way, even if the user's line-of-sight direction or viewpoint position changes, the information display object can be arranged and displayed at a given depth position in the user's line-of-sight direction so as to follow the change. become. Therefore, it is possible to prevent a situation in which the virtual reality of the user is impaired or the displayed image becomes difficult to see.

Further, in one aspect of the present invention, the virtual camera control unit may control the virtual camera so as to follow the change in the viewpoint of the user based on the tracking information of the viewpoint information of the user.

In this way, for example, the head-mounted display device displays the image seen from the virtual camera by changing the line-of-sight direction and the viewpoint position of the virtual camera in the virtual space in accordance with the change in the viewpoint of the user in the real world. It will be possible to display in.

Further, in one aspect of the present invention, a game processing unit that performs game processing is included (a computer functions as a game processing unit), and the game processing unit includes information on a character that is the target of game play by the user and the user's information. A process of invoking the effect of the command corresponding to the viewpoint information may be performed.

In this way, it is possible to realize the process of invoking the effect of the command that reflects the relationship between the character and the user, and it is possible to realize an unprecedented type of game.

The block diagram which shows the configuration example of the simulation system of this embodiment. 2 (A) and 2 (B) are examples of the HMD used in this embodiment. 3 (A) and 3 (B) are other examples of the HMD used in this embodiment. An example of a game image generated by this embodiment. An example of a game image generated by this embodiment. An example of a game image generated by this embodiment. Explanatory drawing about the display of the status of a character. 8 (A) to 8 (D) are explanatory views of a display method of an information display object. 9 (A) and 9 (B) are explanatory views of the collision between the information display object and the object. 10 (A) to 10 (C) are explanatory views of the display mode changing method of the present embodiment. 11 (A) and 11 (B) are explanatory views of a display mode changing method according to the type, priority, and importance of the information display object. An explanatory diagram of a display mode changing method according to the elapsed time from the display of an information display object. 13 (A) to 13 (C) are specific examples of the display mode changing process of the information display object. 14 (A) to 14 (C) are specific examples of the display mode changing process of the information display object. 15 (A) and 15 (B) are specific examples of the positional relationship determination method. 16 (A) and 16 (B) are specific examples of the positional relationship determination method. 17 (A) and 17 (B) are specific examples of the positional relationship determination method. 18 (A) and 18 (B) are explanatory views of the process of invoking the effect of the command according to the positional relationship information between the user and the character. 19 (A) and 19 (B) are explanatory views of the process of invoking the effect of the command according to the positional relationship information between the user and the character. 20 (A) and 20 (B) are explanatory views of the process of invoking the effect of the command according to the line-of-sight relationship information between the user and the character. Explanatory drawing of the activation processing of the effect of a command according to the gaze information of a user. The flowchart which shows the detailed processing example of this embodiment.

Hereinafter, this embodiment will be described. The present embodiment described below does not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present invention.

1. 1. Simulation system FIG. 1 is a block diagram showing a configuration example of a simulation system (simulator, game system) of the present embodiment. The simulation system of the present embodiment is, for example, a system that simulates virtual reality (VR), and provides a game system that provides game content, a real-time simulation system such as a sports competition simulator or a driving simulator, and content that provides content such as video. It can be applied to various systems such as a system or an operating system that realizes remote work. The simulation system of the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting a part of its constituent elements (each part) or adding other constituent elements are possible.

The imaging unit 150 is composed of one or a plurality of cameras. Each camera is composed of an optical system such as a lens (wide-angle lens) and an image sensor such as a CCD or CMOS sensor. Further, the imaging unit 150 may be provided with one or more microphones. By using the imaging unit 150, it is possible to detect user movement information (movement information of each part, skeleton information) and realize user recognition processing (face recognition, etc.). Further, by providing the imaging unit 150 with a plurality of cameras, it is possible to recognize the depth direction of the space and determine the context of two users playing in front of the imaging unit 150. Further, by providing a plurality of microphones in the imaging unit 150, the direction of the sound source and the like can be detected. For example, by using the imaging unit 150 provided with a plurality of cameras and a plurality of microphones, it is possible to realize a game in which the user's movements and voices can be intuitively played as operation information.

The operation unit 160 is for the user (player) to input various operation information (input information). The operation unit 160 can be realized by various operation devices such as operation buttons, turn signal keys, joysticks, handles, pedals or levers.

The storage unit 170 stores various types of information. The storage unit 170 functions as a work area such as the processing unit 100 and the communication unit 196. The game program and game data necessary for executing the game program are stored in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), an HDD (hard disk drive), an SSD, an optical disk device, or the like. The storage unit 170 includes an object information storage unit 172 and a drawing buffer 178.

The information storage medium 180 (a medium that can be read by a computer) stores programs, data, and the like, and its function can be realized by an optical disk (DVD, BD, CD), an HDD, a semiconductor memory (ROM), or the like. .. The processing unit 100 performs various processes of the present embodiment based on the program (data) stored in the information storage medium 180. That is, the information storage medium 180 is a program for causing a computer (a device including an input device, a processing unit, a storage unit, and an output unit) to function as each part of the present embodiment (a program for causing the computer to execute processing of each part). Is memorized.

The HMD200 (head-mounted display device) is a device that is worn on the user's head and displays an image in front of the user's eyes. The HMD 200 is preferably a non-transparent type, but may be a transparent type. Further, the HMD 200 may be a so-called eyeglass type HMD.

The HMD 200 includes a sensor unit 210, a display unit 220, and a processing unit 240. The sensor unit 210 is for realizing tracking processing such as head tracking. For example, the position and direction of the HMD 200 are specified by a tracking process using the sensor unit 210. By specifying the position and direction of the HMD 200, it is possible to specify the viewpoint position and the line-of-sight direction, which is the user's viewpoint information.

Various methods can be adopted as the tracking method. In the first tracking method, which is an example of the tracking method, a plurality of light emitting elements (LEDs) are provided as the sensor unit 210, as will be described in detail in FIGS. 2 (A) and 2 (B) described later. Then, by capturing the light from these plurality of light emitting elements with the imaging unit 150, the position and direction of the HMD 200 (user's head) in the three-dimensional space in the real world are specified. In the second tracking method, a plurality of light receiving elements (photodiodes and the like) are provided as the sensor unit 210 as described in detail in FIGS. 3 (A) and 3 (B) described later. Then, the position and direction of the HMD 200 are specified by receiving light (laser or the like) from an externally provided light emitting element (LED or the like) by these plurality of light receiving elements. In the third tracking method, a motion sensor is provided as the sensor unit 210, and the position and direction of the HMD 200 are specified using this motion sensor. The motion sensor can be realized by, for example, an acceleration sensor or a gyro sensor. For example, by using a 6-axis motion sensor using a 3-axis acceleration sensor and a 3-axis gyro sensor, the position and direction of the HMD200 in a three-dimensional space in the real world can be specified. The position and direction of the HMD 200 may be specified by a combination of the first tracking method and the second tracking method, or a combination of the first tracking method and the third tracking method. Further, instead of specifying the user's viewpoint position and line-of-sight direction by specifying the position and direction of the HMD 200, a tracking process that directly specifies the user's viewpoint position and line-of-sight direction may be adopted.

The display unit 220 of the HMD 200 can be realized by, for example, an organic EL display (OEL) or a liquid crystal display (LCD). For example, the display unit 220 of the HMD 200 is provided with a first display or a first display area set in front of the user's left eye and a second display or a second display area set in front of the right eye. It is possible to display stereoscopically. In the case of stereoscopic display, for example, an image for the left eye and an image for the right eye having different parallax are generated, the image for the left eye is displayed on the first display, and the image for the right eye is displayed on the second display. Alternatively, the image for the left eye is displayed in the first display area of one display, and the image for the right eye is displayed in the second display area. Further, the HMD 200 is provided with two eyepieces (fisheye lens) for the left eye and the right eye, thereby expressing a VR space that extends over the entire circumference of the user's field of view. Then, a correction process for correcting the distortion generated in the optical system such as the eyepiece is performed on the left eye image and the right eye image. This correction process is performed by the display processing unit 120.

The processing unit 240 of the HMD 200 performs various processes necessary for the HMD 200. For example, the processing unit 240 performs control processing of the sensor unit 210, display control processing of the display unit 220, and the like. Further, the processing unit 240 may perform three-dimensional sound (stereophonic sound) processing to reproduce the three-dimensional sound direction, distance, and spread.

The sound output unit 192 outputs the sound generated by the present embodiment, and can be realized by, for example, a speaker or headphones.

The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC or the like for I / F processing. The portable information storage medium 195 is for the user to store various types of information, and is a storage device that retains the storage of such information even when the power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

The communication unit 196 communicates with an external device (another device) via a wired or wireless network, and its functions include hardware such as a communication ASIC or a communication processor, and communication firmware. Can be realized by.

The program (data) for operating the computer as each part of the present embodiment is distributed from the information storage medium of the server (host device) to the information storage medium 180 (or storage unit 170) via the network and the communication unit 196. You may. The use of information storage media by such a server (host device) can also be included within the scope of the present invention.

The processing unit 100 (processor) provides operation information from the operation unit 160, tracking information in the HMD 200 (information on at least one of the position and direction of the HMD, information on at least one of the viewpoint position and the line-of-sight direction), a program, and the like. Based on this, game processing (simulation processing), moving object processing, virtual camera control processing, display processing, sound processing, and the like are performed.

Each process (each function) of the present embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the functions of each part may be realized by individual hardware, or the functions of each part may be realized by integrated hardware. For example, a processor includes hardware, which hardware can include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor may be composed of one or more circuit devices (for example, ICs) mounted on a circuit board, or one or more circuit elements (for example, resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. Further, the processor may be a hardware circuit by ASIC. Further, the processor may include an amplifier circuit, a filter circuit, and the like for processing an analog signal. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD), or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and when the instructions are executed by the processor, the processing (function) of each part of the processing unit 100 is realized. The instructions here may be an instruction set that constitutes a program, or may be an instruction that instructs the hardware circuit of the processor to operate.

The processing unit 100 includes an input processing unit 102, an arithmetic processing unit 110, and an output processing unit 140. The arithmetic processing unit 110 includes a game processing unit 112, a moving body processing unit 114, a determination unit 115, a virtual space setting unit 116, a virtual camera control unit 118, a display processing unit 120, and a sound processing unit 130. As described above, each process of the present embodiment executed by each of these parts can be realized by a processor (or a processor and a memory). It is possible to carry out various modifications such as omitting a part of these components (each part) or adding other components.

The input processing unit 102 performs a process of receiving operation information and tracking information, a process of reading information from the storage unit 170, and a process of receiving information via the communication unit 196 as input processes. For example, the input processing unit 102 stores the operation information input by the user using the operation unit 160, the processing of acquiring the tracking information detected by the sensor unit 210 of the HMD 200, and the information specified by the read command in the storage unit 170. The process of reading from the computer and the process of receiving information from an external device (server, etc.) via the network are performed as input processes. Here, the reception process includes instructing the communication unit 196 to receive information, acquiring the information received by the communication unit 196 and writing it to the storage unit 170.

The arithmetic processing unit 110 performs various arithmetic processes. For example, arithmetic processing such as game processing (simulation processing), mobile processing, virtual camera control processing, display processing, or sound processing is performed.

The game processing unit 112 (game processing program module) performs various game processing for the user to play the game. In other words, the game processing unit 112 (simulation processing unit) executes various simulation processes for the user to experience virtual reality (virtual reality). The game process calculates, for example, a process of starting a game when the game start condition is satisfied, a process of advancing the started game, a process of ending the game when the game end condition is satisfied, or a game result. Processing etc.

The moving body processing unit 114 (program module for moving body processing) performs various processing on the moving body moving in the virtual space. For example, a process of moving a moving body in a virtual space (object space, game space) or a process of operating a moving body is performed. For example, the mobile body processing unit 114 is based on the operation information input by the user by the operation unit 160, the tracking information acquired, the program (movement / motion algorithm), various data (motion data), and the like. Performs control processing to move the model object) in the virtual space and to move the moving body (motion, animation). Specifically, a simulation in which movement information (position, rotation angle, velocity, or acceleration) and motion information (position or rotation angle of a part object) of a moving body are sequentially obtained for each frame (for example, 1/60 second). Perform processing. A frame is a unit of time for performing movement / motion processing (simulation processing) and image generation processing of a moving body. The moving body is, for example, a virtual user (virtual player) in a virtual space corresponding to a user (player) in the real space, a boarding moving body (operating moving body) on which the virtual user is boarding (operating), or a play target of the virtual user. Such as a character that becomes.

The determination unit 115 (program module for determination processing) performs various determination processes. The details of the determination unit 115 will be described later.

The virtual space setting unit 116 (program module for virtual space setting processing) performs setting processing for a virtual space (object space) in which a plurality of objects are arranged. For example, various display objects such as moving objects (people, robots, cars, trains, airplanes, ships, monsters or animals), maps (topography), buildings, spectators' seats, courses (roads), trees, walls, water surfaces, etc. Performs a process of arranging and setting an object (an object composed of primitive surfaces such as polygons, free-form surfaces, or subdivision surfaces) in a virtual space. That is, the position and rotation angle (synonymous with orientation and direction) of an object in the world coordinate system are determined, and the rotation angle (rotation angle around the X, Y, Z axis) is used at that position (X, Y, Z). Place the object. Specifically, in the object information storage unit 172 of the storage unit 170, object information which is information such as the position, rotation angle, movement speed, and movement direction of an object (part object) in the virtual space is associated with the object number. Is remembered. The virtual space setting unit 116 performs, for example, a process of updating this object information for each frame.

The virtual camera control unit 118 (program module of virtual camera control processing) performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the virtual space. For example, processing is performed to control the viewpoint position or line-of-sight direction (position or posture of the virtual camera) of the virtual camera. Specifically, the process (viewpoint position, line-of-sight direction or angle of view) for controlling the position (X, Y, Z) of the virtual camera and the rotation angle (rotation angle around the X, Y, Z axis) which is the attitude information. Process to control). This virtual camera corresponds to the viewpoint of the user (virtual user). In the case of stereoscopic display, a first viewpoint for the left eye (first virtual camera for the left eye) and a second viewpoint for the right eye (second virtual camera for the right eye) are set.

The display processing unit 120 (display processing program module) performs display processing of a game image (simulation image). For example, drawing processing is performed based on the results of various processing (game processing, simulation processing) performed by the processing unit 100, an image is generated by this, and the image is displayed on the display unit 220 of the HMD 200. Specifically, geometry processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, fluoroscopic conversion, or light source processing is performed, and drawing data (positions of vertices on the primitive surface) is performed based on the processing results. Coordinates, texture coordinates, color data, normal vector or α value, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is subjected to image information in pixel units such as a drawing buffer 178 (frame buffer, work buffer, etc.). Draw in a memorable buffer). As a result, an image seen from a virtual camera (given viewpoint. First and second viewpoints for the left eye and the right eye) is generated in the virtual space. The drawing process performed by the display processing unit 120 can be realized by a vertex shader process, a pixel shader process, or the like.

The sound processing unit 130 (sound processing program module) performs sound processing based on the results of various processing performed by the processing unit 100. Specifically, a game sound such as a musical piece (music, BGM), a sound effect, or a voice is generated, and the game sound is output to the sound output unit 192. A part of the sound processing of the sound processing unit 130 (for example, three-dimensional sound processing) may be realized by the processing unit 240 of the HMD 200.

The output processing unit 140 performs output processing of various information. For example, the output processing unit 140 performs a process of writing information to the storage unit 170 and a process of transmitting information via the communication unit 196 as an output process. For example, the output processing unit 140 performs a process of writing the information specified by the write instruction to the storage unit 170 and a process of transmitting the information to an external device (server or the like) via the network. The transmission process is a process of instructing the communication unit 196 to transmit information, instructing the communication unit 196 to transmit information, and the like.

As shown in FIG. 1, the simulation system of the present embodiment includes a virtual space setting unit 116, a virtual camera control unit 118, and a display processing unit 120. The virtual space setting unit 116 performs setting processing of a virtual space in which a plurality of objects are arranged. For example, the position and direction of each object of a plurality of objects are set and arranged in the virtual space. An object is a display object that appears in a game or is a game element that constitutes a game space.

The virtual camera control unit 118 performs control processing of the virtual camera corresponding to the viewpoint of the user who wears the HMD 200. For example, the virtual camera control unit 118 controls the virtual camera set as the user's first-person view. For example, by setting a virtual camera at a position corresponding to the viewpoint of a virtual user moving in virtual space and setting the viewpoint position and line-of-sight direction of the virtual camera, the position (position coordinates) and posture (rotation axis) of the virtual camera can be set. (Rotation angle around) is controlled.

The display processing unit 120 generates an image that can be seen from a virtual camera (user's viewpoint) in the virtual space as a display image (display image) of the HMD 200. For example, an image that can be seen from a given viewpoint is generated in an object space that is a virtual space. The generated image is, for example, an image for stereoscopic viewing.

Then, in the present embodiment, the virtual space setting unit 116 arranges at least one information display object in the user's line-of-sight direction (user's visual field range). Specifically, at least one information display object controlled to follow the change in the viewpoint of the user (change in the viewpoint of the virtual camera) is arranged in the virtual space. Here, the viewpoint change is, for example, a change in at least one of the line-of-sight direction and the viewpoint position. That is, the virtual space setting unit 116 arranges an information display object controlled so as to follow at least one of the user's line-of-sight direction and the viewpoint position. For example, when the line-of-sight direction or the viewpoint position of the user changes, the arrangement position (three-dimensional position) of the information display object in the virtual space also changes in conjunction with the change. For example, when the user's line-of-sight direction changes to the left, right, up, or down, the position or direction of the information display object in the virtual space also changes so as to follow the change in that direction. Alternatively, when the user's viewpoint position (user position) moves to the left, right, up, or down, the position of the information display object in the virtual space changes so as to follow the movement (change in position). .. For example, when the information display object is composed of billboard polygons, the billboard polygons constituting the information display object are arranged so as to face the user's viewpoint (orthogonally to the user's line-of-sight direction).

Then, the display processing unit 120 displays the information display object or the object display mode (display method, display method) when the information display object and the object in the virtual space (each object of a plurality of objects) have a given positional relationship. ) Is changed. For example, the display processing unit 120 displays the HMD 200 so that when the information display object and the object have the positional relationship, the image looks different from the image when the information display object and the object have the positional relationship. Generate a display image of. That is, as described above, when the line-of-sight direction or the viewpoint position of the user changes, the position or direction of the information display object changes in the virtual space so as to follow the change. Therefore, the positional relationship between the information display object and each object of the plurality of objects in the virtual space changes variously. Then, when the positional relationship between the information display object and the object becomes a predetermined positional relationship (collision, approach positional relationship, etc.), the display mode (display image) of the information display object or the object is changed. It should be noted that the display mode of both the information display object and the object may be changed.

In this case, the display processing unit 120 may change the display mode according to the type, priority, or importance of the information display object. For example, the display mode of the information display object or object that reflects the type, priority, or importance of the information display object is changed. The types of information display objects are classified according to, for example, the type of information presented by the information display objects. For example, the storage unit 170 stores information of the type associated with each information display object, and the display mode is changed using this type of information. The priority or importance of the information display object is information indicating how high the priority or importance of the information presented by the information display object is. For example, the storage unit 170 stores information on the priority and importance of each information display object in association with each other, and the display mode is changed using the information on the priority and importance. ..

For example, when the information display object is the first type, the display processing unit 120 changes the display mode of the object. For example, when the information display object is the first type (for example, an information display object for command selection), the display mode of the object is changed (erased, translucent, etc.), while the information display object is displayed. The display mode will not be changed. On the other hand, when the information display object is of the second type (for example, an information display object for explanatory display), the display mode of the information display object is changed. For example, when the information display object is the second type, the display mode of the information display object is changed (erased, translucent, etc.), but the display mode of the object is not changed. Whether the information display object is the first type or the second type is stored in the storage unit 170 in association with the information display object.

Further, the display processing unit 120 performs a display mode change process using the priority or importance associated with the information display object. For example, the storage unit 170 stores information of priority and importance in association with the information display object. When the display processing unit 120 performs the display mode change processing, the display processing unit 120 reads out the priority and importance information associated with the information display object to be processed from the storage unit 170, and reads the information display object or the object (positional relationship). Performs a process of changing the display mode of the object) to be determined.

In addition, the display processing unit 120 performs processing for changing the display mode according to the elapsed time from the display of the information display object. For example, the elapsed time since each information display object is displayed is measured and recorded. Then, when the processing for changing the display mode of the information display object or the object is performed, the processing that reflects the elapsed time is performed. For example, an information display object having a long elapsed time since being displayed is prioritized for changing the display mode of the information display object having a short elapsed time.

Further, the display processing unit 120 includes processing for erasing the information display object or object, processing for making the information display object or object semi-transparent, processing for canceling tracking of the information display object in the user's line of sight, and information as processing for changing the display mode. Performs movement processing of the display object or object, resizing processing of the information display object or object, color information change processing of the information display object or object, and the like.

The process of erasing the information display object or object can be realized by a process of excluding the information display object or object from the display target (display list), a process of making it transparent, or the like. The semi-transparency processing of the information display object or object can be realized by changing the semi-transparency (α value) of the information display object or object so as to be more translucent, for example. Further, as described above, the information display object is controlled so as to follow the change in the user's line-of-sight direction, but by canceling (reducing) this tracking control, the information display object can be tracked in the user's line-of-sight direction. Release processing can be realized. In the movement processing and resizing processing of the information display object or object, the information display object or object is moved or the information display object is moved so that the positional relationship between the information display object and the object does not become, for example, the positional relationship of collision or approach. Alternatively, it can be realized by changing the size of the object. The color information change process of the information display object or object can be realized by, for example, a process of changing the color brightness, saturation, hue, or the like of the information display object or object. For example, as a process of changing the color information of the information display object or the object, a process of bringing the color information of the information display object or the object closer to the background color information and blending it into the background can be considered. Such processing can be realized by, for example, an α blending process or a depth skewing process based on the color information of the information display object or the object and the color information of the background.

The information display object is, for example, a display object for explanation display, a display object for status display, a display object for conversation display, a display object for command selection, a display object for menu display, and the like. The display object for the explanatory display is, for example, a display object for giving various explanations about the game or the like to the user. The display object for displaying the status is, for example, a display object for informing the user of various situations regarding a game or the like. The display object for displaying the conversation is, for example, a display object for communicating various conversation contents to the user. For example, it is a display object for conveying the conversation content of the character to be played by the user. The display object for command selection is a display object used for the user to select a command in, for example, a game. For example, it is a display object for presenting command options to the user. The display object for menu display is, for example, a display object for presenting a menu screen in a game or the like to a user. For example, it is a display object for presenting a menu screen such as a game mode, a game setting (option setting), or a game end instruction. The information display object is a display object that conveys information to the user using, for example, characters, icons, or figures. As an example, the information display object can be realized by a billboard polygon or the like on which information to be presented to the user is drawn.

On the other hand, the object (the object for which the positional relationship with the information display object is to be determined) is, for example, a moving object, a fixed object, an item, or a background object appearing in the game. The moving body appearing in the game is, for example, a character to be played by the user or a display object that moves in various directions and positions in the game. A fixed object is a display object that is fixed at a predetermined position in a virtual space without moving like a moving body. The item is a display object such as various tools appearing in the game, and is a display object to be played, possessed, or grasped by the user. The background object is an object that constitutes the background of a game or the like. For example, the object is an object other than the information display object among a plurality of objects arranged in the virtual space, and the object can be a target for determining the positional relationship with the information display object.

Further, the simulation system includes a determination unit 115, and the determination unit 115 determines the positional relationship between the information display object and the object. For example, the determination unit 115 determines whether the information display object and the object have a given positional relationship (collision or approach positional relationship, etc.). Then, when the determination unit 115 determines that the information display object and the object have a given positional relationship, the display processing unit 120 changes the display mode of the information display object or the object.

Specifically, the display processing unit 120 changes the display mode when the determination unit 115 determines that the information display object and the object have a collision or approach (proximity) positional relationship. For example, whether or not the positional relationship is close is determined by determining whether or not the distance between the information display object and the object (distance in three-dimensional coordinates, distance in the depth direction, etc.) is less than or equal to a predetermined distance. it can. Whether or not the collision position has been established can be determined by performing a known collision process on the information display object and the object. For example, it can be realized by performing hit determination (intersection determination) between an information display object and an object.

Further, the determination unit 115 determines the positional relationship based on the three-dimensional coordinate values of the information display object and the object, the depth value of the information display object and the object, or the two-dimensional coordinate values when the information display object and the object are projected on a plane. Can be judged. For example, based on the three-dimensional coordinate values of the information display object and the object, size information (shape information), and the like, it is determined whether or not the information display object and the object have a collision or approach positional relationship. Alternatively, by comparing the depth value (Z value) of the information display object with the depth value (Z value) of the object, it is determined whether or not the information display object and the object have a given positional relationship. Alternatively, the information display object and the object are projected onto a plane (screen), and the information display object and the object collide with each other using the two-dimensional coordinate values (two-dimensional shape) of the information display object and the object on the plane (projected image). Or, it is determined whether or not the positional relationship of approach is reached. For example, by determining the overlap or distance between the information display object and the object on the plane, it is possible to determine whether or not there is a positional relationship of collision or approach.

Further, the virtual space setting unit 116 arranges an information display object at a given depth position in the user's line-of-sight direction so as to follow a change in the user's viewpoint (change in at least one of the line-of-sight direction and the viewpoint position). For example, the information display object is arranged so as to follow the viewpoint change (change in the line-of-sight direction or the viewpoint position) while keeping the depth position (depth distance) of the information display object with respect to the user's viewpoint (virtual camera) unchanged. .. For example, an information display object composed of, for example, a billboard polygon is arranged at a given depth position (position on the Z coordinate axis) from the user's viewpoint (virtual camera).

Further, the virtual camera control unit 118 controls the virtual camera so as to follow the change in the viewpoint of the user based on the tracking information of the viewpoint information of the user.

For example, the input processing unit 102 (input receiving unit) acquires tracking information of the viewpoint information of the user who wears the HMD 200. For example, the tracking information (viewpoint tracking information) of the viewpoint information which is at least one of the user's viewpoint position and line-of-sight direction is acquired. This tracking information can be obtained, for example, by performing tracking processing of HMD200. The user's viewpoint position and line-of-sight direction may be directly acquired by tracking processing. As an example, the tracking information includes change information of the viewpoint position from the initial viewpoint position of the user (change value of the coordinates of the viewpoint position) and change information of the line-of-sight direction from the initial line-of-sight direction of the user (rotation axis in the line-of-sight direction). At least one of (change value of rotation angle in rotation) can be included. The user's viewpoint position and line-of-sight direction (information on the position of the user's head and posture) can be specified based on the change information of the viewpoint information included in such tracking information.

Then, the virtual camera control unit 118 changes the viewpoint position and the line-of-sight direction of the virtual camera based on the acquired tracking information (information on at least one of the user's viewpoint position and the line-of-sight direction). For example, the virtual camera control unit 118 virtually changes the viewpoint position and line-of-sight direction (position, posture) of the virtual camera in the virtual space according to changes in the user's viewpoint position and line-of-sight direction in the real space. Set the camera. By doing so, the virtual camera can be controlled so as to follow the change in the viewpoint of the user based on the tracking information of the viewpoint information of the user.

In this embodiment, virtual reality simulation processing is performed as game processing of a game played by a user. The virtual reality simulation process is a simulation process for simulating an event in the real space in the virtual space, and is a process for allowing the user to experience the event virtually. For example, a virtual user corresponding to a user in the real space and a moving body such as a boarding mobile body thereof are moved in the virtual space, and processing is performed to allow the user to experience changes in the environment and surroundings due to the movement.

The simulation system also includes a game processing unit 112 that performs game processing. Then, the game processing unit 112 performs a process of invoking the effect of the command according to the information of the character to be the target of the user's game play and the viewpoint information of the user.

For example, in the present embodiment, a process of accepting a command selected by the user from a plurality of commands is performed. For example, information on N commands that can be selected by the user is stored in the storage unit 170 (command information storage unit). Then, for example, M (M <N) commands are automatically or manually selected from these N commands, and a set of commands called a so-called deck is organized. During gameplay, the user selects a command he / she desires from among these M commands. Then, the selected command is accepted as the command to be activated.

Then, the game processing unit 112 performs a process of invoking the effect of the received command as a game process. For example, the effect of each command is associated with each command and stored in the storage unit 170 as command information. The game processing unit 112 performs a process of invoking an effect associated with a command selected by the user.

Then, the game processing unit 112 calculates a game parameter, advances the game, calculates a game result (game result), or controls an icon corresponding to the command, based on the result of invoking the effect of the command. Perform game processing such as.

Specifically, the game processing unit 112 performs arithmetic processing for increasing or decreasing the value of the game parameter. Then, based on the game parameters after the arithmetic processing, various game processing such as controlling the movement (behavior) of the character, performing the branch processing of the story, and performing the processing of generating the game event is performed. For example, processing of a game in which a story progresses by a character targeted for game play by the user and the user is performed.

The display processing unit 120 performs display processing of the game image based on the result of the game processing described above. For example, a process for displaying a game image according to a game parameter calculation result, a game progress process result, a game result calculation result, and an icon control process result is performed on the display unit 220 of the HMD 200. Further, the sound processing unit 130 also performs processing for outputting game sounds such as BGM, sound effects, and voice from the sound output unit 192 based on the result of the game processing.

Then, in the present embodiment, the game processing unit 112 performs a process of invoking the effect of the command according to the information of the character to be the target of the game play of the user and the viewpoint information of the user. For example, processing is performed to activate the effect of a command according to information set (specified) by character information and user's viewpoint information (for example, positional relationship information, line-of-sight information, gaze information, etc.).

Specifically, the game processing unit 112 performs a process of changing at least one of the degree of command effect and the content of the command effect according to the character information and the user's viewpoint information. For example, when the information (for example, positional relationship information, line-of-sight information, gaze information, etc.) set (specified) by the character information and the user's viewpoint information changes, the game processing unit 112 changes the effect of the command accordingly. The command is activated so that at least one of the degree and the content of the effect of the command changes.

Here, the character information is, for example, at least one of the character's position information, direction information, game parameter information, part information, posture information, and type information. A character is a gameplay element that appears in a game, representing, for example, a person, animal, robot, or moving object in the real world, and is displayed as an object in a game image.

Specifically, the character that is the target of the user's gameplay is, for example, the character that is the opponent of the user's gameplay. Taking a communication game (human relations simulation game) with a character of the opposite sex as an example, it is a character that is a communication partner (candidate) of a user. For example, in the present embodiment, the display processing unit 120 generates a game image of the user, for example, from the first-person viewpoint, and displays the generated image from the first-person viewpoint on the display unit 220 of the HMD 200. By doing so, it is possible to give the user a virtual reality as if he / she entered the game space (the world of CG animation). Then, the character appears in this game space (object space, virtual three-dimensional space) as a target of the user's game play. As a result, the user can feel the virtual reality as if a real human being (character) exists in front of the user. As the character to be the target of the user's game play, a character appearing as an enemy or ally of the user in a war game, an RPG game, a sports game, or the like may be used.

The game parameter is a parameter used for game processing (game progress processing, etc.). For example, game parameters are parameters that represent the status and abilities of characters and users. For example, as the game parameters of the character, various parameters such as status parameters representing the psychology and state of the character, parameters representing the ability of the character, parameters related to the behavior of the character, and parameters related to the belongings of the character can be assumed. Further, the game parameter of the character may be a parameter representing the evaluation of the character with respect to the user.

The part information of the character includes the type information (head, chest, waist, legs, hands, etc.) of the part (part object) that constitutes the character, the position information of the part (relative position to the representative position of the character), or the part of the part. Shape information (shape of part object) and so on. The posture information of the character is posture information that specifies the motion of the character such as sitting, standing, walking, or running. For example, when a character is represented by a skeleton model, the shape information of this skeleton corresponds to the posture information of the character. The character type information is, for example, what is called a character type or attribute, and if it is a character of the opposite sex, it may be an active type, a serious type, a sweetheart type, a neat type, or the like.

Further, the user's viewpoint information is, for example, at least one of the user's viewpoint position information, line-of-sight direction information, and virtual camera attribute information. The viewpoint position information is, for example, information on a position set as a user's viewpoint in a game space (object space, virtual three-dimensional space). The line-of-sight direction information is information indicating the line-of-sight direction of the user at the viewpoint. The virtual camera attribute information is information about the virtual camera set from the user's viewpoint, and various information such as angle of view information and binocular parallax information in stereoscopic vision can be assumed.

More specifically, the game processing unit 112 performs a process of invoking the effect of the command according to the positional relationship information between the user and the character. For example, in the case of the first-person viewpoint, the effect of the command corresponding to the positional relationship information between the user's viewpoint position and the character's position (representative position) is activated. The positional relationship information is information representing the relationship between the user's position (viewpoint position, user character's position) and the character's position (representative position, viewpoint position). The positional relationship information can be assumed, for example, the distance between the user and the character (distance in the game space), but is not limited to this. For example, it may be information on the directional relationship between the user and the character.

Further, the game processing unit 112 performs a process of invoking the effect of a command according to the line-of-sight relationship information indicating the relationship between the user's line of sight and the character. The line-of-sight relationship information is information that represents the relationship between the user's line of sight and the character, and is, for example, information that represents the relative relationship between the user's line of sight and the character. For example, the line-of-sight-related information is information on which direction the user's line of sight is facing with reference to the position (representative position) of the character and each part.

Further, the game processing unit 112 performs a process of invoking the effect of the command according to the gaze information of the user. The gaze information represents gaze part information which is information on the part of the character that the user is gaze at, gaze presence / absence information which is information on whether or not the user is gaze at the character, and time when the user is gaze at the character. It is at least one of the gaze time information and the gaze distance information indicating the distance between the user and the character at the time of gaze.

2. 2. Tracking process Next, an example of tracking process will be described. FIG. 2A shows an example of the HMD200 used in the simulation system of the present embodiment. In FIG. 2A, a plurality of light emitting elements 231 to 236 are provided for the HMD 200. These light emitting elements 231 to 236 are realized by, for example, LEDs. The light emitting elements 231 to 234 are provided on the front side of the HMD 200, and the light emitting element 235 and the light emitting element 236 (not shown) are provided on the back side. These light emitting elements 231 to 236 emit (emit) light in the visible light band, for example. Specifically, the light emitting elements 231 to 236 emit light of different colors from each other. Then, the imaging unit 150 shown in FIG. 2B is installed on the front side of the user PL, and the light emitting elements 231 to 236 are imaged by the imaging unit 150. That is, the spot light of these light emitting elements 231 to 236 is reflected in the image captured by the imaging unit 150. Then, by performing image processing of this captured image, tracking of the head (HMD) of the user PL is realized. That is, the three-dimensional position and the direction (viewpoint position, line-of-sight direction) of the head of the user PL are detected.

For example, as shown in FIG. 2B, the imaging unit 150 is provided with the first and second cameras 151 and 152, and the first and second cameras 151 and 152 of these first and second cameras 151 and 152 are provided. By using the captured image, the position of the head of the user PL in the depth direction can be detected. Further, the rotation angle (line of sight) of the head of the user PL can also be detected based on the motion detection information of the motion sensor provided in the sensor unit 210 of the HMD 200. Therefore, by using such an HMD200, no matter which direction the user PL faces in all directions of 360 degrees around it, the image (user's) in the corresponding virtual space (virtual three-dimensional space) is used. The image seen from the virtual camera corresponding to the viewpoint) can be displayed on the display unit 220 of the HMD 200. Infrared LEDs may be used as the light emitting elements 231 to 236 instead of visible light. Further, the position and movement of the user's head may be detected by another method such as using a depth camera or the like.

Further, the HMD 200 is provided with a headband 260 and a headphone terminal (not shown). By connecting the headphones 270 (sound output unit 192) to the headphone terminal, the user PL can listen to, for example, a game sound processed by three-dimensional sound.

Further, as shown in FIG. 2A, the user PL possesses the game controllers 166 and 167. The game controllers 166 and 167 correspond to the operation unit 160 in FIG. The game controllers 166 and 167 are provided with light emitting elements 237 and 238. By capturing the light from the light emitting elements 237 and 238 with the imaging unit 150, the positions of the game controllers 166 and 167 and the like are detected by the same method as in the case of the HMD 200. This allows the user PL to input various operation information. It should be noted that the user's operation information may be input by detecting the nodding motion or the swinging motion of the user's head by the tracking process.

FIG. 3A shows another example of the HMD200 used in the simulation system of the present embodiment. As shown in FIG. 3A, the HMD 200 is provided with a plurality of light receiving elements 201, 202, 203 (photodiodes). The light receiving elements 201 and 202 are provided on the front surface side of the HMD 200, and the light receiving elements 203 are provided on the right side surface of the HMD 200. Further, a light receiving element (not shown) is also provided on the left side surface, the upper surface, and the like of the HMD. Further, the HMD 200 is provided with a headband 260, and the user PL is wearing headphones 270.

As shown in FIG. 3B, base stations 280 and 284 are installed around the user PL. The base station 280 is provided with light emitting elements 281 and 282, and the base station 284 is provided with light emitting elements 285 and 286. The light emitting elements 281, 282, 285, and 286 are realized by, for example, an LED that emits a laser (infrared laser or the like). The base stations 280 and 284 use these light emitting elements 281, 282, 285 and 286 to emit, for example, a laser radially. Then, the light receiving elements 201 to 203 and the like provided in the HMD200 of FIG. 3A receive the laser from the base stations 280 and 284 to realize the tracking of the HMD200, and the position and the direction of the head of the user PL. (Viewpoint position, line-of-sight direction) can be detected.

Further, as shown in FIG. 3A, the game controllers 166 and 167 possessed by the user PL are provided with light receiving elements 204 to 206 and 207 to 209. Then, by receiving the lasers from the light emitting elements 281, 282, 285, and 286 of the base stations 280 and 284 by the light receiving elements 204 to 206 and 207 to 209, the game controllers 166 and 167 can be used in the same manner as in the case of the HMD200. The position of is detected. This allows the user PL to input various operation information.

The tracking processing method for detecting the user's viewpoint position and line-of-sight direction (user's position and direction) is not limited to the methods described in FIGS. 2A to 3B. For example, the tracking process may be realized by the HMD 200 alone by using a motion sensor or the like provided in the HMD 200. That is, the tracking process is realized without providing an external device such as the image pickup unit 150 of FIG. 2B or the base station 280 and 284 of FIG. 3B. Alternatively, viewpoint information such as the user's viewpoint position and line-of-sight direction may be detected by various viewpoint tracking methods such as known eye tracking, face tracking, or head tracking. For example, in eye tracking, the position and shape of the eyes of the user's left eye and right eye are recognized. Then, the tracking process is realized by specifying the position of the left eye, the position of the right eye, the line-of-sight direction of the left eye, the line-of-sight direction of the right eye, and the like, and acquiring the viewpoint information of the user. This eye tracking can be realized, for example, by photographing the user's left eye and right eye with an imaging unit and performing image recognition processing such as pupils on the captured image. Further, in face tracking, the image pickup unit captures the user's face and performs face image recognition processing. Then, based on the result of the image recognition process, the position and the direction of the user's face are specified, the viewpoint position and the line-of-sight direction of the user are obtained, and the tracking process is realized.

3. 3. Method of this embodiment Next, the method of this embodiment will be specifically described.

3.1 Description of the game First, an outline of the game realized by the method of the present embodiment will be described. In this game, the user communicates with the characters of the opposite sex appearing in the game, teaches study, gives various advices, answers questions, clears various events, and the character's liking for the user. It is a communication game (human relations simulation game) that enhances such things. In the following, a case where the game to which the method of the present embodiment is applied is a communication game with a character of the opposite sex or the like will be mainly described as an example. However, the game to which the method of the present embodiment is applied is not limited to this, and can be applied to various games such as a battle game, an RPG game, a sports game, and an action game.

4 to 6 show examples of game images generated by the simulation system (game system) of the present embodiment. In the game realized by the present embodiment, the user (player) visits the house as a tutor of the character CH and teaches study and the like. A plurality of stories are prepared in the game, and different types of character CHs appear in each story. The user enjoys intimate communication with the character CH while achieving the purpose in each story.

The user who plays the game wears the HMD200. Then, when the user turns his / her line of sight in each direction of the surrounding 360 degrees, the image to be seen in each direction is displayed on the display unit 220 of the HMD 200. For example, in FIG. 4, when the user turns his / her line of sight toward the character CH, the image of the character CH studying is displayed on the display unit 220. On the other hand, when the user turns his / her head and directs his / her line of sight to the back side of the user, an image of furniture or the like in the room located on the back side of the character CH is displayed on the display unit 220. Therefore, the user can experience the virtual reality as if he / she has entered the game world. That is, it is possible to give the user a virtual reality as if he / she actually entered the room and taught the opposite sex in front of his / her eyes to study, and it is possible to enhance the user's immersive feeling in the game.

In the game image of FIG. 4, the character CH (moving object in a broad sense), the book possessed by the character CH (item in a broad sense), the curtain CT (fixed object in a broad sense), the wall WL (background in a broad sense), etc. The object is displayed. In addition, information display objects DSA and DSB are displayed.

Information display object DSA is a display object for explanatory display. For example, the information display DSA explains that the character CH is reading a book. The information display object DSB is a display object for displaying the status. As will be described later with reference to FIG. 7, various status information of the character CH is displayed by the information display object DSB.

When the user approaches the character CH in FIG. 4 and turns his / her gaze, the game image shown in FIG. 5 is displayed. In FIG. 5, the information display objects DSC, DSD1 and DSD2 are further displayed. Information display object DSC is a display object for conversation display. For example, on the information display object DSC, "What do you want?", Which is a conversation of the character CH, is displayed. Information display items DSD1 and DSD2 are display items for command selection (command input). For example, a command for replying to a conversation from the character CH (YES, NO) is displayed.

When the user selects the "YES" command in FIG. 5, a game image as shown in FIG. 6 is displayed. In FIG. 6, information displays DSE1, DSE2, and DSE3 are displayed. Information display objects DSE1, DSE2, and DSE3 are display objects (icons) for command selection. For example, commands CM1, CM2, and CM3 are associated with the information display objects DSE1, DSE2, and DSE3. For example, when the user selects the information display object DSE1, the effect of the command CM1 "more concentrated" is activated. When the information display object DSE2 is selected, the effect of the command CM2 "scolding" is activated. When the information display object DSE3 is selected, the effect of the command CM3 "call out" is activated.

The information display objects (DSE1, DSE2, DSE3, DSD1, DSD1) corresponding to the commands are selected by using the operation unit 160 of FIG. For example, the user uses the operation unit 160 to select an information display object (icon) corresponding to the command desired by the user, and performs a command operation of invoking the effect of the command corresponding to the selected information display object. Taking FIGS. 2 (A) and 3 (A) as an example, the user performs such a command operation by moving the game controllers 166 and 167 held in both hands. Alternatively, a command operation may be performed using a game controller (not shown) connected by wire. Alternatively, a user wearing the HMD200 on his / her head may perform a command operation by performing an action such as turning the line of sight toward an information display object corresponding to the command or shaking his / her head vertically or horizontally. May be good.

In FIG. 6, the user selects the information display object DSE3 corresponding to the command CM3 to "call out". As a result, the effect of the command CM3 to "call out" is activated. For example, the command CM3 is associated with voice data of words corresponding to "call out" (for example, "hey!", "Are you doing your best?", "Are you studying?"). Then, when the command CM3 is activated, the voice data is reproduced, and the voice of the word (sentence) is output from the sound output unit 192 of FIG.

In addition, when a command is activated (the effect of the command is activated), the game parameters of the character CH are also affected. For example, when the command CM3 "call out" is activated, the game parameters of the corresponding character CH also change.

For example, in FIGS. 4 to 6, an information display object DSB for informing the user of the status of game parameters, which are various states of the character CH, is displayed. As shown in FIG. 7, for example, the information display object DSB conveys to the user the current psychological state, motivation, concentration, pulse, and the like of the character CH. These psychological states, motivation, concentration, and pulse are prepared as game parameters of the character CH.

The psychological state of the character CH is expressed using, for example, color. For example, in the game of the present embodiment, a lesson task is given, and the lesson ends, for example, when a predetermined time elapses. Then, for example, the game result of the user is determined according to the amount of achievement of the task at the end of the lesson. For example, when the user's game performance is high, for example, a card with a high effect or a command (action) with a high degree of rarity is given to the user.

For example, the attributes of red, green, and blue are given to each task given in the lesson, and the psychological state of the character CH also has the components of red, green, and blue. For example, when the psychological state of the character CH has a large amount of red components, the tasks with the red attribute can be digested immediately, but the tasks with the green and blue attributes are slow to digest. Similarly, when the psychological state of the character CH has a large amount of green component, the task of the green attribute can be digested immediately, but the task of the red and blue attributes is slow to digest. Therefore, the user can increase the amount of tasks to be achieved and properly complete the lesson by looking at the psychological state of the character CH displayed on the information display object DSB and digesting the tasks of the attributes suitable for the psychological state. Will be.

Further, as shown in FIG. 7, game parameters of motivation and concentration are set in the character CH. The values of these motivation and concentration game parameters vary from 0 to 100 (%), for example, and the larger the value, the higher the motivation and concentration. The higher the motivation value, the faster the task will be digested. On the other hand, the value of concentration decreases with the passage of time, but the degree of decrease becomes large or difficult to decrease depending on the psychological state of the character CH. In addition, when the value of concentration becomes lower than the predetermined value and the concentration is lost, a break from the lesson is required.

Further, the pulse of the information display object DSB in FIG. 7 functions as a game parameter indicating, for example, the degree of excitement and the favorable impression of the character CH to the user. For example, when the degree of excitement increases, the pulse also increases, motivation increases, concentration becomes difficult to decrease, and the digestion speed of tasks increases.

3.2 Processing for changing the display mode As shown in FIGS. 4 to 6, in the game realized by the present embodiment, various information display objects are displayed. In a conventional image generation system that displays a pseudo three-dimensional image on a display unit without using an HMD, a two-dimensional display object (sprite or the like) that is displayed two-dimensionally on a projection screen is used as such an information display object. It was used.

However, in a VR (Virtual Reality) system using an HMD, if a two-dimensional display is used as an information display such as an explanation display, a status display, a command selection, and a menu display, the user's virtual reality is impaired. It also causes problems such as the screen becoming difficult to see. For example, in a VR space where three-dimensional objects such as the character CH, the book BK, the curtain CT, and the wall WL in FIG. 4 are three-dimensionally displayed, only the information display object is displayed two-dimensionally on the projection screen. , It spoils the sense of virtual reality. Further, when a large number of two-dimensional information display objects are displayed, these information display objects are two-dimensionally overlapped, or the objects are hidden by the information display objects and cannot be seen, resulting in a very difficult-to-see screen.

Therefore, in the present embodiment, at least one information display object is arranged in the user's line-of-sight direction (user's field of view range, front direction). For example, the information display object is composed of objects having three-dimensional coordinate values and arranged in the virtual space like other objects in the virtual space. Specifically, an information display object controlled to follow a change in the user's viewpoint (change in at least one of the line-of-sight direction and the viewpoint position) is arranged in the virtual space. More specifically, the information display object is arranged at a given depth position in the line-of-sight direction of the user so as to follow the change in the viewpoint of the user.

8 (A) to 8 (D) are explanatory views of a display method for such an information display object. For example, in FIG. 8A, VP is the viewpoint of the user PL wearing the HMD, and VL is the line-of-sight direction of the user PL. These viewpoint VP and line-of-sight direction VL correspond to the viewpoint and line-of-sight direction of the virtual camera corresponding to the user PL.

Then, in FIG. 8A, the information display objects DS1 and DS2 are arranged at the positions of the depth values Z1 and Z2 in the line-of-sight direction VL of the user PL, respectively. The depth values Z1 and Z2 are, for example, coordinate values of the Z coordinate axes in the viewpoint coordinate system (camera coordinate system). The information display objects DS1 and DS2 are realized by, for example, billboard polygons. That is, by mapping images such as characters, icons, or figures for explanation display, status display, conversation display, command selection display, and menu display as described in FIGS. 4 to 6 to the billboard polygon. , Information display objects DS1 and DS2 are realized. The billboard polygon (bilboard primitive surface) is a polygon (primitive surface) arranged so as to face (substantially face) the viewpoint VP (virtual camera) of the user PL. For example, the billboard polygons of the information display objects DS1 and DS2 are arranged so as to be orthogonal (substantially orthogonal) to the line-of-sight direction VL (line-of-sight direction vector). For example, the normal direction of the surface of the billboard polygon is arranged so as to be parallel to the line-of-sight direction VL. The arrangement direction of the billboard polygon is not limited to such a direction, and can be set in various arrangement directions. Further, the information display object may be realized by a display object having a format different from that of the billboard polygon. For example, an information display object may be realized by a three-dimensional object composed of a plurality of primitive surfaces (polygons and the like).

Then, in the present embodiment, the information display objects DS1 and DS2 are controlled so as to follow the change in the line-of-sight direction VL of the user PL. For example, in FIG. 8B, the line-of-sight direction VL of the user PL changes to the right side. For example, the user PL wearing the HMD is facing the right side, and the line-of-sight direction of the virtual camera in the virtual space is also changed to the right side. In this case, as shown in FIG. 8B, the positions and directions of the information display objects DS1 and DS2 change so as to follow the change in the line-of-sight direction VL. For example, also in FIG. 8B, the billboard polygons constituting the information display objects DS1 and DS2 are arranged so as to face the viewpoint VP of the user PL.

Further, in FIG. 8C, the line-of-sight direction VL of the user PL changes to the left side. For example, the user PL wearing the HMD faces the left side, and the line-of-sight direction of the virtual camera in the virtual space also changes to the left side. Also in this case, as shown in FIG. 8C, the positions and directions of the information display objects DS1 and DS2 change so as to follow the change in the line-of-sight direction VL. For example, also in FIG. 8C, the billboard polygons constituting the information display objects DS1 and DS2 are arranged so as to face the viewpoint VP of the user PL.

Further, in FIG. 8D, the line-of-sight direction of the user PL changes in the upward direction. Also in this case, as shown in FIG. 8D, the positions and directions of the information display objects DS1 and DS2 change so as to follow the change in the line-of-sight direction VL. Similarly, when the line-of-sight direction of the user PL changes downward, the positions and directions of the information display objects DS1 and DS2 change so as to follow the change.

As described above, in the display methods of FIGS. 8A to 8D, the information display object is three-dimensionally arranged in the virtual space like other objects in the virtual space. Therefore, it is possible to solve the problem that the user's virtual reality is impaired due to the display of the information display object. That is, in the conventional method in which a two-dimensional information display object is displayed in a plane, there is a problem that the virtual reality is impaired due to the existence of the information display object, but the information display object is three-dimensionally displayed in the virtual space. According to the method of the present embodiment arranged in, the occurrence of such a problem can be prevented. Further, by arranging the information display objects three-dimensionally in this way, even when displaying a plurality of information display objects, it is possible to make the display suitable and easy for the user to see. For example, as shown in FIGS. 8A to 8D, it is possible to arrange a plurality of information display objects (DS1, DS2) with different depth values (Z1, Z2), and the user. It is possible to display an appropriate information display that is easy for the user to see. Further, according to the display methods of FIGS. 8A to 8D, the information display object is always displayed on the front side of the user. Therefore, even when the line-of-sight direction of the user is directed to various directions, the information display object is displayed on the front side of the user so that various information necessary for the user can be appropriately presented by using the information display object. become.

The display method of the information display object is not limited to the methods shown in FIGS. 8 (A) to 8 (D), and various modifications can be performed. Further, it is not necessary to display the information display object so as to always follow the change in the line-of-sight direction, and it may be exceptionally not followed. Alternatively, the information display object may be displayed so as to follow the change in the line-of-sight direction with a time delay. For example, when the line-of-sight direction changes significantly (when the change angle, change speed, or change acceleration in the line-of-sight direction is equal to or greater than a predetermined value), the information display object may not follow the change in the line-of-sight direction, or may be delayed in time. You may try to follow. Then, when a given time has elapsed, a return process such as rearrangement of the information display object may be performed. Alternatively, the specific information display object may not follow the change in the line-of-sight direction of the user. For example, an information display object that displays the status of a specific object (for example, a character to be played by the user) may be controlled so as to be displayed in the vicinity of the specific object.

Further, in FIGS. 8A to 8D, the case where the information display object follows the change in the line-of-sight direction of the user has been described, but the information display object may follow the change in the viewpoint position of the user. .. For example, when the user's viewpoint position (user position) moves up, down, left, and right, the information display object is moved up, down, left, and right so as to follow this movement. The information display object may follow only the change in the line-of-sight direction of the user, may follow only the change in the viewpoint position, or may follow both the change in the line-of-sight direction and the change in the viewpoint position. .. However, in the following description, in order to simplify the explanation, the case where the change in the viewpoint of the user is the change in the direction of the user's line of sight will be mainly taken as an example.

According to the methods of FIGS. 8A to 8D, it is possible to appropriately display an information display object that does not impair the virtual reality of the user. However, since there are many objects in the virtual space other than the information display object, it has been found that the relationship with these objects becomes a problem.

For example, in FIG. 9A, the information display objects DS1 and DS2 are arranged in the front direction of the user PL, and other objects OB1 and OB2 are also arranged in the virtual space. These objects OB1 and OB2 are objects corresponding to, for example, the character CH (moving body), the main BK (item), the curtain CT (fixed object), the wall WL (background), and the like in FIG. In FIG. 9A, the information display objects DS1 and DS2 do not collide with the objects OB1 and OB2.

On the other hand, when the line-of-sight direction VL of the user PL changes as shown in FIGS. 9A to 9B, the positions of the information display objects DS1 and DS2 change following the change in the line-of-sight direction VL. In this case, in FIG. 9B, a situation occurs in which the information display objects DS1 and DS2 collide (approach, interfere with) the objects OB1 and OB2. When such a situation occurs, for example, the information display objects DS1 and DS2 are sunk into the objects OB1 and OB2, and a part of the information display objects DS1 and DS2 is missing or displayed, or one of the objects OB1 and OB2. Problems such as missing parts are displayed. For example, if the line-of-sight direction of the user PL changes slightly in the state of FIG. 9B, the display flickers and the display quality is greatly deteriorated.

Further, even when the information display object follows the change of the user's viewpoint position (user position), the information display object collides with the object due to the movement of the user's viewpoint position up, down, left, and right.

Therefore, in the present embodiment, when the information display objects DS1 and DS2 and the objects OB1 and OB2 have a positional relationship as shown in FIG. 9B, the information display objects DS1 and DS2 or the display modes of the objects OB1 and OB2 are changed. I do. That is, the display mode is changed to solve the display defect due to the collision (approach, interference) as shown in FIG. 9B.

For example, in FIG. 10A, the display modes of the information display objects DS1 and DS2 are changed. Specifically, the display of the information display objects DS1 and DS2 is erased, and the information display objects DS1 and DS2 are semi-transparent. On the other hand, the objects OB1 and OB2 are not subjected to such a display mode change process. The process of erasing the information display objects DS1 and DS2 can be realized by, for example, a process of excluding the information display objects DS1 and DS2 from the display list (display object list). The translucency processing of the information display objects DS1 and DS2 can be realized, for example, by performing a process of making the translucency (α value) of the information display objects DS1 and DS2 close to transparency. Alternatively, the information display objects DS1 and DS2 may be changed to, for example, a frame model or the like to erase characters, icons, or figures such as explanation display and status display. Alternatively, the color information of the information display objects DS1 and DS2 may be changed. For example, the colors of the information display objects DS1 and DS2 are changed to a color having low visibility (a color that is inconspicuous) or a color that blends into the background.

In FIG. 10B, the display mode of the objects OB1 and OB2 is changed. Specifically, the display of the objects OB1 and OB2 is erased, and the objects OB1 and OB2 are made translucent. On the other hand, the information display objects DS1 and DS2 are not subjected to such a display mode change process. The erasing process of the objects OB1 and OB2 can be realized by, for example, a process of excluding the objects OB1 and OB2 from the display list. The translucency processing of the objects OB1 and OB2 can be realized, for example, by performing a process of making the translucency of the objects OB1 and OB2 close to transparent. Alternatively, the objects OB1 and OB2 may be changed to, for example, a frame model so that properties such as color and texture are not displayed. Alternatively, the color information of the objects OB1 and OB2 may be changed. For example, the colors of the objects OB1 and OB2 are changed to a color having low visibility (a color that is inconspicuous) or a color that blends into the background.

In FIG. 10C, in the relationship between the information display object DS1 and the object OB1, the display mode of the object OB1 is changed, and the object OB1 is erased or made translucent. For example, when the information display object DS1 is a display object having a high priority or importance, the display mode of the object OB1 is changed instead of the information display object DS1. On the other hand, in the relationship between the information display object DS2 and the object OB2, the display mode of the information display object DS1 is changed, and the information display object DS1 is erased or made translucent. For example, when the information display object DS2 is a display object having a low priority or importance, the display mode of the information display object DS2 is changed instead of the object OB2.

That is, when the information display object DS1 is the first type (for example, a display object of a type having high importance or priority), the display mode of the object OB1 is changed. On the other hand, when the information display object DS2 is of the second type (for example, a display object of a type having low importance or priority), the display mode of the information display object DS2 is changed.

For example, in the table data of FIG. 11A, the types TPA, TPB, and TPA are associated with the information display objects DSF1, DSF2, and DSF3. This table data is stored in the storage unit 170 of FIG. Since the type TPA is, for example, a type having a low priority or importance, the display mode of the information display object is changed when a collision (approach, interference) with an object occurs. For example, processing is performed to erase or make the information display object semi-transparent. On the other hand, since the type TPB is, for example, a type having a high priority or importance, the display mode of the information display object is not changed when a collision with an object occurs. For example, the information display object is not erased or translucently processed, but the object is erased or translucently processed.

In the table data of FIG. 11B, the priority (importance) PRA, PRB, and PRC are associated with the information display objects DSG1, DSG2, and DSG3. This table data is stored in the storage unit 170 of FIG. Then, when a collision between the information display objects DSG1, DSG2, DSG3 and another object occurs, the priority (importance) PRA, PRB, and PRC associated with each of the information display objects DSG1, DSG2, and DSG3 are used. The display mode is changed. For example, for an information display object to which a high priority (importance) is associated, the display mode is not changed even when the object is in a positional relationship of collision or approach. Then, for example, the display mode of the object is changed. On the other hand, for the information display object associated with a low priority (importance), the display mode is changed when the object is in a positional relationship of collision or approach. And, for example, the display mode of the object is not changed. In this case, the priority (importance) is also associated with the object, and the priority (importance) associated with the information display object is compared with the priority (importance) associated with the object. You may try to do. Then, based on the result of the comparison process, the display mode is changed. For example, when the information display object has a higher priority (importance) than the object, the display mode of the object is changed. On the other hand, when the object has a higher priority (importance) than the information display object, the display mode of the information display object is changed.

As described above, in the present embodiment, it is possible to realize an appropriate display mode change process according to the type, priority or importance of the information display object. For example, for a specific type of information display object, the display mode is changed only for the object, or conversely, the display mode is changed only for the information display object even when there is a collision or approach positional relationship with the object. become able to. Alternatively, if the priority or importance of the information presented by the information display object is high, change the display mode of the information display object even if there is a collision or approach positional relationship with the object. Instead, the display mode can be changed only for the object.

Further, in the present embodiment, as shown in FIG. 11B, the display mode is changed by using the priority or importance associated with the information display object. In this way, the priority or importance can be associated with the information display object in advance, and the appropriate display mode change process can be executed using this priority or importance.

For example, in FIGS. 4 to 6, the information display objects (DSD1, DSD2, DSE1 to DSE3) for command selection are information display objects having high priority or importance. For example, when the information display object and the object collide or approach each other due to a change in the user's line-of-sight direction, the information display object for command selection is erased or translucent. You will not be able to select (input) commands, and the game will stop progressing. Therefore, a high priority or importance is associated with the information display object for such command selection, and when the positional relationship of collision or approach with the object occurs, the information display object is used. Do not perform the erasing process or semi-transparency processing of the object, but perform the erasing process or semi-transparency processing of the object.

On the other hand, the information display object (DSA) for displaying the explanation is an information display object having a low priority or importance, and is used for displaying the explanation when the object is in a positional relationship of collision or approach. Even if the information display object is erased or made semi-transparent, it does not hinder the progress of the game so much. Therefore, a low priority or importance is associated with such an information display object for explanatory display, and when a collision or approaching positional relationship is reached with respect to the object, the information display object is used. Perform erasing processing and semi-transparency processing. By doing so, it becomes possible to realize an appropriate display process that reflects the priority or importance of the information display object.

Further, in the present embodiment, the display mode may be changed according to the elapsed time from the display of the information display object.

For example, in FIG. 12, the information display objects DSH1 and DSH2 are subject to the display mode change processing at the timing when a long elapsed time TE1 and TE2 have elapsed since they were displayed. On the other hand, the information display object DSH3 is subject to the display mode change process at the timing when a short elapsed time TE3 has elapsed since it was displayed. In this case, the information display objects DSH1 and DSH2 having a long elapsed time TE1 and TE2 after being displayed are subjected to display mode change processing such as erasing and translucency. On the other hand, for the information display object DSH3 having a short elapsed time TE3 after being displayed, the display mode change processing such as erasing and semi-transparency is not performed.

For example, for the information displays DSH1 and DSH2 having long elapsed times TE1 and TE2 after being displayed, there is a possibility that the user is viewing these information displays DSH1 and DSH2 during the long elapsed times TE1 and TE2. It is considered that the information transmission by the information display objects DSH1 and DSH2 is sufficiently performed. Therefore, with respect to such information display objects DSH1 and DSH2, when the positional relationship of collision or approach with respect to the object occurs, the display mode is changed such as erasure or translucency.

On the other hand, for the information display object DSH3 having a short elapsed time TE3 since it is displayed, it is unlikely that the user has sufficiently viewed the information display object DSH3 during the short elapsed time TE3, and the information by the information display object DSH3 Communication is considered inadequate. Therefore, with respect to such an information display object DSH3, when the object is in a positional relationship of collision or approach, the display mode change processing such as erasing or translucency is not performed.

That is, when a large number of information display objects are displayed, if all the information display objects that collide with or approach the object are erased or made translucent, the information transmission by the information display objects is performed. Will be inadequate. In this regard, according to the method of FIG. 12, since it is determined whether or not to perform the display mode change process by using the elapsed time from the display as a judgment material, it is effective to cause such a problem. Can be prevented.

3.3 Specific examples of display mode change processing and positional relationship determination Next, specific examples of display mode change processing and positional relationship determination will be described. For example, as the display mode change process, the information display object or object is erased or translucently processed, the tracking of the information display object in the user's line of sight is canceled, the information display object or object is moved, or the information display is performed. There is a process of resizing an object or an object.

For example, in FIG. 13A, as the display mode change process, the information display object DS or the object OB is erased or translucent. For example, when the information display object DS and the object OB are in a positional relationship of collision or approach, one of them is erased or semi-transparent is performed.

In FIGS. 13 (B) and 13 (C), as a process of changing the display mode, a process of canceling the tracking of the information display object DS in the line-of-sight direction VL of the user PL is performed. For example, in the present embodiment, as described with reference to FIGS. 8A to 8D, the information display object DS is controlled so as to follow the change in the line-of-sight direction VL of the user PL.

On the other hand, in FIG. 13B, when the information display object DS and the object OB collide or approach each other, the tracking control is released. As a result, even when the line-of-sight direction VL of the user PL changes as shown in FIG. 13C, the position of the information display object DS does not change. Therefore, it is possible to prevent the occurrence of problems such as the information display object DS colliding with the object OB and being sunk into it.

Further, in FIG. 14A, when the information display object DS and the object OB are in a positional relationship of collision or approach, for example, the information display object DS is moved so as to separate them. For example, in FIG. 14A, since the object OB is located on the right side of the information display object DS, the information display object DS is moved to the left so that the two are separated from each other. As a result, even when the information display object DS and the object OB are in a positional relationship of collision or approach, a more appropriate image can be displayed by separating them by movement. Although the information display object DS is moved in FIG. 14A, the object OB may be moved. Specifically, the object OB is moved to the right so that the information display object DS and the object OB are separated from each other.

In FIG. 14B, the size of the information display object DS is changed when the information display object DS and the object OB are in a positional relationship of collision or approach. For example, by reducing the size of the information display object DS, the distance between the right end portion of the information display object DS and the left end portion of the object OB is increased, and the collision between the two is eliminated. The size of the object OB may be reduced.

In FIG. 14C, the information display object DS has a positional relationship of being located closer to the user's viewpoint than the object OB. In this case, as a process of changing the display mode, for example, a process of reducing the size of the information display object DS is performed. By doing so, it is possible to prevent the occurrence of a situation in which the object OB is hidden behind the information display object DS and cannot be seen. If the object OB is located closer to the information display object DS, the size of the object OB may be reduced.

Next, a specific example of determining the positional relationship between the information display object and the object will be described. In the present embodiment, the determination unit 115 of FIG. 1 determines the positional relationship between the information display object and the object, and when the determination unit 115 determines that the information display object or the object has a positional relationship of approach or collision, the display mode of the information display object or the object. Change processing is performed. Specifically, the determination unit 115 is based on the three-dimensional coordinate values of the information display object and the object, the depth value of the information display object and the object, or the two-dimensional coordinate value when the information display object and the object are projected on a plane. Such a positional relationship is determined.

For example, in FIG. 15 (A), the three-dimensional coordinate values (XA, YA, ZA) of the information display object DS and the three-dimensional coordinate values (XB, YB, ZB) of the object OB are used to establish a positional relationship between the two. It is judged whether or not it has become. Specifically, in addition to these three-dimensional coordinate values, the positional relationship of approach is determined by using the size information and shape information of the information display object DS and the object OB. For example, by determining whether or not the distance between the end of the information display object DS (for example, the right end) and the end of the object OB (for example, the left end) is equal to or less than a predetermined distance, the positional relationship of approach can be determined. Can be judged. Alternatively, the positional relationship of approach may be determined based on the three-dimensional coordinate values of a plurality of vertices of the polygons constituting the information display object DS and the object OB.

In FIG. 15B, the three-dimensional coordinate values (XA, YA, ZA) of the information display object DS and the three-dimensional coordinate values (XB, YB, ZB) of the object OB are used, and the two are in a collision positional relationship. It is judged whether or not it is. The determination of the positional relationship of the collision can be realized by a known collision process (hit determination). For example, by determining whether or not the information display object DS and the object OB intersect by a known collision process, it is possible to determine the positional relationship of the collision.

In FIGS. 16A and 16B, the positional relationship between the depth value ZA (Z value) of the information display object DS and the depth value ZB of the object OB is determined. For example, in FIG. 16A, ZA <ZB, and it is determined that the information display object DS is located in front of the object OB from the user's point of view. Therefore, the display mode of the information display object DS is changed. For example, by performing a process of erasing or making the information display object DS semi-transparent, it is possible to prevent a situation in which the object OB is hidden behind the information display object DS and cannot be seen. On the other hand, in FIG. 16B, ZA> ZB, and it is determined that the object OB is located on the front side of the information display object DS from the user's point of view. Therefore, the display mode of the object OB is changed. For example, by performing a process of erasing or translucenting the object OB, it is possible to prevent a situation in which the information display object DS is hidden behind the object OB and cannot be seen.

Further, in FIGS. 17 (A) and 17 (B), the two-dimensional coordinate values (X and Y coordinate values in the plane SPL) when the information display object DS and the object OB are projected onto the plane SPL such as a projection screen. Based on this, the positional relationship between the two is determined. For example, in FIGS. 17A and 17B, it is determined that the information display object DS and the object OB overlap on the plane SPL. In FIG. 17A, the information display object DS is located on the front side when viewed from the viewpoint, and in FIG. 17B, the object OB is located on the front side when viewed from the viewpoint. By determining such a positional relationship, one of the display modes of the information display object DS and the object OB is changed. For example, in FIG. 17A, by performing the process of erasing or making the information display object DS semi-transparent, it is possible to prevent a situation in which a part of the image of the object OB is lost behind the information display object DS. On the other hand, in FIG. 17B, by performing the process of erasing or translucenting the object OB, it is possible to prevent a situation in which a part of the image of the information display object DS is hidden behind the object OB and is missing.

By determining the positional relationship as described above and performing the process of changing the display mode of the information display object or the object, it becomes possible to realize an appropriate process of changing the display mode in consideration of the positional relationship between the two. Appropriate display images can be displayed on the HMD.

3.4 Command processing Next, the details of the command processing of the present embodiment described with reference to FIG. 6 will be described. In conventional communication games using characters and the like, so-called two-dimensional games have been the mainstream, and relationships such as the positional relationship between the user and the character and the line-of-sight relationship have not been taken into consideration. Therefore, for example, when the user exercises the command CM3 "call out" in FIG. 6, the effect activated by this command CM3 is the same regardless of whether the distance between the user and the character is short or long. It ends up. Therefore, there is a problem that the user cannot feel a sense of distance from the character and cannot improve the virtual reality of the user.

Therefore, in the present embodiment, a method of changing the effect of the command is adopted according to the positional relationship between the user and the character, the line-of-sight relationship, the gaze state, and the like. For example, in the present embodiment, the effect of a command corresponding to the information of the character to be the target of the user's game play and the viewpoint information of the user is activated. For example, the degree of effect of the command and the content of the effect are changed according to the character information and the user's viewpoint information. Specifically, character information such as character position information, direction information, game parameter information, part information, posture information or type information, and viewpoint information such as user's viewpoint position information, line-of-sight direction information or virtual camera attribute information. Invokes the effect of the command according to.

18 (A) to 19 (B) are explanatory views of a method of invoking the effect of a command according to the positional relationship information between the user PL and the character CH.

For example, in FIGS. 18A and 18B, the degree of effect of the command is changed according to the positional relationship information between the user PL and the character CH. Here, the positional relationship information is the distance LC between the user PL and the character CH, and this distance LC is obtained based on the viewpoint VP which is the viewpoint information of the user PL and the position PC which is the information of the character CH. is there. The positional relationship information may be information indicating the relationship between the direction of the user PL and the direction of the character CH.

Then, in FIGS. 18A and 18B, the user selects and exercises the command CM1 "more concentrated!", But in FIG. 18A, the distance LC between the user PL and the character CH is far. On the other hand, in FIG. 18B, the distance LC is short.

In this case, in the present embodiment, even if the user exercises the same command CM1 as shown in FIGS. 18A and 18B, the degree of the effect activated by the command CM1 changes according to the distance LC. I'm letting you. For example, in FIG. 18 (A), since the distance LC is long, the degree of effect of the command CM1 is reduced accordingly, and in FIG. 18 (B), since the distance LC is short, the degree of effect of the command CM1 is correspondingly low. Will be higher.

Specifically, when the command CM1 "more concentrated!" Is exercised, in FIG. 18 (A), since the distance LC is long, as shown in A1, the game parameter of the concentration power of the character CH increases. The degree is small. On the other hand, in FIG. 18B, since the distance LC is short, the degree of increase in the game parameter of the concentration power of the character CH becomes large as shown in A2.

In this way, when the user exercises the command CM1 "more concentrated!", The user goes near the character CH and exercises the command CM1 to further enhance the concentration of the character CH. Becomes possible. Therefore, it is possible to give the user the feeling of saying the word "more concentrated" near the character CH, and it is possible to improve the virtual reality of the user.

19 (A) and 19 (B) change the content of the effect of the command according to the positional relationship information between the user PL and the character CH. That is, the degree of the effect was changed in FIGS. 18 (A) and 18 (B), but the content of the effect was changed in FIGS. 19 (A) and 19 (B).

For example, in FIG. 19A, the user is exercising the command CM3 to "call out" in a state where the distance LC from the character CH is far. In this case, for example, a voice such as "Hey!" Is output as a user's "call out" word, or a telop (speech balloon) of "Hey!" Is displayed on the game screen.

On the other hand, in FIG. 19B, the user exercises the command CM3 "call out" while the distance LC from the character CH is close. In this case, as the user's "calling" words, for example, the voices of the words "OK!" And "Are you doing your best?" Are output, and the telops of these words are displayed on the game screen.

That is, in FIG. 19A, the content of the effect of the command CM3 "call out" is the content of voice output (telop display) of the word "hey!". On the other hand, in FIG. 19B, as the distance LC gets closer, the content changes to the voice output of the words "OK!" And "Are you doing your best?".

In this way, even with the same command CM3 of "calling out", the content of the effect of the command changes according to the positional relationship between the user PL and the character CH. Therefore, it is possible to improve the virtual reality of the user and the variety of game expressions.

20 (A) and 20 (B) are explanatory views of a method of invoking the effect of a command according to the line-of-sight relationship information between the user PL and the character CH.

For example, in FIGS. 20A and 20B, the degree of effect of the command and the content of the effect are changed according to the line-of-sight relationship information between the user PL and the character CH. In FIGS. 20A and 20B, the line-of-sight relationship information is information representing the relationship between the line-of-sight direction VL of the user PL and the position PC of the character CH and the line-of-sight direction VLC.

Then, in FIGS. 20 (A) and 20 (B), the user selects and exercises the command CM5 "to give advice", but in FIG. 20 (A), the line-of-sight direction VL of the user PL is a character. CH position Not facing the direction of the PC. On the other hand, in FIG. 20B, the line-of-sight direction VL of the user PL faces the direction of the position PC of the character CH. Further, in FIG. 20A, the line-of-sight direction VL of the user PL and the line-of-sight direction VLC of the character CH do not match. On the other hand, in FIG. 20B, the line-of-sight direction VL and VLC match.

In this case, in the present embodiment, even if the user exercises the same command CM5 as shown in FIGS. 20A and 20B, the degree of the effect activated by the command CM2 is determined according to the line-of-sight information. The content of the effect is changing.

For example, in the case of the line-of-sight relationship shown in FIG. 20A, "ordinary advice" is given as the content of the effect of invoking the command CM5 to "give advice". For example, mediocre advice is given in a normal tone. Then, as an effect of "ordinary advice" by the command CM5, the motivation which is a game parameter of the character CH increases, but in FIG. 20A, the degree of the increase is small as shown in B1.

On the other hand, in the case of the line-of-sight relationship shown in FIG. 20B, "gentle advice" is given as the content of the effect of the command CM5 "give advice". For example, motivational advice is given in a gentle tone. Then, as an effect of the "gentle advice" by the command CM5, the motivation of the character CH increases, and in FIG. 20B, the degree of the increase is large as shown in B2.

In this way, as shown in FIG. 20 (A), as compared with the case where the user looks away from the character CH and exercises the “advice” command CM5, FIG. 20 (B). ), When the user looks at the character CH properly and exercises the command CM5, the advice has better content, and the degree of increase in motivation, which is the effect of the advice, also increases. Therefore, the user can feel as if he / she is giving advice to the opposite sex in front of him / her, and the virtual reality of the user can be greatly improved.

FIG. 21 is an explanatory diagram of a method of invoking the effect of a command according to the gaze information of the user. For example, in FIG. 21, the effect of the command corresponding to the gaze part information, which is the information of the part of the character CH that the user is gaze at, is activated.

Specifically, in FIG. 21, the command CM4 "poke" is exercised. Here, as shown in C1 of FIG. 21, it is assumed that the user exercises the command CM4 of "poke" while gazing at the shoulder which is the part of the character CH. In this case, an image (video) expressing a scene in which the user pokes the shoulder of the character CH is displayed. For example, a user's hand (object) appears on the screen, and an image is displayed in which the hand pokes the shoulder of the character CH. On the other hand, as shown in C2 of FIG. 21, it is assumed that the user exercises the command CM4 "poke" while gazing at the head (forehead) which is the part of the character CH. In this case, an image representing a scene in which the user pokes the head of the character CH is displayed. For example, a user's hand appears on the screen, and an image is displayed in which the hand pokes the head of the character CH.

By doing so, even when the same command CM4 of "poke" is exercised, "poke" of different contents of operation is performed according to the part of the character CH that the user is gazing at. Then, since the user's action is performed on the part that the user gazes at, the user can feel as if he / she is actually poking the character CH, and the virtual reality of the user can be improved. ..

According to the command processing method of the present embodiment as described above, the virtual reality of the user can be further improved. On the other hand, in the command processing method as in the present embodiment, a large number of information display objects are arranged and displayed as shown in FIGS. 6 and 21. Therefore, problems such as collision between the information display object and other objects in the virtual space are likely to occur. In this regard, in the present embodiment, even when the information display object and the object have a positional relationship such as a collision, the process of changing the display mode of the information display object or the object is performed, so that a more appropriate display image can be displayed by the HMD. It becomes possible to display in.

3.5 Processing Example Next, a processing example of the present embodiment will be described with reference to the flowchart of FIG. First, the tracking information of the user's viewpoint information is acquired (step S1). For example, the tracking information obtained by the tracking process described with reference to FIGS. 2 (A) to 3 (B) and the like is acquired. Then, based on the acquired tracking information, the virtual camera is controlled so as to follow the change in the viewpoint of the user (step S2). For example, when the user turns the line-of-sight direction in the vertical and horizontal directions in the real world, the virtual camera is controlled so that the line-of-sight direction of the virtual camera in the virtual space also faces the vertical and horizontal directions. Further, when the user's viewpoint position moves in the vertical and horizontal directions in the real world, the virtual camera is controlled so that the viewpoint position of the virtual camera in the virtual space also moves in the vertical and horizontal directions accordingly.

Next, the positional relationship between the information display object and the object is determined (step S3). For example, the positional relationship between the two is determined by the methods described in FIGS. 15 (A) to 17 (B). Then, it is determined whether or not the information display object and the object are in a positional relationship of collision or approach (step S4). Then, when there is a positional relationship of collision or approach, a process of changing the display mode of the information display object or the object is performed (step S5). For example, the display mode changing process described with reference to FIGS. 13 (A) to 14 (C) is performed. Then, an image that can be seen from the virtual camera in the virtual space is generated and displayed on the HMD (step S6). For example, as an image that can be seen from a virtual camera, stereoscopic left-eye and right-eye images are generated and displayed on the HMD.

Although the present embodiment has been described in detail as described above, those skilled in the art will easily understand that many modifications that do not substantially deviate from the novel matters and effects of the present invention are possible. Therefore, all such modifications are included in the scope of the present invention. For example, in a specification or drawing, a term (change in line-of-sight direction, character, book, etc.) described at least once with a different term (viewpoint change, moving object, item, etc.) in a broader sense or synonymous is a specification or drawing. Can be replaced with the different term anywhere in. Further, the virtual space setting process, the virtual camera control process, the information display object placement process, the display mode change process, the positional relationship determination process, and the like are not limited to those described in the present embodiment, and are equivalent to these. The method, processing, and configuration are also included in the scope of the present invention. The present invention can also be applied to various games. Further, the present invention can be applied to various simulation systems such as a commercial game device, a home game device, or a large attraction system in which a large number of users participate.

DSA, DSB, DSC, DSD1, DSD2, DSE1-DSE3 information display,
DS, DS1, DS2 information display, CH character (moving body),
BK book (item), CT curtain (fixed object), WL wall (background),
OB, OB1, OB2 object, PL user, VP viewpoint, VL line-of-sight direction,
100 processing unit, 102 input processing unit, 110 arithmetic processing unit, 112 game processing unit,
114 Mobile processing unit, 115 Judgment unit, 116 Virtual space setting unit,
118 virtual camera control unit, 120 display processing unit, 130 sound processing unit,
140 output processing unit, 150 imaging unit, 151, 152 first and second cameras,
160 operation unit, 166, 167 game controller,
170 storage, 172 object information storage, 178 drawing buffer,
180 Information storage medium, 192 sound output section, 194 I / F section,
195 Portable information storage medium, 196 Communication unit,
200 HMD (Head-mounted display), 201-209 light receiving element,
210 sensor unit, 220 display unit, 231 to 238 light emitting element, 240 processing unit,
260 headband, 270 headphones, 280, 284 base station

Claims (14)

A virtual space setting unit that performs setting processing for a virtual space in which multiple objects are placed,
A virtual camera control unit that performs control processing of the virtual camera corresponding to the viewpoint of the user who wears the head-mounted display device,
A display processing unit that generates an image that can be seen from the virtual camera in the virtual space as a display image of the head-mounted display device.
Including
The virtual space setting unit
At least one information display object controlled to follow the change in the viewpoint of the user is arranged in the virtual space.
Each of the information display objects is associated with importance information indicating how important the information presented by the information display object is.
The display processing unit
When the information display object and the object in the virtual space have a given positional relationship, the information display object or the display mode of the object is used by using the importance information associated with the information display object. A simulation system characterized by performing change processing. In claim 1,
Including a storage unit in which the importance information is associated with the information display object and stored.
The display processing unit
A simulation system characterized in that the display mode is changed by using the importance information associated with the information display object in the storage unit. In claim 1 or 2,
The display processing unit
When the importance information indicating that the information display object has high importance is associated with the information display object, the display mode of the object is changed and the importance is low with respect to the information display object. When the importance information indicating the above is associated, the simulation system is characterized in that the display mode of the information display object is changed. In any one of claims 1 to 3 ,
The display processing unit
A simulation system characterized in that the display mode is changed according to the elapsed time from the display of the information display object. In any of claims 1 to 4 ,
The display processing unit
As the process of changing the display mode,
Erase processing of the information display object or the object, translucency processing of the information display object or the object, cancellation processing of tracking of the information display object in the line-of-sight direction of the user, movement of the information display object or the object. A simulation system characterized by performing processing, resizing processing of the information display object or the object, or color information changing processing of the information display object or the object. In any of claims 1 to 5 ,
The information display object is a display object for explanation display, a display object for status display, a display object for conversation display, a display object for command selection, or a display object for menu display. A featured simulation system. In any of claims 1 to 6 ,
A simulation system characterized in that the object is a moving object, a fixed object, an item, or a background object appearing in a game. In any of claims 1 to 7 ,
Includes a determination unit that determines the positional relationship between the information display object and the object.
The display processing unit
A simulation system characterized in that when the determination unit determines that the information display object and the object have a given positional relationship, the display mode is changed. In claim 8 .
The display processing unit
A simulation system characterized in that when the determination unit determines that the information display object and the object have a collision positional relationship or an approaching positional relationship, the display mode is changed. In claim 8 or 9 ,
The determination unit
The positional relationship is based on the three-dimensional coordinate values of the information display object and the object, the depth value of the information display object and the object, or the two-dimensional coordinate values when the information display object and the object are projected on a plane. A simulation system characterized by determining. In any of claims 1 to 10 ,
The virtual space setting unit
A simulation system characterized in that the information display object is arranged at a given depth position in the line-of-sight direction of the user so as to follow a change in the viewpoint of the user. In any of claims 1 to 11 ,
The virtual camera control unit
A simulation system characterized in that the virtual camera is controlled so as to follow a change in the viewpoint of the user based on tracking information of the viewpoint information of the user. In any of claims 1 to 12 ,
Including the game processing unit that performs game processing
The game processing unit
A simulation system characterized in that a process of invoking the effect of a command corresponding to the information of a character to be the target of the game play of the user and the viewpoint information of the user is performed. A virtual space setting unit that performs setting processing for a virtual space in which multiple objects are placed,
A virtual camera control unit that performs control processing of the virtual camera corresponding to the viewpoint of the user who wears the head-mounted display device,
As a display processing unit that generates an image that can be seen from the virtual camera in the virtual space as a display image of the head-mounted display device.
Make your computer work
The virtual space setting unit
At least one information display object controlled to follow the change in the viewpoint of the user is arranged in the virtual space.
Each of the information display objects is associated with importance information indicating how important the information presented by the information display object is.
The display processing unit
When the information display object and the object in the virtual space have a given positional relationship, the information display object or the display mode of the object is used by using the importance information associated with the information display object. A program characterized by performing change processing.

Images