JP2018025955A - Information processing method, and program for causing computer to implement information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a virtual experience in which a user interacts with a virtual object.SOLUTION: A method for using a computer to provide in a head-mounted display (HMD, in the following) a virtual space in which a user is immersed, includes the steps of: moving a first manipulation object in accordance with the movement of a first portion of the user's body; moving a second manipulation object in accordance with the movement of a second portion of the user's body; selecting a target object in accordance with the movement of the first portion; and, with the target object selected, deforming the target object in accordance with the movement of the second portion.SELECTED DRAWING: Figure 12

Description

  The present disclosure relates to an information processing method and a program for causing a computer to execute the information processing method.

  Non-Patent Document 1 changes the state of a hand object in a virtual reality (VR) space and manipulates the hand object according to the state (position, inclination, etc.) of the user's hand in the real space. Discloses that a predetermined action is given to a predetermined object in the virtual space.

“Toybox Demo for Oculus Touch”, [online], October 13, 2015, Oculus, [search August 6, 2016], Internet <https: // www. youtube. com / watch? v = iFEMiyGMa58>

  In Non-Patent Document 1, there is room for improvement in operating a predetermined object with a hand object. For example, there is room for improvement in providing the user with a virtual experience that cannot be experienced by manipulating actual objects in real space, and the user needs to be able to manipulate the desired virtual object at the desired timing There is. Accordingly, in addition to the VR space, the virtual experience in which the user interacts with the virtual object in various environments such as augmented reality (AR) space and mixed reality (MR) space can be improved.

  An object of the present disclosure is to provide an information processing method capable of improving a virtual experience and a program for causing a computer to realize the information processing method.

According to one aspect of the present disclosure, a head-mounted display, a position sensor configured to detect the position of the head-mounted display and the positions of the first and second parts of the body other than the user's head; An information processing method in a system comprising:
(A) identifying virtual space data defining a virtual space including a virtual camera, a first operation object, a second operation object, and a target object;
(B) moving the virtual camera in response to movement of the head mounted display;
(C) moving the first operation object in accordance with the movement of the first part;
(D) moving the second operation object in accordance with the movement of the second portion;
(E) selecting the target object according to the movement of the first part;
(F) With the target object selected, deforming the target object according to the movement of the second operation object;
(G) defining a visual field of the virtual camera based on the movement of the virtual camera, and generating visual field image data based on the visual field and the virtual space data;
(H) displaying a visual field image on the head-mounted display based on the visual field image data. Is provided.

  According to the present disclosure, it is possible to provide an information processing method capable of improving a virtual experience and a program for causing a computer to realize the information processing method.

It is the schematic which shows a head mounted display (Head Mounted Display: HMD) system. It is a figure which shows the head of the user with which HMD was mounted | worn. It is a figure which shows the hardware constitutions of a control apparatus. It is a figure which shows an example of a specific structure of an external controller. It is a flowchart which shows the process which displays a visual field image on HMD. It is xyz space figure which shows an example of virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. State (a) is a diagram showing a user wearing the HMD and an external controller. The state (b) is a diagram illustrating a virtual space including a virtual camera, a hand object, and a target object. It is a flowchart for demonstrating the information processing method which concerns on this embodiment. It is a flowchart for demonstrating the information processing method which concerns on this embodiment. A state in which the target object 500 selected by the right hand object 400R is deformed by the left hand object 400L is shown. A state in which the target object 500 selected by the right hand object 400R is deformed by the left hand object 400L is shown. A state in which the orientation of the target object 500 is changed by moving the right hand object 400R after the target object 500R is selected by the right hand object 400R is shown. The change of the coordinate information of the target object 500 before and after the deformation of the target object 500 is shown. It is a figure which shows an example of the visual field image displayed on HMD.

[Description of Embodiments Presented by the Present Disclosure]
An overview of an embodiment indicated by the present disclosure will be described.
(Item 1)
An information processing method in a system comprising: a head mounted display; and a position sensor configured to detect a position of the head mounted display and positions of a first part and a second part of a body other than a user's head. And
(A) identifying virtual space data defining a virtual space including a virtual camera, a first operation object, a second operation object, and a target object;
(B) moving the virtual camera in response to movement of the head mounted display;
(C) moving the first operation object in accordance with the movement of the first part;
(D) moving the second operation object in accordance with the movement of the second portion;
(E) selecting the target object according to the movement of the first part;
(F) With the target object selected, deforming the target object according to the movement of the second operation object;
(G) defining a visual field of the virtual camera based on the movement of the virtual camera, and generating visual field image data based on the visual field and the virtual space data;
(H) displaying a field image on the head-mounted display based on the field image data;
Including an information processing method.
According to the information processing method of this item, the target object can be deformed by moving the first operation object, selecting the target object, and moving the second operation object. A virtual experience that can be manipulated can be provided.
(Item 2)
In (e), the target object is selected by contact between the first operation object and the target object,
The method according to item 1, wherein in (f), after the second operation object and the target object are in contact with each other, the target object is deformed based on a direction in which the second operation object has moved.
Thereby, the target object can be easily deformed based on the movement of the second operation object, and a virtual experience that allows the user to manipulate the target object at will can be provided.
(Item 3)
The target object includes coordinate information in the virtual space,
The method according to item 2, wherein the coordinate information of the target object is updated based on a direction in which the second operation object has moved.
Thereby, it is possible to prevent the positional relationship between the target object and the first operation object from becoming unnatural due to the deformation of the target object.
(Item 4)
The method according to item 3, wherein when the second operation object moves in a predetermined direction by a predetermined distance, the coordinate information is changed so as to move by a half of the predetermined distance in the predetermined direction.
Thereby, it is possible to prevent the positional relationship between the target object and the first operation object from becoming unnatural due to the deformation of the target object.
(Item 5)
In (e), when the target object is selected, a direction display indicating a deformation direction in which the target object can be deformed according to the movement of the second operation object is displayed in association with the target object. , Any one of items 1-4.
Thereby, the deformation | transformation rule of a target object can be clearly shown to a user.
(Item 6)
The method of item 5, wherein in (f), the direction display is erased by detecting the movement of the second operation object.
Thereby, the time which displays a direction display can be reduced and it can prevent that a direction display obstructs a user's virtual experience.
(Item 7)
The virtual camera defines a visual field coordinate system that moves according to the movement of the head-mounted display, and the visual field coordinate system includes a vertical direction, a horizontal direction, and a depth direction,
In (e), when the target object is selected, the first operation object in the target object among the upper direction and / or the lower direction, the depth direction and / or the front direction, and the horizontal direction of the target object. The method according to any one of items 1 to 6, wherein at least one of the unselected directions is specified as a deformation direction capable of deforming the target object in accordance with a movement of the second operation object.
Thereby, the deformation direction of the target object can be limited to a dimension that is easy for the user to operate, and the processing load on the computer can be reduced without impairing the virtual experience of the user.
(Item 8)
Even if the direction of the target object is changed by moving the first operation object after the deformation direction capable of deforming the target object according to the movement of the second operation object is specified, The method of item 7, wherein the deformation direction is not changed.
Accordingly, even when the orientation of the target object is changed so that the user can easily view the predetermined deformation direction in order for the user to deform the target object in the predetermined deformation direction, the user does not change the deformation direction. Can provide a consistent feeling of operation.
(Item 9)
A program for causing a computer to execute any one of items 1 to 8.

[Details of Embodiments Presented by the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, about the member which has the same reference number as the member already demonstrated in description of this embodiment, the description is not repeated for convenience of explanation.

  First, the configuration of the head mounted display (HMD) system 1 will be described with reference to FIG. FIG. 1 is a schematic diagram showing an HMD system 1. As shown in FIG. 1, the HMD system 1 includes an HMD 110 mounted on the head of the user U, a position sensor 130, a control device 120, and an external controller 320.

  The HMD 110 includes a display unit 112, an HMD sensor 114, and a gaze sensor 140. The display unit 112 includes a non-transmissive display device configured to cover the field of view (field of view) of the user U wearing the HMD 110. Thereby, the user U can immerse in the virtual space by viewing the visual field image displayed on the display unit 112. The display unit 112 includes a display unit for the left eye configured to provide an image to the left eye of the user U and a display unit for the right eye configured to provide an image to the right eye of the user U. Also good. The HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance. Further, the visual field image may include a configuration for presenting the real space in a part of the image configuring the virtual space. For example, an image captured by a camera mounted on the HMD 110 may be displayed so as to be superimposed on a part of the field-of-view image, or by setting the transmittance of a part of the transmissive display device to be high. Real space may be visible from a part.

  The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (such as an angular velocity sensor and a gyro sensor), and can detect various movements of the HMD 110 mounted on the head of the user U.

  The gaze sensor 140 has an eye tracking function that detects the direction of the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor irradiates, for example, infrared light to the right eye of the user U, and detects reflected light reflected from the right eye (particularly the cornea and iris), thereby acquiring information related to the rotation angle of the right eye's eyeball. May be. On the other hand, the left eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly the cornea and iris), thereby providing information on the rotation angle of the left eye's eyeball. May be obtained.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 by wireless or wired communication, and is configured to detect information on the position, inclination, or light emission intensity of a plurality of detection points (not shown) provided in the HMD 110. . Further, the position sensor 130 is configured to detect information on the position, inclination, and / or emission intensity of a plurality of detection points 304 (see FIG. 4) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may include an infrared sensor and a plurality of optical cameras.

  The control device 120 acquires motion information such as the position and orientation of the HMD 110 based on information acquired from the HMD sensor 114 and the position sensor 130, and based on the acquired motion information, a virtual viewpoint (virtual The position and orientation of the camera) can be accurately associated with the position and orientation of the user U wearing the HMD 110 in the real space. Further, the control device 120 acquires the movement information of the external controller 320 based on the information acquired from the position sensor 130, and based on the acquired movement information, a finger object (described later) is displayed in the virtual space. Can be accurately associated with the relative relationship between the position and orientation between the external controller 320 and the HMD 110 in the real space. Note that the motion information of the external controller 320 may be a geomagnetic sensor, an acceleration sensor, a tilt sensor, or the like mounted on the external controller 320, as with the HMD sensor 114.

  Based on the information transmitted from the gaze sensor 140, the control device 120 identifies the gaze of the right eye and the left gaze of the user U, and identifies the gaze point that is the intersection of the gaze of the right eye and the gaze of the left eye. be able to. Furthermore, the control device 120 can specify the line-of-sight direction of the user U based on the specified gaze point. Here, the line-of-sight direction of the user U is the line-of-sight direction of both eyes of the user U and coincides with the direction of a straight line passing through the middle point of the line segment connecting the right eye and the left eye of the user U and the gazing point.

  With reference to FIG. 2, a method for acquiring information related to the position and orientation of the HMD 110 will be described. FIG. 2 is a diagram illustrating the head of the user U wearing the HMD 110. Information on the position and orientation of the HMD 110 linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands up is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 is an angle around each uvw axis (that is, a yaw angle indicating rotation around the v axis, a pitch angle indicating rotation around the u axis, and a center around the w axis). The inclination determined by the roll angle indicating rotation) is detected. The control device 120 determines angle information for defining the visual axis from the virtual viewpoint based on the detected angle change around each uvw axis.

  The hardware configuration of the control device 120 will be described with reference to FIG. FIG. 3 is a diagram illustrating a hardware configuration of the control device 120. The control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via a bus 126 so as to communicate with each other.

  The control device 120 may be configured as a personal computer, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. In addition, some functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

  The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and a program specified from various programs incorporated in the ROM is expanded on the RAM. It is comprised so that various processes may be performed in cooperation with.

  The control unit 121 allows the control unit 121 to execute a program for causing the computer to execute the information processing method according to this embodiment (described later) on the RAM and execute the program in cooperation with the RAM. Various operations of 120 may be controlled. The control unit 121 displays a virtual space (field-of-view image) on the display unit 112 of the HMD 110 by executing predetermined application programs (including game programs and interface programs) stored in the memory and the storage unit 123. . Thereby, the user U can be immersed in the virtual space displayed on the display unit 112.

  The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a program that causes a computer to execute the information processing method according to the present embodiment. In addition, a user U authentication program, a game program including data on various images and objects, and the like may be stored. Furthermore, a database including tables for managing various data may be constructed in the storage unit 123.

  The I / O interface 124 is configured to connect the position sensor 130, the HMD 110, and the external controller 320 to the control device 120 in a communicable manner. For example, a USB (Universal Serial Bus) terminal, a DVI (Digital) The terminal includes a Visual Interface (HDMI) terminal, a High-Definition Multimedia Interface (HDMI) (registered trademark) terminal, and the like. The control device 120 may be wirelessly connected to each of the position sensor 130, the HMD 110, and the external controller 320.

  The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with external devices on the network via the communication network 3 and various processing circuits for wireless connection, and for communicating via the communication network 3. It is configured to conform to the communication standard.

  An example of a specific configuration of the external controller 320 will be described with reference to FIG. The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, which is the user U's hand in the present embodiment), thereby moving the hand object displayed in the virtual space. Used to control. The external controller 320 is operated by a right hand external controller 320R (hereinafter simply referred to as a controller 320R) operated by the right hand (first part of the body) of the user U and a left hand (second part of the body) of the user U. Left-hand external controller 320L (hereinafter simply referred to as controller 320L). The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the finger of the right hand. Further, the right hand object 400R (see FIG. 9) that exists in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the finger of the left hand. Further, the left hand object 400L (see FIG. 9) that exists in the virtual space moves according to the movement of the controller 320L. Since the controller 320R and the controller 320L have substantially the same configuration, only the specific configuration of the controller 320R will be described below with reference to FIG. In the following description, the controllers 320L and 320R may be simply referred to as the external controller 320 for convenience. Further, the left hand object 400L and the right hand object 400R may be simply referred to as the hand object 400 in some cases. Further, the right hand object 400R may be referred to as an example of a first operation object, and the left hand object 400L may be referred to as an example of a second operation object.

  As shown in FIG. 4, the controller 320R includes an operation button 302, a plurality of detection points 304, a sensor (not shown), and a transceiver (not shown). Only one of the detection point 304 and the sensor may be provided. The operation button 302 includes a plurality of button groups configured to accept an operation input from the user U. The operation button 302 includes a push button, a trigger button, and an analog stick. The push-type button is a button operated by an operation of pressing with a thumb. For example, two push buttons 302 a and 302 b are provided on the top surface 322. It is preferable to change the thumb of the hand object 400 from a stretched state to a bent state in response to the thumb placed on the top surface 322 or the push buttons 302a and 302b being pressed. The trigger type button is a button operated by an operation of pulling a trigger with an index finger or a middle finger. For example, a trigger button 302 e is provided on the front surface portion of the grip 324, and a trigger button 302 f is provided on the side surface portion of the grip 324. The trigger type button 302e is assumed to be operated by an index finger, and is preferably changed from a stretched state to a bent state by pushing the index finger of the hand object 400. The trigger button 302f is assumed to be operated by the middle finger, and is preferably changed from a stretched state to a bent state by the middle finger, the ring finger, and the little finger in the hand object 400 when pressed. The analog stick is a stick-type button that can be operated by being tilted 360 degrees from a predetermined neutral position in an arbitrary direction. For example, it is assumed that an analog stick 320i is provided on the top surface 322 and is operated using a thumb.

  The controller 320R includes a frame 326 that extends from both side surfaces of the grip 324 in a direction opposite to the top surface 322 to form a semicircular ring. A plurality of detection points 304 are embedded on the outer surface of the frame 326. The plurality of detection points 304 are, for example, a plurality of infrared LEDs arranged in a line along the circumferential direction of the frame 326. After the position sensor 130 detects information on the position, inclination, or light emission intensity of the plurality of detection points 304, the control device 120 detects the position or orientation (inclination / posture) of the controller 320R based on the information detected by the position sensor 130. The motion information including information on (direction) is acquired.

  The sensor of the controller 320R may be, for example, a magnetic sensor, an angular velocity sensor, an acceleration sensor, or a combination thereof. When the user U moves the controller 320R, the sensor outputs a signal (for example, a signal indicating information related to magnetism, angular velocity, or acceleration) according to the direction or movement of the controller 320R. The control device 120 acquires information related to the position and orientation of the controller 320R based on the signal output from the sensor.

  The transceiver of the controller 320R is configured to transmit and receive data between the controller 320R and the control device 120. For example, the transceiver may transmit an operation signal corresponding to the operation input of the user U to the control device 120. The transceiver may receive an instruction signal for instructing the controller 320 </ b> R to emit light at the detection point 304 from the control device 120. Further, the transceiver may send a signal to the controller 120 indicating the value detected by the sensor.

  A process for displaying the field-of-view image on the HMD 110 will be described with reference to FIGS. FIG. 5 is a flowchart showing a process for displaying the visual field image on the HMD 110. FIG. 6 is an xyz space diagram showing an example of the virtual space 200. The state (a) in FIG. 7 is a yx plan view of the virtual space 200 shown in FIG. The state (b) in FIG. 7 is a zx plan view of the virtual space 200 shown in FIG. FIG. 8 is a diagram illustrating an example of the visual field image M displayed on the HMD 110.

  As shown in FIG. 5, in step S1, the control unit 121 (see FIG. 3) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as an omnidirectional sphere with the center position 21 as the center (in FIG. 6, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 21 as the origin is set. The virtual camera 300 defines a visual axis L for specifying a visual field image M (see FIG. 8) displayed on the HMD 110. The uvw coordinate system that defines the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system that is defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 according to the movement of the user U wearing the HMD 110 in the real space. The various objects in the virtual space 200 include, for example, a left hand object 400L, a right hand object 400R, and an operation object 500 (see FIG. 9).

  In step S2, the control unit 121 identifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on information regarding the position and tilt of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300 and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a partial area of the virtual space 200 that can be viewed by the user U wearing the HMD 110. In other words, the visual field CV corresponds to a partial area of the virtual space 200 displayed on the HMD 110. The visual field CV is viewed in the first region CVa set as an angular range of the polar angle α around the visual axis L in the xy plane shown in the state (a) and in the xz plane shown in the state (b). And a second region CVb set as an angle range of the azimuth angle β around the axis L. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and determines the direction of the virtual camera 300 based on the line of sight of the user U. May be.

  The control unit 121 can identify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 changes the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can change the visual field CV according to the movement of the HMD 110. Similarly, when the line-of-sight direction of the user U changes, the control unit 121 can move the visual field CV of the virtual camera 300 based on the data indicating the line-of-sight direction of the user U transmitted from the gaze sensor 140. That is, the control unit 121 can change the visual field CV according to the change in the user U's line-of-sight direction.

  In step S <b> 3, the control unit 121 generates visual field image data indicating the visual field image M displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

  In step S4, the control unit 121 displays the field image M on the display unit 112 of the HMD 110 based on the field image data (see FIG. 7). As described above, the visual field CV of the virtual camera 300 is updated according to the movement of the user U wearing the HMD 110, and the visual field image M displayed on the display unit 112 of the HMD 110 is updated. It is possible to immerse in the space 200.

  The virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye view image and the right-eye view image on the display unit 112 of the HMD 110 based on the left-eye view image data and the right-eye view image data. In this way, the user U can visually recognize the visual field image as a three-dimensional image from the left-eye visual field image and the right-eye visual field image. In the present disclosure, for convenience of explanation, the number of virtual cameras 300 is one, but the embodiment of the present disclosure is applicable even when the number of virtual cameras is two.

  The left hand object 400L, the right hand object 400R, and the operation object 500 included in the virtual space 200 will be described with reference to FIG. The state (a) shows the user U wearing the HMD 110 and the controllers 320L and 320R. The state (b) shows the virtual space 200 including the virtual camera 300, the right hand object 400R (an example of the first operation object), the left hand object 400L (an example of the second operation object), and the target object 500.

  As shown in FIG. 9, the virtual space 200 includes a virtual camera 300, a left hand object 400 </ b> L, a right hand object 400 </ b> R, and a target object 500. The control unit 121 generates virtual space data that defines the virtual space 200 including these objects. As described above, the virtual camera 300 is linked to the movement of the HMD 110 worn by the user U. That is, the visual field of the virtual camera 300 is updated according to the movement of the HMD 110. The right hand object 400R is a first operation object that moves according to the movement of the controller 320R attached to the right hand (the first part of the body) of the user U. The left hand object 400L is a second operation object that moves according to the movement of the controller 320L attached to the user U's left hand (second body part). Hereinafter, for convenience of explanation, the left hand object 400L and the right hand object 400R may be simply referred to as the hand object 400 in some cases. The left hand object 400L and the right hand object 400R each have a collision area CA. The collision area CA is used for collision determination (hit determination) between the hand object 400 and a target object (for example, the target object 500). For example, when the collision area CA of the hand object 400 and the collision area of the target object 500 are in contact, it is determined that the hand object 400 and the target object 500 are in contact. As shown in FIG. 9, the collision area CA may be defined by, for example, a sphere having a diameter R with the center position of the hand object 400 as the center. In the following description, it is assumed that the collision area CA is formed in a spherical shape having a diameter R with the center position of the object as the center.

  The target object 500 can be moved by the left hand object 400L and the right hand object 400R. For example, in the virtual space 200 shown in FIG. 8, a game is illustrated in which the character object CO is automatically moved on the passage RW from the START point toward the GOAL point. A hollow portion is provided on the passage RW, and the character object CO falls into the hollow on the way to the GOAL point, cannot reach the GOAL point, and the game is over. The user can guide the character object CO to the GOAL point by operating the target object 500 by operating the hand object 400 and covering the depression with the target object 500.

  The target object 500 is set with coordinate information that defines an arrangement position in the xyz space, as will be described later. In the virtual space 200, a grid GR is set so as to correspond to the xyz space coordinate system. The user selects the target object 500 by bringing the hand object 400 into contact with the target object 500 (may perform a grabbing action), and moves the hand object 400 in contact with the target object 500 in this state to move the target object 500. The coordinate information of the target object 500 can be changed by moving the object 500. When the user cancels the selection of the target object 500 by the hand object 400 (the operation may be performed to release the grasped hand), the target object 500 is arranged in the grid closest to the coordinates of the target object 500 at the time of the cancellation. Is done.

  In the present embodiment, the initial shape of the target object 500 is set so that the depression is not covered simply by moving the target object 500 to the depression of the passage RW. As described below, the shape of the target object 500 is set. It is necessary to move to the recess of the passage RW.

  The information processing method according to the present embodiment will be described with reference to FIGS. 10 and 11 are flowcharts for explaining the information processing method according to the present embodiment. 12 and 13 show a state in which the target object 500 selected by the right hand object 400R is deformed by the left hand object 400L. FIG. 14 illustrates a state in which the orientation of the target object 500 is changed by moving the right hand object 400R after the target object 500 is selected by the right hand object 400R. FIG. 15 shows changes in coordinate information of the target object 500 before and after the deformation of the target object 500. FIG. 16 shows how the character object CO is successfully guided to the GOAL point using the deformed target object 500.

  As shown in FIG. 10, in step S <b> 10, the visual field image presented on the HMD 110 is specified. In the present embodiment, as shown in the state (b) of FIG. 9, the target object 500 and the hand objects 400L and 400R exist in front of the virtual camera 300. Therefore, as shown in FIG. 8, the target object 500 and the hand object 400 are displayed in the visual field image M. The other objects such as the character object CO and the passage RW are not shown in FIGS. 9 and 12 to 15.

  In step S <b> 11, the control unit 121 moves the hand object 400 as described above according to the hand movement of the user U detected by the controller 320.

  In step S12, the control unit 121 determines whether or not the target object 500 and the first operation object 400 satisfy a predetermined condition. In this embodiment, based on the collision area CA set for the left hand object 400L and the right hand object 400R, it is determined whether or not each hand object 400 and the target object 500 are in contact with each other. If contacted, the process proceeds to step S13. If it is not in contact, the control waits for the user's hand movement information again and continues to move the hand object 400.

  In step S13, the control unit 121 sets the target object 500 in contact with the first operation object 400 to the selected state. In the present embodiment, as shown in the state (a) of FIG. 12, the contact between the right hand object 400R and the target object 500 is determined, and each finger of the right hand object 400R is bent by the operation as described above. The target object 500 is selected by being grasped by the right hand object 400R.

  In step S <b> 14, the control unit 121 specifies a deformation direction in which the target object 500 can be deformed according to the movement of the left hand object 400 </ b> L (second operation object). In the present embodiment, as shown in the state (a) of FIG. 12, the deformation direction TD is specified in the upward direction, the lateral direction (left direction), and the depth direction (not shown) of the target object 500. When the user U operates the right hand and selects the target object 500 with the right hand object 400R, the user U operates the left hand and deforms the target object 500 with the left hand object 400L as described later. Therefore, by limiting the deformation direction TD of the target object 500 to a dimension that is easy for the user to operate, the processing load on the computer can be reduced without impairing the user's virtual experience.

  In the present embodiment, when the target object 500 is selected, the right hand object 400R (first operation) in the target object among the upper direction and / or the lower direction, the depth direction and / or the front direction, and the horizontal direction of the target object 500 is selected. Preferably, at least one of the left directions not selected by the object) is specified as the deformation direction TD. In the state (a) of FIG. 12, as an example, the upper direction, the depth direction, and the left direction that is the direction on the side where the left hand object 400L opposite to the right hand object 400R used for selection is present is the deformation direction TD. Have been identified. In specifying the deformation direction TD, the vertical direction, the horizontal direction, and the depth direction of the target object 500 are preferably specified based on the visual field coordinate system uvw. The vertical direction of the target object 500 is the direction perpendicular to the plane (upper surface / lower surface) that intersects the v axis, which is the vertical direction of the visual field coordinate system, at the angle closest to the vertical, and is the highest in the horizontal direction of the visual coordinate system. The direction perpendicular to the plane (left plane / right plane) intersecting at an angle close to vertical is the horizontal direction (lateral direction) of the target object 500, and the plane intersects at the angle closest to the w axis, which is the depth direction of the visual field coordinate system. A direction perpendicular to (the back surface / the front surface of the hand) is defined as the depth direction (front side) of the target object 500.

  In step S <b> 15, the control unit 121 displays a direction display OD for presenting the deformation direction so as to be visible to the user in association with the target object 500 based on the specified deformation direction. In the state (a) of FIG. 12, the direction display OD extends in the deformation direction TD from the top surface, the left surface, and the back surface specified based on the visual field coordinate system uvw as described above in the target object 500. Is displayed. Thereby, the deformation | transformation rule of the target object 500 can be clearly shown to a user.

  As shown in FIG. 11, in step S16, the control unit 121 determines whether or not the target object 500 and the second operation object 400 satisfy a predetermined condition. In this embodiment, based on the collision area CA set for each hand object 400, it is determined whether or not each hand object 400 and the target object 500 are in contact. If contacted, the process proceeds to step S17. If it is not in contact, the control waits for the user's hand movement information again and continues to move the hand object 400.

  In step S <b> 17, the control unit 121 sets the target object 500 in contact with the second operation object 400 to the selected state. In the present embodiment, as shown in the state (b) of FIG. 12, the contact between the left hand object 400L and the target object 500 is determined, and each finger of the left hand object 400L is bent by the operation as described above. The target object 500 is selected by being grasped by the left hand object 400L.

  In step S18, when the target object 500 is selected by the left hand object 400L, the control unit 121 deletes the direction display OD as shown in the state (b) of FIG. When the user U selects the target object 500 with the left-hand object 400L, it is assumed that the user U has already determined a mode for deforming the target object 500. Therefore, by reducing the time for displaying the direction display without impairing the convenience of the user U, it is possible to prevent the virtual experience from being disturbed by the direction display being displayed unnecessarily in the user's field of view.

  In step S19, the control unit 121 detects whether or not the left hand object 400L is moved in any one of the deformation directions TD in a state where the target object 500 is selected. If detected, the process proceeds to step S20. If it is not detected, it continues to wait for the left hand object 400L to move.

  In step S20, the control unit 121 deforms the target object 500 in accordance with the movement of the left hand object 400L. Specifically, as shown in the state (c) of FIG. 12, the control unit 121 deforms the target object based on the movement direction MO and the movement amount MD of the left hand object 400L. Specifically, since the moving direction MO in which the left hand object 400L is moved is along the left direction of the deformation direction TD, the target object 500 is enlarged along the left direction. The enlargement amount is determined based on the movement amount MD, and the target object 500 is enlarged along the left direction so that the surface (left surface) in the enlargement direction of the target object 500 moves to the left by the movement amount MD. Is done. At this time, it is preferable that the surface (right surface) opposite to the surface in the enlarged direction is not moved. Thereby, it is possible to prevent the positional relationship between the target object 500 and the right hand object 400R from becoming unnatural due to the deformation of the target object 500.

  In step S <b> 21, the control unit 121 deforms the target object 500 and updates the coordinate information of the target object 500 to be deformed. The coordinate information is information for specifying the arrangement position of the target object in the virtual space 200 and is defined based on the space coordinate system xyz in the virtual space 200.

  In the present embodiment, as shown in FIG. 15, the coordinate information of the target object 500 is updated based on the moving direction and moving amount of the left hand object 400L. As shown in the state (a) of FIG. 15, if the target object 500 before deformation has the arrangement coordinates O1 (L, 0, 0) defined based on the center of gravity, the horizontal width of the target object 500 is 2L. As shown in the state (b) of FIG. 15, when the movement direction MO of the left hand object 400L is the left direction and the movement amount MD is 2D, the movement direction MO of the arrangement coordinates O1 is the same as the movement direction MO of the left hand object 400L. It is preferable that the movement amount of the arrangement coordinate O1 is D in the left direction. The deformed target object 500 has arrangement coordinates O2 (L + D, 0, 0) and a horizontal width of 2L + 2D. Thus, it is preferable that the movement amount of the arrangement coordinates O1 is half of the movement distance of the left hand object 400L. As a result, even if the target object 500 is deformed, the target object 500 can be enlarged without changing the position of the right end of the target object 500 selected (gripped) by the right hand object 400R. The arrangement coordinate O2 of the object 500 can be easily determined. Therefore, it is possible to prevent the positional relationship between the target object 500 and the right hand object 400R (first operation object) from becoming unnatural due to the deformation of the target object 500.

  In step S <b> 22, the control unit 121 updates the visual field image M based on the moved hand object 400 and the deformed target object 500, and outputs the visual field image M to the HMD 110. Thereby, a virtual experience that allows the user U to manipulate the target object 500 at will can be provided.

  Then, it is good also as returning to step S10 and waiting for the user's U hand movement again. Further, when the selection of the target object 500 by the right hand object 400R is continued, the process of returning to step 16 and continuously accepting the movement of the left hand object 400L and further deforming the target object 500 may be executed.

  The state (a) in FIG. 13 shows a state in which the selection of the target object 500 by the left hand object 400L is canceled after the target object 500 is enlarged in the left direction as described above. This state corresponds to the state (a) in FIG. 12, and as shown in the state (b) in FIG. 13, after the upper surface of the target object is selected by the left hand object 400L, as shown in the state (c) in FIG. By moving the left hand object 400L upward in the deformation direction, the target object 500 can be further enlarged upward. In this way, by repeating the above process, the user U can deform the target object 500 at will based on the hand object 400.

  In this embodiment, as shown in FIG. 14, even if the direction of the target object 500 is changed by moving the right hand object 400R after the deformation direction of the target object 500 is specified by the right hand object 400R, the deformation is performed. It is preferable not to change the direction TD. When the user U enlarges the target object 500 in the depth direction after selecting the target object 500 with the right hand object 400R, the user U can visually recognize the depth direction of the target object 500 in order to finely adjust the length to be enlarged. Therefore, it is expected that the direction of the target object 500 is changed. Also in this case, by not changing the deformation direction TD, the user U can enlarge the target object 5000 in the depth direction by the left hand object 400L while visually recognizing the depth direction, and a consistent operational feeling can be provided.

  As shown in FIG. 16, by repeating the above operation, the user U changes the target object 500 into a shape suitable for covering the recess of the passage RW, and then the target object 500 is moved into the recess by the hand object 400. To place. Thereby, the character object CO can be made to reach the GOAL point, and the game can be cleared.

  As mentioned above, although embodiment of this indication was described, the technical scope of this invention should not be limitedly interpreted by description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications can be made within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalents thereof.

  In the present embodiment, the movement of the hand object is controlled according to the movement of the external controller 320 indicating the movement of the user U's hand, but the hand in the virtual space is controlled according to the movement amount of the user U's hand itself. The movement of the object may be controlled. For example, instead of using an external controller, by using a glove-type device or a ring-type device worn on the user's finger, the position sensor 130 can detect the position and movement amount of the user U's hand, The movement and state of the user's U finger can be detected. Further, the position sensor 130 may be a camera configured to image the user U's hand (including fingers). In this case, by capturing the user's hand using a camera, the position and movement of the user's U hand can be determined based on the image data on which the user's hand is displayed without directly attaching any device to the user's finger. The amount can be detected, and the movement and state of the finger of the user U can be detected.

  In the present embodiment, the collision effect that defines the influence of the hand object on the target object is set according to the position and / or movement of the hand that is a part of the body other than the head of the user U. The present embodiment is not limited to this. For example, depending on the position and / or movement of a foot that is a part of the body other than the head of the user U, the influence of a foot object (an example of an operation object) linked to the movement of the user U's foot on the target object A prescribed collision effect may be set. Thus, in this embodiment, the relative relationship (distance and relative speed) between the HMD 110 and a part of the body of the user U is specified, and the user U is determined according to the specified relative relationship. A collision effect may be set that regulates the influence of an operation object linked to a part of the body on the target object.

  In the above description, the right hand object 400R is described as the first operation object and the left hand object 400L is described as the second operation object. However, when the target object 500 is selected by the left hand object 400L and the target object 500 is deformed by the right hand object 400R. Can handle the left hand object 400L as the first operation object and the right hand object 400R as the second operation object.

  In the present embodiment, the virtual space (VR space) in which the user is immersed by the HMD 110 has been described as an example, but a transmissive HMD may be adopted as the HMD 110. In this case, a virtual experience as an AR space or an MR space may be provided by synthesizing and outputting an image of the target object 500 in a real space visually recognized by the user U via the transmissive HMD 110. Then, instead of the first operation object and the second operation object, the selection of the target object 500 based on the movement of the first part of the user's body and the second part (both hands of the user U), and Deformation may be performed. In this case, the coordinate information of the real space and the first part and the second part of the user's body is specified, and the coordinate information of the target object 500 is defined in relation to the coordinate information in the real space. Thus, an action can be given to the target object 500 based on the movement of the user U's body.

1: HMD system 3: Communication network 21: Center position 112: Display unit 114: HMD sensor 120: Control device 121: Control unit 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140 : Gaze sensor 200: virtual space 300: virtual camera 302: operation buttons 302a, 302b: push buttons 302e, 302f: trigger buttons 304: detection point 320: external controller 320i: analog stick 320L: external controller for left hand (controller)
320R: External controller for right hand (controller)
322: Top surface 324: Grip 326: Frame 400: Hand object 400L: Left hand object 400R: Right hand object 500: Target object CA: Collision area CV: Field of view CVa: First area CVb: Second area

Claims (9)

An information processing method in a system comprising: a head mounted display; and a position sensor configured to detect a position of the head mounted display and positions of a first part and a second part of a body other than a user's head. And
(A) identifying virtual space data defining a virtual space including a virtual camera, a first operation object, a second operation object, and a target object;
(B) moving the virtual camera in response to movement of the head mounted display;
(C) moving the first operation object in accordance with the movement of the first part;
(D) moving the second operation object in accordance with the movement of the second portion;
(E) selecting the target object according to the movement of the first part;
(F) With the target object selected, deforming the target object according to the movement of the second operation object;
(G) defining a visual field of the virtual camera based on the movement of the virtual camera, and generating visual field image data based on the visual field and the virtual space data;
(H) displaying a field image on the head-mounted display based on the field image data;
Including an information processing method. In (e), the target object is selected by contact between the first operation object and the target object,
2. The method according to claim 1, wherein, in (f), after the second operation object contacts the target object, the target object is deformed based on a direction in which the second operation object has moved. The target object includes coordinate information in the virtual space,
The method according to claim 2, wherein the coordinate information of the target object is updated based on a direction in which the second operation object has moved.   The method according to claim 3, wherein when the second operation object moves in a predetermined direction by a predetermined distance, the coordinate information is changed so as to move by a half of the predetermined distance in the predetermined direction.   In (e), when the target object is selected, a direction display indicating a deformation direction in which the target object can be deformed according to the movement of the second operation object is displayed in association with the target object. The method in any one of Claims 1-4.   6. The method according to claim 5, wherein in (f), the direction indication is erased by detecting the movement of the second operation object. The virtual camera defines a visual field coordinate system that moves according to the movement of the head-mounted display, and the visual field coordinate system includes a vertical direction, a horizontal direction, and a depth direction,
In (e), when the target object is selected, the first operation object in the target object among the upper direction and / or the lower direction, the depth direction and / or the front direction, and the horizontal direction of the target object. The method according to claim 1, wherein at least one of the unselected directions is specified as a deformation direction capable of deforming the target object in accordance with a movement of the second operation object.   Even if the direction of the target object is changed by moving the first operation object after the deformation direction capable of deforming the target object according to the movement of the second operation object is specified, The method of claim 7, wherein the deformation direction is not changed. The program which makes the said computer perform the method in any one of Claims 1-8.