WO2021029256A1

WO2021029256A1 - Information processing device, information processing method, and program

Info

Publication number: WO2021029256A1
Application number: PCT/JP2020/029699
Authority: WO
Inventors: 貴光後藤; 西堀　一彦
Original assignee: ソニー株式会社
Priority date: 2019-08-13
Filing date: 2020-08-03
Publication date: 2021-02-18

Abstract

In order to achieve the purpose of the invention, an information processing device according to one embodiment of the present technology comprises a detection unit and a display control unit. The detection unit detects, on the basis of an image of an indication tool used by a user, the position/orientation of the indication tool. The display control unit displays, in a display space of a stereoscopic display device and on the basis of the position/orientation of the indication tool, an indication object which represents a virtual insertion portion of the indication tool with respect to the display space.

Description

Information processing equipment, information processing methods, and programs

This technology relates to information processing devices, information processing methods, and programs used for input operations.

Non-Patent Document 1 describes a pen-type input device (stylus) whose tip expands and contracts. In this stylus, a telescopic shaft is housed inside the grip. The stylus is also equipped with an acceleration sensor and a magnetic sensor for calculating its orientation. For example, when the shaft is pressed against the surface of a touch screen, a virtual stylus is drawn in the digital space inside the screen along the axis of the actual stylus. This makes it possible to perform a three-dimensional input operation on an object in the digital space.

In recent years, a technique for performing a stereoscopic display that allows an observer to perceive a stereoscopic image has been developed, and a technique for realizing an intuitive input operation for the stereoscopic image is required.

In view of the above circumstances, the purpose of this technology is to provide an information processing device, an information processing method, and a program capable of realizing an intuitive input operation for a stereoscopic image.

In order to achieve the above object, the information processing apparatus according to one embodiment of the present technology includes a detection unit and a display control unit.
The detection unit detects the position and orientation of the indicator based on the image of the indicator used by the user.
The display control unit causes the display space of the stereoscopic display device to display an instruction object representing a virtual insertion portion of the indicator with respect to the display space, based on the position and orientation of the indicator.

In this information processing device, the position and orientation of the indicator are detected from the image of the indicator used by the user. Based on this detection result, the virtual insertion portion of the indicator into the display space of the stereoscopic display device is displayed as an instruction object. As a result, it becomes possible to easily operate the instruction object in the display space by using the indicator, and it is possible to realize an intuitive input operation for the stereoscopic image.

The display control unit may calculate a virtual insertion amount of the indicator into the display space based on the position and orientation of the indicator, and control the display of the instruction object according to the insertion amount. ..

The display control unit may control the display of the instruction object so that the length of the instruction object is proportional to the insertion amount.

The stereoscopic display device may have a display panel for displaying a stereoscopic image. In this case, the display space may be a virtual space with the display panel as a boundary. Further, the detection unit may detect the position and orientation of the indicator with respect to the display panel.

The indicator may expand and contract in one direction in response to pressing. In this case, the display control unit may calculate the amount of expansion and contraction of the indicator in response to the contact between the display panel and the indicator as a virtual insertion amount of the indicator into the display space.

The display control unit may display the instruction object starting from the contact position between the display panel and the indicator in the display space.

The stereoscopic display device may have an imaging unit directed to the observation range of the display panel. In this case, the detection unit may detect the position and orientation of the indicator used by the user in the observation range based on the image of the observation range captured by the imaging unit.

The detection unit may detect the viewpoint of the user who observes the display panel based on the image of the observation range captured by the imaging unit. In this case, the display control unit may display the stereoscopic image according to the viewpoint of the user.

The indicator may have a marker portion. In this case, the detection unit may detect the position and orientation of the marker unit as the position and orientation of the indicator.

The indicator may have a tip portion directed at the target and a grip portion connected to the tip portion. In this case, the marker portion may be connected to the side opposite to the side to which the tip portion of the grip portion is connected.

The display control unit may display a target object to be an input operation using the indicator in the display space.

The display control unit may set the type of the instruction object according to the type of the target object.

The display control unit may control the display of the target object according to the contact between the target object and the instruction object.

The indicator may have a sensation presenting unit that presents a skin sensation to the user. In this case, the information processing device may further include a sensory control unit that controls the sensory presentation unit in response to contact between the target object and the instruction object.

The sensation presenting unit may present at least one of vibration sensation and thermal sensation.

The sensory control unit may control the sensory presentation unit according to at least one of the type of the target object, the type of the instruction object, or the amount of contact between the target object and the instruction object.

The information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes detecting the position and orientation of the indicator based on an image of the indicator used by the user.
Based on the position and orientation of the indicator, an instruction object representing a virtual insertion portion of the indicator with respect to the display space is displayed in the display space of the stereoscopic display device.

A program according to a form of the present technology causes a computer system to perform the following steps.
A step of detecting the position and orientation of the indicator based on the image of the indicator used by the user.
A step of displaying an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.

It is a schematic diagram for demonstrating the outline of this technique. It is a schematic diagram which shows the configuration example of a pointing device. It is a block diagram which shows the functional configuration example of a stereoscopic display device. It is a flowchart which shows an example of the information processing method which concerns on this Embodiment. It is a schematic diagram for demonstrating the arrangement relation between a display panel and a pointing device. It is a schematic diagram which shows an example of the calculation method of the position and orientation of a pointing device. It is a schematic diagram which shows the display example of a virtual pointer.

Hereinafter, embodiments relating to the present technology will be described with reference to the drawings.

[Outline of this technology]
FIG. 1 is a schematic diagram for explaining an outline of the present technology.
The present technology can realize an input operation for the stereoscopic image 10 displayed by the stereoscopic display device 100, and can provide feedback according to the input operation.
FIG. 1A is a schematic diagram showing a state before the input operation of the user 1 is performed on the stereoscopic image 10, and FIG. 1B is a schematic diagram showing a state in which the input operation of the user 1 is performed. 1A and 1B schematically show a user 1 observing a stereoscopic image 10 (rabbit object) displayed by the stereoscopic display device 100.

The three-dimensional display device 100 is a device that three-dimensionally displays content or the like to be viewed. The stereoscopic display device 100 performs stereoscopic display (stereoscopic stereoscopic vision) using binocular parallax by displaying, for example, an image for the right eye and an image for the left eye. As a result, the user 1 observing the stereoscopic display device 100 can perceive the target content as a stereoscopic image 10 having a depth.
In the following, the space in which the stereoscopic image 10 is displayed will be referred to as the display space 11 of the stereoscopic display device 100. The display space 11 is, for example, a three-dimensional virtual space that extends in the vertical direction, the horizontal direction, and the depth direction. Therefore, the stereoscopic image 10 can be said to be a virtual object that is stereoscopically displayed in the display space 11.

In the present embodiment, the stereoscopic image 10 is displayed by the display panel 20 of the stereoscopic display device 100. In FIG. 1A, a display panel 20 arranged obliquely with respect to the horizontal direction (vertical direction) is schematically illustrated as a shaded area.
The display panel 20 is a display that displays an image for the right eye and an image for the left eye on the right eye (right viewpoint) and the left eye (left viewpoint) of the user 1, respectively.
As a result, the user 1 can perceive the stereoscopic image 10 simply by looking at the display panel 20. As described above, the stereoscopic display device 100 is a naked-eye stereoscopic display device that displays the stereoscopic image 10 to the user 1 in the state of the naked eye without using a glasses-type device or the like.
Further, in the stereoscopic display device 100, the position of the viewpoint 2 of the user 1 is tracked by using a camera 21 (for example, a monocular RGB camera) arranged in the vicinity of the display panel 20, and the right according to the position of the viewpoint 2 of the user 1. An eye image and a left eye image are generated.
As a result, even when the position of the viewpoint 2 is moved, it is possible to perceive the stereoscopic image 10 (rabbit object or the like) seen from the position of the viewpoint 2 after the movement.

In the stereoscopic display device 100, for example, an image for the right eye and an image for the left eye are generated so that the stereoscopic image 10 is perceived in the space on the opposite side of the display panel 20 from the user 1. That is, the stereoscopic image 10 is displayed in the space inside the display panel 20 when viewed from the user 1. As described above, in the present embodiment, the display space 11 of the stereoscopic display device 100 is a virtual space whose boundary is the display panel 20.
FIG. 1A schematically illustrates a rabbit object (stereoscopic image 10) displayed in the display space 11 inside the display panel 20. In this embodiment, the rabbit object is an example of the target object 14. In the present disclosure, the target object 14 is an object (stereoscopic image 10) that is the target of the input operation of the user 1.

The pointing device 40 is used for the input operation on the target object 14. The pointing device 40 is a rod-shaped device configured to be expandable and contractible (see FIG. 2).
For example, the user 1 who has grasped the pointing device 40 can perform an input operation on the target object 14 by operating the tip of the pointing device 40 so as to approach the target object 14.
Further, the pointing device 40 is provided with a marker 41 for detecting the position and orientation thereof. FIG. 1A schematically illustrates the pointing device 40 used by the user 1. In this embodiment, the pointing device 40 corresponds to an indicator. The marker 41 corresponds to a marker portion.

FIG. 1B shows how the user 1 uses the pointing device 40 to perform an input operation on the rabbit object, which is the target object 14.
It is assumed that the user 1 brings the tip of the pointing device 40 close to the target object 14. At this time, the camera 21 captures a two-dimensional image of the marker 41 of the pointing device 40, and the three-dimensional position and orientation of the marker 41 are estimated based on the captured two-dimensional image. As a result, the position and orientation of the pointing device 40, that is, from which position to which direction the pointing device 40 is directed is estimated.

When the tip of the pointing device 40 comes into contact with the display panel 20 and the user 1 further pushes the pointing device 40, the pointing device 40 shrinks and its length becomes shorter.
At this time, the virtual pointer 15 is displayed in the display space 11 in the display panel 20. The virtual pointer 15 is a stereoscopic image 10 displayed in the display space 11, and is a virtual object representing a virtual insertion portion of the pointing device 40 in the display space 11. In this embodiment, the virtual pointer 15 corresponds to an instruction object.
As shown in FIG. 1B, the virtual pointer 15 corresponds to the insertion amount (expansion / contraction amount) of the pointing device 40 from the contact point between the pointing device 40 and the display panel 20 along the direction in which the pointing device 40 is directed. It is displayed so that it is the length. That is, it can be said that the virtual pointer 15 is an object in which the pointing device 40 used by the user 1 in the real space is extended to the virtual space (display space 11) and displayed.

For example, if the pointing device 40 is moved while the pointing device 40 is in contact with the display panel 20, the contact point moves and the virtual pointer 15 moves. Further, when the orientation of the pointing device 40 is changed, the orientation of the virtual pointer 15 changes along the orientation of the pointing device 40. Further, when the insertion amount (expansion / contraction amount) of the pointing device 40 is changed, the length of the virtual pointer 15 changes.
Therefore, the user 1 can intuitively operate the virtual pointer 15 in the virtual space (display space 11) by operating the pointing device 40 which is an object in the real space.

When the user 1 operates the pointing device 40 and the virtual pointer 15 comes into contact with the target object 14, for example, a process of changing the state of the target object 14 is executed. In the example shown in FIG. 1B, the rabbit object, which is the target object 14, is displayed so as to look in the direction in which the virtual pointer 15 is in contact.
As a result, it is possible to provide the user 1 with a virtual experience as if the rabbit (target object 14) in the display space 11 was reacted by using the pointing device 40. That is, it is possible to realize the interaction between the target object 14 in the display space 11 and the user 1.

Further, as will be described later, the pointing device 40 is equipped with a vibration presenting unit (see FIG. 2) that generates vibration. For example, when the virtual pointer 15 and the target object 14 come into contact with each other, vibration of a predetermined vibration pattern is presented via the vibration presenting unit.
As a result, the user 1 can perceive the sensation of contact with the target object 14 as vibration. By vibrating the pointing device 40 in this way, it is possible to feed back to the user 1 the feeling as if the object object 14 which is the stereoscopic image 10 was actually touched.

The operation of bringing the virtual pointer 15 into contact with the target object 14 is an example of an input operation for the target object 14 by the user 1. In this operation, the display of the target object 14 changes according to the contact with the virtual pointer 15, and the feeling of contact is fed back to the user 1 as vibration.
In this way, the user 1 can perform an intuitive input operation on the target object 14 displayed by the stereoscopic display device 100 by using the existing pointing device 40. In addition, since vibration is fed back according to the state of operation, it is possible to provide an experience as if the operation is being applied to a real object.

[Pointing device configuration example]
FIG. 2 is a schematic view showing a configuration example of the pointing device 40. 2A and 2B schematically show a state in which the pointing device 40 is extended and a state in which the pointing device 40 is contracted.
First, the configuration of the pointing device 40 will be described with reference to FIG. 2A.
The pointing device 40 has an expansion / contraction portion 42, a grip portion 43, and a marker 41. Further, inside the pointing device 40, a communication unit 44 and a vibration presentation unit 45 are provided.

The telescopic portion 42 has a rod-shaped structure extending in one direction as a whole, and is a portion directed to the target. One end of the telescopic portion 42 is connected to the grip portion 43, and the other end is a tip 46 directed at the target. Hereinafter, the direction in which the telescopic portion 42 extends is referred to as an axial direction. In the present embodiment, the telescopic portion 42 corresponds to the tip portion directed to the target, and the axial direction corresponds to one direction.
The telescopic portion 42 is configured to contract in the axial direction when the tip 46 is pushed. As the telescopic portion 42, for example, a telescopic structure in which a plurality of cylinders arranged in a nested manner along the axial direction expand and contract is used. In addition, the stretchable portion may be configured using any stretchable structure.
The grip portion 43 is a tubular longitudinal member connected to the telescopic portion 42, and is a portion gripped by the user 1 who uses the pointing device 40. For example, the outermost cylinder in the telescopic portion 42 of the telescopic structure is used as the grip portion 43.
The marker 41 has a plurality of detection points 47 used for detecting the position and orientation. For example, the position and orientation of the marker 41 (pointing device 40) can be detected from the positional relationship of each detection point 47 displayed in the image obtained by capturing the marker 41.
The marker 41 is typically provided with three or more detection points 47. As the detection point 47, for example, a vertex of a solid, a coloring point or a light emitting point having different colors from each other is used.
In the present embodiment, the marker 41 is connected to the side opposite to the side to which the telescopic portion 42 of the grip portion 43 is connected. That is, the marker 41 is provided at the end opposite to the tip 46 directed to the target. This makes it possible to properly detect the marker 41.
In the example shown in FIG. 2, the marker 41 on the pentagon is schematically illustrated. In this case, for example, each vertex of the pentagon becomes the detection point 47. The specific configuration of the marker 41 is not limited.

The communication unit 44 and the vibration presentation unit 45 are provided inside, for example, the grip unit 43 (or the marker 41).
The communication unit 44 is a communication module for communicating with other devices, and communicates with, for example, the stereoscopic display device 100. As the communication unit 44, for example, a wireless LAN (Local Area Network) module such as WiFi or a communication module for short-range wireless communication such as Bluetooth (registered trademark) is used.
The vibration presenting unit 45 presents the vibration sensation to the user 1. As the vibration presenting unit 45, for example, a vibration element such as a voice coil motor, a linear actuator, an eccentric motor, and a piezo element is used.
For example, when the vibration presenting unit 45 generates vibration based on a predetermined vibration signal, the vibration sensation corresponding to the vibration signal is presented to the user 1 via the grip unit 43. This makes it possible for the user 1 to perceive various tactile sensations (haptics). In the present embodiment, the vibration presenting unit 45 functions as a sensation presenting unit that presents a skin sensation to the user.
In addition to the vibration presenting unit 45, a sensation presenting unit (heat presenting unit) for presenting a thermal sensation to the user 1 may be mounted. As the heat presenting unit, for example, a heater element, a Peltier element, or the like is used. By installing a heat presenting unit, it is possible to present a hot sensation or a cold sensation.
In addition, the pointing device 40 may be equipped with an element (sensory presentation unit) capable of presenting any skin sensation such as tactile sensation, heat sensation, and force sensation.

FIG. 2B shows a state in which the pointing device 40 is pressed against the display panel 20 and contracted. For example, suppose that the user 1 presses the tip 46 against the display panel 20 while grasping the grip portion 43. In this case, the telescopic portion 42 contracts when the tip 46 is pushed by the display panel 20.
For example, in the telescopic structure, the cylinders provided on the tip 46 side of the telescopic portion 42 are pushed toward the grip portion 43 in order from the side closer to the tip 46. FIG. 2B schematically shows a cylinder pushed toward the grip portion 43 side.

Further, the telescopic portion 42 is provided with a restoring mechanism (not shown) that generates a restoring force that pushes back the tip 46 when the tip 46 is pushed. The specific configuration of the restoration mechanism is not limited, and for example, an elastic body such as a spring or rubber, a pressure cylinder, or the like is used.
For example, when the grip portion 43 is operated in a direction away from the display panel 20, the tip 46 of the telescopic portion 42 is pushed back and extended by the restoring mechanism. Therefore, while the tip 46 is pressed against the object (for example, the display panel 20), the telescopic portion 42 is contracted, but when the tip 46 is not pressed, the telescopic portion 42 is as shown in FIG. 2A. It returns to the stretched state.

In this way, the pointing device 40 can be expanded and contracted in the axial direction in response to pressing. That is, the pointing device 40 is configured such that when it comes into contact with the display panel 20, the tip portion is reduced by the expansion / contraction mechanism (expansion / contraction portion).
As will be described later, the expansion / contraction amount of the pointing device 40 is used for display control of the virtual pointer 15 as a virtual insertion amount of the pointing device 40 into the display panel 20. For example, the reduced tip is displayed as the virtual pointer 15.
As a result, an intuitive input operation as if the pointing device 40 is inserted as it is into the display space 11 in the display panel 20 becomes possible. In addition, input can be intuitively added to the target object 14 localized on the back side of the surface of the display panel 20 via the virtual pointer 15.

[Configuration example of stereoscopic display device]
FIG. 3 is a block diagram showing a functional configuration example of the three-dimensional display device 100.
The stereoscopic display device 100 includes a display panel 20, a camera 21, a speaker 22, an operation unit 23, an I / F (interface) unit 24, a communication unit 25, and a storage unit 26 described with reference to FIG. , And a controller 27.

The display panel 20 displays the stereoscopic image 10. Specifically, the right eye image and the left eye image capable of stereoscopic stereoscopic viewing are displayed on the right eye and the left eye of the user 1, respectively.
For example, the display panel 20 emits light rays forming an image for the right eye toward a certain direction of the right eye, and emits light rays forming an image for the left eye toward a certain direction of the left eye. This makes it possible to give the user 1 binocular parallax to perceive the stereoscopic image 10.
Further, the light rays for perceiving the stereoscopic image 10 (right eye image and left eye image) are emitted toward a predetermined observation range set in front of the display panel 20. Therefore, it can be said that the display panel 20 is a device that allows the user 1 who observes the display panel 20 from the observation range to perceive the stereoscopic image 10.

The display panel 20 is, for example, a rectangular panel in a plan view. In the present embodiment, the display panel 20 is arranged at an angle when viewed from the user 1 so that the pair of sides are parallel to the horizontal direction. Since the surface of the display panel 20 is arranged at an angle in this way, the user 1 can observe the stereoscopic image 10 from the horizontal direction and the vertical direction, and the observation range is expanded.
The method of the display panel 20 is not limited. For example, a parallax barrier method in which a shielding plate is provided for each set of display pixels to separate light rays incident on each eye, a lenticular lens method in which the emission direction is controlled for each display pixel, and the like are used. Further, a light ray reproduction type display or the like may be used. In addition, any method capable of stereoscopic display with the naked eye may be adopted.

The camera 21 is arranged toward the observation range of the display panel 20. In the example shown in FIG. 1, the camera 21 is arranged at the center of the upper end of the display panel 20 so as to capture the space (observation range) in front of the display panel 20 arranged diagonally. In this embodiment, the camera 21 corresponds to an imaging unit.
The camera 21 is typically a monocular RGB camera capable of capturing color moving and still images. The present technology can be applied even when a plurality of cameras 21 such as a stereo camera 21 are used.
As the camera 21, a digital camera including an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be used. In addition, any configuration may be adopted.

The speaker 22 can output various sounds. The specific configuration of the speaker 22 is not limited.
The operation unit 23 is an operation device that performs physical operations such as a keyboard and a mouse.
The I / F unit 24 is an interface to which other devices and various cables are connected, such as a USB (Universal Serial Bus) terminal and an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.
The communication unit 25 is a communication module for communicating with another device, and communicates with, for example, the pointing device 40. As the communication unit 25, for example, a wireless LAN module such as WiFi or a communication module for short-range wireless communication such as Bluetooth (registered trademark) is used.

The storage unit 26 is a non-volatile storage device, and for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like is used.
The object library 28 is stored in the storage unit 26. The object library 28 is, for example, a library in which the virtual pointer 15 and the design data of the object to be the target object 14 are stored. For example, when displaying the virtual pointer 15, the target object 14, or the like, the design data stored in the object library 28 is appropriately read out.
Further, the storage unit 26 stores a control program 29 for controlling the overall operation of the stereoscopic display device 100. The control program 29 includes a program related to the present technology.
The method of installing the control program 29 on the stereoscopic display device 100 is not limited. For example, the installation may be executed via various recording media, or the program may be installed via the Internet or the like.
The type of recording medium on which the program according to the present technology is recorded is not limited, and any computer-readable recording medium may be used. For example, any recording medium for recording data non-temporarily may be used.

The controller 27 controls the operation of each block included in the stereoscopic display device 100. The controller 27 has hardware necessary for configuring a computer, such as a processor such as a CPU or GPU and a memory such as ROM or RAM. The information processing method according to the present technology is executed by loading and executing the control program (program according to the present technology) 29 recorded in the storage unit 26 by the CPU or the like into the RAM. In this embodiment, the controller 27 corresponds to an information processing device.
The specific configuration of the controller 27 is not limited, and any hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit) may be used.

In the present embodiment, the CPU of the controller 27 or the like executes the program according to the present embodiment to perform the image acquisition unit 30, the viewpoint detection unit 31, the pointer detection unit 32, the display control unit 33, and the vibration control as functional blocks. Part 34 is realized. Then, the information processing method according to the present embodiment is executed by these functional blocks.
In addition, in order to realize each functional block, dedicated hardware such as an IC (integrated circuit) may be appropriately used.
In the present embodiment, the viewpoint detection unit 31 and the pointer detection unit 32 function as detection units.

The image acquisition unit 30 acquires an image of the observation range captured by the camera 21 (hereinafter, simply referred to as an captured image). For example, the camera 21 reads the captured images generated at a predetermined frame rate. The captured image acquired by the image acquisition unit 30 is output to the viewpoint detection unit 31 and the pointer detection unit 32, which will be described later, respectively.

The viewpoint detection unit 31 detects the viewpoint 2 of the user 1 who observes the display panel 20 based on the captured image captured by the camera 21. More specifically, the viewpoint detection unit 31 detects the position of the viewpoint 2 of the user 1.
For example, the face recognition of the user 1 observing the display panel 20 is executed, and the three-dimensional coordinates of the viewpoint position of the user 1 in the observation range are calculated. The method for detecting the viewpoint 2 is not limited, and for example, a viewpoint estimation process using machine learning or the like, or a viewpoint detection using pattern matching or the like may be executed.

The pointer detection unit 32 detects the position and orientation of the pointing device 40 based on the image of the pointing device 40 used by the user 1.
As described above, since the pointing device 40 is provided with the marker 41 for detecting the position and orientation, it is possible to easily detect the position and orientation of the pointing device 40 from the two-dimensional image.
As the position of the pointing device 40, a three-dimensional position in a predetermined coordinate system is detected. Further, as the posture of the pointing device 40, an angle parameter representing the direction of the pointing device 40 (axial direction) in a predetermined coordinate system is detected. The method of detecting the position and orientation of the pointing device 40 (marker 41) will be described in detail later.

In the present embodiment, the observation image read by the image acquisition unit 30 is used as the image of the pointing device 40. That is, the pointer detection unit 32 detects the position and orientation of the pointing device 40 used by the user 1 in the observation range based on the captured image captured by the camera 21.
That is, it can be said that the stereoscopic display device 100 detects the position and orientation of the pointing device 40 used by the user 1 by using the camera 21 that tracks the viewpoint 2 of the user 1. As a result, it is possible to detect the position and orientation of the pointing device 40 without adding a special sensor or the like, and it is possible to reduce the cost of the device or the like.
The case is not limited to the case where the camera 21 for viewpoint detection is used, and for example, an image obtained by the pointing device 40 by another imaging device may be used. Further, both the camera 21 for viewpoint detection and the image captured by another imaging device may be used. As a result, it is possible to improve the detection accuracy of the position and orientation and expand the detection range.

The display control unit 33 displays the stereoscopic image 10 according to the viewpoint 2 of the user 1. As described with reference to FIG. 1, the object displayed as the stereoscopic image 10 includes the target object 14 and the virtual pointer 15.
For example, the display control unit 33 calculates the direction in which the object is observed based on the position of the viewpoint 2 of the user 1 detected by the viewpoint detection unit 31. Then, a parallax image (right eye image and left eye image) for displaying the object viewed from the calculated direction as a stereoscopic image 10 is generated, and the image data of the parallax image is output to the display panel 20.
As a result, even when the viewpoint 2 of the user 1 changes, it is possible to perceive the stereoscopic image 10 of the object viewed from that direction.

Specifically, the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40. For example, when the content for displaying the rabbit object (see FIG. 1) is executed, a parallax image for displaying the rabbit object as a stereoscopic image 10 is generated and output to the display panel 20.

The display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display a virtual pointer 15 representing a virtual insertion portion of the pointing device 40 with respect to the display space 11 based on the position and orientation of the pointing device 40.
More specifically, display parameters such as the position, posture, and length of the virtual pointer 15 are calculated based on the position and orientation of the pointing device 40 detected by the pointer detection unit 32. Based on this display parameter, a parallax image for displaying the virtual pointer 15 viewed from the viewpoint 2 of the user 1 as a stereoscopic image 10 is generated and output to the display panel 20.
The target object 14 and the virtual pointer 15 are generated by using the design data stored in the object library 28.

The vibration control unit 34 controls the vibration presentation unit 45 provided in the pointing device 40 in response to the contact between the target object 14 and the virtual pointer 15.
For example, when the target object 14 and the virtual pointer 15 come into contact with each other, the vibration control unit 34 generates a predetermined vibration signal and outputs it to the vibration presenting unit 45 via the

communication units

25 and 44. As a result, at the timing when the virtual pointer 15 comes into contact with the target object 14, the user 1 holding the pointing device 40 is presented with a predetermined vibration sensation.
If a heat presentation unit or the like is provided in addition to the vibration presentation unit 45, a heat control unit or the like that controls the heat presentation unit may be configured. In this case, the heat control unit generates a control signal for controlling the temperature of the heat presentation unit and the like in response to the contact between the target object 14 and the virtual pointer 15. This makes it possible to present the user 1 with a predetermined thermal sensation.
In the present embodiment, the vibration control unit 34 (heat presentation unit) corresponds to the sensory control unit.

[Virtual pointer display processing]
FIG. 4 is a flowchart showing an example of the information processing method according to the present embodiment. FIG. 5 is a schematic diagram for explaining the arrangement relationship between the display panel 20 and the pointing device 40. Hereinafter, the display processing of the virtual pointer 15 will be described with reference to FIGS. 4 and 5.

In the present embodiment, the display control unit 33 calculates the virtual insertion amount of the pointing device 40 into the display space 11 based on the position and orientation of the pointing device 40. Then, the display of the virtual pointer 15 is controlled according to the calculated insertion amount.
The virtual insertion amount is the length at which the user 1 pushes the pointing device 40 into the display panel 20, and is the amount of expansion and contraction of the pointing device 40, as described with reference to FIG. 1B. The larger the expansion / contraction amount, the longer the length of the virtual pointer 15.
For example, the user 1 operates the grip portion 43 so that the tip 46 of the pointing device 40 is brought closer to the target object 14. At this time, a virtual pointer 15 that moves in conjunction with the pointing device 40 is displayed inside the display panel 20. Therefore, the user 1 can bring the virtual pointer 15 closer to or in contact with a desired point on the target object 14 by adjusting the position and orientation of the pointing device 40 and the virtual pointer 15 which is an extension thereof. ..
As described above, in the stereoscopic display device 100, the target object 14 displayed in the display space 11 is displayed by the visual information of the real or virtual displayed pointers (pointing device 40 and virtual pointer 15) according to the state of the pointer. An input operation that gives input is realized. Hereinafter, a specific description will be given.

FIG. 5A is a perspective view showing a user 1 who performs an input operation on the stereoscopic display device 100 (display panel 20). FIG. 5B is a top view of the display panel 20 when viewed from the front. FIG. 5C is a side view of the display panel 20 when viewed from the side.
In the following, the horizontal and vertical directions of the image (captured image) in the observation range captured by the camera 21 are defined as the x direction and the y direction, respectively. Further, the coordinate system set based on the position of the camera 21 is referred to as a camera coordinate system. The camera coordinate system is, for example, a coordinate system represented by two-dimensional coordinates (x, y) and the z direction orthogonal to the two-dimensional coordinates, and the position of the camera 21 is set as the origin.
Further, the horizontal direction (horizontal direction) of the display panel 20 is the X direction, and the vertical direction of the display panel 20 is the Y direction. Further, the direction orthogonal to the surface (XY plane) of the display panel 20 is defined as the Z direction. Further, the coordinate system represented by the X direction, the Y direction, and the Z direction is described as a display coordinate system (X, Y, Z).
Further, as shown in FIG. 5A, the rotation angle with the X direction as the rotation axis is defined as θ, the rotation angle with the Y direction as the rotation axis is φ, and the rotation angle with the Z direction as the rotation axis is described as ω. Therefore, the position and orientation in the display coordinate system are represented by (X, Y, Z, θ, φ, ω).
In the following, the display coordinate system will be used as the world coordinates in the stereoscopic display device 100. For example, when displaying the stereoscopic image 10 (target object 14 or virtual pointer 15), it is assumed that the position and orientation in the display coordinate system are calculated.
Further, the position of the marker 41 (pointing device 40) in the display coordinate system is described as M (X _M , Y _M , Z _M ), and the axial direction of the pointing device 40 and the surface (XY surface) of the display panel 20 are defined. The intersection is described as D (X _D , Y _D , 0), and the contact point between the virtual pointer 15 and the target object 14 is described as O (X _obj , Y _obj , Z _obj ).
When the pointing device 40 is in contact with the display panel 20, the intersection D is the contact position between the display panel 20 and the pointing device 40. 5A to 5C show the marker position M, the intersection D, and the contact point O, respectively.

As shown in FIG. 4, the pointer detection unit 32 first detects the marker 41 from the captured image captured by the camera 21 (step 101). Image recognition processing using, for example, feature point detection, template matching, or the like is used to detect the marker 41. The method for detecting the marker 41 is not limited, and the marker 41 may be detected by using, for example, machine learning.
The pointer detection unit 32 determines whether or not the marker 41 has been detected (step 102). If the marker 41 is not detected (No in step 102), step 101 is executed again, and the detection process of the marker 41 is executed for the next image. When the marker 41 is detected (Yes in step 102), the pointer detection unit 32 executes a process of estimating the position and orientation of the marker 41 (step 103).
In the present embodiment, the position and orientation of the marker 41 is used as the position and orientation of the pointing device 40. That is, the pointer detection unit 32 detects the position / orientation of the marker 41 as the position / orientation of the pointing device 40. By using the marker 41, it is possible to estimate the position and orientation of the pointing device 40 with high accuracy. The position and orientation of the pointing device 40 may be calculated by appropriately converting the position and orientation of the marker 41.

For example, the two-dimensional coordinates (x _i , y _i ) of each detection point 47 of the marker 41 in the captured image are calculated. Here, i is an index representing each detection point 47. These two-dimensional coordinates are coordinates in which the three-dimensional coordinates of the detection point 47 in the real space are projected (transparent projection conversion) in the captured image by being imaged by the camera 21.
Here, it is assumed that the internal parameters and the external parameters of the camera 21 have been calibrated in advance. The internal parameter is a parameter that changes according to zoom and the like, and includes, for example, a parameter that represents the principal point (typically the center of the image) of the captured image and a parameter that represents the focal length corresponding to each pixel. The external parameter is a parameter that changes according to the position and orientation of the camera 21, and includes, for example, a matrix element of a simultaneous transformation matrix of translation and rotation in fluoroscopic projection transformation. These parameter values are pre-calibrated with reference to, for example, the display coordinate system and stored in the storage unit 26.

In the estimation of the position and orientation of the marker 41, for example, the translation and rotation vectors of the marker 41 appearing in the two-dimensional image (captured image) are derived by using the solution of the PNP problem. For example, using the transformation matrix described in internal parameters and external parameters of the camera 21, the two-dimensional coordinates (x _i, y _i) of each detection point 47 when converted into three-dimensional coordinates, in the actual marker 41 The translation vector and the rotation vector of the marker 41 are calculated so as to satisfy the three-dimensional arrangement relationship of each detection point 47.
In the present embodiment, first, vectors (x _c , y _c , z _c , θ _c , φ _c , ω _c ) representing the position and orientation of the marker 41 in the camera coordinate system are calculated.

Next, the pointer detection unit 32 converts the position and orientation of the marker 41 in the camera coordinate system calculated in step 103 into the position and orientation of the display coordinate system (step 104). For the transformation of the coordinate system, for example, a transformation matrix set according to the arrangement relationship between the camera 21 and the display panel 20 is used. The vector representing the position and orientation of the marker 41 is converted into a vector (X _M , Y _M , Z _M , θ _M , φ _M , ω _M ) in the display coordinate system by the transformation matrix.
The vector representing the position and orientation of the marker 41 in the display coordinate system is the position and orientation of the pointing device 40. In this way, the pointer detection unit 32 detects the position and orientation of the pointing device 40 in the display coordinate system, that is, the position and orientation of the pointing device 40 with respect to the display panel 20.

FIG. 6 is a schematic view showing an example of a method of calculating the position and orientation of the pointing device 40. FIG. 6 shows the position M of the marker 41 (pointing device 40) in the display coordinate system (X, Y, Z) with the intersection D between the axial direction of the pointing device 40 and the display panel 20 as the origin. ..

First, the pointer detection unit 32 calculates the intersection D between the pointing device 40 and the display panel 20 (step 105).
The position M (X _M , Y _M , Z _M ) of the marker 41 and the rotation angle Ω (θ _M , φ _M , ω _M ) of the marker 41 in the display coordinate system are calculated in step 104 and are known values. is there. Here, from these positions M and the rotation angle Ω (position and orientation of the marker 41), the intersection D (X _D , Y _D , 0) between the axial direction of the pointing device 40 and the surface (Z = 0) of the display panel 20. The method of calculating is described.

First, the X and Y coordinates of the intersection D are set as X _D and Y _D , and the Z coordinate (Z _M ) of the marker 41 is represented by using the position M and the rotation angle Ω of the marker 41. The Z coordinate of the marker 41 is represented by the following equation using, for example, the length (X _M − X _D ) in the X direction from the intersection D to the position M of the marker 41.

Similarly, the Z coordinate of the marker 41 is expressed by the following equation using, for example, the length in the Y direction (Y _M − Y _D ) from the intersection D to the position M of the marker 41.

As described above, the Z coordinate (Z _M ) of the marker 41 in the display coordinate system is described by two methods as the height seen from the XY plane.

By squared both sides of equation (1), the following equation is obtained for Z _M.

By modifying the equation (3) with respect to the length (X _M − X _D ) in the X direction from the intersection D to the position M of the marker 41, the following equation is obtained.

The left side of the equation (4) includes the X coordinate (X _D ) of the intersection D to be calculated. Therefore, by modifying equation (4) for X _D , the following equation is obtained.

Similarly, when the Y coordinate (Y _D ) of the intersection D, which is the calculation target, is transformed from the equation (2), Y _D is expressed by the following equation.

As shown in Eqs. (5) and (6), the X coordinate (X _D ) and Y coordinate (Y _D ) of the intersection D are the positions M (X _M , Y _M , Z) of the marker M that have already been calculated. _M), and the rotation angle of the marker 41 (θ _M, can be expressed using phi _M). That is, the intersection D between the axial direction of the pointing device 40 and the surface of the display panel 20 is represented as follows.

The pointer detection unit 32 calculates the X coordinate and the Y coordinate of the intersection D in the display coordinate system according to the equation (7). The intersection D may be calculated by using another method without being limited to the above method.

When the intersection D is calculated, the display control unit 33 calculates the length of the pointing device 40 (step 106). The length of the pointing device 40 is, for example, the length from the position of the tip 46 to the position M of the marker 41. The position of the tip 46 changes due to expansion and contraction. In the following, the length from the tip 46 to the marker 41 in the non-contact state in which the pointing device 40 is not shrunk is described as the reference length d.
For example, when the pointing device 40 is not in contact with the display panel 20, the length of the pointing device 40 is the reference length d. On the other hand, when the pointing device 40 is in contact with the display panel 20, the telescopic portion 42 expands and contracts, so that the length of the pointing device 40 is equal to or less than the reference length d.
In the present embodiment, the length of the current pointing device 40 is calculated based on the distance MD from the intersection D to the marker position M. The distance MD is expressed by the following equation using the coordinates of the intersection D calculated by the equation (7) and the position M of the marker 41.

When the distance MD is longer than the reference length d, the length of the pointing device 40 becomes the reference length. When the distance MD is equal to or less than the reference length d, the length of the pointing device 40 is the distance MD.

Based on the distance MD calculated by the equation (8), it is determined whether or not the pointing device 40 and the display panel 20 are in contact with each other (step 107). Specifically, using the reference length d as a threshold value, it is determined whether or not the distance MD is equal to or less than the reference length d. This is a determination as to whether or not the current length of the pointing device 40 is shorter than the length of the pointing device 40 in the non-contact state (the original length of the pointing device 40).
When the distance MD is longer than the reference length d (MD> d), it is determined that the pointing device 40 is not in contact with the display panel 20 (No in step 107). In this case, the processes after step 101 are executed for the next image.
When the distance MD is equal to or less than the reference length d (MD ≦ d), it is determined that the pointing device 40 is in contact with the display panel 20 (Yes in step 107). In this case, the process of drawing the virtual pointer 15 is executed according to the length (distance MD) of the pointing device 40 (step 108).

FIG. 7 is a schematic diagram showing a display example of the virtual pointer 15. On the left and right sides of FIG. 7, a pointing device 40 in a non-contact state and a pointing device 40 that has shrunk in contact with the display panel 20 are schematically illustrated.

First, the display control unit 33 calculates a virtual insertion amount into the display panel 20 (display space 11) of the pointing device 40, which is the length of the virtual pointer 15. In the present embodiment, the expansion / contraction amount Δ of the pointing device 40 in response to the contact between the display panel 20 and the pointing device 40 is calculated as a virtual insertion amount of the pointing device 40 into the display space 11.
As shown on the right side of FIG. 7, the pointing device 40 shrinks by the amount that the tip 46 is pressed against the display panel 20. The expansion / contraction amount Δ at this time is calculated as a virtual insertion amount of the pointing device 40 into the display panel 20 (display space 11).
The expansion / contraction amount Δ is a value (Δ = d−MD) obtained by subtracting the distance MD, which is the length of the current pointing device 40, from the reference length d.
The display control unit 33 generates a virtual pointer 15 based on the expansion / contraction amount Δ, the posture of the pointing device 40, and the contact position (intersection point D) with the display panel 20. Hereinafter, the position, orientation, and length of the virtual pointer 15 will be described.
The virtual pointer 15 is generated based on, for example, the design data stored in the object library 28.

The display control unit 33 displays the virtual pointer 15 starting from the contact position (intersection D) between the display panel 20 and the pointing device 40 in the display space 11. For example, as shown on the right side of FIGS. 1B and 7, the virtual pointer 15 is displayed so as to extend from the intersection D where the pointing device 40 contacts the display panel 20 to the display space 11.

Further, the display control unit 33 displays the virtual pointer 15 along the axial direction of the pointing device 40. That is, the virtual pointer 15 that extends and displays the pointing device 40 in the direction in which the pointing device 40 is directed is displayed in the display space 11.
As a result, it is possible to realize a display as if the pointing device 40 is inserted into a virtual display space, and an intuitive input operation is possible.

In the present embodiment, the display of the virtual pointer 15 is controlled so that the length of the virtual pointer 15 is proportional to the expansion / contraction amount Δ. For example, as shown on the right side of FIG. 7, a virtual pointer 15 having the same length as the expansion / contraction amount Δ is displayed in the display space 11. In this case, the proportional coefficient between the length of the virtual pointer 15 and the expansion / contraction amount Δ is 1, and the offset or the like is not set.
Further, for example, a virtual pointer 15 having a length obtained by adding a predetermined offset to the expansion / contraction amount Δ may be displayed. In this case, the virtual pointer 15 is an object longer (or shorter) than the expansion / contraction amount Δ. Even when an offset is added, the length of the virtual pointer 15 is set in proportion to the expansion / contraction amount Δ.
The proportional coefficient between the length of the virtual pointer 15 and the expansion / contraction amount Δ may be appropriately set. For example, a proportional coefficient of 1 or more may be set. As a result, it is possible to amplify the operation amount (expansion / contraction amount Δ) of the user 1, and for example, it is possible to perform an input operation on an object or the like arranged at a depth exceeding the maximum value of the expansion / contraction amount Δ of the pointing device 40. Is.
Further, for example, a proportional coefficient smaller than 1 may be set. As a result, the operation amount of the user 1 can be reduced, and for example, fine input operations can be easily performed. In any case, by appropriately setting the proportional coefficient and the offset, it is possible to display the virtual pointer 15 that is linearly linked to the operation of the user 1, and it is possible to perform an intuitive input operation.
In this way, when the length of the pointing device 40 is shorter than the original length (reference length d), the display control unit 33 draws the virtual pointer 15 according to the difference (expansion / contraction amount Δ).

When the virtual pointer 15 is drawn, it is determined whether or not the virtual pointer 15 and the target object 14 are in contact with each other (step 109). For example, whether or not a part (tip, side surface, etc.) of the virtual pointer 15 is in contact with the target object 14 is determined based on the coordinate values of the display coordinates.
When it is determined that the virtual pointer 15 and the target object 14 are not in contact with each other (No in step 109), step 108 is executed and the drawing of the virtual pointer 15 is continued. In addition, instead of step 108, the processing after step 101 may be executed again.
When it is determined that the virtual pointer 15 and the target object 14 are in contact with each other (Yes in step 109), the display control unit 33 executes a process of changing the drawing of the target object 14 (step 110).

It can be said that the process of changing the drawing of the target object 14 is a process of displaying the reaction (interaction) of the operation of the virtual pointer 15 with respect to the target object 14. This process is executed according to, for example, the contact position and the contact amount with the virtual pointer 15.
Here, the contact amount is, for example, an amount representing the degree to which the virtual pointer 15 interferes with the target object 14, and is expressed using, for example, the amount of penetration of the virtual pointer 15 into the coordinate space inside the target object 14.
The type of interaction with the target object 14 is not limited. For example, there is a process of deforming the target object 14 itself, such as denting the contact position with which the virtual pointer 15 is in contact according to the contact amount of the virtual pointer 15.
Further, for example, a process of changing the position of the target object 14, such as moving the target object 14 or changing the direction of the target object 14, may be executed in response to the contact of the virtual pointer 15. In this case, the movement amount and the rotation amount of the target object 14 are appropriately calculated according to the contact position, the contact amount, and the like.
Further, a process of drawing the target object 14 so as to make a predetermined reaction according to the contact may be executed. For example, the process of looking back at the rabbit object described with reference to FIG. 1 is a process of drawing a reaction in response to contact with the virtual pointer 15.
As described above, in the present embodiment, the display control unit 33 controls the display of the target object 14 according to the contact between the target object 14 and the virtual pointer 15. This makes it possible to present various interactions according to input operations (touching, poking, hitting, cutting, pushing, etc.) with respect to the target object 14.

When the drawing of the target object 14 is changed, the vibration control unit 34 executes a feedback process for presenting vibration to the user 1 (step 111).
For example, the vibration control unit 34 generates a vibration signal according to the content and the like, and outputs the vibration signal to the vibration presenting unit 45 mounted on the pointing device 40. As a result, the pointing device 40 generates vibration in response to the vibration signal, and the user 1 is presented with a tactile sensation.

In the present embodiment, the vibration control unit 34 controls the vibration presentation unit 45 according to at least one of the type of the target object 14, the type of the virtual pointer 15, or the contact amount between the target object 14 and the virtual pointer 15. ..
For example, vibration signals that present various vibration sensations are generated according to the types of the target object 14 and the virtual pointer 15. Further, the intensity of these vibration sensations is adjusted according to the contact amount. In addition, the method of controlling the vibration presenting unit 45, the method of generating a vibration signal, and the like are not limited.
As described above, in the present embodiment, the skin sensation is fed back to the user 1 through the pointing device 40. As a result, it is possible to present a feeling of being in contact with an existing object, and it is possible to improve the operability of the input operation.

[Application example]
Hereinafter, an application example of an input operation in the stereoscopic display device 100 using the pointing device 40 will be described.
In FIGS. 1B and 7 described above, a rod-shaped virtual pointer 15 that is simply an extension of the pointing device 40 has been described as an example. The shape of the virtual pointer 15 and the like are not limited, and any design can be applied according to the use case.
For example, a knife-shaped object may be used as the virtual pointer 15. In this case, it is possible to perform an operation of disconnecting the target object 14 using the virtual pointer 15.
Further, an object having a pointed tip may be used. In this case, it is possible to perform an operation of stabbing the target object 14.
Further, an object having a tip shaped like a brush or a brush may be used. In this case, it is possible to paint the target object 14 or to draw a picture or a character.
Further, an object having a tip in the shape of a tool such as a screwdriver or a spanner may be used. In this case, the operation of assembling or disassembling the target object 14 becomes possible.
Further, by using a plurality of pointing devices at the same time, it may be possible to perform an operation such as pinching the target object 14 with chopsticks. Also, by using a plurality of devices, it is possible to use a plurality of tools at the same time in the display space.
Further, for example, the type of the virtual pointer 15 may be set according to the type of the target object 14. For example, when the target object 14 in the shape of food is displayed, the virtual pointer 15 in the shape of a knife, fork, or the like is automatically displayed.
Alternatively, it is also possible to display the object candidates of the plurality of virtual pointers 15 on the stereoscopic display device 100 so as to be selectable, and the user 1 can select the objects.

As an example of the application of the input operation using the pointing device 40, medical training can be mentioned. For example, the pointing device 40 is used to input to the human body or the affected part (target object 14) displayed in three dimensions. At this time, the virtual pointer 15 displays an object of a medical instrument such as a scalpel or forceps. Further, when the user 1 performs an operation such as cutting the affected portion, a predetermined vibration is presented via the device. This makes it possible to simulate surgery.
Other examples of use include architecture, model design, and work. In this case, a model, a part (target object 14), or the like is three-dimensionally displayed, and a virtual pointer 15 that imitates a tool or the like for precision machining is used. In the present embodiment, since the pointing device 40 can be used for intuitive operation, it is possible to perform a complicated input operation on the object to be processed or assembled.

The input interface using the pointing device 40 and the virtual pointer 15 can also be used as a communication tool.
For example, it is possible to perform interactive communication with a person in a remote place by using a plurality of stereoscopic display devices 100. In this case, for example, it is possible to intuitively operate each other on a shared object displayed in three dimensions.
In addition, a target object 14 (virtual pet) that becomes a pet such as a dog, a cat, or a rabbit is three-dimensionally displayed. As a result, the user 1 can come into contact with the virtual pet virtually displayed through the pointing device 40 and experience the breeding of the pet.
The scope of application of this technology is not limited to the above-mentioned examples, and can be applied to various scenes such as amusement, education, and product development.

As described above, in the information processing apparatus according to the present embodiment, the position and orientation of the pointing device 40 are detected from the image of the pointing device 40 used by the user 1. Based on this detection result, the virtual insertion portion of the pointing device 40 into the display space 11 of the stereoscopic display device 100 is displayed as a virtual pointer 15. As a result, the virtual pointer 15 in the display space 11 can be easily operated by using the pointing device 40, and an intuitive input operation for the stereoscopic image 10 can be realized.

In the present embodiment, by operating the pointing device 40 so as to bring it into direct contact with the target object 14, it is possible to perform an input operation on the target object 14 via the virtual pointer 15. In this way, it is possible to operate a virtual object as it is using a real device, and it is possible to demonstrate high usability.
When the virtual pointer 15 comes into contact with the target object 14, sensory feedback is performed by the vibration presenting unit 45 or the like. This makes it possible to provide a virtual experience as if operating a real object.

For example, as a method of performing an input operation in the depth direction, there is a method of detecting the position and orientation of an input device using a sensor. For example, a method of calculating the position and orientation of the input device by using a touch sensor mounted on the display, a posture sensor mounted on the input device, or the like can be considered. In such a method, it is necessary to use a special sensor, which may lead to an increase in equipment cost.

In the present embodiment, the image of the pointing device 40 is captured by using the camera 21 provided in the stereoscopic display device 100. From this image, the position and orientation of the pointing device 40 are calculated, and the display control of the virtual pointer 15 and the like is performed. As described above, in the present embodiment, the display processing of the virtual pointer 15 can be performed without providing a touch sensor, a position / orientation sensor, or the like. As a result, the cost of the device can be sufficiently suppressed.
In particular, in a type of device such as the stereoscopic display device 100 described above that displays a stereoscopic image according to the viewpoint 2 of the user 1, a camera 21 for tracking the viewpoint 2 is often provided. By using such a camera 21, it is possible to easily implement the display processing of the virtual pointer 15.

<Other Embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be realized.

In the above, the virtual pointer is displayed so that its length is proportional to the amount of expansion and contraction of the pointing device (the amount of virtual insertion into the display panel). Not limited to this, for example, the virtual pointer may be controlled non-linearly with respect to the amount of expansion and contraction.
For example, control may be performed such that the larger the expansion / contraction amount, that is, the larger the amount of pushing the pointing device, the larger the rate of increasing the length of the virtual pointer. As a result, for example, it is possible to bring the virtual pointer into contact with an object localized at a deep position without pushing the pointing device to the end, and it is possible to improve the operability of the user.
Further, the virtual pointer may be displayed along a direction different from the axial direction of the pointing device. This makes it possible to perform complicated input operations in a comfortable posture.

In the above embodiment, a stereoscopic display device that performs stereoscopic display with the naked eye has been described. Not limited to this, a glasses-type stereoscopic display device that displays a stereoscopic image using a glasses-type device such as polarized glasses or liquid crystal shutter glasses may be used.
In the glasses-type stereoscopic display device, the user wears a glasses-type device for perceiving a parallax image and observes the display panel. Even in this case, it is possible to display the virtual pointer according to the amount of expansion and contraction of the pointing device.

The present technology can also be applied when stereoscopic display is performed using a head-mounted display (HMD) or the like. In this case, the pointing device does not need to expand or contract.
For example, in an HMD capable of AR display, it is possible to display a stereoscopic image (right eye image and left eye image) by superimposing it on the field of view in front of the user. In this case, the range of the field of view in which the stereoscopic image is displayed is the display space of the HMD. The image of the pointing device is captured by using a front camera or the like mounted on the HMD.
Suppose the user brings the tip of the pointing device closer to the target object. In this case, the virtual pointer is displayed by superimposing it on the pointing device that has entered the user's field of view (display space). This enables intuitive input operations via the virtual pointer. Further, for example, when a boundary surface is set in the display space and the pointing device exceeds the boundary surface, a virtual pointer having a length corresponding to the insertion amount is displayed. For example, such a process may be executed.
In addition, the type of device that performs stereoscopic display is not limited.

In the present disclosure, "center", "center", "uniform", "equal", "same", "orthogonal", "parallel", "symmetrical", "extended", "axial", "cylindrical", "cylindrical", "ring", and "circle". Concepts that define shape, size, positional relationship, state, etc., such as "ring shape," are "substantially centered," "substantially centered," "substantially uniform," "substantially equal," and "substantially the same." "Substantially orthogonal" "substantially parallel" "substantially symmetric" "substantially extending" "substantially axial" "substantially cylindrical" "substantially cylindrical" "substantially cylindrical" The concept includes a ring shape, a substantially ring shape, and the like.

For example, "perfectly centered", "perfectly centered", "perfectly uniform", "perfectly equal", "perfectly identical", "perfectly orthogonal", "perfectly parallel", "perfectly symmetric", "perfectly extending", "perfectly extending" Includes states included in a predetermined range (for example, ± 10% range) based on "axial direction", "completely cylindrical shape", "completely cylindrical shape", "completely ring shape", "completely annular shape", etc. Is done.

It is also possible to combine at least two feature parts among the feature parts related to the present technology described above. That is, the various feature portions described in each embodiment may be arbitrarily combined without distinction between the respective embodiments. Further, the various effects described above are merely examples and are not limited, and other effects may be exhibited.

In addition, this technology can also adopt the following configurations.
(1) A detection unit that detects the position and orientation of the indicator based on the image of the indicator used by the user, and
An information processing device including a display control unit that displays an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.
(2) The information processing device according to (1).
The display control unit calculates a virtual insertion amount of the indicator into the display space based on the position and orientation of the indicator, and controls the display of the instruction object according to the insertion amount. ..
(3) The information processing device according to (2).
The display control unit is an information processing device that controls the display of the instruction object so that the length of the instruction object is proportional to the insertion amount.
(4) The information processing device according to any one of (1) to (3).
The stereoscopic display device has a display panel for displaying a stereoscopic image.
The display space is a virtual space with the display panel as a boundary.
The detection unit is an information processing device that detects the position and orientation of the indicator with respect to the display panel.
(5) The information processing device according to (4).
The indicator can be expanded and contracted in one direction in response to pressure.
The display control unit is an information processing device that calculates the amount of expansion and contraction of the indicator in response to contact between the display panel and the indicator as a virtual insertion amount of the indicator into the display space.
(6) The information processing device according to (4) or (5).
The display control unit is an information processing device that displays the instruction object starting from the contact position between the display panel and the indicator in the display space.
(7) The information processing apparatus according to any one of (4) to (6).
The stereoscopic display device has an image pickup unit directed to an observation range of the display panel.
The detection unit is an information processing device that detects the position and orientation of the indicator used by the user in the observation range based on the image of the observation range captured by the imaging unit.
(8) The information processing apparatus according to (7).
The detection unit detects the viewpoint of the user who observes the display panel based on the image of the observation range captured by the imaging unit.
The display control unit is an information processing device that displays the stereoscopic image according to the viewpoint of the user.
(9) The information processing apparatus according to any one of (1) to (8).
The indicator has a marker portion and has a marker portion.
The detection unit is an information processing device that detects the position and orientation of the marker unit as the position and orientation of the indicator.
(10) The information processing apparatus according to (9).
The indicator has a tip that is directed at the object and a grip that is connected to the tip.
The marker portion is an information processing device connected to a side opposite to the side to which the tip portion of the grip portion is connected.
(11) The information processing apparatus according to any one of (1) to (10).
The display control unit is an information processing device that displays a target object that is a target of an input operation using the indicator in the display space.
(12) The information processing apparatus according to (11).
The display control unit is an information processing device that sets the type of the instruction object according to the type of the target object.
(13) The information processing device according to (11) or (12).
The display control unit is an information processing device that controls the display of the target object in response to contact between the target object and the instruction object.
(14) The information processing apparatus according to any one of (11) to (13).
The indicator has a sensation presenting portion that presents a skin sensation to the user.
Further, an information processing device including a sensory control unit that controls the sensory presentation unit in response to contact between the target object and the instruction object.
(15) The information processing apparatus according to (14).
The sensation presenting unit is an information processing device that presents at least one of a vibration sensation and a heat sensation.
(16) The information processing apparatus according to (14) or (15).
The sensory control unit is an information processing device that controls the sensory presentation unit according to at least one of the type of the target object, the type of the instruction object, or the amount of contact between the target object and the instruction object.
(17) The position and orientation of the indicator are detected based on the image of the indicator used by the user.
An information processing method in which a computer system executes an instruction object representing a virtual insertion portion of the indicator to the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.
(18) A step of detecting the position and orientation of the indicator based on an image of the indicator used by the user, and
A program that causes a computer system to perform a step of displaying an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.

1 ... User 2 ... Viewpoint 10 ... Stereoscopic image 11 ... Display space 14 ... Target object 15 ... Virtual pointer 20 ... Display panel 21 ... Camera 27 ... Controller 31 ... Viewpoint detection unit 32 ... Pointer detection unit 33 ... Display control unit 34 ... Vibration Control unit 40 ... Pointing device 41 ... Marker 45 ... Vibration presentation unit 100 ... Three-dimensional display device

Claims

A detection unit that detects the position and orientation of the indicator based on the image of the indicator used by the user, and
An information processing device including a display control unit that displays an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.
The information processing device according to claim 1.
The display control unit calculates a virtual insertion amount of the indicator into the display space based on the position and orientation of the indicator, and controls the display of the instruction object according to the insertion amount. ..
The information processing device according to claim 2.
The display control unit is an information processing device that controls the display of the instruction object so that the length of the instruction object is proportional to the insertion amount.
The information processing device according to claim 1.
The stereoscopic display device has a display panel for displaying a stereoscopic image.
The display space is a virtual space with the display panel as a boundary.
The detection unit is an information processing device that detects the position and orientation of the indicator with respect to the display panel.
The information processing device according to claim 4.
The indicator can be expanded and contracted in one direction in response to pressure.
The display control unit is an information processing device that calculates the amount of expansion and contraction of the indicator in response to contact between the display panel and the indicator as a virtual insertion amount of the indicator into the display space.
The information processing device according to claim 4.
The display control unit is an information processing device that displays the instruction object starting from the contact position between the display panel and the indicator in the display space.
The information processing device according to claim 4.
The stereoscopic display device has an image pickup unit directed to an observation range of the display panel.
The detection unit is an information processing device that detects the position and orientation of the indicator used by the user in the observation range based on the image of the observation range captured by the imaging unit.
The information processing device according to claim 7.
The detection unit detects the viewpoint of the user who observes the display panel based on the image of the observation range captured by the imaging unit.
The display control unit is an information processing device that displays the stereoscopic image according to the viewpoint of the user.
The information processing device according to claim 1.
The indicator has a marker portion and has a marker portion.
The detection unit is an information processing device that detects the position and orientation of the marker unit as the position and orientation of the indicator.
The information processing device according to claim 9.
The indicator has a tip that is directed at the object and a grip that is connected to the tip.
The marker portion is an information processing device connected to a side opposite to the side to which the tip portion of the grip portion is connected.
The information processing device according to claim 1.
The display control unit is an information processing device that displays a target object that is a target of an input operation using the indicator in the display space.
The information processing device according to claim 11.
The display control unit is an information processing device that sets the type of the instruction object according to the type of the target object.
The information processing device according to claim 11.
The display control unit is an information processing device that controls the display of the target object in response to contact between the target object and the instruction object.
The information processing device according to claim 11.
The indicator has a sensation presenting portion that presents a skin sensation to the user.
Further, an information processing device including a sensory control unit that controls the sensory presentation unit in response to contact between the target object and the instruction object.
The information processing apparatus according to claim 14.
The sensation presenting unit is an information processing device that presents at least one of a vibration sensation and a heat sensation.
The information processing apparatus according to claim 14.
The sensory control unit is an information processing device that controls the sensory presentation unit according to at least one of the type of the target object, the type of the instruction object, or the amount of contact between the target object and the instruction object.
The position and orientation of the indicator are detected based on the image of the indicator used by the user.
An information processing method in which a computer system executes an instruction object representing a virtual insertion portion of the indicator to the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.
A step of detecting the position and orientation of the indicator based on the image of the indicator used by the user, and
A program that causes a computer system to perform a step of displaying an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.