JP2017091433A - Head-mounted display device, method for controlling head-mounted display device, computer program - Google Patents

Abstract

A head-mounted display device capable of prompting a user to pay attention to a real object that exists in the real world while ensuring visibility in the real world.
A head-mounted display device in which a user can visually recognize a virtual image includes an image display unit that allows the user to visually recognize a virtual image, and an outside scene in a user's visual field direction when the head-mounted display device is mounted. An image acquisition unit that acquires an image, and guidance for a target object that is real in the real world and that is included in the acquired outside scene image and that is a real object that prompts the user to pay attention And a guide unit that performs sound.
[Selection] Figure 4

Description

  The present invention relates to a head-mounted display device.

  2. Description of the Related Art A head-mounted display device that is used by being mounted on a user's head and that forms a virtual image in the user's visual field region is known. This head-mounted display device is also called a head mounted display (HMD). In recent years, in such an HMD, an augmented reality (AR) technology has been used in order to attract the user's attention to a real object that exists in the real world.

  In order to implement AR in the HMD, the HMD captures an outside scene with, for example, a camera, recognizes an image obtained by the imaging, and generates or acquires a virtual object. In a transmissive HMD in which the user's field of vision is not blocked when the HMD is worn, the user is allowed to visually recognize only a virtual image including a virtual object. The user can experience AR by looking at both real objects in the real world and virtual objects represented by virtual images. Patent Document 1 discloses an image of an arrow that represents a direction in which a user should move next in accordance with the position of a work target that is a real object that exists in the real world and the current position of the user in the HMD. A technique for displaying is described.

JP 2014-71756 A JP 2002-159019 A

  A virtual image of the HMD is formed in the visual field region of the user. For this reason, in the technique described in Patent Document 1, the larger the amount of information displayed as a virtual image (image), the narrower the user's field of view, in other words, the real world visibility decreases. There was a problem. As described above, there is room for improvement in ensuring the visibility of the real world in the technology using AR in order to attract the user's attention to a real object that exists in the real world. In the technique described in Patent Document 2, no consideration is given to prompting the user to pay attention to a real object that exists in the real world.

  For this reason, a head-mounted display device capable of prompting a user's attention to a real object that exists in the real world while ensuring visibility in the real world has been desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device in which a user can visually recognize a virtual image is provided. The head-mounted display device includes: an image display unit that allows the user to visually recognize the virtual image; and an image acquisition unit that acquires an outside scene image in the user's visual field direction when the head-mounted display device is mounted. Guidance that uses sound to guide a target object that is real in the real world and that is at least one real object included in the acquired outside scene image, and that is a real object that prompts the user to pay attention A section.
According to the head-mounted display device of this aspect, the guide unit is an actual object that exists in the real world and is at least one real object included in the acquired outside scene image, and that prompts the user to pay attention. Guidance for the object of interest, which is an object, can be performed by sound. Compared with guidance using a virtual image formed in the user's visual field area, the guidance using sound reduces the risk of narrowing the user's visual field, in other words, lowering the visibility of the real world. be able to. As a result, it is possible to provide a head-mounted display device that can prompt the user to pay attention to a real object that actually exists in the real world while ensuring visibility in the real world.

(2) The head-mounted display device according to the above aspect further includes: a line-of-sight detection unit that detects a direction of the user's line of sight when the head-mounted display device is mounted; If it is estimated that the object of interest is not in the detected direction of the line of sight, the guidance is performed by the sound; if the object of interest is estimated to be in the direction of the detected line of sight, the sound In addition, the guidance may be performed by causing the image display unit to form the virtual image including a virtual object to be additionally displayed on the object of interest.
According to the head-mounted display device of this aspect, the guidance unit performs guidance using sound when it is estimated that the object of interest is not in the direction of the user's line of sight. For this reason, the guide unit can suppress the formation of a virtual image in the visual field region of the user for guiding an object of interest that is not in the direction of the line of sight, and can ensure visibility in the real world. . On the other hand, when it is estimated that the target object is in the direction of the user's line of sight, the guide unit forms a virtual image including a virtual object for additionally displaying the target object in addition to the sound. I will guide you. For this reason, the guide unit can perform guidance for the object of interest in the direction of the line of sight using both sound and virtual images. That is, the user can enjoy guidance through visual and auditory senses for the object of interest in the direction of his / her line of sight. As a result, in the head-mounted display device that can attract the user's attention to the real object that exists in the real world while ensuring the visibility in the real world, the convenience of the user is further improved. Can be improved.

(3) In the head-mounted display device of the above aspect; the guidance; guidance for alerting the user to the attention object; guidance of information regarding the attention object; guidance regarding the position of the attention object And at least one of: guidance of the user's work contents for the object of interest based on procedure information prepared in advance.
According to the head-mounted display device of this form, guidance by the guide unit can be realized in various modes. For example, when performing guidance for alerting the user to the attention object, the guidance unit can perform guidance for avoiding danger to the user. When guiding information related to the object of interest, the guiding unit can perform guidance for providing information to the user. When performing guidance regarding the position of the object of interest, the guidance unit can provide guidance for assisting the operation of the user. When guiding the user's work content for the object of interest, the guide unit can provide guidance for work support to the user.

(4) The head-mounted display device according to the above aspect, wherein the guide unit includes: the alert, the information, the position, and the work content as the guide. A head-mounted display device that is determined according to an operation input from a user.
According to this form of the head-mounted display device, the guidance unit determines the type of guidance (attention, information, position, work content) in accordance with the operation input from the user. Can be improved.

(5) In the head-mounted display device of the above aspect; the image acquisition unit repeatedly acquires the outside scene image; the guide unit further recognizes each of the acquired plurality of outside scene images; A real object included in a plurality of outside scene images and having a large change with time may be used as the object of interest.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image. In addition, since the guidance unit sets a real object having a large change over time in a plurality of repeatedly acquired outside scene images as a target object, for example, a real object having a large movement amount or a high movement speed is selected as the target object. It can be. The attention object identified in this way is particularly suitable for guidance for avoiding danger.

(6) In the head-mounted display device according to the above aspect; the guide unit further recognizes the acquired outside scene image; an actual object included in the outside scene image and has a predetermined feature An object may be the attention object.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image. In addition, in order to make a real object having a predetermined characteristic an attention object in an outside scene image, for example, by preparing a “predetermined characteristic” in advance, various kinds of real objects are set as attention objects. Can do. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(7) In the head-mounted display device according to the above aspect, the head-mounted display device further includes: a line-of-sight detection unit that detects a direction of the user's line of sight in a state in which the head-mounted display device is mounted; The outside scene image is repeatedly acquired; the line-of-sight detection unit repeatedly detects the direction of the line of sight; the guide unit is further: a plurality of the acquired outside scene images and the direction of the line of sight detected multiple times And a real object that is included in the plurality of outside scene images and that is estimated to have a large proportion at the end of the line of sight may be used as the attention object.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image and the detected direction of the line of sight. In addition, since the guide unit uses a real object that is estimated to have a large proportion of the point of the line of sight in a plurality of repeatedly acquired outside scene images and the direction of the line of sight as an attention object, for example, An object can be a target object. The object of interest identified in this way is particularly suitable for types of guidance such as information provision, movement assistance, and work assistance.

(8) The head-mounted display device of the above aspect further includes: an input information acquisition unit that acquires an operation input from the user; and the guide unit is further acquired with the acquired outside scene image. The operation input may be collated; a real object included in the outside scene image and designated by the user may be the attention object.
According to this form of the head-mounted display device, the guide unit can automatically identify the object of interest from the acquired outside scene image and the acquired operation input. In addition, since the guidance unit uses the real object specified by the user as the target object, the target object can be specified reflecting the intention of the user. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(9) The head-mounted display device according to the above aspect; the image acquisition unit repeatedly acquires the outside scene image; and the input information acquisition unit recognizes the repeatedly acquired outside scene image. The movement of the user's body part is acquired; the guide unit is configured to collate the acquired outside scene image with the movement of the body part acquired as the operation input; A head-mounted display device that uses a real object intended by the user based on the movement of a part of the body as the object of interest.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image and the movement of a part of the user's body. For this reason, the user can intuitively specify the object of interest by performing an operation such as surrounding or pointing to the intended real object. As a result, it is possible to identify the attention object reflecting the user's intention and to improve the convenience for the user.

(10) The head-mounted display device according to the above aspect, wherein the guide unit is further a virtual object serving as a mark when the user designates the object of interest, and the body of the user A head-mounted display that causes the image display unit to form the virtual image including a virtual object in which at least one of position, size, and shape is changed over time so as to follow a part of the movement of apparatus.
According to this type of head-mounted display device, the guide unit causes the image display unit to form a virtual image including a virtual object that serves as a mark when the user designates the target object. For this reason, the user can specify the target object while confirming which real object is to be specified as the target object from among the plurality of real objects within his field of view. As a result, it is possible to further improve convenience for the user.

(11) The head-mounted display device according to the above aspect, wherein the input information acquisition unit is configured to acquire the operation input from another device connected to the head-mounted display device. Display device.
According to the head-mounted display device of this aspect, the guide unit can acquire an operation input from another device connected to the head-mounted display device, thereby improving convenience for the user. be able to.

(12) The head-mounted display device according to the above aspect, wherein the image display unit forms the virtual image representing a virtual operation unit for operating the head-mounted display device; The unit repeatedly acquires the outside scene image; the input information acquisition unit; the movement of the part of the user's body acquired by recognizing the repeatedly acquired outside scene image and the virtual operation A head-mounted display device that acquires the operation input by collating with the position of each part in the part.
According to the head-mounted display device of this aspect, the guide unit can acquire an operation input using a virtual operation unit for operating the head-mounted display device. Can be improved.

(13) In the head-mounted display device of the above aspect; the image acquisition unit repeatedly acquires the outside scene image; the guide unit; the time-lapse of the object of interest in the plurality of acquired outside scene images The guidance may be repeatedly performed according to the change.
According to this form of the head-mounted display device, the guidance unit repeatedly performs guidance according to the temporal change of the object of interest in the plurality of repeatedly acquired outside scene images. As a result, the guide unit can perform guidance corresponding to the latest state of the target object at any time. In other words, the guide unit can perform guidance while tracking the target object.

(14) The head-mounted display device according to the above aspect, wherein the guide unit further acquires additional information used for the guidance.
According to the head-mounted display device of this embodiment, the guide unit can acquire additional information used for guidance, and therefore can perform guidance more accurately in consideration of this additional information. .

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including some or all of the three elements of the image display unit, the image acquisition unit, and the guide unit. That is, this apparatus may or may not have an image display unit. In addition, this apparatus may or may not have an image acquisition unit. Moreover, this apparatus may have a guide part and does not need to have it. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device. For example, an apparatus according to an embodiment of the present invention has an object of urging a user to pay attention to a real object that exists in the real world while ensuring visibility in the real world. However, there are other demands for this device such as downsizing of the device, improvement in convenience, cost reduction during device manufacture, resource saving, and ease of manufacture.

  The present invention can be realized in various modes. For example, a head-mounted display device, a head-mounted display device control method, a system including a head-mounted display device, and these methods The present invention can be realized in the form of an apparatus, a computer program for realizing the functions of the system, a server apparatus for distributing the computer program, a storage medium storing the computer program, and the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus in one Embodiment of this invention. It is a block diagram which shows the structure of HMD functionally. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is a flowchart which shows the procedure of a guidance process. It is a figure which shows an example of the information for guidance. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 2nd specific example of guidance processing. It is a figure explaining the 2nd specific example of guidance processing. It is a figure explaining the 2nd specific example of guidance processing. It is a figure explaining modification 1 of guidance processing. It is a figure explaining modification 1 of guidance processing. It is a figure explaining modification 2 of guidance processing. It is a figure which shows an example of the information for guidance in the deformation | transformation 3. FIG. It is a figure which shows an example of an input device. It is a block diagram which shows the structure of an input device functionally. It is a figure which shows the other example of an input device. It is a figure which shows an example of a virtual input device. It is explanatory drawing which shows the structure of the external appearance of HMD in a modification.

A. Embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device 100 of the present embodiment is a display device that is mounted on the head, and is also called a head mounted display (HMD). The HMD 100 is an optically transmissive head mounted display that allows a user to visually recognize a virtual image and at the same time also visually recognize an outside scene.

  The HMD 100 according to the present embodiment performs guidance for an object of interest by “voice only” or “a combination of voice and augmented reality (AR) processing” by performing a guidance process described later. Specifically, the HMD 100 detects the direction of the user's line of sight, and when it is estimated that the object of interest is not in the direction of the line of sight, the HMD 100 performs guidance only by voice and estimates that the object of attention is in the direction of the line of sight. If so, guidance is provided by a combination of voice and augmented reality processing. Details of the guidance process will be described later. In this embodiment, “voice” functions as “sound”.

  The “attention object” means at least one real object identified from real objects that are objects that exist in the real world, and means a real object that prompts the user to pay attention. The real object means an arbitrary person, an arbitrary animal or plant, an arbitrary object (including an artificial object, a natural object, or the like). A real object is an “real object of interest” that is an object that the user is interested in (for example, the line of sight), and is outside the user's interest (for example, the line of sight that is in the field of view). It can include both “real background objects” that are objects. That is, in the guidance process, the entire real object including both the real interest object and the real background object can be processed. A method for specifying the object of interest will be described later.

“Augmented reality processing” is processing for adding information to a real object by the HMD 100. In the augmented reality processing, information displayed in addition to the real object is referred to as a “virtual object”. “Additional” means to include all of the following. The real object and the virtual object additionally displayed on the real object may or may not be related.
Add information to real objects using virtual objects.
Emphasize real objects using virtual objects.
-Delete and attenuate information (color, shape, etc.) held by a real object using a virtual object.
Any combination of two or more of addition, emphasis, deletion, and attenuation described above.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control unit (controller) 10 that controls the image display unit 20. In the following description, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience. In addition, the HMD 100 emitting the image light generated based on the image data is also referred to as “displaying an image”.

A-1-1. Configuration of image display:
FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment (FIG. 1). The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a line-of-sight detection unit 62, a microphone 63, and a nine-axis sensor 66. Hereinafter, the positional relationship and function of each unit of the image display unit 20 in a state where the user wears the image display unit 20 will be described.

  As shown in FIG. 1, the right optical image display unit 26 and the left optical image display unit 28 are disposed so as to be positioned in front of the user's right eye and in front of the left eye, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected at a position corresponding to the eyebrow of the user. As shown in FIG. 2, the right optical image display unit 26 includes a right light guide plate 261 and a light control plate (not shown). The right light guide plate 261 is formed of a light transmissive resin material or the like, and guides the image light output from the right display driving unit 22 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plate 261 and suppresses damage, dirt adhesion, and the like. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the external light quantity entering the user's eyes and adjust the ease of visual recognition of the virtual image. The dimming plate can be omitted.

  The left optical image display unit 28 includes a left light guide plate 262 and a light control plate (not shown). These details are the same as those of the right optical image display unit 26. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”. The optical image display unit can use any method as long as it forms a virtual image in front of the user's eyes using image light. For example, the optical image display unit may be realized by using a diffraction grating, or a transflective film may be used. It may be realized using.

  As shown in FIG. 1, the right holding unit 21 extends from the other end ER of the right optical image display unit 26 to a position corresponding to the user's temporal region. The left holding unit 23 is provided to extend from the other end EL of the left optical image display unit 28 to a position corresponding to the user's temporal region. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses. The right holding unit 21 and the left holding unit 23 are collectively referred to simply as a “holding unit”.

  As shown in FIG. 1, the right display driving unit 22 is disposed inside the right holding unit 21 (the side facing the user's head). The left display driving unit 24 is disposed inside the left holding unit 23. As shown in FIG. 2, the right display driving unit 22 functions as a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, and a display element. A right LCD (Liquid Crystal Display) control unit 211, a right LCD 241, and a right projection optical system 251 are provided. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED (Light Emitting Diode) or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data for the right eye input via the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right projection optical system 251 is a collimating lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state.

  The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. These details are the same as those of the right display drive unit 22. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”.

  As shown in FIG. 1, the camera 61 is a stereo camera disposed at a position corresponding to the upper left and right corners of the user. The left and right cameras 61 each capture an outside scene (external scenery) in the front side direction of the image display unit 20, in other words, the user's viewing direction when the HMD 100 is worn, and two outside scene images corresponding to the left and right sides. To get. The camera 61 is a so-called visible light camera, and the outside scene image acquired by the camera 61 is an image representing the shape of the object from the visible light emitted from the object. The camera 61 in the present embodiment is a stereo camera, but may be a monocular camera. The camera 61 functions as an “image acquisition unit”.

  As shown in FIG. 1, the line-of-sight detection unit 62 is disposed at a position corresponding to the lower part of the left and right corners of the user. The left and right line-of-sight detection units 62 each include an infrared light emitting unit and an infrared light receiving unit (not shown). The right gaze detection unit 62 receives the infrared light emitted from the infrared light emitting unit and reflected by the right eye of the user. The CPU 140 of the control unit 10 acquires the movement of the line of sight of the user's right eye based on the intensity of infrared light received by the right line of sight detection unit 62. Similarly, the left line-of-sight detection unit 62 receives the infrared light emitted from the infrared light emitting unit and reflected by the left eye of the user. The CPU 140 detects the line of sight of the user's left eye based on the intensity of infrared light received by the left line-of-sight detection unit 62. The reflectance of infrared rays is different depending on whether the infrared ray hits the iris (black eye), the infrared ray hits the eyelid, or the infrared ray hits the white eye. Therefore, the CPU 140 can acquire the movement of the user's line of sight based on the intensity of infrared rays received by the line-of-sight detection unit 62. In this case, the line-of-sight detection unit 62 and the CPU 140 function as a “line-of-sight detection unit” that detects the direction of the user's line of sight. Although the line-of-sight detection unit 62 in the present embodiment is provided on each of the left and right sides, it may be on either the left or right side.

  As shown in FIG. 1, the microphone 63 is a microphone disposed at a position corresponding to the upper side of the user's nose. The microphone 63 acquires external sound. “Voice” means not only a human voice but also a broad sense of sound including mechanical sounds.

  As shown in FIG. 1, the 9-axis sensor 66 is disposed at a position corresponding to the temple on the right side of the user. The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of detecting the movement of the user's head on the head mounted display 100. Functions as a detection unit. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  As shown in FIG. 1, the image display unit 20 includes a connection unit 40 for connecting the image display unit 20 and the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 branched from the main body cord 48, and a connecting member 46 provided at the branch point. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. For example, a metal cable or an optical fiber can be employed for each cord of the connection unit 40.

A-1-2. Configuration of control unit:
The control unit 10 is a device for controlling the HMD 100. As shown in FIG. 1, the control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power source. And a switch 18. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated in the control unit 10. The lighting unit 12 is realized by, for example, an LED, and notifies the operation state of the HMD 100 (for example, ON / OFF of the power source) according to the light emission state. The display switching key 13 detects a pressing operation, and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various methods such as an electrostatic method, a pressure detection method, and an optical method can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  2, the control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and a transmission unit (Tx ) 51 and 52, and the respective parts are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires signals corresponding to operation inputs to the enter key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18. The input information acquisition unit 110 can acquire operation inputs using various methods other than those described above. For example, the user's gesture may be detected using the camera 61, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical instrument held by the user's hand, or the like can be used as a mark for motion detection. For example, an operation input by a command associated with the user's line of sight detected by the line-of-sight detection unit 62 or the movement of the eyes may be acquired. This command may be set to be addable by the user. For example, an operation input by recognizing voice acquired by the microphone 63 may be acquired. For example, an operation input by a foot switch (not shown) (a switch operated by a user's foot) may be acquired. If the operation input by these methods can be acquired, the input information acquisition unit 110 can acquire the operation input from the user even in the work where it is difficult for the user to release his / her hand.

  The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 stores various computer programs including an operating system (OS). The storage unit 120 stores guidance information 122 and a guidance mode 124. The guidance information 122 is information used in the guidance process. Details will be described later.

  The guidance mode 124 is information for designating a guidance method in guidance processing. Guidance methods in the guidance process include a “with tracking mode” and a “without tracking mode”. In the no-tracking mode, the guide unit 144 performs guidance for the object of interest only once. In the tracking mode, the guide unit 144 repeatedly performs guidance on the target object as the target object changes with time. In the guidance mode 124, information representing either one of the no tracking mode or the tracking mode is stored. The user can change the content of the guidance mode 124 by an operation input by the input information acquisition unit 110.

  The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with an external device in accordance with a predetermined wireless communication standard. The predetermined wireless communication standard is, for example, infrared, short-range wireless communication exemplified by Bluetooth (registered trademark), wireless LAN exemplified by IEEE 802.11, or the like. The GPS module 134 detects the current position of the user of the HMD 100 by receiving a signal from a GPS satellite, and generates current position information representing the current position information of the user. The current position information can be realized by coordinates representing latitude and longitude, for example.

  The CPU 140 reads out and executes a computer program stored in the storage unit 120, thereby executing an augmented reality processing unit 142, a guide unit 144, an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, and a display. It functions as the control unit 190.

  The augmented reality processing unit 142 executes augmented reality processing when performing guidance by “a combination of voice and augmented reality processing” in the guidance processing. In the augmented reality processing, the augmented reality processing unit 142 adds a virtual object to the object of interest (real object) and displays it. Details will be described later. The guide unit 144 executes guidance processing to implement guidance for the object of interest by “voice only” or “combination of voice and augmented reality processing”. Details will be described later.

  The image processing unit 160 performs signal processing for image display. Specifically, when content (video) is input via the interface 180 or the wireless communication unit 132, the image processing unit 160 generates image data Data based on the content. When the image processing unit 160 receives image data from another functional unit of the HMD 100, the image processing unit 160 sets the received data as image data Data. The image processing unit 160 may execute image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data. The image processing unit 160 transmits the image data Data, the clock signal PCLK, the vertical synchronization signal VSync, and the horizontal synchronization signal HSync to the image display unit 20 via the transmission units 51 and 52. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data Data2”.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 turns on and off the left and right LCDs 241 and 242 by the left and right LCD control units 211 and 212 and the left and right backlights by the left and right backlight control units 201 and 202 according to the control signal. The generation and emission of image light by the right display drive unit 22 and the left display drive unit 24 are controlled by individually controlling the driving ON / OFF of the 221 and 222, respectively. The display control unit 190 transmits these control signals to the image display unit 20 via the transmission units 51 and 52.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to a speaker (not shown) of the right earphone 32 and a speaker (not shown) of the left earphone 34.

  The interface 180 communicates with the external device OA in accordance with a predetermined wired communication standard. The predetermined wired communication standards include, for example, MicroUSB (Universal Serial Bus), USB, HDMI (High Definition Multimedia Interface, HDMI is a registered trademark), DVI (Digital Visual Interface), VGA (Video Graphics Array), Composite, RS- The wired LAN is exemplified by 232C (Recommended Standard 232) and IEEE802.3. As the external device OA, for example, a personal computer PC, a mobile phone terminal, a game terminal, or the like can be used.

  FIG. 3 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 3A illustrates the user's visual field VR when the augmented reality processing is not executed. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the virtual image VI. In the example of FIG. 3A, the virtual image VI is a standby screen of the OS 150 of the HMD 100. The user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the HMD 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. In addition, the portion of the visual field VR where the virtual image VI is not displayed can be seen through the optical scene display portion and directly through the outside scene SC.

  FIG. 3B illustrates the visual field VR of the user when the augmented reality process is executed. By executing the augmented reality process, the user visually recognizes the virtual image VI including the virtual object VO. The virtual object VO is an image of an apple arranged so as to be superimposed on the foot of the mountain of the outside scene SC. In this way, the user can experience augmented reality by seeing both the virtual object VO included in the virtual image VI and the real object in the outside scene SC that can be seen behind the virtual image VI. it can.

A-2. Guidance process:
FIG. 4 is a flowchart showing the procedure of the guidance process. The guidance process is a process for performing guidance for the object of interest by “voice only” or “a combination of voice and augmented reality processing”. The start trigger of the guidance process can be arbitrarily determined. For example, a power ON of the HMD 100 may be used as a start trigger, and a process start request from the OS 150 of the HMD 100 or a specific application may be used as a start trigger.

  In step S102, the guide unit 144 causes the camera 61 to acquire an outside scene image.

  In step S104, the guide unit 144 identifies at least one real object from among the real objects included in the outside scene image acquired in step S102, as a target object that should attract attention from the user. Specifically, the guide unit 144 specifies the object of attention using any one of the following methods a1 to a4. Methods a1 to a4 may be used alone or in combination.

(A1) A method of using the change over time of a real object:
The guide unit 144 recognizes the outside scene image acquired in step S102, and models various objects (for example, glasses, bottles, human bodies, cars, trees, etc.) stored in the storage unit 120 or the like in advance. Then, all real objects included in the outside scene image are recognized by performing comparison by pattern matching or statistical identification method (procedure a11). In step a11, instead of recognizing all real objects, only main real objects may be recognized. In step a11, the outside scene image may be divided into a plurality of areas for image recognition. The plurality of areas may be, for example, four areas obtained by dividing the outside scene image into two equal parts in the horizontal direction and the vertical direction, or nine areas obtained by dividing the outside scene image into three parts in the horizontal direction and the vertical direction, respectively. There may be.
The guiding unit 144 repeats the acquisition of the outside scene image (step S102) and the recognition of the real object with respect to the acquired latest outside scene image (procedure a11) a plurality of times (procedure a12).
From the result of step a12, the guide unit 144 identifies a real object having a large change over time in a plurality of repeatedly acquired outside scene images as a target object (step a13). The large change over time includes both a case where the amount of movement over time is large and a case where the movement speed over time is high. The number of objects of interest specified here may be one or more.

  According to the method a1, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image. In addition, in the plurality of repeatedly acquired outside scene images, the guide unit 144 sets a real object having a large change over time as a target object. For example, the guide unit 144 pays attention to a real object having a large movement amount or a high movement speed. It can be an object. The attention object identified in this way is particularly suitable for guidance for avoiding danger.

(A2) Method of using features of real object:
The guide unit 144 recognizes the outside scene image acquired in step S102, and features (for example, color features, shape features, size features, etc.) stored in the storage unit 120 or the like in advance. Functions as a “predetermined feature”) and performs comparison by pattern matching or statistical identification (step a21).
The guide unit 144 identifies a real object that matches the comparison result of the procedure a21 in the outside scene image as a target object (procedure a22). The number of objects of interest specified here may be one or more.

  According to the method a2, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image. In addition, since the guide unit 144 uses a real object having a predetermined feature in the outside scene image as a target object, for example, by preparing the “predetermined feature” in advance as described above, various real objects can be displayed. It can be an object of interest. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(A3) Method of using the user's line of sight:
The guide unit 144 recognizes the outside scene image acquired in step S102 and recognizes all (or main) real objects included in the outside scene image (procedure a31). Details are the same as in step a11.
The guide unit 144 causes the line-of-sight detection unit 62 to detect the direction of the user's line of sight. The guide unit 144 collates the real object recognized in the procedure a31 with the detected gaze direction, and extracts a real object estimated to be ahead of the gaze direction (procedure a32).
The guide unit 144 acquires an outside scene image (step S102), recognizes a real object with respect to the latest acquired outside scene image (procedure a31), and extracts a real object estimated to be ahead of the direction of the line of sight ( Procedure a32) is repeated a plurality of times (procedure a33).
The guide unit 144 identifies, as a target object, a real object that is estimated to have a higher proportion of the direction of the line of sight in the plurality of repeatedly acquired outside scene images and the direction of the line of sight from the result of the procedure a33. (Procedure a34). The number of objects of interest specified here may be one or more.

  According to the method a3, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image and the detected line-of-sight direction. In addition, the guide unit 144 uses a real object that is estimated to have a large proportion of the point of view in the plurality of repeatedly acquired outside scene images and the direction of the line of sight as an attention object. A real object can be a target object. The object of interest identified in this way is particularly suitable for types of guidance such as information provision, movement assistance, and work assistance.

(A4) Method of using user designation:
The guide unit 144 recognizes the outside scene image acquired in step S102 and recognizes all (or main) real objects included in the outside scene image (procedure a41). Details are the same as in step a11.
The guide unit 144 acquires an operation input from the user acquired by the input information acquisition unit 110. The guiding unit 144 collates the real object recognized in the procedure a41 with the acquired operation input, and specifies the real object designated by the user as the attention object (procedure a42). The number of objects of interest specified here may be one or more. As described above, the operation input from the user via the input information acquisition unit 110 may be, for example, an input using a gesture, an input using a movement of eyes, or a voice. The input may be an input using a foot switch.

  According to the method a4, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image and the acquired operation input. In addition, since the guide unit 144 uses the real object specified by the user as the attention object, the guidance object 144 can identify the attention object reflecting the intention of the user. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

  In step S106 in FIG. 4, the guide unit 144 acquires content to be guided to the user (hereinafter also referred to as “guidance content”) for the object of interest specified in step S104. Specifically, the guidance unit 144 acquires the guidance content by searching the guidance information 122 using the name, feature, model, and the like of the object of interest.

  FIG. 5 is a diagram illustrating an example of the guidance information 122. The guidance information 122 includes object 1, object 2, and guidance content.

  In “Object 1” and “Object 2”, information for collation using the name, feature, model, etc. of the object of interest is stored. In the example of FIG. 5, the names of the objects of interest are stored in the objects 1 and 2. Therefore, when the guidance information 122 illustrated in FIG. 5 is used, a search using the name of the object of interest is performed in step S106 (FIG. 4) of the guidance process. When a search using the feature of the target object is performed in step S106 of the guidance process, information representing the feature of the target object is stored in object 1 and object 2. When a search using the model of the object of interest is performed in step S106 of the guidance process, the object 1 and the object 2 store information representing the model of the object of attention.

  In “Guidance contents”, information representing contents to be guided to the user is stored. In the example of FIG. 5, the guidance content stores one or two character strings representing the content to be guided to the user. In FIG. 5, a character string without “(*)” notation at the end of the character string represents guidance information used in both the guidance in the mode without tracking and the guidance in the mode with tracking. What has the notation “(*)” at the end of the character string represents guidance information used only for guidance in the tracking mode. The guidance content may be stored as a character string as illustrated, may be stored as a graphic or an image, or may be stored as program data executed in the guide unit 144.

The “guidance content” may include the following information b1 to b4.
(B1) Guidance for alerting the user to the object of interest: According to the information b1, the guidance unit 144 can implement guidance for avoiding danger to the user.
(B2) Information guidance on the object of interest: According to the information b2, the guidance unit 144 can carry out guidance for providing information to the user.
(B3) Guidance on the position of the object of interest: According to the information b3, the guiding unit 144 can perform guidance for assisting the operation of the user.
(B4) User's work content guidance for the object of interest: According to the information b4, the guide unit 144 can implement guidance for work support for the user.

  Specifically, in the example of the entry E1 in FIG. 5, when the object 1 is “bottle” and the object 2 is “glass”, the “bottle mouth glass” is set in both the non-tracking mode and the tracking mode. “Guide to the center of the screen (corresponding to information b3)” and “Guiding the amount of liquid in the glass (corresponding to information b2)” in the tracking mode. Further, in the example of entry E5, when the object 1 is “car”, “notice guidance to the car (corresponding to information b1)” in both the non-tracking mode and the tracking mode and tracking In the mode, “Guide distance with car (corresponding to information b2)” is stored. Thus, the object 2 may not store data.

  In step S108 in FIG. 4, the guide unit 144 causes the line-of-sight detection unit 62 to detect the direction of the user's line of sight.

  In step S110, the guide unit 144 collates the position of the target object specified in step S104 in the outside scene image with the detected line-of-sight direction, and estimates whether the target object is in the line-of-sight direction. .

  When it is estimated that the object of interest is not in the line of sight (step S110: no line of sight), in step S112, the guide unit 144 performs guidance based on the guidance content acquired in step S106 only by voice. Specifically, the guidance unit 144 generates or acquires voice data for guidance, and transmits the voice data to the voice processing unit 170. The sound processing unit 170 outputs sound based on the received sound data from the speaker in the right earphone 32 and the speaker in the left earphone 34.

  When it is estimated that the object of interest is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 performs guidance based on the guidance content acquired in step S106 with voice and augmented reality processing. It carries out by the combination. A specific method of voice guidance is the same as in step S112. A specific method of guidance by augmented reality processing is as shown in the following procedures c1 to c5.

(C1) The guide unit 144 performs augmented reality processing on information (name, feature, model, etc. of the target object) for specifying the target object specified in step S104 and the guide content acquired in step S106. To the unit 142.

(C2) The augmented reality processing unit 142 causes the camera 61 to acquire an outside scene image. The augmented reality processing unit 142 acquires the position of the target object in the acquired outside scene image using the information for specifying the target object acquired in step c1. The “position” includes the distance from the user and the horizontal and vertical positions in the user's field of view. At this time, the augmented reality processing unit 142 may calculate the position of the object of interest using two or more outside scene images acquired by the camera 61 that is a stereo camera. In addition, the augmented reality processing unit 142 uses one or more outside scene images acquired by the camera 61 and various sensors (not shown) (for example, a depth sensor, a distance measuring sensor, etc.) in combination to determine the position of the target object. May be calculated.

(C3) The augmented reality processing unit 142 displays a virtual object to be additionally displayed on the object of interest according to the content of the guidance content acquired in step c1 that does not have “(*)” at the end. Get or generate images, characters, graphic symbols, etc. to represent. Examples of images, characters, graphic symbols and the like representing virtual objects are, for example, arrows, lines, scales, scales, pointers, shortcuts, menus, radio buttons, selection buttons, soft keyboards, and the like. The augmented reality processing unit 142 may acquire a virtual object stored in advance in the storage unit 120 or may acquire a virtual object stored in advance in another device connected to the HMD 100 via a network.

(C4) The augmented reality processing unit 142 arranges the virtual object acquired or generated in step c3 in accordance with the position of the target object acquired in step c2, and generates additional image data in which black is arranged in other portions. To do. At the time of alignment, the augmented reality processing unit 142 may use a characteristic part (such as an edge) of the target object, or may use a marker such as a marker attached to the target object. At the time of alignment, the augmented reality processing unit 142 may use image recognition using a model (or image) of the target object stored in the storage unit 120 in advance. At the time of alignment, the augmented reality processing unit 142 may be based on an instruction from the user. The augmented reality processing unit 142 may perform image processing such as enlargement, reduction, rotation, and color conversion on the virtual object when arranging the virtual object.

(C5) The augmented reality processing unit 142 transmits an image based on the generated additional image data to the image processing unit 160 and causes the optical image display units 26 and 28 of the image display unit 20 to display the images.

  After step S112 or step S114 in FIG. 4 is completed, the guide unit 144 acquires the content stored in the guide mode 124. When the content of the guidance mode 124 is “no tracking mode”, the guidance unit 144 ends the guidance process. When the content of the guidance mode 124 is “tracking mode”, the guide unit 144 waits for an arbitrary predetermined time, and then repeats the process of step S102 and the processes of steps S108 to S114. In the second and subsequent steps S108 to S114, the guide unit 144 includes “(*) at the end of the attention object identified in the first step S104 and the guidance content acquired in the first step S106. ”And the content of what has the notation. That is, step S104 and step S106 are not executed after the second time.

A-3. Specific examples of guidance processing:
A-3-1. First specific example:
6-8 is a figure explaining the 1st specific example of a guidance process. In the first specific example, a description will be given of guidance processing in a scene where a user is pouring liquid from a bottle to a glass. The first specific example is an example of operation assistance and information provision. In step S102 of the guidance process (FIG. 4), the guidance unit 144 acquires an outside scene image. FIG. 6 is an example of an outside scene image acquired in step S102. The outside scene image IM1 includes OB11 to OB13 which are real objects.

  In step S104 of the guidance process, the guidance unit 144 identifies an object of interest from any of the real objects OB11 to OB13 using any of the methods a1 to a4 described above. For example, it is assumed that the glass OB12 and the bottle OB11 are specified as the target object. In step S106 of the guidance process, the guidance unit 144 searches the guidance information 122 using the glass OB12 and the bottle OB11, and the guidance contents “entry the bottle mouth to the center of the glass” and “ “Guid the amount of liquid (*)”.

  In step S108 of the guidance process, the guidance unit 144 detects the direction of the user's line of sight. When both the glass OB12 and the bottle OB11 are not in the line of sight (step S110: no line of sight), in step S112, the guidance unit 144 obtains the guidance content “bottle mouth in the middle of the glass in step S106. Guidance based on “Guidance on” is performed only by voice. As a result, only the guidance voice SO11 (FIG. 7) “Please move the bottle mouth to the left” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO11 is not displayed.

  When at least one of the glass OB12 and the bottle OB11 is in the direction of the line of sight (step S110: in the direction of the line of sight), the guide unit 144 in step S114 displays the guidance content “bottle mouth” acquired in step S106. Guidance based on “Guided in the center of the glass” is implemented by a combination of voice and augmented reality processing. As a result, as shown in FIG. 7, the guidance sound SO11 “Please move the bottle mouth to the left” is reproduced from the speakers in the left and right earphones 32 and 34, and the center line of the glass OB12 is represented. A virtual image VI including the virtual object VO11 is displayed.

  When the content stored in the guidance mode 124 is the “no tracking mode”, the guidance unit 144 ends the process as described above. When the content stored in the guidance mode 124 is the “tracking mode”, the guidance unit 144 waits for a predetermined time. In step S102 of the second guidance process, the guidance unit 144 acquires an outside scene image.

  In step S108 of the second guidance process, the guidance unit 144 detects the direction of the user's line of sight. When both the glass OB12 and the bottle OB11 are not in the line of sight (step S110: no line of sight), the guide 144 in the second step S112 displays the guidance content “liquid in the glass” acquired in step S106. Guidance based on “Guidance of amount (*)” is performed only by voice. As a result, only the guidance voice SO12 (FIG. 8) “It will be full soon” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO12 is not formed.

  When at least one of the glass OB12 and the bottle OB11 is in the direction of the line of sight (step S110: in the direction of the line of sight), in the second step S114, the guide unit 144 displays the guidance content “glass” acquired in step S106. Guidance based on the guidance of the amount of liquid in (*) "is performed by a combination of voice and augmented reality processing. As a result, as shown in FIG. 8, the guidance voice SO12 “It is almost full” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO11 representing the center line of the glass OB12 and the current A virtual image VI including a virtual object VO12 in which the amount of liquid is indicated by an arrow is displayed.

A-3-2. Second specific example:
FIGS. 9-11 is a figure explaining the 2nd specific example of a guidance process. In the second specific example, guidance processing in a scene where the user is driving a car will be described. The second specific example is an example of danger avoidance and information provision. In step S102 of the guidance process (FIG. 4), the guidance unit 144 acquires an outside scene image. FIG. 9 is an example of an outside scene image acquired in step S102. The outside scene image IM2 includes OB21 to OB26 that are real objects.

  In step S104 of the guidance process, the guidance unit 144 specifies the object of interest from any of the real objects OB21 to OB26 using any of the methods a1 to a4 described above. For example, it is assumed that the signboard OB21 and the car OB23 are specified as the attention object. In step S106 of the guidance process, the guidance unit 144 searches the guidance information 122 using the signboard OB21 and the car OB23, and guides the entry contents “Estimation guidance for the signboard” and “Guide the contents of the signboard”. Then, the guidance contents of entry E5 “Guidance for alerting the car” and “Guide distance to car (*)” are acquired.

  In step S108 of the guidance process, the guidance unit 144 detects the direction of the user's line of sight. When the signboard OB21 is not in the direction of the line of sight (step S110: no line of sight), in step S112, the guide unit 144 obtains the guidance contents “guidance for calling attention to the signboard” and “the contents of the signboard” acquired in step S106. Guidance based on “guidance” is performed only by voice. As a result, only the guidance voice SO21 (FIG. 10) “Please pay attention to the rear-end collision” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual image VI including the virtual object VO21 is not displayed. When the vehicle OB23 is not in the line of sight (step S110: no line of sight), in step S112, the guide unit 144 provides voice guidance for guidance based on the guidance content “guidance for alerting the vehicle” acquired in step S106. Implemented only by. As a result, only the guidance voice SO22 (FIG. 10) “Please pay attention to the car ahead” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO22 is not displayed.

  When the signboard OB21 is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 obtains the guidance contents “guidance for alerting the signboard” and “the contents of the signboard” acquired in step S106. Guidance based on “guidance” is performed by a combination of voice and augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO21 “Please pay attention to the rear-end collision” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO21 for emphasizing the signboard OB21 is included. A virtual image VI is displayed. When the vehicle OB23 is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 provides guidance based on the guidance content “guidance for alerting the vehicle” acquired in step S106. And a combination of augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO22 “Please pay attention to the car ahead” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual object VO22 for emphasizing the car OB23. A virtual image VI including is displayed.

  When the content stored in the guidance mode 124 is the “no tracking mode”, the guidance unit 144 ends the process as described above. When the content stored in the guidance mode 124 is the “tracking mode”, the guidance unit 144 waits for a predetermined time. In step S102 of the second guidance process, the guidance unit 144 acquires an outside scene image. Since the acquired outside scene image does not include the signboard OB, the guide unit 144 excludes the signboard OB21 from the object of interest in the second process.

  In step S108 of the second guidance process, the guidance unit 144 detects the direction of the user's line of sight. When the vehicle OB23 is not in the line of sight (step S110: no line of sight), in the second step S112, the guidance unit 144 obtains the guidance content “Guide distance from car (*)” acquired in step S106. Guidance based on is performed only by voice. As a result, only the guidance voice SO23 (FIG. 10) “5 m ahead vehicle is approaching” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO21 is not displayed.

  When the vehicle OB23 is in the direction of the line of sight (step S110: in the direction of the line of sight), in the second step S114, the guide unit 144 obtains the guidance content “Guide distance from the car (*)” acquired in step S106. Based on the combination of voice and augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO23 “5 m ahead vehicle is approaching” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO22 for emphasizing the car OB23. And a virtual image VI including the virtual object VO23 indicating the distance between the user and the car OB23.

  As described above, according to the guidance process of the above embodiment, the guide unit 144 is at least one real object that exists in the real world and is included in the outside scene image acquired in step S102. On the other hand, guidance for the attention object, which is a real object that prompts attention, can be implemented by at least the voices SO11 to SO23 (steps S112 and S114). Compared with guidance using the virtual image VI formed in the user's visual field area, guidance using the voices SO11 to SO23 may reduce the visual field of the user, in other words, visibility in the real world is reduced. The fear can be reduced. As a result, it is possible to provide a head-mounted display device (HMD 100) that can prompt the user's attention to a real object that exists in the real world while ensuring visibility in the real world.

  Furthermore, according to the guidance processing of the above embodiment, when it is estimated that the object of interest is not in the direction of the user's line of sight (step S110: none in the direction of the line of sight), only the voices SO11 to SO23. The guidance is implemented by (Step S112). For this reason, the guide unit 144 can suppress the formation of the virtual image VI in the user's visual field region for guiding an object of interest that is not in the direction of the line of sight, and ensure visibility in the real world. Can do. On the other hand, when it is estimated that the target object is in the direction of the user's line of sight (step S110: in the direction of the line of sight), the guide unit 144 further adds to the target object in addition to the voices SO11 to SO23. Guidance is implemented by forming a virtual image VI including the virtual objects VO11 to VO23 to be displayed (step S114). For this reason, the guide part 144 can perform guidance with respect to the object of interest in the direction of the line of sight using both the sound SO11 to SO23 and the virtual image VI. That is, the user can enjoy guidance through visual and auditory senses for the object of interest in the direction of his / her line of sight. As a result, in the head-mounted display device (HMD100) capable of prompting the user's attention to a real object that exists in the real world while ensuring visibility in the real world, the user can further Convenience can be improved.

  Furthermore, according to the guidance process of the above embodiment, when the guidance mode 124 is the “tracking mode”, the guidance unit 144 changes the temporal change of the object of interest in the plurality of outside scene images repeatedly acquired in step S102. Accordingly, the guidance in step S112 or step S114 is repeatedly performed. As a result, the guide unit 144 can perform guidance corresponding to the latest state of the target object at any time. In other words, the guide unit 144 can perform guidance while tracking the target object.

A-4. Variations on the guidance process:
The guidance process described with reference to FIG. 4 is merely an example, and various modifications are possible. For example, you may give the various deformation | transformation demonstrated as the deformation | transformation 1-6 below. Modifications 1 to 6 may be employed alone or in combination with the above-described guidance process.

A-4-1. Deformation 1 (Specifying object of interest):
In the first modification, the real object specified by the user using a part of the body or the like is specified as the object of interest. In the first modification, the following procedure is executed in place of each procedure described above in the method a4 (identification of an object of interest using user designation).

12 and 13 are diagrams for explaining a first variation of the guidance process.
The guide unit 144 recognizes the outside scene image acquired in step S102 and recognizes all (or main) real objects included in the outside scene image (procedure a41). Details are the same as in step a11. For example, FIG. 12 shows an example of the outside scene image IM3 acquired in step S102. The guide unit 144 recognizes the real objects OB31 to OB33 included in the outside scene image IM3 by the procedure a41.

As described above, the input information acquisition unit 110 uses a part of the user's body (for example, the entire hand, the fingertip, the palm, the arm, the hand, etc.) using the outside scene image taken (repetitively) by the camera 61 over time. And the movement of a ring, a bracelet attached to an arm, a tool device held in a hand). The guide unit 144 identifies one or more real objects intended by the user from the real objects recognized in the procedure a41 by the movement of a part of the user's body detected by the input information acquisition unit 110. Then, the one or more real objects are set as the attention object (procedure a42). Here, the “real object intended by the user” means a real object that the user wants to designate as the object of interest. The guide unit 144 can grasp the user's intention by, for example, an operation of enclosing with both hands or one hand, an operation of pointing a finger, an operation of pointing using a tool or the like. For example, in the example of FIG. 13, the guide unit 144 sets the real object OB32 surrounded by the user's right hand RH and left hand LH as the object of interest. When the input information acquisition unit 110 detects the movement of a part of the user's body, the input information acquisition unit 110 is a feature point such as a part of the user's body (for example, fingertips CH1, CH2, rings, bracelets, pointers, etc. The tip of the instrument or the like may be used as a mark for motion detection.

  According to the first modification, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image and the movement of a part of the user's body. Therefore, the user can intuitively specify the object of interest by performing an operation such as surrounding or pointing to the intended real object. As a result, according to the first modification, it is possible to identify the attention object reflecting the intention of the user and improve the convenience for the user. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

A-4-2. Deformation 2 (target object specification method):
In the modification 2, in the configuration of the modification 1, a virtual image for assisting the user to specify the object of interest is displayed.

  FIG. 14 is a diagram for explaining modification 2 of the guidance process. The virtual image VI includes a virtual object VO31 that serves as a mark indicating the real object designated as the “target object” by the user among the real objects in the visual field VR of the user. The aspect and shape of the virtual object VO31 can be arbitrarily determined. As modes of the virtual object VO31, modes such as a frame surrounding the real object, a cover superimposed on the real object, and a pointer pointing to the real object can be adopted. As the shape of the virtual object VO31, a circle, a triangle, a rectangle, or the like can be adopted. In the example of FIG. 14, the virtual object VO31 is a circular frame surrounding the real object.

  In the initial state (that is, immediately after the start of procedure a42), the guide unit 144 displays the virtual object VO31 having a default size at the default position. Thereafter, the guide unit 144 changes at least one of the position, size, and shape of the virtual object VO31 so as to follow the movement of a part of the user's body in the procedure a42. The default position can be, for example, a predetermined position (center, etc.) on the screen, a position overlapping with any real object on the screen, or the like. In the example of FIG. 14, the guide unit 144 changes the position and size of the virtual object VO31 so as to follow the movement of the user's right hand RH and left hand LH. For example, the guide unit 144 gradually decreases (increases) the diameter of the virtual object VO31 when the circle formed by the right hand RH and the left hand LH gradually decreases (increases). Further, when the right hand RH and the left hand LH move upward (downward), the guide unit 144 moves the virtual object VO31 upward (downward) so as to follow it.

  According to the modification 2, the guide unit 144 displays the virtual image VI for assisting the user in specifying the object of interest. Therefore, the user uses the virtual object VO31 displayed as the virtual image VI as a mark, and checks which of the plurality of real objects in the field of view VR is the real object to be designated as the “target object”. However, the object of interest can be specified. As a result, according to the modification 2, it is possible to further improve convenience for the user.

A-4-3. Modification 3 (instruction by voice):
In the third modification, the target object and the guidance content can be specified by voice.

  FIG. 15 is a diagram illustrating an example of the guidance information 122a in the third modification. The only difference from the guidance information 122 shown in FIG. 5 is that each guidance content (character string representing the content to be guided to the user) is stored in association with the type of the guidance content. It is. In this modification, alerting, information, position, and work content are assumed as types of guidance content. The guidance content that is associated with “calling attention” as the type means guidance for avoiding danger to the user. Similarly, the guidance content associated with “information” as a type means guidance for providing information to the user. The guidance content associated with “position” as the type means guidance for assisting the operation of the user. The guidance content associated with “work content” as the type means guidance for work support for the user.

  As described above, the input information acquisition unit 110 can acquire the operation input by the voice from the user by recognizing the voice acquired by the microphone 63. Therefore, in step S104 of the guidance process shown in FIG. 4, the guidance unit 144 can identify the real object instructed by voice as the “target object” (method a4). Furthermore, the guidance unit 144 can limit the guidance content acquired and guided in steps S106 to S114 to the type (attention, information, position, work content) designated by voice.

  Moreover, the guide part 144 may acquire the additional information regarding the guidance content in steps S106 to S114 by voice. As the additional information, for example, the type of liquid contained in the bottle (whether it is carbonic acid), the height of the user, the size of the tire mounted on the vehicle, and the like are assumed. In particular, in the “tracking mode”, the guide unit 144 may change the amount of liquid to be guided and the safety factor in the distance from the target object (car, person, wall, etc.) according to the acquired additional information. it can. Specifically, the guide part 144 can increase the safety factor when the type of liquid is carbonic acid compared to the case where the liquid type is not carbonic acid. Similarly, the guide part 144 can increase the safety factor as the height of the user increases, and can increase the safety factor as the size of the tire increases. In this way, the guide unit 144 can perform the guidance more accurately in consideration of the additional information given by the user. Note that the additional information is not limited to voice but can be given by various input means (key input, gesture, line of sight, etc.) realized by the input information acquisition unit 110.

  Furthermore, in the first and second modifications described above, the guide unit 144 can designate one or more real objects from a plurality of real objects specified by the movement of a part of the user's body, etc. You may acquire by. For example, in the example of FIG. 13, it is assumed that the real objects OB31 and OB32 are surrounded by both hands. In this state, when the user utters “tree”, the guide unit 144 specifies the real object OB32 as the target object. In addition, in the second modification described above, the guide unit 144 uses voice to designate a real object that is a default position for displaying the virtual object VO31 (a virtual object that serves as a mark indicating the real object that the user designates as the object of interest). You may get it. For example, in the example of FIG. 14, when the user utters “the cloud on the right side”, the guide unit 144 displays the virtual image VI in which the virtual object VO31 is arranged at a position superimposed on the real object OB33. In this way, since the object of interest can be designated by the body movement and the voice, the convenience for the user can be further improved.

  According to the third modification, the guide unit 144 can execute a guidance process in response to a voice instruction. For this reason, since the user can perform an operation input even during work in which it is difficult to release his / her hands, convenience for the user can be improved.

A-4-4. Modification 4 (instruction by a small input device):
In the fourth modification, the object of interest and the guidance content can be specified using a small input device (hereinafter also simply referred to as “input device”) provided separately from the HMD 100.

  FIG. 16 is a diagram illustrating an example of the input device. FIG. 17 is a block diagram functionally showing the configuration of the input device. The modified input device 300 has a clock shape and includes an input surface 310, a microphone 320, a ROM 330, a RAM 340, a CPU 350, a storage unit 360, and a communication interface 370. The input surface 310 is a touch panel that combines a display device such as a liquid crystal panel and a position input device such as a touch pad, and detects information on a position touched by the user. The input surface 310 is arranged over the entire upper surface of the case of the input device 300. The microphone 320 is built in the case of the input device 300. The storage unit 360 includes a ROM, a RAM, a DRAM, a hard disk, and the like.

The CPU 350 functions as the control unit 351 by reading and executing the computer program stored in the storage unit 360. The control unit 351 realizes the functions shown in the following d1 to d3.
(D1) The control unit 351 performs pairing with the HMD 100. The control unit 351 stores the information of the paired HMD 100 in the storage unit 360, and thereafter blocks the execution of the function a2 and the execution of input processing with the other HMD 100.
(D2) The control unit 351 displays the status of the paired HMD 100 on the input screen 310. The status includes, for example, the presence / absence of an incoming mail, the presence / absence of a telephone call, the remaining battery level, the state of an application running on the HMD 100, and the like.
(D3) The control unit 351 transmits the content of the input operation performed on the input surface 310 to the HMD 100 via the communication interface 370. In addition, the control unit 351 transmits the content of the input operation acquired by recognizing the voice acquired by the microphone 320 to the HMD 100 via the communication interface 3370.

  In the configuration of the modification 4, the guide information 122a shown in FIG. 15 is provided in place of the guide information 122 shown in FIG. In Modification 4, as in Modification 3, the object of interest is specified in Step S104, the type of guidance content is specified in Steps S106 to S114, the additional information regarding the guidance content is specified in Steps S106 to S114, and the real object in Modifications 1 and 2 is specified. Can be acquired from the input device 300.

  According to the modification 4, the guide unit 144 can execute the guidance process in response to an instruction using the input device 300. For this reason, since the user can operate HMD100 using devices other than HMD100, the convenience in a user can be improved.

  FIG. 18 is a diagram illustrating another example of the input device. An input device 300a illustrated in FIG. 18 includes a cross key 311 and an enter button 312 instead of the input screen 310. Even with such a configuration, the same effect as that of the input device 300 shown in FIG. 16 can be obtained.

A-4-5. Modification 5 (instruction by virtual small input device):
In the fifth modification, the object of interest and the guidance content can be specified using a virtual small-sized input device (hereinafter simply referred to as “virtual input device”) realized by the HMD 100.

  FIG. 19 is a diagram illustrating an example of a virtual input device. The virtual input device 300b is realized by superimposing and displaying the virtual image VI by the HMD 100 on the surface of a normal clock (which may be an analog clock or a digital clock). Specifically, the augmented reality processing unit 142 of the HMD 100 causes the virtual image VI to be superimposed and displayed on the board surface when the user views the board surface of the virtual input device 300b. The augmented reality processing unit 142 can determine whether or not the user has seen the board surface of the virtual input device 300b by recognizing the outside scene image acquired by the camera 61.

  The virtual image VI includes a menu button B1, a forward button B2, a home button B3, a return button B4, and a confirmation button B5. The menu button B1 is a button for displaying the menu screen of the HMD 100 as a virtual image VI. The forward button B2 is a button for proceeding to the next screen or item currently displayed or selected. The home button B3 is a button for returning the currently displayed screen and items to the initial state. The return button B4 is a button for returning to the previous screen and item currently displayed or selected. The confirmation button B5 is a button for confirming the selection contents. These buttons B1 to B5 function as a “virtual operation unit”. The input information acquisition unit 110 detects the movement of a part of the user's body (for example, a finger, a pointer, and the like) using an outside scene image photographed with time (repetitively) by the camera 61. The input information acquisition unit 110 can acquire the type of button operated by the user by comparing the detected movement of a part of the body of the user with the positions of the buttons B1 to B5. it can. The collation can be performed by, for example, calculating which button region in the virtual operation unit is (or close to) the position (coordinates) of the user's fingertip in the outside scene image.

  In the configuration of the modification 5, the guide information 122a shown in FIG. 15 is provided instead of the guide information 122 shown in FIG. In modification 5, as in modification 3, the object of interest in step S104 is designated, the type of guidance content in steps S106 to S114 is designated, additional information relating to the guidance content in steps S106 to S114 is designated, and the real object in transformations 1 and 2 is designated. Can be acquired from the virtual input device 300b.

  According to the fifth modification, the guide unit 144 can execute a guide process in response to an instruction using the virtual input device 300b. For this reason, since the user can operate HMD100 using virtual devices other than HMD100, the convenience for a user can be improved.

A-4-6. Modification 6 (acquisition of additional information):
In the modification 6, in the configuration using the “additional information” described in the modification 3, the additional information is automatically acquired regardless of the input from the user. Specifically, the guide unit 144 may estimate the size and capacity of the object of interest by combining a distance measuring sensor mounted on the HMD 100 and image recognition of an outside scene image acquired by the camera 61. In addition, the guide unit 144 may estimate a temperature change for each part of the object of interest by combining a temperature sensor mounted on the HMD 100 and image recognition of an outside scene image acquired by the camera 61.

  According to the modification 6, since the guide part 144 can acquire additional information automatically, it can implement guidance more correctly.

B. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of HMD. However, the configuration of the HMD can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted.
(I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. Aspect to be mounted (iii) A mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer)
(Iv) A mode in which a smartphone or a portable game machine is used instead of the control unit. (V) The control unit and the image display unit are connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth. The aspect which abolished the connection part (code). In this case, power supply to the control unit or the image display unit may be performed wirelessly.

  For example, the configuration of the control unit exemplified in the above embodiment can be arbitrarily changed. Specifically, for example, both the transmission unit (Tx) of the control unit and the reception unit (Rx) of the image display unit have a function capable of bidirectional communication, and may function as a transmission / reception unit. For example, a part of an operation interface (such as various keys and a track pad) provided in the control unit may be omitted. For example, the control unit may be provided with another operation interface such as an operation stick. For example, the control unit may be configured such that a device such as a keyboard or a mouse can be connected, and an input may be received from the keyboard or the mouse. For example, each processing unit (for example, an image processing unit, a display control unit, an augmented reality processing unit, or the like) included in the control unit is an ASIC (Application Specific Integrated Circuit) designed to realize the function. (Integrated circuit). For example, the control unit is mounted with a secondary battery as a power source, but the power source is not limited to a secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, the configuration of the image display unit exemplified in the above embodiment can be arbitrarily changed. Specifically, for example, the image light generation unit of the image display unit may be combined with the above-described configuration unit (backlight, backlight control unit, LCD, LCD control unit) or in place of the above-described configuration unit. You may provide the structure part for implement | achieving a system. For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. For example, the image generation unit may include a digital micromirror device or the like instead of the LCD. For example, the present invention can be applied to a laser retinal projection type head-mounted display device. For example, a bone conduction speaker may be provided instead of the earphone or together with the earphone. According to sound reproduction using a bone conduction speaker, sound can be clearly transmitted to the user even in a noisy environment (for example, in a factory).

  FIG. 20 is an explanatory diagram showing the configuration of the appearance of the HMD in the modification. An image display unit 20x in FIG. 20A includes a right optical image display unit 26x and a left optical image display unit 28x. These are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely above the left and right eyes of the user when the HMD is mounted. The image display unit 20y shown in FIG. 20B includes a right optical image display unit 26y and a left optical image display unit 28y. These are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely below the left and right eyes of the user when the HMD is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is arbitrary, and the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit completely covers the user's eyes. There may be no aspect.

  For example, the HMD is a binocular transmissive HMD, but may be a monocular HMD. For example, it may be configured as a non-transparent HMD that blocks the transmission of outside scenes when the user wears the HMD, or may be configured as a video see-through in which a camera is mounted on the non-transparent HMD. For example, the earphone may be an ear-hook type or a headband type, or may be omitted. For example, instead of the image display unit worn like glasses, a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc.) may be employed. Also in this case, the connection between the control unit and the image display unit may be either wired or wireless. In this way, the control unit can be used as a remote controller for a normal flat display device. For example, instead of an image display unit worn like glasses, an image display unit of another aspect such as an image display unit worn like a hat or an image display unit built in a body protective device such as a helmet is used. It may be adopted. For example, you may comprise as a head-up display (HUD, Head-Up Display) mounted in vehicles, such as a motor vehicle and an airplane, or another transportation means.

Modification 2
In the above embodiment, an example of the guidance process has been described. However, the processing procedure shown in the above embodiment is merely an example, and various modifications are possible. For example, some steps may be omitted, and other steps may be added. The order of the steps performed may be changed.

  For example, the guide unit may omit the detection of the line-of-sight direction (step S108). In this case, the guidance unit may perform guidance for the object of interest only by voice regardless of the direction of the user's line of sight (step S112).

  For example, the guidance unit obtains information provided by DSSS (Driving Safety Support Systems) instead of obtaining the guidance content (step S106) or together with obtaining the guidance content, and performs guidance in step S112 or step S114. May be implemented. In addition, the guide unit may acquire information provided by “support system for vehicles, pedestrians, etc.” other than DSSS, and may perform the guidance in step S112 or step S114.

  For example, the guidance unit may perform only one of guidance in the tracking mode and guidance in the non-tracking mode in the guidance processing (steps S112 and S114). In this case, the guidance mode of the storage unit may be omitted. In addition, for example, the guide unit may perform guidance in the mode without tracking in principle, and may perform guidance in the mode with tracking only when a predetermined condition is satisfied. The predetermined condition is, for example, whether or not a change with time of the object of interest is greater than or equal to a predetermined amount (the detailed determination method is the same as the method a1 described above), and whether or not the feature of the object of interest matches the predetermined feature (The detailed determination method is the same as the method a2 described above), whether the user's line of sight is directed to the object of interest (the detailed determination method is the same as the method a3 described above), and whether the user is specified for the object of interest. (The detailed determination method is the same as the method a4 described above).

  For example, the guidance unit may shorten the content of the guidance voice when performing guidance using a predetermined number or more of guidance voices in the guidance processing (steps S112 and S114). Specifically, in the example of FIG. 10, the guidance unit may shorten the guidance voice “Please pay attention to the rear-end collision” like “Warning rear-end collision”. The guidance voice may be shortened as “forward attention”. The predetermined number may be two or more, may be three or more, and may be settable by the user. In this way, the time required for guidance using a plurality of guidance voices can be shortened.

  For example, when performing guidance using a plurality of guidance voices in the guidance processing (steps S112 and S114), the guidance unit changes the order in which guidance is performed according to the importance of the object of interest corresponding to the guidance voice. You may decide. In determining the importance of the attention object, the above-described methods a1 to a4 are used. For example, the attention object identified by the method a4 has the highest priority, and the attention object identified by the method a1 has the priority 2. Number, etc. Also, the user may be able to change the correspondence between the method of identifying the object of interest and the priority order. In this way, it is possible to first guide the guidance voice for the attention object with a high priority, so that convenience can be improved.

  For example, in the guidance process (steps S112 and S114), the guidance unit determines whether the speaker in the right earphone or the speaker in the left earphone is in accordance with the positional relationship between the object of interest and the user. A guidance voice may be output. Specifically, for example, when the attention object is located on the right side of the user, the guidance unit outputs guidance voice from the speaker in the right earphone, and when the attention object is located on the left side of the user. The guidance voice may be output from a speaker in the left earphone. For example, when the head-mounted display device includes a speaker having a wide frequency response characteristic and directivity, an audio processing system such as Dolby (registered trademark) can be used. In this case, the guidance unit can generate voice data that reproduces the guidance voice from the position where the target object exists, and can supply the voice data to the voice processing system for output. In this way, the user who has heard the guidance voice can intuitively know the positional relationship between the object and the object of interest.

  For example, in the guidance process (steps S112 and S114), the guidance unit may reproduce a warning sound such as a buzzer, a warning sound such as an alarm, a melody, or the like instead of the guidance voice.

  For example, in the guidance process (steps S112 and S114), the guidance unit may perform guidance for the object of interest by vibration instead of sound (sound wave) or together with sound. In this case, for example, the head-mounted display device includes a vibration unit with a built-in vibration motor or the like, and the guide unit can drive the vibration motor instead of reproducing the guidance voice or together with the reproduction of the voice. Good. In this case as well, the guide unit may select one of, for example, a vibration motor built in the right holding unit and a vibration motor built in the left holding unit, depending on the positional relationship between the object of interest and the user. Either of them may be driven.

  For example, the guidance unit may add parallax to the virtual object displayed in the guidance process (step S114).

  For example, the following guidance processing may be performed together with the guidance processing described above. The head-mounted display device includes a distance acquisition unit that acquires a distance between the user and the object of interest. The guidance unit reproduces the guidance voice when the distance between the user and the object of interest is within a predetermined distance (for example, 20 m). The guidance voice may be a warning sound, a warning sound, or a melody. The guide unit may change the predetermined distance according to the moving speed of the user (for example, 20 m when the speed is 50 km / h or more). The distance acquisition unit may acquire the distance by a stereo camera method using the stereo camera shown in FIG. 2, for example, or acquire the distance by a monocular camera method using the movement distance between the monocular camera and the monocular camera. May be. The distance acquisition unit may acquire the distance by a pattern irradiation method using an infrared (or ultrasonic) light receiving / emitting unit, or by a ToF (Time of Flight) method using an infrared (or ultrasonic) light receiving / emitting unit. The distance may be acquired.

  For example, the following adjustment process may be performed as a pre-process of the above-described guidance process or as a pre-process of step S114 of the guidance process. The head-mounted display device includes an IMU (Inertial Measurement Unit). Using the IMU, the CPU associates an area in the user's visual field direction (for example, an imaging area of a camera) with a virtual image display area in the image display unit. In this way, the augmented reality processing unit can improve the accuracy of alignment between the virtual object displayed in step S114 of the guidance process and the real object (target object). In addition, it is possible to suppress the display blur of the virtual object due to the fine shake of the user's head. Also, when performing mask processing of unnecessary images in augmented reality processing, the speed of mask processing can be improved.

  For example, in an environment where there are a plurality of head-mounted display device users, the guidance process may be performed using the relative positional relationship of each user. Specifically, for example, in specifying the object of interest (step S104), the guide unit of the HMD 100x also uses other HMDs 100y, 100z, etc., having a predetermined relative positional relationship with the user of the HMD 100x, as the object of attention. Good. For example, in the guidance (steps S112 and S114), the guidance unit of the HMD 100x is connected to the speaker in the right earphone and the left earphone in accordance with the relative positional relationship between the user of the HMD 100x and other users of the HMDs 100y and 100z. The guidance voice may be output from any one of the speakers, and the position where the virtual object is displayed may be adjusted. In this way, guidance based on the relative positional relationship of each user can be realized in an environment where there are users of a plurality of head-mounted display devices (for example, in factories, schools, etc.). .

  For example, in the above-described modification 1 of the guidance process (designation of an object of interest by movement of a part of the body, etc.), the guide unit further enlarges the virtual object attached to the object of attention by movement of a part of the user's body, etc. Further, reduction, rotation, adjustment of the arrangement position, and the like may be performed.

・ Modification 3:
In the above embodiment, an example of guidance information has been shown. However, the configuration and content of the guidance information shown in the above embodiment is merely an example, and various modifications can be made. For example, a part of the configuration may be omitted or another configuration may be added. The data stored in the guide information may be changed.

  For example, there may be one “object” field, which is information for collation using the name of the object of interest, or may be three or more.

  For example, procedure information may be stored instead of or together with the guidance content. “Procedure information” is information used to teach a series of operation procedures to an HMD user. The operation procedures widely include various categories of operation procedures. For example, the procedure of the producer's operation in cooking and sewing, the procedure of the worker's operation in factory work, the procedure of the athlete's operation in sports such as tennis are included. Further, the operation procedure includes a procedure of thinking operation that does not involve the movement of the user's body, such as a procedure of thinking such as a thought process and a procedure of thinking in the learning process. The procedure of these operations may be continuous or discontinuous at regular intervals.

-Modification 4:
In the above embodiment, a specific example of the guidance process has been shown. However, the specific examples shown in the above embodiment are merely examples, and various modifications are possible. For example, the voice guidance mode exemplified and the virtual object mode in augmented reality processing can be arbitrarily changed.

  For example, in the first and second specific examples, the real object is described as one “thing” unit. However, the guide unit may perform the guide process on the assumption that there are a plurality of real objects in one “thing”. Specifically, the guidance process may be performed on the assumption that there are a plurality of real objects such as a body, a bumper, a tire, and a tail lamp in one “car”.

  For example, in the first specific example, the guide unit may display a virtual object representing a scale for guiding the liquid amount to the user in the vicinity of the glass. Similarly, the guiding unit may display a virtual object representing a mark for guiding a position where the liquid injection should be stopped at an arbitrary point on the glass. Similarly, the guide unit may display a scale for guiding weight to the user in the vicinity (preferably the lower part) of the glass. The guide unit may change the value of the scale according to the liquid amount estimated by the image recognition.

  For example, in the second specific example, the guide unit may change the aspect of the virtual object for emphasizing the car according to the distance from the car ahead. Specifically, the guide unit can make the virtual object more conspicuous (larger and darker) as the distance from the vehicle ahead is shorter.

-Modification 5:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

    DESCRIPTION OF SYMBOLS 10 ... Control part, 11 ... Decision key, 12 ... Lighting part, 13 ... Display switch key, 14 ... Trackpad, 15 ... Luminance switch key, 16 ... Direction key, 17 ... Menu key, 18 ... Power switch, 20 ... Image Display unit, 21 ... right holding unit, 22 ... right display driving unit, 23 ... left holding unit, 24 ... left display driving unit, 26 ... right optical image display unit, 28 ... left optical image display unit, 30 ... earphone plug, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connector, 42 ... Right cord, 44 ... Left cord, 46 ... Connector, 48 ... Body cord, 51 ... Transmitter, 53 ... Receiver, 61 ... Camera, 62 ... gaze detection unit, 63 ... microphone, 66 ... 9-axis sensor, 100 ... head mounted display (head mounted display device), 110 ... input information acquisition unit, 120 ... storage unit, 122, 122a ... information for guidance, 124 … Draft Mode: 130: Power source 132: Wireless communication unit 134: GPS module 140: CPU 142: Augmented reality processing unit 144: Guide unit 150: OS 160: Image processing unit 170: Audio processing unit 180 ... interface, 190 ... display control unit, 201 ... right backlight control unit, 202 ... left backlight control unit, 211 ... right LCD control unit, 212 ... left LCD control unit, 221 ... right backlight, 222 ... left backlight Light, 241 ... Right LCD, 242 ... Left LCD, 251 ... Right projection optical system, 252 ... Left projection optical system, 261 ... Right light guide plate, 262 ... Left light guide plate, 300, 300a ... Input device, 300b ... Virtual input device , 310 ... input surface, 311 ... cross key, 312 ... decision button, 320 ... microphone, 330 ... ROM, 340 ... RAM, 350 ... CP 351: control unit, 360: storage unit, 370: communication interface, VSync ... vertical synchronization signal, HSync ... horizontal synchronization signal, Data ... image data, OA ... external device, PC ... personal computer, VI ... virtual image, SC ... outside scene , VR ... field of view, RE ... right eye, LE ... left eye, IM1, IM2 ... outside image, OB11-OB26 ... real object, VO11-VO23 ... virtual object, SO11-SO23 ... audio

Claims (16)

A head-mounted display device that allows a user to visually recognize a virtual image,
An image display unit for allowing the user to visually recognize the virtual image;
An image acquisition unit for acquiring an outside scene image in the visual field direction of the user in a state in which the head-mounted display device is mounted;
A guidance unit that uses sound to guide a target object that is present in the real world and that is at least one real object included in the acquired outside scene image and is a real object that prompts the user to pay attention; ,
A head-mounted display device comprising: The head-mounted display device according to claim 1, further comprising:
A line-of-sight detection unit that detects the direction of the line of sight of the user in a state in which the head-mounted display device is mounted;
The guide part is
When it is estimated that the object of interest is not in the direction of the detected line of sight, the guidance is performed with the sound,
When the object of interest is estimated to be in the direction of the detected line of sight, in addition to the sound, the image display unit further includes the virtual image including a virtual object for additionally displaying the object of interest. A head-mounted display device that performs the guidance by forming the head-mounted display device. The head-mounted display device according to claim 1 or 2,
The guidance is
Guidance for alerting the user to the object of interest;
Information guidance on the object of interest;
Guidance on the position of the object of interest;
Based on procedure information prepared in advance, guidance of the user's work content for the object of interest;
A head-mounted display device including at least one of the above. The head-mounted display device according to claim 3,
The guide part is
A head-mounted display device that determines which of the alert, the information, the position, and the work content is included as the guidance according to an operation input from the user. The head-mounted display device according to any one of claims 1 to 4,
The image acquisition unit repeatedly acquires the outside scene image,
The guide unit further includes:
Recognizing each of the plurality of acquired outside scene images,
A head-mounted display device that uses a real object that is included in the plurality of outside scene images and has a large change over time as the object of interest. The head-mounted display device according to any one of claims 1 to 4,
The guide unit further includes:
Image recognition of the acquired outside scene image,
A head-mounted display device that uses a real object included in the outside scene image and having a predetermined characteristic as the attention object. The head-mounted display device according to any one of claims 1 to 4, further comprising:
A line-of-sight detection unit that detects the direction of the line of sight of the user in a state in which the head-mounted display device is mounted;
The image acquisition unit repeatedly acquires the outside scene image,
The line-of-sight detection unit repeatedly detects the direction of the line of sight,
The guide unit further includes:
The plurality of acquired outside scene images and the direction of the line of sight detected multiple times are collated,
A head-mounted display device in which a real object that is included in the plurality of outside scene images and is estimated to have a high proportion of the line of sight is the attention object. The head-mounted display device according to any one of claims 1 to 4, further comprising:
An input information acquisition unit for acquiring an operation input from the user;
The guide unit further includes:
The acquired outside scene image is collated with the acquired operation input,
A head-mounted display device in which a real object included in the outside scene image and designated by the user is the attention object. The head-mounted display device according to claim 8,
The image acquisition unit repeatedly acquires the outside scene image,
The input information acquisition unit acquires a motion of a part of the user's body by recognizing the external scene image acquired repeatedly,
The guide part is
Collating the acquired outside scene image with the movement of the body part acquired as the operation input;
A head-mounted display device that uses a real object that is included in the outside scene image and that is intended by the user based on the movement of a part of the body as the attention object. The head-mounted display device according to claim 9,
The guide unit further includes:
A virtual object that serves as a mark when the user designates the object of interest, and at least one of a position, a size, and a shape so as to follow the movement of a part of the body of the user A head-mounted display device that causes the image display unit to form the virtual image including a virtual object whose shape has been changed over time. A head-mounted display device according to any one of claims 4 or 8 to 10,
The input information acquisition unit
A head-mounted display device that acquires the operation input from another device connected to the head-mounted display device. A head-mounted display device according to any one of claims 4 or 8 to 10,
The image display unit forms the virtual image representing a virtual operation unit for operating the head-mounted display device;
The image acquisition unit repeatedly acquires the outside scene image,
The input information acquisition unit
The operation input is acquired by collating the movement of a part of the user's body acquired by recognizing the repeatedly acquired outside scene image with the position of each unit in the virtual operation unit. Head mounted display device. The head-mounted display device according to any one of claims 1 to 12,
The image acquisition unit repeatedly acquires the outside scene image,
The guide part is
A head-mounted display device that repeatedly performs the guidance according to a change with time of the object of interest in the plurality of acquired outside scene images. The head-mounted display device according to any one of claims 1 to 13,
The guide unit is a head-mounted display device that further acquires additional information used for the guidance. A method for controlling a head-mounted display device,
(A) acquiring an outside scene image in the user's field of view in a state where the head-mounted display device is mounted;
(B) Guidance is given by sound to at least one real object that exists in the real world and is an actual object that is included in the acquired outside scene image and that prompts the user to pay attention. Process,
A method comprising: A computer program,
A function of acquiring an outside scene image in the direction of the user's field of view when the head-mounted display device is mounted;
A function for performing sound guidance on an object of interest that is an actual object that exists in the real world and that is at least one real object included in the acquired outside scene image and that attracts attention to the user;
A computer program that enables a computer to realize

Images