JP2019053714A - Head mounted display device and control method for head mounted display device - Google Patents

Abstract

To provide a user interface with high usability enabling a state of a display device to be recognized in an operation using an indication body.SOLUTION: An HMD 100 includes: an image display unit 20 that is mounted on a head of a user and has a display region VR configured to display an image while an outside scene is visible; a first camera 61; a detecting unit 154 configured to detect a movement of the user's fingers based on an image captured by the first camera 61 and to receive an operation based on the detected movement of the fingers; and a display control unit 155 configured to determine whether or not the detecting unit 154 is ready to detect the operation based on the movement of the fingers, and to display the determination result in the display region VR.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a head-mounted display device and a method for controlling the head-mounted display device.

  2. Description of the Related Art Conventionally, head-mounted display devices are capable of various operations such as inputting characters and figures using a pointing device, and selecting icons and windows by touching the screen. For this reason, a display device that displays auxiliary information that assists the user's operation is known (see, for example, Patent Document 1). The terminal disclosed in Patent Document 1 displays an operation guide image indicating auxiliary information for using an application in a superimposed manner on an image being displayed.

JP 2017-16370 A

In the case of an operation using a pointer, the detection unit detects the movement, shape, position, and the like of the pointer to determine the input operation. Therefore, for example, when the detection unit does not detect the indicator, when the state of the display device is busy, or when the operation is performed outside the range in which the detection unit can detect the operation, Even if an operation is performed using an indicator, the operation may not be input to the display device. For this reason, in an operation using an indicator, a user-friendly user interface that can grasp the state of the head-mounted display device is required.
An object of the present invention is to provide an easy-to-use user interface capable of grasping the state of a head-mounted display device in an operation using an indicator.

In order to solve the above problems, the present invention is based on a display unit that is mounted on a user's head and has a display area that displays an image so that an outside scene can be viewed, an imaging unit, and a captured image of the imaging unit. Detecting a movement of the indicator and receiving a manipulation based on the detected movement of the pointer; and determining whether the detection section is in a state in which an operation based on the movement of the pointer can be detected. And a display control unit that displays the determination result in the display area.
According to this configuration, the determination result indicating whether or not the detection unit can detect an operation based on the movement of the indicator is displayed in the display area. For this reason, in the operation using the indicator, it is possible to grasp whether or not the head-mounted display device is in a state where the operation can be detected, and a user interface that is easy to use can be provided.

Further, according to the present invention, the display control unit may include, as the determination result, a figure, a character, an icon indicating that the detection unit can detect the operation, or the indicator detected by the detection unit. A range image indicating the range is displayed in the display area.
According to this configuration, since a figure, a character, an icon, and a range image are displayed as the determination result, it is possible to notify the outside whether or not the operation can be detected.

Further, according to the present invention, when the display unit detects the indicator, the display control unit displays the range image in the display area, and the detection unit is in a state in which the operation can be detected. The display mode of the figure, the character, or the icon displayed in the display area is changed depending on whether or not.
According to this configuration, the range image is displayed when the indicator is detected, and the display mode of the graphic, the character, or the icon is changed depending on whether or not the detection unit can detect the operation. . For this reason, it is possible to notify the outside that the detection unit has detected the indicator, and it is possible to notify the outside whether or not the detection unit is in a state in which an operation can be detected.

Further, in the present invention, the display control unit is configured such that when the detection unit is in a state in which the operation can be detected, the graphic or the icon indicates the movement of the indicator detected by the detection unit. Display icon.
According to this structure, since the figure or icon which shows the motion of the said indicator which a detection part detects is displayed, a user can be made to recognize operation using the indicator.

In addition, the present invention includes a processing control unit that controls a control target in accordance with the operation received by the detection unit, and the display control unit displays a preset movement of the indicator detected by the detection unit. The figure or the icon to be shown is changed according to the operation state of the control target.
According to this configuration, the user can cause the indicator to move according to the operation state of the controlled object, and the controlled object can be operated by the movement of the indicator. For this reason, the convenience of operation using a pointer can be improved.

In the present invention, the detection unit determines whether the detected movement of the indicator corresponds to a preset movement of the indicator, and the processing control unit determines whether the instruction is preset. When the detection unit detects the movement of the indicator corresponding to the movement of the body, the control target is caused to execute processing associated with the movement of the indicator detected by the detection unit.
According to this configuration, it is possible to cause the control target to execute processing associated with the movement of the indicator by performing the movement of the indicator set in advance.

In the present invention, the display control unit displays the range image superimposed on the indicator when the indicator is detected by the detector.
According to this configuration, the range image can be displayed superimposed on the indicator. For this reason, the object recognized as the indicator by the head-mounted display device can be notified to the outside.

Further, according to the present invention, even if a first preset time elapses after the display control unit displays the graphic, the character, or the icon in the display area, the detection unit performs the operation. Is not received, an operation guide for guiding the operation by the indicator is displayed in the display area.
According to this configuration, an operation guide can be displayed to guide the user to perform operations using the indicator.

Further, according to the present invention, the display control unit displays a remaining time until the operation guide is displayed in the display area in the display area.
According to this configuration, a user who does not know the operation can be notified of the time for which the operation guide is displayed.

In the present invention, it is preferable that the display control unit detects a movement of the indicator when the preset second set time has elapsed after the operation guide is displayed in the display area. When an operation based on the operation guide is detected, the display of the operation guide is deleted.
According to this configuration, when it is determined that there is no need to display the operation guide, the display of the operation guide can be erased, and the visibility of the display area can be improved.

Further, according to the present invention, the processing control unit controls the device as the control target by the operation accepted by the detection unit, or executes the application program as the control target and performs the operation accepted by the detection unit. The function of the corresponding application program is executed.
According to this configuration, the device can be controlled by operating the indicator, and a function can be selected and executed by an application program as a control target.

In order to solve the above-described problem, the present invention is a method for controlling a head-mounted display device that is mounted on a user's head and displays an image in a display area so that an outside scene can be visually recognized. The step of detecting the movement of the indicator based on the above, the step of accepting an operation based on the detected movement of the indicator, and the step of accepting the operation are in a state where the operation based on the movement of the indicator can be detected. Determining whether or not there is, and displaying the determination result in the display area.
According to this configuration, the determination result as to whether or not the detection unit can detect an operation based on the movement of the indicator is displayed in the display area. Therefore, it is possible to provide an easy-to-use user interface that can grasp the state of the head-mounted display device in the operation using the indicator.

In order to solve the above problem, the display device displays an image superimposed on an outside scene, or displays the image around the outside scene, a detection unit that detects a pointer, and a detection unit that detects the display unit. A control unit that receives an operation based on at least one of the movement, shape, and position of the indicator, and the control unit determines whether or not an input operation by the indicator is possible. Display the judgment result.
According to this configuration, it is determined whether or not the input operation by the indicator is possible, and the determination result is displayed. For this reason, the user of a display apparatus can grasp | ascertain the state of a display apparatus and the usability of operation using a pointer can be improved.

In the display device, the control unit causes the display unit to display an image, a graphic, a character, or an icon indicating that the operation can be received at a position set in the display area of the display unit.
According to this configuration, an image, graphic, character, or icon indicating that an operation can be accepted is displayed. Therefore, the user can intuitively recognize that the display device is in a state where it can accept an operation.

Further, in the display device, the control unit is configured to be able to control a control target according to the operation, and controls an image, a figure, a character, or an icon indicating that the operation can be accepted according to the operation. It changes according to the state of the controlled object.
According to this configuration, the image, figure, character, or icon indicating that the operation can be accepted changes depending on the state of the control target. For this reason, the user can intuitively recognize the state of the control target.

In the display device, the detection unit detects a range including the outside scene that is visible together with the image displayed by the display unit, and the control unit detects the indicator within the range by the detection unit. When it is done, the display unit displays a message indicating that the operation can be accepted.
According to this configuration, since the detection unit detects the indicator for the range including the viewable outside scene together with the image displayed on the display unit, the user can set the indicator within the range including the viewable outside scene. By moving the position, the display device can be in a state in which an operation can be accepted. Further, when an indicator is detected within the range, a display indicating that the operation can be accepted is displayed, so that the state of the display device can be grasped.

In the display device, the detection unit includes an imaging unit that captures a range including the outside scene that is visible together with the image displayed by the display unit, and the control unit receives the instruction from the captured image of the imaging unit. The operation can be received by detecting a body, and when the indicator is detected from a captured image of the imaging unit, the display unit displays that the operation can be received.
According to this configuration, the detection unit includes the imaging unit that images the range including the outside scene that can be visually recognized together with the image displayed by the display unit and detects the indicator. For this reason, the user can make the display device a state in which an operation can be accepted by moving the position of the indicator within a range including a visually recognizable outside scene. Further, when an indicator is detected within the range, a display indicating that the operation can be accepted is displayed, so that the state of the display device can be grasped.

Further, in the display device, the display unit has a transmissive display region that allows the outside scene to be visually recognized by transmitting outside light, and the detection unit displays the outside scene that is visible through the display unit. The indicator is detected within the range.
According to this configuration, the detection unit detects the indicator within a range including the outside scene that is visible through the display unit, and thus the user moves the position of the indicator within the range including the outside scene that is visible. By doing so, the display device can be set in a state in which an operation can be accepted.

In the display device, the control unit displays an image superimposed on the indicator detected by the detection unit when the indicator is detected by the detection unit.
According to this configuration, since the image superimposed on the indicator detected by the detection unit is displayed, it is possible to notify the user of the position of the detected indicator. Accordingly, when an object other than the indicator is detected as the indicator, the user can prevent the erroneous operation from being detected by causing the display device to detect the indicator again.

Further, in the display device, the control unit performs the operation when the indicator is detected by the detection unit and the operation is acceptable, when the indicator is not detected by the detection unit, and when the operation is performed. The display mode of the display indicating that the operation can be accepted is changed depending on whether the operation is not acceptable.
According to this configuration, the display mode of the display indicating that the operation can be accepted is changed depending on whether the indicator is detected and whether the operation is acceptable. For this reason, it becomes easy to grasp the state of the display device.

In the display device, the control unit detects a gesture as the input operation based on at least one of the movement, shape, and position of the indicator detected by the detection unit, and displays the gesture by the display unit. Any one of an image, a figure, a character, or an icon indicating that the operation can be accepted, and the display mode of any of the image, the figure, the character, or the icon associated with the detected input operation is changed.
According to this configuration, the display device can be operated by a gesture using an indicator. In addition, since the display mode of any of the images, graphics, characters, or icons associated with the gesture detected by the display device changes, the detected gesture can be notified to the user.

Further, in the display device, the control unit starts the display indicating that the operation can be accepted, and when the operation by the indicator is not detected within a preset first set time, An image for guiding the operation is displayed on the display unit.
According to this configuration, the operation of the indicator can be guided by the image that guides the operation.

Further, in the display device, when the second setting time set in advance has elapsed after displaying the image for guiding the operation, or when the operation by the indicator is detected, The display of the image for guiding the operation is terminated.
According to this configuration, when it is determined that the display of the image for guiding the operation is unnecessary, the display of the image for guiding the operation can be terminated to prevent the visibility of the outside scene from being deteriorated.

In the display device, the control unit controls a device as a control target according to the operation, or executes an application program to execute processing according to the operation.
According to this configuration, it is possible to control a device as a control target by operating a pointer or to execute a process corresponding to the operation by executing an application program.

The display device control method is a display device control method including a display unit that displays an image superimposed on an outside scene, or displays the image around the outside scene. An operation based on at least one of the movement, shape, and position of the indicator is accepted, it is determined whether or not the input operation by the indicator is possible, and the determination result is displayed.
According to this configuration, it is determined whether or not the input operation by the indicator is possible, and the determination result is displayed. For this reason, the user of a display apparatus can grasp | ascertain the state of a display apparatus and the usability of operation using a pointer can be improved.

FIG. The principal part top view which shows the structure of the optical system of HMD. The perspective view which shows the structure of an image display part. The block diagram of HMD. The functional block diagram of a control apparatus. The figure which shows a user's visual field range. The flowchart which shows operation | movement of a control part. The figure which shows the display state of a display area. The figure which shows the display state of a display area. The figure which shows the display state of a display area. The figure which shows the other display state of a display area. The flowchart which shows the other operation | movement of a control part.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an external configuration of an HMD (Head Mounted Display) 100 to which the present invention is applied. The HMD 100 is a display device that includes the control device 10 and the image display unit 20. The image display unit 20 corresponds to the “display unit” of the present invention.

  The control device 10 includes an operation unit that receives a user's operation, and functions as a controller for the user to operate the HMD 100. The control device 10 receives a user operation and controls the image display unit 20 according to the received operation. The image display unit 20 is attached to the user's head and makes the user visually recognize a virtual image. The user means a person who wears the image display unit 20 on the head.

  The control device 10 includes a flat box-shaped case 10A (also referred to as a housing or a main body) as shown in FIG. The case 10 </ b> A includes the operation button 11, LED indicator 12, track pad 14, up / down key 15, changeover switch 16, and power switch 18. The operation button 11, the up / down key 15, the changeover switch 16, and the power switch 18 are collectively referred to as an operator 13 (FIG. 4). The user can operate the HMD 100 by operating the operator 13 and the track pad 14.

  The operation button 11 includes a menu key, a home key, a return key, and the like, and particularly includes a key or a switch that is displaced by a pressing operation. The LED indicator 12 lights up or blinks in accordance with the operating state of the HMD 100.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. Various detection methods, such as an electrostatic method, a pressure detection method, and an optical method, can be adopted as the detection method of the operation on the operation surface of the track pad 14.

  The up / down key 15 is used to input an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34 and to input an instruction to increase / decrease the display brightness of the image display unit 20. The changeover switch 16 is a switch for changing an input corresponding to the operation of the up / down key 15. The power switch 18 is a switch for switching on / off the power of the HMD 100, and is configured by, for example, a slide switch.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27.

  Each of the right holding unit 21 and the left holding unit 23 extends rearward from both ends of the front frame 27, and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is mounted. The left holding unit 23 is provided to extend from the end EL to a position corresponding to the left side of the user when the image display unit 20 is mounted.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is in a wearing state in which the user wears the image display unit 20. Corresponding to the eyebrows. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) that contacts the back of the user when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the person's head.

  The right display unit 22 realizes image display by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user in the mounted state. The left display unit 24 realizes image display by the left light guide plate 28. The left display unit 24 is provided in the left holding part 23 and is located in the vicinity of the left side of the user in the wearing state.

  The right light guide plate 26 and the left light guide plate 28 are optical units formed of a light-transmitting resin or the like, and are configured by prisms, for example. The right light guide plate 26 and the left light guide plate 28 guide the image light output from the right display unit 22 and the left display unit 24 to the eyes of the user.

  A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the front side of the front frame 27 which is the side opposite to the eye side of the user. By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 is a transmissive display device that displays an image so that an outside scene can be visually recognized by transmitting external light. The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively. The image light guided to the right light guide plate 26 and the left light guide plate 28 enters the user's right eye and left eye, and makes the user visually recognize a virtual image. Thereby, the image display unit 20 displays an image. An area in which the image display unit 20 can display an image in the visual field range FV of the user wearing the image display unit 20 on the head is referred to as a display area VR (FIGS. 6, 8 to 10 and 12).

  When the external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image are incident on the user's eyes. Therefore, the visibility of the virtual image is affected by the intensity of external light. For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the ease of visual recognition of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28, the adhesion of dirt, and the like. The light control plate may be configured to be detachable from the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28, or a plurality of types of light control plates may be replaced and mounted. . Moreover, it is good also as the image display part 20 of a structure which does not provide a light control board.

A first camera 61 and a second camera 62 are disposed on the front frame 27 of the image display unit 20. The configuration and arrangement of the first camera 61 and the second camera 62 are determined so as to capture an outside scene visually recognized by the user wearing the image display unit 20. The outside scene refers to an outside scene in the direction of the line of sight of the user wearing the image display unit 20 on the head. For example, the first camera 61 and the second camera 62 are provided on the front surface of the front frame 27 at positions that do not block external light that passes through the right light guide plate 26 and the left light guide plate 28. In the example shown in FIG. 1, the first camera 61 is disposed on the end ER side of the front frame 27 and is a part of the user's body or an object attached to the body as an indicator (hereinafter referred to as a detected object). ) To detect. The second camera 62 is disposed at the boundary between the right light guide plate 26 and the left light guide plate 28, that is, at the middle position of the left and right eyes of the user, and images the front of the HMD 100 that is the user's line-of-sight direction. . The arrangement of the second camera 62 is not limited to the example illustrated in FIG. 1. For example, the second camera 62 may be arranged so as to face the inside of the image display unit 20 and image the user's eyes. You may arrange | position so that the back may be imaged. Alternatively, the second camera 62 may be disposed on the end EL side of the front frame 27 and the first camera 61 and the second camera 62 may constitute a stereo camera.
The first camera 61 corresponds to the “imaging unit” of the present invention. The second camera 62 corresponds to a “control target” of the present invention.

  The first camera 61 and the second camera 62 are digital cameras that include an imaging element such as a CCD or CMOS, an imaging lens, and the like. In the present embodiment, the first camera 61 and the second camera 62 are monocular cameras, but may be configured with a stereo camera. The first camera 61 and the second camera 62 image at least a part of the outside scene in the front side direction of the HMD 100, in other words, in the user's viewing direction when the HMD 100 is worn. In another expression, the first camera 61 and the second camera 62 capture a range or direction that overlaps the user's field of view, and capture a direction in which the user gazes. The direction and width of the angle of view of the first camera 61 and the second camera 62 can be set as appropriate. In the present embodiment, as will be described later, the angle of view of the first camera 61 and the second camera 62 includes the outside world that the user visually recognizes through the right light guide plate 26 and the left light guide plate 28. More preferably, the angle of view of the first camera 61 and the second camera 62 is set so that the entire field of view of the user that can be seen through the right light guide plate 26 and the left light guide plate 28 can be imaged.

The first camera 61 and the second camera 62 execute imaging according to the control of the imaging control unit 153 (FIG. 5) provided in the control unit 150.
When the power switch 18 is turned on and the main processor 140 is activated upon receiving power supply from the power supply unit 130, the main processor 140 causes the power supply unit 130 to start supplying power to the first camera 61 and turns on the first camera 61. The main processor 140 supplies power to the second camera 62 when an operation of a preset indicator is detected from the captured image data of the first camera 61 or when an operation by the operation element 13 is received. And the power of the second camera 62 is turned on.

  The HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 in the front frame 27. In this case, when the image display unit 20 is mounted, the position of the distance sensor is substantially in the middle between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction. The measurement direction of the distance sensor can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping the imaging direction of the first camera 61. The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference in accordance with the control of the control unit 150. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 150.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light and a lens group that guides the image light L emitted from the OLED unit 221 as a configuration that allows the user's right eye RE to visually recognize an image. Right optical system 251. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by arranging, in a matrix, light emitting elements that emit light of R (red), G (green), and B (blue) by emitting light by organic electroluminescence. It is. The OLED panel 223 includes a plurality of pixels, each of which includes one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element under the control of the control unit 150 (FIG. 5), and causes the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be formed of a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
Note that the OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are stacked. Further, an OLED panel 223 having a WRGB configuration provided with a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be used.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE. The image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

  In addition, the left display unit 24 has a left optical system including an OLED unit 241 that emits image light and a lens group that guides the image light L emitted from the OLED unit 241 as a configuration that allows the user's left eye LE to visually recognize an image. 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED driving circuit 245 causes the light emitting elements of the OLED panel 243 to emit light by selecting a light emitting element included in the OLED panel 243 and energizing the light emitting element according to the control of the control unit 150 (FIG. 5). The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be configured by a semiconductor device that drives the OLED panel 243, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 includes a collimating lens that converts the image light L emitted from the OLED panel 243 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element formed with a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) located in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE. The image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

  According to this configuration, the HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. As described above, the HMD 100 causes the image light L of the image processed inside and the external light OL to overlap and enter the user's eyes. For the user, the outside scene can be seen through the right light guide plate 26 and the left light guide plate 28, and an image of the image light L is displayed so as to be visible over the outside scene or around the outside scene. The half mirrors 261 and 281 are image extraction units that extract image by reflecting image light output from the right display unit 22 and the left display unit 24, respectively, and can be called display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

Returning to FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector (not shown) provided at the lower part of the case 10 </ b> A, and is connected to various circuits provided in the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable that transmits digital data or an optical fiber cable, and may include a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40. The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the configuration example shown in FIG. 1, a headset 30 having a right earphone 32 and a left earphone 34 and a microphone 63 constituting a stereo headphone is connected to a connector 46.
The control device 10 and the image display unit 20 may be wirelessly connected. For example, the control device 10 and the image display unit 20 mutually transmit and receive control signals and data by wireless communication complying with standards such as Bluetooth (registered trademark) and wireless LAN (including Wi-Fi (registered trademark)). It is good also as a structure.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound interface 180 (FIG. 4). To do. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

FIG. 3 is a perspective view showing a configuration of the image display unit 20 and shows a main configuration of the image display unit 20 as viewed from the user's head side. FIG. 3 shows the side of the image display unit 20 that contacts the user's head, in other words, the side that can be seen by the user's right eye RE and left eye LE. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3, the half mirror 261 that irradiates the user's right eye RE with the image light and the half mirror 281 that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. For this reason, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees a rectangular display image at the positions of the half mirrors 261 and 281.

  The first camera 61 is disposed at the right end of the image display unit 20 and images the direction in which the user's eyes are facing, that is, the front for the user. The 2nd camera 62 is arrange | positioned in the boundary part with the right light-guide plate 26 and the left light-guide plate 28, and images the front for a user. The optical axes of the first camera 61 and the second camera 62 are directions including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when a user looks at an object located in front of the user with both eyes, the distance from the user to the object is often about 30 cm to 10 m, and is about 1 m to 4 m. There are more. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object during normal use may be determined. This standard may be obtained by investigation or experiment, or may be set by the user. The optical axes and angles of view of the first camera 61 and the second camera 62 are determined when the distance to the object during normal use corresponds to the set upper limit guide and the lower limit guide. It is preferable that the object is set to be included in the angle of view.

  In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. When the gazing point is an object located in front of the user, in the user's field of view, about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction are effective around the line of sight of each of the right eye RE and the left eye LE. It is a field of view. Further, the stable viewing field is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction, and the viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, the actual visual field that the user sees through the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV: Field Of View). In the configuration of the present embodiment shown in FIGS. 1 and 2, the actual visual field corresponds to an actual visual field that is transmitted through the right light guide plate 26 and the left light guide plate 28 and visually recognized by the user. The real field of view is narrower than the viewing angle and stable field of view, but wider than the effective field of view.

  The first camera 61 and the second camera 62 image a range including an outside scene that can be viewed together with the image displayed by the image display unit 20. The angle of view of the first camera 61 and the second camera 62 is preferably capable of capturing a wider range than the user's field of view. Specifically, the angle of view is at least wider than the effective field of view of the user. Is preferred. Further, it is more preferable that the angle of view is wider than the actual field of view of the user. More preferably, the angle of view is wider than the stable viewing field of the user, and most preferably, the angle of view is wider than the viewing angle of both eyes of the user.

  The first camera 61 and the second camera 62 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, or may be a single focus lens or a zoom lens. The lens group composed of a plurality of lenses may be a first camera 61 and a first lens. The structure with which 2 cameras 62 are provided may be sufficient.

FIG. 4 is a block diagram illustrating a configuration of each unit included in the HMD 100.
The control device 10 includes a main processor 140 that executes a program and controls the HMD 100. A memory 121 and a non-volatile storage unit 123 are connected to the main processor 140. In addition, a 6-axis sensor 111 and a magnetic sensor 113 are connected to the main processor 140 as sensors. The main processor 140 is connected with a GPS receiver 115, a communication unit 117, an audio codec 182, an external connector 184, an external memory interface 186, a USB connector 188, a sensor hub 192, and an FPGA 194. These function as an interface with the outside. In addition, the LED indicator 12, the LED display unit 17, the vibrator 19, the operation unit 110, and the power supply unit 130 are connected to the main processor 140.

  The main processor 140 is mounted on the controller board 120 built in the control device 10. In addition to the main processor 140, a memory 121, a nonvolatile storage unit 123, a six-axis sensor 111, a magnetic sensor 113, a GPS reception unit 115, a communication unit 117, an audio codec 182, and the like are further mounted on the controller board 120. In this embodiment, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 are mounted on the controller board 120.

  The memory 121 constitutes a work area that temporarily stores a control program to be executed and data to be processed when the main processor 140 executes the control program. The nonvolatile storage unit 123 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 123 stores a program executed by the main processor 140 and various data processed by the main processor 140 executing the program.

  Although FIG. 4 shows a configuration in which the function of the control device 10 is realized by one main processor 140, the function of the control device 10 may be realized by a plurality of processors or semiconductor chips. For example, a SoC (System-on-a-Chip), an MCU (Micro Control Unit), an FPGA (Field-Programmable Gate Array), or the like may be further mounted on the controller board 120. . The control device 10 may perform various controls by causing both the main processor 140 and the sub-processing device to cooperate or selectively using one of the two.

  The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which the above sensors are modularized. The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.

  The 6-axis sensor 111 and the magnetic sensor 113 output the detection value to the main processor 140 according to a sampling period specified in advance. The 6-axis sensor 111 and the magnetic sensor 113 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

  The GPS receiving unit 115 includes a GPS antenna (not shown) and receives a GPS signal transmitted from a GPS satellite. The GPS receiving unit 115 outputs the received GPS signal to the main processor 140. Further, the GPS receiving unit 115 measures the signal strength of the received GPS signal and outputs it to the main processor 140. Information such as received signal strength (RSSI: Received Signal Strength Indication), electric field strength, magnetic field strength, signal to noise ratio (SNR) can be used as the signal strength.

  The communication unit 117 performs wireless communication with an external device. The communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication complying with standards such as Bluetooth and wireless LAN (including Wi-Fi), for example.

  The audio interface 180 is an interface for inputting and outputting audio signals. In the present embodiment, the audio interface 180 includes a connector 46 (FIG. 1) provided on the connection cable 40. The connector 46 is connected to the headset 30. The audio signal output from the audio interface 180 is input to the right earphone 32 and the left earphone 34, whereby the right earphone 32 and the left earphone 34 output audio. Further, the microphone 63 provided in the headset 30 collects sound and outputs a sound signal to the sound interface 180. An audio signal input from the microphone 63 to the audio interface 180 is input to the external connector 184.

  The audio codec 182 is connected to the audio interface 180 and encodes and decodes an audio signal input / output via the audio interface 180. The audio codec 182 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. For example, the HMD 100 according to the present embodiment outputs sound by the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 182 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 180. The audio codec 182 converts an analog audio signal input to the audio interface 180 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector that connects an external device that communicates with the main processor 140. The external connector 184 is an interface for connecting an external device when, for example, an external device is connected to the main processor 140 to debug a program executed by the main processor 140 or collect a log of the operation of the HMD 100. It is.

  The external memory interface 186 is an interface to which a portable memory device can be connected. For example, the external memory interface 186 includes a memory card slot in which a card type recording medium is mounted and data can be read and an interface circuit. In this case, the size, shape and standard of the card type recording medium are not limited and can be changed as appropriate.

  The USB (Universal Serial Bus) connector 188 includes a connector conforming to the USB standard and an interface circuit. The USB connector 188 can connect a USB memory device, a smartphone, a computer, or the like. The size and shape of the USB connector 188 and the compatible USB standard version can be appropriately selected and changed.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196.

  The LED indicator 12 lights up or blinks in accordance with the operating state of the HMD 100. The LED display unit 17 controls lighting and extinguishing of the LED indicator 12 according to the control of the main processor 140. The LED display unit 17 may include an LED (not shown) disposed immediately below the track pad 14 and a drive circuit that lights the LED. In this case, the LED display unit 17 turns on, blinks, and turns off the LED according to the control of the main processor 140.

  The vibrator 19 includes a motor and an eccentric rotor (both not shown), and may have other necessary configurations. The vibrator 19 generates vibration by rotating the motor according to the control of the main processor 140. The HMD 100 generates vibration by the vibrator 19 with a predetermined vibration pattern, for example, when an operation on the operation unit 110 is detected, when the power of the HMD 100 is turned on or off, and in other cases.

  The operation unit 110 includes an operation element 13 and a track pad 14. The operation element 13 includes an operation button 11, an up / down key 15, a changeover switch 16, and a power switch 18. When the operation unit 13 or the track pad 14 is operated, the operation unit 110 controls an operation signal including identification information of the operated operation unit 13 or the track pad 14 and information indicating the received operation content. Output to the unit 150.

  The control device 10 includes a power supply unit 130 and operates with power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls charging of the battery 132. The power control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 or a detected value of the voltage to the main processor 140. Further, the power may be supplied from the control device 10 to the image display unit 20 based on the power supplied from the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are each connected to the control device 10. As shown in FIG. 1, in the HMD 100, a connection cable 40 is connected to the left holding unit 23, wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is Connected to the control device 10.

The right display unit 22 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213 that receives data input from the control device 10 via the interface 211, and an EEPROM 215 are mounted. .
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the first camera 61, the second camera 62, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data relating to light emission characteristics and display characteristics of the OLED units 221 and 241 provided in the image display unit 20, data relating to characteristics of sensors provided in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215. After the shipment, the main processor 140 can perform processing using the data in the EEPROM 215.

  The first camera 61 and the second camera 62 execute imaging according to a signal input via the interface 211, and output captured image data or a signal indicating the imaging result to the control device 10.

  As shown in FIG. 1, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity).

  As shown in FIG. 1, the LED indicator 67 is disposed near the first camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the first camera 61 or the second camera 62 to notify that imaging is being performed.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 2). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The receiving unit 213 receives data transmitted from the main processor 140 via the interface 211. When receiving the image data of the image displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

  The left display unit 24 has a display unit substrate 230. On the display unit substrate 230, an interface (I / F) 231 connected to the interface 196 and a receiving unit (Rx) 233 that receives data input from the control device 10 via the interface 231 are mounted. In addition, a six-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210. The interface 231 connects the receiving unit 233, the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

  The 6-axis sensor 235 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

  The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 2). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243. Further, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 and the like, the temperature sensor 239 may be mounted on the semiconductor chip.

  The first camera 61, the second camera 62, the illuminance sensor 65, the temperature sensor 217, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. Is done.

The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in accordance with the output timing to the main processor 140. In addition, the sensor hub 192 may have a function of converting the data of the output value of each sensor or the data format into data of a unified data format and outputting the data to the main processor 140.
The sensor hub 192 starts and stops energization of the LED indicator 67 according to the control of the main processor 140, and lights or blinks the LED indicator 67 in accordance with the timing when the first camera 61 starts and ends imaging.

  FIG. 5 is a functional block diagram of the storage unit 160 and the control unit 150 that constitute the control system of the control device 10. A storage unit 160 illustrated in FIG. 5 is a logical storage unit configured by the nonvolatile storage unit 123 (FIG. 4), and may include an EEPROM 215 and a memory 121. The control unit 150 and various functional units included in the control unit 150 are formed by cooperation of software and hardware by the main processor 140 executing a program. The control unit 150 and each functional unit constituting the control unit 150 include, for example, a main processor 140, a memory 121, and a nonvolatile storage unit 123.

  The storage unit 160 stores a control program executed by the control unit 150 and various data processed by the control unit 150. Specifically, the storage unit 160 stores a control program 161, setting data 162, content data 163, a pattern recognition dictionary 164, captured image data 165, and the like.

  The control program 161 is an application program having a specific function that is executed on the OS 151. The application programs include, for example, browser software, business programs, office programs such as document creation software and spreadsheet software. In addition, the application program includes a program that detects the movement, shape, and position of the user's hand as a gesture and executes processing associated with the detected gesture. Details of the operation when the control unit 150 executes the application program will be described later.

  The setting data 162 includes various setting values that set the operation of the HMD 100. Further, when the control unit 150 uses the parameters, determinant, arithmetic expression, LUT (Look Up Table), etc. when controlling the HMD 100, these may be included in the setting data 162.

The content data 163 is content data including a display image and video displayed by the image display unit 20 under the control of the control unit 150, and includes image data or video data. Further, the content data 163 may include music and audio data.
Further, the content data 163 may be bidirectional content data. That is, the image display unit 20 displays the image data and video data included in the content data 163, the operation unit 110 receives an operation on the displayed image data and video data, and the control unit 150 performs a process corresponding to the received operation. Run. In this case, the content data 163 may include image data of a menu screen that is displayed when an operation is received, data that defines processing corresponding to items included in the menu screen, and the like.

  The pattern recognition dictionary 164 is a dictionary in which information for specifying a gesture and processing executed by the control unit 150 are registered in association with each other. The gesture is a preset movement or shape of a hand (in this specification, a portion from the wrist to the tip of the finger is referred to as a “hand”). In addition, an operation of designating a preset position of the display area VR by the user's hand can be included in the gesture.

  For example, assume that the gesture is an action of moving the hand from left to right. In this case, information indicating the moving direction of the hand is registered in the pattern recognition dictionary 164 as information for specifying a gesture. In addition, it is assumed that the gesture is an operation of opening the grasped hand, that is, changing the shape of the hand from the state of grasping the palm to the state of expanding the hand. In this case, information indicating the shape of the gripped hand, information indicating the shape of the open hand, and information indicating the detection order of the information indicating these shapes are registered in the pattern recognition dictionary 164. Further, when it is assumed that the gesture is an operation of pointing a preset position of the display area VR by hand, the pattern recognition dictionary 164 includes, for example, information indicating the position or range of the display area VR indicated by the hand. Is registered. For example, when it is determined that the position indicated by the user's hand is a position or range corresponding to the four corners of the display area VR, the control unit 150 determines that a gesture has been detected.

  In addition, the process registered in the pattern recognition dictionary 164 in association with the information for specifying the gesture is performed by controlling a device as a control target or by executing an application program. Etc. are included.

  The registration information in the pattern recognition dictionary 164 can be updated. For example, it is possible to change the association between the information specifying the gesture and the process executed by the control unit 150. Further, information specifying a gesture may be downloaded from a server device (not shown) on the Internet. The user operates the operation unit 110 to associate processing desired to be executed by the control unit 150 with information specifying the downloaded gesture. In accordance with the operation received by the operation unit 110, the control unit 150 associates information for specifying a gesture with a process to be executed by the control unit 150 and registers the information in the pattern recognition dictionary 164.

  The captured image data 165 is captured image data captured by at least one of the first camera 61 and the second camera 62. The captured image data 165 is data temporarily stored in the storage unit 160. For example, when the processing by the control unit 150 ends, the captured image data 165 is deleted from the storage unit 160. The captured image data captured by the first camera 61 corresponds to the “captured image” of the present invention.

  The control unit 150 executes various processes using data stored in the storage unit 160 and controls the HMD 100. The control unit 150 includes an OS (Operating System) 151, an image processing unit 152, an imaging control unit 153, a detection unit 154, a display control unit 155, and a processing control unit 156 as functional blocks. The function block is a function that is realized by executing a control program by the main processor 140 for convenience, and does not indicate a specific application program or hardware.

  The function of the OS 151 is a function of a control program stored in the storage unit 160, and the other units are functions of application programs executed on the OS 151.

For example, the image processing unit 152 reads the content data 163 from the storage unit 160 and separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the read content data 163. The image processing unit 152 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. In addition, the image processing unit 152 may perform various image processing such as resolution conversion, brightness and saturation adjustment, and 2D / 3D conversion processing on the image data included in the content data 163 as necessary. .
The image processing unit 152 develops the image data subjected to the image processing in the DRAM in the storage unit 160 for each frame that is an image display unit. A DRAM area where one frame of image data is expanded is hereinafter referred to as a frame area. The image processing unit 152 reads the image data from the frame area and outputs the read image data to the image display unit 20 as display image data.

  The image processing unit 152 may be configured by hardware (for example, a DSP (Digital Signal Processor)) different from the main processor 140 in addition to the configuration realized by the main processor 140 executing a program.

  The imaging control unit 153 controls the first camera 61 and the second camera 62 to execute imaging, and generates captured image data. The imaging control unit 153 temporarily stores the generated captured image data in the storage unit 160. When the first camera 61 and the second camera 62 are configured as a camera unit including a circuit that generates captured image data, the imaging control unit 153 acquires the captured image data from the first camera 61 and the second camera 62. , Temporarily stored in the storage unit 160.

The detection unit 154 detects the movement of the indicator based on the captured image data captured by the first camera 61, and accepts an operation based on the detected movement of the indicator.
The operation accepted by the detection unit 154 is an operation based on a preset movement of the indicator. The movement of the indicator includes, for example, an operation for moving the indicator in the vertical direction and the horizontal direction, an operation for rotating the indicator, and the like. The detection unit 154 detects the movement of the indicator based on the captured image data of the first camera 61, and determines whether or not the detected movement of the indicator corresponds to a preset movement of the indicator. . The detection unit 154 accepts an operation based on the movement of the indicator when the detected movement of the indicator corresponds to a preset movement of the indicator. When receiving the operation, the detection unit 154 executes a process associated with a preset movement of the indicator.

  First, the detection unit 154 reads the captured image data of the first camera 61 from the storage unit 160, and detects the indicator imaged in the read captured image data. In addition, the detection unit 154 detects an indicator from captured image data continuously captured by the first camera 61, and detects at least one of the movement, shape, and position of the indicator. The detection unit 154 receives an operation based on at least one of the detected movement, shape, and position of the indicator.

  In the present embodiment, a case where the indicator is a user's hand will be described, but the indicator is not limited to the user's hand. For example, a part of the body such as the user's finger or arm, an indicator held by the user's hand, or a detected object attached to the body may be used. It is also possible to use a ring attached to the user's finger, a watch or a band attached to the user's arm as the indicator. When using a pointer, a clock, a band, or the like as the indicator, the image data of the indicator is stored in advance in the storage unit 160, and the indicator is detected by a method such as pattern matching.

  The detection unit 154 detects an area in which the user's hand is imaged (hereinafter referred to as a hand area) in the captured image data. For example, the detection unit 154 extracts the skin color area from the captured image data area by skin color extraction processing, and detects the hand area by comparing the pixel value of the extracted skin color area with the threshold value by threshold processing. . The detection unit 154 may extract the contour of the obtained hand region by edge detection processing, and detect the shape and position of the hand using feature point matching or curvature determination based on the contour unevenness. The detection unit 154 instructs the display control unit 155 to display the object 300 when detecting a hand region in the captured image data. The object 300 will be described with reference to FIG.

  In addition, the detection unit 154 detects the hand region from each of the captured image data continuously captured by the first camera 61 or the captured image data for every predetermined number of times, and detects the movement, shape, and position of the hand region. To do. When detecting the movement, shape, and position of the hand area, the detection unit 154 refers to the pattern recognition dictionary 164 based on the detected movement, shape, and position of the hand area, and identifies a gesture corresponding to the detected movement of the hand area. To do. The detection unit 154 receives an operation based on at least one of the movement, shape, and position of the indicator by specifying a gesture corresponding to the movement, shape, and position of the hand region. When the detection unit 154 specifies a gesture and receives an operation, the detection unit 154 acquires a process associated with the specified gesture from the pattern recognition dictionary 164. The detection unit 154 causes the process control unit 156 to execute a process associated with the acquired gesture.

  The display control unit 155 generates control signals for controlling the right display unit 22 and the left display unit 24, and controls generation and emission of image light by the right display unit 22 and the left display unit 24, respectively, based on the generated control signals. To do. Specifically, the display control unit 155 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 155 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243 based on the signal output from the image processing unit 152, controls the brightness of the OLED panels 223 and 243, and the like.

  Further, the display control unit 155 determines whether or not the detection unit 154 is ready to accept an operation, and displays the determination result. Specifically, the display control unit 155 determines whether or not the HMD 100 is in a state in which an operation by a gesture can be accepted when the indicator is detected by the detection unit 154. If the display control unit 155 determines that the operation can be received, the display control unit 155 controls the image display unit 20 to display a display indicating that the operation can be received in the display area VR.

FIG. 6 is a diagram showing a visual field range FV of a user wearing the image display unit 20 on the head.
When the indicator is detected by the detection unit 154, the display control unit 155 displays a preset object 300 such as an image, a character, a figure, a symbol, a mark, or an icon in the display region VR. Further, the display control unit 155 displays the displayed image, character, figure, symbol, mark, or icon when the indicator is detected by the detection unit 154 and the detection unit 154 is in a state where the operation of the indicator can be received. The display mode of the object 300 is changed. For example, the display control unit 155 determines that the operation of the indicator is acceptable when the indicator is detected and other processing with a large processing load such as image processing is not being executed.

  FIG. 6 shows a case where the first object 310 and the second object 320 are displayed as the object 300. In the present embodiment, a range image 311 indicating the range of the hand region is displayed as the first object 310, and an image simulating the shape of the indicator (hereinafter referred to as an icon) 321 is displayed as the second object 320. Indicates. The range image 311 corresponds to the “image superimposed on the indicator” of the present invention.

The range image 311 as the first object 310 is an image indicating the range of the user's hand detected from the captured image data. The range image 311 is displayed on the display region VR by the display control unit 155 when a hand region is detected from the captured image data. When the hand region is no longer detected from the captured image data, the display control unit 155 deletes the range image 311 from the display region VR.
The display control unit 155 may display the range image 311 so as to surround the periphery of the hand region detected by the detection unit 154, or may display the range image 311 so as to overlap a part of the hand region. . The shape of the range image 311 may be a rectangle, a circle or a polygon, or a shape corresponding to the shape of the body.
The user can determine whether or not the object detected by the HMD 100 as the indicator is a hand by displaying the range image 311 in the display area VR. Therefore, when the HMD 100 recognizes and detects an erroneous object as a hand, the HMD 100 can detect the indicator again so that the erroneous operation is not executed.

  An icon 321 that is the second object 320 is an icon that indicates the state of the HMD 100, and indicates that a hand region has been detected and that the HMD 100 is in a state in which an operation by a gesture can be accepted. In this embodiment, a case where an icon 321 simulating the shape of a hand as an indicator is displayed as the second object 320 is shown. For example, when the indicator is an indicator, the shape of the indicator is indicated. An icon may be displayed as the second object 320. In the present embodiment, the case where the icon 321 is displayed as the second object 320 will be described. However, the second object 320 is not limited to an icon, and may be an image, a character, a figure, a symbol, or a mark.

  The display control unit 155 changes the display mode of the second object 320. For example, the second object 320 is displayed in the display area VR when the application program is activated. In the display control unit 155, the visibility of the second object 320 when the hand region is detected and the control unit 150 is ready to accept an operation is higher than the visibility of the second object 320 in other cases. As described above, the display mode of the second object 320 is changed. Other cases include a case where a hand region is not detected from the captured image data, and a case where a hand region is detected but the control unit 150 is not in a state where it can accept an operation.

For example, the display control unit 155 can change the display mode of the second object 320 by changing the transmittance of the second object 320. The display control unit 155 sets the transmittance of the second object 320 to be lower than the transmittance of the second object 320 in other cases when the hand region is detected and the control unit 150 is in a state where the operation can be accepted. . When the transmittance of the second object 320 is set low, the second object 320 is displayed conspicuously with respect to the outside scene of the display area VR. Further, by setting the transmittance of the second object 320 high, the second object 320 becomes inconspicuous with respect to the outside scene of the display area VR.
Further, the display control unit 155 can change the display mode of the second object 320 by changing the display color of the second object 320. The display control unit 155 makes the display color of the second object 320 when the hand region is detected and the control unit 150 is ready to accept an operation to be more conspicuous than the display color of the second object 320 in other cases. change. For example, the display control unit 155 displays the second object 320 in white when a hand region is detected and the control unit 150 is ready to accept an operation, and in other cases, the second object 320 is displayed. Display in black. The display control unit 155 turns on the display of the second object 320 when the hand region is detected and the control unit 150 is ready to accept an operation, and otherwise displays the second object 320. May be turned off.

  Although FIG. 6 shows a case where the first object 310 and the second object 320 are displayed as the object 300, only one of them may be displayed, for example, display of the LED indicators 12, 67, etc. You may display in combination with other displays. For example, the first object 310 may be displayed in the display area VR, and the LED indicator 12 may be turned on. FIG. 6 shows the case where the first object 310 and the second object 320 are displayed as the object 300. For example, text such as “a hand has been detected” may be displayed in the display area VR. Good. Further, as the object 300, a symbol or mark indicating that a hand has been detected may be displayed.

  The process control unit 156 controls the operation of the device as the control target according to the control of the detection unit 154. In addition, the process control unit 156 performs processing by executing an application program according to the control of the detection unit 154. Examples of devices controlled by the processing control unit 156 include the second camera 62, the microphone 63, the six-axis sensors 111 and 235, the magnetic sensors 113 and 237, the temperature sensors 217 and 239, the illuminance sensor 65, and the LED indicators 12 and 67. Etc. are included. Further, the processing performed by executing the application program includes, for example, processing for reproducing the content data 163 such as a moving image or music. The processing performed by executing the application program includes processing for starting browser software as an application program and processing for executing a search based on a keyword input in a search field of the browser software.

FIG. 7 is a flowchart showing the operation of the control unit 150.
The operation of the control unit 150 will be described with reference to the flowchart of FIG. First, the control unit 150 determines whether an operation for starting an application program has been received (step S1). In step S <b> 1, the control unit 150 determines whether an operation for starting an application program that controls the HMD 100 with a gesture (hereinafter simply referred to as an operation) has been received. If no operation is received (step S1 / NO), control unit 150 waits for the start of the process until an operation is received, and executes another process if there is another process that can be executed.

  When the operation is received (step S1 / YES), the control unit 150 executes the application program to display the second object 320 and the third object 330 in the display area VR (step S2).

FIG. 8 is a diagram illustrating a display state of the display area VR. In particular, FIG. 8 is a diagram showing a state in which the second object 320 and the third object 330 are displayed as the object 300 in the display area VR.
FIG. 8 shows a case where the second object 320 and the third object 330 are displayed on the upper right side of the display area VR. The second object 320 and the third object 330 are displayed on the upper left, lower right, lower left, etc. of the display area VR. It may be displayed at other positions. The second object 320 and the third object 330 are preferably displayed at a set position in the display area VR. By always displaying the second object 320 and the third object 330 at a fixed position, it is possible to reduce the trouble for the user to search for the second object 320 and the third object 330 displayed in the display area VR.

The third object 330 is an image indicating a gesture to be detected that is detected by the control unit 150, and is an image whose display state changes according to the state of the device to be controlled.
The control unit 150 displays, as the third object 330, arrow images 331 and 333 indicating the moving direction of the indicator as a gesture, and characters 332 and 334 indicating the control target and the control action in the display area VR. Characters 332 indicating the control target and the control action are displayed in the arrow image 331. Similarly, a character 334 indicating a control target and a control operation is displayed in the arrow image 333. The arrow image 331 is an arrow image indicating the right direction of the display area VR as viewed from the user side. The arrow image 333 is an arrow image indicating the left direction of the display area VR as viewed from the user side.

  Characters 332 and 334 indicating control targets are device names to be controlled by the control unit 150, and characters 332 and 334 indicating control operations indicate control operations of the control unit 150 performed on the control targets. In the present embodiment, the characters “camera” are displayed as the characters 332 and 334 indicating the control target. In the arrow image 331 indicating the right direction of the display area VR, a character “ON” is displayed as a character 332 indicating a control operation. In the arrow image 333 indicating the left direction of the display area VR, a character “OFF” is displayed as the character 332 indicating the control operation. The control unit 150 turns on the power of the second camera 62 when a gesture for moving the hand from the left to the right is detected as viewed from the user side. The control unit 150 turns off the power of the second camera 62 when a gesture for moving the hand from right to left is detected as viewed from the user side.

  Further, the control unit 150 changes the display mode of the third object 330 according to the state of the device to be controlled. The control unit 150 sets the transmittance of the arrow image 331 to a value lower than the transmittance of the arrow image 331 when the power of the second camera 62 is on when the power of the second camera 62 is off. When the power of the second camera 62 is off, the transmittance of the arrow image 331 is low, so that the arrow image 331 becomes a conspicuous display with respect to the outside scene in the display area VR, and the visibility of the arrow image 331 is enhanced. In addition, the control unit 150 may change the display color of the arrow image 331 depending on whether the power of the second camera 62 is on or off. The controller 150 displays the arrow image 331 in an inconspicuous display color such as gray when the power of the second camera 62 is on, and the arrow image 331 is conspicuous as white or the like when the power of the second camera 62 is off. Display in display color.

  In addition, when the power of the second camera 62 is on, the control unit 150 sets the transmittance of the arrow image 333 to a value lower than the transmittance of the arrow image 333 when the power of the second camera 62 is off. . When the power of the second camera 62 is turned on, the transmittance of the arrow image 333 is low, so that the arrow image 333 becomes a conspicuous display with respect to the outside scene in the display area VR, and the visibility of the arrow image 333 is improved. The control unit 150 may change the display color of the arrow image 333 depending on whether the power of the second camera 62 is turned on or turned on. The control unit 150 displays the arrow image 333 in an inconspicuous display color such as gray when the power of the second camera 62 is off, and the arrow image 333 is conspicuous as white or the like when the power of the second camera 62 is on. Display in display color.

  In the present embodiment, arrow images 331 and 333 and characters 332 and 334 indicating control targets and control actions are shown as examples of the third object 330. However, the third object 330 is not limited to an image. , Graphics, symbols, marks, icons.

  When the second object 320 and the third object 330 are displayed in the display area VR, the control unit 150 reads captured image data to be processed from the storage unit 160. Here, among the captured image data for which the detection of the hand area has not been completed, the captured image data having the oldest captured time is read out. The control unit 150 performs skin color extraction processing and threshold processing on the read captured image data to detect a hand region (step S3). When the hand region is not detected (step S3 / NO), the control unit 150 reads captured image data to be processed next from the storage unit 160, and performs a skin color extraction process or a threshold on the read captured image data. Processing is performed to detect a hand region (step S3). The control unit 150 may detect the hand area using all of the captured image data captured by the first camera 61 as a processing target, or may select the captured image data to be processed at a predetermined interval to detect the hand area. Detection may be performed. The control unit 150 sets the captured image data captured next to the captured image data selected as the processing target last time or the captured image data captured after a predetermined number of images from the captured image data selected as the processing target as the next processing target. Select as captured image data.

  Further, when a hand region is detected from the captured image data (step S3 / YES), the control unit 150 changes the display color of the second object 320 displayed in the display region VR (step S4). As a result, the visibility of the second object 320 can be improved compared to before the hand region is detected. In addition, the control unit 150 displays the range image 311 in the display area VR corresponding to the hand area detected from the captured image data (step S5).

FIG. 9 is a diagram showing a display state of the display area VR. In particular, FIG. 9 is a diagram illustrating an example of the range image 311.
When detecting the hand region from the captured image data, the control unit 150 determines a display position for displaying the range image 311 based on the detected hand region. For example, the control unit 150 may set the display position of the range image 311 so as to surround the detected hand region, or set the display position of the range image 311 so as to overlap a part of the hand region. Also good. When the display position of the range image 311 is set, the control unit 150 displays the range image 311 as the first object 310 at the position of the display area VR corresponding to the set display position (step S5).

  When the range image 311 is displayed in the display area VR, the control unit 150 analyzes the captured image data continuously captured by the first camera 61 and detects a gesture. The gesture indicated by the image of the third object 330 displayed in the display area VR is a detection target. More specifically, when the power of the second camera 62 is not turned on, a gesture for moving the hand from the left to the right as viewed from the user side is the gesture to be detected. In addition, when the power of the second camera 62 is turned on, a gesture to move the hand from the right to the left as viewed from the user side is a gesture to be detected.

  The control unit 150 detects a hand region from each of the captured image data continuously captured by the first camera 61, and determines whether or not there is a change in position of the detected hand region. When there is a change in the position of the hand area, the control unit 150 determines whether the change in the position of the hand area corresponds to the gesture registered in the pattern recognition dictionary 164. When the change in the position of the hand region corresponds to the gesture registered in the pattern recognition dictionary 164, the control unit 150 determines that a gesture has been detected (step S6). When detecting a gesture, the control unit 150 acquires a process corresponding to the detected gesture pattern from the pattern recognition dictionary 164. When the control unit 150 acquires process information corresponding to the gesture pattern, the control unit 150 executes a process indicated by the acquired process information (step S7). For example, when the power of the second camera 62 is not turned on and a gesture for moving the hand from the left to the right as viewed from the user side is detected, the control unit 150 turns on the power of the second camera 62.

  When the process corresponding to the gesture is executed (step S7), the control unit 150 determines whether there is a gesture associated with the next acceptable operation (step S8). For example, when the power of the second camera 62 is turned on in step S7, it is determined whether or not there is a gesture associated with the operation of releasing the shutter of the second camera 62. When there is a gesture associated with the operation of releasing the shutter of the second camera 62 (step S8 / YES), the control unit 150 displays an image of the third object 330 associated with the gesture in the display area VR ( Step S2). When it is determined that there is no gesture associated with the operation (step S8 / NO), the control unit 150 ends this processing flow.

  If no gesture is detected in step S6 (step S6 / NO), the controller 150 determines whether or not the elapsed time since the display of the range image 311 in the display area VR has exceeded the first set time. Is determined (step S9). When the elapsed time has not passed the first set time (step S9 / NO), the control unit 150 returns to step S6 and detects a gesture from the captured image data (step S6). In addition, the control unit 150 causes the operation guide to be displayed in the display area VR when the elapsed time has passed the first set time (step S9 / YES) (step S10).

FIG. 10 is a diagram illustrating the operation guide displayed in the display area VR.
The operation guide is an image that guides the gesture to the user, and is an image that represents the movement of the gesture detected by the control unit 150. In addition to the operation guide, images, figures, symbols, marks, and icons that image the user's hand as indicators are displayed in the display area VR, and the display positions of the displayed images, figures, symbols, marks, and icons are displayed for a predetermined time. The movement of the gesture may be indicated by changing each time. FIG. 10 shows an operation guide corresponding to a gesture for turning on the power of the second camera 62. In this operation guide, an icon 307 indicating the user's hand is displayed in the display area VR, and the display position of the displayed icon 307 is moved from left to right as viewed from the user side at predetermined time intervals. The operation guide may be a moving image captured in advance.

  When the operation guide is displayed in the display area VR, the control unit 150 performs gesture detection based on the captured image data of the first camera 61 (step S11). When the gesture is not detected (step S11 / NO), the control unit 150 determines whether or not the second set time has elapsed after the operation guide is displayed in the display area VR (step S12). When the second set time has not elapsed (step S12 / NO), the control unit 150 returns to the determination of step S11 and continues to detect the gesture (step S9). Moreover, the control part 150 complete | finishes the display of an operation guide, when 2nd setting time passes (step S12 / YES) (step S13). Thereafter, the control unit 150 returns to the determination in step S9, and determines whether or not the first set time has elapsed since the display of the operation guide was terminated (step S9). When the first set time has elapsed (step S9 / YES), the control unit 150 displays the operation guide again (step S10), and when the first set time has not elapsed (step S9 / NO), step S6. Returning to, gesture detection is performed from the captured image data.

FIG. 11 is a diagram showing another display example of the display area VR.
In the above-described flowchart, the example in which the operation guide is displayed when the first set time has elapsed after the range image 311 is displayed in the display area VR has been described. As another operation, a display 309 indicating the remaining time until the display of the operation guide (referred to as remaining time display) 309 may be displayed during the measurement of the first set time. By displaying the remaining time display 309, the user can select whether or not to display the operation guide.

As described above, the HMD 100 according to the first embodiment includes the image display unit 20, the first camera 61 and the imaging control unit 153 that operate as the detection unit, the detection unit 154 that operates as the control unit, the display control unit 155, and the process control. Part 156.
The image display unit 20 displays the image superimposed on the outside scene, or displays the image around the outside scene. The first camera 61 and the imaging control unit 153 detect the indicator. The detection unit 154 receives an operation based on at least one of the detected movement, shape, and position of the indicator. The display control unit 155 determines whether or not an input operation using an indicator is possible, and displays the determination result.
Therefore, the user can grasp the state of the HMD 100 and improve the usability of the operation using the indicator.

Further, the display control unit 155 causes the image display unit 20 to display an image, a figure, a character, or an icon indicating that an operation can be accepted at a position set in the display area of the image display unit 20.
Therefore, the user can intuitively recognize that the HMD 100 is in a state where the operation can be accepted.

In addition, the display control unit 155 is configured to be able to control a control target according to an operation, and displays an image, a figure, a character, or an icon indicating that the operation can be accepted according to the state of the control target controlled according to the operation. Change.
Therefore, the user can intuitively recognize the state of the control target.

Further, the first camera 61 detects a range including an outside scene that can be visually recognized together with the image displayed by the image display unit 20. The display control unit 155 causes the image display unit 20 to display that the operation can be accepted when an indicator is detected within the range from the captured image data of the first camera 61.
Therefore, the user can make the HMD 100 ready to receive an operation by moving the position of the indicator within a range including a visually recognizable outside scene. Further, when an indicator is detected within the range, a display indicating that the operation can be accepted is displayed, so that the state of the HMD 100 can be grasped.

  The image display unit 20 includes a transmissive display area VR that allows the outside scene to be visually recognized by transmitting outside light. The first camera 61 detects the indicator within a range including an outside scene that is visible through the image display unit 20. Therefore, the user can make the HMD 100 ready to receive an operation by moving the position of the indicator within a range including a visually recognizable outside scene.

The display control unit 155 operates when the indicator is detected in the captured image data of the first camera 61 and the operation can be accepted, and when the indicator is not detected in the captured image data of the first camera 61. The display mode of the display indicating that the operation can be accepted is changed in the case where the operation is not acceptable.
Therefore, it is possible to notify the user whether or not the HMD 100 has detected the indicator.

  Further, the display control unit 155 detects a gesture as an input operation based on at least one of the movement, shape, and position of the indicator detected in the captured image data of the first camera 61. The display control unit 155 changes the display mode of any one of an image, a figure, a character, or an icon associated with the detected input operation. Therefore, the HMD 100 can be operated by a gesture using an indicator. Moreover, since the display mode of any of images, graphics, characters, or icons associated with the gesture detected by the HMD 100 changes, the detected gesture can be notified to the user.

In addition, when the indicator is detected by the detector 154, the display controller 155 displays the range image 311 so as to be superimposed on the indicator.
Therefore, the range image 311 can be displayed superimposed on the indicator. For this reason, the object recognized by the HMD 100 as the indicator can be notified.

In addition, the display control unit 155 displays an image, a graphic, a character, or an icon in the display area VR, and when the detection unit does not accept an operation even after a preset first setting time has elapsed, An operation guide for guiding the operation by the indicator is displayed in the display area VR.
Therefore, an operation guide can be displayed to guide the user to perform operations using the indicator.

The display control unit 155 displays the remaining time until the operation guide is displayed in the display area in the display area VR.
Therefore, it is possible to notify the user who does not know the operation of the time when the operation guide is displayed.

Further, the display control unit 155 displays the operation guide in the display area VR, when a preset second set time has elapsed, or when the detection unit 154 detects an operation based on the movement of the indicator Then, clear the operation guide display.
Therefore, when it is determined that it is not necessary to display the operation guide, the display of the operation guide can be erased, and the visibility of the display area VR can be improved.

In addition, the process control unit 156 controls a device as a control target by an operation received by the detection unit 154, or executes an application program as a control target, and functions of an application program corresponding to the operation received by the detection unit 154 Execute.
Therefore, the device can be controlled by operating the indicator, and the function can be selected and executed by the application program as the control target.

[Second Embodiment]
The second embodiment of the present invention will be described below with reference to the accompanying drawings.

  The display control unit 155 displays the object 300 in the display area VR when the detection unit 154 detects the image of the indicator from the captured image data of the first camera 61. In the present embodiment, a range image 311 that is the first object 310 is displayed as the object 300.

  When detecting the indicator from the captured image data, the detecting unit 154 outputs coordinate data indicating a range on the captured image data in which the indicator is captured to the display control unit 155. The display control unit 155 performs calculation based on the coordinate data input from the detection unit 154, and obtains the coordinate data of the display area VR surrounding the indicator as shown in FIG. The display control unit 155 displays the range image 311 in the display area VR corresponding to the obtained coordinate data.

  The setting data 162 stored in the storage unit 160 includes calibration data. The calibration data is data that associates the captured image data of the first camera 61 with the display area VR of the image display unit 20. More specifically, it is data for converting coordinates on the captured image data into coordinates on the display area VR. For example, when a real object such as a marker is imaged by the first camera 61 and the image is displayed by the image display unit 20 at a position overlapping the captured real object, the coordinates on the captured image data are the coordinates on the display area VR. Calibration data 145 that is data to be converted to is required. Therefore, calibration data for converting the coordinates on the captured image data into the coordinates on the display area VR is generated in advance by causing the HMD 100 to perform calibration.

  In addition, when the detection unit 154 detects the image of the indicator from the captured image data of the first camera 61, the display control unit 155 displays the icon 321 that is the second object 320 and the arrow image 331 that is the third object 330. 333 and characters 332 and 334 may be displayed together with the first object 310. In addition, the display timing of the second object 320 and the third object 330 may be a timing at which an application program for executing processing for turning on or off the second camera 62 is started. The display timing of the second object 320 or the third object 330 may be a timing at which the detection unit 154 determines that an operation can be accepted.

  Next, when the detection unit 154 detects the indicator, the display control unit 155 determines whether the detection unit 154 is in a state where it can accept an operation by a gesture, and displays the determination result in the display area VR. To do. For example, the display control unit 155 determines that the detection unit 154 cannot accept an operation by a gesture when a process with a large processing load such as image processing is being executed and the HMD 100 is busy.

  When the display control unit 155 determines that the HMD 100 is not in a busy state and the detection unit 154 is in a state in which an operation by a gesture can be received, the display control unit 155 displays a display indicating that the operation can be received in the display region VR. Further, when the display control unit 155 determines that the HMD 100 is in a busy state and the detection unit 154 is not in a state in which an operation by a gesture can be received, a display indicating that the operation cannot be received is displayed in the display region VR. To do. For these displays, for example, images, figures, and icons are used. Moreover, you may display the character and symbol which show whether it is in the state which can accept operation.

In the present embodiment, the display control unit 155 changes the display mode of at least one of the range image 311 and the icon 321 so that the detection unit 154 can accept the operation and can accept the operation. When not.
For example, the display control unit 155 displays the range image 311 and the icon 321 with a solid line when the detection unit 154 is ready to accept an operation, and the range image when the detection unit 154 is not ready to accept an operation. 311 and the icon 321 are displayed with a broken line.
The display control unit 155 blinks the range image 311 and the icon 321 when the detection unit 154 is not in a state where the operation can be accepted, and displays the range image 311 when the detection unit 154 is in a state where the operation can be received. The icon 321 is turned on.
In addition, the display control unit 155 may change the display color of the range image 311 and the icon 321 depending on whether the detection unit 154 is in a state where the operation can be accepted or not. .

  In addition, the display control unit 155 displays the range image 311 and the icon 321 when the detection unit 154 is in a state where the operation can be accepted, and displays the range image when the detection unit 154 is not in a state where the operation is acceptable. The display of 311 and the icon 321 may be deleted. In this case, the display control unit 155 may switch between displaying and not displaying the range image 311 and the icon 321 by changing the transmittance. That is, when displaying the range image 311 or the icon 321 in the display area VR, the display control unit 155 sets the transmittance of the range image 311 or the icon 321 low. Further, when the range image 311 and the icon 321 are not displayed in the display region VR, the display control unit 155 sets the transmittance of the range image 311 and the icon 321 high. By changing the transmittance to a low value, the range image 311 and the icon 321 are displayed more conspicuously than the outside scene visually recognized through the image display unit 20. Further, by changing the transmittance to a high value, the range image 311 and the icon 321 are displayed more conspicuously than the outside scene visually recognized through the image display unit 20.

  Further, the first object 310 and the second object 320 may be displayed in the display area VR at different timings. For example, when the detection unit 154 detects the hand area of the user from the captured image data, the range image 311 as the first object 310 is displayed. Alternatively, the second object 320 may be displayed when a hand region is detected from the captured image data and the HMD 100 is ready to accept an operation. Further, the third object 330 may be displayed together with the second object 3320.

Further, the display control unit 155 changes the display mode of the third object 330 in accordance with the operation state of the device or application program as the control target. The third object 330 is an image, figure, or icon indicating the movement of the indicator detected by the detection unit 154. That is, an image, a figure, or an icon indicating a gesture registered in the pattern recognition dictionary 164.
For example, when the control target is the second camera 62 and the operation state of the second camera 62 is off, the display control unit 155 indicates the gesture associated with the operation for turning on the second camera 62. The three objects 330 are blinked or lit. In addition, the display control unit 155 displays the transmittance of the third object 330 indicating the gesture associated with the operation for turning on the second camera 62, and the third object indicating the gesture associated with the operation for turning off the second camera 62. A value lower than the transmittance of 330 may be set. Therefore, the third object 330 associated with the operation of turning on the second camera 62 can be displayed more conspicuously than the third object 330 associated with the operation of turning off the second camera 62.

  In addition, when the control target is an application program for playing a movie and the movie is being played, the display control unit 155 is associated with operations for stopping, fast-forwarding, and fast-rewinding the movie. The gesture is displayed as the third object 330. In addition, the display control unit 155 displays a gesture associated with an operation for playing a moving image as the third object 330 when the moving image playing application program is activated but the moving image is not played.

FIG. 12 is a flowchart showing another operation of the control unit 150.
When the power switch 18 is turned on (step S21), the control program 161 stored in the nonvolatile storage unit 123 is loaded into the memory 121. The control unit 150 is activated (step S22) and starts executing the loaded control program 161. The control unit 150 causes the first camera 61 to start imaging (step S23).

  In addition, when the first camera 61 starts imaging, the control unit 150 displays the second object 320 and the third object 330 in the display area VR (step S24). The second object 320 is displayed in such a manner that the transmittance is set high and the second object 320 is not conspicuous with respect to the outside scene of the display area VR. When detecting the hand area of the user from the captured image data of the first camera 61, the control unit 150 changes the transmittance of the second object 320 to a low value. Accordingly, the second object 320 is displayed more conspicuously than the outside scene visually recognized through the image display unit 20, and the second object 320 is visually recognized by the user.

In the display area VR, as shown in FIG. 9, as the third object 330, an arrow image 331 in which a “camera ON” character 332 is displayed and an arrow image in which a “camera OFF” character 334 is displayed. 333 is displayed.
In a state before the hand area is detected, the second camera 62 is not turned on. Therefore, the transmittance of the arrow image 331 on which the character 332 “camera ON” is displayed is low, and “camera OFF” is displayed. The arrow image 333 on which the character 334 is displayed is displayed with high transmittance.
As a result, the arrow image 331 on which the characters “332” of “camera ON” are displayed is more conspicuous than the outside scene visually recognized through the image display unit 20. In addition, the arrow image 333 on which the characters “334” of “camera OFF” are displayed is less conspicuous than the outside scene visually recognized through the image display unit 20. Accordingly, the user does not seem to display the arrow image 333 in which only the arrow image 331 in which the character “332” of “camera ON” is displayed is displayed in the display area VR, and in which the character 334 of “camera OFF” is displayed. Recognized.

  Next, the control unit 150 acquires captured image data of the first camera 61, and detects a hand area for the acquired captured image data (step S25). When the hand region cannot be detected from the captured image data (step S25 / NO), the control unit 150 acquires the next captured image data and detects the hand region with respect to the acquired captured image data. .

  Further, when detecting a hand region from the captured image data (step S25 / YES), the control unit 150 determines whether or not the control unit 150 is in a state where it can accept an operation (step S26). For example, the control unit 150 determines that the operation cannot be accepted when another process is being executed and the new command is not acceptable (step S26 / NO). In this case, the control unit 150 determines whether or not the elapsed time from the start of detection of the hand region in step S25 has passed the timeout time (step S27).

  When the time-out period has not elapsed (step S27 / NO), the control unit 150 waits for a predetermined time (step S28). When a predetermined time has elapsed, the control unit 150 determines again whether or not the control unit 150 is ready to accept an operation in step S26 (step S26). In addition, when the preset timeout time has elapsed (step 26 / YES), the control unit 150 displays an error display on the image display unit 20 (step S29).

  In addition, when the control unit 150 is in a state where the operation can be accepted (step S26 / YES), the control unit 150 changes the display mode of the second object 320 or causes the second object 320 to be displayed in the display region VR ( Step S30). Specifically, the control unit 150 increases the visibility of the second object 320 by changing the transmittance of the second object 320 to a lower value, and displays the second object 320 than the outside scene visually recognized through the image display unit 20. Make it stand out. When the second object 320 is not displayed in step S24, the control unit 150 displays the second object 320 in step S30.

  In addition, the control unit 150 displays the range image 311 in the display area VR corresponding to the area of the captured image data in which the hand area is detected (step S31). For example, the control unit 150 displays the range image 311 on the display region VR so as to surround the detected hand region.

  Next, the control unit 150 starts detecting a gesture (step S32). When the range image 311 is displayed in the display area VR, the control unit 150 detects a hand area from each of the captured image data continuously captured by the first camera 61 and detects a gesture input by the user. To do. The image of the third object 330 displayed in the display area VR is a gesture to be detected. Specifically, when the power of the second camera 62 is not turned on, a gesture for moving the hand from the left to the right as viewed from the user side is the gesture to be detected. In addition, when the power of the second camera 62 is turned on, a gesture to move the hand from the right to the left as viewed from the user side is a gesture to be detected.

The control unit 150 detects the hand area from each of the captured image data continuously captured by the first camera 61, and detects the movement of the hand area (step S33). When there is no change in the position of the hand area detected from each of the captured image data, the control unit 150 determines that there is no movement in the hand area (step S33 / NO), and continues to detect the movement of the hand area.
In addition, when there is a change in the position of the detected hand region and there is a movement in the hand region (step S33 / YES), the control unit 150 registers a gesture corresponding to the detected movement of the hand region in the pattern recognition dictionary 164. It is determined whether or not (step S34).

  When the gesture corresponding to the detected movement of the hand area is not registered in the pattern recognition dictionary 164 (step S34 / NO), the control unit 150 returns to step S33 and detects the movement of the hand area again. When the gesture corresponding to the detected movement of the hand region is registered in the pattern recognition dictionary 164 (step S34 / YES), the control unit 150 executes a process associated with the registered gesture (step S34). S35).

As described above, the HMD 100 according to the second embodiment includes the image display unit 20, the first camera 61, the detection unit 154, and the display control unit 155.
The image display unit 20 is mounted on the user's head and has a display area VR that displays an image so that the outside scene can be visually recognized.
The detection unit 154 detects the movement of the indicator based on the captured image data of the first camera 61, and accepts an operation based on the detected movement of the indicator.
The display control unit 155 determines whether or not the operation based on the movement of the indicator detected by the detection unit 154 can be detected, and displays the determination result in the display region VR.
Therefore, the determination result as to whether or not the detection unit 154 can detect an operation based on the movement of the indicator is displayed in the display region VR. Therefore, it is possible to provide an easy-to-use user interface that can grasp the state of the HMD 100 in the operation using the indicator.

In addition, the display control unit 155 displays, as a determination result, a range image 311 indicating a range of a figure, a character, an icon, or an indicator detected by the detection unit 154 indicating that the detection unit 154 can detect an operation. Displayed in the display area VR.
Therefore, since a figure, a character, an icon, and a range image are displayed as the determination result, it is possible to notify the outside whether or not the operation is detectable.

Further, the display control unit 155 displays the range image 311 in the display region VR when the detection unit 154 detects the indicator, and the display region depends on whether or not the detection unit 154 can detect the operation. The display mode of the graphic, character, or icon displayed on the VR is changed. The graphic, character, or icon includes an icon 321 of the second object 320, arrow images 331 and 333, and characters 332 and 334 of the third object 330.
Therefore, it is possible to notify the outside that the detection unit 154 has detected the indicator, and it is possible to notify the outside whether or not the detection unit 154 is in a state in which an operation can be detected.

In addition, when the detection unit 154 is in a state in which an operation can be detected, the display control unit 155 displays a graphic or icon indicating the movement of the indicator detected by the detection unit 154 as a graphic or an icon.
Accordingly, since the graphic or icon indicating the movement of the indicator detected by the detection unit 154 is displayed, the user can recognize the operation using the indicator.

The HMD 100 also includes a process control unit 156 that controls a control target in accordance with an operation received by the detection unit 154. The display control unit 155 changes the graphic or icon indicating the movement of the preset indicator detected by the detection unit 154 according to the operation state of the control target.
Therefore, the user can move the indicator corresponding to the operation state of the controlled object, and the controlled object can be operated by the movement of the indicator. For this reason, the convenience of operation using a pointer can be improved.

In addition, the detection unit 154 determines whether or not the detected movement of the indicator corresponds to a preset movement of the indicator. When the detection unit detects the movement of the indicator corresponding to the preset movement of the indicator, the process control unit 156 sets the process associated with the movement of the indicator detected by the detection unit 154 as a control target. Let it run.
According to this configuration, it is possible to cause the control target to execute processing associated with the movement of the indicator by performing the movement of the indicator set in advance.

The above-described embodiment is a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.
For example, in the above-described embodiment, the movement, shape, and position of the indicator are detected based on the captured image data of the first camera 61, but the sensors such as an infrared sensor, an electromagnetic wave sensor, an optical sensor, and an ultrasonic sensor are used. The movement, shape, and position of the indicator may be detected.

In addition, the display control unit 155 detects the hand region and the control unit 150 is in a state where the operation can be accepted, the case where the hand region is not detected, and the state where the hand region is detected but the operation can be accepted. The display mode of the second object 320 may be changed depending on whether it is not present.
For example, the display control unit 155 sets the transmittance of the second object 320 to be the lowest when the hand region is detected and the control unit 150 is in a state where the operation can be accepted. Further, the display control unit 155 sets the second object 320 to have the highest transmittance when no hand region is detected. In addition, the display control unit 155 determines the transmittance of the second object 320 when the hand region is detected but the control unit 150 is not ready to accept the operation, The transmittance is set to an intermediate value between the case where the area is not detected.

  Further, the HMD 100 may include an interface (not shown) for connecting various external devices that are content supply sources. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface may be used, or a wireless communication interface may be used. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on the content data 163 input from these external devices.

  In the above-described embodiment, the configuration in which the control device 10 is connected to the image display unit 20 by wire is illustrated. However, the present invention is not limited to this, and the image display unit 20 is wirelessly connected to the control device 10. It may be a configuration. In this case, the wireless communication method may be the method exemplified as the communication method supported by the communication unit 117, or may be another communication method.

  In addition, some functions of the control device 10 may be provided in the image display unit 20, and the control device 10 may be realized by a plurality of devices. That is, the control device 10 is not limited to the configuration including the box-shaped case 10A. For example, instead of the control device 10, a user's body, clothes, or a device worn by the user may be used. The device worn by the user may be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, a pen-type device, or the like.

  Furthermore, in the above embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.

  As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like.

Further, for example, instead of the image display unit 20, an image display unit of another method such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye LE of the user is displayed. What is necessary is just to provide the display part which displays, and the display part which displays an image corresponding to a user's right eye RE. In addition, by using a head-up display instead of the image display unit 20, it can be mounted on a vehicle such as an automobile or an airplane. For example, when a head-up display is mounted on a vehicle, an operation surface corresponding to the operation surface of the track pad 14 is provided on the handle of the vehicle.
For example, it may be configured as a head-mounted display device built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

In addition, as an optical system that guides image light to the user's eyes, a configuration in which virtual images are formed by half mirrors 261 and 281 on part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and an image may be displayed on a display region having an area that occupies the entire or most of the right light guide plate 26 and the left light guide plate 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.
Furthermore, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  In addition, at least a part of the functional blocks shown in FIGS. 4 and 5 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 150 may be stored in the nonvolatile storage unit 123 or another storage device (not shown) in the control device 10. A program stored in an external device may be acquired and executed via the communication unit 117 or the external connector 184. Moreover, the operation part 110 may be formed as a user interface (UI) among the structures formed in the control apparatus 10.

  7 and 12 are divided according to main processing contents in order to make the processing of the control unit 150 of the HMD 100 easy to understand. The present invention is not limited by the name. Further, the processing of the control unit 150 can be divided into more processing units according to the processing content, or can be divided so that one processing unit includes more processing. Further, the processing order of the above flowchart is not limited to the illustrated example.

  When the method for controlling the head-mounted display device of the present invention is realized by a computer, the program is executed by a computer, a recording medium in which the program is recorded so as to be readable by the computer, or a transmission medium that transmits the program. can do. As the recording medium, a magnetic or optical recording medium or a semiconductor memory device can be used. Specifically, flexible disk, HDD (Hard Disk Drive), CD-ROM (Compact Disk Read Only Memory), DVD (Digital Versatile Disk), Blu-ray (registered trademark) Disc, magneto-optical disk, flash memory, card A portable recording medium such as a mold recording medium or a fixed recording medium can be used. The recording medium may be a non-volatile storage device such as a random access memory (RAM), a read only memory (ROM), or an HDD that is an internal storage device included in the image display device.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 10A ... Case, 11 ... Operation button, 12, 67 ... LED indicator, 13 ... Operator, 14 ... Track pad, 15 ... Up / down key, 16 ... Changeover switch, 17 ... LED display part, 18 ... Power supply Switch, 19 ... Vibrator, 20 ... Image display unit, 21 ... Right holding unit, 22 ... Right display unit, 23 ... Left holding unit, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... headset, 32 ... right earphone, 34 ... left earphone, 40 ... connection cable, 46 ... connector, 61 ... first camera, 62 ... second camera, 63 ... microphone, 65 ... illuminance sensor, 100 ... head-mounted display device, 111 ... 6-axis sensor, 113 ... magnetic sensor, 110 ... operation part, 115 ... GPS receiver, 117 ... communication part, 120 ... con Roller board, 121 ... Memory, 123 ... Non-volatile storage unit, 127 ... Setting data, 130 ... Power supply unit, 132 ... Battery, 134 ... Power supply control circuit, 140 ... Main processor, 150 ... Control unit, 151 ... OS, 152 ... Image processing unit 153 ... Imaging control unit, 154 ... detection unit, 155 ... display control unit (control unit), 156 ... processing control unit (control unit), 160 ... storage unit, 161 ... control program, 162 ... setting data, 163 ... Content data, 164 ... Pattern recognition dictionary, 165 ... Captured image data, 180 ... Audio interface, 182 ... Audio codec, 184 ... External connector, 186 ... External memory interface, 188 ... USB connector, 192 ... Sensor hub, 194 ... FPGA, 196, 211, 231 ... Interface 213, 233 ... receiving unit, 215 ... EEPROM, 217 ... temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit, 235 ... 6-axis sensor, 237 ... magnetic sensor, 239 ... temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 251 ... right optical system, 252 ... left optical system, 261,281 ... half mirror, 307 ... icon, 309 ... remaining time display, 310 ... first object, 311... Range image (image superimposed on indicator) 320 320 second object 321 icon 3 330 object 331 333 arrow image 332 334 character

Claims (12)

A display unit that is mounted on the user's head and has a display area that displays an image so that the outside scene can be visually recognized;
An imaging unit;
A detection unit that detects a movement of the indicator based on a captured image of the imaging unit and receives an operation based on the detected movement of the indicator;
A head-mounted display device comprising: a display control unit that determines whether or not the detection unit is capable of detecting an operation based on the movement of the indicator and displays the determination result in the display region.   The display control unit, as the determination result, a graphic, a character, an icon indicating that the detection unit can detect the operation, or a range image indicating the range of the indicator detected by the detection unit The head-mounted display device according to claim 1, wherein the head-mounted display device displays on the display area.   The display control unit displays the range image in the display area when the detection unit detects the indicator, and the display unit displays the display depending on whether the detection unit is in a state where the operation can be detected. The head-mounted display device according to claim 2, wherein a display mode of the graphic, the character, or the icon displayed in the area is changed.   The said display control part displays the figure or icon which shows the motion of the said indicator which the said detection part detects as the said figure or the said icon, when the said detection part is a state which can detect the said operation. Item 4. The head-mounted display device according to Item 2 or 3. A processing control unit that controls a control object according to the operation received by the detection unit;
The head according to claim 4, wherein the display control unit changes the figure or the icon indicating the preset movement of the indicator detected by the detection unit according to an operation state of the control target. Wearable display device. The detection unit determines whether or not the detected movement of the indicator corresponds to a preset movement of the indicator,
The processing control unit, when the detection unit detects a movement of the indicator corresponding to a preset movement of the indicator, a process associated with the movement of the indicator detected by the detection unit The head-mounted display device according to claim 5, wherein the control target is executed.   The head mounted according to any one of claims 2 to 6, wherein the display control unit displays the range image superimposed on the indicator when the indicator is detected by the detection unit. Type display device.   The display control unit displays the graphic, the character, or the icon in the display area, and the detection unit does not accept the operation even after a preset first set time has elapsed. The head-mounted display device according to claim 2, wherein an operation guide that guides an operation by the indicator is displayed in the display area.   The head-mounted display device according to claim 8, wherein the display control unit displays a remaining time until the operation guide is displayed in the display area in the display area.   The display control unit, when a preset second set time has elapsed after displaying the operation guide in the display area, or when the detection unit detects an operation based on the movement of the indicator The head-mounted display device according to claim 8 or 9, wherein the display of the operation guide is erased.   The processing control unit controls the device as the control target by the operation received by the detection unit, or executes the application program as the control target and executes the application program corresponding to the operation received by the detection unit. The head-mounted display device according to claim 5 or 6, which executes a function. A method for controlling a head-mounted display device that is mounted on a user's head and displays an image in a display area so that an outside scene can be visually recognized,
Determining whether the operation based on the movement of the indicator is in a detectable state, and displaying the determination result in the display area;
Detecting the movement of the indicator based on the captured image of the imaging unit;
Receiving an operation based on the detected movement of the indicator;
A method for controlling a head-mounted display device.

Images