JP2018146966A - Information input/output device and information input/output method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information input/output device and an information input/output method capable of encouraging a user to confirm information displayed on a display unit.SOLUTION: An information input/output device includes: a display unit that displays information; an observation propriety determination unit that determines whether or not the display unit can be observed by a user; a notification unit that notifies by sound; and a control unit that controls, when the display unit is determined to be incapable of being observed by the user on the basis of result determined by the observation propriety determination unit, the notification unit to notify the user of the effect.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information input / output device and an information input / output method.

The display unit of the navigation device that performs route guidance displays information such as a map of the vicinity where the user is moving, the current position, and the direction of travel to the destination. When such a navigation device is mounted on a vehicle, the user drives the vehicle while viewing the information displayed on the display unit. For example, in a city center with many landmarks, when a user searches for a parking lot or the like from a map displayed on the display unit, it takes time to search for necessary information from information displayed on the screen. As described above, when the user keeps looking at the display unit for a long time in order to search for necessary information, the concentration of driving and the attention may be reduced.
For this reason, for example, in the technique of Patent Document 1, when the user keeps looking at the display unit for a long time, a notification is made to prompt the user not to look at the display unit. Furthermore, in the technique of Patent Document 1, if the user continues to look at the display unit even after notification, the display on the display unit is turned off.

JP 2001-307298 A

  However, in the technique of Patent Document 1, when the user keeps looking at the display unit for a long time, it only gives notification to urge the user not to look at the display unit or turns off the display. When displayed on the display, it did not prompt the user to pay attention. Information important to the user is, for example, accident information, speed limit information, and the like. For this reason, the user may miss information displayed on the display unit.

  The present invention has been made in view of the above circumstances, and provides an information input / output device and an information input / output method capable of prompting a user to check information displayed on a display unit. It is an object.

  In order to achieve the above object, an information input / output device according to an aspect of the present invention includes a display unit that displays information, and an observability determination unit that determines whether the display unit is observable by a user. And when the display unit is determined to be in a state that is not observable by the user, based on the determination result by the notification unit that performs notification by sound and the observation availability determination unit, the notification unit notifies the user. And a control unit for controlling as described above.

In order to achieve the above object, an information input / output method according to an aspect of the present invention is an information input / output method in an information input / output device, in which an observation availability determination unit can observe a display unit that displays information by a user Based on the result of determination in the observation propriety determination procedure, the control unit determines that the display unit is not observable by the user based on the result of the determination in the observation propriety determination procedure. In this case, the control unit includes a procedure for controlling so as to perform notification by sound from the notification unit.
It is said.

  According to the present invention, the user can be prompted to confirm the information displayed on the display unit.

It is a perspective view of the head mounted display of a 1st embodiment. It is the perspective view which looked at the head mounted display in the state where the display main part and headband concerning a 1st embodiment were removed from the back side. It is a horizontal sectional view of the display main part concerning a 1st embodiment. It is operation | movement explanatory drawing of the headband which concerns on 1st Embodiment. It is a figure which shows the mounting | wearing form (1st display mode) of the head mounted display which concerns on 1st Embodiment. It is a perspective view which shows the stereo earphone attached to the head mounted display which concerns on 1st Embodiment. It is a figure explaining lighting and extinction of the eyeball illumination light concerning a 1st embodiment. It is sectional drawing of the display part for demonstrating the structure and optical system of a display part which concern on 1st Embodiment. It is a figure explaining the difference of the display area in each of the 1st-3rd observation state. It is a figure explaining the observation range and imaging range in a 1st observation state. It is a figure explaining an example of the captured image in the 1st observation state. It is a figure explaining an example of the captured image in the 2nd observation state. It is a figure explaining the observation range and imaging range in a 2nd observation state. It is a figure explaining an example of the captured image in the 3rd observation state. It is a figure explaining the observation range and imaging range in a 3rd observation state. It is a figure explaining the example which arrange | positions a display part in front of eyes and images the surrounding image containing eyes. It is a figure explaining the example which arrange | positions a display part to the lower side of the eye which concerns on 1st Embodiment, and images an eye from the bottom. It is a figure explaining the angle of view and selection range of the image sensor concerning a 1st embodiment. It is a figure explaining the imaging area and selection area concerning a 1st embodiment. It is a figure explaining the mechanism which performs selection of the selection range concerning a 1st embodiment mechanically. It is a functional block diagram of the head mounted display concerning a 1st embodiment. It is a functional block diagram of the process part which concerns on 1st Embodiment. 1 is a system configuration diagram of a communication system including an HMD according to a first embodiment. It is a flowchart of the operation | movement performed after the power supply state of HMD which concerns on 1st Embodiment. It is a flowchart of the detection process procedure of the right and left eyes which concerns on 1st Embodiment. It is a figure explaining an example of the picked-up image of the imaging range S in the case of a right eye observation form. It is a figure explaining an example of the captured image of the imaging range S in the case of a left eye observation form. It is a figure explaining an example of the captured image in a left-eye observation form. It is a flowchart of the process sequence of the mounting guide of HMD which concerns on 1st Embodiment. It is a flowchart of the selection processing procedure of the HMD eyeball illumination light according to the first embodiment. It is a flowchart of the procedure of the calibration for the gaze point detection which concerns on 1st Embodiment. It is a flowchart of the procedure of the calibration for the gaze point detection which concerns on 1st Embodiment. It is a figure explaining lighting and extinguishing of the eyeball illumination light in the right eye observation mode according to the first embodiment. It is a figure explaining the relationship between the position of the display part which concerns on 1st Embodiment, and an eyeball illumination light, and is the figure which looked at the user's face from the horizontal front. It is an example of the password input screen which concerns on 1st Embodiment. It is a figure which shows an example of the menu screen which concerns on 1st Embodiment. It is an example of the screen which prompts the user who concerns on 1st Embodiment. It is a figure explaining the screen of the application of the eyes | visual_axis input which concerns on 1st Embodiment. It is a flowchart of the procedure of the business logging using HMD which concerns on 1st Embodiment. It is a figure which shows an example of the movement log memorize | stored in the memory | storage part which concerns on 1st Embodiment. It is a figure which shows an example of the barcode attached to the test object apparatus which concerns on 1st Embodiment. It is a figure explaining an example of the information displayed on the display panel which concerns on 1st Embodiment. It is a figure which shows the mounting form (2nd display mode, retreat mode) of the head mounted display which concerns on 2nd Embodiment. It is a figure explaining the relationship between the position of the display part which concerns on 2nd Embodiment, and an eyeball illumination light, and is the figure which looked at the user's face from the horizontal front. It is a flowchart of the selection processing procedure of the eyeball illumination light of HMD which concerns on 2nd Embodiment. It is sectional drawing of the display part for demonstrating the optical system which concerns on 3rd Embodiment. It is a figure explaining the observation range and imaging range which concern on 3rd Embodiment.

In the head mounted display (information input / output device) according to the present embodiment, it is determined by the observation availability determination unit whether the display unit is observable by the user, and it is determined that the display unit is not observable by the user. In this case, control is performed so that the control unit notifies from the notification unit. Accordingly, the user notices that the display unit is observed by the notification even when the display unit is not in an observable state. Therefore, the user can confirm without missing the information displayed on the display unit.
Hereinafter, an example in which the information input / output device is applied to a head mounted display will be described, but the present invention is not limited to this.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following description, an XYZ rectangular coordinate system is set as necessary, and the positional relationship of each part of the head mounted display will be described with reference to the XYZ rectangular coordinate system. A predetermined direction in the horizontal plane is defined as an X-axis direction, a direction orthogonal to the X-axis direction in the horizontal plane is defined as a Y-axis direction, and a direction orthogonal to each of the X-axis direction and the Y-axis direction (that is, a vertical direction) is defined as a Z-axis direction. Further, the rotation (inclination) directions around the X axis, Y axis, and Z axis are the θX, θY, and θZ directions, respectively.

[First Embodiment]
FIG. 1 is a perspective view of a head mounted display (head mounted device, hereinafter also referred to as HMD) 1 according to the present embodiment. FIG. 2 is a perspective view of the head mounted display 1 as viewed from the back side in a state where the display main body and the headband according to the present embodiment are removed.
FIG. 3 is a horizontal sectional view of the display main body 20 according to the present embodiment. FIG. 4 is an explanatory diagram of the operation of the headband according to the present embodiment. FIG. 5 is a diagram illustrating a mounting mode (first display mode) of the head mounted display according to the present embodiment.

The HMD 1 is a monocular head mounted display that includes a display main body 20 and a headband 40 that is attached to the user's head and supports the display main body 20. The HMD 1 of the present embodiment can be used with either eye as shown in FIG. FIG. 5A shows a state where the user is viewing the display unit 60 with the right eye (right eye), and FIG. 5B shows a state where the user is viewing with the left eye (left eye).
As shown in FIG. 2, the display main body 20 and the headband 40 are configured to be detachable via a connecting pin 41. 1 and 2, the longitudinal direction of the display main body 20 is defined as the Y-axis direction, and the direction in which the headband 40 sandwiches the user's head is defined as the X-axis direction.

Hereinafter, the configuration of each part of the HMD 1 will be described in detail.
The display main body 20 includes a device main body 21 that includes a main circuit and includes an operation unit and various interfaces, and a display unit 60 that is connected to the tip of the device main body 21.

  The apparatus main body 21 has a substantially plate-shaped casing 21A (see FIG. 3). In the present embodiment, an end portion (+ Y side end portion) on the side where the connection portion with the headband 40 is provided in the apparatus main body portion 21 is a base end portion, and an end portion on the opposite side of the base end portion ( The end on the -Y side) is the tip. Further, in a state where the apparatus main body 21 is attached to the headband 40, the headband 40 side (+ X side) of the apparatus main body 21 is the inside, and the opposite side (−X side) from the headband 40 is the outside.

As shown in FIG. 1, a main switch 28, a touch switch 34, and a sound collecting microphone (sound acquisition unit) 24 are arranged on the outer surface of the apparatus main body 21 along the longitudinal direction of the housing 21A. Yes.
The main switch 28 is a switch for turning on / off the power of the display body 20. The touch switch 34 is a touch panel that can perform various operations of the HMD 1 by touching the surface with a finger or the like. The sound collection microphone 24 is an external microphone that collects environmental sounds.

  As shown in FIG. 2, an ear speaker 23, an audio connector 26, and a headband hinge 32 having a connection hole 31 are provided on the base end side of the inner surface of the apparatus main body 21. A heart rate sensor 137 is provided at the center of the inner surface of the apparatus main body 21. A call microphone 37 is provided at the tip of the inner side surface of the apparatus main body 21.

  The ear speaker 23 is disposed in the vicinity of the user's ear. Audio information is transmitted from the ear speaker 23 to the user. The audio connector 26 is, for example, an audio input / output terminal to which the earphone shown in FIG. 6 is connected. The headband hinge 32 is a joint part with the headband 40. The voice of the user is input to the call microphone 37.

  The heart rate sensor 137 is a sensor that measures a user's heart rate by contacting the surface of the user's face. The heart rate sensor 137 includes a light emitting unit including a light emitting diode and the like, and a light receiving unit that detects light reflected inside the user's skin. The heart rate sensor 137 counts the heart rate by detecting a change in the amount of reflected light due to a change in blood flow. The heart rate sensor 137 is disposed near the user's eyes, but if the infrared light is emitted from the light emitting unit, the user will not feel glare.

A USB connector 25, an operation switch 30, and a video connector 27 are provided on the side end surface of the apparatus main body 21 on the base end side.
The USB connector 25 is a connection terminal of a USB (Universal Serial Bus) device. In this embodiment, for example, a remote controller (not shown) is connected.
The operation switch 30 is a pointing device such as a trackball or a stick. The operation switch 30 is provided so as to face the screen displayed on the display unit 60. As a result, the left-right direction of the operation on the operation switch 30 matches the left-right direction of the screen, so that the user can intuitively operate the operation switch 30 while looking at the screen.
The video connector 27 is a video input / output terminal.

  As shown in FIG. 3, the apparatus main body 21 incorporates a plate-like circuit board 29 extending along the longitudinal direction of the housing 21 </ b> A and a battery 33. A control circuit, a power supply circuit, and the like (not shown) are mounted on the circuit board 29, and are electrically connected to each part of the display body 20 via wiring (not shown).

  A display panel 36 made of a liquid crystal panel and a backlight 35 are arranged inside the touch switch 34 exposed on the outer surface of the apparatus main body 21. In the present embodiment, the display image on the display panel 36 is displayed through the touch switch 34. The display panel 36 and the backlight 35 may be an organic EL panel or an electrophoresis panel.

  The headband hinge 32 is a ball joint composed of a concavely curved housing portion 32a provided in the housing 21A and a spherical portion 32b fitted to the housing portion 32a. The spherical portion 32b has a spherical side surface portion and two plane portions formed in parallel to each other so as to sandwich the side surface portion. A connecting hole 31 is formed so as to penetrate the two flat portions vertically. The connection hole 31 is formed in a hexagonal shape when viewed in the axial direction. The display main body 20 and the headband 40 are connected by inserting the connection pin 41 of the headband 40 into the connection hole 31.

  By providing the headband hinge 32, the display main body 20 can be rotated in the A direction (around the X axis centering on the headband hinge 32) shown in FIG. In the present embodiment, the rotatable range of the display body 20 is about 270 °. By the rotation operation around the X axis, a switching function between a form in which an image is observed with the right eye shown in FIG. 5A and a form in which an image is observed with the left eye shown in FIG. 5B is realized.

  Further, since the headband hinge 32 is a ball joint, the display main body 20 can also be swung in the B direction (around the Z axis centering on the headband hinge 32) shown in FIG. By this swinging operation, the position of the display main body 20 relative to the user's eyes and ears can be adjusted.

  The heart rate sensor 137 in the vicinity of the headband hinge 32 is provided so as to protrude from the inner surface of the apparatus main body 21, and can be brought into contact with the surface of the user's face when the HMD 1 is worn. The battery 33 may be either a primary battery or a secondary battery.

As shown in FIGS. 1 and 2, the display unit 60 is connected to the distal end portion of the apparatus main body unit 21. In the present embodiment, the headband 40 side of the display unit 60 is the inside, and the side opposite to the headband 40 is the outside. The display unit 60 is an arm member that has a curved shape in a top view (Z-axis view), and has a shape that curves inward from the connecting portion with the apparatus main body 21 toward the distal end side. A finder opening 67 is provided on the inner surface of the display unit 60. A camera 64 is provided on the outer surface of the display unit 60. Moreover, as shown in FIG. 2, the shape of the finder opening 67 is a rectangle.
On the housing surface of the display unit 60, eyeball illumination lights 166 a to 166 d are provided in the vicinity of the finder opening 67. In FIG. 2, the eyeball illumination lights 166 a and 166 b are provided at positions along one of the long sides of the finder opening 67 and in the vicinity of the corner of the finder opening 67. The eyeball illumination lights 166 c and 166 d are provided along the other of the long sides of the finder opening 67 and near the corner of the finder opening 67.

  The eyeball illumination lights 166a to 166d are, for example, LEDs (light emitting diodes) that emit light in the infrared wavelength region. The eyeball illumination lights 166a to 166d are turned on or off according to the positional relationship between the user and the display unit 60. Hereinafter, the eyeball illumination lights 166a to 166d are collectively referred to as an eyeball illumination light 166.

  As shown in FIG. 3, the display unit 60 is connected to the apparatus main body unit 21 via a display hinge 61. The display hinge 61 is a ball joint composed of a concave curved housing 61a formed in the housing 60A of the display 60 and a spherical portion 61b formed in the apparatus main body 21 and fitted in the housing 61a. .

  The spherical portion 61b of the apparatus main body 21 has a normal direction of the inclined surface 21a (in FIG. 3) to an inclined surface 21a extending obliquely with respect to the longitudinal direction of the casing 21A formed at the outer surface tip of the apparatus main body 21. It is provided so as to protrude in the Y′-axis direction).

  The display unit 60 can be freely rotated around the Y ′ axis with respect to the spherical portion 61 b of the display hinge 61. Accordingly, the display unit 60 can be rotated from the viewable state illustrated in FIG. 3A to shift to the accommodation state illustrated in FIG. In the present embodiment, the display hinge 61 is provided on the inclined surface of the apparatus main body 21, and the display unit 60 has a shape curved inward. Thereby, the display part 60 is arrange | positioned so that the outer surface of the apparatus main body part 21 may be followed in the accommodation state shown to FIG. 3 (B).

  Further, since the display hinge 61 is a ball joint, the display unit 60 can swing in a direction orthogonal to the rotation plane around the Y ′ axis. This swinging operation facilitates position adjustment of the finder opening 67 by the user. If the interval between the display unit 60 and the apparatus main body unit 21 is increased, the swinging width of the display unit 60 can be increased.

  The spherical portion 61b of the display hinge 61 is formed with a through hole 61c that penetrates the spherical portion 61b in the height direction (Y′-axis direction). The inside of the display unit 60 and the inside of the apparatus main body unit 21 are communicated with each other through the through hole 61c. A cable (not shown) is inserted through the through hole 61c. The circuit board 29 and each part of the display unit 60 are electrically connected via the inserted cable.

  Inside the display unit 60, there are a backlight 62, a display panel 63, a camera (second imaging unit) 64, a prism 65, a reflection mirror 66, a front light 68, a front speaker 70, and an imaging lens. 71 and an imaging element (first imaging unit) 72 are provided. The configuration of the optical system of the display unit 60 will be described later.

  The camera 64 includes, for example, an image sensor with 5 to 10 million pixels, and is configured to be able to perform an autofocus operation. The camera 64 captures an image (optical image) formed on the light receiving surface of the image sensor. For example, the camera 64 can capture the front direction for the user. Note that the captured image may be signal-processed in either a moving image format or a still image format.

The front light 68 is, for example, an LED light. The front light 68 may include red, green, and blue light emitting elements, and may be configured to emit any color at any timing. The front light 68 may be used as a device that displays information to the outside depending on the light emission color and the light emission timing, or may be used as an illumination device when photographing with the camera 64.
The image sensor 72 captures an image (optical image) formed on the light receiving surface. The image sensor 72 can image a user's face, for example. Note that the captured image may be signal-processed in either a moving image format or a still image format.

  Although not shown in FIG. 3, the display unit 60 according to the present embodiment is provided with a laser transmitter. The laser transmitter can be provided in the vicinity of the front light 68, for example. By irradiating, for example, red laser light emitted from a laser transmitter forward, pointing (display of an index) with the laser light is possible.

Next, the headband 40 will be described with reference to FIGS.
As shown in FIG. 2, the headband 40 includes a pair of head pads (mounting members) 46 and 47 that sandwich the user's head, a first headband 43, a second headband 44, and a rotation mechanism 56. , 57.

  The first headband 43 is an elastic member having an arc shape as a whole. A joint portion 43 a for bending the first headband 43 is provided on the top of the first headband 43. At both ends of the first headband 43, bearing portions 43b and 43c constituting the rotation mechanisms 56 and 57 are provided, respectively. Bearing portions 43d and 43e to which the head pads 46 and 47 are connected are provided further on the front end side of the band than the bearing portions 43b and 43c.

  The second headband 44 is an elastic member having an arc shape as a whole. A joint portion 44 a for bending the second headband 44 is provided on the top of the second headband 44. At both ends of the second headband 44, shaft members 44b and 44c constituting rotation mechanisms 56 and 57 are provided.

  In the present embodiment, the second headband 44 has a configuration in which the surface of the spring member 48 made of a metal such as stainless steel is covered with a flexible material such as a resin. The spring member 48 of the second headband 44 generates a spring force that holds the user's head. The second headband 44 is formed wider than the first headband 43. The formation part of the joint part 44a is a mark display part 49 formed wider than the other band parts. A product tag or the like is attached to the mark display section 49 by sticker or printing. The entire second headband 44 may be made of metal.

  The head pad 46 includes a plate-like support plate 46a and an elastic member 46b having an arched cross section provided on one surface side of the support plate 46a. On the surface of the support plate 46a opposite to the elastic member 46b, a substantially hexagonal column-shaped connecting pin 41 is erected in a posture perpendicular to the surface. The connecting pin 41 is pivotally supported by a bearing portion 43 d provided at one end of the first headband 43. As a result, the head pad 46 can rotate around the connecting pin 41.

  The head pad 47 includes a plate-like support plate 47a and an elastic member 47b having an arcuate cross section provided on one surface of the support plate 47a. A shaft member 47c is provided on the surface of the support plate 47a opposite to the elastic member 47b. The shaft member 47 c is pivotally supported by a bearing portion 43 e provided at the tip of the first headband 43. Thereby, the head pad 47 can rotate around the shaft member 47c.

  With this configuration, the second headband 44 is urged to rotate in a direction away from the first headband 43 by the rotation mechanism 56, while being within a predetermined angle with respect to the first headband 43 by the rotation mechanism 57. The rotation is restricted to the range. As a result, the second headband 44 is held at a predetermined angle with respect to the first headband 43. Therefore, the rotation mechanisms 56 and 57 function as an angle holding mechanism in the headband 40.

Next, the operation of the headband in the HMD 1 having the above configuration will be described with reference to FIGS. Hereinafter, the right side for the user is referred to as “the right side of the user”. Further, the left side for the user is referred to as “the left side of the user”. The upper side for the user is referred to as “the upper side of the user”. Further, the lower side for the user is referred to as “the lower side of the user”. The clockwise direction for the user is referred to as “clockwise direction”.
Further, the counterclockwise direction for the user is referred to as “counterclockwise direction”.

  As shown in FIG. 5, the HMD 1 of the present embodiment can be used by switching between the right-eye observation form shown in FIG. 5A and the left-eye observation form shown in FIG. Here, the right-eye observation form is a form in which the user observes the display unit 60 with the right eye. The left-eye observation form is a form in which the user observes the display unit 60 with the left eye. Hereinafter, the form shown in FIGS. 5A and 5B, that is, the form in which the finder opening 67 is arranged horizontally when viewed from the user is referred to as a first display mode.

  For example, to switch from the right-eye observation mode to the left-eye observation mode, first, the display body 20 connected to the headband 40 is moved around the rotation axis of the headband hinge 32 (see FIG. 2) (A in FIG. 5). Direction). Further, in the headband 40, the position of the second headband 44 and the position of the first headband 43 are obtained by swinging the second headband 44 around the axes of the rotation mechanisms 56 and 57 (see FIG. 4). Swap the context. By this operation, as shown in FIG. 5B, the left-eye observation form in which the display main body 20 is arranged on the left eye side of the user and the second headband 44 is arranged on the occipital side of the user is performed. Can be switched.

  In the right-eye observation mode, the eyeball illumination lights 166a and 166b in the vicinity of the right eye are turned on, and the eyeball illumination lights 166c and 166d are turned off. On the other hand, in the left-eye observation mode, the eyeball illumination lights 166c and 166d in the vicinity of the left eye are turned on, and the eyeball illumination lights 166a and 166b are turned off.

In the present embodiment, as shown in FIG. 4, the arc height r <b> 1 of the first headband 43 is smaller than the arc height r <b> 2 of the second headband 44. As a result, even if the second headband 44 is moved so as to intersect the first headband 43, the front and rear can be switched smoothly without interfering with each other.
The arc height r1 shown in FIG. 4 is the maximum value of the distance from the center position C of the rotation axis L of the rotation mechanisms 56 and 57 arranged coaxially to the first headband 43. The arc height r2 is the maximum value of the distance from the center position C of the rotation axis L to the second headband 44.

  The arc height r1 of the first headband 43 may be larger than the arc height r2 of the second headband 44, and the second headband 44 may be swung inside the first headband 43.

FIG. 6 is a perspective view showing a stereo earphone attached to the head mounted display according to the present embodiment.
Stereo earphone 100 includes a connector 101, a cable 102, a first speaker 103, a second speaker 104, a sound collecting microphone 105, and a plurality of clips 106.

  The connector 101 is provided at one end of the cable 102. The connector 101 is a general 4-pole φ3.5 mm mini plug. The breakdown of the four poles is the sound collection microphone 105, the first speaker 103, the second speaker 104, and the ground (GND). The cable 102 is bifurcated in the vicinity of the connector 101, and a first speaker 103 is provided at the end of the branched cable. A second speaker 104 and a sound collecting microphone 105 are provided at the other end of the cable 102. The plurality of clips 106 are arranged on the cable 102 at a predetermined interval.

  The stereo earphone 100 is used by connecting the connector 101 to the audio connector 26 of the display body 20. When the stereo earphone 100 is connected, the ear speaker 23 of the display body 20 and the call microphone 37 of the display unit 60 are invalidated. Further, the front speaker 70 of the display unit 60 is also invalidated as necessary. Then, the first speaker 103, the second speaker 104, and the sound collecting microphone 105 of the stereo earphone 100 are activated.

  The first speaker 103 of the stereo earphone 100 is attached to the ear of the user on the side where the display main body 20 is arranged, and the second speaker 104 is attached to the ear on the opposite side to the first speaker 103. At this time, the cable 102 can be fixed to the second headband 44 by the clip 106.

Further, stereo recording can be performed by the sound collecting microphone 105 of the stereo earphone 100 and the sound collecting microphone 24 provided on the outer surface of the housing 21 </ b> A in the display main body 20.
For example, if the display main body 20 is arranged on the right eye side as shown in FIG. 5A, the sound collecting microphone 24 of the display main body 20 collects sounds on the right side of the user and is attached to the left ear. The sound collecting microphone 105 collects the sound on the left side of the user.
In the right-eye observation mode, the right channel sound is output from the first speaker 103 of the stereo earphone 100, and the left channel sound is output from the second speaker 104.

  On the other hand, when the display main body 20 is arranged on the left eye side of the user as shown in FIG. 5B, the sound collecting microphone 24 of the display main body 20 collects the sound on the left side of the user and puts it in the right ear. The attached sound collecting microphone 105 collects sound on the right side of the user. In the left-eye observation mode, left channel sound is output from the first speaker 103 of the stereo earphone 100, and right channel sound is output from the second speaker 104.

Next, lighting and extinguishing of the eyeball illumination lights 166a to 166d will be described.
FIG. 7 is a diagram for explaining turning on and off of the eyeball illumination light 166 according to the present embodiment. FIG. 7A is a diagram for explaining turning on and off of the eyeball illumination light 166 in the right-eye observation mode. FIG. 7B is a diagram for explaining turning on and off of the eyeball illumination light 166 in the left-eye observation mode. In FIG. 7A and FIG. 7B, the front direction of the user's face is denoted as HZ.

  In the example shown in FIG. 7A, the display unit 60 is disposed obliquely below the right eye. Further, the front direction HZ and the normal line of the screen of the display unit 60 form an angle α. The HMD 1 selects an eyeball illumination light on the side close to the user's eye among the eyeball illumination lights 166a to 166d, and turns on the selected eyeball illumination light. On the other hand, the HMD 1 turns off the eyeball illumination lights other than the selected one, that is, the eyeball illumination lights far from the user's eyes.

  In FIG. 7A, the eyeball illumination lights 166a and 166b are disposed below the front direction HZ by an angle α1. On the other hand, the eyeball illumination lights 166c and 166d are arranged at an angle α2 below the front direction HZ. Here, the angle α1 is smaller than the angle α, and the angle α2 is larger than the angle α. That is, the eyeball illumination lights 166a and 166b are disposed closer to the user's right eye than the eyeball illumination lights 166c and 166d.

  In this case, the HMD 1 turns on the eyeball illumination lights 166a and 166b and turns off the eyeball illumination lights 166c and 166d. In this lighting state, the image sensor 72 images around the right eye of the user. Hereinafter, an image captured around the right eye of the user is referred to as a “right eye peripheral image”.

  The peripheral image of the right eye may include an image of the right eye, an image of the right eyebrow, and an image of a part of the face (head, hair, etc.). The peripheral image of the left eye may include a left eye image, a left eyebrow image, and a part of the face (head, hair, etc.). Hereinafter, the peripheral image of the right eye and the peripheral image of the left eye are collectively referred to as “eye peripheral image”.

  In the example shown in FIG. 7A, the eyeball illumination lights 166c and 166d far from the user's right eye may not be able to illuminate the user's right eye due to the user's cheek getting in the way. Therefore, the HMD 1 turns off the eyeball illumination lights 166c and 166d far from the user's right eye. Thereby, HMD1 can reduce power consumption.

  On the other hand, in FIG. 7B, the eyeball illumination lights 166c and 166d are arranged on the lower side by an angle α3 from the front direction HZ. On the other hand, the eyeball illumination lights 166c and 166d are disposed at an angle α4 below the front direction HZ. Here, the angle α3 is smaller than the angle α, and the angle α4 is larger than the angle α. That is, the eyeball illumination lights 166a and 166b are disposed closer to the user's left eye than the eyeball illumination lights 166c and 166d.

  In this case, the HMD 1 turns on the eyeball illumination lights 166c and 166d and turns off the eyeball illumination lights 166a and 166b. In this lighting state, the image sensor 72 images around the left eye of the user. Hereinafter, an image in which the periphery of the user's left eye is captured is referred to as a “left eye peripheral image”.

  In the example shown in FIG. 7B, the eyeball illumination lights 166a and 166b far from the user's left eye may not be able to illuminate the user's left eye due to the user's cheek getting in the way. Accordingly, the HMD 1 turns off the eyeball illumination lights 166a and 166b on the side far from the user's left eye. Thereby, HMD1 can reduce power consumption.

  That is, the HMD 1 turns on an eyeball illumination light that can illuminate the user's eyes from the front, among the eyeball illumination lights 166a to 166d, according to the positional relationship between the user and the display unit 60, and other eyeballs. Turn off the illumination light.

Next, the configuration of the display unit 60 and the optical system will be described.
FIG. 8 is a cross-sectional view of the display unit 60 for explaining the configuration and optical system of the display unit 60 according to the present embodiment. As shown in FIG. 8, the display unit 60 of this embodiment includes a backlight 62, a display panel (display unit) 63, a first prism 65a, a second prism 65b, a reflection mirror 66, a finder opening 67, and an imaging lens. 71, an image sensor 72, and a quarter wavelength plate 75. In FIG. 8, symbols H, I, J, K, L, M, and N each represent a light beam.

First, the configuration of the display unit 60 will be described.
The prism 65 has a configuration in which a substantially triangular first prism 65a and a second prism 65b are bonded to each other on a top view (Z-axis view). A display panel 63 made of a liquid crystal panel is provided at a position facing one of the other two surfaces of the first prism 65a other than the bonding surface. A backlight 62 that transmits and illuminates the display panel 63 is disposed on the back surface of the display panel 63. A reflection mirror 66 is disposed at a position facing the other surface of the first prism 65a. The reflection mirror 66 is located substantially in front of the finder opening 67.

  Of the two surfaces other than the bonding surface of the second prism 65 b, one surface is a viewfinder eyepiece disposed in the viewfinder opening 67. An imaging element 72 is disposed opposite to the other surface of the second prism 65b via the imaging lens 71.

Next, the optical system of the display unit 60 will be described.
In the display unit 60, the image displayed on the display panel 63 is emitted from the finder opening 67 through the first prism 65a, the reflection mirror 66, and the second prism 65b, and is observed by the user. Further, the image looking at the viewfinder opening 67 is imaged on the image sensor 72 via the second prism 65 b and the imaging lens 71.
The captured image acquired via the image sensor 72 is used for analysis of the direction of the user's line of sight, blinking, facial expression, and posture of the display unit 60. The captured image includes a peripheral image of the eye. Note that the backlight 62 may also be used as auxiliary illumination when an image around the eye is captured by the image sensor 72.

  The light emitted from the backlight 62 passes through the display panel 63 and enters the first prism 65a. At this time, due to the characteristics of the liquid crystal of the display panel 63, the light irradiated by the backlight 62 is polarized, and the S-polarized light of the polarized light beam H is incident on the first prism 65a.

  The light beam H is totally reflected by the influence of the air layer provided on the back surface 65d of the first prism 65a, and enters the joint surface 65c of the first prism 65a and the second prism 65b as the light beam I. Here, a polarizing reflection film that reflects S-polarized light and transmits P-polarized light is deposited on the joint surface 65c of the two prisms. Here, since the light ray H is S-polarized light, it is reflected by the joint surface 65c to become a light ray J. Since the light beam J is incident on the back surface 65d at an incident angle of 0 degree, it passes through the back surface 65d and enters the quarter-wave plate 75.

The light beam J passes through the quarter wave plate 75 after passing through the air layer, and becomes circularly polarized light clockwise with respect to the traveling direction. The light beam K reflected by the reflecting mirror 66 becomes counterclockwise circularly polarized light by reflection. Then, the light K passes through the quarter-wave plate 75 again and becomes P-polarized light.
Since the light K has been converted to P-polarized light, it can pass through the joint surface 65c, and thus can pass through the finder opening 67. With such a configuration, the user can observe the image data displayed on the display panel 63 through the viewfinder opening 67.

  On the other hand, the light beam L incident from the finder opening 67 is reflected by the joint surface 65c and converted into an S-polarized light beam M. The light ray M is totally reflected by the air layer on the front surface 65e of the second prism 65b to become a light ray N. The light beam N is collected by the imaging lens 71, and the collected light beam N enters the image sensor 72. With such a configuration, when the finder opening 67 is disposed in front of the user's eye, a peripheral image (optical image) of the user's eye is formed on the image sensor 72.

Next, an image observed by the user through the finder opening 67 and an image captured by the image sensor 72 in the right-eye observation mode will be described.
In FIGS. 10, 13, and 15, the reflecting mirror 66 (see FIG. 8) is replaced with a convex lens so as to be optically equivalent. 10, 13, and 15, the alternate long and short dash line CT represents the optical axis of the imaging lens 71.

Hereinafter, a state in which the display unit 60 is disposed within a predetermined distance from the user's eyes is referred to as a “first observation state”.
FIG. 9 is a diagram for explaining a difference in display area in each of the first to third observation states.
In the first observation state, since the display unit 60 is disposed within a predetermined distance from the user's eyes, the entire display area of the display panel 63 can be observed by the user. Therefore, the display panel 63 can display an image in the entire display area. That is, the entire display area of the display panel 63 becomes the display area S.

  Hereinafter, the horizontal length of the display area S is expressed as Sw. Also. The vertical length of the display area S is expressed as Sh. In the following description, as an example, the description will be continued assuming that Sw is 800 pixels and Sh is 600 pixels. In addition, the vertical length of the display area T is expressed as Th. Further, the horizontal length of the display area T is denoted as Tw. Further, the vertical length of the display area U is denoted as Uh. Further, the horizontal length of the display area U is denoted as Uw.

  FIG. 10 is a diagram illustrating the observation range and the imaging range in the first observation state. When the user's eyes are in the observation range 67 a, the user can observe the display area S corresponding to the entire display area of the display panel 63 through the finder opening 67. The observation range 67a is in a range from the finder opening 67 to the distance P on the optical axis of the reflection mirror 66 (convex lens in FIG. 10). On the other hand, the imaging range 67c that can be imaged by the imaging device 72 includes an observation range 67a. That is, even when the user's eyes are not in the observation range 67a, the imaging element 72 can capture the user's eyes including eyebrows if the user has the eyes in the shooting range 67c.

  FIG. 11 is a diagram illustrating an example of a captured image in the first observation state. In the approximate center of FIG. 11, the eye of the user is imaged by the imaging element 72. In this case, as described with reference to FIG. 7A, the eyeball illumination lights 166a and 166b are turned on. In the case of the right eye observation mode, the + Z side in FIG. 11 is the vertically upward direction, the −Z side is the vertically downward direction, the + X side is the left side of the user, and the −X side is the right side of the user. In the case of the left-eye observation form, the display unit 60 is inverted 180 degrees, so that the peripheral image of the eye is inverted horizontally and vertically as compared with the right-eye observation form.

  In FIG. 11, reference numeral 601 </ b> R indicates a right eye image, and reference numeral 602 </ b> R indicates a right eyebrow image. Reference numeral 604A represents a bright spot image by the eyeball illumination light 166a (see FIG. 2), and reference numeral 604B represents a bright spot image by the eyeball illumination light 166b (see FIG. 2). Reference sign L41 is a sign indicating the width of the user's eyes in the X-axis direction. For example, when the resolution of the image sensor 72 is 640 pixels in the horizontal direction (X-axis direction) in the first observation state, the eye width L41 in the image is about 350 pixels.

In FIG. 11, since the distance between the user's eyes and eyebrows is equal to or smaller than the width L41, it can be said that an appropriate range is captured in the captured image. Thus, it is effective to photograph the user's eyes from below.
Thus, in the first observation state, the finder opening 67 does not get in the way when the user looks at the front. As will be described later, in the first observation state, the HMD 1 can detect the user's line-of-sight direction based on the movements of the eyebrows, the eyes, and the eyeballs captured in the image captured by the image sensor 72.

Hereinafter, a state in which the user's eyes are separated from the display unit 60 as compared with the first observation state is referred to as a “second observation state”.
Next, a 2nd observation state is demonstrated using FIG.9, FIG12 and FIG.13.

FIG. 12 is a diagram illustrating an example of a captured image in the second observation state. Reference numeral 601R1 represents a right eye image, and reference numeral 602R1 represents a right eyebrow image.
Reference numeral 604C represents a bright spot image by the eyeball illumination light 166a, and reference numeral 604D represents a bright spot image by the eyeball illumination light 166b. Reference sign L71 is a sign indicating the width of the user's eyes in the X-axis direction.
In the second observation state, since the user's eyes are separated from the display unit 60, an image of the user's eyes is captured smaller than in the first observation state. When the width L71 of the user's eye in the captured image is equal to or smaller than the width L41 of the user's eye in the first observation state (for example, 300 pixels or less), the HMD 1 determines that the user is taking his eyes off the display unit 60. To do.

  FIG. 13 is a diagram illustrating an observation range and an imaging range in the second observation state. In FIG. 13, the user's eye is located on the optical axis of the reflection mirror 66 (convex lens in FIG. 13) at a position within the distance P2 from the finder opening 67 and within the distance P2, and the observation range 67b in the first observation state. There is no distance (within the distance P from the finder opening 67 on the optical axis). For this reason, the user can observe only a part of the display area S (see FIG. 9).

  Therefore, in the second observation state, the HMD 1 displays an image reduced at the reduction rate η in the display area T (see FIG. 9). In the second observation state, the reduction rate η (ratio with the first observation state) of the image displayed in the display region T is expressed by the following equation (1).

  η = L71 / L41 (1)

  For example, when the user's eye width L41 (see FIG. 11) is 350 pixels and L71 (see FIG. 12) is 220 pixels, HMD1 is about 0.63 times (= (L71 / L41) = (Th / Sh) = (Tw / Sw)), an image reduced in length and width is displayed in the display area T of the display panel 63. The entire image (optical image) displayed in the display area T reaches the user's eyes without being blocked by the edge of the finder opening 67. In the second observation state, the HMD 1 may increase the luminance of the eyeball illumination light 166 compared to the luminance in the first observation state.

Hereinafter, a state in which the relative position between the display unit 60 and the user's eye is shifted in either the left or right direction compared to the second observation state is referred to as a “third observation state”.
Next, a 3rd observation state is demonstrated using FIG.9, FIG14 and FIG.15.
FIG. 14 is a diagram illustrating an example of a captured image in the third observation state. In the third observation state, it is assumed that, as an example, an image of the user's eye is captured with a vector V1 shifted from the vicinity of the center of the image in the lower right direction as compared with the second observation state.

Reference numeral 601R2 represents a right eye image, and reference numeral 602R2 represents a right eyebrow image. Reference numeral 604E represents a bright spot image by the eyeball illumination light 166a, and reference numeral 604F represents a bright spot image by the eyeball illumination light 166b.
Also, 605A indicated by a one-dot chain line represents an outline image of an eyeball that is captured when the position of the user's eye is not displaced. Reference numeral 605B represents an image of the eyeball when the user's eye is at the center of the optical axis.

  FIG. 15 is a diagram illustrating an observation range and an imaging range in the third observation state. In FIG. 15, the user's eye eye is at a position within the observation range 67e, that is, at a distance P or more from the finder opening 67 and shifted inward from the optical axis CT by the X component amount (= V1x) of V1. exist. Since the user's eye in the observation range 67e is not in the observation range 67b (within the distance P from the finder opening 67 on the optical axis) in the first observation state, one of the display areas S (see FIG. 9). Only the part can be observed.

  Therefore, in the third observation state, the HMD 1 reduces the image at a predetermined reduction rate (= (Uh / Sh) = (Uw / Sw)) so that the entire display image can be observed from the user. Further, the HMD 1 displays the reduced image in the display area U (see FIG. 9). Here, the HMD 1 determines the position of the display area U in the display area S (see FIG. 9) according to the positional deviation amount and direction of the user's eyes. In the third observation state, the display area U is defined at the upper right of the display area S.

  Here, it is assumed that the number of recording pixels of the imaging system including the imaging lens 71 and the imaging element 72 is, for example, 640 pixels wide and 480 pixels vertically. Further, it is assumed that the angle of view of the imaging system including the imaging lens 71 and the imaging element 72 is 60 degrees lateral and 45 degrees longitudinal. In this case, the angle of view per pixel is about 0.094 degrees.

  Therefore, if the amount of horizontal displacement of the user's eye position (X component amount of the vector V1) is, for example, 50 pixels, the user's eye position is 4.7 degrees (= 50 pixels × 0.094). Degree). If the viewing angle in the horizontal direction (Sw: 800 pixels) of the display area S (see FIG. 9) is, for example, 35 degrees, the HMD 1 is 108 pixels (= Sw × 4. The position of the display area U is determined to the right in the horizontal direction by 7 degrees / 35 degrees. Similarly, in the vertical direction, the HMD 1 defines the display area U at a position shifted by a predetermined number of pixels.

  14 and 15, the HMD 1 reduces the image until the horizontal direction Uw of the display area U is 500 pixels and the vertical direction Uh is 375 pixels, 108 pixels to the right in the horizontal direction from the center of the display area S, and A display area U is defined at a position shifted vertically by the predetermined number of pixels corresponding to the z component amount (= V1z) of the vector V1. In the display area U, an image reduced at the same reduction ratio as the display area U is displayed. Thereby, even when the user's eye eye is shifted from the optical axis, the HMD 1 can display the image so that the entire image can be seen.

  The HMD 1 extracts, for example, an eye contour image from the captured image, and detects an eye displacement amount and direction (vector V1) based on the extracted position of the eye contour image. Here, the detection of the eye position deviation amount and direction may be performed by extracting an eyeball image from the captured image and detecting the eye position deviation amount and direction based on the extracted deviation amount of the outline image of the eyeball. Good. Further, the HMD 1 detects the vector V1 by appropriately turning on the eyeball illumination lights LEDs 166a to 166d (see FIG. 2) and measuring spots on the eyeball of the user projected by the eyeball illumination lights. You may do it.

  As described above, even when the user's eyes are separated from the display unit 60 or deviated from the optical axis, in the HMD 1, the user can select the entire area of the image according to the positional relationship between the user and the display unit 60. An image can be displayed on the display unit 60 so that the image can be observed. Note that the timing for determining the display area may be before or after calibration processing (described later) for detecting a point of interest.

Next, arrangement of the display unit 60 and imaging of the user's eyes will be described.
FIG. 16 is a diagram illustrating an example in which the display unit 60 is arranged in front of the eyes and a peripheral image including the eyes is captured. FIG. 17 is a diagram illustrating an example in which the display unit 60 is arranged on the lower side of the eye according to the present embodiment and the eye is imaged from below.

  In FIG.16 and FIG.17, the code | symbol L11 represents the distance from a user's lower eyelid to eyebrows. Further, in FIGS. 16 and 17, the distance L11 is 40 [mm]. 16 and 17, when the user's face is viewed from the side, the vertical direction is referred to as the Z-axis direction, and the front-back direction is referred to as the Y-axis direction.

  As shown in FIG. 16, when the display unit 60 is arranged in front of the eyes, in order to image from the user's lower eyelid to eyebrows, the display unit 60 is captured until a width L11 or more (for example, width L31) can be imaged. It is necessary to keep the imaging lens 71 away from the user's face. Here, the width L31 is, for example, 70 [mm]. When the range of the width L31 is imaged, the distance L21 needs to be separated from the user's eye to the center of the imaging lens 71. Here, the distance L21 is, for example, 85 [mm].

  The distance L21 also includes the optical path lengths of the first prism 65a, the second prism 65b, etc. in the optical system shown in FIG. Assuming that the total optical path length of the first prism 65a, the second prism 65b, etc. is 20 [mm], the display unit 60 is only 65 (= 85-20) [mm] in the Y-axis direction from the user's face. It is necessary to keep away.

  Therefore, when the display unit 60 is disposed in front of the eyes, the display unit 60 needs to be disposed at a position away from the user's face in order to capture an image from the user's lower eyelid to the eyebrows.

  On the other hand, as shown in FIG. 17, when the display unit 60 is arranged on the lower side of the eye, in order to image the range of the distance L11 from the user's lower eyelid to the eyebrows, the imaging range of the width L61 is captured There is a need. Here, since the image is picked up from below, the width L61 is shorter than the width L31 shown in FIG. For example, the width L61 is 20 [mm]. An imaging distance (distance L51) necessary for securing the imaging range 20 [mm] is, for example, 45 [mm].

  Similarly to FIG. 16, when the total optical path length of the first prism 65a, the second prism 65b, etc. is 20 [mm], the display unit 60 is moved away from the user's face by 25 [mm]. When the display unit 60 is inclined by an angle β with respect to the Y-axis direction, the distance in the Y-axis direction from the face to the imaging lens 71 is a value obtained by multiplying cos β by 45 [mm].

  In the HMD 1, the display unit 60 arranged under the eyes can capture a peripheral image of the eye (see FIG. 17), so that the HMD 1 is compared with the case where the display unit 60 is arranged in front of the eyes (see FIG. 16). See), and the distance from the face to the display unit 60 is short. Thereby, HMD1 does not enlarge. Moreover, since the display part 60 is not arrange | positioned in front of a user, HMD1 does not block the front view of a user. Furthermore, since the display part 60 can also be arrange | positioned in the vicinity of a user's face, the display part 60 does not become obstructive, for example in a narrow work space.

  In addition, the HMD 1 lowers the luminance of the eyeball illumination light 166 when the display unit 60 is disposed under the eyes as compared with the case where the display unit 60 is disposed in front of the eyes (see FIG. 16). May be. Thereby, HMD1 can reduce power consumption.

Next, adjustment of the installation angle of the image sensor 72 that accompanies switching between the right eye observation mode and the left eye observation mode will be described.
FIG. 18 is a diagram illustrating the angle of view and the selection range of the image sensor 72 according to the present embodiment. FIG. 18A is a diagram for explaining the angle of view and the selection range of the image sensor 72 in the right-eye observation mode.
FIG. 18B is a diagram for explaining the angle of view and the selection range of the image sensor 72 in the left-eye observation mode. In FIGS. 18A and 18B, the first headband 43, the second headband 44, the apparatus main body 21 and the like are not shown.

  FIG. 19 is a diagram illustrating an imaging area and a selection area according to the present embodiment. FIG. 19A is a diagram illustrating an imaging region by the imaging element 72 (see FIG. 3) when the headband hinge 32 (see FIG. 2) is swung. FIG. 19B is a diagram for explaining a selection area selected from the imaging area. Here, the symbol P is a symbol indicating a range in which the image sensor 72 can capture an image. The image sensor 72 has, for example, an aspect ratio of 4 to 3.

  In the case of the right-eye observation mode, in the first display mode (see FIG. 5A), the image sensor 72 is provided in the display unit 60 so that the aspect of the imaging region is vertically long. In this case, as shown in FIGS. 18 (A) and 19 (B), the image sensor 72 is a region indicated by a symbol Q among regions indicated by a symbol P (hereinafter referred to as “imaging region P”). The image data of only the range to be imaged is taken in (hereinafter referred to as “selected area Q”). Thereby, the image sensor 72 can appropriately capture the image data of the right eye including the eyebrows in the right eye observation mode.

  On the other hand, in the case of the left-eye observation mode, in the first display mode (see FIG. 5B), the image sensor 72 is provided in the display unit 60 so that the aspect of the imaging region is vertically long. In this case, as shown in FIGS. 18B and 19B, the imaging element 72 is imaged in an area indicated by a symbol R (hereinafter referred to as “selection area R”) in the imaging area P. Image data of only the range. Thereby, the image sensor 72 can appropriately capture the image data of the left eye including the eyebrows in the left-eye observation mode.

  Moreover, HMD1 determines appropriately whether it is a right eye observation form or a left eye observation form based on the picked-up image by the image pick-up element 72 instead of the detection value (after-mentioned) which the acceleration sensor 132 detected. Can do. A method for determining whether the observation mode is the right eye observation mode or the left eye observation mode will be described later.

FIG. 20 is a diagram illustrating a mechanism for mechanically selecting a selection range according to the present embodiment.
A rotation shaft 72 b is fixed to the main body 72 a of the image sensor 72. A dial 72c is provided for the rotating shaft 72b. The user can tilt the imaging device 72 about the rotation shaft 72b by rotating the dial 72c within the rotatable range (D direction in FIG. 20).

  When the imaging lens 71 is tilted in the upward direction, the selection area Q (see FIG. 19B) is determined as an upper area in the imaging area P. Thereby, the image sensor 72 can image the user's eyes and eyebrows in the imaging region Q. On the other hand, when tilted in the direction toward the bottom of the imaging lens 71, the selection region R (see FIG. 19B) is defined as a lower region of the imaging region P. Thereby, the image sensor 72 can image the user's eyes and eyebrows in the imaging region R.

  The HMD 1 can also finely adjust the position of the imaging area Q or the selection area R with the dial 72c. When a captured image by the image sensor 72 is displayed on the display unit 60, the user observes the captured image displayed on the display unit 60 and determines the image sensor according to the positional relationship between the display unit 60 and the eyes. 72 imaging areas can be selected. As a result, the HMD 1 can increase the accuracy of calibration for HMD 1 alignment (described later) and gaze point detection using a captured image that is captured so that the range is appropriate.

Next, functional blocks of the HMD 1 will be described.
FIG. 21 is a functional block diagram of the head mounted display 1 according to the present embodiment.
The HMD 1 includes various electric circuits centering on the processing unit 123. The processing unit 123 is, for example, a CPU (Central Processing Unit), and is connected to various circuits of the HMD 1 and comprehensively controls the HMD 1. The processing unit 123 performs, for example, “detection processing on left and right eyes”, “mounting guide processing”, “eyeball illumination light selection processing”, “gazing point calibration processing”, “business logging processing”, and the like. Do. Each process performed by the processing unit 123 will be described later.

The battery 33 supplies power to the power supply circuit 120.
The power supply circuit 120 supplies the power supplied from the battery 33 to each unit of the HMD 1 under the control of the processing unit 123.

  In the present embodiment, the processing unit 123 includes an encoder 129 and a decoder 121, a power supply circuit 120, an operation switch 30, a flash memory 122, a BL driver 126, a BT communication circuit 130, a WiFi communication circuit 131, an acceleration sensor 132, and a geomagnetic sensor. 133, front light 68, 3G / LTE communication circuit 138, laser transmitter 73, angular velocity sensor 134, GPS sensor (positioning sensor) 135, temperature / humidity sensor 136, heart rate sensor 137, memory 127, main switch 28, touch switch 34 The headband sensor 151 and the eyeball illumination lights 166a to 166d are connected.

  The encoder 129 encodes (encodes) the audio signal and the image signal into audio data and image data of a predetermined method. Connected to the encoder 129 are a camera 64, an image sensor 72, a sound collection microphone 24, a call microphone 37, an audio connector 26, and a video connector 27.

  The encoder 129 includes an audio signal input from the sound collection microphone 24 and the call microphone 37, an image signal input from the camera 64, an audio signal input from the audio connector 26, an image signal input from the video connector 27, and Then, an image signal of the image sensor 72 that captures a peripheral image of the eye is input. The audio signal and image signal input to the encoder 129 are encoded into audio data and image data, and then input to the processing unit 123. The input audio data and image image data are used for a reproduction operation by the processing unit 123 or recorded in the flash memory 122.

  The decoder 121 decodes (decodes) audio data and image data into audio signals and image signals. An LCD driver 125, a speaker amplifier 162, an audio connector 26, and a video connector 27 are connected to the decoder 121. The LCD driver 125 is a drive control device for a liquid crystal panel, and is connected to the display panel 36 and the display panel 63. The speaker amplifier 162 is a device that amplifies an audio signal and outputs it to the speaker, and is connected to the ear speaker 23 and the front speaker 70.

  When reproducing the image data, the image data recorded in the flash memory 122 or the image data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The image data input to the decoder 121 is decoded into an image signal and then supplied to the display panels 36 and 63 via the LCD driver 125. Then, an image based on the image data is displayed on the display panel 36 or the display panel 63 to which the image signal is input. The image signal output from the decoder 121 to the video connector 27 is output to an external device via the video connector 27.

  When displaying an image, the processing unit 123 turns on the backlight 35 for the display panel 36 and the backlight 62 for the display panel 63 as necessary. The BL driver 126 is a drive control device for the backlight, and is connected to the backlight 35 and the backlight 62. The processing unit 123 transmits a drive signal to the BL driver 126, and the BL driver 126 lights the backlights 35 and 62 individually.

  When reproducing the audio data, the audio data recorded in the flash memory 122 or the audio data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The audio data input to the decoder 121 is decoded into an audio signal and then output to one or both of the ear speaker 23 and the front speaker 70 via the speaker amplifier 162. Then, sound is output from the ear speaker 23 or the front speaker 70 to which the sound signal is input. The audio signal output from the decoder 121 to the audio connector 26 is output to the stereo earphone 100 via the audio connector 26.

  In the case of this embodiment, the ear speaker 23 and the front speaker 70 are assumed to use monaural sound, and the ear speaker 23 and the front speaker 70 emit a sound in which left and right audio signals are synthesized.

  On the other hand, when an audio signal is output to the stereo earphone 100, the sound of the left channel or the right channel of the first speaker 103 and the second speaker 104 is output. Here, since the HMD 1 of this embodiment is for both left and right, the channel of the audio signal to the stereo earphone 100 is switched according to the mounting position. That is, when the display body 20 is attached to the right eye, the right channel sound is output to the first speaker 103 and the left channel sound is output to the second speaker 104. When the display body 20 is attached to the left eye, the left channel sound is output to the first speaker 103 and the right channel sound is output to the second speaker 104.

The memory 127 stores a control program executed by the processing unit 123.
When the user operates the main switch 28 for turning on / off the entire power source, the operation switch 30 for performing a pointing operation in the screen, or the touch switch 34 for performing various operations by a touch operation, these switches are used for the operation. A control signal is output to the processing unit 123. The processing unit 123 detects an operation based on the control signal, and executes an operation defined in the control program.

The BT communication circuit 130 is a communication circuit for performing Bluetooth (registered trademark) communication with other devices. The WiFi communication circuit 131 is a wireless LAN communication (IEEE) with other devices.
It is a communication circuit for performing 802.11). The 3G / LTE communication circuit 138 is a communication circuit for performing mobile communication with other devices.

  The acceleration sensor 132 is used for detecting the inclination of the HMD 1. The acceleration sensor 132 is a triaxial sensor, for example, and detects gravitational acceleration. The acceleration sensor 132 is attached, for example, in the apparatus main body 21.

  Hereinafter, the coordinates of the acceleration sensor 132 are as follows: when the user standing upright on the ground is wearing the HMD 1 on the head, the vertical direction for the user is the Z-axis direction, the horizontal direction is the X-axis direction, and the front-back direction is the Y-axis direction. To do. The acceleration sensor 132 is attached so that the detected value in the Z-axis direction becomes a negative value in the case of the right-eye observation form, and the detected value in the Z-axis direction becomes a positive value in the case of the left-eye observation form. .

The geomagnetic sensor 133 is used for detecting the direction of the HMD 1. An angular velocity sensor (gyro sensor) 134 is used to detect rotation of the HMD 1. The GPS sensor 135 is used for positioning detection using GPS (Global Positioning System). The temperature / humidity sensor 136 is used to detect the temperature and humidity of the environment.
The heart rate sensor 137 contacts the user's cheek, detects the user's heart rate, and outputs the detected detection value to the wearing processing unit 311. For example, the heart rate sensor 137 irradiates infrared rays, measures the heart rate based on the reflected wave of the infrared rays, and outputs information indicating the measured heart rate to the control unit 302 as a detection value. Further, the heart rate sensor 137 detects that the HMD 1 is within several centimeters of the vicinity of the head based on the infrared reflected wave, and outputs it to the control unit 302.

  The headband sensor 151 is a sensor that detects the connecting pin 41 provided on the headband 40. Specifically, the headband sensor 151 detects a state where the connection pin 41 is inserted into the connection hole 31, that is, a state where the headband 40 and the display main body 20 are connected, and processes a detection signal indicating the detection result. Output to the unit 123. Note that the headband sensor 151 may detect the connecting pin 41 provided on the helmet on the assumption that the display body 20 is mounted on a helmet or the like instead of the headband 40. good.

  FIG. 22 is a functional block diagram of the processing unit 123 according to the present embodiment. The processing unit 123 includes an input detection unit 301, a control unit 302, a storage unit 303, an eyeball illumination unit 304, an acoustic control unit (notification unit) 305, an image switching unit 306, an image display unit 307, a transmission / reception unit 308, and an action history registration. Unit 309 and an observation availability determination unit 310. In FIG. 22, the WiFi communication circuit 131 and the 3G / LTE communication circuit 138 (see FIG. 21) are collectively referred to as “communication interfaces (131, 138)”.

  The input detection unit 301 determines the type of operation performed on the touch switch 34, and outputs a control signal based on the determined result to the control unit 302, the image switching unit 306, and the transmission / reception unit 308. The types of operations performed on the touch switch 34 are a transmission instruction, a reception instruction, and the like. When a predetermined area on the touch switch 34 is touched, the input detection unit 301 determines the type of these operations based on the touched area.

  The control unit 302 determines the right-eye observation form or the right-eye observation form based on at least one of the detection value detected by the acceleration sensor 132 and the image captured by the image sensor 72. Here, with respect to the image captured by the image sensor 72, the control unit 302, for example, the relative position between the eyebrows and the eye, the behavior of eyelids in blinking, the eyebrow contour, the eye contour, and the lowering of the left and right corners of the eye Based on the above, it is determined whether the user's eye shown in the captured image is an image of the left or right eye (right eye observation form or right eye observation form).

  The control unit 302 controls lighting and extinguishing of the eyeball illumination light 166 based on the result of determining whether the user's eye shown in the captured image is an image of the left or right eye. Further, the control unit 302 controls the sound output to the stereo earphone 100 connected to the audio connector 26 based on the determined result. Further, the control unit 302 controls an image projected from the display unit 60 based on the determined result.

  The storage unit 303 stores a count value i (i is an integer of 0 or more) for counting the inspection position corresponding to the inspection target apparatus. The inspection target device is a device to be inspected by the user. Information based on the positioning information measured by the GPS sensor 135 is stored in the storage unit 303. The storage unit 303 stores a user ID. The storage unit 303 stores parameters for operation based on the line of sight generated by the control unit 302.

  The eyeball illumination unit 304 drives the eyeball illumination lights 166a to 166d to be turned on or off based on the control of the control unit 302 or the action history registration unit 309.

  The sound control unit 305 receives sound data output from the encoder 129 (user HMD1 sound data) and a reception signal output from the transmission / reception unit 308. The acoustic control unit 305 extracts the voice data of the user of the other terminal (hereinafter also referred to as voice data of the other terminal) from the reception signal output from the transmission / reception unit 308. The sound control unit 305 is based on the information indicating the wearing state output by the control unit 302, and is based on the sound data output from the encoder 129 (the sound data of the user of the HMD 1) or the sound data of the user of another terminal Is output to the left or right speaker of the stereo earphone 100, and the volume of each of the left and right speakers is adjusted.

  The image switching unit 306 acquires the image data read from the flash memory 122 or the image data output from the encoder 129 based on the control signal output from the input detection unit 301. The image switching unit 306 switches the display mode of the acquired image data based on the information indicating the mounting state output by the control unit 302, and outputs the image data to the image display unit 307 in the switched display mode. Here, the display mode is, for example, a first display mode or a second display mode (described later in FIG. 43).

The image switching unit 306 extracts an image switching signal from the reception signal output from the transmission / reception unit 308. When the image switching signal can be extracted, the image switching unit 306 outputs the image data read from the flash memory 122 or the image data output by the encoder 129 to the image display unit 307 based on the extracted image switching signal. The image switching unit 306 converts the resolution of the image data output from the encoder 129 or the image data read from the flash memory 122 based on the control signal output from the input detection unit 301, and converts the converted image data into an image. The data is output to the display unit 307.
The image switching unit 306 switches an angle for rotating an image to be displayed on the display unit 60 (hereinafter referred to as a display angle), for example, 0 degree, 90 degrees, 180 degrees, and 270 degrees, depending on the observation form. Thereby, the image switching unit 306 appropriately switches the display direction of the image displayed on the display unit 60 according to the relationship between the display unit 60 and the user's eyes regardless of the user's posture. Note that the observation mode can be recognized based on an image of a region including one eye of the user included in the captured image. Specifically, when the display angle of the image in the right-eye observation mode is set to 0 degrees and when the display angle of the image in the left-eye observation mode is set to 180 degrees, the control unit 302 displays the right-eye observation mode. The display angle of the image at the time of is switched to 0 degrees, and the display angle of the image in the left eye observation mode is switched to 180 degrees.

  The image data output from the image switching unit 306 is input to the image display unit 307. The image display unit 307 generates image data to be displayed on the display panel 63 based on these input image data. The image display unit 307 outputs the generated image data to the transmission / reception unit 308, the video connector 27, and the LCD driver 125.

  The transmission / reception unit 308 transmits the image data output from the image display unit 307 via the communication interface (131, 138) based on the control signal output from the input detection unit 301. The transmission / reception unit 308 outputs a reception signal received via the communication interfaces (131, 138) to the image switching unit 306.

The action history registration unit 309 stores the information in the storage unit 303 in response to the user of the HMD 1 operating the operation switch 30 or the touch switch 34 to start “business logging processing” in order to inspect the inspection target device. The transmitted user ID is transmitted to the server 200 (see FIG. 23) via the transmission / reception unit 308.
The action history registration unit 309 receives the inspection information transmitted from the server 200 according to the transmitted user ID. The inspection information is information including information indicating a position where the inspection target device is disposed. When there are a plurality of inspection target devices, the inspection information includes information indicating the positions where the plurality of inspection target devices are arranged and information indicating the order of inspection.
The action history registration unit 309 displays information indicating the direction in which the inspection target device is arranged on the display panel 63 based on the received inspection information and the detection value detected by the geomagnetic sensor 133.

In the present invention, the “business logging process” means any one of data acquired by executing a movement log, an observation log, or a script while the user is inspecting the inspection target device, This includes processing in which the action history registration unit 309 transmits to the server 200 via the transmission / reception unit 308. Further, the “business logging process” includes a process in which the server 200 stores the movement log, the observation log, and the data received from the HMD 1 in the storage unit 202 (see FIG. 23).
The movement log is a history including positioning information (information indicating latitude, information indicating longitude, information indicating altitude, and time information) measured by the GPS sensor 135 of the HMD 1 while the inspection target apparatus is inspected. Information. For example, the action history registration unit 309 may transmit the movement log to the server 200 at predetermined time intervals, or the movement log when the user operates the operation switch 30 or the touch switch 34. May be transmitted to the server 200.
The information indicating the observation log is history information indicating the contents processed by the HMD 1 while inspecting the inspection target device. The contents processed by the HMD 1 include information indicating that the operation switch 30 or the touch switch 34 has been operated, time information on the operation, and the action history registration unit 309 transmitted a movement log, an observation log, and data to the server 200. And information indicating the result of determining whether the user was observing the display unit 60, and the like. The action history registration unit 309 may transmit an observation log to the server 200 every time the inspection of the inspection target device ends, or may transmit an observation log to the server 200 at the end of the inspection to all inspection target devices. The observation log may be transmitted to the server 200 at a predetermined time interval.

  The script file is, for example, a file in which a program that causes the HMD 1 to execute a predetermined operation is described. The script file is included in information related to the inspection target device that the server 200 transmits to the HMD 1. Information relating to the inspection target apparatus will be described later. The predetermined operation to be executed for the HMD 1 is an operation for operating a functional unit included in the HMD 1. As an example of the predetermined operation, the HMD 1 captures a still image or a moving image with the camera 64 and transmits the captured image data to the server 200. As another example of the predetermined operation, the HMD 1 stores the sound collected by the sound collecting microphone 24 in the flash memory 122 and records it. As another example of the predetermined operation, the HMD 1 transmits the sound collected by the sound collecting microphone 24 or the sound data stored in the flash memory 122 to the server 200. As another example of the predetermined operation, the HMD 1 acquires the temperature and humidity of the position to be inspected by the temperature / humidity sensor 136 and transmits information indicating the acquired temperature and humidity to the server 200.

Returning to FIG. 22, the action history registration unit 309 controls turning on and off of the eyeball illumination light 166, and controls turning on and off of the backlight 62. When the information is displayed on the display panel (display unit) 63 according to the result determined by the observation availability determination unit 310, the action history registration unit 309 is in a situation where the displayed information is not observable by the user. The sound control unit 305 is controlled to notify the user to observe the information displayed on the display panel 63 from the ear speaker 23.
The situation in which the information displayed on the display panel 63 can be observed by the user is at least one of the following conditions (a) to (e). (A) A situation where the user is observing information displayed on the display panel 63. (B) A situation where the display body 20 is not folded as shown in FIG. (C) The situation in which the headband hinge 32 (see FIG. 3A) is not rotated 90 degrees in the A direction from the state shown in FIG. 5A and the display unit 60 is not arranged on the top of the user's head. . (D) A situation in which the headband 40 and the display main body 20 are connected. (E) A situation in which a captured image captured by the image sensor 72 includes a region including the user's eyes.
The situation where the information displayed on the display panel 63 is not observable by the user is at least one of the following conditions (f) to (j). (F) A situation where the user does not observe the information displayed on the display panel 63. (G) The display main body 20 is folded as shown in FIG. (H) A situation in which the headband hinge 32 (see FIG. 3A) is rotated 90 degrees in the A direction from the state shown in FIG. 5A and the display unit 60 is arranged on the top of the user's head. . (I) A situation where the headband 40 and the display main body 20 are not connected. (J) A situation in which a region including the user's eyes is not included in the captured image captured by the image sensor 72.

The action history registration unit 309 extracts a barcode image from the image data captured by the camera 64. When the barcode image can be extracted, the action history registration unit 309 extracts unique information (hereinafter referred to as an inspection target device ID) for determining the inspection target device embedded in the extracted barcode image. The action history registration unit 309 acquires information related to the inspection target device corresponding to the extracted inspection target device ID from the server 200 or the network. For example, when the acquired barcode includes address information indicating the address on the network where the information related to the inspection target device is located, the action history registration unit 309 extracts the address information from the barcode and extracts the address information. Based on the address information, information related to the inspection target device is acquired through the network. The address information is, for example, a URL (Uniform Resource Locator) indicating the location of the Web page. Also, the address information refers to the location of the Web page where the information is described when the precautions or priority inspection items for the inspection target device are described on the homepage of the manufacturer that sells the inspection target device. The URL shown may be included. The action history registration unit 309 displays information based on the acquired information related to the inspection target device on the display panel 63.
The action history registration unit 309 extracts a script file that executes a predetermined operation on the HMD 1 from the acquired information related to the inspection target device, and executes the script.

Here, information related to the inspection target apparatus will be described. The information related to the inspection target device includes at least one of the following information.
(1) Device name of the device to be inspected (2) Number of the device to be inspected (3) Items to be inspected for the device to be inspected (4) Precautions for inspection (5) Priority inspection items (6) Items to be inspected, inspection When any one of the precautions and priority inspection items is updated, information on the date and time when each of these inspection items was updated (hereinafter referred to as update date and time information)
(7) When any one of the inspection item of the inspection target device, the precautions in the inspection, and the priority inspection item is updated, information indicating that these pieces of information have been updated (hereinafter referred to as updated information).
(8) Information indicating that there is a note when there is a note (hereinafter, information with a note) (9) Information indicating that there is a check item when there is a check item (hereinafter, information with a check item)
(10) Script file for causing the HMD 1 to perform a predetermined operation The precautions in the inspection are, for example, a user who performs the inspection when the administrator who manages the inspection target device needs to pay attention to the inspection. It is information to call attention to. The priority inspection item is, for example, an inspection item for intensively inspecting a problem location when there is a problem in the inspection result with the same type of apparatus.
Note that the inspection item for the inspection target device, the precautions for inspection, and the priority inspection item are updated by the administrator as necessary for each information stored in the server 200 or each information on the network.

Returning to FIG. 22, the observability determination unit 310 determines whether the information displayed on the display panel 63 is observable by the user, and outputs the determined result to the action history registration unit 309.
The observation availability determination unit 310 determines whether the information displayed on the display panel 63 is in a state that can be observed by the user. The observability determination unit 310 determines, for example, whether the information displayed on the display panel 63 can be observed by the user as follows. The observation propriety determination unit 310 determines the second coordinates (Ex (i ), Ez (i)), it is determined whether or not the situation of (a) described above. Here, the coordinates based on the user's line of sight at step S605 in FIG. 39 are set as the first coordinates. Further, the second coordinate is the coordinate of the image indicating the orientation of the inspection position displayed in the information displayed on the display panel 63 in step S603.
The observability determination unit 310 compares the first coordinate and the second coordinate. As a result of the comparison, if it is determined that the first coordinate and the second coordinate match, the situation (a) described above is obtained. judge.
On the other hand, as a result of the comparison, if it is determined that the first coordinate and the second coordinate do not match, the observability determination unit 310 determines that the situation (f) described above is present.

The observability determination unit 310 determines whether or not the user is wearing the HMD 1 on the head based on the detection value detected by the heart rate sensor 137, so that the situation of (a) described above. Or the situation (f) may be determined. The observability determination unit 310 is in a situation where the information displayed on the display panel 63 can be observed by the user based on the result of determining whether the situation is (a) or (f). It may be determined whether or not.
In addition, the observability determination unit 310 determines, for example, whether the situation is any of (b), (c), (g), and (h) described above based on the detection value detected by the acceleration sensor 132. Thus, it may be determined whether or not the information displayed on the display panel 63 is in a state that can be observed by the user.

Further, the observability determination unit 310 determines, for example, whether the situation is the above-described situation (d) or the situation (i) based on the detection signal detected by the headband sensor 151. It may be determined whether the information displayed in 63 is in a state that can be observed by the user.
Further, the observability determination unit 310 determines whether the above-described situation (e) or (j) is based on the captured image, whereby the information displayed on the display panel 63 is changed by the user. It may be determined whether or not the situation is observable.

  FIG. 23 is a system configuration diagram of a communication system including the HMD 1 according to the present embodiment. As illustrated in FIG. 23, the communication system includes an HMD 1, an HMD 500, a PC 210, a PC 220, and a server 200. The HMD 1, HMD 500, PC 210, PC 220, and server 200 are connected to the network 230. The network 230 is a network such as the Internet connected via a wireless LAN (Local Area Network), a 3G line, an LTE line, or the like.

In FIG. 23, the HMD 1 has the configuration described in FIG. 21, but only shows functional units necessary for explaining the processing procedure. The display panel 63 of the HMD 1 is hereinafter referred to as an LCD 63.
As shown in FIG. 23, the HMD 500 mainly includes a flash memory 522, a processing unit 523, communication interfaces (531, 538), a geomagnetic sensor 533, a touch switch 534, a display panel (display unit) 563 (hereinafter referred to as an LCD 563), A camera 564 is provided. Each function part of HMD500 is equivalent to each function part of HMD1.

  Next, the configuration of the server 200 will be described. As shown in FIG. 23, the server 200 includes a processing unit 201, a storage unit 202, and a communication interface 203.

  The processing unit 201 of the server 200 includes image data (moving image, still image, line drawing, etc.) received from each terminal (HMD1, HMD500, PC210, and PC220) via the communication interface 203, audio data, environment information of each terminal, In addition, a communication record such as a request from each terminal is recorded in the storage unit 202. The processing unit 201 reads image data recorded in the storage unit 202 in response to a request from each terminal, and transmits the read image data to each terminal.

  The storage unit 202 stores information relating to the user ID and the inspection target device in association with each other. In the storage unit 202, information indicating the movement log received by the server 200 from the HMD 1 and information indicating the observation log are stored in association with the user ID. The storage unit 202 stores image data (still images, moving images), audio data, and detection values detected by various sensors received from the HMD 1 in association with user IDs. In the storage unit 202, personal data corresponding to the user ID of each terminal, for example, data such as the user's name, gender and nationality is associated with information such as a telephone number or an IP address of each terminal that can identify the terminal. It is remembered.

  The communication interface 203 includes, for example, a WiFi communication circuit, a 3G communication circuit, an LTE communication circuit, a wired LAN circuit, and the like.

Next, the configuration of the PC 210 and the PC 220 will be described. The PC 210 and the PC 220 are, for example, personal computers.
The PC 210 includes a processing unit 211, a storage unit 212, a communication interface 213, an input unit 214, a display unit 215, and a camera 216.
The PC 220 includes a processing unit 221, a storage unit 222, a communication interface 223, an input unit 224, a display unit 225, and a camera 226. Each functional unit of the PC 220 is equivalent to each functional unit of the PC 210.

The processing unit 211 of the PC 210 includes a CPU (Central Processing Unit) and a GPU (Graphics).
Processing Unit). The processing unit 211 controls the communication interface 213, the input unit 214, the display unit 215, and the camera 216. The processing unit 211 switches the image data stored in the storage unit 212, the image data acquired through the communication interface 213, and the image data captured by the camera 216 through the communication interface 213 as will be described later. Switch based on signal. The processing unit 211 outputs the switched image data to the communication interface 213 and the network 230. The processing unit 211 outputs the image data to the display unit 215. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the operation of the input unit 214. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the received signal acquired via the communication interface 213.

  The storage unit 212 is, for example, a memory or an HDD (hard disk). The storage unit 212 stores an OS (operation system) of the PC 210, an application for making a call with the HMD1, the HMD 500, and the PC 220, and the like.

  The communication interface 213 is an interface that performs communication such as a wireless LAN, a wired LAN, a 3G line, an LTE line, or the like. The communication interface 213 outputs information received via the network 230 to the processing unit 211. The communication interface 213 transmits information output from the processing unit 211 to the HMD 1, the HMD 500, the PC 220, and the server 200 via the network 230.

  The input unit 214 is, for example, a keyboard, a mouse, a tablet, and an operation pen. The input unit 214 outputs the input information to the processing unit 211. Note that when the display unit 215 has a touch panel function, the display unit 215 may also function as the input unit 214.

The display unit 215 displays an image based on the image data output from the processing unit 211.
As an example, an image captured by the HMD 1 and a line drawing image corresponding to an operation of the input unit 214 of the PC 210 are combined and displayed on the display unit 215.

  The camera 216 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) having an image sensor with 5 to 10 million pixels.

Next, the operation of the HMD 1 will be described. FIG. 24 is a flowchart of the operation performed after the power-on state of the HMD 1 according to the present embodiment.
(Step S1) The processing unit 123 waits for completion of activation of the system of the HMD 1, and proceeds to step S2 when the activation of the system is completed. The activation means that the main switch 28 is operated to cold start the HMD 1 and the HMD 1 is not used for a predetermined time to reach a sleep state, and then the touch switch 34 or the like is operated. For example, when it starts.

(Step S <b> 2) The processing unit 123 performs a process of detecting whether the user is observing the display unit 60 of the HMD 1 with the left or right eye (hereinafter referred to as a left / right eye detection process). Detailed processing of the “left and right eye detection processing” will be described later. The processing unit 123 switches the orientation of the image displayed on the display unit 60 based on the detected result.

(Step S <b> 3) The processing unit 123 performs a process of performing a guide (hereinafter referred to as a mounting guide process) so that the display unit 60 of the HMD 1 is mounted at the correct mounting position. The detailed processing of “mounting guide processing” will be described later.

(Step S4) The processing unit 123 performs a process of selecting an eyeball illumination light 166 to be turned on and an eyeball illumination light to be turned off (hereinafter referred to as an eyeball illumination light selection process). Details of the “eyeball illumination light selection process” will be described later.

(Step S5) The processing unit 123 performs calibration for gazing point detection. Details of the calibration process for detecting the point of interest will be described later.

(Step S <b> 6) The processing unit 123 performs “business logging processing” that inspects an inspection apparatus that is an inspection target and registers the inspection result in the server 200 as an observation log. Details of the “business logging process” will be described later.

Next, a procedure for “left and right eye detection processing” will be described.
FIG. 25 is a flowchart of the left and right eye detection processing procedure according to the present embodiment.

(Step S101) The control unit 302 of the processing unit 123 determines whether or not the detection value detected by the acceleration sensor 132 is stronger in the Z-axis direction than in the X-axis direction and the Y-axis direction.
When it is determined that the detected value in the Z-axis direction is stronger than the X-axis direction and the Y-axis direction (step S101; Yes), the process proceeds to step S102. When it is determined that the detected value in the Z-axis direction is not stronger than the X-axis direction and the Y-axis direction (step S101; No), the process proceeds to step S103.

(Step S102) When it is determined that the detected value in the Z-axis direction is stronger than the X-axis direction and the Y-axis direction, the display unit 60 displays the first value as shown in FIG. 5 (A) or FIG. 5 (B). This is a display mode wearing form. Therefore, the control unit 302 determines whether the observation mode is the right eye observation mode or the left eye observation mode based on the detection value of the acceleration sensor 132. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.

(Step S103) When it is determined that the detected value in the Z-axis direction is not stronger than the X-axis direction and the Y-axis direction, the control unit 302 drives the image sensor 72 because the user is sleeping or lying on the back. To do.
FIG. 26 is a diagram illustrating an example of a captured image in the imaging range P in the case of the right eye observation form. FIG. 27 is a diagram illustrating an example of a captured image in the imaging range P in the case of the left eye observation form.
In order to discriminate the observation form, the control unit 302 captures a peripheral image of the eye with the image sensor 72 using the imaging region P described in FIG. For example, the captured image is based on the captured image (hereinafter referred to as the first right-eye image) when used in the first display mode and the right-eye observation mode illustrated in FIG. Therefore, as shown in FIG. 26, the first right eye image 610R is an erect image. As shown in FIG. 26, in the first right eye image 610R, the right eyebrow image 602R3 is in the + Z direction with respect to the right eye image 601R3.
On the other hand, as shown in FIG. 27, an eye peripheral image (hereinafter referred to as a first left eye image) 610L when used in the first display mode and the left eye observation form is an inverted image. As shown in FIG. 27, in the first left eye image 610L, the right eyebrow image 602L is in the −Z direction with respect to the left eye image 601L.

(Step S104) Returning to FIG. 25, the control unit 302 determines whether or not an image of eyebrows can be extracted from the captured image. If it is determined that the image of the eyebrows can be extracted (step S104; Yes), the process proceeds to step S105. For example, when it is determined that the eyebrow image cannot be extracted, such as when the display unit 60 is close to the eyes or when it is difficult to determine the eyebrows by makeup or the like (step S104; No), the process proceeds to step S106. Note that the control unit 302 extracts an image of eyebrows using, for example, a pattern matching technique.
For example, when the user is using the right-eye observation mode, the image captured in the selection area Q (see FIG. 19A) is an image as shown in FIG. On the other hand, when the user is using the left-eye observation form, the image captured in the selection region R (see FIG. 19B) is an image as shown in FIG. FIG. 28 is a diagram illustrating an example of a captured image in the left-eye observation mode. In FIG. 28, the image of the selection region R is shown rotated by 180 degrees. Therefore, the −Z side in FIG. 28 is the user's upper side, the + Z side is the user's lower side, the −X side is the user's left side, and the + X side is the user's right side. In FIG. 28, reference numeral 601L1 is a left-eye image, and reference numeral 602L1 is a left eyebrow image.
Note that during the processing in steps S101 to S108, the control unit 302 may turn on or turn off the eyeball illumination light 166. During the processes in steps S101 to S108, the control unit 302 may turn on the backlight 62.

(Step S105) When the image of the eyebrows can be extracted from the captured image, the control unit 302 uses, for example, a pattern matching technique or an image feature amount from the position of the eyebrow image and the eye image or the shape of the eyebrows. Using the image comparison technique, it is determined whether the captured image is a right-eye image or a left-eye image of the user. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.
For example, as illustrated in FIG. 26, when it is determined that the image of the eyebrows is in the + Z direction with respect to the image of the eyeball, the control unit 302 has the right eye in which the user observes the display unit 60 with the right eye. It is determined that it is an observation form. On the other hand, as illustrated in FIG. 27, when it is determined that the image of the eyebrows is in the −Z direction with respect to the image of the eyeball, the control unit 302 displays the left where the user observes the display unit 60 with the left eye. It is determined that it is an eye observation form.

(Step S106) When the eyebrow image cannot be extracted from the captured image, the control unit 302 uses the pattern matching technique or the image comparison technique based on the image feature amount, for example, based on the behavior of eyelids in blinking. It is discriminated whether it is the right eye or the left eye. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.
For example, since the upper eyelid moves during blinking, when the eyelid moves from the + Z direction to the −Z direction in FIG. 11 and immediately returns from the −Z direction to the + Z direction, the control unit 302 displays the right eye. It is determined that this is the right-eye observation mode in which the unit 60 is observed. Further, when the movement start direction and the return direction of the eyelid are reversed, the control unit 302 determines that the left-eye observation mode is that the user observes the display unit 60 with the left eye.
As described above, according to the present embodiment, even when the user is lying in the horizontal direction, it is possible to detect the movement of the eyebrows and the eyelashes without depending on the direction of gravity detected by the acceleration sensor 132. The direction of the eye that the user is observing can be determined.

(Step S107) When it is determined that the user is using the right-eye observation mode based on the determination results up to Step S106, the control unit 302 selects the selection region Q from the imaging region P captured by the imaging element 72. Select and proceed to step S108. On the other hand, when it is determined that the user is using the left-eye observation form, the control unit 302 selects the selection region R from the imaging region P, and proceeds to step S108.

(Step S <b> 108) If it is determined that the right-eye observation mode is set based on the determination results up to step S <b> 106, the image switching unit 306 sets the display direction of the display unit 60 for the right eye. Further, when the stereo earphone 100 is connected, the acoustic control unit 305 sets the left and right of the stereo sound output so that the first speaker 103 is for the right ear and the second speaker 104 is for the left ear.
On the other hand, when the left-eye observation mode is determined, the image switching unit 306 sets the display direction of the display unit 60 for the left eye. The display direction for the left eye is a direction in which the display face for the right eye is rotated 180 degrees and turned upside down. When the stereo earphone 100 is connected, the acoustic control unit 305 sets the left and right stereo sound output so that the first speaker 103 is for the left ear and the second speaker 104 is for the right ear.

In addition, although the example which discriminate | determines whether it is a right-eye observation form based on the detected value of the acceleration sensor 132 and a captured image in step S101-S106 was demonstrated, determination of whether it is a right-eye observation form The method is not limited to this.
For example, the HMD 1 may be detected by a switch or the like (not shown) attached to the display main body 20. Of course, the user may manually set the right-eye observation form or the left-eye observation form.

Next, the “mounting guide processing” procedure will be described.
FIG. 29 is a flowchart of the processing procedure of the mounting guide for the HMD 1 according to the present embodiment. In the following description, the display panel 63 is referred to as an LCD 63. In this process, when the user uses the HMD 1 and the LCD 63 is being displayed, this process may be appropriately performed to prompt the user to correct the displacement of the LCD 63. Alternatively, it may be performed when the user instructs to perform alignment by operating the touch switch 34 or the like.

  In the present embodiment, as described with reference to FIGS. 18A and 18B, the display unit 60 including the image sensor 72 is disposed obliquely below the user's eye. For this reason, as shown in FIG. 17, the image sensor 72 captures an image around the user's eye obliquely from below. However, when the distance between the display unit 60 and the eyes is not appropriate, and when the screen center of the display unit 60 and the center of the eyes are shifted, the user cannot properly observe the image displayed on the LCD 63. For this reason, in the present embodiment, the control unit 302 detects that the position of the display unit 60 is deviated from the position of the user's eye based on the image captured by the image sensor 72 and announces to the user. To do. Note that the control unit 302 determines whether or not the position of the user's eyes is shifted from the position of the display unit 60 using the image of the selection region (image region) Q or R selected from the captured image. Alternatively, it may be performed using an image of a region including one eye of the user included in the captured image. In the following description, an example in which the control unit 302 determines whether or not the position of the user's eye is shifted from the position of the display unit 60 using the image of the selection region Q or R selected from the captured image. Will be explained. Therefore, in the following description, the captured image is an image of the selection region Q or R selected in step S107 (see FIG. 25).

(Step S201) The control unit 302 extracts an eye image using, for example, a pattern matching technique. When it is determined that the user is using the right-eye observation form based on the determination result of “left and right eye detection process” (step S201; Yes), the control unit 302 proceeds to step S202, and the user When it is determined that the left-eye observation mode is used (step S201; No), the process proceeds to step S214.

(Step S202) The control unit 302 determines whether or not the position of the user's eyes is shifted in the + X direction based on the captured image (see, for example, FIG. 11). When it is determined that the position of the user's eye is shifted in the + X direction (step S202; Yes), the process proceeds to step S203. When it is determined that the position of the user's eye is not shifted in the + X direction (step S202; No), the process proceeds to step S204.

(Step S203) When it is determined that the position of the user's eye is shifted in the + X direction, the display unit 60 is shifted to the right side of the user with respect to the user's eye. For this reason, in such a case, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the left side of the user.

(Step S <b> 204) The control unit 302 determines whether or not the position of the user's eye is shifted in the −X direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −X direction (step S204; Yes), the process proceeds to step S205. When it is determined that the position of the user's eye is not shifted in the −X direction (step S204; No), the process proceeds to step S206.

(Step S205) When it is determined that the position of the user's eyes is shifted in the -X direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's right side. .

(Step S206) The control unit 302 determines whether or not the position of the user's eye is shifted in the + Z direction based on the captured image. When it is determined that the user's eyes are displaced in the + Z direction (step S206; Yes), the process proceeds to step S207. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S206; No), the process proceeds to step S208.

(Step S207) When it is determined that the position of the user's eye is shifted in the + Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user.

(Step S208) The control unit 302 determines whether or not the position of the user's eye is shifted in the −Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −Z direction (step S208; Yes), the process proceeds to step S209. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S208; No), the process proceeds to step S210.

(Step S209) When it is determined that the position of the user's eyes is shifted in the -Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's lower side. Do.

(Step S210) The control unit 302 determines whether the image of the user's eye is larger than a predetermined size based on the captured image. When it is determined that the image of the user's eye is larger than the predetermined size (step S210; Yes), the process proceeds to step S211. When it is determined that the image of the user's eye is not larger than the predetermined size (step S210; No), the process proceeds to step S212. For example, the control unit 302 extracts an eye image, and extracts the eye width based on the extracted image. The control unit 302 determines whether or not the extracted eye width is larger than a predetermined size.

(Step S211) If it is determined that the image of the user's eyes is larger than the predetermined size, it is estimated that the user's eyes are too close to the display unit 60, so that the processing unit 123 moves the display unit 60 away from the eyes. Announcement is made from the ear speaker 23.

(Step S212) The control unit 302 determines whether the image of the user's eye is smaller than a predetermined size based on the captured image. When it is determined that the image of the user's eye is smaller than the predetermined size (step S212; Yes), the process proceeds to step S213. When it is determined that the image of the user's eye is not smaller than the predetermined size (step S212; No), the wearing state determination process is terminated.

(Step S213) If it is determined that the image of the user's eyes is smaller than the predetermined size, it is estimated that the user's eyes are too far from the display unit 60, so that the processing unit 123 brings the display unit 60 closer to the eyes. Announcement is made from the ear speaker 23.

(Step S214) When it is determined that the user is using the left-eye observation form, the control unit 302 shifts the position of the user's eyes in the + X direction based on the captured image (for example, see FIG. 27). It is determined whether or not. When it is determined that the position of the user's eye is shifted in the + X direction (step S214; Yes), the process proceeds to step S215. When it is determined that the position of the user's eye is not shifted in the + X direction (step S214; No), the process proceeds to step S216.

(Step S215) When it is determined that the position of the user's eye is shifted in the + X direction, the display unit 60 is shifted to the left side of the user with respect to the user's eye. For this reason, in such a case, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the right side of the user.

(Step S216) The control unit 302 determines whether the position of the user's eye is shifted in the −X direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −X direction (step S216; Yes), the process proceeds to step S217. When it is determined that the position of the user's eye is not shifted in the −X direction (step S216; No), the process proceeds to step S218.

(Step S217) When it is determined that the position of the user's eyes is shifted in the −X direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the left side of the user. .

(Step S218) The control unit 302 determines whether or not the position of the user's eye is shifted in the + Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the + Z direction (step S218; Yes), the process proceeds to step S219. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S218; No), the process proceeds to step S220.

(Step S219) When it is determined that the position of the user's eyes is shifted in the + Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's lower side. .

(Step S <b> 220) The control unit 302 determines whether the position of the user's eye is shifted in the −Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −Z direction (step S220; Yes), the process proceeds to step S221. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S220; No), the process proceeds to step S210.

(Step S221) When it is determined that the position of the user's eye is shifted in the -Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user. .
After the processing in step S221, the processing unit 123 performs steps S211 to S213, and ends the processing for mounting the HMD1.

  Next, the eyeball illumination light selection process will be described. FIG. 30 is a flowchart of the selection processing procedure of the eyeball illumination light of the HMD 1 according to the present embodiment.

(Step S401) When the control unit 302 determines that the user is using the right-eye observation mode based on the determination result of the “left and right eye detection process” (step S401; Yes), the process proceeds to step S402. If it is determined that the user is using the left-eye observation form (step S401; No), the process proceeds to step S403.

(Step S402) When it is determined that the user is using the right-eye observation mode, the control unit 302 selects the eyeball illumination lights 166a and 166b and stores the selected result in the storage unit 303 as the eyeball illumination light selection setting. Remember me. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

(Step S403) When it is determined that the user is using the left-eye observation mode, the control unit 302 selects the eyeball illumination lights 166c and 166d and stores the selected result in the storage unit 303 as the eyeball illumination light selection setting. Remember me. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

  As described above, the HMD 1 of the present embodiment selects and turns on the eyeball illumination lights 166a to 166d according to the positional relationship between the display unit 60 and the user. As a result, the HMD 1 of the present embodiment can appropriately illuminate the user's eyes by appropriately selecting a light for illuminating the eyeball, and can reduce power consumption.

Next, the details of the calibration for gazing point detection will be described.
In the HMD 1 according to the present embodiment, the user does not set the calibration as in the calibration comparative example, but operates the information displayed on the display panel 63 as will be described later. ) To obtain the parameters for operation based on), and perform calibration for gazing point detection.
FIG. 31 and FIG. 32 are flowcharts of a calibration procedure for gazing point detection according to the present embodiment. FIG. 33 is a diagram for explaining turning on and off of the eyeball illumination light 166 in the right-eye observation mode according to this embodiment. FIG. 34 is a diagram for explaining the relationship between the position of the display unit 60 and the eyeball illumination light 166 according to the present embodiment, and is a diagram of the user's face viewed from the horizontal front.
In the following, as shown in FIG. 7A, FIG. 7B, and FIG. 34, a state in which the display unit 60 is arranged in the right-eye observation form and under the eye, or the left-eye observation form and An example will be described in which a state in which the display unit 60 is placed under the eyes is detected and calibration for detecting a point of interest is performed.

  In FIG. 34, positions 600a and 600b indicated by alternate long and short dash lines are examples of positions where the display unit 60 is disposed. As will be described in the following processing, when the display unit 60 is at the position 600a, the HMD 1 turns on the eyeball illumination lights 166a and 166b in the vicinity of the eye and turns off the others. On the other hand, when the display unit 60 is at the position 600b, the HMD 1 turns on the eyeball illumination lights 166c and 166d in the vicinity of the eyes and turns off the others.

(Step S <b> 501) The control unit 302 determines whether or not the user is using the right-eye observation mode based on the determination result of “left and right eye detection processing”. When it is determined that the user is using the right-eye observation form (step S501; Yes), the control unit 302 proceeds to step S502, and when it is determined that the user is not using the right-eye observation form (step S501). Step S501; No), the process proceeds to Step S503.

(Step S <b> 502) When the right eye observation mode is determined, the control unit 302 reads the selection setting of the eyeball illumination light stored in the storage unit 303. Next, the control unit 302 turns on the eyeball illumination lights 166a and 166b via the eyeball illumination unit 304 based on the selection setting of the eyeball illumination light to be read.
As shown in FIG. 33, when the display unit 60 is placed under the user's right eye for observation, the eyeball illumination lights 166c and 166d cannot be illuminated because they are arranged very close to the cheek. Therefore, in this embodiment, as shown at a position 600a in FIG. 34, the eyeball illumination unit 304 turns off the eyeball illumination lights 166c and 166d in order to reduce power consumption.

(Step S <b> 503) When it is determined that the left-eye observation mode is selected, the control unit 302 reads the selection setting of the eyeball illumination light stored in the storage unit 303. Next, the control unit 302 turns on the eyeball illumination lights 166c and 166d via the eyeball illumination unit 304 based on the read setting of the eyeball illumination light to be read, as indicated by a position 600b in FIG. The eyeball illumination unit 304 turns off the eyeball illumination lights 166a and 166b in order to reduce power consumption.
With the above processing, the HMD 1 of the present embodiment can appropriately select a light that illuminates the eyeball according to the positional relationship between the user and the display unit 60. Furthermore, the HMD 1 of the present embodiment can efficiently take measures for power saving by turning off the eyeball illumination light 166 that is not used.

(Step S504) After step S502 or S503 is completed, the image switching unit 306 displays a password input screen (display image) as shown in FIG. This is a method for inputting a password set by a user, which is used in PCs and mobile phones that are widely used for mobile purposes, and is intended to prevent others from using the password.
FIG. 35 is an example of a password input screen 400e according to the present embodiment. In FIG. 35, the longitudinal direction of the password input screen 400e is defined as the X-axis direction, and the short direction is defined as the Z-axis direction. As shown in FIG. 35, on the LCD 63, alphabet keys A to Z (operation images) 461, symbol keys (operation images) 462, shift keys (operation images) 463, 10-key switching keys (operation images) 464 are displayed. , A space key (operation image) 465, a determination (Done) key (operation image) 466, an area 467 for displaying input characters, and a cursor 468 are displayed. In FIG. 35, the symbol Dx represents the coordinate in the X-axis direction, and the symbol Dz represents the coordinate in the Z-axis direction.

  The display image is an image displayed on the LCD 63, such as a password input screen shown in FIG. 35, a menu screen described later, or the like. The operation image is displayed on the display image and can be selected by the user via the operation unit (operation switch 30 and touch switch 34). Each key image, each button image, each icon image, cursor image, etc. It is. As described above, these display images include images that prompt the user to select an operation image by operating the operation unit in response to the display.

(Step S <b> 505) Returning to FIG. 31, the control unit 302 initializes “i” as a parameter suffix (character string) to zero.

(Step S506) The control unit 302 adds 1 to the suffix “i”.

(Step S507) The control unit 302 acquires a captured image (including video) captured by the image sensor 72. Next, the control unit 302 determines the coordinates (second coordinates) (Ex (i), Ez (i)) of the bright spot by the eyeball illumination lights 166a to 166d irradiated on the eyeball based on the acquired captured image. get. The second coordinates are coordinates that correspond one-to-one with the rotation of the eyeball, that is, the direction of the line of sight.

  The captured image acquired by the control unit 302 is, for example, FIG. As shown in FIG. 11, a bright spot 604A by the eyeball illumination light 166a and a bright spot 604B by the eyeball illumination light 166b are shown in the image, and the bright spot moves in conjunction with the movement of the eyeball. That is, when the eyeball moves to the left side, the bright spots 604A and 604B move to the + X side. Further, when the eyeball moves to the left side, the same bright spot moves to the −X side. When the eyeball moves upward, the bright spot moves to the + Z side, and when the eyeball moves downward, the bright spot moves to the −Z side. As for the movement of the eyeball, the position in the LCD 63 and the position of the bright spot are uniquely determined depending on where the user looks at the display in the LCD 63.

A cursor 468 is displayed on the password input screen 400e of FIG. The cursor 468 can be moved up, down, left, and right by the user operating the operation switch 30 by tilting it up, down, left, or right. As shown in FIG. 35, for example, when “P” is selected as one character of the password, the cursor 468 is moved to the character “P”, and the operation switch 30 is further pressed to change the character “P”. A character is entered.
Immediately before the operation switch 30 is pressed, the user has a very high probability of gazing at the letter “P”. Note that the control unit 302 gazes at the first character of the password by using an average value of the coordinates (Xa, Za) of the two bright spots 604A and the coordinates (Xb, Zb) of the bright spots 604B in FIG. The measured values Ex (1) and Ez (1) of the X-axis and Y-axis of the bright spot of the eye are calculated as in the following expressions (2) and (3).

  Ex (1) = (Xa + Xb) / 2 (2)

  Ez (1) = (Za + Zb) / 2 (3)

  The control unit 302 causes the storage unit 303 to store the calculated coordinates (Ex (i), Ez (i)) of the measurement value.

(Step S508) Returning to FIG. 31, the control unit 302 acquires (Dx (i), Dz (i)), which are coordinates (first coordinates) on the LCD 63 on which characters are displayed. For example, when the character “P” is selected in FIG. 35, the control unit 302 acquires the coordinates of the character “P” and acquires (Dx (1), Dz (1)) as the coordinates of the first character of the password. To do. Thus, the first coordinates are the coordinates of the operation image in the display image displayed on the LCD 63. In addition, a predetermined reference point may be stored in the storage unit 303 in each operation image. The predetermined reference points are, for example, the corners of the operation image (upper left, upper right, lower left, or lower right), the center of the operation image, and the like. The control unit 302 acquires the coordinates of a reference point that is predetermined for the selected operation image. Alternatively, the control unit 302 may acquire the reference point of the tip of the arrow of the cursor 468 in the character “P” instead of the coordinates, and the acquired coordinates may be the first coordinates.

(Step S509) The control unit 302 determines whether or not the authentication of the password set by the user has been completed. If it is determined that password authentication has been completed (step S509; Yes), the process proceeds to step S510. If it is determined that password authentication has not been completed (S509; No), steps S506 to S509 are repeated.
In steps S507 to S509, a plurality of password characters are selected, so that the measurement coordinates of the bright point of the eye (Ex (i), Ez (i)) and the position coordinates of the characters on the LCD 63 ( The value of Dx (i), Dz (i)) is obtained. When the user determines that the password has been input, the user selects and presses the Done key 466 with the operation switch 30. At this time, the control unit 302 also acquires the coordinates of the Done key 466.

(Step S510) By the password input operation up to Step S312, the measured value of the bright point of the eye and the position information of the character corresponding to this are obtained. Therefore, after this, if the coordinates of the acquired bright spot are measured, it becomes possible to know where the user is gazing on the LCD 63. That is, by these processes, an operation by line-of-sight input becomes possible. Therefore, the control unit 302 generates function expressions for obtaining the display position (Dx, Dz) based on the measured values (Ex, Ez) as the following expressions (4) and (5).

  Dx = F1 (Ex, Ez) (4)

  Dz = F2 (Ex, Ez) (5)

  In the equations (4) and (5), F1 is a function of Dx and (Ex, Ez), and F2 is a function of Dz and (Ex, Ez). Next, the control unit 302 stores the generated expressions (4) and (5) in the storage unit 303.

(Step S511) Upon completion of the password input, the image switching unit 306 displays a menu icon as shown in FIG.
FIG. 36 is a diagram showing an example of a menu screen (display image) 400f according to the present embodiment. In the example shown in FIG. 36, nine icons (operation images) 471 to 479 and a cursor (operation image) 480 are displayed on the menu screen 400f. For example, the icon 471 is an icon that indicates an instruction to capture a moving image. Icons 471 to 479 are referred to as applications. Of these, the icon 471 is an application having a function of inputting with a line of sight, and is particularly referred to as a line-of-sight input application.
In addition, in the application for taking a moving image represented by the icon 471, it is possible to perform moving image shooting and recording with the camera 64, and to move a marker displayed on the screen and to operate a button by a line-of-sight input as described later. It has become.
In the initial state, a cursor 480 is displayed at the position of the icon 471 as shown in FIG.

The menu screen in FIG. 36 may be a menu screen (display image) for performing various settings in an OS (operation system) that controls the processing unit 123, for example. The menu screen for performing various settings includes, for example, various setting buttons (or icons, keys, etc.) for adjusting the brightness of the display panel 63, adjusting the contrast of the display panel 63, setting the date and time, and setting whether or not to enter the standby state. This is the displayed screen.
Further, an application whose icon is displayed on the menu screen 400f is not limited to an application executed inside the HMD 1. For example, the application in which the icon is displayed may be an application executed on the HMD or PC used by the communication partner that is performing communication. In addition, the application in which the icon is displayed may be, for example, a Web application that is executed on a network such as the Internet.

(Step S512) Returning to FIG. 31, the control unit 302 determines whether or not the icon 471 is manually selected by the operation switch 30 on the menu screen. When it is determined that the icon 471 has not been selected manually by the operation switch 30 (step S512; No), the process proceeds to step S513. When it is determined that the icon has been manually selected by the operation switch 30 (step S512; Yes), the process proceeds to the circle C (FIG. 32, step S519).

(Step S513) The control unit 302 determines whether an application other than the icon 471 has been selected. When it is determined that an application other than the icon 471 has been selected (step S513; Yes), the process proceeds to step S514, and the control unit 302 performs calibration to further improve the accuracy of the line-of-sight input in the menu operation performed by the user. Do. When it is determined that an application other than the icon 471 has not been selected (step S513; No), the process returns to step S512.

(Step S514) When it is determined that an application other than the icon 471 has been selected, the control unit 302 adds 1 to the suffix “i”.

(Step S515) The control unit 302 acquires a captured image. Next, the control unit 302 acquires the coordinates (Ex (i), Ez (i)) of the bright spot by the eyeball illumination lights 166a to 166d that are irradiated to the eyeball based on the acquired captured image.

(Step S516) The control unit 302 acquires coordinates (Dx (i), Dz (i)) on the LCD 63 on which the selected icon is displayed.

(Step S517) The control unit 302 recalculates the relational expressions (4) and (5) stored in the self unit, and stores the relational expression generated by the recalculation in the storage unit 303.

(Step S518) The control unit 302 executes the selected application (or menu). After executing the application, the process returns to step S511.
That is, the control unit 302 obtains the luminance coordinates and recalculates the relational expressions (4) and (5) every time an icon, a check box, or the like is selected by the operation switch 30 or the like, so that the gaze point is obtained. Perform calibration for detection. Note that each time an icon or check box is selected during execution of various applications including the menu screen 400f, calibration for detecting a point of interest is sequentially performed using the icon or check box as the point of interest. You may go.

(Step S519) Moving to FIG. 32, when it is determined that the icon 471 has been manually selected by the operation switch 30, the control unit 302 adds 1 to the suffix “i”.

(Step S520) The control unit 302 acquires (Dx (i), Dz (i)), which are coordinates on the LCD 63 on which the selected icon is displayed.

(Step S <b> 521) The control unit 302 recalculates the relational expressions (4) and (5) stored in the self unit, and stores the relational expression generated by the recalculation in the storage unit 303.

(Step S <b> 522) The control unit 302 determines whether or not calibration for detecting a gazing point has been sufficiently performed before executing the gaze input application. Specifically, for example, if the password registration is one character, there is a possibility that the calibration is not sufficiently performed. For example, if at least two different gazing points are detected on the password input screen 400e or the menu screen 400f, the control unit 302 determines that the calibration has been sufficiently performed. Note that the two different positions are positions arranged at diagonal positions in the password input screen 400e or the menu screen 400f.
When it is determined that the calibration has been sufficiently performed (step S522; Yes), the process proceeds to step S527. When it is determined that the calibration is not sufficiently performed (step S522; No), the process proceeds to step S523.

(Steps S <b> 523 to S <b> 526) The control unit 302 presents text prompting the user to input again as illustrated in FIG. 37 via the image switching unit 306. The control unit 302 additionally presents the screen (display image) shown in FIG. 37, and when the user operates this screen, the coordinates on the LCD 63 (Dx (i), Dz (i)), and eyeball illumination The coordinates (Ex (i), Ez (i)) of the bright spot by the lights 166a to 166d are acquired, and calibration for gazing point detection is performed.
FIG. 37 is an example of a screen for prompting the user to input according to the present embodiment. As shown in FIG. 37, a character string “Do you use the line-of-sight input mode?” 481, “Yes” button (operation image) 482, and “No” button (operation image) 483 are displayed on the screen 400g. Yes.
For example, in FIG. 37, when the “Yes” button 482 is selected by line-of-sight input, the Done key 466 in FIG. 35 and the moving image imaging instruction icon 471 in FIG. Three buttons (or keys, icons) have been pressed, and a certain degree of reliable calibration has been completed. Then, the control unit 302 performs the same processing as steps S519 to S521 by the processing of steps S523 to S526, and derives a function expression for obtaining (Dx, Dz).

  The display for input includes each key (461-466) in FIG. 35, each icon 471-479 in FIG. 36, each button (482, 483) in FIG. Here, when the position where the calibration is insufficient is in the screen, the Done key 466 may be displayed at a position where calibration is necessary by appropriately changing the display position.

(Step S527) Returning to FIG. 32, the control unit 302 displays the image captured by the camera 64 on the LCD 63 via the image switching unit 306 as shown in FIG.
FIG. 38 is a diagram for explaining the screen 400h of the line-of-sight input application according to the present embodiment. In the example shown in FIG. 38, the screen 400 (display image) h includes a captured image 491, a marker (operation image) 492, an “end button” (operation image) 493, and a “record button” (operation image) 494. , And a “menu button” (operation image) 495 are displayed. The marker 492 is a marker that indicates a required position in the image 491 by the user. The “end button” 493 is a button for ending the application and returning to the menu screen 400f shown in FIG. The “record button” 494 is a button for starting to record the image signal captured by the camera 64 in the flash memory 122. The “menu button” 495 is a button for displaying a menu (not shown) for performing settings specific to the application.
When the HMD 1 is communicating with another terminal, the captured image 491 is transmitted to the other terminal via the transmission / reception unit 308.
As shown in the screen 400h, the HMD 1 of the present embodiment is specific to the product 496 by transmitting an image indicating the product 496 with a cursor to another terminal in the image that the user is observing and capturing with the camera 64. Can be communicated to the other party.

(Step S528) Returning to FIG. 32, the controller 302 determines whether or not the user has selected the buttons 493 to 495 with the operation switch 30. When it is determined that the user has selected the buttons 493 to 495 with the operation switch 30 (step S528; Yes), the process proceeds to step S535. When it is determined that the user has not selected the buttons 493 to 495 with the operation switch 30 (step S528; No), the process proceeds to step S529.

(Step S529) When it is determined that the user has not selected the buttons 493 to 495 with the operation switch 30, the control unit 302 acquires the coordinates (Ex, Ez) of the bright spot in the user's eye. Next, the control unit 302 calculates where the user is gazing using the acquired coordinates and the functions of Expressions (4) and (5).

(Step S530) The control unit 302 displays a marker 492 at the calculated coordinate position.
Note that the HMD 1 transmits the image on which the image of the marker 492 is superimposed to another terminal. Since the marker 492 moves when the user moves the gazing point, the gazing point can be transmitted to users of other terminals in a hands-free manner.

(Step S531) The control unit 302 determines whether or not the marker 492 is on the buttons 493, 494, and 495. When it is determined that the marker 492 is on the buttons 493, 494, and 495 (step S531; Yes), the process proceeds to step S532. When it is determined that the marker 492 is not on the buttons 493, 494, and 495 (step S531; No), the process returns to step S528.

(Step S532) When it is determined that the marker 492 is on the buttons 493, 494, and 495, the control unit 302 meditates the user's eyelid twice in a short time based on the captured image. It is determined whether or not. Note that the control unit 302 performs determination using a pattern matching technique or an image comparison technique based on image feature amounts. When it is determined that the user's eyelid has been meditated twice in a short time (step S532; Yes), the process proceeds to step S533. When it is determined that the user's eyelid has not been meditated twice in a short time (step S532; No), the process returns to step S528.

(Step S533) When it is determined that the user's eyelid has been meditated twice in a short time, the control unit 302 determines whether or not the “end button” 493 has been selected by blinking twice. When it is determined that the “end button” 493 has been selected by blinking twice (step S533; Yes), the process proceeds to the circle D (FIG. 31, step S511). When it is determined that the “end button” 493 is not selected by two blinks, that is, it is determined that another button is selected (step S533; No), the process proceeds to step S534.

(Step S534) If it is determined that the “end button” 493 is not selected by two blinks, the control unit 302 performs an operation corresponding to the selected button, and returns to step S528.

(Steps S535 to S538) When it is determined that the user has selected the buttons 493 to 495 with the operation switch 30, the control unit 302 performs step S535 based on the coordinates of the buttons 493 to 495 and the bright spot of the user's eyes. To S538 to perform gazing point detection calibration.

(Step S <b> 539) The control unit 302 determines whether or not the “end button” 493 is selected by the operation switch 30. When it is determined by the operation switch 30 that the “end button” 493 has been selected (step S539; Yes), the process proceeds to the circle D (FIG. 31, step S511). If it is determined by the operation switch 30 that the “end button” 493 has not been selected (step S539; No), the process proceeds to step S540.

(Step S540) When the “end button” 493 is not selected by the operation switch 30, that is, when it is determined that no other button is selected, the control unit 302 performs an operation corresponding to the selected button. Return to step S331.

In the HMD 1 of the embodiment, even when the eye gaze input application is started and when the application is being executed, the gaze point detection is performed every time an operation image is selected by an operation of the operation unit. Calibrate the calculation formula for. As a result, according to the HMD 1 of the embodiment, the gaze point detection is automatically performed every time an operation image is selected at the timing when the user operates the operation unit. Is possible. Further, in the calibration for detecting the gazing point, as described above, the HMD 1 of the present embodiment uses the image of the region including the one eye of the user and does not need to capture the image including both eyes. Therefore, the HMD 1 can be downsized.
In the HMD 1 of the present embodiment, the user's eyes can be irradiated with appropriate light by turning on and off the eyeball illumination light 166 set in the “eyeball illumination lamp selection process”. For this reason, it is possible to accurately perform calibration for detecting a gazing point.

In this embodiment, the password input image and the menu image have been described as examples of the display image, but the present invention is not limited to this. The display image may be an image that prompts the user to select an operation image by operating the operation unit. The operation image may be an image that prompts the user to select an operation image by operating the operation unit.
If it is determined in step S522 in FIG. 32 that the calibration is not sufficient, the display position of the operation image displayed on the display image may be changed and displayed. It may be changed and displayed. Alternatively, a new display image including the operation image may be displayed on the LCD 63.

  As described with reference to FIGS. 9 to 15, when the size of the image is reduced and the position is moved before the calibration for detecting the gazing point, for example, in the selected display region T. The password input screen 400e, the menu screen 400f, and the like may be reduced and displayed. In addition, after performing calibration for gazing point detection, when a gaze input is performed by reducing the size of the image and moving the position, the memory is stored based on the display position and the reduction ratio of the image. The coordinates may be converted by correcting the value of the relational expression stored in the unit 303.

Note that the processing unit 123 may turn on the eyeball illumination light 166 when detecting the position of the user's eye based on the captured image. In addition, when the user's eyes are separated from the display unit 60 (see FIGS. 12 and 14), the processing unit 123 calculates the interval between the bright spots 604A and the bright spots 604B (see FIG. 12) in order to calculate the reduction ratio of the display area. 11) and the distance between the bright spot 604C and the bright spot 604D (see FIG. 12).
By reducing the display area S shown in FIG. 9 to the display area T by this process, the processing unit 123 can allow the user to look over the entire display. Further, when it is determined that the intermediate position between the bright spot 604E and the bright spot 604F (see FIG. 14) is shifted from the intermediate position between the bright spot 604A and the bright spot 604B, the processing unit 123 It can be seen that the user's eyes are shifted by the vector V1. In this way, the image switching unit 306 (see FIG. 22) of the processing unit 123 can shift the display area T in FIG. 9 to the display area U.

  In this embodiment, the example in which the password input screen shown in FIG. 35 is used for calibration for detecting a point of interest has been described, but the present invention is not limited to this. Keys, buttons, icons, etc. that can be selected by the operation switch 30 may be displayed on the display panel 63. Also in this case, unlike the prior art, keys, buttons, icons, and the like relating to operations that the user is required to perform are displayed, not just for calibration. Since the user has a very high probability of gazing at these keys, buttons, icons, etc. during operation, the user can perform calibration for gazing point detection based on the captured image at the time of operation.

Next, “business logging processing” will be described.
FIG. 39 is a flowchart of a procedure of business logging processing using the HMD 1 according to the present embodiment. Hereinafter, a case where the user wears the HMD 1 on the head will be described.

(Step S <b> 601) When the user of the HMD 1 performs a task of inspecting a device to be inspected, the user operates the operation switch 30 or the touch switch 34 to start “task logging process”. Next, the processing unit 123 of the HMD 1 transmits a user ID to the server 200 as an initial operation. Next, as an initial operation, the server 200 transmits inspection information stored in the storage unit 202 in association with the user ID to the HMD 1 in response to receiving the user ID from the HMD 1.
Next, the action history registration unit 309 of the processing unit 123 initializes the suffix “i” for counting the examination positions stored in the storage unit 303 to “0”. This suffix is associated with each inspection position. For example, i = 1 is associated with inspection position 1 and i = 2 is associated with inspection position 2. The relationship between the suffix and the inspection position may be stored in advance in the storage unit 303, or may be stored in advance in the storage unit 202 of the server 200, and the server 200 may include the inspection information and transmit it to the HMD1. May be.

(Step S602) The action history registration unit 309 adds “1” to the suffix “i”.

(Step S603) The action history registration unit 309 extracts the location information of the inspection target device corresponding to the inspection position 1 from the received inspection information. Next, the action history registration unit 309 displays the direction in which the inspection target device is installed (hereinafter referred to as the direction of the inspection position) on the display panel 63 based on the extracted location information of the inspection target device. For example, the action history registration unit 309 reads and reads a plan view of the floor on which the inspection target device stored in advance in the storage unit 303 is associated with the inspection target device ID based on the extracted location information. The position where the inspection target apparatus is installed is superimposed on the plan view and displayed on the display panel 63. In addition, the action history registration unit 309 displays on the display panel 63 which direction (east, west, north, and north) the inspection target device is installed in the plan view based on the detection value detected by the geomagnetic sensor 133.
The action history registration unit 309 reads an image corresponding to the inspection target device stored in advance in the storage unit 303 in association with the inspection target device ID, and also superimposes the read image corresponding to the inspection target device on the plan view. Then, it may be displayed on the display panel 63.

(Step S604) The action history registration unit 309 selects the eye illumination light selection processing procedure (see FIG. 9) in order to determine whether or not the user is observing information displayed on the display panel 63. The eyeball illumination light 166 is turned on.

(Step 605) Next, the action history registration unit 309 outputs an instruction to the observation availability determination unit 310 to determine whether or not the user is observing the image displayed on the display panel 63. .
Next, the observability determination unit 310 determines whether or not the user is observing information displayed on the display panel 63 in response to an instruction to perform observability determination being input, and determines the determination result. It outputs to the action history registration unit 309. Next, the action history registration unit 309 determines whether the user is observing information displayed on the display panel 63 based on the determination result output by the observation availability determination unit 310.
If it is determined that the user has not observed the information displayed on the display panel 63 (step S605; No), the action history registration unit 309 proceeds to step S606, and the user observes the information displayed on the display panel 63. When it determines with having carried out (step 605; Yes), it progresses to step S607.

(Step S606) If the action history registration unit 309 determines that the user has not observed the information displayed on the display panel 63 based on the determination result output by the observation availability determination unit 310, the behavior history registration unit 309 The backlight 62 is turned off. As described above, when the HMD 1 determines that the user does not observe the information displayed on the display panel 63, the backlight 62 is turned off, so that the power of the HMD 1 can be saved.
(Step S <b> 607) When the action history registration unit 309 determines that the user is observing information displayed on the display panel 63 based on the determination result output by the observation availability determination unit 310, the action history registration unit 309 is for the display panel 63. The backlight 62 is turned on.
(Step S608) The action history registration unit 309 turns off the eyeball illumination light 166 because the determination as to whether or not the user is observing the information displayed on the display panel 63 has been completed.

(Step S609) The action history registration unit 309 acquires the positioning information measured by the GPS sensor 135. The positioning information to be acquired is information indicating latitude, information indicating longitude, information indicating altitude, and information indicating time.

(Step S610) The action history registration unit 309 transmits a movement log (hereinafter referred to as a movement log) of the HMD 1 in which the user ID is associated with the acquired positioning information to the server 200 via the transmission / reception unit 308. The action history registration unit 309 transmits the movement log to the server 200 at a predetermined time interval, for example. Next, the server 200 stores the received movement log of the HMD 1 in the storage unit 202.
FIG. 40 is a diagram illustrating an example of a movement log stored in the storage unit 202. As shown in FIG. 40, the storage unit 202 indicates the user ID using the HMD1, the device ID that is identification information of the HMD1, time information, position information (latitude information, longitude information, and altitude). Information) is associated and stored.

(Step S611) The action history registration unit 309 acquires image data captured by the camera 64. The behavior history registration unit 309 determines whether the user has arrived in front of the inspection target device based on the barcode included in the acquired image data.
When the action history registration unit 309 determines that the user has arrived in front of the inspection target device (step S611; Yes), the process proceeds to step S612 and determines that the user has not arrived in front of the inspection target device ( Step S611; No), it returns to step S603.
FIG. 41 is a diagram illustrating an example of a barcode 850 attached to the inspection target apparatus 851.
Here, the barcode is, for example, a two-dimensional barcode (for example, a QR code (registered trademark)) as shown by the barcode 850 in FIG. As shown in the figure, the barcode 850 is attached to, for example, the side surface of the inspection target device 851. First, the action history registration unit 309 determines whether or not such an image of the barcode 850 is included in the image data. The action history registration unit 309 extracts an image of the barcode 850 from the image data when the image of the barcode 850 is included in the image data. Next, the action history registration unit 309 extracts the inspection target device ID embedded in the barcode 850. If the inspection target device ID can be extracted from the barcode 850, the behavior history registration unit 309 determines that the user has arrived in front of the inspection target device. Further, the action history registration unit 309 matches the inspection target device ID from the barcode 850 even when the barcode 850 cannot be extracted from the image data or the barcode 850 can be extracted from the image data. If the user cannot be extracted, it is determined that the user has not arrived in front of the inspection target device.

(Step S612) The action history registration unit 309 stores parameters relating to the position in advance in its own unit. The parameter La (i) is a longitude parameter, and the parameter Lo (i) is a latitude parameter. The parameter Al (i) is an advanced parameter, and the parameter T (i) is a time parameter. The parameter V (i) is an average speed parameter, and the parameter X (i) is a movement distance parameter. The parameter P (i) is a parameter for position information.
When it is determined that the user has arrived at the inspection target device, the action history registration unit 309 inputs the longitude into the parameter La (i) and the latitude into the parameter Lo (i) based on the positioning information acquired in step S609. input. The action history registration unit 309 inputs the altitude into the parameter Al (i) and the time into the parameter T (i) based on the positioning information acquired this time. In addition, the action history registration unit 309 causes the storage unit 303 to store each parameter input with each positioning information. In addition, the action history registration unit 309 transmits each parameter to which the positioning information is input, included in the movement log, to the server 200.
Next, the server 200 stores each parameter included in the received movement log in the storage unit 202 in association with the movement log of the HMD 1.

(Step S613) The action history registration unit 309 extracts address information extracted from the barcode. Next, the action history registration unit 309 acquires information related to the inspection target device through the network based on the address information extracted from the barcode.
(Step S614) The action history registration unit 309 displays information related to the inspection target device acquired through the network on the display panel 63.

FIG. 42 is a diagram illustrating an example of information displayed on the display panel 63 according to the present embodiment. FIG. 42A is a diagram illustrating an example of an image 800 displayed on the display panel 63 when the line of sight is input and when the operation switch 30 or the touch switch 34 is operated. FIG. 42B is a diagram illustrating an example of an image 800a displayed on the display panel 63 when a selection item with a pointing finger is selected.
As shown in FIGS. 42A and 42B, the image 800 and the image 800a include an information image 801 indicating the number and name of the inspection target device, and an information image indicating the inspection item (confirmation item). 802 includes an information image 803 indicating precautions and an image 804 indicating the result of the inspection. The image 804 indicating the result of the inspection includes an image “OK” 804a indicating that there is no problem in the result of the inspection, and an image “NG” 804b indicating information indicating that there is a problem in the result of the inspection. Is included. As shown in FIG. 42A, the image 800 includes a cursor image 805 at the time of line-of-sight input. As shown in FIG. 42B, the image 801a includes an image 806 including the user's fingertip when selecting a selection item by pointing. Note that the image 806 including the user's fingertip is obtained by detecting the user's fingertip captured by the camera 64 by the processing unit 123 and superimposing the image on the image 800a. The image display unit 307 of the processing unit 123 displays the images 800 and 800a so that the displayed images 801 to 804 are not obstructed by the image 806 including the user's fingertip.

Here, an example in which selection items are selected by line-of-sight input and operation of the operation switch 30 or the touch switch 34 will be described. In accordance with the result of the inspection performed on the inspection target device, the user displays an image “□ OK” 804a or an image “□ NG” 804b displayed on the display panel 63 as shown in FIG. Selection is performed by line-of-sight input, operation of the operation switch 30 or the touch switch 34.
Next, an example of selecting a selection item by pointing will be described. As shown in FIG. 42B, the user moves the user's finger in the space captured by the camera 64 so that the user's finger is displayed in the image displayed on the display panel 63, and the inspection is performed. A selection is made by pointing to any of the images 804 showing the results. Then, the action history registration unit 309 determines whether or not item selection with a pointing is performed based on an image captured by the camera 64.
42A and 42B, the image 800 and the image 800a are displayed in Japanese. However, the user operates the operation switch 30 and the touch switch 34 in advance to operate the HMD1. When the language used in is selected, the action history registration unit 309 may display each information on the display panel 63 in the selected language. In addition, the action history registration unit 309 may announce an announcement or the like that prompts the user to observe the information displayed on the display panel 63 with the selected language voice.

(Step S615) Returning to FIG. 39, the action history registration unit 309 determines whether or not information indicating that it is a notification target is set in the acquired information related to the inspection target device.
The information indicating that it is a target of notification is any information among update date / time information, information with update information, information with notes, and information with check items. Next, when it is determined that the update date information is included, the behavior history registration unit 309 compares the current date acquired in, for example, step S609 with the updated date. Next, as a result of the comparison, if the action history registration unit 309 determines that the update has not been performed for the past three days (step S615; No), the action history registration unit 309 proceeds to step S621. If the action history registration unit 309 determines that none of the updated information, the caution information, and the check item information is included (step S615; No), the action history registration unit 309 proceeds to step S621.
Here, the reason why the steps S616 to S620 are skipped when the update has not been performed in the past three days, or when any of the update information, the precaution information, and the check item information is not included. To do. When the user inspects the inspection target device every day, there is a high possibility that the user already remembers the contents of the inspection item. For this reason, when there is no change in the contents of the inspection item or when important precautions are not added, the HMD 1 dares to display information related to the inspection target device displayed on the display panel 63 to the user. You may not be prompted to observe. Therefore, in such a case, the HMD 1 of this embodiment skips steps S616 to S620. Note that the action history registration unit 309 of the HMD 1 may perform steps S616 to S620 without skipping when the frequency of the inspection by the user is low, or when the user who performs the inspection changes depending on the inspection date.
On the other hand, if the action history registration unit 309 determines that any one of the update date / time information, the update presence information, the caution item presence information, and the check item presence information is included (step S615; Yes), the step The process proceeds to S616.
Next, the action history registration unit 309, for example, if the information related to the acquired device to be inspected includes check item presence information, the check for inputting the result of checking the check item on the display panel 63 A box (hereinafter referred to as a check box) image is displayed. For example, as shown in FIG. 42A, an image 804 includes a “□ OK” image 804a indicating that there is no problem as a result of the inspection, and “□ NG” indicating that there is a problem as a result of the inspection. An image 804b is included. Of each image, “□” is called a check box.

(Step S616) If the information acquired via the network has been updated, if there are cautions, or if there are check items, the user checks the information displayed on the display panel 63 and performs an inspection. There is a need. For this reason, the action history registration unit 309 outputs an announcement from the ear speaker 23 by voice, prompting the user to observe the information displayed on the display panel 63. The announcement output from the ear speaker 23 is in accordance with the updated content, for example, “Document has been changed. Look at the screen”. When the stereo earphone 100 is connected to the HMD 1, the action history registration unit 309 outputs an announcement from the stereo earphone 100 by voice to urge the user to observe the information displayed on the display panel 63. To do.

(Step S617) The action history registration unit 309 follows an eyeball illumination light selection processing procedure (see FIG. 9) in order to determine again whether or not the user is observing information displayed on the display panel 63. The selected eyeball illumination light 166 is turned on.
(Step S618) The action history registration unit 309 turns on the backlight 62. In response to the user's announcement to observe the information displayed on the display panel 63, when the user suddenly turns his / her eyes on the information displayed on the display panel 63, the user's line of sight is detected. When the backlight 62 is turned on, a time lag occurs until the user confirms the image displayed on the display panel 63. For this reason, the HMD 1 of this embodiment turns on the backlight 62 at this time in order to eliminate a time lag until the user confirms the image displayed on the display panel 63.

(Step S619) The observability determination unit 310 determines whether the user is observing information displayed on the display panel 63, as in the process of step S605. When it is determined that the user is observing information displayed on the display panel 63 (step S619; Yes), the observation availability determination unit 310 proceeds to step S621, and the user observes information displayed on the display panel 63. When it determines with not having carried out (step S619; No), it progresses to step S620.

(Step S620) When it is determined that the user does not observe the information displayed on the display panel 63, the action history registration unit 309 performs steps S619 and S620 until the user observes the information displayed on the display panel 63. Is repeated, and a warning sound that prompts the user to observe the information displayed on the display panel 63 is output from the ear speaker 23.

(Step S621) The action history registration unit 309 determines whether or not the information related to the acquired inspection target device includes check item presence information. If the action history registration unit 309 determines that the information related to the acquired inspection target device includes check item presence information (step S621; Yes), the action history registration unit 309 proceeds to step S622, and acquires the information related to the acquired inspection target device. If it is determined that the check item presence information is not included (step S621; No), the process proceeds to step S624.

(Step S622) When it is determined that the information related to the acquired device to be inspected includes check item presence information, the action history registration unit 309 performs an inspection on the check item for the user and checks the result of the inspection. An announcement that prompts the user to output is output from the ear speaker 23 by voice.
Next, in accordance with the announcement, the user inspects the inspection target device with respect to the inspection target device, and selects the inspection result by line-of-sight input. As an example, the user gazes at a “□ OK” image 804 a or a “□ NG” image 804 b displayed on the display panel 63. The action history registration unit 309 displays, on the display panel 63, an image in which the check box of the “□ OK” image 804a or the “□ NG” image 804b is input according to the user's gaze. Alternatively, the user operates the operation switch 30 or the touch switch 34. The action history registration unit 309 displays, on the display panel 63, an image in which a check is input to the check box of the “OK” image 804a or the “OK” image 804b displayed on the display panel 63 in accordance with a user operation. It may be. Alternatively, the user selects the “□ OK” image 804a or the “□ NG” image 804b displayed on the display panel 63 by rotating the user's head. The action history registration unit 309 may input a check in the check box according to the rotation of the user's head as described above, and display the image on which the check is input on the display panel 63.

Here, selection of the “□ OK” image 804a or the “□ NG” image 804b by turning the head will be described. In such a case, it is desirable that the “□ OK” image 804a and the “□ NG” image 804b are arranged not to the same position in the left-right direction but to be shifted from side to side. For example, when the user rotates the head in the right direction at an angular velocity equal to or higher than a predetermined value, the action history registration unit 309 determines the acceleration of the right rotation based on the detection value of the angular velocity sensor 134. The movement direction and the movement amount due to the part rotation are calculated. Next, the action history registration unit 309 moves the cursor image 805 (FIG. 42) to the “□ OK” image 804a or the “□ NG” image 804b on the display panel 63 based on the calculated moving direction and moving amount. To display.
As described above, the action history registration unit 309 moves the cursor image 805 by detecting the movement of the user's head based on the detection value of the angular velocity sensor 134 in the selection based on the rotation of the head. Select. Then, for example, when the user moves his / her face in the front-rear direction as if the user whispered, the control unit 302 determines that the selection item has been determined.

(Step S623) The action history registration unit 309 determines whether or not the check item inspection has been completed. Specifically, the action history registration unit 309 checks the check box when either the “□ OK” image 804a or the “□ NG” image 804b displayed on the display panel 63 is checked. It is determined that the inspection of the item has been completed, and if no check is input in any of the check boxes, it is determined that the inspection of the check item has not been completed. If the action history registration unit 309 determines that the check item inspection has been completed (step S623; Yes), the process proceeds to step S624. If the action history registration unit 309 determines that the check item inspection has not been completed (step S623; No), Steps S622 and S623 are repeated.

(Step S624) The action history registration unit 309 determines whether or not a script file for executing a predetermined operation on the HMD 1 is included from information related to the inspection target device acquired based on the barcode. When it is determined that the script file is included in the information related to the inspection target device, the action history registration unit 309 executes the script and transmits data obtained as a result of the execution to the server 200.
Note that the script file may be stored in advance in the flash memory 122 in the HMD 1. In this case, the action history registration unit 309 extracts a barcode from the image data captured by the camera 64, and extracts information corresponding to the script file embedded in the extracted barcode. Next, the action history registration unit 309 may read a script file stored in the flash memory 122 based on information corresponding to the extracted script file and execute the script.

In the present embodiment, the HMD 1 performs a predetermined operation based on the script file embedded in the barcode as described above. That is, the user of the HMD 1 images the barcode attached to the inspection target device using the camera 64, so that the HMD 1 has information on the position P (i) where the inspection target device is installed, and the inspection target. Data corresponding to the device can be acquired and automatically transmitted to the server 200. Therefore, the administrator of the system including the HMD 1 and the server 200 can automatically acquire information on the position P (i) where the inspection target device is installed, data corresponding to the inspection target device, and the like.
As described above, the predetermined operation to be executed by the HMD 1 includes, for example, the temperature of the position where the still image or moving image is captured by the camera 64, the sound is collected by the sound collecting microphone 24, and the inspection by the temperature / humidity sensor 136 is performed. For example, acquisition of humidity, acquisition of information on the position P (i) by the GPS sensor 135, and the like.
Therefore, the HMD 1 of the present embodiment corresponds to the information on the position P (i) where the inspection target device is installed and the inspection target device without depending on the memory, intuition, experience, etc. of the user who uses the HMD 1. Data can be collected. Further, the HMD 1 of the present embodiment can update the information updated to the user who is using the HMD 1 based on the barcode attached to the inspection target device even when the displayed information is changed. Since the notification can be made, it becomes possible to perform the inspection according to the environment and situation for the inspection. Note that the information stored in the server 200 may be stored in advance in the storage unit 202 of the server 200 by, for example, the system administrator operating the PC 210.

(Step S625) Returning to FIG. 39, the action history registration unit 309 transmits all information indicating the contents processed by the HMD 1 in steps S611 to S624 to the server 200 in association with the user ID as an observation log. In this observation log, for example, information indicating whether or not the user using the HMD 1 was observing information displayed on the display panel 63 in step S619, and a warning sound was output in step S620. Information indicating time is included.
Next, the server 200 receives the observation log and the user ID, and stores the received observation log in the storage unit 202 in association with the user ID.
(Step S626) The action history registration unit 309 turns off the eyeball illumination light 166 and the backlight 62.

(Step S627) The action history registration unit 309 confirms the suffix value stored in the storage unit 303, and determines whether or not the inspection has been completed for all predetermined inspection target devices. . If the action history registration unit 309 determines that the inspection has not been completed for all the inspection target devices (step S627; No), the process returns to step S602, and hereinafter, until the inspection of all the inspection target devices is completed. , Steps S602 to S627 are repeated. If the action history registration unit 309 determines that the inspection has been completed for all the devices to be inspected (step S627; Yes), the action history registration unit 309 ends the process.

  As described above, the HMD 1 in the present embodiment includes a display unit (display panel 63) that displays information, and an observation availability determination unit (observation availability) that determines whether the display unit is in a state that can be observed by the user. It is determined that the display unit is not observable by the user based on the determination unit 310), the notification unit (sound control unit 305) that performs notification by sound, and the result of determination by the observation availability determination unit. A control unit (behavior history registration unit 309) that controls to notify from the notification unit. With this configuration, the HMD 1 in the present embodiment can prompt the user to observe the information displayed on the display panel 63 when the user cannot observe the information displayed on the display panel 63. Therefore, in the HMD 1 of the present embodiment, the user can check the information displayed on the display panel 63, thereby preventing the user from overlooking the information displayed on the display unit.

Moreover, in HMD1 in this embodiment, the action history registration part 309 acquires the information regarding a test object apparatus from the server 200 based on the barcode stuck on the test object apparatus. The action history registration unit 309 determines whether or not it is necessary to display (notify) the information related to the acquired inspection target device on the display panel 63 based on the acquired information related to the inspection target device. When it is determined that it is necessary to display the acquired information related to the inspection target device on the display panel 63, and the observation availability determination unit 310 determines that the user does not observe the information displayed on the display panel 63. In such a case, a notification is made to prompt the user to observe the information displayed on the display panel 63.
Thereby, the HMD 1 in the present embodiment can prompt the user to observe the information displayed on the display panel 63. Therefore, in the HMD 1 of the present embodiment, the user can confirm without missing the information displayed on the display panel 63.

Further, in the HMD 1 in the present embodiment, the action history registration unit 309 determines that it is not necessary for the user to check the information displayed on the display panel 63 based on the information related to the inspection target device, and determines whether or not to observe. When the unit 310 determines that the user is not observing the display panel 63, the backlight 62 for the display panel 63 is turned off.
Thereby, since HMD1 in this embodiment lights a backlight only when the user is observing the information displayed on the display panel 63, it can save power.

In the HMD 1 of the present embodiment, the action history registration unit 309 transmits to the server 200 the movement log and the observation log of the user who is inspecting the inspection target device. In the HMD 1 of the present embodiment, the action history registration unit 309 causes the HMD 1 to perform a predetermined operation according to the script file extracted from the information related to the inspection target device, and causes the server 200 to store the data acquired through the predetermined operation. Send. The server 200 stores the movement log, the observation log, and the data received from the HMD 1 in the storage unit 202 in association with the user ID.
Thereby, the HMD 1 in the present embodiment can record in the server 200 when and where the user inspected which device. Therefore, in the HMD 1 of the present embodiment, the system supervisor can monitor the progress of the user's inspection work.

In the HMD 1 of the present embodiment, first, “left and right eye detection processing” is performed. Thereby, the HMD 1 of the present embodiment uses the image of the region including the user's one eye included in the captured image, and the eyes observing the image displayed on the display unit 60 are the right eye and the left eye. Recognize which one of them.
As a result, regardless of the posture of the HMD mounted on the user's head, the user can recognize whether the eye observing the image displayed on the display unit 60 is the right eye or the left eye. HMD1 can be realized. Further, the HMD 1 of the present embodiment includes the entire face or both eyes of the user in order to determine whether the eye observing the image displayed on the display unit 60 is the right eye or the left eye. It is not necessary to image the area. For this reason, HMD1 of this embodiment can suppress the enlargement of an own apparatus.
Furthermore, the HMD 1 of the present embodiment displays the display unit 60 according to the result of recognizing which of the right eye and the left eye is observing the image displayed on the display unit 60 in this way. Change the display angle of the image to be displayed. And HMD1 of this embodiment displays an image on the display part 60 according to this display angle. As a result, according to the HMD 1 of the present embodiment, the user can observe the image displayed on the display unit 60 at an appropriate display angle regardless of the posture of the HMD 1 mounted on the user's head.

Furthermore, the HMD 1 of the present embodiment performs “gaze point calibration processing” after “left and right eye detection processing”. In this “gaze point calibration process”, the user operates the operation image included in the display image displayed on the display panel 63 using the operation unit (the operation switch 30 and the touch switch 34). The HMD 1 according to the embodiment acquires a captured image in which an operation image included in a display image displayed on the display unit 60 is being watched when the operation unit is operated.
Further, the HMD 1 of the present embodiment selects an image in the selected imaging range Q or R (see FIG. 19) based on the determination result of “left and right eye detection processing”. That is, the HMD 1 according to the present embodiment uses the captured image for “gazing point calibration processing” based on the result of recognizing whether the eye observing the image is the right eye or the left eye. Select an image containing the user's eye area. Then, the HMD 1 of the present embodiment performs “gazing point calibration processing” using the image in the selected imaging range. And HMD1 of this embodiment acquires the 1st coordinate based on the operated operation image and the 2nd coordinate based on a captured image which are parameters for operating based on a user's look. Then, the HMD 1 of the embodiment automatically performs calibration for gazing point detection based on the acquired combination of the first coordinate and the second coordinate.
For this reason, in the HMD 1 of the present embodiment, the calibration for gaze point detection is performed by simply operating the operation image included in the display image without the user having to select the calibration setting from the menu or the like. It can be performed. Further, in the HMD 1 of the present embodiment, as in the past, calibration for gazing point detection is performed in which the user is gazing at a calibration image for gazing point detection as shown in FIG. Even if it is not necessary, calibration for detecting a gazing point can be performed by simply operating the operation image included in the display image. As a result, according to the present embodiment, convenience can be improved.

In the HMD 1 of the embodiment, even when the eye gaze input application is started and when the application is being executed, the gaze point detection is performed every time an operation image is selected by an operation of the operation unit. Calibrate the calculation formula for. As a result, according to the HMD 1 of the embodiment, the gaze point detection is automatically performed every time an operation image is selected at the timing when the user operates the operation unit. Is possible. Further, in the calibration for detecting the gazing point, as described above, the HMD 1 of the present embodiment uses the image of the region including the one eye of the user and does not need to capture the image including both eyes. Therefore, the HMD 1 can be downsized.
In the HMD 1 of the present embodiment, the user's eyes can be irradiated with appropriate light by turning on and off the eyeball illumination light 166 set in the “eyeball illumination lamp selection process”. For this reason, it is possible to accurately perform calibration for detecting a gazing point.

  As described with reference to FIGS. 9 to 15, when the size of the image is reduced and the position is moved before the calibration for detecting the gazing point, for example, in the selected display region T. The password input screen 400e, the menu screen 400f, and the like may be reduced and displayed. In addition, after performing calibration for gazing point detection, when a gaze input is performed by reducing the size of the image and moving the position, the memory is stored based on the display position and the reduction ratio of the image. The coordinates may be converted by correcting the value of the relational expression stored in the unit 303.

  In this embodiment, the example in which the password input screen shown in FIG. 35 is used for calibration for detecting a point of interest has been described, but the present invention is not limited to this. Keys, buttons, icons, etc. that can be selected by the operation switch 30 may be displayed on the display panel 63. Also in this case, unlike the prior art, keys, buttons, icons, and the like relating to operations that the user is required to perform are displayed, not just for calibration. Since the user has a very high probability of gazing at these keys, buttons, icons, etc. during operation, the user can perform calibration for gazing point detection based on the captured image at the time of operation.

Note that the processing unit 123 may turn on the eyeball illumination light 166 when detecting the position of the user's eye based on the captured image. In addition, when the user's eyes are separated from the display unit 60 (see FIGS. 12 and 14), the processing unit 123 calculates the interval between the bright spots 604A and the bright spots 604B (see FIG. 12) in order to calculate the reduction ratio of the display area. 11) and the distance between the bright spot 604C and the bright spot 604D (see FIG. 12).
By reducing the display area S shown in FIG. 9 to the display area T by this process, the processing unit 123 can allow the user to look over the entire display. Further, when it is determined that the intermediate position between the bright spot 604E and the bright spot 604F (see FIG. 14) is shifted from the intermediate position between the bright spot 604A and the bright spot 604B, the processing unit 123 It can be seen that the user's eyes are shifted by the vector V1. In this way, the image switching unit 306 (see FIG. 22) of the processing unit 123 can shift the display area T in FIG. 9 to the display area U.

[Second Embodiment]
In the second embodiment, the eyeball illumination light 166 is switched on and off for many positions on the display unit 60, including the position 600a and the position 600b of the display unit 60 (see FIG. 34 of the first embodiment). explain.

  FIG. 43 is a diagram illustrating a mounting form of the head mounted display (second display mode, save mode) according to the present embodiment. In FIG. 43, the apparatus main body 21 is inclined downward by an angle α with respect to the XY plane including a line that virtually connects the left and right eyes of the user.

  FIG. 43A is a diagram illustrating a mounting form in which the display unit 60 is rotated 90 degrees clockwise from the first display mode of the right-eye observation form. This mounting form is referred to as a second display mode. In the second display mode, the display unit 60 rotates clockwise about the Y ′ axis (see FIG. 3), with the storage unit 61 a and the spherical unit 61 b as rotation axes, so that the tip of the display unit 60 is located on the upper side. Rotated by 90 degrees. Thereby, the display part 60 and the finder opening part 67 are vertically arranged in the Z-axis direction when viewed from the user side. Further, the display panel 63 displays an image having an aspect ratio that is vertically long.

  That is, in the second display mode, the image displayed on the display panel 63 of the display unit 60 is displayed at a position in the diagonally forward right direction with respect to the user's right eye. In the second display mode, the eyeball illumination lights 166c and 166d in the vicinity of the right eye are turned on, and the eyeball illumination lights 166a and 166b are turned off.

  FIG. 43B is a diagram illustrating a mounting form in which the display unit 60 is rotated 90 degrees counterclockwise from the first display mode of the right-eye observation form. This wearing form is referred to as a save mode. In the evacuation mode, the display unit 60 is rotated counterclockwise by 90 degrees around the Y ′ axis (see FIG. 3), with the accommodating portion 61 a and the spherical portion 61 b as rotation axes so that the tip of the display portion 60 is positioned on the lower side. It is rotated in the turning direction. Thereby, the display unit 60 and the viewfinder opening 67 are retracted from the visual field of the user's right eye. In the retreat mode, the eyeball illumination lights 166a to 166d may all be turned off.

  The rotation axis of the display unit 60 is common in the first display mode, the second display mode, and the save mode. For this reason, if the display part 60 is rotated 90 degree | times from 2nd display mode or save mode, it can return to 1st display mode. That is, the user can retract the display unit 60 from the visual field of the right eye of the user or return it to the observation position without removing the HMD 1 or changing the angle of the apparatus main body unit 21.

  In the example shown in FIGS. 43A and 43B, the example in which the display unit 60 is arranged vertically from the first display mode of the right-eye observation mode has been described, but FIG. Similarly, from the first display mode of the left-eye observation mode shown, the display unit 60 can be arranged vertically. That is, from the first display mode of the left-eye observation mode, the display unit 60 is rotated by 90 degrees in the counterclockwise direction on the XZ plane with the accommodating portion 61a and the spherical portion 61b as the central axes, so that the user can The HMD 1 can be switched to the second display mode of the eye observation form. Similarly, from the first display mode of the right-eye observation mode, the display unit 60 is rotated by 90 degrees in the clockwise direction around the Y ′ axis (see FIG. 3), with the accommodating portion 61a and the spherical portion 61b as the central axes. By doing so, the user can switch the HMD 1 to the save mode of the left-eye observation mode.

  Based on the detection value detected by the acceleration sensor 132 and an image (see FIGS. 26 and 27) captured by the image sensor 72 (see FIG. 22), the control unit 302 uses the right-eye observation form or the right-eye observation form. And the first display mode, the second display mode, or the save mode. Here, with respect to the image captured by the image sensor 72, the control unit 302, for example, the relative position between the eyebrows and the eye, the behavior of eyelids in blinking, the eyebrow contour, the eye contour, and the lowering of the left and right corners of the eye Based on the above, the right-eye observation form or the right-eye observation form and the first display mode, the second display mode, or the save mode are determined. In addition, when the user's eyeball is not captured in the image captured by the image sensor 72, the control unit 302 determines that the mode is the save mode.

  FIG. 44 is a diagram for explaining the relationship between the position of the display unit 60 and the eyeball illumination light 166 according to the present embodiment, and is a diagram when the user's face is viewed horizontally from the front. 44, positions 600a to 600h indicated by alternate long and short dashed lines are examples of positions where the display unit 60 is arranged.

<First display mode>
The position 600a is a position of the display unit 60 that is disposed sideways below the right eye of the user. In addition, the position 600b is a position of the display unit 60 that is disposed sideways below the left eye of the user. The position 600c is a position of the display unit 60 that is disposed sideways above the right eye of the user. The position 600d is a position of the display unit 60 that is disposed sideways above the left eye of the user.

<Second display mode>
The position 600e is the position of the display unit 60 that is arranged vertically on the right side of the user's right eye. The position 600f is the position of the display unit 60 that is arranged vertically on the left side of the user's left eye.

  When the display unit 60 is located at any of the positions 600a, 600d, and 600f, the control unit 302 (see FIG. 22) of the HMD1 lights the eyeball illumination lights 166a and 166b located closer to the user's eyes at this position. The eyeball illumination lights 166c and 166d are turned off. On the other hand, when the display unit 60 is in any of the positions 600b, 600c, or 600e, the eyeball illumination lights 166c and 166d located closer to the user's eyes are turned on at this position, and the eyeball illumination lights 166a and 166b are turned off. .

  As described above, the control unit 302 irradiates the user's eyes from the front among the eyeball illumination lights 166a to 166d arranged in the vicinity of the corner of the finder opening 67 (within a predetermined distance). The lighting state is controlled according to the positional relationship between the user and the display unit 60 so that the light is turned on.

<Evacuation mode>
The position 600g is the position of the display unit 60 that is arranged vertically on the lower right side of the user's right eye. The position 600h is the position of the display unit 60 that is arranged vertically on the lower left side of the user's left eye.
When the display unit 60 is located at the positions 600g and 600h, the control unit 302 turns off all of the eyeball illumination lights 166a to 166d.

Next, a selection process for selecting an eyeball illumination light to be turned on at the time of calibration for detecting a gazing point or at the time of business logging process will be described.
FIG. 45 is a flowchart of the selection processing procedure of the eyeball illumination light of the HMD 1 according to this embodiment.

(Step S801) The control unit 302 turns on the eyeball illumination lights 166a and 166b via the eyeball illumination unit 304. Next, the control unit 302 drives the image sensor 72 to acquire a captured image, and stores the acquired captured image in the storage unit 303.

(Step S802) The control unit 302 turns off the eyeball illumination lights 166a and 166b and turns on the eyeball illumination lights 166c and 166d via the eyeball illumination unit 304. Next, the control unit 302 drives the image sensor 72 to acquire a captured image, and stores the acquired captured image in the storage unit 303.

(Step S803) The control unit 302 reads the captured image stored in step S801, and extracts images of bright spots corresponding to the eyeball illumination lights 166a and 166b from the read captured image. Next, the control unit 302 calculates the luminance of the extracted bright spot image. Next, the control unit 302 reads the captured image stored in step S802, and extracts bright spot images corresponding to the eyeball illumination lights 166c and 166d from the read captured image. Next, the control unit 302 calculates the luminance of the extracted bright spot image.
Next, the control unit 302 determines whether the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d. When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d (step S803; Yes), the process proceeds to step S804. When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is not higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d (step S803; No), the process proceeds to step S805.

(Step S804) When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d, the control unit 302 166a and 166b are selected, and the selected result is stored in the storage unit 303. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

(Step S805) When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is not higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d, the control unit 302 performs the eyeball illumination. The lights 166c and 166d are selected, and the selected result is stored in the storage unit 303. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

Also in the second embodiment, the controller 302 adjusts the eyeball according to the positional relationship between the user and the display unit 60 as described above, instead of step S502 or S503 in FIG. The lighting light 166 is turned on and off. Further, the control unit 302 controls the turning on and off of the eyeball illumination light 166 according to the positional relationship between the user and the display unit 60 as described above in step S604 and step S617 in FIG. .
As described above, since the eyeball illumination light 166 is controlled based on the luminance of the eyeball illumination light 166, the HMD 1 of the present embodiment illuminates the eyeball according to the positional relationship between the user and the display unit 60. The light can be selected as appropriate. Further, the HMD 1 of the present embodiment can efficiently take measures for power saving by turning off the unused eyeball illumination light 166.

  In the present embodiment, the example in which turning on and off of the eyeball illumination light 166 is controlled according to the luminance of the eyeball illumination light 166 and the positional relationship between the user and the display unit 60 has been described. An example of controlling the turning on and off of light is not limited to this. For example, the control unit 302 compares the acquired captured image with the discrimination image stored in the storage unit 303 by pattern matching, and according to the positional relationship between the user and the display unit 60 based on the comparison result. Thus, the turning on and off of the eyeball illumination light 166 may be controlled.

As described above, the HMD 1 of the present embodiment is in a positional relationship between the user and the display unit 60 even when the display unit 60 is disposed below, above or outside the user with respect to the user's eyes. In response, the turning on and off of the eyeball illumination light 166 is controlled. As a result, according to the positional relationship between the user and the display unit 60, the HMD 1 of the present embodiment can accurately detect the user's viewpoint based on the bright spot by the light of the eyeball illumination light 166 appropriate for the user's eyes. Thus, calibration for detecting a gazing point can be performed with high accuracy. Accordingly, as in the first embodiment, in the “business logging process”, the HMD 1 of the present embodiment allows the user to work with both hands by selecting the selection items displayed on the display panel 63 by gazing. Even while doing so, it is possible to record logs related to business execution.
Furthermore, since the HMD 1 of the present embodiment turns on the appropriate eyeball illumination light 166 for the user's eyes, it is determined whether or not the user is observing information displayed on the display panel 63 in the “business logging process”. It can be determined with high accuracy. As a result, as in the first embodiment, the HMD 1 of the present embodiment observes the information displayed on the display panel 63 when the user cannot observe the information displayed on the display panel 63. Can be urged.
In addition, the HMD 1 of the present embodiment can reduce the power consumption because the eyeball illumination light 166 can be turned off according to the positional relationship between the user and the display unit 60 as in the first embodiment.

  Note that if the display unit 60 is in front of the user's face and all of the eyeball illumination lights 166a to 166d are determined to be at an approximately equidistant position from the user's horizontal line of sight, the HMD 1 is in principle which eyeball. The illumination light 166 may be turned on. However, normally, there are ceiling lighting, sky, sun, and the like above the eyes, and bright spots from these light sources may exist on the eyeball. Since these bright spots and the bright spot of the eyeball illumination light 166 are difficult to discriminate, when it is determined that the display unit 60 is substantially in front of the user's face, the HMD 1 Control may be performed to turn on the eyeball illumination light 166 arranged at.

  In addition, although the example provided with the eyeball illumination lights 166a to 166d near the long side of the finder opening 67 including the display unit 60 (see FIG. 2), the example provided with the eyeball illumination lights 166a to 166d Not limited. That is, the eyeball illumination light 166 may be disposed in the vicinity of the short side of the finder opening 67.

[Third Embodiment]
Next, the case where the display unit is a see-through optical system will be described.
The difference from the first and second embodiments is only the configuration of the display unit 60.
FIG. 46 is a cross-sectional view of the display unit 60a for describing the optical system according to the present embodiment. The members having the same functions as those in FIG.

  As shown in FIG. 46, the display unit 60a of the present embodiment includes a backlight 62, a display panel 63, a first prism 65a, a second prism 65b, a reflection mirror 66, a viewfinder opening 67, an imaging lens 71, and an image sensor. 72, a quarter-wave plate 75, a polarizing plate 90, a concave lens 91, a quarter-wave plate 92, a liquid crystal 93, a polarizing plate 94, and an opening 95. In FIG. 46, symbols H, I, J, K, O, and T each represent a light beam, and symbol S represents a light beam emitted from the user's eye.

As shown in FIG. 46, the scene in front of the user becomes a light ray O and enters the opening 95. Next, the light beam O passes through the opening 95 and passes through the polarizing plate 94. Next, the light beam O transmits only S-polarized light.
Next, the light beam O enters the liquid crystal 93. Note that the polarization direction of the light beam O is changed by the action of the liquid crystal 93. That is, when the liquid crystal 93 is energized (ON state), the light beam O enters the quarter-wave plate 92 with the S-polarized light incident on the liquid crystal 93 becoming P-polarized light. Next, the light beam O becomes counterclockwise polarized light by the action of the quarter-wave plate 92.

In the present embodiment, the reflection mirror 66 is a semi-transmissive type, for example, transmits 50% light and reflects 50% light. The following description will be made on the assumption that the reflection mirror 66 transmits 50% of light. As a result, 50% of the light beam O enters the quarter-wave plate 75.
The concave lens 91 is a member that is paired with a convex lens constituting the reflection mirror 66 to become a parallel plate with zero power.

Next, the light beam O becomes P-polarized light by the action of the quarter-wave plate 75 and enters the first prism 65a. Next, the light beam O passes through the bonding surface 65 c as it is, passes through the second prism 65 b, and enters the polarizing plate 90. The polarizing plate 90 is configured to transmit P-polarized light.
With the above configuration, an image of the outside world (an image viewed through the opening 95) can be observed together with an image projected from the display panel 63 by the user.

Next, a case where the image of the outside world is erased and only the image projected from the display panel 63 is observed will be described.
In FIG. 46, when the liquid crystal 93 is turned off (off state), the polarization direction of the light beam O is not changed by the liquid crystal 93. For this reason, the light beam O becomes S-polarized light at the polarizing plate 94. Next, the light beam O is incident on the quarter-wave plate 92 as it is. Next, the light beam O becomes clockwise polarized light by the action of the quarter wave plate.

  Next, 50% of the light beam O transmitted through the reflection mirror 66 enters the quarter-wave plate 75. Next, the light beam O becomes S-polarized light by the action of the quarter-wave plate 75. Next, the light beam O that has become S-polarized light enters the first prism 65a. The polarization reflection film provided on the bonding surface 65c reflects S-polarized light. As a result, external light rays do not enter the user's eyes.

  In this way, by switching the liquid crystal 93 between the on state and the off state, it is possible to control whether or not the image of the outside world can be transmitted through the finder and observed by the user. Note that the processing unit 123 performs switching between the ON state and the OFF state of the liquid crystal 93, for example.

In FIG. 46, the action of the polarizing plate 90 is arranged so that the user does not enlarge and observe his / her eyes by the reflecting mirror 66. Explain why users do not observe their eyes enlarged.
The light beam S emitted from the user's eyes transmits only the P-polarized light through the polarizing plate 90, passes through the second prism 65 b, the bonding surface 65 c, and the first prism 65 a and enters the quarter-wave plate 75. Here, the light beam S becomes counterclockwise circularly polarized light by the action of the quarter-wave plate 75, and is reflected as the light beam T by the reflection mirror 66. Next, the light beam T becomes clockwise circularly polarized light by reflection, and becomes S-polarized light when passing through the quarter-wave plate 75.
Next, the light beam T passes through the first prism 65a and enters the joint surface 65c. Here, the joint surface 65c reflects and does not transmit S-polarized light. Therefore, the user's eyes do not receive the luminous flux from his / her own eyes, and his / her eyes cannot be seen on the display screen.

  Further, the imaging lens 71 and the image sensor 72 for taking a peripheral image of the eye are provided adjacent to the finder opening 67. For this reason, the light rays of the imaging lens 71 and the imaging element 72 system can directly photograph the vicinity of the user's eyes without passing through the second prism 65b.

FIG. 47 is a diagram for explaining an observation range and an imaging range according to the present embodiment. In FIG. 47, reference numeral 67a represents the observation range of the image projected from the display panel 63. Reference numeral 67 b represents an imaging range of the imaging element 72.
FIG. 47 is an equivalent view in which the reflecting mirror 66 is replaced with a convex lens, as in FIG.

In FIG. 47, the focal length of the imaging lens 71 is made shorter than the focal length of the reflection mirror 66. For this reason, when FIG. 46 is cut in a cross section perpendicular to the paper surface, as shown in FIG.
In FIG. 47, the imaging lens 71 and the image sensor 72 are offset from the optical center of the reflection mirror 66. However, the user can sufficiently separate his / her eyes from the viewfinder opening 67 by lowering the display unit 60a not toward the front of the eyes but toward the mouth, and the user's eyes can be removed as shown in FIG. It is at the position of the observation distance L81. For this reason, when the user is observing the display panel 63, according to this embodiment, the image sensor 72 can photograph the user's eyes and the surroundings without any problem.

By providing the above-described configuration, it is possible to perform imaging including the user's eyes, and to see through the HMD 1 wearing state and line-of-sight calibration based on the image captured by the image sensor 72 as in the first embodiment. A head-mounted display including an optical system display unit can be provided.
In the second embodiment, as described in the first embodiment, the eyeball illumination light may be arranged in the vicinity of two opposing sides of the four sides of the finder opening 67.

Similarly to the first embodiment, the HMD 1 of the third embodiment provides a notification for prompting the user to observe the display unit when the user determines that the information displayed on the display panel 63 is not observable. Perform by sound. As a result, as in the first embodiment, the HMD 1 of the third embodiment allows the user to observe the information displayed on the display panel 63 when the user cannot observe the information displayed on the display panel 63. Can be prompted by sound. Therefore, the HMD 1 of the third embodiment provides a head mounted display including a display unit 60a of a see-through optical system that allows a user to check information displayed on the display panel 63.
Further, the HMD 1 of the third embodiment determines that the user needs to confirm the information displayed on the display panel 63 based on the information related to the inspection target device, as in the case of the HMD 1 of the first embodiment. If the user determines that the information displayed on the display panel 63 is not observable by the observability determination unit 310, a notification for prompting the user to observe the information displayed on the display panel 63 Perform by sound. As a result, the HMD 1 of the third embodiment prompts the user to observe the information displayed on the display panel 63 when the information displayed on the display panel 63 is important, as in the first embodiment. I can do it. Therefore, the HMD 1 of the third embodiment can provide a head mounted display including the display unit 60a of the see-through optical system that allows the user to check the information displayed on the display panel 63.

  In the first to third embodiments, an example of a head-mounted display that can be used by switching between a right-eye observation mode and a left-eye observation mode has been described as an example of a head-mounted display. For example, it may be a head-mounted display dedicated to the right-eye observation form or dedicated to the left-eye observation form. In this case, among the processes described in the present embodiment, only the process relating to the right eye observation form or the left eye observation form may be performed.

  In the first to third embodiments, the controller 302 may turn on the backlight 62 when recognizing an eye observing the image of the display unit 60 (or the display unit 60a). . As a result, the image including the region of the user's eye included in the captured image becomes clearer, and the arithmetic processing in the “left and right eye detection processing” is simplified. Alternatively, the control unit 302 may perform the “left and right eye detection process” using both the captured image in which the backlight 62 is turned off and the captured image in which the backlight 62 is turned on. In this case, the control unit 302 may perform the “left and right eye detection process” using a captured image in which the region including the user's eyes is clearer.

  In the first to third embodiments, as an example of the captured image, an area including one eye of the user is captured as illustrated in FIGS. 11, 12, 14, 26, 27, and 28. However, the present invention is not limited to this. In the captured image, not only a region including one eye but also a region including a part of one other eye of the user may be captured in both eyes of the user. In this case, the control unit 302 extracts the image area of the eye that occupies the larger area in the captured image, and sets the extracted image area as the area including one eye of the user. Then, the control unit 302 may perform “left and right eye detection processing” based on the extracted region including one eye of the user. Alternatively, the control unit 302 extracts the image region of the eye whose eyeball image is closer to the center of the captured image in the captured image, and performs “left and right eye detection processing” based on the extracted image region. May be. Further, the control unit 302 extracts an eyebrow image, extracts an eye image in the vicinity of the extracted eyebrow image, and performs a “left and right eye detection process”. Thus, by extracting the image of the eye in the vicinity of the image of the eyebrows, in the HMD 1 according to the present invention, a part of the other eye of the user's eyes is added to the image of the region including the user's one eye Even if the image of the eyebrows corresponding to the eye or the image of the eyebrows corresponding to the eye is included, the image of the eye of the user and the image of the eyebrows corresponding to the eye can be appropriately extracted. As a result, the HMD 1 according to the present invention appropriately determines whether the eye observing the image displayed on the display unit 60 is the right eye or the left eye based on the region including one eye of the user. Can be recognized.

  In the first to third embodiments, FIG. 2 illustrates an example in which the display unit 60 and the finder opening 67 are rectangular. However, the display unit 60 and the finder opening 67 are square, oval, circular, or the like. It may be. Even if the display unit 60 and the viewfinder opening 67 have such a shape, a plurality of eyeball illumination lights are arranged in the vicinity of two opposing sides of the four sides of the viewfinder opening 67 as shown in FIG. May be arranged respectively.

In the first to third embodiments, an example in which two eyeball illumination lights 166a to 166d or 166e to 166h are provided on each of the upper and lower sides or the left and right sides of the finder opening 67 as shown in FIG. Although explained, it is not limited to this. For example, one each may be provided above and below or right and left of the finder opening 67, or three or more may be provided above and below or right and left of the finder opening 67.
Further, the positions where the eyeball illumination lights 166a to 166d are provided may not be above and below the finder opening 67 as shown in FIG. Any position that irradiates the user's eyeball may be used. For example, the eyeball illumination lights 166 a to 166 d may be provided above and below the viewfinder opening 67. In other words, when the user places the HMD 1 in the vicinity of the user's eyes, the eyeball illumination lights 166a to 166d may be arranged in the vicinity of the user's eyes. For this reason, the eyeball illumination lights 166a to 166d may be arranged in a range of one surface of the display unit 60 where the finder opening 67 is arranged, or a range inside the finder opening 67. Further, the vicinity of the user's eye where the display unit 60 is arranged is, for example, below, above, outside, or the like of the user's eye, such as positions 600a to 600f shown in FIG. As long as it is within a range where a peripheral image of the eye can be captured.

In the first to third embodiments, the example in which the detection value of the acceleration sensor 132 or the image captured by the image sensor 72 is used to determine whether the observation mode is the right eye observation mode or the left eye observation mode is described. It is not limited to this. For example, a type including a light receiving unit may be used for the eyeball illumination lights 166a to 166d, and the reflected light from each of the eyeball illumination lights 166a to 166d may be detected. For example, in the case of the right-eye observation form, as shown in FIG. 34, the eyeball illumination lights 166a and 166b are arranged in front of the user's eye rather than the eyeball illumination lights 166c and 166d. For this reason, the brightness of the reflected light from the eyeball illumination lights 166c and 166d is lower than the brightness of the reflected light from the eyeball illumination lights 166a and 166b.
As described above, the processing unit 123 may detect the reflected light from each of the eyeball illumination lights 166a to 166d and determine the observation form based on the detected result.

The image sensor 72 may capture a captured image at a predetermined time interval. In this case, the processing unit 123 determines whether the user has removed the head-mounted display from the head without turning the main switch 28 based on the image captured by the image sensor 72. Also good. If it is determined that the user has removed the head-mounted display from the head for a predetermined time or longer without turning off the main switch 28, the processing unit 123 stops imaging by the camera 64 and the imaging element 72. Also good. The processing unit 123 may turn off the eyeball illumination lights 166a to 166d. Further, the processing unit 123 may control each communication circuit or the like so as to enter a power saving state. Whether or not to perform these controls may be set by the user by operating the touch switch 34, for example.
Alternatively, the processing unit 123 may determine whether or not the head mounted display is removed from the head based on the detection value of the heart rate sensor 137, for example.
Further, when a detection switch (not shown) detects that the apparatus main body 21 is not attached to the headband 40, the processing unit 123 determines that the apparatus main body 21 is removed from the head, and the eyeball All of the illumination lights 166a to 166d may be turned off.
Thereby, the process part 123 can reduce the power consumption of a head mounted display.

  In the first to third embodiments, the monocular type has been described as an example of the head mounted display. However, the head mounted display may be a binocular type. In this case, based on the image captured by the image sensor 72, the processing unit 123 may be in a state where the display unit is disposed at positions 600c and 600d on the eye as illustrated in FIG. It may be determined whether or not the display unit is located at the positions 600a and 600b. The processing unit 123 may control lighting or extinguishing of the eyeball illumination lights 166 provided in the display units corresponding to the left and right eyes, based on the determined result.

  Further, when the head mounted display is a binocular type, the eyeball illumination light 166 and the image sensor 72 may be provided only on one display unit 60 of the left and right display units, or on both display units 60. It may be provided. The reason why the eyeball illumination light 166 may be provided only on one display unit 60 is that, when a person looks at an object, the eyeballs of both eyes generally face the same direction.

  In the first to third embodiments, the example in which the HMD 1 includes the functions illustrated in FIG. 21 has been described. However, the HMD 1 may not have all the functions. Depending on the application, for example, the temperature / humidity sensor 136 and the heart rate sensor 137 may not be provided. Conversely, the HMD 1 may include other functional units other than those shown in FIG. 22 as necessary.

  In the first to third embodiments, the example in which the display main body 20 is mounted on the headband 40 has been described, but the present invention is not limited to this. The user may use the display body 20 alone without depending on the headband 40 depending on the application.

  The technical scope of the present invention is not limited to the above embodiment. At least one of the requirements described in the above embodiments may be omitted. The requirements described in the above embodiments can be combined as appropriate.

  The HMD 1 described above has a computer system (for example, the processing unit 123) inside. The operation process of each functional unit is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing the program. Here, the computer system includes a CPU, various memories, an OS, and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it also includes those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Head mounted display, 20 ... Display main body, 26 ... Audio connector, 34 ... Touch switch, 60 ... Display part, 63 ... Display panel (display part), 64 ... Camera, 72 ... Imaging element (imaging part), 100 ... Earphone, 103 ... first speaker, 104 ... second speaker, 121 ... decoder, 123 ... processing unit, 131 ... WiFi communication circuit, 132 ... acceleration sensor, 133 ... geomagnetic sensor, 137 ... heart rate sensor, 138 ... 3G / LTE Communication circuit, 166a to 166h ... eyeball illumination light, 301 ... input detection unit, 302 ... control unit, 303 ... storage unit, 304 ... eyeball illumination unit, 305 ... acoustic control unit, 306 ... image switching unit, 307 ... image display unit 308 ... Transmission / reception unit 309 ... Behavior history registration unit 310 ... Observability determination unit

Claims (1)

A display for displaying information;
An observability determination unit that determines whether or not the display unit is observable by a user;
An informing unit for informing by sound;
Based on the result determined by the observability determination unit, when it is determined that the display unit is not observable by the user, a control unit that controls to notify from the notification unit;
An information input / output device comprising: