JP6701631B2 - Display device, display device control method, display system, and program - Google Patents

Description

  The present invention relates to a display device, a display device control method, a display system, and a program.

  Conventionally, a head-mounted display device is known. Among display devices of this type, there is one in which a user wearing the device can make settings relating to the operation of the device (for example, refer to Patent Document 1). Patent Literature 1 discloses a configuration in which a display device mounted on the head of a player who is a user can perform a more natural display by aligning the display positions of images.

JP, 2002-15919, A

As described above, by adjusting the head-mounted display device, it is possible to prevent deterioration in the feeling of use due to individual differences among users. However, it is complicated to make adjustments each time the display device is used. Further, for example, when a plurality of users share the display device, adjustments must be made each time each user wears the display device, which imposes a heavy operational burden.
The present invention has been made in view of the above circumstances, and an object of the present invention is to reduce the burden on the user when setting the display device according to the user.

In order to achieve the above object, the display device of the present invention is a display device including a head-mounted display unit, wherein the setting information regarding the display of the display unit is associated with identification information for identifying a user. And a storage unit for storing the identification information, an input unit for inputting the identification information, and a control unit for controlling display of the display unit based on setting information corresponding to the input identification information.
According to the present invention, since the setting relating to the display of the display device can be performed based on the setting information, the setting suitable for the user can be promptly performed.

Further, the present invention is, in the above display device, comprising an imaging section, wherein the input section inputs the identification information based on a captured image of the imaging section, and the control section stores the identification information stored in the storage section. Of the above, the identification information corresponding to the identification information input by the input unit is specified, and based on the setting information stored in association with the specified identification information, the display of the display unit is controlled, It is characterized by
According to the present invention, since the setting relating to the display of the display device can be performed using the captured image, the setting suitable for the user can be promptly performed.

The present invention is also characterized in that, in the display device, the identification information stored in the storage unit includes a part of the features included in the biometric information of the user.
According to the present invention, it is possible to identify a user by using biometric information as identification information, reduce the amount of information by including a part of the features included in the biometric information of the user, and It enables simple settings according to.

In the display device according to the present invention, the control unit may include some of the characteristics of the biometric information input as the identification information by the input unit and some of the characteristics of the biometric information stored in the storage unit. And setting information is specified by collating with.
According to the present invention, it is possible to make simple settings according to physical characteristics.

Further, in the display device according to the present invention, the identification information includes image information regarding an image extracted from a captured image of the imaging unit.
According to the present invention, an image extracted from a captured image can be used to make settings relating to display on the display device.

Further, in the display device according to the present invention, the control unit collates the image information included in the identification information input by the input unit with the image information stored in the storage unit, The setting information is specified.
According to the present invention, it is possible to easily perform authentication by using an image extracted from a captured image and set the display of the display device.

Further, in the display device according to the present invention, the identification information includes first identification information including a part of characteristics included in the biological information of the user, and second identification information including the image information. The storage unit is characterized in that the storage unit stores the first identification information, the second identification information, and the setting information in association with each other.
According to the present invention, it is possible to perform the setting related to the display of the display device by using the biometric information and the information related to the image extracted from the captured image. Further, by combining a plurality of pieces of information, it is possible to use more setting information properly.

In the display device according to the present invention, the control unit may be stored in the storage unit based on the first identification information and the second identification information included in the identification information input by the input unit. The setting information is specified.
According to the present invention, it is possible to use more settings properly by combining biometric information and information related to an image extracted from a captured image.

Further, in the display device according to the present invention, the control unit stores in the storage unit in association with a combination of the first identification information and the second identification information included in the identification information input by the input unit. It is characterized in that the stored setting information is specified.
According to the present invention, it is possible to use more settings properly by combining biometric information and information related to an image extracted from a captured image.

Further, in the display device according to the present invention, the control unit stores the memory based on one of the first identification information and the second identification information included in the identification information input by the input unit. Selecting a plurality of the setting information from the setting information stored in the unit, and among the selected setting information, the first identification information and the second identification information included in the identification information input by the input unit The setting information corresponding to the other of the above is specified.
According to the present invention, setting information can be specified in two stages using a plurality of pieces of information, and more settings can be used efficiently.

Further, in the display device according to the present invention, the image information is non-living body specific information that is information that can be extracted or detected from the captured image and that does not include information that can individually identify a living body. , Is characterized.
According to the present invention, it is possible to select a pre-stored setting and make a setting related to the display of the display device by using information that does not include information that can individually identify a living body.

Further, in the display device according to the present invention, the image information is extracted from outside image information in which information including a shape of a building is extracted from the captured image including a building, and the captured image in which an inanimate object for identification is captured. And the locus information relating to the locus of the object extracted from the plurality of captured images of the moving object.
According to the present invention, a captured image including a building, a captured image of a non-living object for identification, or a plurality of captured images of a moving object is used to select a prestored setting, Settings related to display on the display device can be performed.

Further, the present invention is the display device, wherein the display unit is a binocular display for displaying an image corresponding to the right eye and the left eye of the user, and either the right eye or the left eye of the user. It is possible to switch between and execute the single-eye display for displaying an image corresponding to, and the control unit displays the display unit based on the setting information corresponding to the identification information input by the input unit. Until the control is performed, the display unit is caused to perform a single-eye display, and when the display is controlled based on the setting information, the display unit is switched to the binocular display.
According to the present invention, it is possible to reduce the user's discomfort while the setting is not reflected on the display of the display unit.

Further, in the display device according to the present invention, the display unit displays an image so that the outside scene is transparent and can be visually recognized together with the outside scene, and the control unit includes at least one of a display position and a display size of the image. Is changed in a plurality of steps corresponding to the setting information.
According to the present invention, the display position and display size of an image can be set in stages according to the user.

Further, in the display device according to the present invention, the display unit transmits an outside scene, and an optical element that allows image light to enter the eye of the user so that the image can be visually recognized together with the outside scene, and a line of sight of the user. An object detection unit that detects an object, and a position detection unit that detects the position of the object with respect to the display area of the display unit, and the control unit is the position of the object detected by the position detection unit. Further, the display position of the image by the optical element is changed according to the positional relationship between the optical element and the position of the pupil of the user.
According to the present invention, the display position of an image can be set according to the user.

Further, in the display device according to the present invention, the setting information includes information about language setting, and the control unit corresponds to the setting information when displaying content including characters on the display unit. Characters of a language are displayed on the display unit.
According to the present invention, the language of characters displayed on the display unit can be set according to the user.

In the display device according to the present invention, a communication unit may be provided, and the control unit may associate the setting information and the identification information stored in the storage unit with each other and transmit the setting information and the identification information to an external device. The communication unit receives the setting information and the identification information, and stores the received setting information and the identification information in the storage unit in association with each other.
According to the present invention, the setting information and the identification information can be transmitted to or received from an external device. Thus, the setting information and the identification information can be easily acquired and stored, and the setting information and the identification information can be shared with an external device.

Further, in order to achieve the above-mentioned object, the display system of the present invention includes a plurality of display devices each including a head-mounted display unit, and the display device uses setting information regarding display of the display unit. A storage unit that stores the identification information for identifying a person in association with the identification information, an input unit that inputs the identification information, and a control unit that controls the display of the display unit based on the setting information corresponding to the input identification information. And a communication unit that communicates with another display device, wherein the control unit associates the setting information and the identification information stored in the storage unit with each other and transmits the communication information by the communication unit. A unit receives the setting information and the identification information, and stores the received setting information and the identification information in the storage unit in association with each other.
According to the present invention, since the setting relating to the display of the display device can be performed based on the setting information, the setting suitable for the user can be promptly performed. Moreover, since the setting information and the identification information are transmitted and received between the display devices, the setting information and the identification information can be shared between the plurality of display devices.

Further, in order to achieve the above object, a control method of a display device of the present invention is a control method of a display device including a head-mounted display unit, the step of inputting identification information, and the display unit A step of controlling the display of the display section based on the setting information corresponding to the input identification information, by referring to a storage section that stores the setting information relating to the display in association with the identification information for identifying the user; , Are provided.
According to the present invention, since the setting relating to the display of the display device can be performed based on the setting information, the setting suitable for the user can be promptly performed.

In order to achieve the above object, the program of the present invention is a program executable by a computer that controls a display device having a head-mounted display unit, and the computer is an input unit that inputs identification information. With reference to the storage unit that stores the setting information regarding the display of the display unit in association with the identification information for identifying the user, based on the setting information corresponding to the input identification information, the display unit of the display unit It functions as a control unit that controls the display.
According to the present invention, since the setting relating to the display of the display device can be performed based on the setting information, the setting suitable for the user can be promptly performed.

1 is a schematic configuration diagram of a communication system and a display system according to a first embodiment. Explanatory drawing which shows the external appearance structure of a head mounted display device. The top view which shows the structure of a light-guide plate. The functional block diagram of each part which comprises a head mounted display device. The schematic diagram which shows the structural example of a user information table. 9 is a flowchart showing the operation of the head-mounted display device in the information registration mode. An example of an image displayed by the head-mounted display device in the information registration mode is shown, (A) shows an example of a silhouette image, and (B) shows an example of adjustment using the silhouette image. 6 is a flowchart showing the operation of the head-mounted display device for registering user information. As an example of an image displayed by the head-mounted display device, an example in which an additional guide is displayed on a silhouette image is shown. The schematic diagram which shows the structural example of communication data. 9 is a flowchart showing the operation of the head-mounted display device according to the second embodiment. The figure which shows the example of the message which a head mounted display device outputs. 9 is a flowchart showing the operation of the head-mounted display device according to the third embodiment. The flowchart which shows operation|movement of the head mounted display apparatus of 4th Embodiment. The figure which shows the example of a display of the head mounted display apparatus of 4th Embodiment. The flowchart which shows operation|movement of the head mounted display apparatus of 5th Embodiment. The figure which shows the example of the user information table which the head mounted display device of 5th Embodiment memorize|stores. The figure which shows the example of the image which the head mounted display apparatus of 5th Embodiment recognizes. The flowchart which shows operation|movement of the head mounted display apparatus of 6th Embodiment. The figure which shows the example of the user information table which the head mounted display device of 6th Embodiment memorize|stores.

  FIG. 1 is a schematic configuration diagram showing a configuration of a communication system 1 as an embodiment to which the present invention is applied. The communication system 1 is a system in which a display system 2 including a plurality of head-mounted display devices 100 is connected to a server 5 via a communication network 4.

The head-mounted display device 100 is a display device that a user wears on the head as shown in FIG. 1, and is also called a head mounted display (HMD). The head-mounted display device 100 is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly visually recognize an outside scene.
In the following description, the virtual image visually recognized by the user by the head-mounted display device 100 is also referred to as “display image” for convenience. In addition, emitting the image light generated based on the image data is also referred to as “displaying an image”.
In FIG. 1, a plurality of head-mounted display devices 100 included in the display system 2 are referred to as head-mounted display devices 100A and 100B, respectively. Below, when it is not necessary to distinguish between the head-mounted display devices 100A and 100B, the head-mounted display device 100 is referred to.

  The communication network 4 is realized by various communication lines such as a public line network, a dedicated line, a wireless communication line including a mobile phone line, and a backbone communication line of these lines, or a combination thereof, and its specific configuration is not limited. .. The communication network 4 may be a wide area communication line network capable of connecting remote locations, or may be a LAN (Local Area Network) laid in a specific facility or building. Further, the communication network 4 may include network devices such as a server device, a gateway device, and a router device that connect the various communication lines. Further, the communication network 4 may be composed of a plurality of communication lines.

  The display system 2 is configured using a LAN or the like installed in a building or the like. The display system 2 includes a wireless access point 3 that performs wireless communication, and a plurality of head-mounted display devices 100A and 100B. The wireless access point 3 is a communication device such as an access point or a router, and performs data communication between the head-mounted display device 100A and the head-mounted display device 100B, and the head-mounted display devices 100A and 100B. And data communication with the communication network 4.

The head-mounted display device 100A can perform data communication with another head-mounted display device 100B via the wireless access point 3. In addition, the head-mounted display devices 100A and 100B perform data communication with the server 5 via the wireless access point 3. Note that each head-mounted display device 100A may directly perform wireless communication with another head-mounted display device 100B in, for example, an ad hoc mode. Further, the head-mounted display devices 100A and 100B may be connected by a wired communication line.
That is, the configuration of the display system 2 is not particularly limited as long as the head-mounted display device 100A can communicate with the head-mounted display device 100B. Further, the communication system 1 may have a configuration capable of communicating with the server 5 and the head-mounted display devices 100A and 100B included in the display system 2.

FIG. 2 is an explanatory diagram showing an external configuration of the head-mounted display device 100. Since the head-mounted display devices 100A and 100B described above have a common configuration, the head-mounted display device 100 will be described below.
The head-mounted display device 100 includes an image display unit 20 that allows the user to visually recognize a virtual image while being mounted on the head of the user, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100.

  The image display unit 20 is a mounting body mounted on the head of the user, and has a spectacle shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a first display unit. The camera 61, the second camera 62, and the microphone 63 are provided. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrows of the user when the user wears the image display unit 20.

  The right holding portion 21 is provided by extending from the end portion ER which is the other end of the right optical image display portion 26 to a position corresponding to the user's temporal region when the user wears the image display portion 20. It is a member. Similarly, the left holding unit 23 extends from the end EL, which is the other end of the left optical image display unit 28, to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided by. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple (vine) of glasses.

  The right display drive unit 22 and the left display drive unit 24 are arranged on the side facing the head of the user when the user wears the image display unit 20. The right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”, and the right optical image display unit 26 and the left optical image display unit 28 are collectively referred to as “optical image display unit”. Also called.

The display drive units 22 and 24 include liquid crystal displays 241, 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251, 252, and the like (see FIG. 4). The details of the configuration of the display drive units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 4) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display drive units 22 and 24 to the eyes of the user. The light control plate 20A is a thin plate-shaped optical element, and is arranged so as to cover the front side of the image display unit 20, which is the side opposite to the eye side of the user. As the light control plate 20A, various ones such as one having almost zero light transmittance, one nearly transparent, one that attenuates the amount of light and transmits light, one that attenuates or reflects light of a specific wavelength, etc. are used. You can By appropriately selecting the optical characteristics (light transmittance or the like) of the light control plate 20A, the amount of external light incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having a light transmissive property that allows a user wearing the head-mounted display device 100 to visually recognize an outside scene is used. The dimming plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage and stains on the right light guide plate 261 and the left light guide plate 262.
The light control plate 20A may be attachable to and detachable from the right optical image display unit 26 and the left optical image display unit 28, and may be attachable by exchanging a plurality of types of light control plates 20A, or may be omitted. ..

The first camera 61 as an input unit is arranged at the end ER which is the other end of the right optical image display unit 26. The first camera 61 captures an outside scene, which is an outside scene in the direction opposite to the side of the user's eyes. The first camera 61 may be a monocular camera or a stereo camera.
The image capturing direction of the first camera 61, that is, the angle of view is the front side direction of the head-mounted display device 100, in other words, at least a part of the external view in the visual direction of the user with the head-mounted display device 100 mounted. Is the direction to image. In addition, although the width of the angle of view of the first camera 61 can be set as appropriate, the imaging range of the first camera 61 is set to the outside world that the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. It is preferably within the range. Furthermore, it is more preferable that the imaging range of the first camera 61 is set so that the entire field of view of the user through the light control plate 20A can be imaged.

  The second camera 62 is arranged at the boundary between the right optical image display unit 26 and the left optical image display unit 28, that is, at the intermediate position between the left and right eyes of the user. The second camera 62 faces the inside of the image display unit 20 and images the side of the user's eyes. The second camera 62 may be a monocular camera or a stereo camera.

FIG. 3 is a plan view showing the configuration of the left light guide plate 262. The configuration in which the virtual image is visually recognized by the left and right eyes of the user is symmetrical. Therefore, only the left light guide plate 262 will be described here, and the right light guide plate 261 has a configuration symmetrical to the left light guide plate 262 in FIG.
The left display drive unit 24 of the image display unit 20 includes a left backlight 222 having a light source such as an LED and a diffusion plate, and a transmissive left arranged on the optical path of light emitted from the diffusion plate of the left backlight 222. And an LCD 242. The left display drive unit 24 of the image display unit 20 also includes a left projection optical system 252 including a lens group that guides the image light L transmitted through the left LCD 242.
The left projection optical system 252 is configured by a collimator lens that converts the image light L emitted from the left LCD 242 into a parallel light flux. The image light L converted into a parallel light flux by the collimator lens is incident on the left light guide plate 262. The left light guide plate 262 is a prism in which a plurality of reflection surfaces that reflect the image light L are formed, and the image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. Then, the image light L reflected by the half mirror 262A located in front of the left eye LE is emitted from the left optical image display unit 28 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE. , Allow the user to view the image.
The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”. This light guide unit can use any method as long as a virtual image is formed in front of the user's eyes by using image light. For example, a diffraction grating or a semi-transmissive reflective film may be used. good.

The user's left eye LE receives not only the image light L reflected by the half mirror 262A but also external light OL transmitted through the light control plate 20A. That is, the head-mounted display device 100 causes the image light L of the internally processed image and the external light OL to overlap and enter the user's eye, and the user can adjust the head-mounted display device 100. An external view is visible through the light plate 20A, and an image formed by the image light L is visually recognized by being superimposed on the external view. That is, it can be said that the head-mounted display device 100 is a see-through display device.
Although not shown, the right light guide plate 261 includes a half mirror 261A. The configuration of the half mirror 261A is symmetrical with the half mirror 262A of FIG.

  The image capturing direction of the second camera 62 shown in FIG. 2, that is, the angle of view, is the back side direction of the head-mounted display device 100, in other words, the face of the user with the head-mounted display device 100 mounted is imaged. Direction. The second camera 62 can capture the position of the eyes of the user, the distance between the eyes, and an image of the face of the user (face image). Further, although the width of the angle of view of the second camera 62 can be set as appropriate, it must be within a range in which the face of the user wearing the head-mounted display device 100 and the half mirrors 261A and 262A can be imaged. Is preferred. The angle of view of the second camera 62 does not have to be an angle of view capable of imaging the half mirrors 261A and 262A, but may be an angle of view capable of imaging an optical element such as the light control plate 20A. .. The control unit 140, which will be described later, calculates the angles of the half mirrors 261A and 262A based on the captured angle of the light control plate 20A.

  Returning to FIG. 2, the head-mounted display device 100 has a connection unit 40 for connecting the image display unit 20 to the control device 10. The connecting portion 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are cords in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the front end portion AP of the right holding unit 21 in the extending direction, and is connected to the right display drive unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the front end portion AP of the left holding unit 23 in the extending direction, and is connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48 and the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided near the earphone plug 30. The earphone plug 30 to the microphone 63 are integrated into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  As shown in FIG. 2, for example, the microphone 63 is arranged such that the sound collecting unit of the microphone 63 faces the direction of the user's line of sight, collects sound, and outputs the sound signal to the sound processing unit 187 (FIG. 4). Output. The microphone 63 may be, for example, a monaural microphone or a stereo microphone, a directional microphone, or an omnidirectional microphone.

  It is also possible to combine the right cord 42 and the left cord 44 into one cord. Specifically, the conducting wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, covered with resin together with the conducting wire inside the left cord 44, and collected into one cord. Good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. A connector (not shown) that fits with each other is provided at each of the end of the body cord 48 on the opposite side of the connecting member 46 and the control device 10. By mating/unmating the connector of the body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 controls the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a brightness switching key 15, a direction key 16, a menu key 17, and a power switch 18. The control device 10 also includes a track pad 14 that is touch-operated by a user with a finger.

  The enter key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operating state of the head-mounted display device 100 by its light emitting state. The operating state of the head-mounted display device 100 includes, for example, turning on/off a power source. As the lighting unit 12, for example, an LED (Light Emitting Diode) is used. The display switching key 13 detects a pressing operation and outputs, for example, a signal for switching the display mode of the content moving image between 3D and 2D.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal according to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The brightness switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the brightness of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up, down, left, and right directions and outputs a signal according to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting the slide operation of the switch.

FIG. 4 is a functional block diagram of each part constituting the head-mounted display device 100.
As shown in FIG. 4, the head-mounted display device 100 is connected to the external device OA via the interface 125. The interface 125 is an interface for connecting the control device 10 to various external devices OA that are content supply sources. As the interface 125, for example, a USB interface, a micro USB interface, an interface for a memory card, or the like compatible with wired connection can be used.
The external device OA is used as an image supply device that supplies an image to the head-mounted display device 100, and for example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like is used.

The control device 10 of the head-mounted display device 100 includes a control unit 140, an operation unit 111, an input information acquisition unit 110, a storage unit 120, an interface 125, a transmission unit (Tx) 51, and a transmission unit (Tx). ) 52 and.
The operation unit 111 detects an operation performed by the user. The operation unit 111 includes the determination key 11, the display switching key 13, the track pad 14, the brightness switching key 15, the direction key 16, the menu key 17, and the power switch 18 shown in FIG.

  The input information acquisition unit 110 acquires a signal according to an operation input by the user. The signals corresponding to the operation inputs include, for example, operation inputs to the track pad 14, the direction keys 16, and the power switch 18. Further, the control device 10 includes a power supply unit (not shown), and supplies power to each unit of the control device 10 and the image display unit 20.

  The storage unit 120 is a non-volatile storage device and stores various computer programs. Further, the storage unit 120 may store image data to be displayed on the image display unit 20 of the head-mounted display device 100.

  The storage unit 120 also stores a user information table for registering user information. The user information table is a table in which biometric information (identification information) and setting information are registered in association with user IDs that identify each user. The biometric information is information on the biometric body that is unique to the user and that can identify the user. In addition, the setting information is information regarding the setting regarding the display of the image display unit 20 and the setting regarding the operation of the head-mounted display device 100. Details of the user information table will be described later.

  For example, the distance between the user's eyes and the half mirrors 261A and 262A has individual differences depending on the height of the user's nose, the size of the eye socket or eyeball, and the like. In addition, some users have different left and right ear heights, and when the head-mounted display device 100 is worn, the image display unit 20 may not be horizontal. As described above, due to individual differences in the anatomical structure of the user's head, the relative positional relationship between each part of the image display unit 20 and the user's eyes when the image display unit 20 is mounted is affected. .. Therefore, if the distance between the user's eyes and the optical element, the angle of the optical element when the head-mounted display device 100 is worn, or the like is adjusted so as to fit the user's body, individual differences are absorbed. Therefore, a good usability can be realized.

  Therefore, the head-mounted display device 100 of the present embodiment performs calibration under the control of the control unit 140 when the user first wears the head-mounted display device 100. Then, the head-mounted display device 100 measures the distance between the user's eyes and the optical element, the angle of the optical element when the head-mounted display device 100 is mounted, and the like. The control unit 140 registers the measured setting information of the user in the user information table. The control unit 140 also registers biometric information that can identify the user in the user information table in association with the setting information of the user. Then, when the user actually uses the head-mounted display device 100, the control unit 140 identifies the user based on the biometric information detected from the image captured by the first camera 61. The head-mounted display device 100 adjusts the display position and size of the image displayed by the image display unit 20 according to the specified setting information of the user.

The biometric information used by the head-mounted display device 100 includes a fingerprint, a palm print, a palm line, a blood vessel pattern of the retina, a face image, a whole body image, a voice, and the like. These are information that can be detected by the first camera 61, the second camera 62, the microphone 63, and the like, which are input sections of the head-mounted display device 100. The palm line is a figure composed of linear wrinkles (recesses) appearing on the palm or a combination thereof, and is a so-called palm line. It is also possible to use the absolute length of the user's hand, the length of the finger, the ratio of the length of the finger, and the like.
In the present embodiment, an example is shown in which the first camera 61 is used as an input unit and biometric information that can be processed as an image is processed. The head-mounted display device 100 includes one or a plurality of biometric feature information based on biometric information itself such as a fingerprint, a palm print, a palm line, a blood vessel pattern of a retina, a captured image of a face, and a captured image of a whole body. Is extracted and the extracted characteristic information is used. For example, when the biometric information is a palm line, the feature information includes a plurality of pieces of information representing the features of the biometric information such as the position coordinates of the starting point and the ending point of the palm line, the length ratio, and the curvature. In the present embodiment, an example is shown in which the absolute length of the hand, the length ratio of the finger, the palm line, and the palm line are used as biometric information.

The biometric information used by the head-mounted display device 100 is registered in the user information table stored in the storage unit 120.
FIG. 5 is a schematic diagram showing a specific example of the user information table stored in the storage unit 120 in this embodiment. FIG. 5A shows a user information table 120A, and FIG. 5B shows a palm line information table 120B that complements the information of the user information table 120A. The user information table 120A plus the palm line information table 120B corresponds to the user information table of this embodiment.

The user information table 120A and the palm line information table 120B include absolute length information of the hand, length ratio information of the finger, angle information of the surface of the palm, and palm line information as biometric information. The biometric information is detected by the control unit 140 based on the captured image of the user's hand captured by the first camera 61. The biometric information registered in the user information table 120A and the palm line information table 120B is particularly referred to as registered biometric information.
The absolute hand length information includes the length of the user's hand in the longitudinal direction and the length in the lateral direction (unit: cm). The length in the longitudinal direction of the hand is, for example, the length from the wrist to the middle finger, and the length in the lateral direction is, for example, the width of the palm. The finger length ratio is the length of the finger based on the length in the longitudinal direction of the hand. As the finger length ratio, the length ratio of each finger of the user may be registered, or only the length ratio of the index finger may be registered, for example.

  The angle information of the palm surface is information indicating the angle of the palm when the user holds the palm over the first camera 61. This angle is an angle based on the image pickup surface of the first camera 61, or the light guide plates 261 and 262 and the light control plate 20A. For example, an angle of 0° when parallel to the image pickup surface, an angle of the half mirrors 261A and 262A of the light guide plates 261 and 262 with respect to the optical axis of the image light L, or 0° when parallel to the light control plate 20A is set. It is the angle to do. For example, when the first camera 61 is a stereo camera or when the head-mounted display device 100 includes a distance sensor, the angle of the palm surface is measured by the first camera 61 or the distance sensor. This can be achieved by measuring the distances to multiple measurement points.

The palm line information is information about the length of the palm line and the shape of the figure formed by the palm line. Since the palm line information has a large amount of information, it is registered in the palm line information table 120B which is another table. The user information table 120A and the palm line information table 120B are linked by the palm line information ID. The head-mounted display device 100 can obtain a plurality of palm line information of the palm of the user based on the image captured by the first camera 61. In the palm line information table 120B of FIG. 5B, a palm line ID is attached to each main palm line, information regarding the position and length of each palm line is stored in association with each other, and the user who has detected this palm line User IDs of are registered in association with each other.
The starting and ending coordinates of the palm line are the starting and ending coordinates of the palm line with respect to the origin. The origin can be set at any position. For example, the upper left corner of the photographed image of the user's hand may be the origin, or the upper left corner of the region of the user's hand detected from the photographed image with a predetermined size may be the origin. Further, the length ratio of the palm line is the length of the palm line based on the length in the longitudinal direction of the hand, and the curvature is, for example, the maximum value of the curvature in one palm line.
The biometric information used as the registered biometric information is not limited to that shown in FIG. 5A, and biometric information that can be easily measured, such as the inter-eye distance, the color of the pupil, and the captured image of the face, is used as the simple biometric information. It may be registered in the person information table 120A.

  The setting information of the user information table 120A includes an inter-eye distance, a convergence angle, a relative position between the eye and an optical element, an angle of the optical element, a dominant eye, color information, brightness, and a language used. And display position adjustment information. The inter-eye distance may be the distance between the inner eyes of the user, that is, the distance between the inner or inner corners of the eyes, or the distance between the centers of the pupils.

  The relative position between the eye and the optical element is, for example, a value that represents the deviation between the center of the pupil and the center of the half mirrors 261A and 262A as the optical element. This value may be the average value of the left and right sides, or may be the value of the center position of the left and right eyes. That is, it is the average value of the deviation between the center of the pupil of the right eye of the user and the center of the right half mirror 261A, and the deviation of the center of the pupil of the left eye of the user and the center of the left half mirror 262A. May be. Further, it may be a value obtained by calculating a shift between the center positions of the pupils of the left and right eyes of the user and the center positions of the center of the half mirror 261A and the center of the half mirror 262A.

  The control unit 140 calculates the deviation between the center of the pupil and the centers of the half mirrors 261A and 262A based on the image captured by the second camera 62, and registers the calculation result in the user information table. In this process, the center of the pupil is represented by the X and Y coordinate values of the center position of the pupil in the captured image. The X coordinate indicates the horizontal direction of the captured image, and the Y coordinate indicates the vertical direction of the captured image. Further, the centers of the half mirrors 261A and 262A are represented by X and Y coordinate values of the center positions of the half mirrors 261A and 262A in the captured image. The control unit 140 calculates how much the center positions of the half mirrors 261A and 262A deviate in the X coordinate direction and the Y coordinate direction with reference to the pupil center position, for example.

  The angle of the optical element is calculated based on the measurement value of the 9-axis sensor 66. The angle of the optical element is represented by the rotation angle (θr, θp, θy) of the half mirror 262A when the image display unit 20 is mounted on the head of the user. The rotation angles (θr, θp, θy) are, for example, when the center of the half mirror 262A is the origin and the roll axis, the pitch axis, and the yaw axis that are orthogonal to each other at the origin are defined, the optical axis of the half mirror 262A. The angle formed by the direction and the roll axis is represented by the roll angle θr. Further, the angle formed by the optical axis direction of the half mirror 262A and the pitch axis is represented by the pitch angle θp, and the angle formed by the optical axis direction of the half mirror 262A and the yaw axis is represented by the yaw angle θy.

  The brightness is information indicating the brightness of the image displayed in the display area of the image display unit 20. The color information is information indicating whether the user has a color vision abnormality. The used language is language information such as Japanese and English used by the user.

  The interface 125 is an interface for connecting the control device 10 to various external devices OA that are content supply sources. As the interface 125, for example, a USB interface, a micro USB interface, an interface for a memory card, or the like that supports wired connection can be used.

  Further, the control unit 140 is connected with the triaxial sensor 113, the GPS 115, the communication unit 117, and the voice recognition unit 114. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and current time obtained based on the GPS signal to the control unit 140. In addition, the GPS 115 may have a function of acquiring the current time based on the information included in the GPS signal and correcting the time measured by the control unit 140 of the control device 10.

The communication unit 117 executes wireless data communication in accordance with standards such as wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), and Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the image display unit 20. On the other hand, when the external device OA is wire-connected to the interface 125, the control unit 140 acquires the content data from the interface 125 and performs control for displaying an image on the image display unit 20. Therefore, the communication unit 117 and the interface 125 are generically referred to as a data acquisition unit DA hereinafter.
The data acquisition unit DA acquires content data from the external device OA. The data acquisition unit DA acquires data of an image displayed by the head-mounted display device 100 (hereinafter, referred to as “display image data”) from the external device OA.

  The voice recognition unit 114 extracts a feature from the digital voice data collected by the microphone 63 and converted into digital data by the voice processing unit 187, which will be described later, and models it. The voice recognition unit 114 recognizes the voices of a plurality of persons separately by extracting the features of the voice and modeling them, and performs speaker recognition for identifying the person who is speaking for each voice, or converting the voice into a text. Perform text conversion. Further, the voice recognition unit 114 may be configured to be able to identify the language type of voice data in the voice recognition process.

  The control unit 140 reads out and executes the computer program stored in the storage unit 120 to thereby execute the operating system (OS) 150, the image processing unit 160, the display control unit 170, the biometric information detection unit 181, and the setting information detection unit. Function as 182. The control unit 140 also functions as the setting unit 183, the target detection unit 184, the position detection unit 185, the information display control unit 186, and the voice processing unit 187.

The image processing unit 160 acquires the image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. The image processing unit 160 also generates a clock signal PCLK by using a PLL (Phase Locked Loop) circuit or the like (not shown) in accordance with the cycle of the separated vertical synchronization signal VSync or horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the sync signal has been separated into a digital image signal using an A/D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 for each frame as image data (Data in the figure) of the target image. This image data is, for example, RGB data.
The image processing unit 160 may perform image processing such as resolution conversion processing, various color tone correction processing such as adjustment of brightness and saturation, and keystone correction processing on the image data as necessary. Good.

  The image processing unit 160 transmits the generated clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. .. The image data Data transmitted via the transmission unit 51 is also referred to as “right-eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left-eye image data”. The transmission units 51 and 52 function as transceivers for serial transmission between the control device 10 and the image display unit 20.

The display controller 170 generates a control signal for controlling the right display driver 22 and the left display driver 24. Specifically, the display control unit 170 individually controls the driving ON/OFF of the right LCD 241 by the right LCD control unit 211 and the driving ON/OFF of the right backlight 221 by the right backlight control unit 201 according to the control signal. .. Further, the display control unit 170 individually controls ON/OFF driving of the left LCD 242 by the left LCD control unit 212 and ON/OFF driving of the left backlight 222 by the left backlight control unit 202. Accordingly, the display control unit 170 controls the generation and emission of image light by the right display drive unit 22 and the left display drive unit 24, respectively. For example, the display control unit 170 controls the right display driving unit 22 and the left display driving unit 24 by a control signal to cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, or Only generate image light. In addition, the display control unit 170 controls the right display driving unit 22 and the left display driving unit 24 by the control signal so that both the right display driving unit 22 and the left display driving unit 24 do not generate image light. You can also
Further, the display control unit 170 transmits the control signals to the right LCD control unit 211 and the left LCD control unit 212, respectively, via the transmission units 51 and 52. In addition, the display control unit 170 transmits respective control signals to the right backlight control unit 201 and the left backlight control unit 202.

The biometric information detection unit 181 as an input unit has an information registration mode and a user identification mode as operation modes. The information registration mode is a mode in which biometric information or setting information of the user is detected and the detected biometric information or setting information is registered in the user information table. The user identification mode is a mode in which the biometric information of the user is detected and it is determined whether or not the biometric information that matches the detected biometric information is registered in the user information table.
In the information registration mode, the biometric information detection unit 181 extracts the biometric information of the user from the image captured by the first camera 61. In the information registration mode, the setting information detection unit 182 calculates the setting information based on the captured image of the second camera 62 and the measurement value of the 9-axis sensor 66. The biometric information registered in the user information table in the information registration mode is called registered biometric information.

  FIG. 6 is a flowchart showing the operation of the head-mounted display device 100 in the information registration mode, and particularly shows the processing procedure of the biometric information detection unit 181 and the setting information detection unit 182. Further, FIG. 7 shows an example of an image displayed by the head-mounted display device 100 in the information registration mode, FIG. 7(A) shows an example of the silhouette image S, and FIG. 7(B) uses the silhouette image S. An example of the adjustment that was performed is shown below.

In the information registration mode, the biometric information detection unit 181 outputs a hand-shaped image (hereinafter, referred to as a silhouette image S) to the display control unit 170 and causes the image display unit 20 to display the image (step S1). FIG. 7A shows the silhouette image S displayed by the image display unit 20. The image display unit 20 displays the visual field VA viewed by the user, the display region T in which the image display unit 20 displays an image, and the silhouette image S displayed in the display region T. The display area T of the image display unit 20 is a range in which the images displayed by the right optical image display unit 26 and the left optical image display unit 28 are visible to the user. The display area T is the maximum range in which the image display unit 20 can visually recognize the image, and the image display unit 20 displays the image in all or part of the display area T.
The user adjusts the position of the hand so that his or her hand matches the silhouette image while visually recognizing the silhouette image S displayed on the image display unit 20. FIG. 7B shows how the position of the user's hand R is adjusted so as to match the silhouette image S displayed in the display area T. After adjusting the position of the hand R, the user operates the operation unit 111 with the other hand to instruct to capture an image of the hand R. When the operation of the operation unit 111 is input, the biological information detection unit 181 causes the first camera 61 to capture an image of the hand R of the user (step S2). The captured image of the first camera 61 is input to the biological information detection unit 181.

The biometric information detection unit 181 performs image processing on the input captured image to extract biometric information (step S3). The biometric information detection unit 181 first converts the captured image into grayscale, and performs edge detection on the grayscaled captured image. The biological information detection unit 181 compares the detected edge shape with the contour shape of the silhouette image S to detect a region in which the user's hand R is shown (hereinafter, a hand region). Since the silhouette image S as a guide is displayed at a predetermined position in the display area T, the position of the hand R shown in the captured image of the first camera 61 is almost constant. Therefore, the biometric information detection unit 181 does not need to perform edge detection on the entire captured image, but may perform edge detection on a preset area.
The biometric information detection unit 181 extracts the biometric information of the user from the captured image of the detected hand region of the user (step S3). The biometric information detection unit 181 detects the absolute length information of the hand, the length ratio information of the finger, and the palm line information as biometric information from the captured image of the hand region.

  When the biometric information detection unit 181 detects biometric information, the biometric information detection unit 181 generates a user ID for identifying the user, and registers the user ID in the user information table in association with the generated user ID. Further, although a plurality of palm line information can be detected from the captured image of the hand, the biometric information detection unit 181 does not have to register the palm line information of all detectable palm lines in the user information table. Good. The biometric information detection unit 181 selects (thinning out) the palm lines from the palm lines that can be detected from the captured image, and registers the palm lines in the user information table. It should be noted that not only the palm line information but also a plurality of pieces of biometric information can be obtained by thinning out the number of registrations registered in the user information table.

  The setting information detection unit 182 also calculates the setting information of the user in the information registration mode (step S4). The setting information detection unit 182 causes the second camera 62 to perform imaging. The second camera 62 images the face of the user and the half mirror 262A (see FIG. 2). The captured image of the second camera 62 is input to the setting information detection unit 182. The setting information detection unit 182 calculates setting information based on the image captured by the second camera 62 (step S4). The setting information detection unit 182 calculates the distance between the eyes of the user, the vergence angle, and the relative position of the pupil and the optical element as the setting information. The setting information detection unit 182 registers the calculated setting information in the user information table in association with the biometric information of the corresponding user (step S5).

  Further, the setting information detection unit 182 calculates the angle of the optical element when the user wears the head-mounted display device 100 based on the measurement value of the 9-axis sensor 66 as the setting information (step S4). .. The angle of the optical element is represented by the rotation angle (θr, θp, θy) of the half mirror 262A when the image display unit 20 is mounted on the head of the user. The setting information detection unit 182 registers the calculated setting information in the user information table in association with the biometric information of the corresponding user (step S5).

In addition to this, the setting information detection unit 182 also sets the setting information such as the dominant eye, brightness setting, color information, and language used, which is input by the user by operating the operation unit 111, to the biometric information of the corresponding user. Is registered in the user information table in association with.
For example, when adjusting the brightness of the image displayed by the image display unit 20, the setting information detection unit 182 controls the display control unit 170 to display a test image for adjusting the brightness on the image display unit 20. Display it. The user operates the brightness switching key 15 of the operation unit 111 while visually recognizing the image displayed by the image display unit 20, and inputs an operation of increasing or decreasing the brightness of the test image. The setting information detection unit 182 controls the display control unit 170 according to the operation input from the operation unit 111, and adjusts the brightness of the test image displayed by the image display unit 20. In addition, when the setting information detection unit 182 receives the operation of the enter key 11, the brightness of the test image displayed by the image display unit 20, that is, the brightness information of the right backlight 221 and the left backlight 222 is stored in the storage unit 120. Remember.
Further, the setting information detection unit 182 controls the display control unit 170 to display the display data for prompting the input of the dominant eye, the color information, and the language used on the image display unit 20, and accepts the operation of the operation unit 111. For example, the setting information detection unit 182 controls the display control unit 170 to cause the image display unit 20 to display a text including a message such as "Is your dominant eye your right eye?" / Or display the image. The setting information detection unit 182 determines the dominant eye of the user and the language used based on the operation received by the operation unit 111, and registers it in the user information table as the setting information of the corresponding user ID.

  Further, the biometric information detection unit 181 detects the biometric information of the user in the user identification mode by the same procedure as in the biometric information registration mode, and passes the detected biometric information to the setting unit 183. The setting unit 183 compares the biometric information passed from the biometric information detection unit 181 with the registered biometric information registered in the user information table, and registers the biometric information that matches the biometric information detected by the biometric information detection unit 181. It is determined whether the information is registered in the user information table.

When the setting unit 183 is in the user identification mode, biometric information is input from the biometric information detection unit 181.
The setting unit 183 collates the input biometric information (hereinafter, referred to as input biometric information) with the registered biometric information registered in the user information table, and the registered biometric information that matches the input biometric information is the user information. Collation processing is performed to determine whether or not it is registered in the table.
When performing the matching process, the setting unit 183 performs a thinning-out process of thinning out biometric information used in the matching process. The setting unit 183 performs the same thinning process on both the input biometric information and the registered biometric information.

  For example, in the user information table 120A and palm line information table 120B shown in FIGS. 5A and 5B, the absolute length of the hand, the length ratio of the finger, and the angle of the palm surface are registered biometric information. The palm line information is registered. The setting unit 183 selects one or more pieces of biometric information to be used for the matching process from these registered biometric information. The setting unit 183 also selects the same biometric information as the biometric information selected from the registered biometric information for the input biometric information. Then, the setting unit 183 performs matching processing between the selected input biometric information and the selected registered biometric information. The setting unit 183 selects, for example, the palm line information and the finger length ratio from the registered biometric information registered in the user information table. Similarly, the setting unit 183 selects the palm line information and the finger length ratio from the input biometric information. The setting unit 183 compares the palm line information as the selected registered biometric information with the palm line information as the selected input biometric information to perform matching. Similarly, the setting unit 183 performs comparison by comparing the finger length ratio as the selected registered biometric information with the finger length ratio as the selected input biometric information.

The setting unit 183 may reduce the load of the matching process by thinning out the characteristic information used in the matching process instead of selecting the registered biometric information used in the matching process. For example, in the user information table shown in FIG. 4, a plurality of palm line information of the same user is registered as palm line information. The setting unit 183 selects one or more pieces of palm line information to be used for matching from the plurality of palm line information registered in the user information table. The setting unit 183 selects the palm line information whose coordinates are closest to the starting point coordinates and the ending point coordinates of the selected palm line information from the input biometric information, and collates these palm line information.
Further, the setting unit 183 may reduce the load of the matching process by thinning out the types of feature information used for matching.
For example, in the user information table shown in FIG. 5A, the starting point coordinates and the ending point coordinates, the length ratio, and the curvature are registered as palm line information. The setting unit 183 selects one or a plurality of types of palm line information used for collation from the plurality of palm line information registered in the user information table. For example, the setting unit 183 selects the length ratio of the palm line from the palm line information as the characteristic information used for the matching. When the palm line information is selected from the input biometric information, the setting unit 183 takes out the palm line information whose coordinates are closest to the starting point coordinates and the ending point coordinates of the selected palm line information from the user information table. The setting unit 183 compares the extracted length ratio of the palm line information with the length ratio of the palm line information selected from the input biometric information to perform matching.

  Note that the thinning-out process performed by the setting unit 183 is one of thinning-out of biometric information used in the matching process, thinning-out of characteristic information used in the matching process, and thinning-out of the type of feature information used in the matching process. May be performed by selecting. Further, the thinning-out process performed by the setting unit 183 may be a process in which a plurality of thinning-outs described above are combined and collation is performed.

  When the setting unit 183 determines that the input biometric information matches the registered biometric information registered in the user information table by matching the input biometric information with the registered biometric information, the setting information corresponding to the matched biometric information is set by the user. Read from the information table. The setting unit 183 outputs the brightness setting registered in the setting information to the display control unit 170. The display control unit 170 adjusts the light amounts of the right backlight 221 and the left backlight 222 according to the brightness setting input from the setting unit 183 so that the light amounts correspond to the brightness setting.

  The setting unit 183 also passes the information on the dominant eye registered as the setting information to the display control unit 170. For example, when the operation unit 111 is operated and an instruction to switch to a two-dimensional image is given when the image display unit 20 is displaying a three-dimensional (3D) image, the display control unit 170 displays the dominant eye information. The image on the side of the dominant eye is displayed. For example, when “left eye” is registered as the dominant eye of the user in the setting information, the display control unit 170 causes the image display unit 20 to display the image for the left eye. Thereby, when switching the display state of the three-dimensional image to the two-dimensional image, it is possible to reduce a sense of discomfort associated with the switching of the display mode.

  The setting unit 183 also passes the color information registered as the setting information and the language used to the information display control unit 186. The information display control unit 186 processes display data (described later) and adds new display data based on the color information passed from the setting unit 183 and the language used.

  Further, the setting unit 183 displays the display position in the display area of the image display unit 20 based on at least one of the inter-eye distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element, or A correction coefficient for correcting the display size is calculated. The setting unit 183 passes the calculated correction coefficient to the position detection unit 185.

Further, the setting information detection unit 182 determines the division to which the user belongs based on the setting information measured by the user wearing the head-mounted display device 100, and sets the information indicating the determined division to the user information table. You can also register at. The storage unit 120 stores, for each section, information in which the range of the interocular distance, the range of the relative distance between the eye and the optical element, the range of the angle of convergence, and the range of the angle of the optical element are set for each section. Remembered. When the biometric information detection unit 181 measures the interocular distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element, the setting information detection unit 182 is based on each measured information. Then, the category to which the corresponding user belongs is determined.
Further, the storage unit 120 stores the correction coefficient corresponding to each section. Further, when the setting unit 183 reads out the information indicating the division from the registered biometric information corresponding to the input biometric information, the setting unit 183 reads out the correction coefficient corresponding to the division from the storage unit 120 based on the read information of the division. The setting unit 183 outputs the read correction coefficient to the position detection unit 185. As a result, at least one of the display position and the display size of the image can be changed in a plurality of steps corresponding to the setting information.

The target detection unit 184 controls the first camera 61 to execute image capturing and acquires a captured image. This captured image is output from the first camera 61 as color image data or monochrome image data, but the first camera 61 outputs an image signal and the target detection unit 184 uses the image signal to conform to a predetermined file format. Data may be generated.
The target detection unit 184 analyzes the acquired captured image data and detects a target included in the captured image data. The target object is an object or a person existing in the image pickup direction of the first camera 61, that is, the line-of-sight direction of the user.

The position detection unit 185, based on the position of the image of the object in the image captured by the first camera 61, is a relative position between the position where the user sees the object and the position where the image displayed by the image display unit 20 is viewed. Ask for. This processing requires information indicating the positional relationship between the display area of the image display unit 20 and the imaging range (angle of view) of the first camera 61. Instead of this information, information indicating the positional relationship between the visual field (field of view) of the user and the imaging range (angle of view) of the first camera 61, and the visual field (field of view) of the user and the display area of the image display unit 20. You may use the information which shows the positional relationship with. These pieces of information are stored in advance in the storage unit 120, for example.
When the position detection unit 185 detects the position of the image of the target object with respect to the display area, the position detection unit 185 corrects this position based on the correction coefficient passed from the setting unit 183.

Further, the position detection unit 185 may detect the display size of the target object in the display area together with the position of the target object.
When the position detection unit 185 detects the display position and the display size with respect to the display area of the image of the object, the position detection unit 185 corrects the detected display position and display size based on the correction coefficient passed from the setting unit 183. Accordingly, when the image displayed by the image display unit 20 and the object in the outside scene are visible to the user, the sizes of the display image and the image of the object perceived by the user are in a predetermined state, Images can be displayed. Furthermore, the size of the display image and the image of the object perceived by the user is predetermined according to the physical characteristics of the user such as the inter-eye distance, the relative distance between the eye and the optical element, and the angle of the optical element. The image can be displayed so that it is in the state.

The information display control unit 186 causes the image display unit 20 to display the display data based on the processing results of the setting unit 183, the target detection unit 184, and the position detection unit 185. The head-mounted display device 100 may be configured to acquire various data such as moving images, still images, characters, and symbols by the data acquisition unit DA, and these data can be used as display data.
The information display control unit 186, based on the display position and/or the display size corrected by the position detection unit 185 based on the correction coefficient for the object detected by the target detection unit 184, the display position of the display data and the display size. To decide. Further, for example, when the display data is text data, the information display control unit 186 sets the display position and the display size of the character, as well as the display color, the font, the presence or absence of character decoration such as bold and italic, and the like. In addition, when the display data is image data, the information display control unit 186 sets the display color, the transparency, and the like in addition to the display position and the display size of the image.

The information display control unit 186 processes the display data based on the color information and the used language input from the setting unit 183. For example, when the input color information is color information indicating that the user has an abnormality with respect to green, the information display control unit 186 converts the green display data into another display color. The information display control unit 186 outputs the converted display data to the display control unit 170 and causes the image display unit 20 to display the display data.
Further, when English is set as the input used language, the information display control unit 186 translates the text data as display data into English, and the translated English text data is displayed as display data in the display control unit 170. Output to. The display data is displayed by the image display unit 20.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. (Not shown). Note that, for example, when the Dolby (registered trademark) system is adopted, the audio signal is processed, and the right earphone 32 and the left earphone 34 output different sounds, for example, with different frequencies.

  Further, the voice processing unit 187 acquires the voice collected by the microphone 63, converts it into digital voice data, and performs a process relating to the voice. For example, the voice processing unit 187 recognizes the voices of a plurality of persons separately by extracting features from the obtained voices and modeling them, and performs speaker recognition to identify the person speaking for each voice. May be.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. (Not shown). Note that, for example, when the Dolby (registered trademark) system is adopted, the processing for the audio signal is performed, and the right earphone 32 and the left earphone 34 output different sounds, for example, with different frequencies.

  The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left light guide plate 262 as a left optical image display unit 28. With. The image display unit 20 also includes a first camera 61, a second camera 62, a vibration sensor 65, and a 9-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor and is arranged inside the image display unit 20 as shown in FIG. In the example of FIG. 1, the right holding unit 21 is built in near the end ER of the right optical image display unit 26. When the user hits the end ER (knock operation), the vibration sensor 65 detects the vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects the knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is attached to the head of the user, the control unit 140 uses the 9-axis sensor 66 based on the detection value of the 9-axis sensor 66. The movement of the person's head can be detected. Since the orientation of the image display unit 20 can be known from the detected movement of the head of the user, the control unit 140 can estimate the line-of-sight direction of the user.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. In addition, the interface 25 outputs the control signal transmitted from the display control unit 170 to the corresponding receiving unit 53, 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface of the first camera 61, the second camera 62, the vibration sensor 65, and the 9-axis sensor 66. The detection result of vibration by the vibration sensor 65 and the detection result of acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes) by the 9-axis sensor 66 are sent to the control unit 140 of the control device 10 via the interface 25. Sent.

  The right display drive unit 22 includes a reception unit 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221, which function as a light source, a right LCD control unit 211 and a right LCD 241, which function as display elements, And a right projection optical system 251. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as a display element. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221, and the right LCD 241 are collectively referred to as an “image light generation unit”.

  The reception unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is, for example, a light emitting body such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data that are input via the reception unit 53. .. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 into parallel light fluxes. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display drive unit 24 includes a reception unit 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as light sources, a left LCD control unit 212 and a left LCD 242 that function as display elements, and And a left projection optical system 252. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as a display element. Further, the left projection optical system 252 is configured by a collimator lens that converts the image light emitted from the left LCD 242 into a parallel light flux. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path.

FIG. 8 is a flowchart showing the operation of the head-mounted display device 100 when registering user information.
The process of step S11 of FIG. 8 corresponds to the process of step S1 described in the flowchart of FIG. 6, step S12 corresponds to step S1 (FIG. 6), and step S13 corresponds to step S3 (FIG. 6).

  That is, the biometric information detection unit 181 outputs the silhouette image S to the display control unit 170 in the information registration mode and causes the image display unit 20 to display the silhouette image S (step S11). The user adjusts the position of the hand while observing the silhouette image S and operates the operation unit 111 to instruct to capture an image of the hand R. When the operation of the operation unit 111 is input, the biological information detection unit 181 causes the first camera 61 to capture an image of the hand R of the user (step S12). The captured image of the first camera 61 is input to the biometric information detection unit 181, and the biometric information detection unit 181 performs image processing on the captured image to extract biometric information (step S13).

In the user identification mode, biometric information is input to the setting unit 183 from the biometric information detection unit 181.
When the input biometric information is input from the biometric information detection unit 181, the setting unit 183 selects the biometric information used for the matching process (step S14). In addition, in this processing flow, the setting unit 183 performs the matching process using the palm line information. The setting unit 183 selects one or a plurality of pieces of palm line information to be used for matching from a plurality of pieces of palm line information included in the input biometric information. As a result, the palm line information used for matching is thinned out. When the palm line information is selected from the input biometric information, the setting unit 183 reads out the palm line information whose coordinates are closest to the starting point coordinates and the ending point coordinates of the selected palm line information from the user information table and performs the matching process (step). S15). The setting unit 183 collates the length ratio and/or the curvature of the palm line information extracted from the user information table with the length ratio and/or the curvature of the palm line information selected from the input biometric information. The setting unit 183 performs collation processing on all the selected palm line information (step S16). When the matching ratio of the length ratio and/or the curvature is lower than the threshold value by the matching process, that is, it is determined that they do not match (step S16/NO), the setting unit 183 causes the silhouette displayed by the image display unit 20 to be displayed. A guide display is additionally displayed on the image S (step S17). The setting unit 183 adds the palm line of the palm line information determined to have a high degree of matching in the matching process to the silhouette image S as an additional guide display (see FIG. 9 ).

  FIG. 9 shows an example in which a palm line is added to the silhouette image S as an additional guide display. In this example, a curved line is added to the silhouette image S displayed in the display area T as an additional guide for display. This additional guide makes it easier to overlap the user's hand R on the silhouette image S. For example, the matching degree of the matching process may be low depending on the distance or angle between the first camera 61 and the hand. Therefore, by causing the setting unit 183 to display the guide display of the palm line on the silhouette image S, it is possible to improve the matching accuracy of the authentication process. The additional guide is not limited to the curve corresponding to the palm line, and may be a straight line or may include characters and other figures. In the operation of overlapping the hand on the silhouette image S, the information for guiding the position of the hand may be included.

When the matching process determines that the matching ratio of the length ratio and/or the curvature is higher than the threshold value (step S16/YES), the setting unit 183 reads the setting information from the user information table ( Step S18). This setting information is the setting information associated with the registered biometric information determined to have a high degree of coincidence with the input biometric information. The setting unit 183 calculates a correction coefficient for correcting the display position or size of the display data based on the read setting information. The setting unit 183 calculates a correction coefficient that corrects the display position or size of the display data based on at least one of the inter-eye distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element. To do. After calculating the correction coefficient, the setting unit 183 outputs the calculated correction coefficient to the position detection unit 185.
The position detection unit 185 detects the position and size with respect to the display area of the image of the object detected by the object detection unit 184 from the captured image. When the position detection unit 185 detects the position and size with respect to the display area, the position detection unit 185 corrects the detected position and size based on the correction coefficient passed from the setting unit 183 (step S19). The information display control unit 186 determines the display position of the display data for the object detected by the object detection unit 184, based on the position and/or size corrected by the position detection unit 185 based on the correction coefficient. After that, the information display control unit 186 outputs the display data to the display control unit 170 and causes the image display unit 20 to perform display.

Next, the operation of transmitting and receiving the contents of the user information table 120A and the palm line information table 120B in the communication system 1 will be described.
As the first communication mode, there is a mode in which the head-mounted display device 100A and the head-mounted display device 100B respectively transmit data to the server 5. The head-mounted display device 100A transmits the user information table 120A and the palm line information table 120B stored in the storage unit 120 to the server 5. Further, the head-mounted display device 100B transmits the user information table 120A and the palm line information table 120B stored in the storage unit 120 to the server 5.
The server 5 stores the data of the user information table 120A and the palm line information table 120B transmitted from the head-mounted display devices 100A and 100B, respectively. Here, the server 5 can form a user information table of the display system 2 by aggregating the tables transmitted from the head-mounted display devices 100A and 100B.

10A and 10B are schematic diagrams showing a configuration example of a user information table generated and stored by the server 5, FIG. 10A shows the user information table 5A, and FIG. 10B shows a palm line information table 5B. 10C shows the device information table 5C.
The user information table 5A is a table in which the data of the storage unit 120 received from the head-mounted display devices 100A and 100B are collected. The palm line information table 5B is a table in which the palm line information tables 120B received from the head-mounted display devices 100A and 100B are integrated.

In the user information table 5A, the device ID is associated with the data of the user information table 120A shown in FIG. The data in which the device ID is set to “1” in the user information table 5A is the data transmitted to the server 5 by the head-mounted display device 100 assigned with the device ID “1”. Further, the data in which the device ID is set to “2” is the data transmitted to the server 5 by the head-mounted display device 100 to which the device ID “2” is assigned. Further, the palm line information table 5B includes the data of the palm line information table 120B in association with the device ID.
In the examples of FIGS. 10A and 10B, there are two pieces of data (records) whose user ID is “1”. This indicates that one user has registered the biometric information on each of the two head-mounted display devices 100A and 100B.

  In this way, the server 5 receives the user information table 120A and the palm line information table 120B from the plurality of head-mounted display devices 100, and collects and stores the received data. Thereby, the biometric information and the setting information registered in the head-mounted display device 100 that configures the communication system 1 can be shared by all the head-mounted display devices 100. For example, the server 5 transmits the user information table 5A and the palm line information table 5B to the plurality of head-mounted display devices 100, so that the biometric information and the setting information registered in the head-mounted display device 100A are displayed. It can be used with the head-mounted display device 100B. That is, if the user performs registration on any one of the head-mounted display devices 100, the registered settings can be realized on the other head-mounted display device 100 based on his/her setting information. Therefore, both when a plurality of users use one head-mounted display device 100 and when one user uses a plurality of head-mounted display devices 100, based on biometric information. It is possible to realize a preset setting suitable for the user.

  Further, the server 5 may have a device information table 5C. The device information table 5C stores data relating to specifications of each head-mounted display device 100 in association with the device ID. According to the device information table 5C, with respect to the plurality of head-mounted display devices 100 included in the communication system 1, it is possible to specify a difference in specifications of the head-mounted display devices 100. For example, when the server 5 transmits the user information table 5A including the setting information registered in the head-mounted display device 100B to the head-mounted display device 100A, the user based on the device information table 5C. The value of the information table 5A can be converted. In this case, the same state as the setting of the head-mounted display device 100B can be realized in the head-mounted display device 100A, taking into account the difference in the specifications of the head-mounted display device 100B and the head-mounted display device 100A. .. Further, for example, the server 5 may transmit the device information table 5C to both the head-mounted display devices 100A and 100B, and the head-mounted display devices 100A and 100B may execute the above conversion. Accordingly, even if the specifications of the head-mounted display device 100 included in the communication system 1 differ, each head-mounted display device 100 can realize a setting state according to the user.

  A second communication mode is a mode in which data is transmitted from the head-mounted display device 100A to the head-mounted display device 100B, or from the head-mounted display device 100B to the head-mounted display device 100A. .. In this case, the head-mounted display devices 100A and 100B communicate with each other directly or via the wireless access point 3 to generate the user information table 5A and the palm line information table 5B, and the storage unit. It may be stored in 120. Also in this case, there is an advantage that the registered biometric information and setting information can be shared by the plurality of head-mounted display devices 100 included in the communication system 1.

  As described above, the head-mounted display device 100 of the first embodiment to which the present invention is applied has the storage unit 120, the first camera 61 as the input unit, the biological information detection unit 181, and the information display control unit. And 186. The storage unit 120 stores the setting information regarding the display of the image display unit 20 in association with the identification information for identifying the user. The first camera 61 as an input unit captures an image of the user's hand, and the biometric information detection unit 181 as an input unit extracts biometric information from the captured image. The information display control unit 186 controls the display of the image display unit 20 based on the setting information corresponding to the input biometric information. Thereby, based on the setting information, the display of the head-mounted display device 100 can be promptly set according to the user.

  In addition, the user information table 120A stored in the storage unit 120 includes some of the characteristics included in the biometric information of the user. The user can be identified by using the biometric information, and since a part of the features included in the biometric information is included, the simple setting according to the physical feature can be performed.

The information display control unit 186 includes a part of the characteristics of the biometric information input as identification information by the first camera 61 and the biometric information detection unit 181, and a part of the characteristics of the biometric information stored in the storage unit 120 as the identification information. And specify the setting information. Therefore, it is possible to make simple settings according to physical characteristics.
Further, the image display unit 20 displays the image so as to be visible through the outside scene and visible together with the outside scene, and the information display control unit 186 sets at least one of the display position and the display size of the image according to the setting information. Change to multiple stages. Therefore, the display position and the display size of the image on the image display unit 20 can be set in stages according to the user.

  In addition, the image display unit 20 includes a right light guide plate 261 and a left light guide plate 262 that transmit the image light to the eyes of the user so that the image light can be visually recognized together with the outside scene. The control unit 140 also includes a target detection unit 184 that detects a target object in the direction of the line of sight of the user, and a position detection unit 185 that detects the position of the target object in the display area of the image display unit 20. The information display control unit 186 displays the image by the optical element according to the position of the object detected by the position detection unit 185 and the positional relationship between the right light guide plate 261, the left light guide plate 262, and the position of the user's pupil. Change the display position. Therefore, the display position of the image can be set according to the user.

In addition, the registration information of the user information table 120A includes, as the setting information, information about language setting. The information display control unit 186 displays the characters in the language corresponding to the setting information when displaying the content including the characters on the image display unit 20. Therefore, the language of the characters displayed on the image display unit 20 can be set according to the user.
Further, the head-mounted display device 100 includes a communication unit 117, and the information display control unit 186 transmits the user information table 120A and the palm line information table 120B stored in the storage unit 120 to the server 5. Further, in the head-mounted display device 100, the communication unit 117 may receive the user information table 5A and the palm line information table 5B and store them in the storage unit 120. In this case, the user information table 120A and the palm line information table 120B can be shared with another head-mounted display device 100. Further, based on the user information table 5A and the palm line information table 5B including the information registered in another device, it is possible to set the display according to the user.

[Second Embodiment]
FIG. 11 is a flowchart showing the operation of the head-mounted display device 100 according to the second embodiment of the present invention. In the second embodiment, an operation for asking the user for confirmation when the authentication using the biometric information (palm line information) is successful will be described.
The configuration of the head-mounted display device 100 of the second embodiment is the same as that of the first embodiment, and thus the common components are designated by the same reference numerals and illustration and description thereof are omitted. Further, in the flowchart of FIG. 11, the same steps as those of the operation of FIG.

In the second embodiment, an example in which the setting unit 183 performs the matching process using the palm line information in the user identification mode will be described.
The setting unit 183 selects one or more pieces of palm line information to be used for matching from a plurality of pieces of palm line information included in the input biometric information, and the start point coordinates and end point coordinates of the selected palm line information have the most coordinates. The near palm line information is read from the user information table, and the matching process is performed (step S15). The setting unit 183 collates the length ratio and/or the curvature of the palm line information extracted from the user information table with the length ratio and/or the curvature of the palm line information selected from the input biometric information. The setting unit 183 performs collation processing for all the selected palm line information (step S16).

  If it is determined by the matching process that they do not match (step S16/NO), the setting unit 183 additionally displays the guide display on the silhouette image S displayed by the image display unit 20 (step S17), and the process returns to step S12.

  On the other hand, when the matching process determines that the length ratio and/or the curvature matching degree of the palm line information are higher than the threshold value, the setting unit 183 determines that the palm line information matches (step S16/YES). ). In this case, the setting unit 183 acquires the user ID from the setting information associated with the registered biometric information determined to have a high degree of coincidence with the input biometric information in the user information table 120A, and uses the acquired user ID as the acquired user ID. Based on this, a confirmation message is output (step S21). The content of the confirmation message is a message for confirming that the setting corresponding to the user ID whose palm line information matches. The output of the message is performed, for example, by displaying a text or an image by the image display unit 20 and/or by outputting a voice from the right earphone 32 and the left earphone 34. As a specific example of the message, the user ID may be used, and “using the setting of ID:1” or the like. The user name corresponding to the user ID may be registered in the user information table 120A. In this case, the user name can be included in the message output in step S21.

The message output in step S21 may include a message asking the user to input an instruction. 12A and 12B show examples of messages displayed on the image display unit 20 as examples of messages output by the head-mounted display device 100.
The message M1 shown in FIG. 12(A) is a content that presents the user ID and requests the user to input an instruction for setting. In the message M1, three types of instruction inputs are possible: “1” for instructing acceptance, “2” for instructing re-authentication, and “3” for instructing new registration of palm line information. ..
The message M2 shown in FIG. 12B is a content that presents the user name corresponding to the user ID and requests the instruction input. In the message M2, like the message M1, three types of instruction inputs are requested. The setting unit 183 outputs the messages M1 and M2 by displaying them on the image display unit 20, for example.

  After outputting a message such as the message M1 or M2 that requests or prompts the user to input an instruction, the setting unit 183 waits for the user's input operation within a predetermined time and determines whether or not there is an input operation ( Step S22). When it is determined that the input operation has not been performed within the predetermined time after outputting the message (step S22/NO), the setting unit 183 proceeds to step S18.

As described with reference to FIG. 8, in step S18, the setting unit 183 reads the setting information from the user information table (step S18). This setting information is the setting information associated with the registered biometric information determined to have a high degree of coincidence with the input biometric information, and the setting information corresponding to the user ID referred to when the confirmation message is output in step S21. Is.
The setting unit 183 calculates a correction coefficient for correcting the display position or size of the display data based on the read setting information. The setting unit 183 calculates a correction coefficient that corrects the display position or size of the display data based on at least one of the inter-eye distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element. To do. After calculating the correction coefficient, the setting unit 183 outputs the calculated correction coefficient to the position detection unit 185.

  The position detection unit 185 detects the position and size with respect to the display area of the image of the object detected by the object detection unit 184 from the captured image. When the position detection unit 185 detects the position and size with respect to the display area, the position detection unit 185 corrects the detected position and size based on the correction coefficient passed from the setting unit 183 (step S19). The information display control unit 186 determines the display position of the display data for the object detected by the object detection unit 184, based on the position and/or size corrected by the position detection unit 185 based on the correction coefficient. After that, the information display control unit 186 outputs the display data to the display control unit 170 and causes the image display unit 20 to perform display.

On the other hand, when it is determined that the input operation is performed within the predetermined time after the confirmation message is output (step S22/YES), the setting unit 183 determines the content of the input operation (step S23). Here, as shown in FIGS. 12A and 12B, it is assumed that it is possible to perform three kinds of instructions for setting (acceptance instruction, re-authentication instruction, new registration instruction). When the setting unit 183 determines in step S23 that the input operation indicating acceptance is performed, the setting unit 183 proceeds to step S18. When it is determined in step S23 that the input operation for instructing re-authentication is performed, the setting unit 183 proceeds to step S17.
If it is determined in step S23 that the input operation for instructing new registration has been performed, the setting unit 183 shifts to the information registration mode (step S24) and executes the operation of FIG.
In the operation of the second embodiment shown in FIG. 11, the input operation performed by the user may be a touch operation on the track pad 14. Further, the voice of the user may be collected by the microphone 63 and recognized by the voice processing unit 187 so that the user can input the voice. In addition, the user moves a body such as a finger, a hand, an arm, or a foot, and the control unit 140 detects the movement as a gesture based on the images captured by the first camera 61 and the second camera 62, thereby inputting a gesture. It may be configured to be possible. As described above, the head-mounted display device 100 can be input by operating the control device 10, a voice input, and a gesture input. These inputs can be adopted not only in the first and second embodiments but also in each of the embodiments described later.

  In this way, in the user identification mode, the usability is improved by outputting a confirmation message to the user before performing the setting based on the matching result using the biometric information such as palm lines. You can That is, by simplifying the biometric information, the user can appropriately deal with the possibility that the collation result is unintended by the user. Further, even when the collation result is in agreement with the fact, when the user desires to re-register the biometric information, the operation of new registration can be easily performed. As a result, the user can use the device as intended by the user, and the convenience can be improved.

[Third Embodiment]
FIG. 13 is a flowchart showing the operation of the head-mounted display device 100 according to the third embodiment of the present invention. In the third embodiment, an operation in which the display on the image display unit 20 is a single-eye display until the setting based on the setting information corresponding to the collation result of the biometric information (palm line information) is performed will be described.
Since the configuration of the head-mounted display device 100 of the third embodiment is common to that of the first embodiment, the same reference numerals are given to the common components, and illustration and description thereof will be omitted. Further, in the flowchart of FIG. 13, the same steps as those of the operation of FIG.

In the third embodiment, an example in which the setting unit 183 performs the matching process using the palm line information in the user identification mode will be described.
When the operation in the user identification mode is started, the setting unit 183 switches the display of the image display unit 20 to the one-eye display before the operation of displaying the silhouette image S on the image display unit 20 in step S11 (step S31). .. The setting unit 183 controls the image processing unit 160 and/or the display control unit 170 to stop the display by one of the right display drive unit 22 and the left display drive unit 24.

The stop of the display means a state in which the display by the right display drive unit 22 or the left display drive unit 24 is not visually recognized by the user. Control may be included in which either the right LCD 241 or the left LCD 242 is displayed in black or in a predetermined color set in advance in the entire display area. It may also include control to turn off either the right backlight 221 or the left backlight 222. Further, the display control unit 170 may include control for stopping the transmission of the image data Data by either one of the transmission units 51 and 52. If at least one of the above controls is executed, one of the display images of the right display driving unit 22 and the left display driving unit 24 cannot be viewed.
Which of the right display drive unit 22 and the left display drive unit 24 is to be stopped may be set in advance or may be selected by the user.
In this way, the state in which the display of either the right display drive unit 22 or the left display drive unit 24 is stopped can be said to be a one-eye (one-eye) display state in which the user visually recognizes the display image with one eye. On the other hand, a state in which both the right display driving unit 22 and the left display driving unit 24 are not stopped can be said to be a binocular (both eyes) display state in which the user visually recognizes the display image with both eyes. Under the control of the control unit 140, the image display unit 20 can switch between single-eye display and binocular display and execute the display.

After switching the display state of the image display unit 20 to the one-eye display, the setting unit 183 executes the operations of steps S11 to S18 described with reference to FIG.
The setting unit 183 reads out the setting information from the user information table 120A in step S18, corrects the display position according to the read setting information, and switches to the binocular display (both-eye display) (step S32). Here, the setting unit 183 starts the display for the side of the right display driving unit 22 and the left display driving unit 24 that has stopped the display in step S31, and switches the image display unit 20 to the binocular display. The timing of switching to the binocular display is not limited as long as the setting information is read from the user information table 120A, but it is preferable that the display position is corrected according to the read setting information as described later. In addition, when the setting information includes information of display attributes or setting items other than the display position, not only the correction of the display position, but also when the display is switched to the binocular display after the correction of the display based on these information is performed, More preferable.

When starting the binocular display, the setting unit 183 calculates a correction coefficient for correcting the display position or size of the display data based on the setting information read from the user information table 120A, and uses the calculated correction coefficient as the position. It is output to the detection unit 185. The details of the process of calculating the correction coefficient are the same as in step S19 (FIG. 8) described above. The position detection unit 185 detects the position and size with respect to the display area of the image of the object detected by the object detection unit 184 from the captured image. When the position detection unit 185 detects the position and the size with respect to the display area, the position detection unit 185 corrects the detected position and size based on the correction coefficient passed from the setting unit 183. The information display control unit 186 determines the display position of the display data for the object detected by the object detection unit 184 based on the position and/or the size corrected by the position detection unit 185 based on the correction coefficient.
The control unit 140 may execute either the process of determining the display position of the display data or the process of switching the display state of the image display unit 20 from the one-eye display to the binocular display. When the control unit 140 first executes the process of determining the display position and then switches the display of the image display unit 20 to the binocular display, it is preferable because the binocular display can be started at a suitable position.

  As described above, in the user identification mode, the display state of the image display unit 20 is set to one-eye display until the setting information is read from the user information table 120A based on the biometric information and the display reflecting the setting information is performed. This can reduce the user's discomfort. That is, if the display position and the like are corrected according to the setting information of the user information table 120A, a display state suitable for the user can be realized, but before the correction, the display position and the like may not be suitable for the user. is there. When this display position mismatch is large, the user may feel uncomfortable until the display position is corrected according to the setting information. In particular, when displaying an image so as to be visually recognized by both eyes of a user, if the parameters relating to binocular vision such as the inter-eye distance (inter-pupil distance) and the angle of convergence do not match, a sense of discomfort is great. In the third embodiment, single-eye display is performed until the setting information is reflected in the display on the image display unit 20. That is, since the one-eye display is performed while the display position may be incompatible, it is possible to suppress the discomfort caused by the incompatibility.

[Fourth Embodiment]
FIG. 14 is a flowchart showing the operation of the head-mounted display device 100 according to the fourth embodiment of the present invention. In the fourth embodiment, an example will be described in which when registration of biometric information (palm line information) fails, new registration or the like is executed according to an input operation by the user.
The configuration of the head-mounted display device 100 of the fourth embodiment is the same as that of the first embodiment described above, and thus the common components are designated by the same reference numerals and illustration and description thereof are omitted. Further, in the flowchart of FIG. 14, the same steps as those of the operation of FIG.

  In the fourth embodiment, an example will be described in which the setting unit 183 performs matching processing using palm line information in the user identification mode. The operations of steps S11 to S16 are the same as the operations described with reference to FIG.

In step S15, the setting unit 183 selects one or more pieces of palm line information used for matching from the plurality of pieces of palm line information included in the input biometric information, and the start point coordinates and end point coordinates of the selected palm line information. The palm line information whose coordinates are closest to is read from the user information table, and the matching process is performed. Then, in step S16, the setting unit 183 determines whether or not the palm line information matches in the matching process.
When it is determined by this collation processing that they match (step S16/YES), the setting unit 183 executes the processing of steps S18 to S19, similarly to the operation described with reference to FIG.

  On the other hand, when it is determined by the collation processing that the palm line information does not match (step S16/NO), the setting unit 183 outputs an error message (step S24). The content of the error message includes a message indicating that they do not match each other by collation, and a message requesting the user to input an instruction regarding the subsequent operation. The output of the message is performed, for example, by displaying a text or an image by the image display unit 20 and/or by outputting a voice from the right earphone 32 and the left earphone 34.

  FIG. 15 illustrates an error message M3 displayed on the image display unit 20 as an example of the message output by the head-mounted display device 100. The error message M3 includes text indicating that an authentication error has occurred. Further, the error message M3 has three input options as an instruction input option: "1" for instructing re-authentication, "2" for selecting from the use history, and "3" for instructing new registration of palm line information. Is possible.

  After outputting the error message M3, the setting unit 183 waits for a user's input operation within a predetermined time, and if there is an input operation, determines whether the input operation is a re-authentication instruction. (Step S42). When it is determined that the input operation is an instruction for re-authentication (step S42/YES), the setting unit 183 proceeds to step S17.

  On the other hand, when it is determined that the input operation is not a re-authentication instruction (step S42/NO), the setting unit 183 determines whether it is a new registration instruction (step S43). Here, when it is determined that the instruction is for new registration (step S43/YES), the setting unit 183 shifts to the information registration mode (step S24) and executes the operation of FIG.

  Therefore, when the user cannot be specified due to the failure of the authentication using the biometric information, the user can select whether to perform the authentication again or newly register the biometric information.

  Further, when it is determined that the input operation is not a new registration instruction (step S43/NO), the setting unit 183 determines that the input operation is an instruction to select the user from the history information, and outputs the history information ( Step S45). The history information is a history of a user who uses the head-mounted display device 100, and is, for example, a history of a user ID that has acquired setting information from the user information table 120A. Every time the head-mounted display device 100 acquires the setting information from the user information table 120A in step S18, the head-mounted display device 100 stores the user ID corresponding to the acquired setting information in the storage unit 120 or the RAM of the control unit 140 ( (Not shown) as history information. Here, when the user name is registered in the user information table 120A in association with the user ID, the setting unit 183 may store the user name as history information. The history information may be added each time the setting unit 183 reads the setting information from the user information table 120A, and an upper limit may be set for the number of history information stored in the storage unit 120 or the RAM (not shown). .. In step S45, the setting unit 183 outputs the history information stored in the storage unit 120 or the RAM by the display by the image display unit 20 or the sound output from the right earphone 32 and the left earphone 34. The history information output here may be a part of the history information stored in the storage unit 120 or the RAM. For example, based on the date and time when the history information is stored, a preset number of history information may be selected and output in ascending order.

  After that, the setting unit 183 waits for an input operation by the user, and when an input operation for selecting any of the output history information is performed (step S46), the setting unit 183 proceeds to step S18 and displays the selected history information. The setting information corresponding to a certain user ID is read from the user information table 120A.

  In this way, in the user identification mode, even if the collation using biometric information such as the palm line fails, the result obtained by identifying the user in the past is used and the operation is performed based on the user's input operation. It is possible to specify a person and adjust the display position and the like. Therefore, even if there is a problem in collating biometric information, a legitimate user can use the head-mounted display device 100, and usability can be improved.

  Further, when outputting the history information, the date and time or the season when the history information is stored may be output in association with the user ID or the like as the history information. In this case, the user can select the setting information used for setting the head-mounted display device 100 based on the date and time and the season. Further, the history information may be stored in the storage unit 120 in association with the position information detected by the GPS 115. In this case, when outputting the history information, the position information may be associated and output. The user can select history information based on where the history information was used in the past.

[Fifth Embodiment]
FIG. 16 is a flowchart showing the operation of the head-mounted display device 100 according to the fifth embodiment. FIG. 17 is a diagram showing a configuration example of a user information table 120C stored in the storage unit 120 instead of the user information table 120A in the head-mounted display device 100 of the fifth embodiment. FIG. 18 is a diagram showing an example of an image that can be used for authentication in the head-mounted display device 100 according to the fifth embodiment. An example using an image of a carrying object is shown, and (C) shows an example using an image of a building.
In the head-mounted display device 100 according to the fifth embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and illustration and description thereof will be omitted.

  In the first to fourth embodiments, the head-mounted display device 100 uses biometric information such as palm line information for the process of identifying the user. On the other hand, in the fifth embodiment, the head-mounted display device 100 uses information other than biometric information. Therefore, as shown in FIG. 17, the table referred to in the authentication is not limited to biometric information, and can be a general image.

  The user information table 120C of FIG. 17 stores registered image information relating to an image used for authentication and setting information in association with each other. The registered image information corresponds to the biometric information in the user information table 120A and is associated with the user ID. The registered image information is information used when detecting a preregistered image from the images captured by the first camera 61 and the second camera 62 and determining the matching with the detected image.

  In the user information table 120C illustrated in FIG. 17, the registered image information includes data regarding the size of the image detected from the captured image, the feature amount, and the text and code included in the image. The text code is text that the control unit 140 can recognize by image recognition, and information encoded by a barcode, a two-dimensional code, or the like. The type of code is not limited as long as the control unit 140 can recognize and decode the captured image. The unit of size is arbitrary, and can be represented by the number of dots (the number of pixels) in the images captured by the first camera 61 and the second camera 62, for example, as illustrated in FIG. The feature amount is a feature amount of an image used for authentication. For example, when an image of an object is used for authentication, the user information table 120C includes the characteristic amount indicating the color, shape, and other characteristics of the captured image when the object is captured. In this case, the setting unit 183 performs the process of extracting the image of the object from the imaged image data of the first camera 61 and/or the second camera 62, calculates the feature amount of the extracted image, and the calculated feature amount, The feature amount included in the user information table 120C is compared and collated. When the feature values are close or the same, it can be determined that the authentication is successful.

The user information table 120C is generated by registering a captured image of an object or the like instead of the image of the hand in the operation of the information registration mode described with reference to FIG. 6 in the first embodiment, for example. it can.
That is, instead of the process of displaying the silhouette image in step S1 of FIG. 6, the reference line SP (FIG. 18) that is an index of the position of the image is displayed, the image is executed in step S2, and the image is captured in step S3. The registered image information including the size and the feature amount is calculated. Then, the setting information may be calculated in step S4, and the setting information may be stored in association with the registered image information in step S5.

  FIG. 18 shows an example of an image used for authentication. In FIG. 18A, the trajectory of the user's hand is used for authentication. The control unit 140 displays the reference line SP in the display area T located in the visual field VA. The user moves his or her hand with the reference line SP as a guide, and the control unit 140 extracts an image of the trajectory of the hand from the imaged image of the first camera 61 and/or the second camera 62 (hereinafter, simply referred to as an imaged image). In the example of FIG. 18A, a locus of a hand that draws a signature with the name “YAMADA” is shown. The control unit 140 calculates the size and the feature amount of the image O1 of this trajectory. Alternatively, the image text recognition process may be executed on the image O1 to obtain the text.

FIG. 18B illustrates an example in which an image of a watch is used for authentication as an example in which an image of a portable object is used for authentication. The user adjusts the orientation of the face and the position of the clock so that the clock overlaps the position of the reference line SP of the display area T. The control unit 140 extracts the clock image O2 from the captured image and uses it for authentication. Not limited to a watch, any object that can be held and moved by a user can be used for authentication in the same manner as a watch.
FIG. 18C shows an example in which an image of a non-portable object such as a real estate or a landscape is used for authentication. In FIG. 18C, the user may move so that the building overlaps the reference line SP. In this case, the control unit 140 extracts the image O3 of the building or the landscape from the captured image and calculates the size and the feature amount.
As another example, for example, an ID card on which numbers or codes are printed, a vehicle or a moving body such as an automobile or a bicycle, clothes or accessories, and the like, which can be imaged by the first camera 61 or the second camera 62, can be used. Good.
As described above, the image used for authentication in the fifth embodiment can be the motion of an object such as a hand or the image of the object regardless of whether it is portable or not, and a specific one The image of the landscape is not limited to the object but may be a plurality of objects.

With the user information table 120C stored in the storage unit 120, the user identification mode of FIG. 16 can be executed.
The process of step S51 of FIG. 16 is the same as step S1 of FIG. 6 or step S11 of FIG. 8, and instead of the silhouette image S, the reference line SP is displayed. In the fifth embodiment, since the size and shape of the image used for authentication are various, it is efficient to use the reference line SP indicating the center position like the reference line SP shown in FIG. 18, for example.

  The biological information detection unit 181 outputs the reference line SP to the display control unit 170 and causes the image display unit 20 to display it (step S51). The user moves the hand while visually recognizing the reference line SP, or moves so as to visually recognize the object, and instructs the imaging. When the biological information detection unit 181 detects a user operation, the biological information detection unit 181 causes the first camera 61 and/or the second camera 62 to perform imaging (step S52). The captured image is input to the biometric information detection unit 181, and the biometric information detection unit 181 performs image processing on the captured image and extracts image information for authentication (step S53).

In the user identification mode, the image information is input to the setting unit 183 from the biometric information detection unit 181.
When the image information is input from the biometric information detection unit 181, the setting unit 183 selects the image information used for the matching process (step S54). The setting unit 183 selects one or a plurality of image information used for matching from the image information extracted by the biometric information detection unit 181 (step S55). As a result, the image information used for matching is thinned out. When the setting unit 183 selects the image information to be used for the authentication, the setting unit 183 reads out the registered image information in which the size, the feature amount, the text, the code, and the like of the selected image information match or approximate from each other, and performs the matching process. Perform (step S56). When a plurality of pieces of image information are selected as the image information for authentication, the setting unit 183 may perform the matching process on all the selected image information in step S56.

  The setting unit 183 determines whether the image information matches the registered image information by the matching process (step S57). When it is determined that they do not match (step S57/NO), the setting unit 183 additionally displays the guide display on the reference line SP displayed by the image display unit 20 (step S58). For the guide display, for example, the contour line of the image information that the setting unit 183 determines to have a high degree of matching in the matching process can be used. After displaying the guide display, the process returns to step S53.

When it is determined by the collation processing that the image information and the registered image information match (step S57/YES), the setting unit 183 reads the setting information from the user information table 120C (step S59). This setting information is the setting information associated with the registered image information determined to have a high degree of coincidence with the image information. The setting unit 183 calculates a correction coefficient for correcting the display position or size of the display data based on the read setting information. The setting unit 183 calculates a correction coefficient that corrects the display position or size of the display data based on at least one of the inter-eye distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element. To do. After calculating the correction coefficient, the setting unit 183 outputs the calculated correction coefficient to the position detection unit 185.
The position detection unit 185 detects the position and size with respect to the display area of the image of the object detected by the object detection unit 184 from the captured image. When the position detection unit 185 detects the position and size with respect to the display area, the position detection unit 185 corrects the detected position and size based on the correction coefficient passed from the setting unit 183 (step S60). The information display control unit 186 determines the display position of the display data for the object detected by the object detection unit 184, based on the position and/or size corrected by the position detection unit 185 based on the correction coefficient. After that, the information display control unit 186 outputs the display data to the display control unit 170 and causes the image display unit 20 to perform display.

  As described above, in the head-mounted display device 100, instead of the biometric information such as the palm line information described in the first to fourth embodiments, the information related to the image extracted from the captured image of the object is displayed. It can be used for authentication. Therefore, unlike the case of using the biometric information that can be normally associated with one person, a plurality of setting information can be associated with one person and registered in the user information table 120C. Therefore, a wider range of setting information can be used by the user.

The image information and registered image information of the fifth embodiment are information other than biometric information, and are information extracted, detected, or calculated from a captured image of an object. More specifically, it is information obtained based on an image of an object extracted from a captured image, and this object includes a living body and is movable, an immovable object such as a building, and a plurality of objects. Including scenery including. The information (image information) obtained based on the image of the object does not include image features such as fingerprints, palm prints, facial features and the like that are peculiar to an individual living body. For example, it is preferable that the individual living body cannot be identified by the image information alone.
In other words, as the image information, the shape of the object obtained by extracting the outline or contour of the object shown in the captured image, the color of the object shown in the captured image, the shape of the object obtained from multiple captured images And the locus of the position of the object obtained from a plurality of captured images.

  From these viewpoints, the image information mentioned as the second identification information can be referred to as non-living body unique information that cannot identify an individual living body, for example. Also, image information is extracted from appearance information obtained from the appearance of an object, outside scene information obtained by extracting information including the shape of a building from a captured image including a building as an object, and captured image obtained by capturing an inanimate object for identification. It can be said that the information includes any one of the identification object information described above and the trajectory information related to the trajectory when a moving object is continuously photographed. However, it is an image unique to an individual living body such as a fingerprint, a palm print, or a facial expression. The information is not limited as long as the information does not include the characteristics.

[Sixth Embodiment]
FIG. 19 is a flowchart showing the operation of the head-mounted display device 100 according to the sixth embodiment. 20 is a diagram showing a configuration example of a user information table 120D stored in the storage unit 120 in the head-mounted display device 100 of the fifth embodiment instead of the user information table 120A. In the head-mounted display device 100 of the sixth embodiment, the same components as those of the first embodiment are designated by the same reference numerals, and illustration and description thereof will be omitted.

In the first to fourth embodiments, the head-mounted display device 100 uses biometric information such as palm line information for the process of identifying the user. In addition, in the fifth embodiment, the head-mounted display device 100 uses information other than biometric information.
In the sixth embodiment, an example will be described in which information other than the biometric information described in the fifth embodiment and the biometric information described in the first to fourth embodiments are combined and used for authentication.
The table referred to in the authentication of the sixth embodiment has a configuration in which setting information is registered in association with a combination of biometric information and image information, as shown in FIG.

  The user information table 120D of FIG. 20 stores registered image information relating to an image used for authentication and setting information in association with each other. The registered image information corresponds to the biometric information in the user information table 120A and is associated with the user ID. The registered image information is information used when detecting a preregistered image from the images captured by the first camera 61 and the second camera 62 and determining the matching with the detected image.

  In the user information table 120D illustrated in FIG. 20, similar to the user information table 120A (FIG. 5A), the registered biometric information is registered in association with the user ID. Further, in the user information table 120D, similar to the user information table 120C (FIG. 17), registered image information is registered in association with the user ID. The configurations of the registered biometric information, the registered image information, and the setting information are the same as those in the above-described embodiments.

  In the user information table 120D, the registered biometric information is associated with one user ID. Also, a plurality of registered image information is associated with one user ID and registered biometric information. Then, the setting information is registered in association with the registered biometric information. Then, in the user information table 120D, the setting information is registered in association with the combination of the registered biometric information and the registered image information. For example, the same registered image information may be associated with different registered biometric information.

  FIG. 19 shows the operation when authentication is performed using the user information table 120D. In FIG. 19, the process of step S60 is the same as the process described with reference to FIG. 16 in the fifth embodiment.

In the operation of FIG. 19, two types of information, that is, biometric information and image information, are collated with the information registered in the user information table 120D, and the setting information corresponding to the combination of these two types of information is read. Either one of the biometric information and the image information is collated first as the first identification information, and after the collation is successful, the other is collated as the second identification information. The image information may be the first identification information and the biometric information may be the second identification information, or vice versa.
The user information table 120D illustrated in FIG. 20 has a configuration in which the same registered image information may be associated with different registered biometric information, whereas the biometric information is uniquely associated with a user ID. Be done. Therefore, in the sixth embodiment, it is more preferable to collate the biometric information as the first identification information. This example will be described below.

  The process of step S71 of FIG. 19 is the same as the process of step S1 of FIG. 6, step S11 of FIG. 8 and sub-flow stomach of step S51 of FIG. 16, and the silhouette image S or the reference line SP is displayed. In the example in which biometric information is used as the first identification information, the silhouette image S is displayed. In addition, in the example in which the image information is used as the first identification information, the reference line SP is displayed.

  The biometric information detection unit 181 outputs the silhouette image S to the display control unit 170 and displays it on the image display unit 20 (step S72). The user adjusts the position of the hand while observing the silhouette image S and operates the operation unit 111 to instruct to capture an image of the hand R. When the operation of the operation unit 111 is input, the biological information detection unit 181 causes the first camera 61 to capture an image of the hand R of the user (step S73). The captured image of the first camera 61 is input to the biological information detection unit 181, and the biological information detection unit 181 performs image processing on the captured image to extract biological information (step S74).

When the input biometric information is input from the biometric information detection unit 181, the setting unit 183 selects the biometric information used for the matching process (step S75). The setting unit 183 selects one or more pieces of palm line information to be used for matching from the plurality of pieces of palm line information included in the biometric information, and the coordinates are closest to the starting point coordinates and the ending point coordinates of the selected palm line information. The palm line information is read out from the user information table and collation processing is performed (step S76).
The setting unit 183 collates the length ratio and/or the curvature of the palm line information extracted from the user information table with the length ratio and/or the curvature of the palm line information selected from the input biometric information. The setting unit 183 performs collation processing for all the selected palm line information (step S77). If the matching process determines that the matching ratio of the length ratio and/or the curvature is lower than the threshold value, that is, it is determined that they do not match (step S77/NO), the setting unit 183 determines that the authentication has not succeeded. The error message shown is output (step S78), and the process returns to step S73. Here, the setting unit 183 may additionally display the guide display on the silhouette image S displayed by the image display unit 20. The guide image is as described in step S17 of FIG.

  When the matching process determines that the matching ratio of the length ratio and/or the curvature is higher than the threshold value (step S77/YES), the control unit 140 starts authentication of the image information that is the second identification information. Yes (step S81). That is, the control unit 140 outputs a message to start recognition of image information.

  The setting unit 183 acquires one or a plurality of pieces of image information registered in the user information table 120D in association with the biometric information determined to match in step S76 (step S82). Thereby, the authentication can be executed efficiently.

  The biometric information detection unit 181 outputs the reference line SP to the display control unit 170 and causes the image display unit 20 to display it (step S83). The user moves the hand while visually recognizing the reference line SP, or moves so as to visually recognize the object, and instructs the imaging. When the biometric information detection unit 181 detects a user's operation, the biometric information detection unit 181 causes the first camera 61 and/or the second camera 62 to perform imaging (step S84). The captured image is input to the biometric information detection unit 181, and the biometric information detection unit 181 performs image processing on the captured image and extracts image information for authentication (step S85).

  In the user identification mode, the image information is input to the setting unit 183 from the biometric information detection unit 181. When the image information is input from the biometric information detection unit 181, the setting unit 183 matches or approximates the size, feature amount, text, code, etc. of the input image information among the registered image information acquired in step S82. The registered image information is collated (step S86).

The setting unit 183 reads, from the user information table 120D, the setting information corresponding to the registered image information that matches the image information by the matching process (step S87). The setting unit 183 calculates a correction coefficient for correcting the display position or size of the display data based on the read setting information. The setting unit 183 calculates a correction coefficient that corrects the display position or size of the display data based on at least one of the inter-eye distance, the vergence angle, the relative distance between the eye and the optical element, and the angle of the optical element. To do. After calculating the correction coefficient, the setting unit 183 outputs the calculated correction coefficient to the position detection unit 185.
The position detection unit 185 detects the position and size with respect to the display area of the image of the object detected by the object detection unit 184 from the captured image. When the position detection unit 185 detects the position and size with respect to the display area, the position detection unit 185 corrects the detected position and size based on the correction coefficient passed from the setting unit 183 (step S60). The information display control unit 186 determines the display position of the display data for the object detected by the object detection unit 184 based on the position and/or the size corrected by the position detection unit 185 based on the correction coefficient. After that, the information display control unit 186 outputs the display data to the display control unit 170 and causes the image display unit 20 to perform display.

As described in the sixth embodiment, the setting information is associated with the combination of the biometric information and the image information and registered in the user information table 120D, so that a plurality of setting information can be associated with one user. Can be attached and memorized. Then, when the setting information is selected and used, simple authentication is performed, so that security can be secured and the setting information can be easily used to realize an appropriate setting for the user.
With this configuration, the user can properly use the plurality of setting information stored in association with his/her biometric information. For example, after the authentication is performed using the palm line information, different setting information may be obtained depending on whether the first camera 61 captures an image of running shoes, an image of a building, or a case where a hand is moved along a name sign. Can be set by calling. Therefore, the setting information can be switched according to the use, the situation, the place, etc. of the head-mounted display device 100.
Since the setting information is specified in two steps using the biometric information and the image information, more settings can be stored and the setting information can be efficiently selected and used.

In the sixth embodiment, as shown in FIG. 19, the setting unit 183 first performs matching using the input biometric information (first identification information) input from the biometric information detection unit 181, and then performs biometric information. An example has been described in which image information (second identification information) input from the detection unit 181 is used to perform matching and read setting information.
Specifically, the input biometric information as the first identification information is input, and the setting unit 183 selects the input biometric information and performs the matching process (steps S72 to S77). After that, the image information as the second identification information is input, collation based on the image information is performed (steps S83 to S86), and the setting information is read from the user information table 120D (step S87).
The embodiment of the present invention is not limited to the configuration for performing the operation shown in FIG. For example, the setting unit 183 may perform the acquisition of the second identification information and the collation with the user information table 120D as in steps S83 to S86 before the operation of step S75. After that, the setting unit 183 may perform the operations of steps S72 to S77 to perform matching using the first identification information, and based on the result of this matching, read the setting information in step S87. Also in this case, the same effect as the configuration described in the sixth embodiment can be obtained.

The configurations of the above-described first to sixth embodiments can be applied not only individually, but also in combination.
In the first embodiment, an example has been described in which the authentication is performed using the biometric information as the identification information, the setting information stored in association with the biometric information is used, and the display is controlled. Further, in the second embodiment, the operation for asking the user for confirmation when the authentication using the identification information is successful has been described. In the third embodiment, the operation in which the display on the image display unit 20 is the one-eye display until the setting based on the setting information specified based on the identification information is performed has been described. Further, in the fourth embodiment, an example has been described in which, when collation of biometric information as identification information fails, new registration or the like is executed according to an input operation by the user. Further, in the fifth embodiment, the head-mounted display device 100 performs authentication using information other than biometric information, for example, a general image, as identification information, identifies stored setting information, and displays an image. The example of controlling the display of the unit 20 has been described. Further, in the sixth embodiment, an example in which information other than the biometric information described in the fifth embodiment and the biometric information described in the first to fourth embodiments are combined and used for authentication has been described. Further, in the fifth and sixth embodiments, as an example of the user information table stored in the storage unit 120, the configuration in which the setting information is registered in association with the combination of the biometric information and the image information is shown.
The present invention can be implemented by combining the configurations described in the embodiments. For example, as described in the fifth and sixth embodiments, in the configuration that uses information other than biometric information as the identification information, as described in the third embodiment, the display of the image display unit 20 based on the setting information is performed. Until one is controlled, one-eye display may be performed. Further, for example, in the configurations of the third, fifth, and sixth embodiments, as described in the second embodiment, an operation of asking the user for confirmation may be performed when the authentication is successful. Further, for example, in the configurations of the third, fifth, and sixth embodiments, as described in the fourth embodiment, when collation of identification information fails, new registration or the like is performed according to a user's input operation. You may execute. Further, the embodiment to which the present invention is applied can be a combination of the configurations of the embodiments other than those exemplified above, and the selection thereof is not limited.

The present invention is not limited to the configurations of the above-described embodiments, and can be implemented in various modes without departing from the scope of the invention.
Further, for example, instead of the image display unit 20, an image display unit of another system such as an image display unit worn like a hat may be adopted, and an image is displayed corresponding to the left eye of the user. It suffices to include a display unit that displays the image and a display unit that displays an image corresponding to the right eye of the user. Further, the display device of the present invention may be configured as, for example, a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head mount display built in a body protection tool such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile.

Further, in the above-described embodiment, the image display unit 20 and the control device 10 are separated and connected to each other via the connection unit 40. However, the control device 10 and the image display unit 20 are integrated. It is also possible to adopt a configuration of being attached to the head of the user.
Moreover, when the control device 10 and the image display unit 20 are connected by a longer cable, a laptop computer, a tablet computer, or a desktop computer may be used as the control device 10. Further, as the control device 10, a portable electronic device including a game machine, a mobile phone, a smartphone, a mobile media player, or other dedicated device may be used.

  Further, for example, the image display unit 20 may be configured to include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit as a configuration for generating image light. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), digital micromirror device, or the like can be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, causes the laser light to enter the user's eye, scans the retina, and forms an image on the retina. Alternatively, a configuration may be adopted in which the user visually recognizes the image. When a laser retinal projection type head mounted display is adopted, the “region where the image light can be emitted in the image light generating section” can be defined as an image region recognized by the eyes of the user.

  As an optical system that guides the image light to the user's eye, it is possible to adopt a configuration in which an optical member that transmits external light that is incident from the outside toward the device is transmitted and is incident on the user's eye together with the image light. Moreover, you may use the optical member located ahead of a user's eyes and overlapping a part or all of a user's visual field. Further, a scanning type optical system in which laser light or the like is scanned to form image light may be adopted. The optical member is not limited to the one that guides the image light inside, but may have only the function of refracting and/or reflecting the image light toward the eye of the user.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses a MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans the light emitted by the signal light forming unit, and an optical member that forms a virtual image by the light scanned by the scanning optical system. May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. The MEMS mirror scans the light to form a virtual image on the virtual image forming surface, and the user recognizes the virtual image by the eye. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above-described embodiment, or may use a half mirror surface. ..

  Further, the optical element of the present invention is not limited to the right light guide plate 261 and the left light guide plate 262 having the half mirrors 261A and 262A, and may be any optical component that allows image light to enter the eyes of the user. May use a diffraction grating, a prism, or a holographic display unit.

  Further, the display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by the image light together with the external light, and includes, for example, a configuration in which the image by the image light is visually recognized by an optical member that transmits the external light. Specifically, in addition to the configuration including an optical member that transmits external light in the above head mounted display, a transparent plane or curved surface fixedly or movably installed at a position distant from the user ( It is also applicable to a display device that projects image light onto glass or transparent plastic). As an example, there is a configuration of a display device that projects image light on a window glass of a vehicle and allows a user who is in the vehicle or a user outside the vehicle to visually recognize the scenery inside and outside the vehicle together with the image by the image light. Be done. Further, for example, by projecting image light on a transparent, semi-transparent, or colored transparent display surface that is fixedly installed, such as a window glass of a building, the image light is displayed to the user around the display surface together with the image by the image light. There is a configuration of a display device that allows a scene to be viewed through a surface.

  Further, at least some of the functional blocks shown in FIG. 4 may be realized by hardware, or may be realized by the cooperation of hardware and software. As shown in FIG. The configuration is not limited to the arrangement of independent hardware resources. The program executed by the control unit 140 may be stored in the storage unit 120 or a storage device in the control device 10, or the program stored in an external device may be acquired via the communication unit 117 or the interface 125. It may be configured to execute the above. Further, among the components formed in the control device 10, only the operation unit 111 may be formed as a single user interface (UI). Moreover, the configuration formed in the control device 10 may be duplicated in the image display unit 20. For example, the control unit 140 shown in FIG. 4 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. The functions performed by and may be separately configured.

In addition, as the setting information, the information of the pupil color is registered in the user information table 120A, and the intensity of the image light output from the display drive units 22 and 24 and the color arrangement are registered according to the user's pupil color. May be changed. In addition, the body type information of the user (standard body type, lean body type, obese type) is registered in the user information table 120A, and the color arrangement of the image light output from the display drive units 22 and 24 according to the body type of the user is registered. May be changed. For example, when the user is obese, and the image display unit 20 displays an image of food, the image color may be changed to a bluish color. Thereby, the user's appetite can be suppressed. Further, the right optical image display unit 26 and the left optical image display unit 28 may be provided with a shade whose transmittance can be changed, and the transmittance of the shade may be changed according to the color of the user's pupil.
In addition, as the setting information, physical characteristics such as the height, weight, and sex of the user are registered in the user information table 120A, and the display position/or the display size in the display area of the image display unit 20 is set to the user. You may correct based on the physical characteristics of.

  DESCRIPTION OF SYMBOLS 1... Communication system, 2... Display system, 5... Server, 10... Control device, 20... Image display part (display part), 21... Right holding part, 22... Right display drive part (2nd display part), 23 ... left holding section, 24... left display drive section (first display section), 25... interface, 26... right optical image display section, 28... left optical image display section, 61... camera, 63... microphone, 100, 100A , 100B... Head-mounted display device (display device), 117... Communication unit, 120... Storage unit, 125... Interface, 140... Control unit, 150... Operating system, 160... Image processing unit, 170... Display control unit, 181... Biological information detecting unit, 182... Setting information detecting unit, 183... Setting unit, 184... Target detecting unit, 185... Position detecting unit, 186... Information display control unit, 187... Voice processing unit, 201... Right backlight control 202... Left backlight control unit, 211... Right LCD control unit, 212... Left LCD control unit, 221... Right backlight, 222... Left backlight, 241,... Right LCD, 242... Left LCD, 251... Right projection Optical system, 252... Left projection optical system, 261... Right light guide plate (optical element), 262... Left light guide plate (optical element).

Claims (20)

Binocular display mounted on the user's head and displaying an image corresponding to the right and left eyes of the user, and displaying an image corresponding to either the right or left eye of the user A display device including a head-mounted display unit that can be switched between one-eye display and
Display position adjustment information for adjusting the display position of the display unit, eye distance of the user, and setting information including a convergence angle suitable for the user, in association with the identification information for identifying the user. A storage unit for storing,
An input unit for inputting the identification information,
Based on the setting information stored in the storage unit in association with the input identification information, the display position on the display unit is determined, and the display on the display unit is set and controlled according to the user. And a control unit,
The control unit switches the display of the display unit to a single-eye display, acquires the setting information stored in the storage unit in association with the identification information input by the input unit, and acquires the setting. Control the display by determining the display position of the display unit based on the display position adjustment information included in the information, then switch the display unit to binocular display,
A display device characterized by the above. Equipped with an imaging unit,
The input unit inputs the identification information based on a captured image of the imaging unit,
The control unit specifies, of the identification information stored in the storage unit, the identification information corresponding to the identification information input by the input unit, and the setting stored in association with the identified identification information. The display device according to claim 1, wherein display of the display unit is controlled based on information.   The display device according to claim 1, wherein the identification information includes a part of features included in the biometric information of the user.   The control unit identifies a part of the characteristics of the biometric information input as the identification information by the input unit and a part of the characteristics of the biometric information stored in the storage unit to identify the setting information. The display device according to claim 3, wherein   The display device according to claim 2, wherein the identification information includes image information regarding an image extracted from a captured image of the imaging unit.   The control unit specifies the setting information by collating the image information included in the identification information input by the input unit with the image information stored in the storage unit. The display device according to claim 5. The identification information includes first identification information including a part of characteristics included in the biometric information of the user, and second identification information including the image information,
7. The display device according to claim 5, wherein the storage unit stores the first identification information, the second identification information, and the setting information in association with each other.   The control unit specifies the setting information stored in the storage unit based on the first identification information and the second identification information included in the identification information input by the input unit. The display device according to claim 7.   The control unit specifies the setting information stored in the storage unit in association with a combination of the first identification information and the second identification information included in the identification information input by the input unit. 9. The display device according to claim 8, wherein: The control unit, based on one of the first identification information and the second identification information included in the identification information input by the input unit, a plurality of the setting information stored in the storage unit. Select the setting information,
Among the selected setting information, the setting information corresponding to the other of the first identification information and the second identification information included in the identification information input by the input unit is specified. The display device according to any one of claims 7 to 9.   The image information is information that can be extracted or detected from the captured image, and is non-living body specific information that does not include information that enables individual identification of a living body by itself. The display device according to any one of 1.   The image information is external scene information obtained by extracting information including a shape of a building from the captured image including a building, identification object information extracted from the captured image obtained by imaging an inanimate object for identification, and moving. The display device according to claim 11, wherein the display device includes any one of trajectory information regarding a trajectory of the object extracted from the plurality of captured images of the object. The setting information, the angular relative position, the optical element of the previous SL user's eye and the display unit comprises an optical element, the user's dominant eye, color vision color information about the user, the display unit brightness of the image to be displayed, and display device according to any one of claims 1 to 12, wherein the user comprises a language to be used, it is characterized. The display unit displays an image so that the image can be visually recognized together with the outside scene through the outside scene.
14. The display device according to claim 1, wherein the control unit changes at least one of a display position and a display size of the image in a plurality of stages corresponding to the setting information. The display unit is an optical element which transmits image light and makes image light incident on the eyes of the user so that the image light can be visually recognized together with the image.
An object detection unit for detecting an object in the direction of the user's line of sight,
A position detection unit that detects the position of the target object with respect to the display area of the display unit,
The control unit changes the display position of the image by the optical element according to the position of the object detected by the position detection unit and the positional relationship between the optical element and the position of the pupil of the user. 15. The display device according to any one of claims 1 to 14, wherein: The setting information includes information about language setting,
16. The control unit causes the display unit to display characters in a language corresponding to the setting information when displaying content including characters on the display unit. The display device according to item 1. Equipped with a communication unit,
The control unit associates the setting information and the identification information stored in the storage unit with each other, and transmits them to an external device by the communication unit,
Receiving the setting information and the identification information by the communication unit, and storing the received setting information and the identification information in the storage unit in association with each other,
The display device according to claim 1, wherein the display device is a display device. Binocular display mounted on the user's head and displaying an image corresponding to the right and left eyes of the user, and displaying an image corresponding to either the right or left eye of the user A plurality of display devices each having a head-mounted display unit that can be executed by switching between single-eye display and
The display device,
Display position adjustment information for adjusting the display position of the display unit, eye distance of the user, and setting information including a convergence angle suitable for the user, in association with the identification information for identifying the user. A storage unit for storing,
An input unit for inputting the identification information,
Based on the setting information stored in the storage unit in association with the input identification information, the display position on the display unit is determined, and the display on the display unit is set and controlled according to the user. A control unit,
A communication unit that communicates with another display device,
The control unit associates the setting information and the identification information stored in the storage unit with each other, and transmits the communication information by the communication unit,
The setting information and the identification information are received by the communication unit, the received setting information and the identification information are stored in the storage unit in association with each other,
After switching the display of the display unit to a single-eye display, the setting information stored in the storage unit in association with the identification information input by the input unit is acquired, and the setting information is included in the acquired setting information. Controlling the display by determining the display position of the display unit based on the display position adjustment information, and then switching the display unit to binocular display,
Display system characterized by. Binocular display mounted on the user's head and displaying an image corresponding to the right and left eyes of the user, and displaying an image corresponding to either the right or left eye of the user A method for controlling a display device including a head-mounted display unit capable of executing switching between one-eye display and
Entering identification information,
Display position adjustment information for adjusting the display position of the display unit, eye distance of the user, and setting information including a convergence angle suitable for the user, in association with the identification information for identifying the user. Control for referring to a storage unit to be stored and setting and controlling the display of the display unit according to the user based on the setting information stored in the storage unit in association with the input identification information. Including steps and
In the control step, after switching the display of the display unit to a single-eye display, the setting information stored in the storage unit in association with the input identification information is acquired, and included in the acquired setting information. The display position of the display unit is determined based on the display position adjustment information to be displayed and the display is controlled, and then the display unit is switched to the binocular display,
A method for controlling a display device, comprising: Binocular display mounted on the user's head and displaying an image corresponding to the right and left eyes of the user, and displaying an image corresponding to either the right or left eye of the user A program that can be executed by a computer that controls a display device that includes a head-mounted display unit that can be executed by switching between one-eye display and
Computer
An input section for inputting identification information,
Display position adjustment information for adjusting the display position of the display unit, eye distance of the user, and setting information including a convergence angle suitable for the user, in association with the identification information for identifying the user. Referring to a storage unit to be stored, the display position on the display unit is determined based on the setting information stored in the storage unit in association with the input identification information, and the display on the display unit is used. A control unit that sets and controls according to a person, and switches the display of the display unit to a single-eye display, and then stores the storage unit in association with the identification information input by the input unit. A control unit that acquires setting information, controls the display by determining the display position of the display unit based on the display position adjustment information included in the acquired setting information, and then switches the display unit to binocular display. When,
And a program to make it work.

Images