JP6488786B2 - Head-mounted display device, head-mounted display device control method, and computer program - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system or a light guide plate This causes the user to visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type.

  Patent Document 1 describes a head-mounted display device that divides a user's field of view into two or more regions and displays a virtual image corresponding to each divided region. Further, Patent Document 2 describes a multimodal interface that displays an image indicating guide information for assisting input in the vicinity of a detected user's line-of-sight position. Patent Document 3 discloses a head-mounted display device that displays an image visually recognized by the user at the same position as the distance to the user's point of gaze that changes in accordance with the movement state of the user such as walking. Is described.

Special table 2014-526157 gazette Japanese Patent Laid-Open No. 11-54778 JP 2014-225727 A

  However, in the technique described in Patent Document 1, when there are many virtual images displayed in the divided areas, the processing load for displaying the virtual images is heavy, and there is a risk of delaying the display of the virtual images. was there. Further, even in the techniques described in Patent Documents 2 and 3, when the capacity of an image to be displayed is large, the processing load for displaying the image may be large. In addition, in the technology of the conventional head-mounted display device, it has been desired to reduce the size, reduce the cost, facilitate the manufacturing, improve the usability, and the like.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.
According to one aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes an image display unit capable of displaying an image in a display area, a line-of-sight direction specifying unit for specifying a user's line-of-sight direction, and the display area according to the specified line-of-sight direction. An image change unit that changes a display image to be displayed, and the image change unit includes the specified line-of-sight direction when the display image is an image that is visually recognized by a user as a three-dimensional image. In the third visual field range, the display image is displayed as a three-dimensional image in the display region, and is displayed in a fourth visual field range that is larger than the third visual field range and does not include the third visual field range. Then, the display image is displayed in the display area so as to be visually recognized by the user as a two-dimensional image.
According to the head-mounted display device of this embodiment, the image display mode is changed according to the position that the user is viewing, without reducing the visibility of the display image of the user. It is possible to reduce the burden of processing for displaying. In addition, since the two-dimensional image is displayed in the display area in the field of view where it is difficult for the user to recognize the distance to the display image, the image is displayed without reducing the ease of recognition by the user who visually recognizes the corresponding image. The processing speed can be further improved.
According to another embodiment of the present invention, a head-mounted display device is provided. The head-mounted display device includes an image display unit capable of displaying an image in a display area, a line-of-sight direction specifying unit for specifying a user's line-of-sight direction, and the display area according to the specified line-of-sight direction. An image change unit that changes a display image to be displayed, and the image change unit includes a fifth line that includes the identified line-of-sight direction when the display image is an image including information of a plurality of colors. In the visual field range, the display image is displayed on the display area as an image including information of a plurality of colors, and is a sixth visual field that is larger than the fifth visual field range and does not include the fifth visual field range. In the range, the display image is displayed in the display area using a single color shading.
According to the head-mounted display device of this embodiment, the image display mode is changed according to the position that the user is viewing, without reducing the visibility of the display image of the user. It is possible to reduce the burden of processing for displaying. In addition, the processing speed for displaying an image can be further improved without reducing the ease of recognition by a user who visually recognizes the image.
In addition, the present invention can be realized as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes an image display unit capable of displaying an image in a display area; a line-of-sight direction specifying unit that specifies a user's line-of-sight direction; and the display area according to the specified line-of-sight direction. An image changing unit that changes a displayed display image. According to the head-mounted display device of this embodiment, the image display mode is changed according to the position that the user is viewing, without reducing the visibility of the display image of the user. It is possible to reduce the burden of processing for displaying.

(2) In the head-mounted display device according to the above aspect, the image change unit may change the display image in accordance with a visual characteristic of a user that changes in accordance with the identified line-of-sight direction. According to the head-mounted display device of this aspect, it is possible to improve the processing speed for displaying an image without impairing visual characteristics such as recognition of a user who visually recognizes the image.

(3) In the head-mounted display device according to the above aspect, when the display image is displayed on a part of the display area, the image change unit, depending on the position of the display image and the line-of-sight direction, The display image may be changed. According to the head-mounted display device of this aspect, it is possible to improve the processing speed for displaying an image without reducing the ease of recognition by a user who visually recognizes the image.

(4) In the head-mounted display device of the above aspect, the image display unit can transmit an outside scene; the head-mounted display device further includes an outside scene imaging unit that images the outside scene; The image changing unit may detect the specific object from the captured image of the outside scene and set the position of the display image in association with the detected position of the specific object. According to this type of head-mounted display device, the correspondence between the specific object and the displayed image can be more easily recognized by the user, and the usability of the user is improved.

(5) In the head-mounted display device of the above aspect, the line-of-sight direction specifying unit includes a display unit direction detection unit that detects the direction of the image display unit, and is based on the detected direction of the image display unit. The line-of-sight direction may be specified. According to this form of the head-mounted display device, for example, the user's line-of-sight direction can be specified easily and inexpensively without using a device that images the user's eyes.

(6) In the head-mounted display device according to the above aspect, the line-of-sight direction specifying unit includes an eye imaging unit that images a user's eyes, and the line of sight is based on the captured image of the user's eyes. The direction may be specified. According to this form of the head-mounted display device, the user's line-of-sight direction can be accurately identified.

(7) In the head-mounted display device according to the above aspect, when the display image includes a character image, the image changing unit displays the character image in the first visual field range including the specified line-of-sight direction. The display image is displayed in the display area as the display image, and the character image is used as the display image in the second visual field range that is larger than the first visual field range and does not include the first visual field range. Does not have to be displayed. According to this type of head-mounted display device, an image that does not include a character image is displayed in the display area in a visual field range in which it is difficult for the user to recognize the character. The speed of processing for displaying an image can be improved without further reducing the image quality.

(8) In the head-mounted display device according to the above aspect, the image changing unit includes a third line of sight that includes the specified line-of-sight direction when the display image is an image that is visually recognized by a user as a three-dimensional image. In the visual field range, the display image is displayed as a three-dimensional image in the display region, and is displayed in a fourth visual field range that is larger than the third visual field range and does not include the third visual field range. The display image may be displayed in the display area so as to be visually recognized by the user as a two-dimensional image. According to this type of head-mounted display device, since the two-dimensional image is displayed in the display area in the visual field range in which it is difficult for the user to recognize the distance to the display image, the user who recognizes the corresponding image is recognized. The speed of processing for displaying an image can be further improved without reducing the ease.

(9) In the head-mounted display device according to the above aspect, when the display image is an image including information of a plurality of colors, the image change unit includes a fifth field of view including the specified line-of-sight direction. In the range, the display image is displayed in the display area as an image including information of a plurality of colors. In the sixth field range that is larger than the fifth field range and does not include the fifth field range. The display image may be displayed in the display area using a single color shade. According to the head-mounted display device of this aspect, it is possible to further improve the processing speed for displaying an image without reducing the ease of recognition by a user who visually recognizes the image.

(10) In the head-mounted display device according to the above aspect, the image changing unit may display only the outline of the display image in the display area in the sixth visual field range. According to this form of the head-mounted display device, only the outer frame of the image is displayed in a range where it is difficult for the user to recognize the color information, thereby reducing the ease of recognition by the user viewing the corresponding image. Without this, the processing speed for displaying an image can be further improved.

(11) In the head-mounted display device according to the above aspect, the image changing unit is a range that is larger than a seventh visual field range including the identified line-of-sight direction and does not include the seventh visual field range. In the visual field range of 8, the display image may be displayed in the display region as a simplified image that is simpler than the image displayed in the seventh visual field range. According to the head-mounted display device of this form, only a simplified image indicating that there is an image is displayed in the display area in the visual field range where the user cannot recognize the detailed contents of the display image. The speed of the process for displaying the corresponding image can be further improved without reducing the ease of recognition by the user who visually recognizes the corresponding image.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including some or all of the three elements of the image display unit, the line-of-sight direction specifying unit, and the image changing unit. That is, this apparatus may or may not have an image display unit. In addition, the device may or may not have a line-of-sight direction specifying unit. Further, the apparatus may or may not have an image changing unit. The image display unit may be capable of displaying an image in a display area, for example. The line-of-sight direction specifying unit may specify the line-of-sight direction of the user, for example. For example, the image changing unit may change the display image displayed in the display area in accordance with the identified line-of-sight direction. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, it is possible to solve at least one of various problems such as improvement and simplification of the operability of the device, integration of the device, and improvement of convenience of the user who uses the device. it can. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a display device, a head-mounted display device, a control method for the display device, a control system, a head-mounted display system, a display device, a computer program for realizing the functions of the control system and the display device, and the computer program It can be realized in the form of a recorded recording medium, a data signal including the computer program and embodied in a carrier wave.

It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus (HMD) in this embodiment. It is a block diagram which shows the structure of HMD in this embodiment functionally. It is explanatory drawing of a relationship with a gaze point and a user's gaze direction. It is the schematic of a horizontal visual field and a vertical visual field centering on a gazing point. It is explanatory drawing which shows the degree of human recognition for every visual field range in a horizontal visual field. It is explanatory drawing which shows the degree of human recognition for every visual field range in a vertical visual field. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is a flowchart which shows the flow of an image display process. It is explanatory drawing which shows the relationship between the image display position of a corresponding image, and the visual field range distinguished. It is explanatory drawing which shows an example of a user's visual field when a corresponding image is displayed. It is explanatory drawing which shows an example of a user's visual field when the corresponding image after the imaging range of a camera changes is displayed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the corresponding image after a user's gaze direction changes is displayed. It is explanatory drawing which shows an example of the visual field visually recognized by the user in the modification of this embodiment. It is explanatory drawing which shows an example of the visual field visually recognized by the user in a modification. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when a corresponding image belongs to a visual field range. It is explanatory drawing which shows the external appearance structure of HMD in a modification.

A. Embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (HMD 100) according to the present embodiment. The HMD 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The HMD 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time visually recognize an outside scene. In this specification, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a first optical display unit. The camera 61, the 2nd camera 62, the right eye imaging camera 37, and the left eye imaging camera 38 are included. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The first camera 61 and the second camera 62 capture an outside scene. Each of the first camera 61 and the second camera 62 is disposed at a different position on the image display unit 20. The first camera 61 is arranged at the end ER, which is the other end of the right optical image display unit 26, in the direction of the user's head when the image display unit 20 is worn by the user. Similarly, the second camera 62 is arranged at the end EL, which is the other end of the left optical image display unit 28, in the direction of the user's head when the image display unit 20 is worn by the user. ing. In the present embodiment, each of the first camera 61 and the second camera 62 has a position where one end of the right optical image display unit 26 and one end of the left optical image display unit 28 that are the center of the image display unit 20 are connected. It is arranged symmetrically with respect to the passing center line. Hereinafter, the first camera 61 and the second camera 62 are also simply referred to as cameras 61 and 62. In the present embodiment, since the location where the first camera 61 is arranged in the image display unit 20 and the location where the second camera 62 is arranged in the image display unit 20 are different, the imaging range of the first camera 61 is different. And the distance to the object imaged in both the imaging range of the second camera 62 can be calculated using trigonometry. The cameras 61 and 62 correspond to the outside scene imaging unit in the claims.

  The right eye imaging camera 37 is a camera that images the right eye of the user wearing the image display unit 20. Similarly, the left eye imaging camera 38 is a camera that images the left eye of the user wearing the image display unit 20. For the right eye image of the user captured by the right eye imaging camera 37 and the left eye image of the user captured by the left eye imaging camera 38, the direction specifying unit 166 described later specifies the direction of the user's line of sight. Used for processing. Note that the right-eye imaging camera 37 and the left-eye imaging camera 38 are monocular cameras, but may be stereo cameras. Hereinafter, the right eye imaging camera 37 and the left eye imaging camera 38 are also collectively referred to as eye imaging cameras 37 and 38. The eye imaging cameras 37 and 38 and a direction specifying unit 166 described later correspond to a line-of-sight direction specifying unit in the claims. The eye imaging cameras 37 and 38 correspond to the eye imaging unit in the claims.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the HMD 100 according to the light emission state. The operation state of the HMD 100 includes, for example, power ON / OFF. For example, an LED is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the HMD 100 in the present embodiment. As illustrated in FIG. 2, the control unit 10 includes a storage unit 120, a power supply 130, a wireless communication unit 132, an operation unit 135, a CPU 140, an interface 180, a transmission unit 51 (Tx51), and a transmission unit 52 ( Tx52). The operation unit 135 receives an operation by the user and includes an enter key 11, a display switch key 13, a track pad 14, a luminance switch key 15, a direction key 16, a menu key 17, and a power switch 18. The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with other devices such as a content server, a television, and a personal computer in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth.

  The storage unit 120 includes a ROM that stores computer programs, and a RAM that is used when the CPU 140 executes writing and reading of various computer programs.

  The CPU 140 reads a computer program stored in the ROM of the storage unit 120 and writes and reads the computer program in the RAM of the storage unit 120, whereby the operating system 150 (OS 150), the display control unit 190, the audio processing unit 170, It functions as an image processing unit 160, an image setting unit 165, a distance measuring unit 168, and a direction specifying unit 166.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light. Note that generating image light is also referred to as “displaying an image”.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data (RGB data) of the target image in the DRAM in the storage unit 120 for each frame. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. . Note that the image data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( An audio signal is supplied to (not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example.

  The direction specifying unit 166 specifies the user's line-of-sight direction by analyzing the image of the user's right eye RE and the image of the left eye LE captured by the eye imaging cameras 37 and 38. Various known techniques can be used as a method for identifying the user's line-of-sight direction based on the captured image of the right eye RE and left eye LE of the user. For example, the direction specifying unit 166 performs pattern matching on the captured eye image with reference to an image database in which the captured image is associated with the user's line-of-sight direction stored in the storage unit 120 in advance. The user's line-of-sight direction can be identified using statistical identification methods.

  The distance measurement unit 168 uses the trigonometry for the object included in the captured image of the first camera 61 and the object included in the captured image of the second camera 62, thereby capturing the images of the cameras 61 and 62. The distance to the image of the object contained in the image is measured. In the present embodiment, the distance to the object is measured by the first camera 61 and the second camera 62, which are stereo cameras having different locations, but in other embodiments, the TOF (Time Of The distance to the object may be measured by different measuring means such as a Flight sensor.

  The image setting unit 165 performs various settings for an image (display image) to be displayed on the image display unit 20. For example, the image setting unit 165 sets the display position of the display image, the size of the display image, the brightness of the display image, or the like, or allows the user to visually recognize the display image as a three-dimensional image three-dimensionally (3D). The right-eye image data and the left-eye image data are set so as to form binocular parallax (also simply referred to as “parallax”). Further, the image setting unit 165 detects a preset detection target image from the captured image by using pattern matching or the like on the captured image.

  In the present embodiment, the image setting unit 165 displays different images depending on the relationship between the identified line-of-sight direction of the user and the position of the image area where the image is displayed, even if the images are based on the same image data. It is displayed on the image display unit 20. For example, the image setting unit 165 has RGB color information in a range of a predetermined visual field centered on a user's gazing point based on three-dimensional image data having RGB (Red Green Blue) color information, It is displayed on the image display unit 20 as a three-dimensional image using the measured distance information. On the other hand, the image setting unit 165 causes the image display unit 20 to display a two-dimensional image having only white and black color information outside the predetermined field of view based on the three-dimensional image data. The image setting unit 165 reads, via the ROM, a look-up table that determines the correspondence of how different images are displayed even with the same image data regarding the relationship between the line-of-sight direction and the position of the image area. Thus, the image to be displayed on the image display unit 20 is determined.

  FIG. 3 is an explanatory diagram of a relationship between the gazing point POR and the user's line-of-sight direction ED. FIG. 3 shows the line-of-sight direction ED when the right eye RE and the left eye LE of the user are looking at the gazing point POR. In the present embodiment, the direction from the left and right eyeball center MP, which is an intermediate point between the right eye RE and the left eye LE, to the gazing point POR is defined as the line-of-sight direction ED. The definition of the user's line-of-sight direction ED can be variously modified. For example, the direction from the dominant eye to the gazing point POR may be defined as the line-of-sight direction ED.

  A human can visually recognize an object existing in a visual field that is a certain range around the point of interest POR. In general, a human being is present at the same distance as the distance to the gazing point POR, and the object existing near the gazing point POR is more clearly visible. There are various forms of visual characteristics in which human visibility changes depending on the range in which the human is present. For example, the outline of the target can be clearly seen up to a certain visual field range, but there is a visual characteristic that the target outline cannot be grasped in other visual field ranges although it can be recognized that the target exists. The same can be said for a display image serving as a substitute for an object. In other words, the ability of a human to recognize an object included in the field of view decreases as the distance from the point of interest POR increases. Therefore, in the present embodiment, the image setting unit 165 may reduce the resolution of the image even if the image displayed at a position away from the user's line-of-sight direction ED is based on the same image data. By the means, processing for displaying the image on the image display unit 20 is reduced, and the image display unit 20 displays the image. The image setting unit 165 corresponds to the image changing unit in the claims.

  A region within about 5 degrees centered on the gazing point POR is called a discrimination visual field having excellent visual functions such as visual acuity. An area within about 30 degrees in the horizontal direction around the point of interest POR and within about 20 degrees in the vertical direction around the viewpoint is called an effective field of view that can instantly recognize the object only by eye movement. Yes. In addition, an area of about 60 to 90 degrees in the horizontal direction around the point of interest POR and about 45 to 70 degrees in the vertical direction around the viewpoint is an eye movement or head movement in which the user moves the eyeball. This is called a stable gaze that can be recognized without difficulty.

FIG. 4 is a schematic diagram of the horizontal visual field VRw and the vertical visual field VRh centering on the gazing point POR. FIG. 4 shows the relationship among the discrimination visual field, the effective visual field, and the stable visual field in the horizontal direction and the vertical direction. FIG. 4 shows a visual field VR that can be viewed around the gazing point POR when the user is looking at the gazing point POR, a horizontal visual field VRw for explaining the visual field VR divided into horizontal components, and a visual field VR. A vertical visual field VRh for explaining the above by dividing it into vertical components is shown. In FIG. 4, the discrimination visual field, the effective visual field, and the stable focus visual field are expressed as the following relationship (1). The angle in parentheses in the relationship (1) is an angle on one side when the gazing point POR is the center.
Discrimination field of view: W0, H0 (up to about 5 degrees)
Effective field of view: W1 (up to about 30 degrees), H1 (up to about 20 degrees) (1)
Stable field of view: W2 (up to about 60 (90) degrees), H2 (up to about 45 (70) degrees)
As shown in FIG. 4, in the vertical direction of the visual field VR, the stable focus visual field on the lower side is wider than the upper side. This indicates that it is easier for a human to visually recognize the lower side than the upper side along the vertical direction.

FIG. 5 is an explanatory diagram showing the degree of human recognition for each field of view in the horizontal field of view VRw. FIG. 5 shows visual field ranges VR1 to VR6 that are divided into six visual fields in the visual axis direction ED of the user US, and the following relationship (2) is established. The angle in parentheses in the relationship (2) is an angle on one side when the gazing point POR is the center.
VR1: The range that can be regarded as the point of interest POR (up to about 1 degree)
VR2: Range in which characters can be identified (up to about 10 degrees)
VR3: Range in which eye movement can be easily performed (up to about 15 degrees) (2)
VR4: Range in which the shape can be identified (up to about 30 degrees)
VR5: Range in which color information can be identified (up to about 60 degrees)
VR6: limit of visual field (up to about 94 degrees)
As described above, as shown in FIG. 5, the information that can be recognized by the user US varies depending on the field of view.

  FIG. 6 is an explanatory diagram showing the degree of human recognition for each visual field range in the vertical visual field VRh. FIG. 6 shows a distinguished user US field of view for a vertical field of view VRh as an alternative to the horizontal field of view VRw of FIG. For this reason, the description of the visual field range that is the distinguished visual field range is omitted.

  An interface 180 illustrated in FIG. 2 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a first A camera 61 and a second camera 62 are provided.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL222) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 7 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 2, the backlight system is adopted in the present embodiment, but the image light may be emitted using a front light system or a reflection system.

A-2. Image display processing:
FIG. 8 is a flowchart showing the flow of the image display process. In the image display process, when the CPU 140 displays the corresponding image associated with the detection target detected from the captured images of the cameras 61 and 62 on the image display unit 20, the user's line-of-sight direction ED is the center. In this process, the display image is changed and displayed on the image display unit 20 based on the distinguished visual field range and the display position of the image.

  In the image display process, first, the image setting unit 165 images an outside scene using the cameras 61 and 62 (step S11). The image setting unit 165 detects the detection target associated with the corresponding image displayed on the image display unit 20 from the captured images of the captured cameras 61 and 62 by pattern matching or the like (step S13). If the detection target is not detected from the captured image (step S13: NO), the image setting unit 165 continues to wait for detection of the detection target from the captured image (step S13).

  In the process of step S13, when the detection unit is detected from the captured image (step S13: YES), the image setting unit 165 causes the image display unit 20 to display a corresponding image associated with the detection target in advance. The image display position is determined (step S15). The image display position of the corresponding image can be variously determined. For example, the fixed position in the image display maximum region PN where the image display unit 20 can display an image regardless of the position of the detection target in the captured image. Or may be displayed near the position of a feature point set as a detection target. The image display maximum area PN corresponds to a display area in claims.

  When the image display position of the corresponding image is determined (step S15), the CPU 140 displays the gazing point POR and the line-of-sight direction ED of the user wearing the image display unit 20 on the head of the eye imaging cameras 37 and 38. It specifies using a captured image (step S17). The CPU 140 determines the distance to the detection target measured by the distance measurement unit 168 using the captured images of the cameras 61 and 62, and the user's line of sight specified by the direction specifying unit 166 using the captured images of the eye imaging cameras 37 and 38. The direction ED is specified.

  The image setting unit 165 specifies which of the plurality of visual field ranges set based on the specified line-of-sight direction ED the image display position of the corresponding image in the maximum image display area PN belongs ( Step S19). For example, as shown in FIG. 5, the image setting unit 165 distinguishes a plurality of visual field ranges centered on the identified line-of-sight direction ED, and identifies which visual field range the image display position belongs to.

  FIG. 9 is an explanatory diagram showing the relationship between the image display position of the corresponding image IM1 and the distinguished visual field range. FIG. 9 shows an example of the visual field VR visually recognized by the user. As shown in FIG. 9, the user can visually recognize the outside scene SC transmitted through the optical image display units 26 and 28. The outside scene SC includes a “sky tree” as one of landmarks as the detection target TW. FIG. 9 shows a user's gazing point POR specified by the CPU 140 and visual field ranges VR2, VR3, VR4, and VR5 set in a predetermined range around the gazing point POR. The visual field range VR3 is a range that is included in the visual field range VR3 and not included in the visual field range VR2. Similarly, the visual field range VR4 refers to a range that is included in the visual field range VR4 and is not included in the visual field ranges VR2 and VR3, and the visual field range VR5 is included in the visual field range VR5 and is also included in the visual field range VR2. , VR3, and VR4. In FIG. 9, the broken line image display maximum area PN, the star indicating the gazing point POR, and the alternate long and short dash line indicating each visual field range are shown for convenience, but these are not visually recognized by the user. . In this embodiment, since the captured image corresponding to the visual field visually recognized by the user is created by the captured images of the cameras 61 and 62, the created captured image matches the visual field VR of the user. In other embodiments, the captured image and the visual field VR of the user may be different.

  When the process of step S19 in FIG. 8 is performed, the image setting unit 165 sets the display mode of the corresponding image to be displayed in the image display maximum region PN based on the image data of the corresponding image associated with the detection target TW ( Step S23). In the present embodiment, the image setting unit 165 changes the setting of the display image mode corresponding to the position where the corresponding image is displayed in the maximum image display area PN, even for display images based on the same image data. . When the display mode of the corresponding image IM1 is set, the image setting unit 165 displays the corresponding image IM1 in which the display mode is set in the image display maximum region PN (Step S23).

  FIG. 10 is an explanatory diagram illustrating an example of the visual field VR of the user when the corresponding image IM1 is displayed. As shown in FIG. 10, in the user's visual field VR, an external scene SC that can be visually recognized by the user through the optical image display units 26 and 28, and a corresponding image IM1 displayed in the image display maximum area PN, It is included. The corresponding image IM1 is an image displayed in the maximum image display area PN in association with the detection target TW detected from the captured image. The corresponding image IM1 is set in advance so as to be displayed near the detection target TW so as not to overlap the detection target TW. In particular, when the identified user's point of interest POR overlaps the detection target TW, the corresponding image IM1 is displayed in accordance with the position of the identified point of interest POR. The detection target TW corresponds to the specific target in the claims.

The image setting unit 165 specifies the positional relationship between the image display position for displaying the corresponding image IM1 associated with the position of the specified gazing point POR and each visual field range. When the image setting unit 165 identifies that at least a part of the corresponding image IM1 is included in the visual field range VR2 closest to the gazing point POR, the image setting unit 165 sets the corresponding image as a display mode of the image to be displayed based on the image data of the corresponding image IM1. The image data of IM1 is set as the display mode showing the most detail. The image setting unit 165 displays an image satisfying all of the following contents (A) to (C) in the image display maximum area PN as the corresponding image IM1 of the set display mode.
(A) Character image (B) Three-dimensional image (including distance information)
(C) Image including RGB color information Corresponding image IM1 of the set display mode includes a character image, and the maximum image display as a three-dimensional image that is visually recognized at the same distance as the distance to the detection target TW. It is displayed in the area PN. Further, “here”, which is underlined in the character image included in the corresponding image IM1, can be selected by moving a cursor displayed in the image display maximum area PN separately. In order to indicate this, the color is displayed in a color different from that of other character images (for example, blue (BLUE)). When “here” included in the corresponding image IM1 is selected by a predetermined operation received by the operation unit 135, the CPU 140 is connected to the homepage (HP) of the detection target TW via the wireless communication unit 132. . In the present embodiment, when the image display position of the corresponding image IM1 is included in the plurality of visual field ranges, the image setting unit 165 displays when the entire corresponding image IM1 is included in the innermost visual field range. In this manner, the corresponding image IM1 is displayed in the image display maximum area PN. In other embodiments, the field of view range in which the display position of the corresponding image IM1 is included can be variously modified. For example, a field of view range having the largest number of display areas included in the image is displayed. You may specify as a visual field range for setting an aspect. The visual field range VR2 corresponds to the first visual field range in the claims, and the visual field ranges other than the visual field range VR2 correspond to the second visual field range in the claims.

  In step S23 of FIG. 8, when the corresponding image IM1 is displayed in the maximum image display area PN, the CPU 140 monitors a change in the imaging range of the cameras 61 and 62 (step S25). CPU140 repeats the process after step S13, when the change of the imaging range of the cameras 61 and 62 is detected (step S25: YES). Note that the CPU 140 may treat the change in the imaging range of the cameras 61 and 62 as a change in the captured image if there is a change below a predetermined threshold using a predetermined threshold.

  FIG. 11 is an explanatory diagram illustrating an example of the visual field VR of the user when the corresponding image IM2 after the imaging ranges of the cameras 61 and 62 are changed is displayed. FIG. 11 shows the visual field VR when the user is present at a position away from the detection target TW and the user's point of interest POR is the moon MN included in the outside scene SC, as compared with FIG. ing. That is, in FIG. 11, compared with FIG. 10, the user's point of interest POR also changes. In the example illustrated in FIG. 11, the detection target TW detected from the captured images of the cameras 61 and 62 is included in a range further outside the visual field range VR5. In this case, the image setting unit 165 simplifies only the position of the detection target TW as the display mode of the image to be displayed in the image display maximum area PN based on the image data of the same corresponding image IM2 as the corresponding image IM1. It is set as a display mode that does not have. As illustrated in FIG. 11, the image setting unit 165 displays the corresponding image IM2 that is a round image overlapping the position of the detection target TW as the simplified corresponding image IM2 based on the image data of the corresponding image IM2. The area PN is displayed at a position at infinity. In the present embodiment, the image setting unit 165 displays an image only representing the position of the detection target TW in the image display maximum region PN as a corresponding image of the detection target TW included in the visual field range VR5. The range outside the visual field range VR5 corresponds to the eighth visual field range in the claims, and the visual field range inside the visual field range VR5 including the visual field range VR5 corresponds to the seventh visual field range in the claims. .

  When the corresponding image IM2 is displayed in step S23 of FIG. 8, the CPU 140 monitors a change in the imaging range of the cameras 61 and 62 (step S25). When determining that there is no change in the captured images of the cameras 61 and 62 (step S25: NO), the CPU 140 monitors the change in the user's line-of-sight direction ED (step S27). When the CPU 140 determines that the user's line-of-sight direction ED has changed (step S27: YES), the CPU 140 repeats the processing after step S17.

  FIG. 12 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when the corresponding image IM3 after the user's line-of-sight direction ED is changed is displayed. FIG. 12 shows the visual field VR when the user's gazing point POR is changed as compared with FIG. 11. In the example shown in FIG. 12, the user's point of interest POR is the building BD1 included in the outside scene SC, and the position of the detection target TW in the captured images of the cameras 61 and 62 is included in the visual field range VR4 and the visual field range VR5. ing. Therefore, the image setting unit 165 determines that the detection target TW belongs to the visual field range VR4, and displays an image based on the image data of the corresponding image IM3 that is the same image data as the corresponding image IM1 associated with the detection target TW. A display mode of an image to be displayed in the maximum area PN is set. The image setting unit 165 is linked to the homepage (HP) based on the set display mode, and the corresponding image IM3 as a selectable button is infinite in the same color as “here” of the corresponding image IM1 in FIG. As an image visually recognized in the distance, the image is displayed in the maximum image display area PN. In other words, the image setting unit 165 does not include (A) the content of the character image and (B) the content of the three-dimensional image including the distance information with respect to the corresponding image IM1 illustrated in FIG. ) The corresponding image IM3 is displayed in the image display maximum area PN as an image including RGB color information. The visual field range VR4 corresponds to the fourth visual field range in the claims, and the visual field range inside the visual field range VR4 corresponds to the third visual field range in the claims.

  In the process of step S23 of FIG. 8, when the corresponding image IM3 is displayed in the maximum image display area PN, the CPU 140 monitors a change in the imaging range of the cameras 61 and 62 (step S25). When determining that there is no change in the imaging range of the cameras 61 and 62 (step S25: NO), the CPU 140 monitors the change in the user's line-of-sight direction ED (step S27). When determining that there is no change in the line-of-sight direction ED of the user US (step S27: NO), the CPU 140 monitors reception of a predetermined operation for ending the image display process on the operation unit 135 (step S29). When the predetermined operation for ending the image display process is not accepted (step S29: NO), the CPU 140 repeats the processes after step S25. In the process of step S29, when a predetermined operation for ending the image display process is accepted (step S29: YES), the CPU 140 ends the image display process.

  Further, in the HMD 100 according to the present embodiment, the image setting unit 165 has different image data for the same corresponding image based on the relationship between the identified line-of-sight direction ED of the user and the display position of the corresponding image. As shown in the image display maximum area PN of the image display unit 20. Therefore, in the HMD 100 of the present embodiment, the processing speed for displaying the corresponding image can be improved without reducing the ease of recognition by the user who visually recognizes the corresponding image.

  In the HMD 100 of the present embodiment, the image setting unit 165 sets the image display position of the corresponding image in association with the position of the detection target TW detected from the captured images of the cameras 61 and 62. For this reason, in the HMD 100 of the present embodiment, the correspondence between the detection target TW and the corresponding image can be more easily recognized by the user, and the usability of the user is improved.

  Further, in the HMD 100 of the present embodiment, the user's line-of-sight direction ED and the gazing point POR can be accurately identified because the user's line-of-sight direction ED is specified by the captured images of the eye imaging cameras 37 and 38.

  Further, in the HMD 100 of the present embodiment, the image setting unit 165 displays the corresponding image IM1 including the character image in the visual field range VR2 in the image display maximum region PN of the image display unit 20 based on the image data of the corresponding image IM1. . On the other hand, based on the image data of the corresponding image IM1, the image setting unit 165 displays a corresponding image that does not include a character image (for example, the corresponding image IM2) in the image display unit 20 in the visual field range outside the visual field range VR2. The image is displayed in the maximum image display area PN. Therefore, in the HMD 100 according to the present embodiment, the corresponding image not including the character image is displayed on the image display unit 20 in the visual field range in which it is difficult for the user to recognize the character, so that the user who recognizes the corresponding image can easily recognize. The speed of the process for displaying the corresponding image can be improved without further reducing the depth.

  In the HMD 100 of the present embodiment, the image setting unit 165 displays the simplified image corresponding to the circle image IM2 in the image display unit 20 in the visual field range VR5 based on the image data of the corresponding image IM1. Display on PN. For this reason, in the HMD 100 of the present embodiment, only a simplified image indicating that there is an image is displayed on the image display unit 20 in the visual field range where the user cannot recognize the detailed contents of the display image. The speed of the process for displaying the corresponding image can be further improved without reducing the ease of recognition by the user who visually recognizes the image.

  In the HMD 100 of the present embodiment, the image setting unit 165 includes a two-dimensional image that does not include distance information in the visual field range VR4 regardless of the distance to the user's point of interest POR, based on the image data of the corresponding image IM1. The corresponding image IM3 as an image is displayed in the image display maximum area PN of the image display unit 20. Therefore, in the HMD 100 of the present embodiment, since the two-dimensional image is displayed on the image display unit 20 in the visual field range in which it is difficult for the user to recognize the distance to the display image, it is easy for the user who visually recognizes the corresponding image to recognize. The processing speed for displaying the corresponding image can be further improved without reducing.

B. Variation:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

B-1. Modification 1:
FIG. 13 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user in the modification. FIG. 13 shows the visual field VR after the user's gazing point POR is changed from the building BD1 to the building BD2, compared to FIG. 12 of the embodiment. In the example shown in FIG. 13, since the user's point of interest POR is the building BD2, the image setting unit 165 determines that the position of the detection target TW in the captured images of the cameras 61 and 62 belongs to the visual field range VR3. The corresponding image IM4 as a display image is displayed in the image display maximum area PN. The corresponding image IM4 is an image displayed as a three-dimensional image including (B) distance information with respect to the corresponding image IM3 shown in FIG. In other words, the corresponding image IM4 is an image including (B) the content of a three-dimensional image including distance information and (C) RGB color information. The visual field range VR4 corresponds to the fourth visual field range in the claims, and the visual field range inside the visual field range VR4 corresponds to the third visual field range in the claims.

  As described above, in the HMD 100 of this modified example, the image setting unit 165 displays the corresponding image IM4 as a three-dimensional image including the distance information in the visual field range VR3 based on the image data of the corresponding image IM1. The image is displayed in the maximum image display area PN. Therefore, in the HMD 100 of this modified example, based on the same image data, a 3D image is displayed in a range that can be recognized as a 3D image by the user, and in a range in which the user cannot easily recognize the 3D image. An image is displayed. Thereby, the speed of the process which displays a corresponding image can be improved more, without reducing the ease of recognition of the user who visually recognizes a corresponding image.

  FIG. 14 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user in the modification. FIG. 14 shows the visual field VR after the user's gazing point POR is changed from the building BD1 to the building BD3, as compared with FIG. 12 of the embodiment. In the example illustrated in FIG. 14, since the gazing point POR of the user US is the building BD3, the image setting unit 165 determines that the position of the detection target TW in the captured images of the cameras 61 and 62 belongs to the visual field range VR5. The corresponding image IM5 as the display image is displayed in the image display maximum area PN. Corresponding image IM5 is an image that is represented by the density of white and black and is created based on the same image data as corresponding image IM1. Corresponding image IM5 does not include RGB color information, and expresses characters of “HP” with light and dark colors against a background of a dark and dark rectangular image in order to improve user visibility. It is an image. In other words, the corresponding image IM5 is an image that does not include (C) RGB color information with respect to the corresponding image IM3. The visual field range VR5 corresponds to the sixth visual field range in the claims, and the visual field range inside the visual field range VR5 corresponds to the fifth visual field range in the claims.

  As described above, in the HMD 100 of this modification, the image setting unit 165 displays the corresponding image IM5 that does not include RGB color information in the field of view range VR5 based on the image data of the corresponding image IM1. The maximum area PN is displayed. Therefore, in the HMD 100 of this modification, a corresponding image using RGB color information is displayed within a range in which the user can recognize various color information including RGB based on the same image data. In a range where it is difficult to recognize the image, a corresponding image having only color information of white and black is displayed. Thereby, the speed of the process which displays a corresponding image can be improved more, without reducing the ease of recognition of the user who visually recognizes a corresponding image.

B-2. Modification 2:
FIG. 15 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when the corresponding image IM5 belongs to the visual field range VR5. FIG. 15 shows a visual field VR in a case where a corresponding image IM6 that is an image of a hat is superimposed on a captured human subject OB and displayed at the position of the head. The subject OB is a person who is trying on ski wear on the upper body and the lower body, and is not actually trying on a hat on the head. As shown in FIG. 15, the gazing point POR of the user of the HMD 100 is a position away from the subject OB, and in this case, the region where the corresponding image IM6 is displayed in the image display maximum region PN is in the visual field range VR5. Belongs. The image data on which the corresponding image IM6 is based also includes color information such as a hat pattern displayed as the corresponding image IM6. However, since the display position of the corresponding image IM6 belongs to the visual field range VR5, the image setting unit 165 does not include color information such as a pattern or RGB, and displays only the outer frame of the hat with a dark dark shade of black. The image is displayed in the image display maximum area PN. In other words, the image setting unit 165 displays an image of only the outer frame of the hat image in the image display maximum region PN in order to make the user recognize the outline of the corresponding image IM6.

  As described above, in the HMD 100 of this modification, the image setting unit 165 displays the corresponding image IM6 of only the outline of the display image in the image display maximum region PN of the image display unit 20 based on the image data. Let For this reason, in the HMD 100 of this modified example, only the outer frame of the image is displayed in a range where it is difficult for the user to recognize the color information, so that the user can easily recognize the corresponding image without reducing the ease of recognition. The processing speed for displaying an image can be further improved.

B-3. Modification 3:
In the above-described embodiment, the display image is displayed in a part of the image display maximum area PN. However, the display image may be an image displayed on the entire image display maximum area PN. The image may be different from the corresponding image associated with the detection target detected from the captured image. For example, the display image may be a moving image such as a movie that is reproduced by content or the like regardless of the detection target. The moving image as the display image may be displayed on the entire image display maximum area PN. When the image is a moving image displayed on the entire image display maximum area PN, the image setting unit 165 hides the moving image when determining that the user's line-of-sight direction ED is not on the image display maximum area PN. Also good. As described above, the image setting unit 165 variously changes and sets the display mode of the display image based on the image displayed in the image display maximum region PN and the identified user's line-of-sight direction ED.

  In the above embodiment, the user's line-of-sight direction ED is specified based on the captured images of the eye imaging cameras 37 and 38, but may be specified by another method. For example, the user's line-of-sight direction ED may be specified by the orientation of the image display unit 20. Specifically, the line-of-sight direction ED of the user wearing the image display unit 20 on the head is specified as the front direction of the image display unit 20, and the center of the image display maximum region PN is always set to the user. May be set as the point of interest POR. According to the HMD 100 of this modification, the user's line-of-sight direction ED can be specified easily and inexpensively without using an apparatus such as the eye imaging cameras 37 and 38. Further, a gyro sensor may be mounted on the image display unit 20 and the orientation of the image display unit 20 may be specified based on the angular velocity detected by the gyro sensor. Further, the line-of-sight direction of the user may be specified by a myoelectric potential detected by an electrode disposed in a portion that contacts the user's temple or nose.

  In the above embodiment, the range for the user's point of interest POR, such as the visual field range VR2, is set according to the visual field of the human and the level of human visibility related to the visual field, but various modifications can be made depending on these ranges. is there. The visual field range VR2 and the like may be set according to the brightness of the outside scene SC and the like, the visual acuity of the user of the HMD 100, the resolution of the displayed image, and the like.

  In the above embodiment, the distance to the object included in the captured image is measured based on the captured images of the cameras 61 and 62. However, the method for measuring the distance to the object and the presence or absence of the measurement are variously modified. Is possible. In addition, the distance to the object need not necessarily be measured.

  In the above-described embodiment, the user who wears the image display unit 20 can visually recognize the transmitted outside scene SC. However, a so-called closed HMD in which the user cannot visually recognize the outside scene SC may be used.

  In the above-described embodiment, a stereo camera including the first camera 61 and the second camera 62 is used as the imaging unit that images the outside scene SC. However, one camera may be used. More than one camera may be used. In addition, the position where the camera is arranged is not necessarily included in the image display unit 20, and the camera may be configured separately from the image display unit 20.

  Further, since there are individual differences in the visual characteristics of the user, the correspondence between the visual field range and the image displayed on the image display unit 20 may be changed by the operation unit 135 receiving a predetermined operation.

  In the above embodiment, a method of forming binocular parallax is used in order to make the user recognize as a three-dimensional image, but the method of making the user recognize as a three-dimensional image can be variously modified. For example, as a method for allowing the user to recognize as a three-dimensional image, an image for converging motion is displayed on the image display unit 20, a shadow is added to the image, or a sense of depth is expressed by changing the size of the image. May be. Also, the two-dimensional image may be an image displayed on the image display unit 20 as infinity not including distance information, for example, and can be variously modified.

  As shown in FIG. 14, in the visual field range VR5, the corresponding image IM5 that does not include RGB color information is displayed in the image display maximum area PN. However, examples that do not include color information are not limited thereto, and various modifications are possible. Is possible. For example, an image provided with light and shade for a single color such as red (RED) may be displayed in the image display maximum area PN. An image with reduced color information may simply be an image displayed in a specific color gray scale or the like.

  In the above embodiment, as shown in FIG. 11, the corresponding image IM2 that is a round image is displayed as a simplified image in the image display maximum area PN. However, the simplified image can be variously modified. For example, the image may be a triangle or square image other than a circle, or may be in the form of some symbol mark that easily associates the image data associated with the image data. The simplified image is not complicated compared to an image (for example, the corresponding image IM1) displayed based on the image data, and the user can recognize that the simplified image exists there. Any image can be used.

  In the above embodiment, the aspect of the image displayed in the image display maximum area PN has changed based on the relationship between the user's line-of-sight direction and the user's visual field range. May change. For example, when it is included in the visual field range VR2, the image is displayed on the entire image display maximum region PN, and when it is a range outside the visual field range VR5, it is simplified to a part of the image display maximum region PN. The displayed image may be displayed.

B-4. Modification 4:
In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, a mode in which a user interface that is the operation unit 135 is provided separately from the control unit 10 may be used. In this case, the operation unit 135 is separate from the control unit 10 in which the power supply 130 and the like are formed. Therefore, the operation unit 135 can be reduced in size and user operability is improved. Further, the cameras 61 and 62 are arranged in the image display unit 20, but the camera 61 may be configured separately from the image display unit 20 and can capture the outside scene SC. Alternatively, the HMD 100 in which the CPU 140 and the power supply 130 that are included in the control unit 10 are all mounted in the image display unit 20 may be used. The HMD 100 can be further downsized because there is no controller configured separately from the image display unit 20. Further, by mounting the CPU 140 on each of the control unit 10 and the image display unit 20, the control unit 10 may be used as a single controller and the image display unit 20 may be used as a single display device.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to the laser retinal projection type HMD 100. The image display maximum area PN may be configured by a MEMS shutter type display that opens and closes a MEMS shutter formed in each pixel.

  Alternatively, the image light generation unit may be an HMD 100 that employs a scanning optical system using a MEMS mirror and utilizes MEMS display technology. As a specific configuration, the HMD 100 forms a virtual image by a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and light scanned by the scanning optical system. You may provide an optical member as a light emission part. When the HMD 100 has this configuration, the light emitted by the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided to the optical member, and reaches a virtual image forming surface (for example, a reflecting surface). When the MEMS mirror scans light, a virtual image is formed on the virtual image forming surface, and the user visually recognizes the formed virtual image so that the user can recognize the image.

  Further, for example, the HMD 100 may be a head mounted display in which the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit does not completely cover the user's eyes. The HMD 100 may be a so-called monocular type head mounted display. Further, instead of being mounted on the user's head instead of the HMD 100, a handheld display in which the user fixes the position by hand like binoculars may be used as the image display device. Although the HMD 100 is a binocular optical transmission type, the present invention can be similarly applied to other types of head-mounted display devices such as a video transmission type.

  Moreover, HMD100 may be used with the display apparatus only for displaying the image based on the image signal received from the other apparatus. Specifically, it is used as a display device corresponding to a desktop PC monitor. For example, by receiving an image signal from a desktop PC, an image is displayed in the maximum image display area PN of the image display unit 20. May be.

  Moreover, HMD100 may be used so that it may function as a part of system. For example, the HMD 100 may be used as a device for executing a part of functions of a system including an aircraft, and the system in which the HMD 100 is used is not limited to a system including an aircraft, but is a system including an automobile or a bicycle. There may be.

  The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-mounted display device mounted on a vehicle such as an automobile or an airplane. For example, it may be configured as a head-mounted display device built in a body protective device such as a helmet.

B-5. Modification 5:
The configuration of the HMD 100 in the above embodiment is merely an example and can be variously modified. For example, the direction key 16 provided in the control unit 10 may be omitted, or another operation interface such as an operation stick may be provided in addition to the direction key 16 and the track pad 14. Moreover, the control part 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. Further, the earphones 32 and 34 can be omitted as appropriate. Moreover, in the said embodiment, although LCD and a light source are utilized as a structure which produces | generates image light, it replaces with these and you may employ | adopt other display elements, such as an organic EL display.

  FIG. 16 is an explanatory diagram showing an external configuration of an HMD in a modified example. In the example of FIG. 16A, the difference from the HMD 100 illustrated in FIG. 1 is that the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, It is a point provided with the left optical image display part 28x instead of the display part 28. The right optical image display unit 26x is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the right eye of the user when the HMD 100x is mounted. Similarly, the left optical image display unit 28x is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the left eye of the user when the HMD 100x is worn. 16B, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and the left optical image. It is a point provided with the left optical image display part 28y instead of the display part 28. The right optical image display unit 26y is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28y is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the left eye of the user when the head mounted display is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can be realized as an HMD 100 in an uncovered mode.

  In the above-described embodiment, the HMD 100 may guide the image light representing the same image to the left and right eyes of the user to make the user visually recognize the two-dimensional image, or may display different images on the left and right eyes of the user. The image light to be represented may be guided to make the user visually recognize the three-dimensional image.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown in FIG. 1 and FIG. 2, the control part 10 and the image display part 20 are formed as a separate structure, However, about the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. In addition, the power source 130 in the above embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be overlapped and formed in the image display unit 20. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switch key 14 ... Track pad 15 ... Luminance switch key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part (image display part)
DESCRIPTION OF SYMBOLS 21 ... Right holding part 22 ... Right display drive part 23 ... Left holding part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display part 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 37 ... Right-eye imaging camera (eye imaging unit, line-of-sight direction identification unit)
38 ... Left-eye imaging camera (eye imaging unit, line-of-sight direction specifying unit)
DESCRIPTION OF SYMBOLS 40 ... Connection part 42 ... Right cord 44 ... Left code 46 ... Connecting member 48 ... Main body code 51, 52 ... Transmission part 53, 54 ... Reception part 61 ... 1st camera (outside scene imaging part)
62 ... Second camera (outside scene imaging unit)
100 ... HMD (head-mounted display device)
DESCRIPTION OF SYMBOLS 120 ... Memory | storage part 130 ... Power supply 132 ... Wireless communication part 135 ... Operation part 140 ... CPU
150: Operating system 160 ... Image processing unit 165 ... Image setting unit (image changing unit)
166... Direction specifying unit (gaze direction specifying unit)
168 ... Distance measuring unit 170 ... Audio processing unit 180 ... Interface 190 ... Display control unit 201 ... Right BL control unit 202 ... Left BL control unit 211 ... Right LCD control unit 212 ... Left LCD control unit 221 ... Right backlight 222 ... Left Backlight 241, right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate AP ... Tip part BD1, BD2, BD3 ... Building ED ... User's line of sight direction EL ... End part of left optical image display part ER ... End of right optical image display HSync ... Horizontal synchronization signal VSync ... Vertical synchronization signal PCLK ... Clock signal IL ... Illumination light IM1, IM2, IM3, IM4, IM5, IM6 ... Corresponding image LE ... Left eye RE ... Right eye MN ... Month MP ... Left and right eyeball center OA ... External device OB ... Subject PL ... Image light PN ... Maximum image display area (display area)
POR ... Gaze point SC ... Outside view TW ... Detection target (specific object)
US ... User VR ... View VR1, VR2, VR3, VR4, VR5, VR6 ... View range VRh ... Vertical view VRw ... Horizontal view

Claims (14)

A head-mounted display device,
An image display unit capable of displaying an image in a display area;
A line-of-sight direction identifying unit that identifies the line-of-sight direction of the user;
An image changing unit that changes a display image displayed in the display area according to the identified line-of-sight direction ,
In the case where the display image is an image that is visually recognized by a user as a three-dimensional image, the image changing unit displays the display image as a three-dimensional image in a third field of view including the specified line-of-sight direction. The display image is displayed as a two-dimensional image in a fourth visual field range that is displayed in a region and is larger than the third visual field range and does not include the third visual field range. A head-mounted display device that displays the image on the display area . A head-mounted display device,
An image display unit capable of displaying an image in a display area;
A line-of-sight direction identifying unit that identifies the line-of-sight direction of the user;
An image changing unit that changes a display image displayed in the display area according to the identified line-of-sight direction ,
When the display image is an image including information of a plurality of colors, the image change unit includes the display image including information of a plurality of colors in the fifth visual field range including the specified line-of-sight direction. In the sixth viewing range that is larger than the fifth viewing range and does not include the fifth viewing range, the display image is displayed in the display region using a single color shading. A head-mounted display device for display.   The head-mounted display device according to claim 2,
  The head-mounted display device, wherein the image changing unit displays only an outline of the display image in the display area in the sixth visual field range. The head-mounted display device according to any one of claims 1 to 3 ,
The image changing unit is configured to display the display image in the seventh visual field range that is larger than the seventh visual field range including the specified line-of-sight direction and in the eighth visual field range not including the seventh visual field range. A head-mounted display device that displays in the display area as a simplified image that is simpler than an image displayed in the visual field range. The head-mounted display device according to any one of claims 1 to 4 ,
The head-mounted display device, wherein the image changing unit changes the display image in accordance with a visual characteristic of a user that changes according to the identified line-of-sight direction. The head-mounted display device according to any one of claims 1 to 5 ,
The head-mounted display device that changes the display image in accordance with the position of the display image and the line-of-sight direction when the display image is displayed in a part of the display area. The head-mounted display device according to any one of claims 1 to 6 ,
The image display unit can transmit an outside scene,
The head-mounted display device further includes:
An outside scene imaging unit for imaging the outside scene;
The head-mounted display device, wherein the image change unit detects a specific object from the captured image of the outside scene and sets the position of the display image in association with the detected position of the specific object. The head-mounted display device according to claim 7 ,
The gaze direction specifying unit includes a display unit direction detecting unit that detects the direction of the image display unit, and specifies the gaze direction based on the detected direction of the image display unit. . The head-mounted display device according to claim 7 ,
The eye-gaze direction specifying unit includes an eye imaging unit that images a user's eye, and specifies the eye direction based on the captured image of the user's eye. The head-mounted display device according to any one of claims 1 to 9 ,
When the display image includes a character image, the image changing unit displays the character image as the display image in the display area in the first visual field range including the specified line-of-sight direction, A head-mounted display device that does not display the character image as the display image in the display area in a second visual field range that is larger than the visual field range and does not include the first visual field range. A control method for a head-mounted display device having an image display unit capable of displaying an image in a display area,
Identifying the user's gaze direction;
Changing the display image displayed in the display area according to the identified line-of-sight direction ,
In the step of changing the display image, in the case where the display image is an image that is visually recognized by a user as a three-dimensional image, the display image is converted into a three-dimensional image in the third visual field range including the specified line-of-sight direction. The display image is displayed as a two-dimensional image in the fourth visual field range that is larger than the third visual field range and does not include the third visual field range. A control method for displaying in the display area . A control method for a head-mounted display device having an image display unit capable of displaying an image in a display area,
Identifying the user's gaze direction;
Changing the display image displayed in the display area according to the identified line-of-sight direction,
In the step of changing the display image, when the display image is an image including information of a plurality of colors, the display image is converted to information of a plurality of colors in the fifth visual field range including the specified line-of-sight direction. In the sixth viewing range that is larger than the fifth viewing range and does not include the fifth viewing range, the display image is displayed using a single color shading. A control method that displays in the display area . A computer program for a head-mounted display device,
An image display function capable of displaying an image in the display area;
A gaze direction identifying function for identifying the gaze direction of the user;
An image change function for changing a display image displayed in the display area according to the identified line-of-sight direction is realized in a head-mounted display device ,
In the case where the display image is an image that is visually recognized by a user as a three-dimensional image, the image change function is configured to display the display image as a three-dimensional image in a third visual field range including the specified line-of-sight direction. The display image is displayed as a two-dimensional image in a fourth visual field range that is displayed in a region and is larger than the third visual field range and does not include the third visual field range. A computer program for causing the computer to display in the display area . A computer program for a head-mounted display device,
An image display function capable of displaying an image in the display area;
A gaze direction identifying function for identifying the gaze direction of the user;
An image change function for changing a display image displayed in the display area according to the identified line-of-sight direction is realized in a head-mounted display device ,
When the display image is an image including information of a plurality of colors, the image change function includes an image including information of a plurality of colors in the fifth visual field range including the specified line-of-sight direction. In the sixth viewing range that is larger than the fifth viewing range and does not include the fifth viewing range, the display image is displayed in the display region using a single color shading. A computer program to be displayed .

Images