WO2019041614A1

WO2019041614A1 - Head-mounted immersive virtual reality display device and immersive virtual reality display method

Info

Publication number: WO2019041614A1
Application number: PCT/CN2017/114192
Authority: WO
Inventors: 李海峰; 陆驰豪; 刘旭; 刘玛丽
Original assignee: 浙江大学
Priority date: 2017-09-04
Filing date: 2017-12-01
Publication date: 2019-03-07
Also published as: CN107462994A

Abstract

A head-mounted immersive virtual reality display device and an immersive virtual reality display method. The head-mounted display device comprises a head tracking and orientation detection system (1), a head-mounted optical display system (2), an image segmentation system (3), an encoding and decoding system (4), and a processing system (5). The head tracking and orientation detection system (1) is used to capture an image of a peripheral environment of a human eye. The encoding and decoding system (4) decodes an image signal of the head tracking and orientation detection system (1) before transmitting the decoded image signal to the processing system (5). Once the same has been processed by the processing system (5), the encoding and decoding system (4) performs encoding, and image information is transmitted to the image segmentation system (3). The image segmentation system (3) divides the received image information into a plurality of display images, and then the head-mounted optical display system (2) performs display. The head tracking and orientation detection system (1) captures an image of a peripheral environment and acquires head position and orientation information, and superimposes virtual video information over the image information of the peripheral environment, thereby combining the real and virtual world.

Description

Immersive virtual reality head-mounted display device and immersive virtual reality display method

Technical field

The present invention relates to a three-dimensional display technology neighborhood, and in particular to an immersive virtual reality head-mounted display device and an immersive virtual reality display method.

Background technique

Virtual reality technology is a computer simulation system that can create and experience virtual worlds. This technology is mostly used for entertainment, such as immersive video games. But virtual reality technology is also very important in other fields, such as medicine, military aerospace and so on.

The current mainstream virtual reality devices on the market, such as HTC Vive, Oculus Rift, etc., do not reach the best field of view (120°) of the head-mounted display. Therefore, it is impossible to maximize user immersion.

And like HTC Vive, the devices used for head tracking and attitude detection are too complicated and require a large space to support the operation of the device. Most of the virtual reality devices on the mobile side use the gyroscope to achieve head tracking, and the tracking accuracy cannot be guaranteed.

Summary of the invention

The object of the present invention is to provide an immersive virtual reality head-mounted display device and an immersive virtual reality display method, which can ensure the head tracking accuracy while realizing a large field of view, and has little space for operation.

In order to achieve the above object, an immersive virtual reality head mounted display device provided by the present invention includes a head tracking and attitude detecting system, a head mounted display optical system, an image dividing system, and an encoding and decoding system. And a processing system; the head tracking and attitude detecting system is used for capturing an image of a peripheral environment of the human eye, and the encoding and decoding system decodes the image signal of the tracking and attitude detecting system and transmits it to the processing system, and processes and decodes the image through the processing system. The system is encoded and transmitted to the image segmentation system, and the image segmentation system divides the received image information into a plurality of display images and displays them through the head-mounted display optical system.

In the above technical solution, the head tracking and attitude detection system is used to capture the peripheral environment scene image and obtain the head position and posture information, and the processing system may be an external computer, and the computer superimposes the peripheral environment scene image information and the virtual video information to realize the real world and At the same time, the computer adjusts the virtual video information according to the change of the head position and the posture information so that the peripheral environment scene image is more integrated in reality.

A specific solution is that the head tracking and attitude detecting system includes a live view unit for actual shooting of a peripheral scene placed at a left and right eye position of a person, a panoramic camera unit for panoramic shooting of a peripheral scene, and a position and attitude sensor.

The environment panorama obtained by panoramic camera shooting is used as a base on which peripheral scenes captured by two conventional cameras located at the left and right eyes are superimposed. Because the panoramic camera unit has a larger shooting range, but the resolution is lower, the conventional camera has a smaller shooting range and a higher resolution. The effect of such overlap is that in a large image with a lower resolution, a part of the area (i.e., the normal camera shooting area, that is, the normal eye observation area) is higher than the surrounding resolution. Adding a surrounding environment with a low resolution can deepen the user's sense of presence.

The above video overlay process is implemented by an FPGA in an image segmentation system in a head mounted display device. The video signals from the conventional camera and the panoramic camera unit are combined into one video signal and encoded, and connected to the encoding and decoding system for decoding by a cable, and the decoded video is input to the processing system through the USB3.0 port for calculation. To achieve the acquisition of peripheral real scene, as well as head tracking and attitude detection.

Another specific solution is that the head-mounted display optical system includes two sets of monocular optical systems arranged symmetrically in a left-right direction; the monocular optical system includes a lens group sequentially disposed in front of the human eye and placed in the lens group a rear display; the lens group includes at least one set of Fresnel lenses, each set of Fresnel lenses consisting of two resin plates engraved with a Fresnel lens sawtooth structure; the display is two, respectively parallel to the corresponding resin plate The angle β between the apex of the splicing of the two resin sheets to the center of the eyeball and the optical axis perpendicular to the eyeball is greater than 0°.

Fresnel lens group is used to magnify the virtual image of the image displayed on the display. Each group of Fresnel lens is spliced by two resin plates engraved with Fresnel lens sawtooth structure. The reason is that the spliced Fresnel lens is used. The group replaces the ordinary lens in order to make the left and right fields of view more convenient when splicing. If an ordinary lens group is used, since the shape is generally circular, seamless stitching cannot be achieved when the left and right imaging systems are spliced. The two resin plates are spliced, and since the shape is rectangular, there is no gap after the two rectangles are spliced.

Set the line connecting the apex of the two resin plates to the center of the eyeball, and the angle between the axis perpendicular to the eyeball is greater than 0°. The reasons for this are as follows:

In the monocular system, if the eye just looks at the splicing of the two resin sheets, since the splicing has no imaging effect, the image displayed by the display is not visible to the human eye at this time. The final complete binocular system contains the above two sets of monocular systems. If this angle requirement is not set, then when the human eye observes infinity, both eyes fall on the Fresnel lens stitching of the two systems. At this point, both eyes cannot get the image from the display, so this location becomes a blind spot. When this angle is set, due to binocular convergence, when one eye looks at the splicing of the Fresnel lens, the other eye will inevitably not look at its splicing place, so there is no blind spot.

A more specific solution is that the two resin sheets are spliced into an L-shape; the lens group includes two sets of Fresnel lenses arranged in parallel; the displays located in the left front and the right front of the human eye are respectively formed by the visual optical system to enlarge two The virtual image has an overlap region δ in space, and within the overlap region δ, the image brightness of the two displays gradually darkens from the overlap of the display to the edge.

It can also be set to other numbers of Fresnel lens groups as needed. The image brightness of the two displays gradually dims from the overlap of the display to the edge for the consistency of the final composite image brightness.

A more specific solution is that each L-type Fresnel lens, each arm and the rear optical lens are composed A visual optical system, and the parameters of the optical system are as follows:

The horizontal and vertical fields of view are greater than 80°, the medial half angle is 45°, and the overlap area is 90°;

The two-arm visual optical system is assembled into a monocular optical system with a field of view greater than 120°.

Thus, the two arms of the L-type Fresnel lens are each a set of optical systems with an angle of view above 80°, and after splicing, they become an optical system with a field of view of 120° or more.

Another more specific solution is that the monocular field of view of the head-mounted display optical system is greater than 120°, the binocular field of view is greater than 150°, and the overlap portion is greater than 90°.

The immersive virtual reality display method provided by the present invention comprises the following steps:

1) Obtain a real-life image of the surrounding environment;

2) superimposing the peripheral environment real image and the virtual video image to obtain a actually played video signal;

3) dividing the actually played video signal and performing screening on multiple displays;

4) analyzing the peripheral environment real image to obtain position and attitude information;

5) Adjusting the virtual video image by the position and posture information to fuse with the peripheral environment real image.

The specific solution is that the step 1) comprises: taking a panorama of the peripheral environment as a base by using a panoramic camera, and superimposing a peripheral environment image captured by a conventional camera on the substrate.

Compared with the prior art, the beneficial effects of the present invention are:

(1) The head tracking and posture detection of the present invention are realized by a combination of a panoramic ring imaging system and a nine-axis motion sensor such as a gyroscope. Therefore, the accuracy of head tracking and attitude detection can be improved without increasing the space occupied by the device;

(2) The wearing optical system of the present invention adopts a multi-display viewing angle splicing method, so that the viewing angle of the virtual display device can be improved to achieve the optimal viewing angle of the head mounted display, thereby improving user immersion;

(3) The present invention utilizes a combination of a panoramic camera system and a real-life camera system to obtain a peripheral real scene, so that it is possible to obtain as large a peripheral real scene as possible without reducing the resolution, thereby strengthening The user's sense of presence.

DRAWINGS

1 is a schematic structural view of an embodiment of the present invention;

2 is a schematic structural view of an immersive head mounted display optical system according to an embodiment of the present invention;

3 is a detailed structural view of one of the arm visual optical systems of the immersive head mounted display optical system according to the embodiment of the present invention.

Detailed ways

The invention will be further described below in conjunction with the embodiments and the accompanying drawings.

Example

Referring to FIGS. 1 and 2, the immersive virtual reality head-mounted display device includes a head tracking and attitude detecting system 1, a head-mounted display optical system 2, an image segmentation system 3, an encoding and decoding system 4, and a processing system 5. The encoding and decoding system 4 first decodes the signal of the head tracking and attitude detecting system 1 and transmits it to the processing system 5. After being processed by the processing system 5, it is encoded by the encoding and decoding system 4 into a signal that can be recognized by the image segmentation system 3. To the image segmentation system 3, the image segmentation system 3 restores the image coded information drawn by the processing system 5, and divides it into a plurality of display images, which are transmitted to the head mounted display optical system 1 for display. The processing system 5 includes an external computer.

The head tracking and attitude detecting system 2 includes a live view unit, a panoramic camera unit, and a position and attitude sensor. The real-life camera unit includes two conventional cameras respectively placed at the left and right eye positions of the person for actual shooting of the peripheral scene, and provides a real-life image for the virtual integration of the video by the head-mounted display device. The panoramic camera unit includes a panoramic imaging unit for panoramic shooting of peripheral scenes to provide an image for the position and posture of the head mounted display device. The position and attitude sensors can be gyroscopes.

In the present embodiment, the head-mounted display optical system 2 includes two sets of monocular optical systems placed symmetrically left and right, wherein a monocular optical system 05 is placed in front of the left eye 07, and a monocular optical system is placed in front of the right eye 08. . Each monocular optical system includes a lens group that is placed in front of the human eye in turn, and is placed on After the lens group, two displays placed in the left front and the right front of the human eye, respectively placed on the left front and the right front of the left eye 07 are the OLED display 01 and the OLED display 02, respectively, placed in the left front of the right eye 08 And the right front are respectively OLED display 03 and OLED display 04, wherein each OLED screen has a resolution of 1000x1200, and a large field of view is synthesized by two OLEDs, each of which has a resolution of 2000x1200. The lens group includes at least one L-type Fresnel lens, and the first lens near the human eye employs an L-type Fresnel lens.

The lens group includes an L-type Fresnel lens and an aspherical lens, wherein the L-type Fresnel lens close to the human eye is composed of two resin plates connected at both ends, and each of the two surfaces of each resin plate has at least one side The surface is engraved with a sawtooth structure of a Fresnel lens, and the angle β between the apex of the two resin plates connecting the L-type Fresnel lens to the center of the eyeball and the optical axis perpendicular to the eyeball is greater than 0°. In order to avoid observing the infinity, both eyes look at the splicing of the two resin plates of the L-type Fresnel lens at the same time, thereby causing blind spots.

Each arm of the L-type Fresnel lens and the optical lens behind it form a complete visual optical system. The specific structure can be as shown in Figure 3. It contains a Fresnel lens and two aspheric surfaces close to the human eye. Plastic lens lens. The left side is the position of the human eye and the right side is the position of the display. The display uses an OLED display with a pixel size of approximately 50 um and a stripline pair of 10 lp/mm.

The OLED display screens located on the left and right sides of the same eye respectively form an enlarged virtual image by L-type Fresnel lenses, and the two virtual images have overlapping regions δ in space. In the overlap region δ, the image brightness of the two displays gradually darkens from the overlap of the display to the edge, so that after the images of the overlapping regions are superimposed, the total brightness of the two displays is the same as the brightness of the non-overlapping regions of the two displays. To facilitate the consistency of the brightness of the final composite image, thereby achieving smooth stitching of the left and right fields of view.

Each of the L-type Fresnel lenses described above, each of which is followed by an optical lens, constitutes a complete visual optical system, and the parameters of the optical system are as follows:

The horizontal and vertical fields of view are greater than 80°, the medial half angle is 45°, and the overlap area is guaranteed to be 90°; the visual optical system of the two arms is combined into a monocular optical system with a field of view greater than 120°. Thus, the two arms of the L-type Fresnel lens are each a set of optical systems with an angle of view above 80°, which becomes a splicing Optical system above 120° field of view.

The specific implementation is as follows:

First, the head tracking and attitude detecting system 1 captures a peripheral real-life image, including a panoramic camera system and a video taken by a conventional camera placed at the left and right eye positions, and uses an FPGA in the encoding and decoding system 4 to synthesize a video signal. The encoding is performed by cable to the encoding and decoding system on the processing system 5 side for decoding, and the decoded video is input to the processing system 5 through the USB 3.0 port for calculation, to obtain peripheral real scene acquisition, and head tracking and attitude detection. . At this time, the processing system 5 superimposes the virtual video signal to be displayed on the obtained actual video signal, encodes it into a format recognizable by the image segmentation system 3 by the encoding and decoding system 4, and transmits it to the image segmentation system 3 through the cable. The image segmentation system 3 restores the image coded information drawn by the processing system 5 and divides it into a plurality of display images, which are connected to the driver of the OLED image source in the immersion head mounted display optical system 2. At this time, the human eye can obtain a virtual image superimposed on the peripheral real environment by wearing the display optical system 2.

While the computer outputs the video signal, the video signal and the sensor signal from the head tracking and attitude detecting system are continuously received to obtain the head position and posture information.

The image signal is drawn by a computer graphics card at a resolution of 2000x1200, and is split into two 1000x1200 resolution images by the segmentation system 3. Aberration correction is achieved by a combination of a Fresnel lens and an aspherical lens. Large field of view immersive display through image fusion. The final effect is that the field of view of a single eye is greater than 120 degrees, the field of view of both eyes is greater than 150 degrees, and the overlap is greater than 90 degrees.

The immersive virtual reality display method includes the following steps:

1) Obtaining a real-life image of the peripheral environment; specifically: capturing a panoramic view of the peripheral environment as a base by using a panoramic camera, and superimposing a peripheral environment image captured by a conventional camera on the substrate;

Claims

An immersive virtual reality head-mounted display device, characterized in that:

Including head tracking and attitude detection system, head-mounted display optical system, image segmentation system, encoding and decoding system, and processing system;

The head tracking and attitude detecting system is configured to capture an image of a peripheral environment of a human eye, and the encoding and decoding system decodes the image signal of the tracking and attitude detecting system and transmits the image signal to the processing system, and after being processed by the processing system After being encoded by the encoding and decoding system and transmitted to the image segmentation system, the image segmentation system divides the received image information into a plurality of display images and displays them through the head-mounted display optical system.
The immersive virtual reality head-mounted display device according to claim 1, wherein:

The head tracking and attitude detecting system includes a live view unit that is placed at a left and right eye position of a person for actual shooting of a peripheral scene, a panoramic camera unit for panoramic shooting of a peripheral scene, and a position and attitude sensor.
The immersive virtual reality head-mounted display device according to claim 1, wherein:

The head mounted display optical system comprises two sets of monocular optical systems arranged symmetrically left and right;

The monocular optical system includes a lens group sequentially disposed in front of a human eye and a display placed behind the lens group;

The lens group includes at least one set of Fresnel lenses, each set of Fresnel lenses being composed of two resin plates engraved with a Fresnel lens sawtooth structure;

The display is two pieces, which are respectively parallel to the corresponding resin plates;

The angle β between the line connecting the apex of the two resin sheets to the center of the eyeball and the optical axis perpendicular to the eyeball is greater than 0°.
The immersive virtual reality head-mounted display device according to claim 3, wherein:

The two pieces of the resin sheets are spliced into an L shape;

The lens group includes two sets of Fresnel lenses arranged in parallel;

The display located at the left front and the right front of the human eye respectively forms an enlarged two virtual image by the visual optical system, and has an overlapping area δ in the space. In the overlapping area δ, the image brightness of the two displays overlaps from the display. Gradually darken towards the edges.
The immersive virtual reality head-mounted display device according to claim 4, wherein:

Each of the L-type Fresnel lenses, each of which is followed by an optical lens, constitutes a visual optical system, and the parameters of the optical system are as follows:

The horizontal and vertical fields of view are greater than 80°, the medial half angle is 45°, and the overlap area is 90°;

The two-arm visual optical system is assembled into a monocular optical system with a field of view greater than 120°.
The immersive virtual reality head-mounted display device according to claim 3, wherein:

The head-mounted display optical system has a monocular field of view greater than 120°, a binocular field of view greater than 150°, and an overlapping portion greater than 90°.
An immersive virtual reality display method, comprising the steps of:

1) Obtain a real-life image of the surrounding environment;

2) superimposing the peripheral environment real image and the virtual video image to obtain a actually played video signal;

3) dividing the actually played video signal and performing screening on multiple displays;

4) analyzing the peripheral environment real image to obtain position and attitude information;

5) Adjusting the virtual video image by the position and posture information to fuse with the peripheral environment real image.
The immersive virtual reality display method according to claim 7, wherein:

Step 1) includes: taking a panoramic image of the peripheral environment as a base by using a panoramic camera, and superimposing a peripheral environment image captured by a conventional camera on the substrate.