KR20190095671A - Virtual reality head mounted display for showing user's status and user status display method and content control method using the system - Google Patents

Abstract

Disclosed are a virtual reality head mounted display showing a user's status, a user status displaying method, and a content management method using the same. According to an embodiment of the present invention, in the virtual reality head-mounted display (VR HMD) that allows the user to watch the audio-visual content played through the internal display installed inside the frame body worn on the head, the rear portion of the frame body A camera unit installed to acquire a user face image including the eyes of the user; A signal processor configured to generate a user state image by image processing the user face image according to a predetermined image processing algorithm; And a front display unit installed at the front side of the frame body and displaying the user state image processed by the frame body.

Description

Virtual reality head mounted display for showing user's status and user status display method and content control method using the system}

The present invention relates to a virtual reality head mounted display (VR HMD) showing a user's status and a user status display and content management method using the same.

Virtual reality (VR) is expanding its application to education, medical care, and nursing care. VR can be used to treat alcoholism, to treat soldiers' war trauma, to provide VR care for dementia patients, or to use VR classes for children and other students.

However, for such a virtual reality experience, a head mounted display (HMD) is required. When wearing a VR HMD, the user's eyes and facial expressions cannot be seen. As a result, when a user is a child or a patient, a caregiver, a therapist, a caregiver, etc., are difficult to grasp the user's condition in real time, and thus, there is a problem in utilization in the fields of education, medical care, and care. For example, when the user is a child, the teacher or the parent may be uneasy about whether the user is properly using the VR content. In elderly care patients, it is difficult to catch signs of motion sickness or seizures due to VR content.

In addition, in the case of VR content used in the treatment of psychosis, content that the treatment target wants to avoid is often used as a treatment element. For example, to show war veterans suffering from the treatment of post-traumatic stress disorder (PTSD), VR in high buildings, high-altitude bridges for the treatment of heights, or to treat alcoholism. After a hangover, you can show a vomiting scene. In this case, the treatment subjects often close their eyes, and many of them fail to achieve the purpose of treatment.

Korean Laid-open Patent No. 10-2017-0139185 (Dec. 19, 2017)-Foldable virtual reality HMD with cooling fan

The present invention shows the user's eyes and / or facial expressions through the front of the virtual manager (guardian, therapist, caregiver, caregiver, etc.) can intuitively check the user status to show the user's status that can immediately respond To provide a realistic head-mounted display and a user status display and content management method using the same.

The present invention conceptually shows information on the eyes and / or expressions of the user through the front surface of the virtual reality head-mounted display showing the user's viewing state that allows the administrator to intuitively check whether the VR content experience is properly performed or not. And to provide a user status display and content management method using the same.

Other objects of the present invention will be readily understood through the following description.

According to an aspect of the present invention, in the virtual reality head-mounted display (VR HMD) that allows the user to watch the audio-visual content played through the internal display installed inside the frame body worn on the head, installed in the rear portion of the frame body A camera unit configured to obtain a user face image including the eyes of the user; A signal processor configured to generate a user state image by image processing the user face image according to a predetermined image processing algorithm; And a front display unit installed at the front side of the frame body and displaying the user state image processed by the frame body.

The signal processor may generate the user state image by scaling up or down the user face image to fit the size of the front display unit.

The signal processor may generate the user state image with an animation conceptualized based on the information acquired from the user face image.

The front display unit may be compactly formed on one side of the front unit, and the signal processor may display an emoticon corresponding to whether the user opens his eyes or the emotional state determination result of the user according to a result of analyzing the user facial image. Can be created as a user status image.

The apparatus further includes a sensor unit for obtaining at least one biosignal of the user's brain wave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, gaze direction, the signal processing unit The signal may be processed into user additional information to be superimposed on the user status image displayed through the front display unit.

A state output unit may further include a state output unit configured to output a user state according to an analysis result of the biosignal, wherein the state output unit may include at least one of a plurality of LEDs capable of expressing colors and a speaker capable of outputting a sound.

The electronic device may further include a controller configured to determine an emotional state of the user according to an analysis result of the biosignal or an analysis result of the user's face image, and to automatically switch or stop the audiovisual content currently being played based on the determination result.

The apparatus may further include an operation unit capable of a content related operation including playing, stopping, and converting the audiovisual content, wherein the operation unit may be implemented as a hardware button on the front side or by a software button on the front display unit.

Meanwhile, according to another aspect of the present invention, a user state display method performed in a virtual reality head mounted display for allowing a user to enjoy audiovisual content played through an internal display installed inside a frame body by wearing the head, wherein the audiovisual content is Acquiring a user's face image of the user's face through a camera unit installed at a rear portion of the frame body when played back; Generating a user state image by image processing the user face image according to a predetermined image processing algorithm; And displaying the user status image on the front display unit installed in the front portion of the frame body.

Acquiring at least one biosignal of an electroencephalogram (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, and gaze direction of the user; And processing the biosignal into user additional information so that the biosignal is superimposed on the user state image displayed through the front display unit.

Meanwhile, according to another aspect of the present invention, a content management method performed in a virtual reality head mounted display for allowing a user to enjoy audiovisual content played through an internal display installed inside a frame body by wearing on the head, wherein the audiovisual content is Acquiring at least one biosignal of an electroencephalogram (EGG), a heart rate (HR), a heart rate variability (HRV), an electrocardiogram (ECG), a pupil size, and a gaze direction of the user when being reproduced; Analyzing the bio-signal and classifying a user response to the audiovisual content into one of a plurality of emotional states according to a predetermined criterion; And determining whether to maintain or switch the audiovisual content according to the discriminated emotional state.

Storing preferences and aversions for the audio-visual content based on the discriminated emotional state, and recommending or avoiding any audio-visual content based on the preferences and aversions when the audio-visual content is switched. can do.

According to another aspect of the invention, a virtual reality head-mounted display system showing the user's state, the user is worn on the head to watch the audio-visual content played through the internal display installed inside the frame body, the frame body A virtual reality head mounted display (VR HMD) for acquiring a user's face image including the eyes of the user through a camera unit installed at the rear of the camera; When connected to the virtual reality head-mounted display through a network to receive the user face image or a user state image corresponding to the user face image, and the user face image is received according to a predetermined image processing algorithm. When the user state image is converted and the user state image is received, a virtual reality head mounted display system including an administrator terminal for outputting the user state image on the screen is provided.

The manager terminal may receive a manager's treatment instruction according to the user state image, and the virtual reality head mounted display may receive the treatment instruction through the network and output the treatment instruction for confirmation by the user.

The virtual reality head-mounted display or the manager terminal further includes an analysis module for analyzing the user face image or the user state image, and as a result of the analysis in the analysis module determines whether the user is evading treatment, avoiding treatment If it is determined that a warning sound or a direction sound can be output through the virtual reality head mounted display worn by the user.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, the administrator (caregiver, therapist, caregiver, caregiver, etc.) can intuitively check the user status by showing an image of the user's eyes and / or facial expressions through the front side, so that immediate response is possible. There is.

In addition, by conceptualizing and displaying information on the eyes and / or facial expressions of the user through the front surface, there is an effect that allows the administrator to intuitively check whether the VR content experience is properly performed.

1 is a schematic block diagram of a VR HMD showing a state of a user according to an embodiment of the present invention;
2 is a flow chart of a user status display method using a VR HMD showing a user's status according to an embodiment of the present invention;
3 and 4 are views of the VR HMD showing the state of the user from the back and front, according to an embodiment of the present invention,
5 is a view showing another example of the front display of the VR HMD;
6 is a view showing another example of the front display of the VR HMD;
7 illustrates an example of displaying a user state in a VR HMD showing a state of a user;
8 is a flowchart of a content management method using a VR HMD according to another embodiment of the present invention;
9 is a schematic structural diagram of a virtual reality head mounted display system showing a state of a user according to another embodiment of the present invention.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

In addition, the components of the embodiments described with reference to the drawings are not limited to the corresponding embodiments, and may be implemented to be included in other embodiments within the scope of the technical spirit of the present invention. Even if the description is omitted, it is obvious that a plurality of embodiments may be reimplemented into one integrated embodiment.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same or related reference numerals and redundant description thereof will be omitted. In the following description of the present invention, if it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

In addition, the terms “… unit”, “… unit”, “… module”, “… unit” described in the specification mean a unit that processes at least one function or operation, which means hardware or software or hardware and software. It can be implemented as a combination of.

1 is a schematic block diagram of a VR HMD showing a state of a user according to an embodiment of the present invention, Figure 2 is a user state display method using a VR HMD showing a state of a user according to an embodiment of the present invention 3 and 4 are views of the VR HMD from the back and the front, showing the state of the user according to an embodiment of the present invention, Figure 5 is a view showing another example of the front display of the VR HMD; 6 is a diagram illustrating still another example of the front display unit of the VR HMD, and FIG. 7 is an exemplary diagram showing a user state in the VR HMD showing the state of the user.

The VR HMD (Virtual Reality Head Mount Display) showing the state of the user (wearer) according to an embodiment of the present invention is an image corresponding to the image of the user's eyes and / or facial expressions or the result of analyzing the image through the front display. Display emoticons so that administrators can easily check user status.

The VR HMD 100 showing the state of the user according to the present embodiment includes a frame body 105, an internal display unit 110, a controller 120, a camera unit 130, a front display unit 140, and a signal processor ( 150). In addition, according to the exemplary embodiment, one or more of the sensor unit 160, the state output unit 170, and the operation unit 180 may be further included.

The frame body 105 is a body in which components such as the internal display 110, the controller 120, the camera 130, the front display 140, and the signal processor 150 are installed. Face) to be worn.

The back portion 106 of the frame body 105 may have a shape that covers the eye periphery of the user's face and corresponds to the face curvature of the user. An opening corresponding to the eyes of the user may be formed in the rear part 106, and an optical system including a lens may be installed in the opening.

The inside of the frame body 105 is installed so that the internal display unit 110 faces the rear side (ie, the rear side) in correspondence with the optical system. Therefore, when worn, the user can view audio-visual content such as VR content reproduced on the internal display unit 110 through the optical system of the rear portion.

The internal display 110 reproduces and displays audiovisual content such as VR content (step S210). For example, the internal display unit 110 may be a flat panel display such as an LCD, an LED, or an OLED that reproduces the transmitted video signal and displays it on the screen. In addition, the internal display unit 110 may also include a speaker for audio output.

The controller 120 controls the playback of audiovisual content through the internal display 110. The controller 120 determines the type of audiovisual content to be reproduced by the user or administrator input through the operation unit 180 or the communication unit to be described later, or according to a preset control algorithm, and reproduces or stops playing the determined audiovisual content.

In the present embodiment, the internal display unit 110 and the control unit 120 may be replaced by a smartphone, in which case the smart phone functioning as the internal display unit 110 and the control unit 120 may be detachably configured. . In this case, the frame body 105 may be designed as a removable structure of the smartphone, the removable smartphone configuration in the head mounted display is obvious to those skilled in the art to which the present invention belongs to the detailed description Omit.

The audiovisual content may be stored in advance in a separate storage unit (not shown). Alternatively, it may be transmitted in real time from a remote management server (not shown) through a communication unit (not shown).

The camera unit 130 is installed at the rear portion (ie, the eyepiece) of the frame body 105. The camera unit 130 acquires a user's face image by capturing a user's face including an eye of a user in contact with audiovisual content reproduced through the internal display unit 110 (step S220).

One camera unit 130 may be installed in the center of the rear portion. Alternatively, two may be provided, one for each position corresponding to the optical system of the rear portion so as to photograph the eyes of the user. Alternatively, three or more may be installed for more precise image acquisition.

The front display unit 140 is a display device installed in the front portion of the frame body 105, the user state image generated by image processing the user face image photographed by the camera unit 130 by a predetermined image processing algorithm, Can be displayed (step S230).

Through the user state image displayed on the front display unit 140, the administrator may intuitively check whether the user is currently viewing the audiovisual content, or the user reaction (user emotion state) with respect to the audiovisual content currently being played.

The signal processor 150 generates a user state image by processing the user face image photographed by the camera unit 130 in a format that can be displayed on the front display unit 140 according to a predetermined image processing algorithm.

The front display unit 140 may be composed of one display as shown in FIG. 4 or two displays 140a and 140b as shown in FIG. 5.

When the number of the camera unit 130 and the number of the front display unit 140 are the same, the signal processor 150 may include a front display unit in which an image captured by an arbitrary camera unit 130 corresponds to the corresponding camera unit 130. It may be displayed immediately through (140).

Here, the user state image may simply be an image that scales up, scales down, or removes noise appropriately to fit the size of the front face display unit 140.

Alternatively, the signal processor 150 may generate a user state image with an animation conceptualized based on the information obtained from the user face image. For example, if the user opens his eyes, the animation is open. If the user closes his eyes, the animation is closed. The animation can be replaced with an animation representing the facial expression corresponding to the determined emotional state of the user. It is. This is an image processing algorithm in consideration of the fact that a user may be reluctant to directly display a face including his / her eyes on the front display unit 140.

Alternatively, the front display unit may be compactly implemented (see 140c) on one side of the front side as shown in FIG. 6. This is implemented when the user is reluctant to expose his or her face.

In this case, the small front display unit 140c does not show the image photographed by the camera unit 130 but determines whether the eyes are opened or the emotional state according to the result of analyzing the image by the signal processor 150. An emoticon corresponding to (refer to (b) and (c) of FIG. 6) may be displayed.

Referring back to FIG. 1, the VR HMD 100 according to the present embodiment may further include a sensor unit 160 that measures additional information about a user.

The sensor unit 160 includes an EEG sensor for measuring an EEG of a user, a pupil measuring instrument for measuring a pupil size of the user, an eye tracker for tracking an eye of the user, and an HR of the user. ), A heart rate variability (HRV), a cardiac sensor for measuring electrocardiogram (ECG), and a skin sensor for measuring skin resistance (GSR) of the user.

The sensor unit 160 may acquire a biosignal such as an electroencephalogram (EGG), a heart rate (HR), a heart rate variability (HRV), an electrocardiogram (ECG), a pupil size, and a gaze direction (step S240).

The biosignal obtained by the sensor unit 160 may be processed into user additional information by the signal processor 150 (step S250), and may be displayed by being superimposed on the user state image displayed through the front display unit 140 ( Step S260).

Referring to FIG. 7A, an example in which patient information (eg, age), pupil size, pupil size change, etc. are superimposed on the front display unit 140 in addition to the user state image is illustrated.

Referring to FIG. 7B, an example in which patient emotion information, an EEG spectrum, etc. are superimposed on the front display unit 140 in addition to the user state image is illustrated.

In addition, the sensor unit 160 may measure an eye blink signal through an EEG sensor. The Ilink signal is a signal about blinking eyes.

The controller 120 may determine that the user closes his eyes when the i-link signal is not continuously detected. In this case, the front display unit 140 displaying the animation or the emoticon by controlling the signal processor 150 may display the current user's state (in particular, whether the eyes are closed). While the iBlink signal is detected, an animation or emoticon that is opening eyes may be displayed, and when the IBLINK signal is not detected for a predetermined time, the animation or emoticon with eyes closed may be displayed.

In addition, the VR HMD 100 according to the present embodiment may further include a state output unit 170 for outputting a user state according to a biosignal analysis result.

The state output unit 170 may be installed at the front of the frame body 105, and may include one or more of LEDs capable of expressing a plurality of colors and speakers capable of sound output.

If the analysis result of the bio-signal acquired by the sensor unit 160 indicates an abnormal state different from the expected state to be obtained through the content (eg, fear, anxiety, etc.), the LED lights a warning light for the administrator to recognize. Can be displayed through the speaker. The administrator may stop content currently playing or release the VR HMD 100 through a warning light or a warning sound.

Alternatively, the controller 120 may automatically switch or stop the content currently being played.

The analysis result of the biosignal may determine that the user is in an abnormal state in the following change situation.

EEG-High Beta, Gamma

Heart Rate (HR)-Increase

Heart rate variability (HRV)-increased RR, LF, HF, FL / HF

Skin Resistance (GSR)-Increase

Pupil Size-Increase

Eye movement-increased irregularity

The increase in the biosignal may be seen as an increase in interest up to a certain range, and may be interpreted as a surprise or anxiety when exceeding a threshold. The threshold may have individual differences for each user, and may be corrected in consideration of personal state information such as a user's disease, a normal state, and the like.

In the present embodiment, the control unit 120 is a user preset time (eg, about 15 seconds) as a result of analyzing the user's face image acquired through the camera unit 130 or as a result of analyzing the biosignal obtained through the sensor unit 160. When the eyes are closed continuously), the administrator may recognize the warning signal (a warning light or a warning sound) through the status output unit 170.

When the user closes the eyes for a predetermined time, the playback of the current audiovisual content does not cause the user to have the intended effect. This is to inform the administrator so that appropriate actions may be taken for the user. In this case, the warning signal output makes it easy to recognize even if the manager does not continuously monitor it, so that a small number of managers can care for a large number of users at the same time.

In addition, the VR HMD 100 according to the present embodiment may further include an operation unit 180 capable of a content related operation such as playing, stopping, and converting audiovisual content from the outside.

The operation unit 180 may be implemented as a button of hardware (HW) as shown in (a) of FIG. 7, or may be implemented as software (SW) as shown in (b) of FIG. 7 so that the front display unit ( 140 may be displayed.

The administrator simply presses a hardware button or a software button suitable for a content-related operation such as switching the audiovisual content to be played back, stopping the audiovisual content being played, or playing the paused audiovisual content according to a user state identified through the front display unit 140. The operation can care for the user.

This is because when an abnormal condition such as psychological instability occurs to the user while viewing the audiovisual content, the user of the physically uncomfortable may not be able to release the VR HMD 100 autonomously. It can be added by the administrator to help outside.

According to the present exemplary embodiment, whether the user continuously closes the eyes through the front display unit 140, what are the emotional states (eg, fear, surprise, fear, etc.) that the user feels about the currently playing VR content, and Administrators can intuitively grasp how the emotion level is recorded through the user state image captured by the camera unit 130 or through an animation or emoticon corresponding to the user state, so that an appropriate response can be immediately made according to the user state. To help.

In particular, in the case of VR content, the scope of application is expanding due to the high degree of reality and immersion in the treatment of mental disorders, long-term care patients or dementia patients. According to the present embodiment, the facial expression, eyes, etc. of the patient wearing the VR HMD 100 can be observed, so that immediate response is possible in case of psychological instability of the patient, and intensive care is possible even for the patient who is uncomfortable. Thus, there is an advantage that a small number of caregivers can care for a large number of patients at the same time.

8 is a flowchart of a content management method using a VR HMD according to another embodiment of the present invention.

The content is played through the internal display unit 110 (step S310).

The user wearing the VR HMD 100 watches the played content, and the camera unit 130 obtains the user's content response as an image (step S320). In addition, the sensor unit 160 may obtain a user's content response as a biosignal.

The acquired image and / or biosignal may be stored in association with the reproduced content for each time zone (step S330). The obtained image and / or biosignal may be represented in real time through the front display unit 140 (step S335).

The user's response to the corresponding content and the corresponding emotion are analyzed by analyzing the acquired image and / or biosignal (step S340).

The result of emotion analysis on the user's content is classified and determined according to a predetermined criterion (step S350). For example, the emotional state may be divided into engagement, boring, restful, fear, and the like.

The maintenance or switching of the content is determined according to the determined emotional state (step S360).

In addition, based on the determined emotional state, the user's preference and dislike of the corresponding content may be stored, and may be recommended or avoided when the content is changed later (step S370).

For example, it is assumed that the pupil size and additional user biometric information are used.

The camera unit 130 measures the pupil size. You can also identify the line of sight and measure your attention.

The sensor unit 160 may measure pulse wave (PPG), electrocardiogram (ECG), brain wave (EGG), and the like.

The signal processor 150 determines a significant biometric information change including the pupil size. This can be combined with pulse waves, electrocardiograms, brain waves, etc. to perform comprehensive judgment. In addition, the movement of the eye can also be used.

The controller 120 may record a time point or section of the corresponding content corresponding to a significant time point of biometric information change or generate a snapshot.

For example, in the case of PTSD treatment for firefighters, the time when the pupil becomes large (in addition, the pulse wave response also changes) is measured in a fire situation, and the image snapshot or the front and rear sections of the time point is saved as a video clip, and additionally, the time point Alternatively, the part where the subject's eyes are concentrated (for example, a curtain that catches fire) may be tracked in the section, and only the corresponding part may be marked.

Alternatively, in the case of caring for dementia patients using contents, a snapshot of a portion of the pupil size or a video clip before and after sections can be used for art therapy education after viewing the contents.

Alternatively, in the case of drug addiction treatment, anger disorder treatment, etc., the contents of the conversation at the time point can be extracted to find out the cause of anger for the sound by counseling, and then may seek further treatment.

Alternatively, in aversive treatment and reinforcement treatment using systematic desensitization, various stimuli and sensory states such as smell, sound, vision, and tactile sense of the section are selected to maximize the effect by selecting the type of sensation to be strengthened in subsequent treatment. Can pursue.

Through a separate storage unit, the corresponding snapshot corresponding to a significant bioinformation change time point may be filed, outputted, and made into DB so as to be utilized for later treatment or education.

It can support the application by file, output, and DB by using the source of sensory stimulus such as snapshot or video, sound, smell medicine, and tactile stimulus at the point of purpose.

For example, in the case of dementia, the output of the snapshot to be used for art therapy after the content care. In the case of PTSD, the video clip of the section where the reaction was strong and the marking of the eye focus point can be stored in the video. In the case of addiction treatment and anger disorder treatment, information on stimulus provision may be analyzed by DB and used for maximizing treatment later.

According to the present embodiment, an image and / or a biosignal of a user may be analyzed in real time, and may be switched to another content when the response to the content is not pleasant (eg, fear, boredom, etc.).

For a particular user, the user may determine the type of content that the user likes and recommend similar content.

By automatically switching and recommending such a content, the user can continuously enjoy the desired content without a separate operation, so that a few administrators can manage a large number of users.

The user status display method and / or content management method using the VR HMD according to the above-described embodiments may be embodied as computer readable codes on a computer readable recording medium. Computer-readable recording media include all kinds of recording media having data stored thereon that can be decrypted by a computer system. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like. The computer readable recording medium can also be distributed over computer systems connected over a computer network, stored and executed as readable code in a distributed fashion.

9 is a schematic structural diagram of a virtual reality head mounted display system showing a state of a user according to another embodiment of the present invention.

Referring to FIG. 9, a plurality of patients wear a virtual reality head mounted display, and a manager terminal 400 may be remotely connected to the plurality of virtual reality head mounted displays through a network such as the Internet.

The manager terminal 400 includes a terminal screen 420 through which a therapist (eg, a doctor, a nurse, a caregiver, etc.) in charge of treatment of a patient can check a patient's condition.

The virtual reality head mounted display acquires a user's facial image about the face (including eyes) of each user (patient). In addition, sensor information may be obtained through various attached sensors. The sensor information may include at least one biosignal among an EEG, a heart rate, a heart rate variability, an electrocardiogram, a pupil size, and a line of sight.

In this case, the image signal analysis or the biosignal analysis of the user's face image may be performed by the analysis module (not shown) of the virtual reality head mounted display which is the client end as described above. Alternatively, an image signal analysis and / or biosignal analysis may be performed through the analysis module 410 at the server terminal manager terminal 400.

The terminal screen 420 may express the state of each patient (the treatment subjects) at a glance. Referring to FIG. 9, it can be seen that patients 1 and 3 are eyes open, and patients 2 and 4 are closed eyes.

If it is determined that the current user's condition such as closing the eyes or turning his head, he / she can give micro treatment instruction to the user (patient). When the microphone icon 430 corresponding to the user is selected by clicking or touching, and the administrator speaks through a microphone provided in the manager terminal 400, the treatment instruction may be transmitted to the user only by voice.

Alternatively, the client side (virtual reality head mounted display) or the server side (administrator terminal 400) automatically detects the user's avoidance behavior, and outputs a warning sound or directs the user through the virtual reality head mounted display for the avoidance behavior. You can also output sound (eg, don't turn your head, keep your eyes closed, etc.).

Information about the eyes of the user may be transmitted as an image and represented as an image on the terminal screen 420 (see patient 1 and 3). Alternatively, the information transmitted as an image may be displayed in the form of an icon or an animation through the analysis module 410 (see patient 2 and 4). Alternatively, it may be interpreted at the client end and may be delivered to the server end as an icon instead of an image. As a result, problems such as privacy can be solved.

Although the above has been described with reference to embodiments of the present invention, those skilled in the art may variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. And can be changed.

100: VR HMD 105: frame body
110: internal display unit 120: control unit
130: camera unit 140: front display unit
150: signal processing unit 160: sensor unit
170: status output unit 180: operation unit

Claims (15)

In a virtual reality head mounted display (VR HMD) for allowing a user to enjoy audiovisual content played through an internal display installed inside a frame body worn by a head,
A camera unit installed at a rear portion of the frame body to acquire a user face image including the eyes of the user;
A signal processor configured to generate a user state image by image processing the user face image according to a predetermined image processing algorithm; And
The virtual reality head-mounted display is installed on the front of the frame body, and including a front display for displaying the user state image processed image.
The method of claim 1,
And the signal processor is configured to generate the user state image by scaling up or down the user face image to fit the size of the front display unit.
The method of claim 1,
And the signal processor generates the user state image with an animation conceptualized on the basis of the information obtained from the user face image.
The method of claim 1,
The front display unit is implemented to be compact on one side of the front portion,
The signal processor may generate an emoticon corresponding to a result of determining whether the user is open or an emotional state of the user according to a result of analyzing the user facial image as the user state image. display.
The method of claim 1,
Further comprising a sensor unit for obtaining at least one bio-signal of the user's brain wave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, eye direction,
And the signal processor processes the biosignal into user additional information so that the signal is superimposed on the user state image displayed through the front display.
The method of claim 5,
Further comprising a state output unit for outputting a user state according to the analysis result of the bio-signal,
The status output unit is a virtual reality head-mounted display, characterized in that it comprises one or more of a plurality of color capable LED and the speaker capable of sound output.
The method of claim 5,
And a control unit configured to determine an emotional state of the user according to an analysis result of the bio-signal or an analysis result of the user's face image, and to automatically switch or stop the audiovisual content currently being reproduced according to the determination result. display.
The method of claim 1,
Further comprising an operation unit capable of operation related to the content, including playback, stop, conversion of the audiovisual content,
The operation unit is a virtual reality head mounted display, characterized in that implemented as a hardware button on the front portion or a software button on the front display.
A user status display method performed in a virtual reality head mounted display for allowing a user to enjoy audiovisual content played through an internal display installed inside a frame body by wearing it on a head.
Acquiring a user's face image of the user's face through a camera unit installed at a rear portion of the frame body when the audiovisual content is reproduced;
Generating a user state image by image processing the user face image according to a predetermined image processing algorithm; And
And displaying the user status image on a front display unit installed in the front portion of the frame body.
The method of claim 9,
Acquiring at least one biosignal of an electroencephalogram (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, and gaze direction of the user; And
And processing the biosignal into user additional information so that the biosignal is superimposed on the user status image displayed through the front display unit.
A content management method performed in a virtual reality head mounted display for allowing a user to enjoy audiovisual content played through an internal display installed inside a frame body by wearing it on a head.
Acquiring at least one biosignal of an electroencephalogram (EGG), a heart rate (HR), a heart rate variability (HRV), an electrocardiogram (ECG), a pupil size, and a gaze direction when the audiovisual content is reproduced;
Analyzing the bio-signal and classifying a user response to the audiovisual content into one of a plurality of emotional states according to a predetermined criterion; And
And determining to maintain or switch the audiovisual content according to the discriminated emotional state.
The method of claim 11,
Storing preferences and aversions for the audio-visual content based on the discriminated emotional state,
And recommending or avoiding any audiovisual content based on the preferences and aversions when switching the audiovisual content.
A virtual reality head mounted display system that shows the user's status.
Virtual reality that allows the user to watch the audiovisual content played through the internal display installed inside the frame body, and obtains a user's face image including the user's eyes through a camera unit installed on the rear of the frame body. Head mounted display (VR HMD);
When connected to the virtual reality head-mounted display through a network to receive the user face image or a user state image corresponding to the user face image, and the user face image is received according to a predetermined image processing algorithm. And a manager terminal for converting the user status image into the user status image and outputting the user status image on the screen as it is.
The method of claim 13,
The manager terminal receives a treatment instruction of the manager according to the user state image,
The virtual reality head mounted display is a virtual reality head mounted display system characterized in that the user receives the treatment instruction via the network and outputs so that the user can confirm.
The method of claim 13,
The virtual reality head mounted display or the manager terminal further comprises an analysis module for analyzing the user facial image or the user state image,
The analysis module may determine whether the user is avoiding the treatment, and if it is determined that the treatment is avoided, the virtual sound may be output through a virtual reality head mounted display worn by the user through a warning sound or a direction sound. Realistic head mounted display system.

Images