KR102635074B1 - Head mounted display apparatus for providing object catching training based on virtual transplant of vr/ar technology, and operation method thereof - Google Patents

Abstract

The HMD device according to one embodiment includes an operation of acquiring information of a tracker worn on a first body, which is an amputated or paralyzed body part of a patient; An operation of outputting a virtual body image to the first body; An operation to output a virtual object; Controlling the shape or movement of the virtual body image based on information from the tracker; and an operation of determining whether the virtual body image makes a preset movement at the location where the object is being output.

Description

Rehabilitation HMD device providing VR/AR body image-based object catching training and its operation method {HEAD MOUNTED DISPLAY APPARATUS FOR PROVIDING OBJECT CATCHING TRAINING BASED ON VIRTUAL TRANSPLANT OF VR/AR TECHNOLOGY, AND OPERATION METHOD THEREOF}

The present invention relates to a rehabilitation HMD device that performs object grasping training based on VR/AR virtual body images and a method of operating the same.

The number of patients who have their arms or legs amputated due to diseases related to modern society, such as diabetes and Buerger's disease, is increasing every year. Likewise, 50% to 80% of amputees who have had their arms or legs amputated suffer from phantom pain. Phantom pain is a disease in which pain occurs as if a specific part of the body is present even though it is not physically present due to an accident or disease.

A representative treatment for phantom pain is mirror therapy. Mirror therapy is a treatment method that relieves phantom pain by having patients with amputated arms use a mirror to reflect the other arm and make them move their hands as if they had two arms. The effectiveness of this treatment has been proven in the medical community as a method of reducing phantom pain felt in the amputated area by giving amputees the visual illusion that the body in the amputated area is moving without pain.

However, mirror therapy has the disadvantage of being inconvenient to apply because it must be performed in a position that precisely matches the angle of the patient's gaze to the mirror, and that only a limited range of movement is possible right next to the mirror, which reduces realism.

In addition, the treatment method using a mirror was difficult to apply to patients with amputated lower limbs or paralyzed patients because it was difficult to realistically match the angle between the patient's gaze and the mirror.

Korean Patent Publication No. 10-200112: Motion recognition device using gyro sensor and motion recognition method thereof Korean Patent Publication No. 10-1980297: 3D VR image processing device, method, and program

The problem to be solved in the embodiment of the present invention is to create an illusion in patients with a visual effect that the body at the amputation area moves without pain, while maintaining the treatment concept of mirror therapy, which is to reduce phantom pain felt at the amputation area, while maintaining the treatment concept of mirror therapy. By implementing VR/AR technology that can perform various movements without a mirror, we provide technology that contributes to improving the effectiveness of treatment and rehabilitation training to relieve phantom pain in patients with limb amputation.

In particular, an embodiment of the present invention is a technology that helps patients' rehabilitation by providing a VR/AR-based object grasping simulation that allows patients to indirectly recognize their own body by controlling the movement of the virtual body visible through the HMD device. We would like to provide.

However, the technical problems to be achieved by the embodiments of the present invention are not limited to the problems mentioned above, and various technical problems can be derived from the contents described below within the range obvious to those skilled in the art.

An operation method performed by an HMD device according to an embodiment includes an operation of acquiring information of a tracker worn on a first body, which is an amputated or paralyzed body part of a patient; An operation of outputting a virtual body image to the first body; An operation to output a virtual object; Controlling the shape or movement of the virtual body image based on information from the tracker; and an operation of determining whether the virtual body image makes a preset movement at the location where the object is being output.

In addition, the information of the tracker includes information about a resistance value that changes in response to morphological deformation of the patient's ligaments or muscles, and the controlling operation is performed based on the information about the resistance value to create the virtual body image. It may include actions that control the shape or movement of.

Additionally, the operation of outputting the virtual object includes: setting a plurality of locations where the object will be created; An operation of creating a virtual object in at least one location among the plurality of locations; and an operation of changing the center point of the virtual object or the size of the object.

In addition, the controlling operation includes controlling the virtual body image to have a first shape having a grasping hand shape, and the determining operation includes the center point when the object has a preset size. When the virtual body image is located in a position and the shape of the virtual body image becomes the first shape, it may include an operation of determining that the object has been caught.

Additionally, the controlling operation may include sensing the shape or movement of a second body, which is a body part symmetrical to the first body; And based on the sensed information, it may include controlling the virtual body image to have a shape or movement that is left and right reversed from the shape or movement of the second body.

In addition, the determining operation includes an operation of determining that the object has been caught when the object becomes a preset size and both the virtual body image and the second body are located within a preset distance from the center point. It can be included.

Additionally, the operation of outputting the virtual object includes: setting a plurality of locations where the object will be created; Creating a first object and a second object at two or more locations among the plurality of locations; and an operation of changing the center point and size of each of the first object and the second object, wherein the controlling operation includes controlling the virtual body image to have a first shape having a grasping hand shape. And the determining operation includes: sensing the shape or movement of a second body, which is a body part symmetrical to the first body; And when the first object has a size corresponding to the first catch area information, the center point of the first object is located within the first catch area information and the shape of the virtual body image becomes the first shape. , When the second object becomes a size corresponding to the second catch area information, the center point of the second object is located within the second catch area information and the second body is sensed as being positioned and in a grasping form. In one case, it may include an operation of determining that the first object and the second object have been caught.

Additionally, the output operation includes: outputting first catch area information of a preset size to the area where the virtual body image is located; and an operation of outputting second catch area information of a preset size to the area where the second body is located, wherein the determining operation is performed when the first object has a size corresponding to the first catch area information. , When the center point of the first object is located within the first catch area information, the shape of the virtual body image becomes the first shape, and the second object has a size corresponding to the first catch area information. , If the center point of the second object is located within the second catch area information and the second body is positioned and takes a grabbing form, it may include an operation of determining that the first object and the second object have been caught. You can.

In addition, the determination operation is based on the following equation 1 using the area (S1) of the first object and the area (S2) of the first catch area at the time when the virtual body image is controlled to the first shape. An operation of calculating and outputting the catch accuracy may be further included.

[Equation 1]

Catch Accuracy = (%)

(S1: Area of the first object at the time when the virtual body image is controlled to the first shape, S2: Area of the first catch area)

Additionally, the controlling operation may include sensing the shape or movement of a second body, which is a body part symmetrical to the first body; And an operation of controlling the virtual body image to have a shape or movement of the second body and a left and right reversed shape or movement based on the sensed information, wherein the operation of determining is the operation of determining whether the object has a preset size. When , if both the virtual body image and the second body are located within a preset distance from the location of the virtual object, it may include an operation of determining that the object has been caught.

An HMD device according to an embodiment includes one or more memories that store instructions for performing a predetermined operation; and one or more processors operably connected to the one or more memories and configured to execute the instructions. and a display that outputs a virtual body image to the user, wherein the processor's operations include: acquiring information on a tracker worn on a first body, which is an amputated or paralyzed body part of the patient; An operation of outputting a virtual body image to the first body; An operation to output a virtual object; Controlling the shape or movement of the virtual body image based on information from the tracker; and an operation of determining whether the virtual body image makes a preset movement at the location where the object is being output.

According to an embodiment of the present invention, a virtual body can be implemented on the amputated part of the patient's body, and in this case, the implemented virtual body is implemented to have a virtual shape or movement based on information from a tracker worn on the amputated part of the body. , or it can be implemented to have movements that are symmetrical to those of an unamputated body. Through this, the patient can directly visually see the movements of the virtual body realized through virtual reality and control the movements, allowing the patient to perceive the body as if it exists at the amputation site, thus allowing treatment similar to mirror therapy. .

In addition, patients can perceive a virtual body image in front of their eyes from the same viewpoint as in real life through a Head Mounted Display (HMD) device, allowing them to perceive it vividly from a natural viewpoint like a real body.

Accordingly, the patient can move the virtual body like a real arm and leg through the idea of moving the body at the amputation site, and can visually perceive the body's movement realistically, which can relieve phantom pain and provide rehabilitation. The effectiveness of training can be improved.

In addition, the patient's movements can be implemented so that the tracker senses the detailed movements of the nerves and muscles of the actual patient's arms or legs, so that the movements of the virtual body more accurately reflect the movements of the actual body.

In addition, by providing simulations that include various missions that patients can perform with their virtual bodies, treatment content that combines interesting elements can be provided to patients.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a configuration diagram of a rehabilitation treatment system according to an embodiment of the present invention.
Figure 2 is a functional block diagram of an HMD device according to an embodiment of the present invention.
Figure 3 is an exemplary diagram showing a patient using a rehabilitation treatment system according to an embodiment of the present invention.
Figure 4 is an exemplary diagram showing the operation of using a rehabilitation treatment system according to an embodiment of the present invention.
Figure 5 is a flowchart of an operation method performed by an HMD device according to an embodiment of the present invention.
FIGS. 6 to 8 are examples showing an operation in which the HMD device 200 outputs a virtual object and the patient catches the virtual object by moving the virtual body image or the second body, respectively, according to an embodiment. am.
FIGS. 9 and 10 are example diagrams illustrating an operation in which the HMD device 200 outputs a virtual object and the patient catches the virtual object by moving the virtual body image and the second body together, according to an embodiment. am.

The advantages and features of the present invention, and methods for achieving them, will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. However, the present embodiments are intended to ensure that the disclosure of the present invention is complete, and those skilled in the art It is provided to fully inform the person of the scope of the invention, and the scope of the invention is only defined by the claims.

In describing embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted unless actually necessary. The terms described below are terms defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

The functional blocks shown in the drawings and described below are only examples of possible implementations. Other functional blocks may be used in other implementations without departing from the spirit and scope of the detailed description. Additionally, although one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

Additionally, the expression that it includes certain components is an open expression and simply refers to the existence of the components, and should not be understood as excluding additional components.

Furthermore, when it is mentioned that a component is connected or connected to another component, it should be understood that although it may be directly connected or connected to the other component, other components may exist in between.

In addition, expressions such as 'first, second', etc. are expressions used only to distinguish a plurality of configurations, and do not limit the order or other characteristics between the configurations.

The system 10 according to an embodiment of the present invention creates the illusion in patients with a visual effect that the body at the amputated area moves without pain, thereby reducing the phantom pain felt at the amputated area, while maintaining the treatment concept of mirror therapy, By implementing a treatment method that allows patients to perform various movements without a mirror, we provide technology that contributes to improving the effectiveness of treatment and rehabilitation training to relieve phantom pain in patients with limb amputation.

The system 10 according to an embodiment of the present invention outputs a virtual body image 11 to the amputated part of the patient's body through VR (Virtual Reality) or AR (Augmented Reality) technology, and displays the virtual body image 11 obtained from the patient's body. Based on the information, the shape or movement of the virtual body image 11 can be implemented. To this end, the components included in the system 10 according to an embodiment of the present invention and the operation of each component will be described with drawings.

Figure 1 is a configuration diagram of a system 10 according to an embodiment of the present invention.

Referring to FIG. 1, a system 10 according to an embodiment of the present invention may include a tracker 100 and a Head Mounted Display (HMD) device 200.

The tracker 100 is a device that is attached to or worn on the patient's body and performs wired and wireless communication with an external device. The tracker 100 may include a physical mark that allows the HMD device 200 to specify an end of the first body, which is a paralyzed or amputated body part of the patient, or a wearable device worn on the first body. there is. For example, wearable devices may include motion capture gloves, VR smart gloves, and motion recognition gloves, and the wearable device may be worn on a part of the patient's body, such as the patient's hands, arms, feet, or ankles.

The tracker 100 is worn on one end of the patient's first body, senses the movement of muscles or ligaments that change when the patient tries to move the amputated body, generates information about the movement, and provides information about the movement. Based on this, the movement of the patient's intended amputation site can be determined. For example, the tracker 100 may include a sensor capable of sensing a change in the shape of the surface through a resistance value that changes as the surface increases or decreases. The tracker 100 may generate information about the shape or movement of the first body sensed by the sensor and transmit it to the HMD device 200 through a wired or wireless network.

The HMD device 200 is a device that can be worn like glasses around the patient's eyes and outputs an image containing virtual information to the patient. The HMD device 200 of an embodiment of the present invention may include a form such as a Face Mounted Display (FMD) or Smart Glasses.

Figure 2 is a functional block diagram of an HMD device 200 according to an embodiment of the present invention. Referring to FIG. 2 , the HMD device 200 according to an embodiment of the present invention may include a memory 210, a processor 220, a display 230, and a camera 240.

The memory 210 may store instructions for operating the processor 220 and data required for the operation of the processor 220.

The processor 220 may perform various data processing and calculations, including a central processing unit (CPU). The processor 220 may generate a virtual body image 11 of the patient based on information sensed by the tracker 100 or the camera 240.

The display 230 may output an image including the virtual body image 11 to the patient. The display 230 may output the virtual body image 11 generated by the processor 220 on one end of the patient's first body along with a virtual reality (VR) or augmented reality (AR) image.

The camera 240 may include a plurality of camera modules and generate 3D depth information of the real object by calculating depth values of image pixels obtained from the real object. The camera 240 can photograph the patient's hands, arms, feet, ankles, etc. For example, the camera 240 may photograph the second body 12, which is a body part symmetrical to the first body corresponding to the amputated or paralyzed body part of the patient, and the second body 12 according to the patient's movement. ) can sense the shape and movement of

Figure 3 is an exemplary diagram showing a patient using the system 10 according to an embodiment of the present invention.

Referring to Figure 3, the patient in Figure 3 has the first body (the left hand of the person in Figure 3) amputated. Accordingly, in order to use the system 10 according to an embodiment of the present invention, the patient may display a mark as a tracker 100 near the cut surface of the first body, or may wear a wearable device. Additionally, the patient may wear the HMD device 200 around the eyes.

Figure 4 is an exemplary diagram showing operations using system 10 according to one embodiment of the present invention.

Referring to FIG. 4, the HMD device 200 generates a virtual body image 11 in the form of a VR image or AR image at the location of the first body, which is the amputated or paralyzed body part of the patient, and displays it on the display 230. It can be visually output to the patient.

According to one embodiment, the processor 220 of the HMD device 200 may output a virtual body image 11 to the first body, and the patient may attempt to move the first body based on the information of the tracker 100. By determining the intended motion, the shape or movement of the virtual body image 11 can be controlled. For example, the information of the tracker 100 may include information about the resistance value that changes in response to the morphological deformation or movement of the patient's ligaments or muscles, and the HMD device 200 may include information about the resistance value. Based on this, the shape or movement of the virtual body image 11 can be controlled. As an example, the tracker 100 may include a plurality of strain gauges that output corresponding resistance values according to the morphological deformation of the patient's muscles or ligaments, and the user can measure the resistance values based on the resistance value patterns of the plurality of strain gauges. The intended operation can be determined. The MD device 200 may reflect and output the movement corresponding to the virtual body image 11 based on the shape of the movement determined by the tracker 100. Accordingly, even if the patient's body is amputated, the patient can control the movement of the virtual body image 11 through the movement of the ligaments and can visually confirm the movement of the virtual body image 11.

According to one embodiment, the processor 220 of the HMD device 200 senses the shape or movement of the second body 12, which is a body part symmetrical to the first body, and detects the second body (12) based on the sensed information. The virtual body image 11 can be controlled to have the shape or movement of 12) with the left and right sides reversed.

Figure 5 is a flowchart of an operation method performed by the HMD device 200 according to an embodiment of the present invention. Each step disclosed in FIG. 5 may be performed through the HMD device 200 according to an embodiment, and specifically, the operation of each step may be understood as an operation performed by the processor 220.

Each step disclosed in FIG. 5 is only a preferred embodiment in achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be performed by being included in another step. The order of each operation disclosed in FIG. 5 is merely an order arranged for convenience of understanding, and this order is not limited to a time-serial order, and the order may be changed and operated differently depending on the designer's selection.

Referring to FIG. 5 , in step S1010, the HMD device 200 may acquire information on the tracker 100 worn on the first body, which is an amputated or paralyzed body part of the patient. For example, the information on the tracker 100 may include the example tracker 100 information described with FIGS. 3 and 4 .

In step S1020, the HMD device 200 may output the virtual body image 11 to the first body. For example, the operation of step S1020 may be performed as the example operation described with FIGS. 3 and 4.

In step S1030, the HMD device 200 may output a virtual object 1 (ex. baseball, soccer ball, etc.) on the display 230 in the form of a VR image or an AR image.

In step S1040, the HMD device 200 may control the shape or movement of the virtual body image 11 (e.g., a movement of opening a hand, a movement of clenching a hand, etc.) based on the information of the tracker 100. .

In step S1050, the HMD device 200 may determine whether the virtual body image 11 makes a preset movement at the location where the virtual object 1 is being output. Additionally, the HMD device 200 can sense the movement of the second body 12 and perform additional judgment using the sensed information.

The operations of steps S1030 to S1050 will be examined in detail with the drawings below.

6 to 8 show that, according to an embodiment, the HMD device 200 outputs a virtual object 1, and the patient moves the virtual body image 11 or the second body 12, respectively, to create a virtual object ( This is an example to explain the operation of catching 1).

First, as a first embodiment, an embodiment of catching a virtual object 1 through a virtual body image 11 will be described with FIGS. 6 to 8.

Referring to FIG. 6, the HMD device 200 can set a plurality of locations where the virtual object 1 will be created. For example, the HMD device 200 may create a virtual object in at least one position among the six rectangular areas indicated by dotted lines in FIG. 6 .

Referring to FIG. 7, the HMD device 200 can control the position and size of the virtual object 1. For example, the HMD device 200 can output the virtual object as if it is approaching the patient by moving the center point of the virtual object toward the patient and controlling the size of the virtual object to gradually increase.

Referring to FIG. 8, when the virtual object 1 becomes a preset size, the HMD device 200 positions the virtual body image 11 at the position of the virtual object 1, If the form in (11) becomes the first form (ex. the form of grasping the hand), it can be determined that the patient has caught the virtual object.

Next, as a second embodiment, the virtual first object 1 is caught through the virtual body image 11, and the virtual second object 2 is caught through the second body 12. will be described with FIGS. 6 to 8.

Referring to FIG. 6, the HMD device 200 can set a plurality of locations where objects will be created. For example, the HMD device 200 may create a virtual first object 1 and a virtual second object 2 in two or more positions among the six rectangular areas indicated by dotted lines in FIG. 6 . At this time, the HMD device 200 may generate the virtual first object 1 and the virtual second object 2 with a preset time difference.

Referring to FIG. 7 , the HMD device 200 can sense the shape or movement of the second body 12, which is a body part symmetrical to the first body, through the camera 240. The HMD device 200 can determine the movement of the second body 12 and the movement of the virtual body image 11 based on the sensed information. At this time, the HMD device 200 outputs first catch area information (A) of a preset size to the area where the virtual body image 11 is located, and outputs first catch area information (A) of a preset size to the area where the sensed second body 12 is located. The size of the second catch area information (B) can be output.

At this time, the HMD device 200 may control the positions of each of the virtual first object 1 and the virtual second object 2 and the size of the virtual objects. For example, the HMD device 200 moves the center point of the virtual first object 1 toward the left side of the patient's body and controls the size of the virtual first object 1 to gradually increase, thereby 1 Object (1) can be output as if it is approaching toward the left side of the patient. The HMD device 200 moves the center point of the virtual second object 2 toward the right side of the patient's body and controls the size of the virtual second object 2 to gradually increase, thereby making the virtual second object 2 ) can be output as if it is approaching toward the right side of the patient. The HMD device 200 controls the speed at which the sizes of the virtual first object 1 and the virtual second object 2 change at different rates, so that the virtual first object 1 and the virtual second object 2 ) can be controlled so that the speed at which it approaches the patient feels different.

Referring to FIG. 8, when the virtual first object 1 becomes a preset size, the HMD device 200 positions the virtual body image 11 at the position of the virtual first object 1 and creates a virtual body image 11. When the shape of the body image 11 is controlled to the first shape (ex. grasped shape) and the virtual second object 2 becomes a preset size, it is placed at the position of the virtual second object 2. 2 When the body 12 is positioned and takes a grasping form, it can be determined that the patient has caught the virtual first object 1 and the virtual second object 2.

For example, when the virtual first object 1 becomes a size corresponding to the first catch area information A, the HMD device 200 determines that the center point of the virtual first object 1 is the first catch area. While located within the information (A), the shape of the virtual body image (11) is controlled to a first shape (ex. grasping shape), and the virtual second object (2) corresponds to the second catch area information (B). When the size is reached, when the center point of the virtual second object 2 is located within the second catch area information B and the second body 12 is sensed to be in a grasping form, the patient It can be determined that the first object (1) and the virtual second object (2) have been caught. At this time, when the catch is determined by comparing the size of the catch area information and the size of the virtual object, a preset amount of error may be allowed when comparing the sizes.

In addition, when the user controls the virtual body image 11 to be in the first shape, the HMD device 200 determines the size of the catch area information (A or B) and the virtual object (first object or second object). By outputting the catch accuracy based on the difference in size, the user can be guided to control his or her body more precisely. For example, the HMD device 200 calculates and outputs catch accuracy based on the difference between the size of the first object at the time when the virtual body image is controlled to the first shape and the size corresponding to the first catch area information. can do. As an example, the HMD device 200 calculates the area S1 of the first object 1 and the first catch area A at the time when the virtual body image is controlled to the first shape, as shown in Equation 1 below: Catch accuracy can be calculated based on the area (S2).

[Equation 1]

Catch Accuracy = (%)

The embodiment of Equation 1 described above is the area S1 of the second object 2 and the area of the second catch area B at the determined point in time when the second body 12 is determined to be in the shape of grasping the hand. The same applies to (S2), so the HMD device 200 can calculate the catch accuracy for the second body.

9 and 10 show an operation in which the HMD device 200 outputs a virtual object and the patient moves the virtual body image 11 and the second body 12 together to catch the virtual object, according to an embodiment. This is an example to explain.

9 and 10, an embodiment of catching a virtual object 1 using the virtual body image 11 and the second body 12 will be described.

Referring to FIG. 9, the HMD device 200 senses the shape or movement of the second body 12, which is a body part symmetrical to the first body, and determines the shape or movement of the second body 12 based on the sensed information. The virtual body image 11 can be controlled to have a shape or movement with the left and right sides reversed.

Referring to FIG. 10, when the virtual object 1 becomes a preset size, the HMD device 200 displays the virtual body image 11 and If all of the sensed second bodies 12 are located within a preset distance, it can be determined that the virtual object 1 has been caught.

According to the above-described embodiment, a virtual body image 11 can be implemented on the amputated part of the patient's body, and the implemented virtual body image 11 is based on the information of the tracker 100 worn on the amputated part of the body. It may be implemented to have a virtual shape or movement, or may be implemented to have a movement that is symmetrical to the movement of an unamputated body. Through this, the patient can directly visually see the movements of the virtual body realized through virtual reality and control the movements, allowing the patient to perceive the body as if it exists at the amputation site, thus allowing treatment similar to mirror therapy. .

In addition, since the patient perceives a virtual body image 11 in front of his eyes from the same viewpoint as in real life through a Head Mounted Display (HMD) device, he can perceive it vividly from a natural viewpoint like a real body.

Accordingly, the patient can move the virtual body like a real arm and leg through the idea of moving the body at the amputation site, and can visually perceive the body's movement realistically, which can relieve phantom pain and provide rehabilitation. The effectiveness of training can be improved.

In addition, the patient's movements are implemented so that the tracker 100 senses the detailed movements of the nerves and muscles of the actual patient's arms or legs, so that the movements of the virtual body can be implemented to more accurately reflect the movements of the actual body. there is.

In addition, by providing simulations that include various missions that patients can perform with their virtual bodies, treatment content that combines interesting elements can be provided to patients.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include a single item or a plurality of items, unless the relevant context clearly indicates otherwise.

In this document, “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “A Each of the phrases such as “at least one of , B, or C” may include all possible combinations of the items listed together in the corresponding phrase. Terms such as "1", "second", or "first" or "second" may be used simply to distinguish one element from another and may refer to those elements in other respects, such as importance or order. ) is not limited to. One (e.g. first) component is said to be "coupled" or "connected" to another (e.g. second) component, with or without the terms "functionally" or "communicatively". Where mentioned, it means that one component can be connected to another component directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document may be implemented as software (e.g., program) including one or more instructions stored in a storage medium (e.g., memory) that can be read by a device (e.g., electronic device). Storage media may include random access memory (RAM), memory buffers, hard drives, databases, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and/or the like. It can be included.

Additionally, the processor of the embodiments of this document may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to at least one instruction called. These one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The processor may be a general-purpose processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), and/or the like.

A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium and data stored temporarily. There is no distinction between cases.

Methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play Store) or on two user devices (e.g. : Smartphones) can be distributed (e.g. downloaded or uploaded) directly or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

According to various embodiments, each component (eg, module or program) of the described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as that performed by the corresponding component of the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or , or one or more other operations may be added.

Claims (10)

Memory for storing instructions; and a processor that performs a predetermined operation according to the command. In an operation method performed by an HMD device including a display that outputs a VR (Virtual Reality) or AR (Augmented Reality) image,
The operation of the processor is,
An operation of acquiring information from a tracker worn on a first body, which is an amputated or paralyzed body part of a patient;
An operation of outputting a virtual body image in the form of a VR image or an AR image to one end of the first body visible through the display;
An operation of outputting a virtual object in the form of a VR image or an AR image to at least one location visible through the display, wherein the virtual object is output to move toward the patient.
Controlling the shape or movement of the virtual body image based on information from the tracker;
An operation of sensing the shape or movement of a second body, which is a body part symmetrical to the first body, through the display; and
At the location where the virtual object is being output, the patient moves the virtual body image and the second body together to make a preset movement, or takes a preset shape with any one of the virtual body image and the second body. Including an operation of determining whether or not the virtual object is caught using different criteria depending on whether the virtual object is caught,
When determining whether or not to catch a catch based on whether the virtual body image and the second body are moved together to make the preset movement,
The above discriminating operation is,
When the virtual object becomes a preset size, and when it is determined that both the virtual body image and the second body are located within a preset distance from the position of the center point of the virtual object, the virtual object is caught. Includes an operation to determine that
When determining whether or not to catch a catch based on whether one of the virtual body image and the second body takes a preset shape,
The operation of outputting the virtual object is,
An operation of outputting a virtual first object and a virtual second object through the display at two or more locations with a preset time difference,
The above discriminating operation is,
Setting a first catch area of a preset size in an area where the virtual body image is located and a second catch area of a preset size in an area where the second body is located;
When the first object becomes a preset size, the virtual body image is located at a position including the center point of the first object, and the shape of the virtual body image becomes a predetermined first shape, the first object It is determined that the object has been caught, and when the second object becomes a preset size, the second body is located at a position including the center point of the second object and takes a predetermined first shape, and the second object is An operation to determine that has been caught; and
The catch accuracy of the first body is calculated by comparing the area of the first object and the area of the first catch area at the time when the virtual body image is controlled to the first shape, and the second body is Comprising the operation of calculating the catch accuracy of the second body by comparing the area of the second object at the time of taking shape 1 and the area of the second catch area,
How the HMD device operates.
According to paragraph 1,
The tracker information is,
Contains information about resistance values that change in response to movements of the patient's ligaments or muscles,
The controlling operation is,
Comprising an operation of controlling the shape or movement of the virtual body image based on the information about the resistance value,
How the HMD device operates.
According to paragraph 1,
The operation of outputting the virtual object is,
An operation to set a plurality of locations;
creating a virtual first object and a virtual second object at two or more locations among the plurality of locations; and
Including the operation of changing the size and position of the center point of each of the first object and the second object, and controlling the speed at which the size of the first object and the second object change at different rates,
How the HMD device operates.
According to paragraph 1,
The first form is
Includes a clenched hand shape,
The controlling operation is,
Comprising an operation of controlling the virtual body image to have the first shape having a grasping hand shape based on the information of the tracker,
How the HMD device operates.
delete delete According to paragraph 1,
The calculation operation is,
Calculate and output catch accuracy based on Equation 1 below using the area (S1) of the first object and the area (S2) of the first catch area at the time when the virtual body image is controlled to the first shape. Including the action of
[Equation 1]
Catch Accuracy = (%)
(S1: Area of the first object at the time when the virtual body image is controlled to the first shape, S2: Area of the first catch area)
How the HMD device operates.
delete A computer-readable recording medium on which a computer program including instructions for causing a processor to perform any one of the methods of claims 1 to 4 and 7 is recorded.
One or more memories that store instructions for performing predetermined operations; and one or more processors operably connected to the one or more memories and configured to execute the instructions. and a display that outputs VR (Virtual Reality) or AR (Augmented Reality) images,
The operation of the processor is
An operation of acquiring information from a tracker worn on a first body, which is an amputated or paralyzed body part of a patient;
An operation of outputting a virtual body image in the form of a VR image or an AR image to one end of the first body visible through the display;
An operation of outputting a virtual object in the form of a VR image or an AR image to at least one location visible through the display, wherein the virtual object is output to move toward the patient.
Controlling the shape or movement of the virtual body image based on information from the tracker;
An operation of sensing the shape or movement of a second body, which is a body part symmetrical to the first body, through the display; and
At the location where the virtual object is being output, the patient moves the virtual body image and the second body together to make a preset movement, or takes a preset shape with any one of the virtual body image and the second body. Including an operation of determining whether the virtual object is caught using different criteria depending on whether the virtual object is caught,
When determining whether or not to catch a catch based on whether the virtual body image and the second body are moved together to make the preset movement,
The above discriminating operation is,
When the virtual object becomes a preset size, and when it is determined that both the virtual body image and the second body are located within a preset distance from the position of the center point of the virtual object, the virtual object is caught. Includes an operation to determine that
When determining whether or not to catch a catch based on whether one of the virtual body image and the second body takes a preset shape,
The operation of outputting the virtual object is,
An operation of outputting a virtual first object and a virtual second object through the display at two or more locations with a preset time difference,
The above discriminating operation is,
Setting a first catch area of a preset size in an area where the virtual body image is located and a second catch area of a preset size in an area where the second body is located;
When the first object becomes a preset size, the virtual body image is located at a position including the center point of the first object, and the shape of the virtual body image becomes a predetermined first shape, the first object It is determined that the object has been caught, and when the second object becomes a preset size, the second body is located at a position including the center point of the second object and takes a predetermined first shape, and the second object is An operation to determine that has been caught; and
The catch accuracy of the first body is calculated by comparing the area of the first object and the area of the first catch area at the time when the virtual body image is controlled to the first shape, and the second body is Comprising the operation of calculating the catch accuracy of the second body by comparing the area of the second object at the time of taking shape 1 and the area of the second catch area,
HMD device.