KR102689063B1 - System and method for effectively providing analysis results on the user's exercise behavior in visual aspect to support decision-making related to exercise and medical services - Google Patents

Abstract

Embodiments include obtaining a personal training video image representing at least one movement motion of a personal training object, analyzing the movement movement of the personal training object in the personal training video image, and generating metadata of the personal training video image, Receive a video image of the user in which the user's exercise motion is captured from the server, analyze the user's exercise motion in the video image of the user to generate exercise state information of the user, and generate exercise state information of the user and the exercise motion of the user In order to compare the corresponding exercise actions of the personal training object, the user's exercise state information is compared with the corresponding exercise state information of the personal training object to generate the user's exercise result information, and the user's exercise state information and analysis of the user's exercise movements to support decision-making regarding exercise and medical services, generating analysis data on the user's exercise movements based on at least one piece of information included in the user's exercise result information. It relates to systems and methods that effectively present results in a visual aspect.

Description

System and method for effectively providing analysis results on the user's exercise behavior in visual aspect to support decision- making related to exercise and medical services}

Embodiments of the present application provide the user's exercise status and exercise results in the form of images that visually express some or all of the user's exercise behavior based on the user's body shape to support decision-making regarding exercise and medical services. It relates to a system and method that effectively provides analysis results in a visual aspect.

Recently, as interest in health and medical care has increased, healthcare services have been attracting attention. In order to provide higher quality healthcare services, it is necessary to have abundant information about the service target.

Starting in 2021, the My Data service has been launched, which allows information subjects to make decisions about access, storage, and utilization of information created while individuals use services such as administration, financial transactions, medical care, communication, and education. Accordingly, medical my data projects are also being attempted.

Medical MyData is not limited to medical services provided by medical institutions and includes several items. Among them, medical support projects also include non-medical institution services such as medical support through exercise.

Unlike the past, modern society's interest in health has greatly increased, so the number of people exercising is expected to increase, and the development of the healthcare industry through exercise is also expected.

However, the current medical mydata project is focused on standardizing and collecting medical data such as diagnosis results and test results generated when patients visit medical institutions, among various medical information items. Regarding exercise data that can be generated in an environment where the information subject (e.g., patient) does not visit a medical institution, such as exercise, progress in the My Data project is insufficient.

Therefore, standardization technology is needed for exercise data. In addition, beyond standardizing exercise data, technology is needed to utilize exercise data to support decision-making related to musculoskeletal exercise or appropriate medical service-related decision-making.

Patent Registration Publication No. 10-1864987 (announced on May 30, 2018)

In order to solve the above-described problem, embodiments of the present application obtain standardized user's exercise state and exercise results from the user's exercise image, and some or all of the user's exercise state and exercise results are based on the user's body shape. By providing it in the form of a visually expressed image, we aim to provide a system and method that effectively provides analysis results of the user's exercise movements in a visual aspect to support decision-making related to exercise and medical services.

In order to support decision-making related to musculoskeletal exercise and medical services according to an aspect of the present application, a method of analyzing a user's exercise motion and effectively providing it in a visual aspect may be performed by a server. The method obtains a personal training video image representing at least one movement motion of a personal training object from a server, analyzes the movement movement of the personal training object in the personal training video image, and generates metadata of the personal training video image. generating step; Receiving a video image of the user in which the user's exercise motion is captured from the server, wherein the user's exercise motion is the user's exercise among at least one exercise motion of the personal training object shown in the personal training video image selected by the user imitates the motor movement corresponding to the movement; Generating posture information of the user by analyzing the user's exercise movements in the video image of the user, and generating exercise state information based on the posture information of the user; In order to compare the exercise motion of the user and the corresponding exercise motion of the personal training object, generating exercise result information of the user by comparing the exercise state information of the user and the corresponding exercise state information of the personal training object. Step - the corresponding exercise state information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's exercise state information; and generating analysis data about the user's exercise movement based on at least one of information included in the user's exercise state information and the user's exercise result information - the analysis data is displayed on the user's electronic device. Used to create analysis screens that are may include.

In one embodiment, the step of generating metadata of the personal training video image includes classifying a set of frame images in the personal training video image by scene to form a subset of frame images associated with the scene, the subset being comprising at least one series of frame images, wherein the series of frame images are continuous frame images representing unit operations of the personal training object; generating posture information of the personal training object appearing in the scene by processing frame images of the personal training video image in subset units; generating exercise state information of the training object based on the posture information of the personal training object; Obtaining exercise result information of the personal training object in the personal training video image based on at least one of operator input information of the server and exercise state information of the personal training object; And it may include generating metadata of the personal training video image including some or all of the exercise state information of the personal training object.

In one embodiment, the step of generating posture information of the personal training object includes selecting at least one posture required to identify the motion as a key pose among a series of postures constituting the motion. step; and generating key posture information describing some postures selected as key postures among a series of postures as posture information of the personal training object. The step of selecting the at least one posture as the key posture may include selecting at least one posture of the unit action as the key posture for the unit action based on the position and movement information of the skeleton and the position and movement information of the joints. there is. The exercise state information of the personal training object may be generated based on the key posture information of the personal training object.

In one embodiment, the step of generating posture information of the personal training object includes analyzing the input image in which the input object appears using a pre-learned pose recognition model to recognize the type of posture to which the posture the input object is taking. , it may be to generate posture information of the object including the recognized posture type. In addition, the pose recognition model extracts feature points of an object in an input posture through a pose estimation algorithm, and provides at least one of joint movement and skeletal motion information of the object based on the extraction result of the feature points of the object. Calculates motion information, and infers the correlation between at least one of the calculated joint motion and skeletal motion information of the object and the posture type of the input object to determine the probability that the input posture will be classified into each of a plurality of preset posture types. It may be a machine learning model configured to calculate and recognize the posture type of the input posture based on the calculated probability.

In one embodiment, the step of receiving the user's video image that captures the user's exercise movement from the server includes providing content profile information for a plurality of personal training video images to display a content list on the user's electronic device. transmitting to the user's electronic device; Receiving a request for a target personal training video image showing a guide operation for the user from the user's electronic device; providing a requested target personal training video image to the user's electronic device; and receiving, from the server, a video image of the user generated by filming the user's exercise movements on the user's electronic device. The video image of the user represents the user's motor movements mimicking the motor movements of the personal training object shown in the corresponding personal training video image.

In one embodiment, the step of generating the user's movement state information includes classifying a set of frame images in the video image of the user by scene to form a subset of frame images associated with the scene, wherein the subset includes at least one Includes a series of frame images, wherein the series of frame images is a continuous frame image representing a unit action of the user; generating posture information of the user appearing in the scene by processing frame images of the video image of the user in subset units; It may include generating exercise state information of the user based on the user's posture information. The step of generating the posture information of the user appearing in the scene includes, for each subset related to the scene included in the video image of the user, inputting consecutive frame images included in each subset into the pose recognition model. Recognizing the type of posture of the user shown in consecutive frame images; And it may include generating the user's posture information based on the recognition result of the posture type and the intermediate result calculated during the operation process to recognize the posture type in the pose recognition model.

In one embodiment, the step of generating the user's exercise result information includes comparing the user's posture information and the corresponding posture information of the personal training object in the user's exercise state information to determine the user's posture information for the personal training object. Calculating a user's posture similarity; and generating exercise result information of the user based on at least one of exercise result information of the personal training object, posture similarity of the user, and exercise state information of the user. In addition, calculating the posture similarity may include determining corresponding joints and corresponding skeletons of each of the personal training object and the user from the posture of the personal training object and the user's posture that correspond to each other; calculating a first similarity between the motion vectors of the user's joints in a series of postures of the user and the motion vectors of the joints of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user; calculating a second similarity between the motion vector of the user's skeleton in a series of postures of the user and the motion vector of the skeleton of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user; and calculating a posture similarity between the user and the corresponding personal training object during the series of postures based on the first similarity and the second similarity.

In one embodiment, the step of generating the user's exercise result information includes applying the motion similarity between the user and the personal training object to the difficulty level, heart rate, or calorie consumption of the personal training object in the exercise result information of the personal training object. calculating the user's difficulty level, heart rate, or calorie consumption; determining the user's motion skeleton and motion joints shown in the user's video image based on the positions of each joint in each posture and the motion vector of the skeleton during a series of postures; and calculating the range of motion of the user's motion skeleton and the range of motion of the user's motion joints by tracking the position of the user's motion skeleton and the positions of the motion joints; Detecting the user's exercise muscles among a plurality of muscles included in the user's body based on at least one of the position of the joint, the position of the skeleton, the movement vector of the joint, and the movement vector of the skeleton in the user's posture information during the unit operation. May include steps.

In one embodiment, the step of generating the user's exercise result information includes, when the user's posture matches the posture of the personal training object, calculating the exercise intensity of the personal training object as the user's exercise intensity; And calculating the user's exercise intensity may include correcting the user's exercise intensity according to the user's posture information based on at least one of the user's body profile data and exercise muscle usage level. there is.

In one embodiment, the step of correcting the user's exercise intensity includes determining the user's exercise intensity according to the user's posture information based on the level of exercise muscle use, and determining the user's exercise intensity from the user's exercise result information. It may include a step of correcting. Here, the step of correcting the user's exercise intensity in the user's exercise result information based on the user's exercise intensity according to the user's posture information and the level of exercise muscle use includes the body surrounding the exercise muscle in the input image. At least one step of acquiring muscle volume change information by analyzing the volume change of the portion, and acquiring skin surface change information by analyzing skin change on the motor muscle area indicating the motor muscle in the input image; predicting the degree of use of the exercise muscle for each exercise muscle area shown in the input image by inferring the correlation between the acquired volume change information of the exercise muscle and the skin surface change information and the degree of use of the corresponding muscle; And it may include correcting the user's exercise intensity based on the predicted degree of use of each exercise muscle.

In one embodiment, the step of generating analysis data on the user's exercise movement includes, for each of the user's exercise movements or exercise muscles, the user's exercise results as exercise muscle information and exercise intensity within the user's exercise result information. generating first analysis data by visualizing the information based on the information; may include. The first analysis data includes the first analysis image of the user.

The step of generating first analysis data for the user's exercise muscles includes identifying the user's exercise muscle area in a preset body region based on the user's exercise muscle information, and based on the user's exercise intensity information. The user's exercise intensity for the corresponding exercise muscle may be indicated in the identified exercise muscle area.

In one embodiment, the step of marking the exercise intensity on the exercise muscle includes adding a preset color to the user's exercise result information to distinguish the user's exercise muscle area from the non-exercise muscle area in the user's body image. It may include applying it to exercise muscles. The step of applying a preset color to the exercise muscles in the user's exercise result information to indicate the exercise muscles is performed so that the exercise intensity of the individual exercise muscles is visually distinguished for the exercise muscles in the user's exercise result information. Retrieving the color mapped to the exercise intensity of the exercise muscle from a pre-stored color table; And it may be a step of applying the color searched for each individual exercise muscle to each of the individual exercise muscles.

In one embodiment, the first analysis data may further include comparative evaluation information on exercise results. The step of generating the first analysis data includes at least one of a combination of information included in the user's exercise state information and exercise result information and information included in the exercise state information and exercise result information of the personal training object. It may further include generating exercise result comparison evaluation information by comparing at least one piece of information among the combinations. The exercise result comparative evaluation information may be a result of comparing exercise amount information in the exercise result information of the user with exercise amount information in the exercise result information of the personal training object, for each exercise muscle or exercise type.

In one embodiment, the second analysis data may include the second analysis image visualizing the positions of the user's entire exercise muscles and the total exercise intensity for each exercise muscle during the specific period. The step of generating the second analysis data includes adding up the exercise intensity for each individual exercise muscle from the user's entire exercise muscles during a specific period of time; With respect to the user's exercise muscles, searching for a color mapped to the summed exercise intensity of individual exercise muscles from all exercise muscles in a pre-stored color table so that each exercise intensity of the individual exercise muscles during a specific period can be visually distinguished; And it may include applying the color searched for each individual exercise muscle to each individual exercise muscle. The step of retrieving the color mapped to the summed exercise intensity of the individual exercise muscles from a pre-stored color table includes calculating the daily average exercise intensity of the individual exercise muscles during a specific period of time; and retrieving the color mapped to the daily average exercise intensity from a pre-stored color table.

In one embodiment, the step of generating analysis data on the user's exercise behavior includes statistics on the user's exercise results during a specific period based on at least some information of the user's exercise status information and exercise result information during the specific period. It may include a step of analyzing and generating third analysis data. The statistical analysis may be a summation or average analysis of values in the exercise result information for each item during the specific period.

In one embodiment, the step of generating analysis data for the user's exercise movements further includes generating fourth analysis data for a user group, which statistically analyzes the exercise results of a plurality of users, The fourth step of generating analysis data includes forming a user group by grouping a plurality of users by certain information included in each user's profile data; and generating third analysis data showing exercise trends for the user group by performing statistical analysis on at least one piece of exercise result information of individual users within the user group. The fourth analysis data may be exercise trend data by region, exercise trend by BMI, exercise trend by device type, exercise trend by body type, exercise trend by height, or exercise trend by country.

According to another aspect of the present application, a computer-readable recording medium performs a method of analyzing a user's exercise motion and effectively providing it in a visual aspect to support decision-making related to musculoskeletal exercise and medical services according to the above-described embodiments. Programs can be recorded.

A system that effectively provides analysis results of a user's exercise movements in a visual aspect to support decision-making related to exercise and medical services according to another aspect of the present application may include a server and one or more user electronic devices. there is. The server acquires a personal training video image representing at least one movement motion of a personal training object, analyzes the movement movement of the personal training object in the personal training video image, and generates metadata of the personal training video image, , receiving a video image of the user in which the user's exercise motion is captured from the server, and - the user's exercise motion is the user's exercise among at least one exercise motion of the personal training object shown in the personal training video image selected by the user It imitates the movement movement corresponding to the movement. The user's movement movement is analyzed in the user's video image to generate the user's posture information, and the user's movement state information is generated based on the user's posture information. And, in order to compare the user's exercise motion and the corresponding exercise motion of the personal training object, the user's posture information and the corresponding posture information of the personal training object are compared to generate exercise result information of the user, - The corresponding posture information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's exercise state information, and the user's exercise state information and the user It can be configured to generate analysis data about the user's exercise movement based on at least one piece of information included in the exercise result information - the analysis data is used to generate an analysis screen displayed on the user's electronic device - there is.

The exercise motion analysis system according to one aspect of the present application can collect exercise state information and exercise result information of a user located remotely, which comprehensively describes the health state of the remotely located user from various aspects. This exercise state information and exercise result information is generated and standardized in a preset data format.

Additionally, the exercise motion analysis system can provide analysis data that visually expresses the exercise state and exercise results so that the user can effectively recognize the exercise state and exercise results.

This analysis data can be usefully used to help patients in making medical decisions by efficiently conveying their information to hospital patients or general users who have difficulty making medical decisions.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

Figure 1 is a schematic diagram of a system that effectively provides analysis results of a user's exercise movements in a visual aspect to support decision-making related to exercise and medical services, according to one aspect of the present application.
2 is a schematic block diagram of an electronic device according to various embodiments of the present application.
3 is a schematic configuration diagram of a server according to various embodiments of the present application.
Figure 4 is a flowchart of a method for effectively providing a user's exercise state or exercise results in a visual aspect, according to another aspect of the present application.
Figure 5 is a detailed flowchart of a process for generating metadata of a personal training video image according to various embodiments of the present application.
Figure 6 is a detailed flowchart of a process for generating exercise result information of the user according to various embodiments of the present application.
Figure 7 schematically explains the structure of a used muscle analysis model according to various embodiments of the present application.
Figure 8 is a detailed flowchart of a process for generating analysis data according to various embodiments of the present application.
Figure 9 shows first analysis data showing exercise results by exercise type, according to some embodiments of the present application.
Figure 10 shows first analysis data showing exercise results for each exercise muscle according to some other embodiments of the present application.
Figure 11 shows a screen displaying second analysis data and third analysis data according to various embodiments of the present application.

Hereinafter, embodiments of the present application will be examined in detail with reference to the drawings.

However, this disclosure is not intended to limit the disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the disclosure. . In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this specification, expressions such as “have,” “may have,” “includes,” or “may include” refer to the corresponding features (e.g., numerical values, functions, operations, steps, parts, elements, and/or components). It refers to the presence of components such as etc.) and does not exclude the presence or addition of additional features.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Expressions such as “first,” “second,” “first,” or “second” used in various embodiments may modify various elements regardless of order and/or importance, and limit the elements. I never do that. The above expressions can be used to distinguish one component from another. For example, the first component and the second component may represent different components, regardless of order or importance.

As used herein, the expression “configured to” may mean, for example, “suitable for,” “having the capacity to,” or “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set) to” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or executing one or more software programs stored on a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing the corresponding operations.

As described below, the system and method according to embodiments of the present application generate metadata for the content by analyzing the actions of the personal training object by processing the content image, and calculate the user's action image. Through processing, exercise status information, exercise results and information that analyze the user's movements can be generated.

The actions of an input object, such as the personal training object or a user, refer to body movements implemented in a series of continuous poses. The actions may be exercise actions, daily life actions, working actions, or other actions.

In this specification, the term "exercise (or exercise operation)" is used to indicate the actions so that the term "actions", which describes the movements of the personal training object and the user, is clearly distinguished from the operations of the computing device. Referring to this, embodiments of the present application are described. However, this is merely illustrative, and it will be clearly understood by those skilled in the art that embodiments of the present application should not be limited to embodiments applied only to athletic movements. For example, the exercise state analysis unit and exercise result analysis unit below may also be understood as an action state analyzing unit and an action result analysis unit.

In this specification, the term “exercise” refers to various movements actively implemented by living organisms such as personal training objects and users. The “exercise” is not limited to the exercise of a physically healthy person, and includes the movement of a person (eg, a patient or a disabled person) who is temporarily or non-temporarily at least partially incapable of moving their body. The movement of the uncomfortable person may be, for example, rehabilitation exercise in a hospital or special exercise.

Figure 1 is a schematic diagram of a system that effectively provides analysis results of a user's exercise movements in a visual aspect to support decision-making related to exercise and medical services, according to one aspect of the present application.

Referring to FIG. 1, a system that effectively provides analysis results of the user's exercise motion in a visual aspect to support decision-making related to exercise and medical services (hereinafter referred to as exercise motion analysis system 1) includes the user's electronic device ( 100), a content provider system 200, and a server 300. The user's electronic device 100 and the server 300 may communicate wired/wireless through a telecommunication network.

The motor motion analysis system 1 according to embodiments may be entirely hardware, entirely software, or may have aspects that are partly hardware and partly software. For example, a system may collectively refer to hardware equipped with data processing capabilities and operating software for running it. In this specification, terms such as “unit,” “system,” and “device” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device that includes a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or other processor. Additionally, software may refer to a running process, object, executable, thread of execution, program, etc.

The telecommunication network provides a wired/wireless telecommunication path through which the electronic device 100 and the server 300 can transmit and receive data with each other. A telecommunication network is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used depending on the implementation. For example, when configured as an Internet Protocol (IP)-based system, the telecommunication network may be implemented as a wired and/or wireless Internet network, and the electronic device 100 and the server 300 are implemented as mobile communication terminals. In this case, the telecommunication network may be implemented as a wireless network, such as a cellular network or a wireless local area network (WLAN) network.

The user's electronic device 100 inputs profile data, including the user's body profile data, into the system 1; Requesting the server 300 to analyze the user's own exercise motion and transmitting an exercise video image showing the user's own exercise motion as the analysis target motion; It may be configured to receive analysis screen data including analysis data obtained by analyzing the user's exercise video image and display an analysis screen.

The electronic device 100 includes at least one processor capable of processing data, a memory for storing data, and a communication unit for transmitting/receiving data. The electronic device 100 may be, for example, a laptop computer, other computing device, tablet, cellular phone, smart phone, smart watch, smart glasses, head mounted display (HMD), other mobile device, or other wearable device. .

The operation of this electronic device 100 will be described in more detail with reference to FIGS. 2 and 4 below.

Figure 2 is a schematic block diagram of an electronic device according to various embodiments of the present application.

Referring to FIG. 2, the electronic device 100 may include a memory 101, a communication unit 102, a processor 103, an input device 104, an output device 105, and a camera module 106. . In some embodiments, the electronic device 100 may include a biosensor (not shown).

The memory 101 is connected to the processor 103 and can store data such as basic programs for operation of the processor 103, application programs, setting information, and information generated by operations of the processor 103. The memory 101 may be comprised of volatile memory, non-volatile memory, or a combination of volatile memory and non-volatile memory. Additionally, the memory 101 may provide stored data according to the request of the processor 103. In addition, the memory 101 permanently stores data received from the communication unit 102, data captured by the camera module 106, sensing data generated by the sensor module 107, and data generated or processed by the processor 103. Or you can save it temporarily.

The memory 101 includes HDD (Hard Disk Drive), SSD (Solid State Drive), CD (Compact Disc), RAM (Random Access Memory), Rom (Read Only Memory), and various types of data that store data permanently, semi-permanently, or temporarily. It may also include other storage devices.

In various embodiments of the present application, the memory 101 is used to perform operations of a system and method that effectively provides a user's exercise state or exercise results in a visual aspect to support decision-making related to musculoskeletal exercise and medical services. Dedicated applications can be stored.

The communication unit 102 is connected to the processor 103 to transmit and receive data, and can transmit and receive data with external devices such as another electronic device 100 or the server 300. All or part of the communication unit 102 may be referred to as a transmitter, receiver, transceiver, communication unit, communication modem, or communication circuit. The transceiver 102 uses wired access systems and wireless access systems such as the IEEE (institute of electrical and electronics engineers) 802.xx system, IEEE Wi-Fi system, 3rd generation partnership project (3GPP) system, and 3GPP LTE (long term evolution) system. , 3GPP 5GNR (new radio) system, 3GPP2 system, and Bluetooth can support at least one of various wireless communication standards.

The processor 103 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 103 may generate or generate information by performing pre-stored arithmetic processing operations and determine at least one executable operation to be subsequently performed based on the determined or generated information. And the processor 103 can control the components of the electronic device 100 to perform the determined operation. To this end, the processor 103 may request, retrieve, receive, or utilize data from the memory 101, and the electronic device 100 may execute an operation that is predicted or is determined to be desirable among the at least one executable operation. ) components can be controlled.

This processor 103 may be implemented with at least one processor. The processor 103 may include a CPU (Central processing unit) or a neuromorphic processor designed to be advantageous for artificial neural network calculations by incorporating neurons and synapses of the human brain.

In various embodiments of the present application, the processor 103 provides the user's exercise video image and health status data to be used for analysis by the server 300 to provide analysis data, and analyzes data provided from the server 300. Display and control the overall operation of the electronic device 100.

The input device 104 is a component configured to receive commands related to user input. The input device 104 may include a touch unit or other input unit.

The touch unit is a component that uses a part of the user's body or another object as a pointing object to input a buyer command. The touch unit may include, but is not limited to, a pressure-sensitive or electrostatic sensor. The other input units include, for example, buttons, keyboards, dials, switches, sticks, etc.

The output device 105 is a component that displays information stored and/or processed in the electronic device 100 and may include, for example, an LCD, OLED, or flexible screen, but is not limited thereto.

Although the input device 104 and the output device 105 are separated in FIG. 2, in many embodiments, the input device 104 and the output device 105 are implemented as a single component to receive input and output information. It can be. For example, the input device 104 and the output device 105 may be a touch panel implemented as a touch screen forming a layer structure with a screen. Touch input is input by a pointing object (eg, including the user's body, a tool, etc.).

In various embodiments of the present application, the user's electronic device 100 may input the user's profile data into the system 1 through the input device 104.

Additionally, the user's electronic device 100 may display a personal training video image used to generate the user's exercise movements through the output device 105. The user's exercise motion may mimic the exercise motion of the personal training object shown in the personal training video image, and the posture may at least partially match.

Additionally, the user's electronic device 100 may receive analysis data analyzing the user's exercise movements through the output device 105. In various embodiments of the present application, the user's electronic device 100 may display the analysis data through an analysis screen.

In various embodiments of the present application, the camera module 106 captures a subject to obtain an image frame such as a still image or video. The acquired image frame may be displayed on the display of the output device 106, transmitted to an external device through the communication unit 102, processed by the processor 103, or stored in the memory 101.

The camera module 106 includes a photographing unit that generates an image of an object in response to wavelengths. For example, the camera module 106 may include an RGB sensor to obtain an RGB image. In addition, the camera module 106 may be further configured to further include an IR sensor or a depth sensor to additionally acquire an IR image or a depth image.

In various embodiments of the present application, the electronic device 100 may generate a user's exercise video image by filming the user's exercise movements through the camera module 106.

Additionally, in various embodiments of the present application, the user's electronic device 100 may include one or more biosensors (not shown). The user's electronic device 100 may obtain the user's health status data through the biosensor.

The biosensor may be an acoustic sensor, a body temperature sensor, a blood pressure sensor, and/or a blood sugar sensor. The user's electronic device 100 may acquire health status data of any one of lung sounds, stomach sounds, heart rate sound data, body temperature data, blood pressure data, blood sugar data, and combinations of two or more thereof through a biosensor.

The electronic device 100 may transmit the user's health status data to the server 300. The health status data, together with the results of analyzing the user's exercise movements, can be used to provide decisions related to musculoskeletal exercise and medical services.

Referring again to FIG. 1, in the system 1, the content provider system 200 may provide a personal training video image to the server 300 or the electronic device 100.

The electronic device 100 may display a personal training video image provided by the content provider system 200. The personal training video image comprises a continuous series of video frames representing at least one motor movement of a personal training object. The personal training video image may be provided in the form of video data or multimedia data combining video and sound data.

A personal training video image may be video content that captures a personal training object performing an action. The personal training content may be guide content for providing correct operation information to users who watch the personal training content. In this case, the personal training object may be a trainer, coach, or other expert for training the user's exercise movements.

The server 300 is connected to one or more content provider systems 200 and can obtain personal training content for various sports events. Personal training content for each exercise event includes images showing the exercise movements of the corresponding sport. The above exercise items are not limited to weights, yoga, Pilates, CrossFit, rehabilitation exercise (or rehabilitation training), and other personal training exercises. The sports events may be ball games such as golf, baseball, soccer, basketball, etc., or other sports events.

In this specification, “Content Provider” refers to an entity that provides video images. Here, the “provider” is an “online service provider” that provides content created outside the system regardless of the production request of the person involved in the system (1), or an “online service provider” that provides content created outside the system according to the production request of the person involved in the system. Refers to at least one content provider among an “outsourced content provider” and an “internal content provider” that provides self-generated content within the system.

The person concerned may be an operator of the system 1 or a collaborator collaborating with the system 1. Collaborators rely on the range of movements to be analyzed provided by the system. For example, if the system is configured to analyze rehabilitation exercises, the collaborator may be a hospital, or a rehabilitation institution (or a member or associate thereof). In this case, the content provider system 200 may provide rehabilitation exercise content produced by hospital personnel according to the prescription code issued in the EMR (Electronic Medical Record) of the patient subject to rehabilitation. Then, the patient subject to rehabilitation can imitate the combination of rehabilitation exercises included in the rehabilitation exercise content created as a user of the present system 1, and the imitation results can be analyzed and provided.

Alternatively, if the system is configured to analyze personal training exercises, the collaborator may be a personal training center (or a member thereof).

In various embodiments of the present application, the operator and collaborator of the present system 1 may be the same entity. For example, if a hospital is the operator of this system (1), the hospital can perform both the role of the operator of the system (1) and the role of a collaborator as a single entity.

In this specification, a video image refers to aggregate image data of video components that are a single type of media (monomedia), multiple types of media (multi-media), or a subset thereof. The video image data may include video and/or audio track information, codec layer information (eg, base layer, enhancement layer), or subtitle information.

The server 300 may generate metadata of the personal training video image by analyzing at least one movement motion of the personal training object shown in the personal training video image of the content provider system 200.

In this specification, “metadata” is data associated with a personal training video image, which is content provided by the content provider system 200, and may include various attribute information describing the movement motion of a personal training object shown in the personal training video image. You can.

The personal training video images are described in more detail below.

The server 300 analyzes the movement motion of the personal training object shown in the personal training video image provided from the content provider system 200 and generates metadata of the personal training video image; The user's posture information is generated by analyzing the user's exercise movements shown in the user's exercise video image requested from the user's electronic device 100, and the user's exercise status information and exercise result information are based on the user's posture information. create; It is configured to generate analysis screen data including analysis data based on the user's exercise state information and exercise result information to be transmitted to the electronic device 100 to display an analysis screen.

The server 300 may be a unitary server, or may be implemented as a distributed server spanning multiple computers or multiple data centers. In various embodiments of the present application, the server is a plurality of computer systems or computer software implemented as a network server. Here, a network server is a computer system and computer that is connected to a sub-device that can communicate with other network servers through a computer network such as a private intranet or the Internet, receives a request to perform a task, performs the task, and provides a performance result. Refers to software (network server program). However, in addition to these network server programs, it should be understood as a broad concept that includes a series of application programs operating on a network server and, in some cases, various databases built within it. The server may be implemented as any type or combination of types of computing devices, such as a network server, web server, file server, supercomputer, desktop computer, etc. For this purpose, the server includes at least one processor capable of processing data, a memory to store data, and a communication unit to transmit/receive data.

The operation of the server 300 will be described in more detail with reference to FIGS. 3 and 4 below.

3 is a schematic configuration diagram of a server according to various embodiments of the present application.

Referring to Figure 3, the server 300 includes a control unit 310, an exercise state analysis unit 330; Exercise result analysis unit 350; It includes a profile repository (301), a metadata repository (303), and an analysis data repository (305). In some embodiments, the server 300 may further include a content storage 302.

The profile storage 301 is a data storage that stores profile data.

A user's profile data includes general profile data describing the user's identity, used to identify the user from other users, and/or body aspect profile data describing the user's unique physical information. The user's profile data may include the user's identification information and information describing the user's physical condition (hereinafter referred to as body profile information).

For example, the user's profile data includes at least one identifying information among the user ID in the system (1), patient ID for patient prescription results, name, address, email address, phone number, date of birth, and other identifying information unique to the user. may include.

The user's body profile information includes the user's gender, height, and weight.

In various embodiments of the present application, the user's profile data is information describing the user's physical condition and may further include one or more of the user's BMI, the user's area of interest, and body shape information. The area of interest may be a specific area of muscle, or another area within the user's body.

If there is no user input, a preset value in the system 1 may be applied to at least some of the user's body profile information. For example, if there is no user input, the basic weight, basic height, and basic BMI values pre-stored in the system 1 may be used as the user's body profile information. The basic BMI may be, for example, BMI data from the National Health Insurance Corporation or other standardized BMI data.

The server 300 may use the stored body profile data to generate the user's exercise state information and exercise result information.

Additionally, in some embodiments, the user's body profile data may be obtained from Electronic Medical Record (EMR) information of a hospital associated with the user. The system 1 can directly obtain some or all of the user's EMR information by linking with the hospital's EMR system.

Additionally, in some embodiments, the user's profile data may further include one or more of the geographical region where the user is located, country, device type, and preferred exercise movement information. The user's geographic area information can be obtained through area information according to the IP band of the electronic device 100.

The content storage 302 may be a data storage that stores personal training video image data provided from the content provider system 200. The content storage 302 may include multimedia data for displaying the personal training video image on the electronic device 100, and a content profile introducing the personal training video image. The content profile includes various information for introducing content. For example, the content profile may include, but is not limited to, the type of exercise shown in the content, the name of the content, the introduction of the personal training object, the thumbnail, the playback time, and the composition of the content by time.

When the server 300 receives a content request from the user's electronic device 100, the server 300 may search the content storage 302 for the requested personal training video image and provide it.

Additionally, the content storage 302 may store body profile data of a personal training object shown in a personal training video image. The server 300 may use the body profile data of the personal training object to generate metadata of the personal training video image.

The metadata storage 303 is a data storage that stores metadata of the personal training video image generated by the server 300.

In order to generate a user's exercise motion analysis result (e.g., exercise state information, exercise result information), the server 300 collects metadata of a corresponding guide video image indicating the exercise motion of a personal training object corresponding to the user's exercise motion. can be used.

The analysis result storage 305 is a data storage that stores the user's exercise motion analysis results (eg, exercise state information, exercise result information, analysis data). The analysis result storage 305 can store analysis data included in analysis screen data, which will be described with reference to FIGS. 9 to 11 below.

In some embodiments, information stored in data stores 301, 302, 303, and 305 may be structured according to a particular data structure. Additionally, in some embodiments, each data store may be a relational, columnar, correlational, or other suitable database. In some embodiments, system 1 may provide an interface that allows a device managing server 300 to manage, search, change, add, or delete information stored in data stores 301, 302, 303, and 305. You can.

The control unit 310 controls other components (eg, 330, 350, and 370) within the server 300 to effectively provide a user's exercise state or exercise results in a visual aspect.

The control unit 310 is configured to control the overall operation of the server 300 and may be implemented with at least one processor and memory. In this specification, the operation of the control unit 310 may be treated as equivalent to the operation of the server 300. That is, the operation of the server 300 may be an operation performed by the control unit 310, excluding the operation by the exercise state analysis unit 330 and the exercise result analysis unit 350.

The movement state analysis unit 330 is configured to generate movement state information by analyzing the state of the movement motion of the object shown in the image to be analyzed. The movement state information is information that describes the state of the movement motion of the object in the analysis target image.

The exercise result analysis unit 350 is configured to generate exercise result information of the object by analyzing the exercise result of the object based on the exercise state information of the object and the body profile data of the object. The motion result information is information that describes the implementation result of the motion motion of the object in the analysis target image.

The data generator 370 may generate user analysis data that variously analyzes the user's exercise effects using the user's exercise state information and exercise result information, and transmit the user analysis data to the electronic device 100 . Information provided by the data generation unit 370 may be provided through the analysis screen of the electronic device 100.

The operations of the control unit 310, exercise state analysis unit 330, exercise result analysis unit 350, and data generation unit 370 will be described in more detail with reference to FIG. 4 below.

According to another aspect of the present application, a method of effectively providing a user's exercise status or exercise results in a visual aspect to support decision-making related to musculoskeletal exercise and medical services is performed by a computing device such as the server 300 of FIG. 1 It can be.

Figure 4 is a flowchart of a method for effectively providing a user's exercise state or exercise results in a visual aspect, according to another aspect of the present application.

Referring to FIG. 4, the method acquires a personal training video image showing at least one exercise motion of a personal training object from the server 300, and analyzes the exercise motion of the personal training object in the personal training video image. Generating metadata of the personal training video image (S100); Step (S300) of receiving a video image of the user in which the user's exercise motion is captured from the server 300 - the user's exercise motion is at least one exercise of the personal training object shown in the personal training video image selected by the user Among the movements, the movement movements corresponding to the user's movement movements are imitated; Analyzing the user's exercise movements in the user's video image to generate exercise state information of the user (S400); In order to compare the exercise motion of the user and the corresponding exercise motion of the personal training object, the posture information of the personal training object is combined with the posture information of the user in the metadata of the personal training video corresponding to the video image of the user. Comparing and generating exercise result information of the user (S500); Generating analysis data on the user's exercise motion based on at least one of the information included in the user's exercise state information and the user's exercise result information (S600) - The analysis data is stored in the user's electronic device Used to generate the analysis screen displayed at 100; Includes.

In some embodiments, the method may further include authenticating the user before receiving the user's video image (S200). Transmission of the user's video image (S300) can be performed only when the user is authenticated.

The personal training video image is content including at least one continuous series of frame images, each representing at least one exercise movement of a personal training object. The personal training video image is guide content that provides reference movements for the user to view and imitate, and is used as a target video for the user to imitate.

The personal training video image may indicate the personal training object taking at least one action. The personal training object is an object that guides the user's exercise movements, and may be, for example, a coach, a professional athlete, or another expert who can accurately implement the posture of the exercise movement.

In the system 1, the metadata of the personal training video image provides a standard for comparing the user's exercise movements and providing feedback results to the user. The metadata for the personal training video image includes various information analyzing exercise movements included in the personal training video image.

In various embodiments of the present application, the metadata of the personal training video image may include some or all of the exercise state information and exercise result information of the personal training object for each action included in the personal training video image. You can.

Figure 5 is a detailed flowchart of a process for generating metadata of a personal training video image according to various embodiments of the present application.

Referring to FIG. 5, the step of generating the metadata (S100) includes classifying the frame image set in the personal training video image by scene to form a subset of frame images related to the scene (S101) - the sub The set includes at least one series of frame images, the series of frame images being consecutive frame images representing unit operations of the personal training object; generating posture information of the personal training object appearing in the scene by processing frame images of the personal training video image in subset units (S102); Generating exercise state information of the personal training object based on posture information of the personal training object (S103); Obtaining exercise result information of the personal training object in the personal training video image based on operator input information of the server 300 and/or exercise state information of the personal training object (S104); and generating metadata of the personal training video image including some or all of the exercise state information of the personal training object (S105).

The server 300 may receive the personal training video image from the content provider system 200 and store it in the content storage 302.

The personal training video image includes at least one scene section. A scene is a series of consecutive video frames of the same context representing a specific location, situation, or event.

The personal training video image may include one or more scenes. For example, a personal training video image may be a video in which a user performs a unit of a first exercise motion for a certain period of time, then takes a ready posture again, and then performs a second exercise motion.

In the personal training video image, an individual scene is one in which a personal training object repeats an action belonging to the corresponding action type one or more times, and may be referred to as a so-called set.

Accordingly, in the personal training video image, the scene section may be defined according to the type of motion to which the exercise motion taken by the object belongs. Personal training content may consist of a scene that changes when there is a change in the type of movement. In personal training content, a single scene may be associated with a single action shown in the single scene. Different types of movements in the same personal training content represent different scenes within the same personal training content.

When a personal training object in the video image performs a plurality of motion types, the control unit 310 classifies the frame image of the video image using the scene for each motion type as a section unit, and classifies the frame image of the video image into a sub related to the scene. A set can be formed (S101). A subset associated with the scene is a continuous series of frame images. A continuous series of frame images in the subset represents a series of poses of the personal training object. Each of the series of postures is the instantaneous posture of the personal training object in each frame image.

The control unit 310 may determine a scene section in a personal training video image according to an operator input of the server 300, or may determine a scene section using a pre-learned scene recognition model (S101). The scene recognition model may be a vision deep learning model learned to recognize scenes by processing continuous input images.

The control unit 310 transmits the frame image of the subset related to the scene formed in step S101 to the movement state analysis unit 330, and the movement state analysis unit 330 analyzes consecutive frames included in the subset. The image can be processed to generate posture information of the personal training object appearing in the scene (S102).

The exercise state analysis unit 330 of FIG. 3 is configured to analyze the state of the exercise movement of the analysis target shown in the analysis target image. The image to be analyzed may be a video image, a personal training video image showing a continuous series of postures constituting the exercise movement of a personal training object, or a user image showing a continuous series of postures making up the exercise movement of the user. there is.

The video image may be composed of multiple frame images representing a series of continuous postures to implement the movement motion of the object. Each frame image may be an image that captures the moment when an object takes a pose.

In various embodiments of the present application, the motion state analysis unit 330 analyzes posture information of an object shown in a video image (S102); Based on the posture information of the analyzed object, exercise state information can be generated, including one or more information among the motion type, motion time, motion speed, and number of motions for the motion motion continuously implemented by the series of postures of the object. (S103). If the image to be analyzed includes exercise movements according to a plurality of movement types, exercise states for each exercise movement type may be generated.

The movement of an object can be expressed as a series of poses. A posture can be defined by expressing the position and direction of each body component (eg, arms, legs, head, etc.) that an object assumes in a specific hand. Therefore, posture types can also be defined according to the positions and directions of components within the body. In some embodiments, posture types may be defined based on the relative positions and directions between components.

The posture information is information that describes the posture that constitutes this operation.

In various embodiments of the present application, the posture information in the exercise state information includes posture type, position of the skeleton and joints in each posture, and movement information for each of a series of postures that constitute the movement of the object in the image to be analyzed. It can be included. In some embodiments, the exercise state information may further include location and movement information of body components.

The exercise state analysis unit 330 may analyze the input image and generate posture information of the input object in the input image. Specifically, the exercise state analysis unit 330 analyzes the input image in which the input object appears using a pre-learned pose recognition model, recognizes the posture type to which the posture the input object is taking, and determines the recognized posture type. It is possible to generate posture information of the object including. When the exercise state analysis unit 330 recognizes a posture, it may determine the type of posture to which the posture taken by the input object belongs.

When the exercise state analysis unit 330 inputs individual frame images constituting an image to be analyzed into the pose recognition model, it can determine the posture type predicted by the pose recognition model as the posture type of the input object.

The pose recognition model may be a model based on a pose estimation algorithm and/or a deep learning structure capable of recognizing objects in an image.

In various embodiments of the present application, the pose recognition model extracts feature points of an object in an input posture through a pose estimation algorithm, and generates joint movement and/or skeletal movement information of the object based on the extraction result of the feature points of the object. Calculate the probability that the input posture providing the calculated movement information will be classified into each of a plurality of preset posture types by inferring the correlation between the calculated joint movement and/or skeletal movement information of the object and the posture type of the input object. It may be configured to calculate and recognize the posture type of the input posture based on the calculated probability. The pose recognition model may include an artificial neural network with a deep learning structure composed of a plurality of nodes to infer the correlation. The posture type of the input posture can be recognized by calculating the probability by inputting the calculated joint movement and/or skeletal movement information of the object into an artificial neural network.

Pose estimation technology is a technology that outputs 2D key points for human body parts by inputting color or black-and-white images of human movement. The key point may be extracted as the feature point. In some embodiments, the feature point may be a joint of the body. Specifically, the pose estimation-based pose recognition model detects the human joint area as a key point in the input image, and additionally detects the human body's skeleton (or may be referred to as a skeleton) based on the joint area, It is possible to calculate the movement of detected joints and skeletal movement information, and predict the type of posture of the person shown in the input image based on the position of the detected joints for each posture, the position of the skeleton, the calculated joint movement, and the skeletal movement. there is. Skeletal movements can be extracted in digital data format.

In the case of humans, joint positions such as head, shoulder, elbow, wrist, hip, knee, and ankle are extracted as key points. In one example, the pose recognition model learns the degree of association between human body parts in an image using PAFs (Part Affinity Filed) to accurately predict individual body parts. For each body part, a confidence map and a 2D vector (part affinity) indicating the degree of association between body parts are simultaneously predicted. In this case, the final 2D key point can be predicted using the confidence map and 2D vector. The prediction result of the final 2D key point is expressed as the position of the joint. The position of the joint may be an absolute position in the body, or a relative position with respect to another specific joint.

Additionally, the pose recognition model can form a human skeleton by utilizing the relative positions and angles between key points. The movement state analysis unit 330 may form the skeleton and obtain the position of the skeleton of the input object.

Additionally, the pose recognition model can predict skeletal movement and joint movement. The prediction results of the movement of the skeleton and joints can be used to recognize the type of posture as movement information of the skeleton and joints.

In various embodiments of the present application, the movement information of the skeleton or joints may be calculated in vector form. The pose recognition model may calculate motion vectors of joints and skeleton within the body of the input object in the input posture based on the detected feature points.

The object's motion vector consists of a skeleton motion vector and a joint motion vector. The motion vector of a skeleton consists of the motion values of each individual skeleton. The motion vector of a joint is composed of the movement values of each individual joint. The motion value is a value that describes the movement when assuming the specific posture in the input image, and may be the instantaneous movement (e.g., speed) of the input object at the moment the input image is captured during the motion shown in the video image. The speed can be expressed in terms of movement direction and change amount. The direction of movement may be a straight direction or a rotation direction in three-dimensional space.

In some embodiments, when a feature point is detected on a specific body of an input object in an input image, the pose recognition model calculates a motion vector of the object corresponding to the movement of the skeleton and joints based on a specific skeleton and joints in the specific body image. You can. The specific body reference point may be an absolute coordinate of a joint, or a relative coordinate between the joint and another joint. For example, based on absolute and relative coordinates between the waist or pelvis and each individual skeleton, a motion vector of the input object corresponding to the movement of the skeleton can be calculated.

The pose recognition model receives and processes the skeleton and its motion vectors as input, infers the correlation between the skeleton and its motion vectors and the input posture provided by the skeleton and its motion vectors, so that the input posture corresponds to It has an artificial neural network configured to determine the type of posture. The model parameters of the artificial neural network can be learned in advance before personal training content and the user's exercise video, which will be described below, are input.

The artificial neural network may be configured to calculate a probability that an input posture providing an input motion vector is classified into each of a plurality of preset posture types, and recognize the posture type of the input posture based on the calculated probability. The pose recognition model may determine the posture type with the highest probability value to be calculated as the posture type of the input posture.

Model parameters of the pose recognition model can be learned so that errors in key point detection accuracy and posture type recognition accuracy are minimized. Learning of the pose recognition model may be performed by an internal component of the system 1 (eg, server 300) or by an external device for learning a deep learning model. A pose recognition model previously learned by the server itself or an external device is supplied to the server 300 and used to generate metadata of personal training content and information on the user's exercise state. Specifically, the pose recognition model based on the pose estimation algorithm is installed in the user's electronic device 100 and/or the server 300 in the form of a library, and key points represent joints from the human image. Key points are found, the human body's skeleton and joint movements are extracted as digital data, and the person's posture type is determined from the extracted digital data. The determined posture type is used to judge a person's movement (ex: push-up, squat, etc.).

The pose recognition model can be implemented using various open source technologies. For example, the pose recognition model may be various open source based models such as "OpenPose", "PoseNet", "Alphapose", 'HRNet", "DeepPose", etc. However, the pose recognition model is not limited thereto. Therefore, the pose recognition model adopts a new technology or algorithm with improved motion extraction results for the user and trainer and motion estimation. Performance can be upgraded.

In various embodiments of the present application, the exercise state analysis unit 330 may include a pose recognition model for each exercise type of personal training content that can be provided. For example, the exercise state analysis unit 330 may use a first pose recognition model learned to recognize an exercise posture related to a weight exercise, a second pose recognition model learned to recognize an exercise posture related to a yoga exercise, and a running exercise. It may include a third pose recognition model learned to recognize the associated exercise posture. That is, the exercise state analysis unit 330 may include as many pose recognition models as the number of serviceable exercise items.

The exercise state analysis unit 330 determines the position of the joint and the position of the skeleton. Based on the movement information, the position and movement information of body components (eg, arms, legs, etc.) can be calculated.

These types of postures, positions of the skeleton and joints in individual postures, movement information, positions of body components, and movement information can be calculated for each individual frame image within the video image.

The exercise state analysis unit 330 sequentially inputs frame images within the video image into a pre-learned pose recognition model, recognizes the posture type for each posture within the frame image, and divides the motion section of the input object into the same posture type section. You can classify and determine the order of the classified posture types. Then, the movement state information for the analysis target image includes the sequence number of the posture type for the analysis target image, the posture type for each frame image, the position and direction of the body components in the individual posture, the position of the joint, and the movement information of the individual posture. can be calculated (S102).

The exercise state analysis unit 330 may generate exercise state information, for example, one or more of the type of motion, speed of motion, and number of motions, based on the posture information of the input object.

The exercise state analysis unit 330 may determine the motion type of the input object based on the connection order of the posture types in the posture information of the input object. The motion type indicates the motion type for each exercise event.

In some embodiments, the exercise state analysis unit 330 may determine the type of motion of the input object using a pre-stored motion-posture table. The motion-posture table may record the relationship between the posture type(s) constituting the motion type and the order of posture types for each motion type. The motion type is a type of unit motion, and may be, for example, a push-up or squat.

The exercise state analysis unit 330 obtains the posture types recognized from the posture information of the input object and the sequence between posture types, and determines the motion type matching the sequence between the obtained posture types and posture types. The motion-posture table may be searched, and the searched motion type may be determined as the motion type (i.e., motion type type) of the input object associated with the scene image set.

The operation time is the time when the personal training object performs at least one unit operation according to the individual operation type in the video image to be analyzed. The operation time for one operation type may be the sum of the times of unit operations according to the operation type.

The number of operations is the number of times the same operation type is performed. The number of operations may be the number of repetitions of a unit operation depending on the type of operation. If the image to be analyzed includes a plurality of motion types, the number of repetitions for each motion type is calculated as the number of motions for each individual motion type. The movement state analysis unit 330 may recognize the type of motion of an object based on the posture information of the object throughout the entire section of the video image, and calculate the number of times each motion type is recognized as the number of motions for each motion type.

The operation speed is the speed of implementing one unit operation, and can be calculated based on the playback speed of a plurality of frame images representing the corresponding operation in the video image.

In addition, the step (S102) includes selecting at least one posture required to identify the corresponding action as a key pose among a series of continuous postures constituting the action; and generating key posture information describing some postures selected as key postures among a series of postures as posture information of the personal training object.

The exercise state analysis unit 330 may be further configured to select, as a key pose, a posture essential for performing the corresponding motion from a series of consecutive postures constituting the motion. Then, the exercise state analysis unit 330 may generate key posture information describing some postures selected as key postures among postures (S102). The key posture information may include a posture type and frame sequence indicating the key posture.

The posture information of the personal training object may include the key posture information.

The exercise state analysis unit 330 may determine the key posture by specifying the key posture for each unit operation according to the user input, or by analyzing a plurality of posture information.

In various embodiments of the present application, the exercise state analysis unit 330 selects at least one posture of the unit action as the key posture for the unit action based on the position and movement information of the skeleton and the position and movement information of the joints. can do. The exercise state analysis unit 330 can select key postures for each type of movement and store the results. Then, posture information about the determined key posture can be generated. In some embodiments, the exercise state analysis unit 330 may determine the key posture based on position and movement information for each body component.

The key posture may be a posture in which the direction of position change or movement change of at least one of the joints, skeleton, and body components changes among the continuous postures that constitute the motion.

Key posture includes the starting posture and ending posture of the movement. The starting posture and ending posture change the direction of position and movement changes based on before/after starting and before/after ending.

Additionally, the key posture may further include at least one posture between the starting posture and the ending posture.

For example, the walking motion consists of several consecutive postures, including successive postures in which one foot is moved forward and the other foot in the movement posture. In consecutive postures in which one foot is moved forward, the direction of movement for each posture can be calculated as the direction in which the knee rises forward and then falls again. In postures in which the knee rises forward, the direction of rotation of the knee joint may be counterclockwise relative to the hip joint (relative to the side), and in postures in which the knee falls forward, the direction of rotation of the knee joint may be clockwise relative to the hip joint. . Accordingly, the exercise state analysis unit 330 may determine the posture at the time when the direction of movement of the knee changes from forward upward to forward downward as the key posture.

To this end, the exercise state analysis unit 330 tracks position changes and movement changes of joints, skeletons, and body components between adjacent postures among consecutive postures, and changes the direction of change in the opposite direction according to position changes and movement changes. The posture can be determined by the height posture.

In some embodiments, the exercise state analysis unit 330 may be further configured to additionally determine one or more postures among the connection postures between the primarily determined key postures as a new key posture.

Additionally, in some embodiments, the new key posture is a middle sequential posture among connection postures between sequentially adjacent key postures in a series of postures of the input object, or a change in the position or movement of a skeleton among the connection postures. may be a relatively large posture.

In alternative embodiments, the posture information in the exercise state information may be information about some postures selected as key postures. For example, the exercise state analysis unit 330 selects the key posture, and the posture information including the posture type, skeleton, joint positions, movement information, positions of body components, and movement information for each height posture. can be created. The exercise state analysis unit 330 may generate posture information that omits information on postures other than the height posture.

In various embodiments of the present application, the step (S102) includes, for each subset associated with a scene included in a personal training video image, inputting consecutive frame images included in each subset into a pose recognition model to Recognizing the type of posture of the personal training object shown in the frame image; And it may include generating posture information of the personal training object based on the recognition result of the posture type and the intermediate result calculated during the operation process to recognize the posture type in the pose recognition model.

The exercise state analysis unit 330 may generate posture information for a personal training video image having one or more scenes by generating posture information of the personal training object in the scene (S102). The posture information for a personal training video image consists of the posture information of the personal training object appearing in an individual scene.

Specifically, the movement state analysis unit 330 inputs a series of frame images representing a scene into a previously learned pose recognition model and calculates the posture type to which the posture of the personal training object appearing in the scene is predicted to belong. As described above, the pose recognition model extracts feature points of the personal training object, calculates movement information of the skeleton and joints of the personal training object based on the extraction results of the feature points of the personal training object, and calculates the calculated personal training object Movement information of the skeleton and joints is input into an artificial neural network with a deep learning structure composed of a plurality of nodes to calculate the probability that the input posture will be classified into each of a plurality of preset posture types, and select the posture type with the highest probability of being calculated. It can be configured to predict and output the posture type to which the input posture of the personal training object belongs (S102).

The exercise state analysis unit 330 may determine the posture type output from the pose recognition model as the posture type of the personal training object, and generate posture information of the personal training object including the posture type of the personal training object. .

In addition, the posture information of the personal training object is an intermediate result calculated in the process of processing to recognize the posture type in the pose recognition model, and includes the skeletal position of the personal training object in each posture during a continuous series of postures, the personal It may include movement information of the skeleton and joint movement information of the training object.

Since this pose recognition model produces intermediate results in the process of calculating the posture type to recognize it has been described above, a detailed explanation will be omitted.

In various embodiments of the present application, the step (S102) may further include selecting at least one posture from the continuous series of postures as the key posture. Then, the posture information of the personal training object may include posture information about the key posture. For example, the posture information of the personal training object may further include a binary label value indicating whether each of the series of postures is a key posture.

Key posture includes the starting posture and ending posture of the movement. The exercise state analysis unit 330 may select at least one posture between the start posture and the end posture among the series of postures of the personal training object as the key posture.

The step of selecting the key posture may be selecting the key posture of the personal training object for the motion shown in the scene based on the position and movement information of the skeleton and the position and movement information of the joints in each of the series of postures. . Additionally, in some embodiments, the step of selecting the height posture may determine the height posture based on position and movement information for each body component.

Additionally, in some embodiments, the step of selecting the key posture may further include the step of additionally selecting one or more postures among the connection postures between the primarily determined key postures as a new key posture. Additionally, in some embodiments, the step of additionally selecting the new key posture may be a middle sequential posture among connection postures between key postures whose sequence is adjacent in the series of postures of the personal training object, or among connection postures. A posture with relatively large changes in skeletal position or movement may be selected as the new key posture.

In some other embodiments, the step of selecting the key posture may be selected according to an operator input of the server 300.

The exercise state analysis unit 330 may generate exercise state information of the personal training object in the scene based on posture information of the personal training object that analyzes a series of postures shown in the scene (S103). The exercise state analysis unit 330 may generate exercise state information of the personal training object in each scene based on posture information of the personal training object associated with each scene. The exercise state analysis unit 330 may generate exercise state information for a personal training video image having one or more scenes by generating exercise state information of the personal training object in an individual scene (S103). The movement state information describes the movement state of a personal training object appearing in one or more scenes included in a personal training video image. Movement state information for a personal training video image consists of movement state information of the personal training object appearing in an individual scene.

In various embodiments of the present application, the exercise state information includes posture information of a personal training object shown in a personal training video image, motion type, motion time, motion speed, and It may contain one or more information about the number of operations. The exercise state analysis unit 330 analyzes one or more of the motion type, motion time, motion speed, and number of motions for the exercise motion continuously implemented by the series of postures of the personal training object in the scene. It can be generated based on posture information (S103).

In various embodiments of the present application, the body profile data of the personal training object in step S103 includes the gender, height, and weight of the personal training object. Additionally, the body profile data of the personal training object may further include one or more information among BMI, region of interest, and body type of the personal training object.

The detailed operating principle and calculation result information (i.e., posture information, exercise state information) of the step (S103) have been described above with reference to the exercise state analysis unit 330 of FIG. 3, and detailed descriptions will be omitted.

In step S104, the control unit 310 may directly receive some information among the exercise result information of the personal training object shown in the personal training video image from the device operating the server 300 or the content provider system 200.

When a personal training object in a personal training video image performs a plurality of exercise movements of different motion types, some of the plurality of exercise result information of the personal training object according to the plurality of motion types is displayed as a personal training object for the personal training video image. It can be input as exercise result information of the training object (S104).

In various embodiments of the present application, the exercise result information of the personal training object includes exercise muscles, range of motion of the exercise skeleton, range of motion of the exercise joint, exercise intensity, amount of exercise, It may include one or more information among exercise difficulty and heart rate.

The step (S104) includes receiving motor muscle, motor skeleton, and motor joint information of a personal training object performing an exercise movement in a personal training video image; Based on the exercise state information of the personal training object, one or more information of the range of motion of the exercise skeleton, range of motion of the exercise joint, exercise intensity, amount of exercise, exercise difficulty, and heart rate is generated, and the exercise muscles of the personal training object are generated. It may include obtaining exercise result information of the personal training object.

If the image to be analyzed includes a plurality of motion types, exercise result information related to each individual motion type may be generated. Exercise result information related to a specific motion type is information that analyzes the motion results that implement a motion motion in which an input object is classified as a specific motion type.

The motor skeleton refers to the skeleton directly used for motor movements in the image subject to analysis. Movement joints refer to joints directly used in movement movements in the image subject to analysis.

The motion skeleton and motion joints are preset according to operator input of the server 300 for each type of motion, or the position of each joint during a unit motion, the motion vector of the joint, the position of the skeleton, and/or the motion state information of the object. It may be determined based on the motion vector of the skeleton.

The range of motion of the motion joints and skeleton can be expressed in terms of rotation direction, rotation angle, movement direction, and movement distance. The direction of movement can be expressed as an orientation in three-dimensional space.

The exercise muscle is an area of the body where the muscle that exerts the effect of exercise depending on whether or not it is exercised is located. In weight exercise, the exercise muscle may be a partial area where the target muscle that is stimulated and pumped when the weight exercise is performed is located. For example, in the case of arm exercise, the exercise muscle may be a body region where the biceps or triceps muscles are located, depending on the type of movement.

The target muscle may be at least one muscle among muscles pre-stored in the server 300. The pre-stored muscles include, for example, the upper body, core, lower body, neck, shoulders, biceps, triceps, and back ( Back, Chest, Upper Abs, Lower Abs, Obliques, Spine, Hips, Glutes, Quads, Hamstrings ( It may be the hamstrings or calves.

The exercise muscles may be preset according to operator input of the server 300 for each type of movement, or at least some of the body regions may be determined to be the exercise muscles based on posture information in the exercise state information.

In various embodiments of the present application, the server 300 specifies the exercise muscles, motion joints, and motion skeleton of the personal training object in the personal training video image according to the operator input of the server 300, and produces the exercise results of the personal training object. Information can be generated. The exercise result analysis unit 350 may preset the exercise skeleton and exercise joints of the personal training object shown in the personal training video image according to the operator input of the server 300 (S104). The exercise result analysis unit 350 may generate the remaining exercise result information based on the exercise skeleton and exercise joint information of the personal training object shown in the personal training video image and the exercise state information of the personal training object set according to operator input. (S104).

The exercise intensity describes the intensity of exercise experienced by the exercise muscle during one unit movement, and may indicate the degree of growth of the exercise muscle. The exercise intensity can be calculated based on the load and exercise speed applied to the exercise muscles during the exercise movement. Here, the movement speed is the speed at which a unit movement is performed.

In some embodiments, the load within the exercise intensity may be expressed as a relative value of the load applied to each muscle. For example, the exercise intensity sets the load most strongly applied to a specific muscle area in the unit movement to 100, and the load applied to the remaining muscle areas is set to the load of the remaining muscles and the specific muscle based on the load of 100. It can be expressed as a value applying the compared ratio.

The level of difficulty, heart rate, and calorie consumption may be pre-specified for each type of movement. The specified difficulty level, heart rate, and calorie consumption may have granular values depending on the values in the body profile data. For example, calorie consumption can be specified as various values depending on body weight and height.

Momentum is a quantification of the scale of performing a specific type of unit action(s) during a scene section in the image subject to analysis, including the total movement time for performing unit movements, the number of movements (i.e., number of repetitions), and the exercise intensity of the movement. It can be calculated based on .

The control unit 310 generates metadata of the personal training video image including some or all of the exercise state information and/or exercise result information of the personal training object shown in the personal training video image. (S105).

When the control unit 310 generates metadata of the personal training video image, it can store it in the metadata storage 303.

The server 300 may generate metadata of the personal training content in advance for each of a plurality of personal training content and store it in the metadata storage 303. When the server 300 provides personal training content to the user's electronic device 100, the server 300 searches the metadata storage 303 for metadata of the provided personal training content, and analyzes the received exercise video of the user. and at least some of the metadata of the personal training content (eg, posture information and/or key posture information of the personal training object) can be compared and analyzed.

In alternative embodiments, the posture information of the personal training object shown in the scene may be posture information indicating only the key posture of the personal training object. The exercise state analysis unit 330 may generate posture information of the personal training object appearing in the scene, omitting information on postures other than the key posture (S102). Then, the exercise state information of step 103 and the metadata of the personal training video image may be generated based on the key posture. For example, the metadata of the personal training content may include type of exercise movement, number of exercise movements, time of exercise movement, intensity of exercise movement, intensity of exercise movement, movement joints related to the exercise movement, muscle information related to the exercise movement, and information about the exercise movement. Key posture information may be included. The metadata of the personal training content includes only the key posture information of the personal training object, so that there is no significant difference in evaluating the user's exercise motion while efficiently utilizing storage space compared to the case where the entire posture information is included. .

Referring again to FIG. 4, the step (S200) includes receiving, at the server 300, a request for analysis of the user's exercise motion including the user's identification information from the electronic device 100; Authenticating the user by the control unit 310 based on the user's identification information included in the user's exercise motion analysis request and identification information pre-stored in the profile storage 301; When the user is authenticated, transmitting a message requesting a video image of the user notifying success of authentication to the electronic device 100; and, if the user fails authentication, transmitting a message notifying the authentication failure to the electronic device 100.

The user's electronic device 100, which has received the authentication success message, may transmit the user's video image, which is the subject of exercise motion analysis, to the server 300 (S300). Specifically, in step S300, content profile information for a plurality of personal training video images stored in the content storage 302 is stored in the user's electronic device 100 to display a content list on the user's electronic device 100. 100) transmitting to; Receiving a request for a target personal training video image showing a guide operation for the user from the user's electronic device 100; providing a requested target personal training video image to the user's electronic device (100); It may include receiving, from the server 300, a video image of the user generated by filming the user's exercise movements on the user's electronic device 100.

The content list may be configured to select one personal training video image from a plurality of personal training video images as content to be imitated that represents the exercise movement the user wants to imitate.

The content list consists of a plurality of listing items, and each list item may be a plurality of personal training video images that can be provided through the content list. A plurality of personal training video images included in the content list are personal training video images whose content profile and multimedia data are pre-stored in the content storage 302, and metadata is created in advance and stored in the metadata storage 303. all.

Contents in the content list may be, for example, exercise content created by a seller of exercise content, or rehabilitation content showing movements required for a user to be rehabilitated in a hospital.

In some embodiments, the content in the content list is a hospital. The user's video image is an image of the user's exercise movement, and may have a source image used to analyze the user's exercise movement and generate analysis data.

The content in the content list may be content about exercise items, as shown in FIG. 6, but is not limited thereto.

On the screen displaying the content list, information on some or all of the metadata of the assigned personal training video image may be displayed in the list item area within the content list.

In one example, the list item area within the content list screen includes the name of the personal training video image, the number of repetitions of the exercise movements of the personal training object shown in the personal training video image, the set of exercise movements, the location of the exercise muscles of the personal training object, and the personal training video image. A personal training visualization image showing the exercise intensity for each exercise muscle of the training object can be displayed. The name of the video image may be the name of the exercise event, but is not limited thereto.

When specific content is selected as content to be imitated from the content list displayed on the user's electronic device 100, a personal training video image of the selected content is provided, and analysis of the user's exercise movements is initiated.

The user's video image may be exercise video content created by filming the user's exercise movements.

The video image of the user includes a continuous series of frame images representing each of the user's motor movements. When the target personal training video image includes a plurality of exercise movements of the personal training object, the video image of the user may also include a plurality of exercise movements of the user corresponding to each exercise movement of the personal training object. The user's video image may include a series of multiple frame images for each athletic action.

The user's exercise motion imitates the exercise motion of the personal training object shown in the personal training video image selected by the user. If a personal training object is shown in a personal training video image taking a plurality of motion types, the user's exercise motion in the video image of the user may sequentially imitate a plurality of exercise motions.

Accordingly, the entire frame image of the user content can be subset by operation. Specifically, the user's video image may be created by filming the user's exercise movements on a scene-by-scene basis, similar to a personal training video image. The user's electronic device 100 may display an interface screen that can capture the user's movements for each scene unit in order to generate one exercise content.

In the target personal training video image, the exercise motion of the personal training object corresponds to the user's exercise motion captured at the same or similar time. Specifically, because the exercise motion of the personal training object shown in the personal training video image selected as the content to be imitated corresponds to the user's exercise motion shown in the user's video image, the user's video image corresponds to the selected personal training video image. can be treated as such. In this way, because the user's exercise video is created by the user imitating the exercise movements of personal training content, the user's exercise video is at least partially similar to the exercise movements of the personal training object shown in the personal training video image at the same or similar time. It can indicate that the user takes the same action. A continuous series of frame images representing a specific motor action in the video image of the user corresponds to the specific motor movement in the corresponding personal training video image, and It can also be treated as Accordingly, each of the user's series of postures classified in scene units corresponds to a series of postures constituting the exercise movements of the corresponding personal training object.

The control unit 310 may generate exercise state information of the user in the same manner as the process of generating exercise state information of the personal training object to generate metadata of the personal training object (S400).

Specifically, the step of generating the user's movement state information (S400) includes classifying the frame image set in the user's video image by scene to form a subset of frame images related to the scene (S401) - the sub-set The set includes at least one series of frame images, wherein the series of frame images is a continuous frame image representing a unit action of the user; Processing frame images of the video image of the user in subsets to generate posture information of the user appearing in the scene (S402); It may include generating exercise state information of the user based on the user's posture information (S403).

The control unit 310 may classify the frame image set within the user's video image by scene based on the scene classification result of the target personal training video image. The control unit 310 may associate frame images in the user's video image with each frame image in the target personal training video image. If the frame image in the user's video image represents a specific posture of the user that imitates the specific posture of the personal training object, the frame image in the user's video image and the frame image in the personal training video image are frame images that correspond to each other. can be handled.

For example, in the target personal training video image, a first scene section is composed of first to tenth frame images representing a series of postures constituting the first movement motion of the personal training object, and the second scene section is , It may be composed of 11th to 20th frame images representing a series of postures that constitute the second exercise movement of the personal training object. Then, the control unit 310 classifies the first to tenth frames, which represent the user's first exercise motion imitating the first exercise motion of the personal training object, in the user's video image into a first scene section, By classifying the 11th to 20th frames into a second scene section, forming a subset of frame images associated with the first scene and representing a second movement movement of the user mimicking the second movement movement of the personal training object. A subset of frame images associated with the second scene may be formed.

The control unit 310 provides a set of frame images within the user's video image classified by scene to the exercise state analysis unit 330 (S401). The exercise state analysis unit 330 may generate user exercise state information by analyzing the user's exercise state in the corresponding scene by processing a set of frame images of the user for each scene (S400).

The exercise state analysis unit 330 may generate exercise state information of the user using the user's posture information and the corresponding posture information of the personal training object. The posture information for the user's video image consists of the user's posture information shown in each individual scene. Here, the corresponding posture information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's posture information. Specifically, the exercise state analysis unit 330 may generate posture information for a video image of a user having one or more scenes by generating posture information of the user on a scene basis (S402).

The process of generating the user's posture information in step S402 and the process of generating the user's exercise state information in step S403 include the process of generating posture information of the personal training object in step S102 of FIG. 5, This is the same as the process of generating exercise state information of the personal training object in step S103.

In various embodiments of the present application, the step (S402) includes, for each subset associated with a scene included in the user's video image, inputting consecutive frame images included in each subset into a pose recognition model to Recognizing the type of posture of the user shown in the frame image; And it may include generating the user's posture information based on the recognition result of the posture type and the intermediate result calculated during the operation process to recognize the posture type in the pose recognition model. The intermediate result may be a joint position, a skeleton position, a joint motion vector, or a skeleton motion vector. Then, the user's posture information may include the user's recognized posture type during the movement, the position of the joint, the position of the skeleton, the motion vector of the joint, and the motion vector of the skeleton.

In various embodiments of the present application, the step (S402) may further include selecting at least one posture from the continuous series of postures as the key posture. Then, the user's posture information may include posture information about key posture. For example, the user's posture information may further include a binary label value indicating whether each of the series of postures is a key posture.

Additionally, in some embodiments, the step of selecting the height posture may determine the height posture based on position and movement information for each body component.

Additionally, in some embodiments, the step of selecting the key posture may further include the step of additionally selecting one or more postures among the connection postures between the primarily determined key postures as a new key posture. Additionally, in some embodiments, the step of additionally selecting the new key posture may be a middle sequential posture among connection postures between key postures whose sequence is adjacent in the series of postures of the personal training object, or among connection postures. A posture with relatively large changes in skeletal position or movement may be selected as the new key posture.

In alternative embodiments, the posture information in the exercise state information may be information about some postures selected as key postures. For example, the exercise state analysis unit 330 selects the key posture, and the posture information including the posture type, skeleton, joint positions, movement information, positions of body components, and movement information for each height posture. can be created. The exercise state analysis unit 330 may generate posture information that omits information on postures other than the height posture.

The exercise state analysis unit 330 may generate exercise state information of the user based on the user's posture information. The exercise state analysis unit 330 may generate the user's exercise state information, for example, motion type, motion speed, and number of motions, based on the user's posture information (S403).

Since the operations of steps S402 and S403 are the same as those of steps S102 and S103, detailed descriptions are omitted.

The user's exercise state information generated in the exercise state analysis unit 330 is provided to the exercise result analysis unit 350.

The exercise result analysis unit 350 generates exercise result information of the user as a result of comparing and analyzing the user's exercise motion and the corresponding exercise motion of the personal training object (S500). The user's posture information obtained in the step (S400) is compared with the posture information of the personal training object in the metadata of the personal training video corresponding to the user's video image (S500).

Figure 7 is a detailed flowchart of a process for generating exercise result information of the user according to various embodiments of the present application.

Referring to FIG. 6, comparing the posture information of a personal training object with the posture information of the user in the metadata of the personal training video corresponding to the video image of the user to generate exercise result information of the user (S500) Comparing the user's posture information with the corresponding posture information of the personal training object in the user's exercise state information, calculating the user's posture similarity to the personal training object (S501); It may include generating exercise result information of the user based on at least one of exercise result information of the personal training object, posture similarity of the user, and exercise state information of the user (S502). The posture similarity may indicate the user's posture imitation accuracy based on the posture of the personal training object.

The exercise result analysis unit 350 stores joint motion vectors, skeletal motion vectors in a series of user postures shown in a subset of frame images classified according to scene units, and a personal motion vector corresponding to the user's series of postures. The posture similarity between the joint motion vectors and the skeletal motion vectors of the personal training object shown in a series of postures of the training object can be calculated (S501).

The step of calculating the posture similarity (S501) includes determining corresponding joints and corresponding skeletons of the personal training object and the user, respectively, from the posture of the personal training object and the user's posture that correspond to each other; calculating a first similarity between the motion vectors of the user's joints in a series of postures of the user and the motion vectors of the joints of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user; calculating a second similarity between the motion vector of the user's skeleton in a series of postures of the user and the motion vector of the skeleton of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user; and calculating a posture similarity between the user and the corresponding personal training object during the series of postures based on the first similarity and the second similarity.

The first similarity and the second similarity are similarities between movement information of corresponding joints and corresponding skeletons.

Because the physical conditions (e.g., height, weight, body type, etc.) between the user and the personal training object are different, the positions of the joints and skeleton may differ despite the accuracy of the posture. Accordingly, the exercise result analysis unit 350 can analyze posture accuracy based on the movement information.

The exercise result analysis unit 350 may store various pose matching algorithms in advance to determine the corresponding joints and corresponding skeletons of each personal training object and user. The pose matching algorithm may be configured to determine corresponding joints and corresponding skeletons, for example, by nearest neighbor-based matching, optimization-based matching, or other matching methods. The control unit 310 may determine the user's corresponding joints and skeleton corresponding to the joints and skeleton of the personal training object from the posture of the user and the posture of the corresponding personal training object through a pre-stored pose matching algorithm.

In various embodiments of the present application, the step of calculating the posture error between the personal training object and the user includes calculating the posture error between the personal training object and the user by adding the calculated position errors of the joints and the movement errors of the skeleton. It could be. In some embodiments, the position error of the joint and the movement error of the skeleton may be calculated as mean square error (MSE).

In some embodiments, the step (S501) includes correcting the posture information of the personal training object based on the user's height and body shape to the posture information of the personal training object having the same height and body shape as the user; And it may include comparing the corrected posture information of the personal training object with the posture information of the user to calculate posture similarity between the user and the personal training object. The control unit 310 can more accurately calculate posture similarity using the corrected joint position and skeletal movement information of the personal training object and the user's joint position and skeletal movement information.

In addition, the step of calculating the motion similarity between the personal training object and the user includes calculating the motion similarity between the personal training object and the user by adding up the individual motion similarities between the user and the personal training object for a series of postures. You can. Motion similarity refers to how accurately the user's exercise motion imitates the exercise motion of the personal training object.

The exercise result analysis unit 350 may use a similarity calculation algorithm to calculate various similarities such as first similarity, second similarity, posture similarity, and motion similarity. The similarity calculation algorithm may be, for example, a Euclidean distance method, a cosine distance method, or another similarity calculation algorithm between vectors.

In some embodiments, the exercise result analysis unit 350 corrects the body shape of the personal training object to the body shape of the user before calculating the posture similarity, and analyzes the personal training object according to the corrected body shape of the personal training object. It may be configured to correct the posture information of the training object. Then, the user's motion accuracy can be calculated based on the personal training object whose body shape matches.

Meanwhile, the exercise result analysis unit 350 may generate the user's exercise result information based on the user's body profile data and the user's exercise state information (S502).

In various embodiments of the present application, the exercise result analysis unit 350 determines the amount of exercise, exercise intensity, difficulty, heart rate, calorie consumption, exercise muscles used in the exercise of the object, and exercise range of the exercise skeleton of the object in the image to be analyzed. , motion result information of the object may be generated, including information on one or more of the range of motion of the motion joint. The exercise result information may be generated for each type of exercise of the object. If the image to be analyzed includes multiple exercise types, exercise result information for each exercise type may be generated (S502).

In the step of generating exercise result information of the user (S502), the similarity in motion between the user and the personal training object is applied to some of the exercise state information of the personal training object to generate information on some items of the exercise state information of the user. calculating a value; calculating values for some other items included in the user's exercise result information based on the user's exercise state information; and calculating values for some other items included in the user's exercise result information based on the user's body profile data and the user's exercise state information.

The step of calculating values for some items of the user's exercise state information by applying the motion similarity between the user and the personal training object to some of the exercise state information of the personal training object includes the operation between the user and the personal training object. The similarity may be applied to the difficulty, heart rate, or calorie consumption of the personal training object from the exercise result information of the personal training object to calculate the user's difficulty, heart rate, or calorie consumption.

The step of calculating values for some other items included in the user's exercise result information based on the user's exercise state information is based on the position of each joint in the posture during a series of postures and/or the motion vector of the skeleton. determining the user's movement skeleton and movement joints shown in the user's video image; And it may include calculating the range of motion of the user's motion skeleton and the range of motion of the user's motion joints by tracking the position of the user's motion skeleton and the positions of the motion joints. Additionally, in some embodiments, the step of calculating values for some other items included in the user's exercise result information based on the user's exercise state information includes the user's posture information during the exercise movement in the user's video image. It may include detecting the user's exercise muscles based on the user's exercise muscles.

The exercise result analysis unit 350 may select the user's exercise skeleton and exercise joints based on the user's posture information (S502). In some embodiments, the exercise result analysis unit 350 may select an exercise skeleton among a plurality of skeletons in the body pre-stored in the server 300 if there is a change in the position or motion vector of the skeleton during a unit operation.

The exercise result analysis unit 350 selects an exercise skeleton among a plurality of pre-stored skeletons in the body, and for the selected exercise skeleton, tracks the position and movement of each exercise skeleton during a unit operation to calculate the range of motion of the exercise skeleton. can do.

The exercise result analysis unit 350 may select a motion joint among a plurality of joints in the body, and for the selected motion joint, track the position and movement of each motion joint during a unit operation to calculate the range of motion of the motion joint. there is.

The movement joint may be determined based on a change in the position of the joint during a unit motion in the movement state information of the object. For example, the exercise result analysis unit 350 may determine a joint having a relatively large joint position change value in an individual posture during a series of consecutive postures as an exercise joint. In some embodiments, a skeleton whose movement during the movement of the input object is greater than or equal to a preset threshold may be detected as a motion skeleton. Additionally, in some embodiments, the threshold movement may be body movement. That is, when the body moves, a skeleton that moves more than the entire body can be detected as a motion skeleton. For example, in an arm rotation motion, the motion skeleton may be the brachialis, radius, and ulna muscles located in the shoulder. Additionally, in some embodiments, the movement of the body may be an average movement of coordinates of bones within the body in individual postures.

The range of motion of the motion joints and skeleton can be expressed in terms of rotation direction, rotation angle, movement direction, and movement distance. The direction of movement can be expressed as an orientation in three-dimensional space.

As for the exercise muscle, at least some of the body regions may be determined to be the exercise muscle based on posture information in the exercise state information for each type of movement.

In various embodiments of the present application, the exercise result analysis unit 350 determines the position of the joint, the position of the skeleton, the motion vector of the joint, and/or the motion vector of the skeleton in the user's posture information during the unit motion in the video image of the user. Based on this, the user's exercise muscles among the plurality of muscles included in the user's body can be determined (S502).

The user's exercise intensity may be calculated from the exercise intensity of the personal training object analyzed from the exercise motion of the corresponding personal training object.

The step (S502) further includes calculating the user's exercise intensity applied to individual exercise muscles based on the difference between the posture information of the personal training object and the user's posture information. The exercise result analysis unit 350 may calculate the exercise intensity applied to each muscle selected as the user's exercise muscle in the user's video image.

In the step of calculating the user's exercise intensity, when the user's posture matches the posture of the personal training object, the exercise result analysis unit 350 calculates the exercise intensity of the personal training object as the user's exercise intensity. It may include steps. Additionally, in some embodiments, calculating the user's exercise intensity may include correcting the user's exercise intensity according to the user's posture information based on the user's body profile data and/or the level of exercise muscle use. may further include. The exercise result analysis unit 350 may generate the user's exercise result information in which the corrected exercise intensity is the user's final exercise intensity (S502).

The step of correcting the user's exercise intensity may include correcting the user's exercise intensity according to the user's posture information based on the user's body profile data.

The amount of load that stimulates the muscles is affected by body weight as well as the accuracy of posture. The heavier the weight, the more load is placed on the muscles. In order to more accurately analyze the user's exercise results, the exercise result analysis unit 350 calculates the user's exercise intensity as the ratio of the user's body weight and the body weight of the personal training object, and calculates the calculated body weight ratio as the By applying the user's exercise intensity according to the user's posture information, the user's exercise intensity can be corrected. Then, the user's exercise result information with the corrected exercise intensity can be generated.

Additionally, the step of correcting the user's exercise intensity may include correcting the user's exercise intensity according to the user's posture information based on the level of exercise muscle use.

The level of use of the exercise muscles relates to the exercise muscles actually used by the object in performing the movement, and is a level that indicates the accuracy of use of the exercise muscles. The usage level may be expressed as a value indicating the accuracy of usage by level according to a preset level range.

The usage level can be calculated for each exercise muscle. In order for the user to accurately obtain the effect of exercise, it is necessary to assume an accurate posture and use muscles accurately. In the case of beginners, even if the positions of the joints and the skeleton are accurately matched, exercise movements may be implemented in the wrong way, applying force to muscles that should not be applied, and failing to properly apply force to muscles that need to be applied. For example, among weight exercises, the lat pull-down exercise is a strength exercise that develops the lat (latissimus dorsi muscle) by pulling the bar down as if doing chin-ups, and strengthens the arm muscles and upper trapezius muscle. You can obtain accurate exercise results by excluding as much as possible and utilizing the strength of the muscles around the lats, which are connected to the lats and scapula. Even if a novice user accurately imitates the putt coach's posture, if the user pulls the bar using only the arm muscles for the wrong muscles, a problem occurs in which the load is not applied correctly to the exercise muscles of the lat pull down, and eventually, instead of developing the expected back muscles, the arm muscles are damaged. You only get muscle development.

In various embodiments of the present application, the step of correcting the user's exercise intensity according to the user's posture information based on the level of use of the exercise muscle includes analyzing the volume change of the body part surrounding the exercise muscle in the input image. At least one step of acquiring muscle volume change information and acquiring skin surface change information by analyzing skin changes on the motor muscle area indicating the motor muscle in the input image; By inferring the correlation between at least one of the acquired volume change information of the exercise muscle and the change information of the skin surface on the exercise muscle area and the degree of use of the corresponding muscle, the degree of use of the exercise muscle is determined for each exercise muscle region shown in the input image. predicting step; And it may include correcting the user's exercise intensity based on the predicted degree of use of each exercise muscle.

The exercise result analysis unit 350 may calculate the level of use of the exercise muscles using a pre-learned muscle use analysis model. The used muscle analysis model is a machine learning model that is pre-trained to calculate the level of use of the exercise muscle, indicating whether the exercise muscle was actually used by analyzing muscle use information visually expressed in the input image.

Muscle use information visually expressed in the input image includes changes in muscle volume and/or changes in skin surface.

When muscles are used with force, their volume expands. In other words, when the volume of a muscle expands, it means that the object is using the muscle. On the other hand, if the muscle volume does not expand, it may mean that the object is not using the muscle. Changes in the volume of the muscle correspond to changes in the volume of the body part surrounding the muscle.

Additionally, when muscles are used with force, the skin surface becomes taut due to muscle expansion, or surface changes occur where muscles and tendons under the skin protrude. In other words, when the skin surface swells or protrudes, it means the object is using its muscles. On the other hand, if the skin surface is not inflated and protruding, it may mean that the object is not using its muscles.

In various embodiments of the present application, the used muscle analysis model analyzes the volume change of the body part surrounding the muscle to be analyzed in an input image (e.g., a series of frame images as a subset related to the user's motion) to provide muscle volume change information. Change information on the skin surface may be obtained by acquiring and/or analyzing changes in the skin of the body part surrounding the muscle in the input image. The use level of the muscle to be analyzed can be calculated based on the acquired muscle volume change information and/or skin surface change information. Here, the muscle to be analyzed is at least one muscle analyzed as an exercise muscle.

Figure 7 schematically explains the structure of a used muscle analysis model according to various embodiments of the present application.

Referring to FIG. 7, the used muscle analysis model includes a first analysis model that divides exercise muscles from continuous input images showing motion and calculates volume changes of the divided exercise muscles during motion; and/or a second analysis model that segments motor muscles in continuous input images showing motion and calculates changes in the skin surface within the segmented motor muscles during motion. Additionally, the used muscle analysis model includes an inference model that calculates the use level of the object's muscles based on the analysis result of the first analysis model and/or the analysis result of the second analysis model.

The first analysis model and the second analysis model detect muscles based on the position of the joint and the skeleton, and analyze the detected muscle and its surrounding area in pixel units to determine the boundary of the muscle and segment it.

In various embodiments of the present application, the first analysis model and the second analysis model may be configured to segment exercise muscles shown in the input image. For example, the first analysis model is the upper body part, core part, lower body part, neck part, and shoulders in the body region (e.g., whole body) of the object shown in the input image. , Biceps, Triceps, Back, Chest, Upper Abs, Lower Abs, Obliques, Spine, Hips , the muscles shown in the input image can be segmented and detected by segmenting the glutes, quads, hamstrings, or calves.

In some embodiments, the first analysis model and the second analysis model may include a segmentation model based on a vision deep learning structure that extracts features from an input image and segments a region of interest. The segmentation model may be a segmentation model based on a CNN structure, but is not limited thereto. CNNs are specialized for image processing and may be suitable for detecting muscles. Additionally, in some embodiments, the CNN structure-based segmentation model primarily detects muscles based on the positions of joints and skeletons, extracts features from the input image, and pixels at the boundary of the primary muscles based on the extracted features. The probability of belonging to this muscle is calculated on a pixel basis, and the surrounding pixels of the muscle are determined to be muscles or non-muscles according to the calculated probability, thereby dividing the muscle with an accurate boundary.

In some embodiments, the used muscle analysis model may be configured to enlarge and process a series of consecutive frame images representing movements and then input them into the first analysis model and the second analysis model. A preprocessed image with an enlarged skin surface compared to the original frame image may be input to the first analysis model and the second analysis model.

Additionally, in some embodiments, when the used muscle analysis model includes the first analysis model and the second analysis model, one of the two analysis models (e.g., the first analysis model) performs a segmentation operation, The division result may be provided to the remaining analysis model (e.g., the second analysis model), and the remaining analysis model (e.g., the second analysis model) may be configured to omit the division operation. For example, as shown in Figure 3, when the first analysis model divides the exercise muscle, the second analysis model receives the segmentation result of the exercise muscle and analyzes the change in the skin surface within the divided exercise muscle. You can use it right away. Hereinafter, for clarity of explanation, the operation of the used muscle analysis model that calculates the level of use of the exercise muscles will be described using some of the above embodiments.

The first analysis model is further configured to calculate the volume of segmented muscles for continuous input images showing motion and output the change in muscle volume. The first analysis model may calculate, for each segmented muscle, the volume of the individual muscle during the movement from continuous input images showing the movement. The volume of a muscle in an individual input image may be calculated based on at least some information (eg, height, body shape, etc.) in the object's body profile data and the number of pixels indicating the individual muscle in the two-dimensional image.

The first analysis model can output muscle volume changes in each successive input image in the form of a two-dimensional matrix. In a two-dimensional matrix, the first axis may indicate the sequence number of the input image, and the second axis may indicate the muscle. In a two-dimensional matrix, a matrix element may be the volume value of the muscle in the input image of the corresponding sequence. The volume analysis results of individual muscles in the matrix form include volume change information of each muscle in continuous input images. This volume change information is used to calculate the level of use for each muscle.

Meanwhile, the second analysis model is configured to calculate changes in the muscle and skin surface according to the segmentation results for continuous input images showing motion. Specifically, the second analysis model contrasts continuous input images representing the motion so that skin surface elements (e.g., pores, hair, etc.), muscles protruding from the skin surface, and blood vessels are displayed more clearly in the image. ) and detect skin surface elements in each of the contrast-processed consecutive input images to calculate the change in spacing between adjacent skin surface elements during operation, and/or protruding blood vessels, etc. in each of the contrast-processed consecutive input images. It is configured to detect the part and calculate the height of the step between the protruding part and the surrounding non-protruding part. As a result of the contrast processing, the light-dark contrast of the image is increased, allowing skin surface elements and muscles to become more vivid. The higher the step height, the higher the degree of protrusion of the muscle or tendon. An increase in the spacing between the skin surface elements indicates swelling of the skin.

The second analysis model is a first sub-model that detects skin surface elements in each of the contrast-processed continuous input images and calculates the change in spacing between adjacent skin surface elements during operation, detects protruding parts such as blood vessels, and protrudes It may include a second sub-model that calculates the height of the step between the part and the surrounding non-protruding part.

In some embodiments, the first sub-model and the second sub-model may be a vision recognition deep learning model. The two submodels may have the same structure or different structures. For example, the first sub-model is a CNN structure and can be learned to extract local characteristics of the skin surface, use this to learn features such as pores and hair on the skin surface, and calculate these intervals. The second sub-model is an FCN (Fully Convolutional Networks) structure that extracts regional characteristics from the skin surface, classifies the protruding and non-protruding parts of blood vessels in the pixel based on the extracted regional characteristics, and determines the step difference between these parts. It can be trained to calculate height.

In some other embodiments, the second sub-model may be an imaging processing-based model. The second sub-model can calculate the step height using an image processing algorithm. For example, the second sub-model extracts three-dimensional information from one two-dimensional image using a single image depth prediction method, and uses the extracted three-dimensional information to determine the height difference between the protruding and non-protruding parts of the skin. It can be calculated.

The inference model infers correlations between muscle use and body profile data of the object, changes in muscle volume and/or skin surface, using the level at which muscles in the input image were directly used to implement the action in the input image. It may be a machine learning model that is trained to output at the same level. The inference model may have an artificial neural network structure composed of a plurality of nodes. For example, the inference model may have an artificial neural network with fully connected layers, or other structures such as a multilayer perceptron (MLP).

In various embodiments of the present application, the output result of the first analysis model, the output result of the second analysis model, and the user's body profile data may be concatenated and input to the inference model in the form of a vector.

In some embodiments, the inference model may extract the value with the largest volume change for each individual muscle from the analysis results of the first analysis model. Additionally, the inference model can extract the value with the largest change in the skin surface of each muscle for each muscle from the analysis results of the second analysis model. The inference model can be trained to calculate usage levels based on the largest change in volume and/or the largest change in skin surface.

In the used muscle analysis model, model parameters (eg, node weights, etc.) of the first analysis model, the second analysis model, and the inference model are learned using a training data set including a plurality of training samples. Each training sample may include a body image of the sample object and a label value for the sample object appearing in the body image. In some embodiments, the label value includes actual muscle information (e.g., location) of the sample object as a label value for the first analysis model, and skin surface element information (e.g., location) as a label value for the second analysis model. Alternatively, it may include the step height and the actual muscle use level as a label value for the inference model. The actual muscle use level may be a result of measuring the signal strength applied to the muscle (eg, nerve current or voltage value), or a survey result felt by the sample object.

The used muscle analysis model may be learned using the training data set so that the error of the first analysis model, the error of the second analysis model, and the error of the inference model are minimized. Here, the error of the first analysis model represents the difference between the predicted partition area and the actual partition area. The error of the first analysis model may be expressed, for example, in the form of an error of a vision recognition-based segmentation model. The error of the second analysis model represents the difference between the spacing of the predicted skin surface elements and the spacing of the actual skin surface elements and/or the difference between the predicted step height and the actual step height. The error of the inference model represents the difference between the predicted muscle use level and the actual muscle use level. The error of the second analysis model or inference model may be expressed, for example, in the form of an error of a vision recognition-based prediction model.

The training data set may be formed from personal training video images. The label value can be set based on the motion of the putt.

By calculating the level of exercise muscle use using the used muscle analysis model, the exercise result analysis unit 350 can more accurately analyze the user's exercise results. The exercise result analysis unit 350 may generate exercise result information including the level of use of exercise muscles.

In alternative embodiments, the exercise result analysis unit 350 may generate exercise result data of the user based on the user's height posture information. That is, the exercise result analysis unit 350 may generate the user's exercise result data using the key posture instead of using all of the continuous series of postures that constitute the user's exercise motion.

The data generator 370 may generate user analysis data that analyzes the user's exercise effects in various ways using the user's exercise state information and exercise result information, and transmit this to the electronic device 100 (S600). .

Analysis data provided to the user's electronic device 100 may be stored in the analysis result storage 305.

The analysis data contains information used to create an analysis screen displayed on the user's electronic device 100. The analysis screen is configured to effectively convey the analysis data visually so that users can more easily recognize the analysis data.

Figure 8 is a detailed flowchart of a process for generating analysis data according to various embodiments of the present application.

Referring to FIG. 8, in the step of generating the user's analysis data (S600), for each of the user's exercise movements or exercise muscles, the user's exercise results are divided into exercise muscle information and exercise intensity in the user's exercise result information. A step of generating first analysis data by visualizing the information based on the information (S610); Visualizing the exercise results for all of at least one exercise movement implemented by the user during a specific period based on at least some of the exercise muscle information and exercise intensity information in the user's exercise result information to generate second analysis data (S620) ); and generating third analysis data in which the exercise results for all of at least one exercise movement implemented by the user during the specific period are summarized and analyzed based on at least some information in the exercise result information of the user (S630). It may include the step of generating analysis data.

In some embodiments, one or more of the first analysis data, second analysis data, and third analysis data may be displayed in the form of a single screen. For example, the third analysis data, second analysis data, and first analysis data may be displayed on the user's smart phone in the form of a screen arranged sequentially in the order from top to bottom. However, the arrangement order of these analysis data is not limited to this.

The server 300 may transmit the first analysis data, second analysis data, and third analysis data to the user's electronic device 100 and display them on the screen. The screen displaying the analysis data may be referred to as an analysis screen. The analysis data can be used to create the analysis screen.

In various embodiments of the present application, the data generation unit 370 may generate first analysis data showing the user's exercise state information or exercise result information for each exercise muscle or exercise movement item.

FIG. 9 shows first analysis data showing exercise results by exercise type according to some embodiments of the present application, and FIG. 10 shows first analysis data showing exercise results by exercise muscle according to some other embodiments of the present application. Show analysis data.

Referring to FIGS. 9 and 10 , the first analysis data may include a first analysis image visualizing the location of the user's exercise muscles and exercise intensity for each exercise muscle as a result of the user's exercise.

The data generator 370 performs a first analysis that integrates and visualizes the location of the user's exercise muscles and the exercise intensity for each exercise muscle according to the most recent exercise or overall exercise results for a certain period of time for each exercise muscle or exercise type. Images can be created.

In various embodiments of the present application, the step of generating the first analysis data (S610) includes identifying the user's exercise muscle area in a preset body region based on the user's exercise muscle information, and the user's exercise muscle area. It may include the step of marking the user's exercise intensity for the corresponding exercise muscle or exercise type in the exercise muscle area identified based on the intensity information. The first analysis data includes the first analysis image of the user.

The first analysis data may also be generated based on the user's exercise state information and exercise result information during the specific period. In some embodiments, the specific period may be a unit period. The unit period is a period that repeats in units over time, and may be, for example, weekly, monthly, yearly, or other unit periods (e.g., quarterly, etc.).

In some embodiments, a screen displaying the first analysis data, the second analysis data described below, and the third analysis data may be configured to select one of a plurality of preset unit periods. The data generator 370 may provide analysis data generated based on at least part of the user's exercise state information and exercise result information for the unit period according to the selected unit period. For example, when the unit period is monthly, the data generation unit 370 may generate and provide first analysis data for September or first analysis data for another monthly period (e.g., October). there is.

Additionally, in some other embodiments, the specific period may be a period designated by the user. The user-specified period may be, for example, a rehabilitation period or other exercise period. The user's terminal device 100 provides a calendar from which the start date and end date of a specific period can be selected, and a specific period can be defined according to the user's selection input. Then, the data generator 370 may generate first analysis data, second analysis data, or third analysis data according to a period defined according to the selection input.

The first analysis image indicates the location of the user's exercise muscles and the exercise intensity for each exercise muscle.

The first analysis image of the user's exercise muscles may be generated for each exercise movement item or exercise muscle. For example, the data generating unit 370 may include chest muscles, shoulders, quadriceps femoris, upper abs, hamstrings, back muscles, calves, Multiple first analysis images can be generated for each exercise muscle, such as the spine, obliques, and biceps. The data generator 370 collects exercise muscle information from the user's exercise result information over a certain period of time, and visualizes the user's exercise intensity for the exercise muscle based on exercise intensity information for each exercise muscle, as shown in Figure 10. Analysis images can be created. Alternatively, the data generator 370 collects exercise muscle information and exercise intensity information for each exercise muscle from the user's exercise result information for a certain period of time, and based on the exercise intensity information for each exercise muscle for each exercise type. Thus, the first analysis image of FIG. 9, which visualizes the exercise intensity of each exercise muscle of the user for the above exercise type, can be generated.

The data generating unit 370 may identify the user's exercise muscle area based on the location of the user's exercise muscle in a body area shaped like a human body.

The first analysis image displays the exercise intensity of the exercise muscles identified in the body area, so that the user who receives the first analysis image can more quickly distinguish between the exercise muscles and non-exercise muscles actually used by the user and view the exercise intensity. It is designed for quick recognition. In other words, even if only the exercise intensity is indicated, the location of the exercise muscles is also naturally recognized.

In various embodiments of the present application, in order to generate a first analysis image of the user's exercise muscles, the user's exercise intensity for the corresponding exercise muscle is applied to the exercise muscle area identified based on the user's exercise intensity information. The marking step is to select a color corresponding to the user's exercise intensity information among preset colors to indicate exercise intensity so that the user's exercise muscle area can be distinguished from the non-exercise muscle area in the user's body image. It is applied to the muscles. In the first analysis image, the non-motor muscle area may be expressed in the same basic color (eg, gray) as the color applied to the body image, and the motor muscle area may be expressed in the basic color.

In some embodiments, the step of applying a preset color to indicate the exercise muscle to the exercise muscle in the user's exercise result information may include, for the exercise muscle in the user's exercise result information, the exercise intensity of the individual exercise muscle. Retrieving the color mapped to the exercise intensity of the individual exercise muscles from a pre-stored color table so that they can be visually distinguished; It may include the step of applying the color searched for each individual exercise muscle to each of the individual exercise muscles.

The data generation unit 370 may store in advance a color table that records the mapping relationship between the exercise intensity value and the brightness and/or saturation value within the color series. The color table consists of a plurality of colors with different brightness and/or saturation values within the same color family, and motion intensity values mapped to each color. In some embodiments, the color table may be composed of daily exercise intensity.

The data generator 370 uses the color table to apply different colors belonging to the same color series, such as the red color series in FIG. 10, to the individual motor muscle area depending on whether the exercise intensity of the individual motor muscle is weak or strong. can do. As shown in Figure 10, the higher the exercise intensity applied to a specific exercise muscle, the higher the saturation and/or brightness value, so that dark red is applied to the exercise muscle area with strong exercise intensity and pink is applied to the exercise muscle area with weak exercise intensity. A mapping relationship between color and exercise intensity may be set to have, but is not limited to this, and the opposite mapping relationship may also be set.

Additionally, the data generation unit 370 may generate first analysis data including the number of exercises or comparative evaluation information along with the first analysis image.

Specifically, in some embodiments, the step of generating the first analysis data (S610) includes at least one combination of information included in the user's exercise state information and exercise result information and the exercise of the personal training object. Generating exercise result comparison evaluation information by comparing at least one piece of information from a combination of information included in the state information and exercise result information; may further include.

The exercise result comparison evaluation information is information that quantifies the user's exercise results imitating the personal training video image for each of the user's exercise muscles or exercise types. The exercise result comparative evaluation information may be used as evaluation data to evaluate the user's exercise results during the specific period.

Similar to the first analysis image, the exercise result comparative evaluation information may be generated for each exercise muscle as shown in FIG. 10 . Then, the first analysis data may include the first analysis image and the exercise result comparative evaluation information.

In some embodiments, the exercise result comparative evaluation information may be a result of comparing exercise amount information in the exercise result information of the user and exercise amount information in the exercise result information of the personal training object for each exercise muscle. In this case, generating the exercise result comparative evaluation information includes calculating a reference exercise amount compared with the user's exercise amount; And it may include generating comparison data that compares and evaluates the amount of exercise of the user based on the reference amount of exercise during the specific period. In some embodiments, the exercise result comparative evaluation information may be a comparative evaluation of exercise results during the specific period.

As described above, the momentum quantifies the scale of performing a specific type of unit operation(s) during the scene section in the image subject to analysis, including the total operation time for performing the unit operations, the number of operations (i.e., number of repetitions), and the corresponding It can be calculated based on the exercise intensity of the movement.

The data generator 370 may calculate the reference exercise amount for the user based on the case where the personal training object performs the user's exercise movement with its own exercise amount equal to the number of movements and movement time of the user during the specific period ( S610). The amount of exercise of a personal training object is a value included in the exercise result information of the personal training object.

The data generator 370 may calculate the reference amount of exercise for the user based on the number of movements of the user, the movement time, and the amount of exercise of the personal training object related to the user's exercise movement.

Additionally, in some embodiments, the exercise result comparative evaluation information may be expressed as a ratio of the user's exercise amount based on the exercise amount of the personal training object. In this case, the data generator 370 may generate evaluation data in the form of a ratio of the user's exercise amount within the user's exercise result information to the reference exercise amount for the user.

The data generator 370 transmits the first analysis data to the user's electronic device 100, and the user's electronic device 100 displays the first analysis data on the screen as shown in FIG. 10, The first analysis image and exercise result comparison evaluation information within the first analysis data may be provided.

When the first analysis data including the exercise result comparative evaluation information is transmitted to the electronic device 100, the amount of exercise actually performed by the user is displayed in the form of a percentage (%) as shown in FIG. 10, so that the user can more easily perform his or her exercise. The results can be recognized visually.

In some embodiments, the first analysis image and exercise result comparison evaluation information for each exercise muscle or exercise type in the first analysis data may be provided according to ranking. The step of generating the first analysis data (S610) may further include ranking each exercise muscle based on the amount of exercise of the user during a specific period. Then, the first analysis data may further include ranking values for each Eundong muscle. Here, the higher the ranking, the greater the amount of exercise achieved using the exercise muscles.

The user's electronic device 100, which has received the first analysis data, may arrange and display the first analysis image and exercise result comparative evaluation information in ranking order for each exercise muscle. Additionally, in some embodiments, the user's electronic device 100 may provide only the first analysis image and exercise result comparison evaluation information of a preset threshold or higher. The electronic device 100 may be preset to a threshold ranking indicating higher momentum. The critical ranking may be top-five rank, but is not limited thereto. Then, the exercise results can be sequentially displayed in percentage form to see how much the exercise amount for the exercise muscles corresponding to 1st to 5th place matches the reference exercise amount.

However, it is not limited to this, and the exercise result comparative evaluation information may be generated for each exercise type, referring to FIG. 9 . Then, the first analysis data may include exercise result comparative evaluation information for each exercise event or exercise number information in addition to the first analysis image.

Figure 11 shows a screen displaying second analysis data and third analysis data according to various embodiments of the present application.

Referring to FIG. 11, the second analysis data may include a second analysis image visualizing the location of all exercise muscles of the user and the total exercise intensity for each exercise muscle as the entire exercise result of the user during the specific period. .

The step of generating the second analysis data (S620) may be to generate the second analysis image that visualizes the positions of the user's entire exercise muscles and the total exercise intensity for each exercise muscle during the specific period.

The second analysis image may correspond at least partially to the personal training visualization images shown in FIGS. 9 and 10, which visualize the location of exercise muscles and exercise intensity for each exercise muscle of the personal training object in the content to be imitated.

When a user selects a plurality of contents to imitate the exercise movements of a personal training object during a specific period of time, the second analysis image is a plurality of personal training visualization images in which the personal training visualization images of the selected plurality of contents overlap each other. It can respond to combinations of .

The step of generating the second analysis image (S620) includes adding up the exercise intensity for each individual exercise muscle from all exercise muscles in the user's body during a specific period of time; For all of the user's exercise muscles, searching for a color mapped to the summed exercise intensity of all exercise muscles from a pre-stored color table so that the exercise intensity of individual exercise muscles during a specific period can be visually distinguished; It may include the step of applying the color searched for each individual exercise muscle to each of the individual exercise muscles. In some embodiments, retrieving the color mapped to the combined exercise intensity of the individual exercise muscles from a pre-stored color table may include calculating the daily average exercise intensity of the individual exercise muscles during a specific period of time; It may include retrieving the color mapped to the daily average exercise intensity from a pre-stored color table.

In this way, using color to indicate exercise intensity in the exercise muscle area in the second analysis image is similar to the process of generating the first analysis image, and therefore detailed explanation will be omitted.

In some embodiments, generating the second analysis data may include generating the 2-1 analysis image from a front body region; and generating the 2-2 analysis image from the back body area. Then, as shown in FIG. 11, the front image or the back image can be provided to the user.

Referring again to FIG. 8, the third analysis data is data used to evaluate the user's exercise results during the specific period, and may be referred to as summary evaluation data or summary analysis data.

The step of generating the third analysis data (S630) may be generated by statistical analysis of the user's exercise results during a specific period based on at least some of the user's exercise status information and exercise result information during the specific period. there is.

In some embodiments, the statistical analysis may be a summation or average analysis of values within the exercise result information for each item during the specific period. The data generation unit 370 may generate third analysis data summarizing the result of adding up at least one item in the exercise result information. In one example, the data generating unit 370 calculates the total number of workouts by adding up the number of workouts of the user during the specific period, indicated as workouts in FIG. Third analysis data can be generated including the total exercise time, which is the sum of the exercise time, and the total calorie consumption, which is the sum of the user's calorie consumption during the specific period, indicated as Kcal in FIG. 11. The statistical analysis items (number of exercises, calories burned, exercise time) shown in FIG. 11 are merely examples and are not limited thereto.

Additionally, in some embodiments, the step (S600) may further include a step (S640) of generating fourth analysis data for the user group by statistically analyzing exercise results of a plurality of users.

The third analysis data is data that analyzes trends in exercise results of multiple users (eg, all service users).

In various embodiments of the present application, the fourth analysis data is analysis data based on exercise result information of a user group consisting of a plurality of users, including exercise trend data by region, exercise trend by BMI, exercise trend by device type, and exercise trend by body type. , exercise trends by height, or exercise trends by country.

The data generation unit 370 groups a plurality of users by region information, BMI information, device type information, body type information, height information, or country information in the user's profile data; Exercise trends of grouped users may be generated by statistical analysis of at least one piece of information in exercise result information of grouped users.

The exercise trend may represent at least one piece of information in exercise result information. The fourth analysis data may be a visualization of this movement trend.

In one example, the data generating unit 370 groups users by region; By statistically analyzing the average calories of grouped users, you can generate regional exercise trend data that visually represents the average calories of users in each region.

Alternatively, the data generating unit 370 groups users by BMI; By statistically analyzing which muscles grouped users mainly use as exercise muscles, it is possible to generate exercise trend data by BMI that visually displays the exercise muscle frequency of grouped users by muscle. In some embodiments, the data generator 370 ranks exercise muscles by frequency in order to generate exercise trend data by BMI - the ranking has a higher rank as the frequency increases; It may be configured to perform movements including generating exercise trend data for each BMI including exercise muscle frequencies above a critical rank.

Alternatively, the data generating unit 370 groups users by device type; By statistically analyzing which exercise movements grouped users prefer, it is possible to generate exercise trend data by device type that visually represents the preferred exercise movements of grouped users. In some embodiments, the data generator 370 ranks the user's preference for each exercise motion based on the user's preferred exercise motion information in order to generate exercise trend data for each device type - the preference ranking is divided into preferred exercise motions. Selected motor movements are given higher priority; It may be configured to perform movements including generating the exercise trend data by arranging exercise movements according to preference ranking.

Likewise, the data generation unit 370 can generate analysis data that visualizes the results of a statistical analysis of what exercise groups grouped by body type, height, and country prefer, as well as exercise trend data by device type. .

In this way, at least one of the metadata of the personal training video image generated by the system 1, the user's exercise status information, exercise result information, and analysis data can be used to make decisions related to the user's musculoskeletal exercise and medical support. there is. In particular, the data used is visualized so that users can more easily recognize the information, supporting faster decision making.

It will be clear to those skilled in the art that the motor motion analysis system 1 may also include other components. For example, the motor motion analysis system 1 may include other hardware elements necessary for the operations described herein, including input devices for data entry and output devices for printing or other data presentation. . In addition, it may further include a network, network interface, and protocol connecting the exercise motion analysis system 1 and an external device (eg, a user terminal, an external database, etc.).

When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs) or digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable PLDs (PLDs) configured to perform the embodiments of the present application. logic devices), field programmable gate arrays (FPGAs), etc. may be included in the components of the present application.

The operations of the exercise motion analysis system and method according to the embodiments of the present application described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented with a program product comprised of a computer-readable medium containing program code, which can be executed by a processor to perform any or all steps, operations, or processes described.

The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention discussed above has been described with reference to the embodiments shown in the drawings, but these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached patent claims.

Claims (20)

In a method of analyzing a user's exercise movements performed by a server to support decision-making related to musculoskeletal exercises and medical services and providing the user's exercise motion effectively in a visual aspect,
Obtaining a personal training video image representing at least one exercise motion of a personal training object from a server, and analyzing the exercise motion of the personal training object in the personal training video image to generate metadata of the personal training video image;
Receiving a video image of the user in which the user's exercise motion is captured from the server, wherein the user's exercise motion is the user's exercise among at least one exercise motion of the personal training object shown in the personal training video image selected by the user imitates the motor movement corresponding to the movement;
Generating posture information of the user by analyzing the user's exercise movements in the video image of the user, and generating exercise state information based on the posture information of the user;
In order to compare the exercise motion of the user and the corresponding exercise motion of the personal training object, generating exercise result information of the user by comparing the exercise state information of the user and the corresponding exercise state information of the personal training object. Step - the corresponding exercise state information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's exercise state information; and
Generating analysis data about the user's exercise movement based on at least one of information included in the user's exercise state information and the user's exercise result information - the analysis data is displayed on the user's electronic device Used to create analysis screens; Including,
The step of generating analysis data on the user's exercise movements is,
For each of the user's exercise movements or exercise muscles, visualizing the user's exercise results based on exercise muscle information and exercise intensity information in the user's exercise result information to generate first analysis data - the first analysis data includes a first analysis image of the user; and
A second analysis image that visualizes the exercise results of all at least one exercise movement implemented by the user during a specific period based on at least some of the exercise muscle information and exercise intensity information in the user's exercise result information is used as second analysis data. Generating step - the second analysis image is an image visualizing the position of the user's entire exercise muscles and the total exercise intensity for each exercise muscle during the specific period, including;
The exercise intensity is based on the load representing the stimulus applied to the exercise muscles during the exercise movement,
The step of generating first analysis data on the user's exercise muscles is:
In order to generate a first analysis image for each exercise muscle, the user's exercise muscle region is identified in a preset body region based on the user's exercise muscle information, and the user's exercise muscle region is selected among preset colors to indicate exercise intensity. Applying a color corresponding to the exercise intensity information to the identified exercise muscle to display the user's exercise intensity for the corresponding exercise muscle in the identified exercise muscle area,
The step of generating the second analysis image is,
Adding up the exercise intensity for each individual exercise muscle from the user's total exercise muscles during a specific period of time;
With respect to the user's exercise muscles, searching for a color mapped to the summed exercise intensity of individual exercise muscles from all exercise muscles in a pre-stored color table so that the exercise intensities of individual exercise muscles during a specific period are visually distinguished; and
Comprising the step of applying the color searched for each individual exercise muscle to each of the individual exercise muscles,
method. The method of claim 1, wherein generating metadata of the personal training video image comprises:
classifying the set of frame images within the personal training video image by scene to form a subset of frame images associated with the scene, the subset comprising at least one series of frame images, the series of frame images comprising a set of frame images It is a continuous frame image representing the unit motion of an object;
generating posture information of the personal training object appearing in the scene by processing frame images of the personal training video image in subset units;
generating exercise state information of the training object based on the posture information of the personal training object;
Obtaining exercise result information of the personal training object in the personal training video image based on at least one of operator input information of the server and exercise state information of the personal training object; and
Generating metadata of the personal training video image including some or all of the exercise state information of the personal training object and including some or all of the exercise result information of the personal training object.
method. The method of claim 1, wherein the step of generating posture information of the personal training object comprises:
Selecting at least one posture required to identify the corresponding motion as a key pose from among a series of continuous postures constituting the motion; and
A step of generating key posture information describing some postures selected as key postures among a series of postures as posture information of the personal training object,
The step of selecting the at least one posture as the key posture,
Based on the position and movement information of the skeleton and the position and movement information of the joints, at least one posture of the unit action is selected as the key posture for the unit action,
Characterized in that the exercise state information of the personal training object is generated based on the key posture information of the personal training object,
method. The method of claim 1, wherein the step of generating posture information of the personal training object comprises:
The input image in which the input object appears is analyzed using a pre-learned pose recognition model to recognize the type of posture to which the posture taken by the input object belongs, and to generate posture information of the object including the recognized posture type,
The pose recognition model extracts feature points of an object in an input posture through a pose estimation algorithm, and provides at least one motion information of joint motion and skeletal motion information of the object based on the extraction result of the feature points of the object. Calculate the probability that the input posture is classified into each of a plurality of preset posture types by inferring the correlation between at least one movement information among the calculated joint movement and skeletal movement information of the object and the posture type of the input object. and is a machine learning model configured to recognize the posture type of the input posture based on the calculated probability.
method. The method of claim 1, wherein the step of receiving, from the server, a video image of the user capturing the user's exercise movements,
transmitting content profile information for a plurality of personal training video images to the user's electronic device to display a content list on the user's electronic device;
Receiving a request for a target personal training video image showing a guide operation for the user from the user's electronic device;
providing a requested target personal training video image to the user's electronic device; and
Receiving, from the server, a video image of the user generated by filming the user's exercise movements on the user's electronic device,
wherein the video image of the user represents the user's motor movements mimicking the motor movements of the personal training object shown in the corresponding personal training video image,
method. The method of claim 3, wherein generating the user's exercise state information comprises:
classifying the set of frame images in the video image of the user by scene to form a subset of frame images associated with the scene, wherein the subset includes at least one series of frame images, the series of frame images of the user It is a continuous frame image representing a unit action;
generating posture information of the user appearing in the scene by processing frame images of the video image of the user in subset units; and
A step of generating exercise state information of the user based on the user's posture information,
The step of generating posture information of the user appearing in the scene is,
For each subset related to a scene included in the user's video image, inputting consecutive frame images included in each subset into a pose recognition model to recognize the user's posture type shown in the consecutive frame images. ; and
Characterized in that it includes the step of generating posture information of the user based on a recognition result of the posture type and an intermediate result calculated in a process of processing to recognize the posture type in the pose recognition model.
method. The method of claim 1, wherein generating the user's exercise result information comprises:
Comparing the user's posture information with corresponding posture information of the personal training object in the user's exercise state information, and calculating the user's posture similarity to the personal training object; and
Generating exercise result information of the user based on at least one of exercise result information of the personal training object, posture similarity of the user, and exercise state information of the user,
The step of calculating the posture similarity is,
determining corresponding joints and corresponding skeletons of the personal training object and the user, respectively, from the postures of the personal training object and the user's postures corresponding to each other; calculating a first similarity between the motion vectors of the user's joints in a series of postures of the user and the motion vectors of the joints of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user;
calculating a second similarity between the motion vector of the user's skeleton in a series of postures of the user and the motion vector of the skeleton of the personal training object shown in the series of postures of the personal training object corresponding to the series of postures of the user; and
Comprising a step of calculating a posture similarity between the user and a corresponding personal training object during the series of postures based on the first similarity and the second similarity,
method. The method of claim 1, wherein generating the user's exercise result information comprises:
Calculating the difficulty level, heart rate, or calorie consumption of the user by applying the similarity in motion between the user and the personal training object to the difficulty level, heart rate, or calorie consumption amount of the personal training object in the exercise result information of the personal training object;
determining the user's motion skeleton and motion joints shown in the user's video image based on the positions of each joint in each posture and the motion vector of the skeleton during a series of postures; and calculating the range of motion of the user's motion skeleton and the range of motion of the user's motion joints by tracking the position of the user's motion skeleton and the positions of the motion joints;
Detecting the user's exercise muscles among a plurality of muscles included in the user's body based on at least one of the position of the joint, the position of the skeleton, the movement vector of the joint, and the movement vector of the skeleton in the user's posture information during the unit operation. comprising steps,
method. The method of claim 8, wherein the step of generating the user's exercise result information includes:
It further includes calculating the user's exercise intensity applied to individual exercise muscles based on the difference between the posture information of the personal training object and the user's posture information,
The step of calculating the user's exercise intensity is,
When the user's posture matches the posture of the personal training object, calculating the exercise intensity of the personal training object as the user's exercise intensity; and
The step of calculating the user's exercise intensity includes correcting the user's exercise intensity according to the user's posture information based on at least one of the user's body profile data and exercise muscle usage level,
method. The method of claim 9, wherein the step of correcting the user's exercise intensity includes determining the user's exercise intensity according to the user's posture information based on the level of exercise muscle use, and determining the user's exercise intensity based on the user's exercise result information. Including the step of correcting the exercise intensity,
The step of correcting the user's exercise intensity in the user's exercise result information based on the user's exercise intensity according to the user's posture information and the level of exercise muscle use,
Obtaining muscle volume change information by analyzing the volume change of the body part surrounding the exercise muscle in the input image, and acquiring skin surface change information by analyzing skin change on the exercise muscle area pointing to the exercise muscle in the input image. At least one step of;
predicting the degree of use of the exercise muscle for each exercise muscle area shown in the input image by inferring the correlation between the acquired volume change information of the exercise muscle and the skin surface change information and the degree of use of the corresponding muscle; and
Characterized by comprising the step of correcting the user's exercise intensity based on the predicted degree of use of each exercise muscle,
method. delete According to paragraph 1,
The step of applying a preset color to indicate the exercise muscles to the exercise muscles in the user's exercise result information,
For the exercise muscles in the user's exercise result information, searching for a color mapped to the exercise intensity of the individual exercise muscle from a pre-stored color table so that the exercise intensity of the individual exercise muscle can be visually distinguished; and
Characterized in the step of applying the color searched for each individual exercise muscle to the individual exercise muscle,
method. The method of claim 12, wherein the first analysis data further includes exercise result comparative evaluation information,
The step of generating the first analysis data is,
Exercise results obtained by comparing at least one piece of information among the combination of information included in the user's exercise state information and exercise result information with at least one piece of information among the combination of information included in the exercise state information and exercise result information of the personal training object. Further comprising the step of generating comparative evaluation information,
The exercise result comparison evaluation information is characterized in that it is a result of comparing exercise amount information in the exercise result information of the user and exercise amount information in the exercise result information of the personal training object, for each exercise muscle or exercise type.
method. According to paragraph 1,
The step of retrieving the color mapped to the combined exercise intensity of the individual exercise muscles from a pre-stored color table is,
Calculating the daily average exercise intensity of individual exercise muscles during a specific period of time; and
Characterized in that it includes the step of retrieving the color mapped to the daily average exercise intensity from a pre-stored color table,
method. The method of claim 1, wherein the step of generating analysis data on the user's exercise movements comprises:
Statistically analyzing the user's exercise results during a specific period based on at least some of the user's exercise status information and exercise result information during a specific period, and generating third analysis data,
method. The method of claim 1, wherein the step of generating analysis data on the user's exercise movements comprises:
It further includes the step of generating fourth analysis data for a user group, which statistically analyzes the exercise results of a plurality of users,
The step of generating the fourth analysis data is,
forming a user group by grouping a plurality of users by information included in each user's profile data; and
A step of generating fourth analysis data showing exercise trends for the user group by performing statistical analysis on at least one of the exercise result information of individual users within the user group,
The fourth analysis data is characterized in that it is exercise trend data by region, exercise trend by BMI, exercise trend by device type, exercise trend by body type, exercise trend by height, or exercise trend by country.
method. A method for effectively providing visual information by analyzing a user's exercise movements to support decision-making related to musculoskeletal exercise and medical services according to any one of claims 1 to 10 and 12 to 16. A computer-readable recording medium that records a program that executes the program. In a system that effectively provides analysis results of a user's exercise movements in a visual aspect to support decision-making related to exercise and medical services,
The system includes a server and one or more user electronic devices, where the server includes:
Obtaining a personal training video image representing at least one movement motion of a personal training object, analyzing the movement movement of the personal training object in the personal training video image to generate metadata of the personal training video image,
A video image of the user in which the user's exercise motion is captured is received from the server, and the user's exercise motion is the user's exercise motion among at least one exercise motion of the personal training object shown in the personal training video image selected by the user. It imitates the movement movement corresponding to,
Generate posture information of the user by analyzing the exercise motion of the user in the video image of the user, and generate exercise state information of the user based on the posture information of the user,
In order to compare the user's exercise motion and the corresponding exercise motion of the personal training object, the user's posture information is compared with the corresponding posture information of the personal training object to generate exercise result information of the user. The corresponding posture information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's exercise state information, and
To generate analysis data on the user's exercise movement based on at least one of the information included in the user's exercise state information and the user's exercise result information - the analysis data is displayed on the user's electronic device Used to create screens - configured,
In order to generate analysis data on the user's exercise movements, the server stores the user's exercise results in the exercise muscle information and exercise intensity information in the user's exercise result information for each exercise motion or exercise muscle of the user. Visualize based on the generated first analysis data, wherein the first analysis data includes a first analysis image of the user;
A second analysis image in which the exercise results for all of at least one exercise movement implemented by the user during a specific period of time is visualized based on at least some of the exercise muscle information and exercise intensity information in the user's exercise result information is used as second analysis data. Generation - the second analysis image is an image visualizing the positions of the user's entire exercise muscles and the total exercise intensity for each exercise muscle during the specific period,
The exercise intensity is based on the load representing the stimulus applied to the exercise muscles during the exercise movement,
The server, in order to generate first analysis data about the user's exercise muscles,
In order to generate a first analysis image for each exercise muscle, the user's exercise muscle region is identified in a preset body region based on the user's exercise muscle information, and the user's exercise muscle region is selected among preset colors to indicate exercise intensity. Configured to display the user's exercise intensity for the exercise muscle in the identified exercise muscle area by applying a color corresponding to the exercise intensity information to the identified exercise muscle,
The server, to generate the second analysis image,
Total exercise intensity for each individual exercise muscle from the user's total exercise muscles during a specific period,
With respect to the user's exercise muscles, the color mapped to the combined exercise intensity of the individual exercise muscles from all exercise muscles is searched in a pre-stored color table so that each exercise intensity of the individual exercise muscles during a specific period is visually distinguished, and
Characterized in that it is configured to apply the color searched for each individual exercise muscle to each of the individual exercise muscles,
system.
In a method of analyzing a user's exercise movements performed by a server to support decision-making related to musculoskeletal exercises and medical services and providing the user's exercise motion effectively in a visual aspect,
Obtaining a personal training video image representing at least one exercise motion of a personal training object from a server, and analyzing the exercise motion of the personal training object in the personal training video image to generate metadata of the personal training video image;
Receiving a video image of the user in which the user's exercise motion is captured from the server, wherein the user's exercise motion is the user's exercise among at least one exercise motion of the personal training object shown in the personal training video image selected by the user imitates the motor movement corresponding to the movement;
Generating posture information of the user by analyzing the user's exercise movements in the video image of the user, and generating exercise state information based on the posture information of the user;
In order to compare the exercise motion of the user and the corresponding exercise motion of the personal training object, generating exercise result information of the user by comparing the exercise state information of the user and the corresponding exercise state information of the personal training object. Step - the exercise state information corresponding to the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the exercise motion of the user described by the exercise state information of the user; and
Generating analysis data about the user's exercise movement based on at least one of information included in the user's exercise state information and information included in the user's exercise result information - the analysis data is displayed on the user's electronic device Used to create analysis screens; Including,
The step of generating the user's exercise result information is,
Calculating the user's difficulty level, heart rate, or calorie consumption by applying the similarity in motion between the user and the personal training object to the difficulty level, heart rate, or calorie consumption of the personal training object in the exercise result information of the personal training object;
determining the user's motion skeleton and motion joints shown in the user's video image based on the positions of each joint in the posture and the motion vector of the skeleton during a series of postures;
calculating the range of motion of the user's motion skeleton and the range of motion of the user's motion joints by tracking the position of the user's motion skeleton and the positions of the motion joints;
Detecting the user's exercise muscles among a plurality of muscles included in the user's body based on at least one of the position of the joint, the position of the skeleton, the movement vector of the joint, and the movement vector of the skeleton in the user's posture information during the unit operation. step; and
Comprising: calculating the user's exercise intensity applied to individual exercise muscles based on the difference between the posture information of the personal training object and the user's posture information,
The step of calculating the user's exercise intensity is,
When the user's posture matches the posture of the personal training object, calculating the exercise intensity of the personal training object as the user's exercise intensity; and
Comprising the step of correcting the user's exercise intensity calculated according to the user's posture based on at least one of the user's body profile data and the level of exercise muscle use,
The step of correcting the user's exercise intensity is to correct the user's exercise intensity in the user's exercise result information based on the user's exercise intensity according to the user's posture information and the level of exercise muscle use,
Obtaining information on the volume change of the muscle by analyzing the change in volume of the body part surrounding the exercise muscle in the input image;
Obtaining change information on the skin surface by analyzing skin changes on the motor muscle area indicating the motor muscles in the input image;
predicting the level of use of the exercise muscle for each exercise muscle area shown in the input image by inferring the correlation between the acquired volume change information of the exercise muscle and the skin surface change information and the use level of the corresponding muscle; and
Comprising the step of correcting the user's exercise intensity based on the predicted level of use for each exercise muscle.
Characterized by
method. In a method of analyzing a user's exercise movements performed by a server to support decision-making related to musculoskeletal exercises and medical services and providing the user's exercise motion effectively in a visual aspect,
Obtaining a personal training video image representing at least one exercise motion of a personal training object from a server, and analyzing the exercise motion of the personal training object in the personal training video image to generate metadata of the personal training video image;
Receiving a video image of the user in which the user's exercise motion is captured from the server, wherein the user's exercise motion is the user's exercise among at least one exercise motion of the personal training object shown in the personal training video image selected by the user imitates the motor movement corresponding to the movement;
Generating posture information of the user by analyzing the user's exercise movements in the video image of the user, and generating exercise state information based on the posture information of the user;
In order to compare the exercise motion of the user and the corresponding exercise motion of the personal training object, generating exercise result information of the user by comparing the exercise state information of the user and the corresponding exercise state information of the personal training object. Step - the corresponding exercise state information of the personal training object is a result of analyzing the exercise motion of the personal training object corresponding to the user's exercise motion described by the user's exercise state information; and
Generating analysis data about the user's exercise movement based on at least one of information included in the user's exercise state information and the user's exercise result information - the analysis data is displayed on the user's electronic device Used to create analysis screens; Including,
The step of generating the user's exercise result information is,
Calculating the difficulty level, heart rate, or calorie consumption of the user by applying the similarity in motion between the user and the personal training object to the difficulty level, heart rate, or calorie consumption amount of the personal training object in the exercise result information of the personal training object;
determining the user's motion skeleton and motion joints shown in the user's video image based on the positions of each joint in each posture and the motion vector of the skeleton during a series of postures;
calculating the range of motion of the user's motion skeleton and the range of motion of the user's motion joints by tracking the position of the user's motion skeleton and the positions of the motion joints;
Detecting the user's exercise muscles among a plurality of muscles included in the user's body based on at least one of the position of the joint, the position of the skeleton, the movement vector of the joint, and the movement vector of the skeleton in the user's posture information during the unit operation. step; and
Comprising: calculating the user's exercise intensity applied to individual exercise muscles based on the difference between the posture information of the personal training object and the user's posture information,
The step of calculating the user's exercise intensity is,
When the user's posture matches the posture of the personal training object, calculating the exercise intensity of the personal training object as the user's exercise intensity; and
Comprising the step of correcting the user's exercise intensity calculated according to the user's posture based on at least one of the user's body profile data and exercise muscle usage level,
The step of correcting the user's exercise intensity is to correct the user's exercise intensity in the user's exercise result information based on the user's exercise intensity according to the user's posture information and the level of exercise muscle use,
Obtaining information on the volume change of the muscle by analyzing the change in volume of the body part surrounding the exercise muscle in the input image;
Obtaining change information on the skin surface by analyzing skin changes on the motor muscle area indicating the motor muscles in the input image;
predicting the level of use of the exercise muscle for each exercise muscle area shown in the input image by inferring the correlation between the acquired volume change information of the exercise muscle and the skin surface change information and the use level of the corresponding muscle; and
Comprising the step of correcting the user's exercise intensity based on the predicted level of use for each exercise muscle,
The step of predicting the level of use of the exercise muscles is,
The level of use of the exercise muscles is calculated using a pre-learned muscle use analysis model, and the muscle use analysis model analyzes muscle use information visually expressed in the input image to indicate whether the exercise muscles were actually used. A machine learning model previously trained to calculate the level of use of the exercise muscles, comprising: a first analysis model that segments exercise muscles in continuous input images showing motion and calculates volume changes of the divided exercise muscles during motion; a second analysis model that segments motor muscles from continuous input images representing small muscles and calculates changes in the skin surface within the segmented motor muscles during movement; And an inference model that calculates the level of use for the muscles of the object based on the analysis results of the first analysis model and the analysis results of the second analysis model,
method.

Images