KR101938361B1 - Method and program for predicting skeleton state by the body ouline in x-ray image - Google Patents

Abstract

The present invention relates to a method and a program for predicting a skeleton state based on a body outline in an X-ray image which can predict a skeleton state without visiting a hospital. According to an embodiment of the present invention, the method for predicting a skeleton state based on a body outline in an X-ray image comprises: a skeleton evaluation index calculation step (S600) of learning, by a computer, big data based on a plurality of X-ray image data to calculate a skeleton evaluation index in accordance with a body image, wherein the big data is constructed by storing a skeleton image and a body image extracted from the X-ray image data, and the body image is a body outline included in the X-ray image; a step (S800) of receiving a body photographing image of a specific test subject from a user client; and a step (S1000) of applying the skeleton evaluation index to the body photographing image to calculate a skeleton state.

Description

METHOD AND PROGRAM FOR PREDICTING SKELETON STATE BY THE BODY OUTPUT IN X-RAY IMAGE < RTI ID = 0.0 >

The present invention relates to a method and a program for predicting a skeletal state based on a body contour in an X-ray image.

The body shape changes abnormally due to the improper posture while growing. As the skeleton changes abnormally, the external appearance of the body changes. Patients with scoliosis or scarring tend to develop during the growing season, and difficulties in correcting if they are not confirmed early.

However, in order to determine whether there is an abnormality in the skeleton, it is necessary to visit the hospital to perform X-ray imaging, but it is difficult to perform X-ray imaging frequently. Therefore, there is a need for a method that can predict the skeletal state without X-ray imaging and guide the user (i.e., the parent of the child).

In the past, the feature points were extracted from the patient 's body, and the body shape was analyzed based on the angle between the lines connecting the feature points. However, if the feature point extraction is not accurate, the prediction result may be inaccurate, and it is difficult to grasp the detailed difference of the body shape with a limited number of feature points.

The present invention applies skeleton image and body contour lines in an X-ray image to machine-learning algorithms and applies the skeletal state evaluation index to an image of a body of a subject to be imaged by a camera to predict a skeletal state And a program for predicting a skeletal state based on a body contour line in an X-ray image.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, there is provided a method of predicting a skeleton state based on a body contour in an X-ray image, the method comprising: acquiring a plurality of X- The computer storing the skeleton image and the body image extracted from the X-ray image data to construct big data; Learning the big data to calculate a skeleton evaluation index according to a body image; A body imaging image receiving step of receiving a body imaging image of a specific examining subject from a user client, wherein the body imaging image is imaged by a camera provided in the user client; And calculating a skeletal state by applying the skeletal evaluation index to the body image, wherein the body image is a body contour included in the X-ray image.

According to another embodiment of the present invention, the skeleton evaluation index calculation step is characterized in that the analysis of the big data is performed based on a deep learning algorithm using a depth neural network.

Further, in another embodiment, the method further includes correcting the body image to be applicable to the skeletal evaluation index.

According to another embodiment of the present invention, the user client displays a body photographing standard for applying the skeleton evaluation index on a screen using a user interface.

According to another embodiment, the X-ray image data receiving step receives the body condition of the person to be photographed and matches the X-ray image data with the X-ray image data, and the skeleton evaluation index calculating step calculates the skeleton evaluation index using the body- And the skeletal state calculating step calculates the skeleton evaluation index by learning the body condition together, and the body image receiving step receives the user's body condition from the user client, and the skeleton state calculating step includes: The above skeletal evaluation standard is applied.

According to another embodiment of the present invention, the method may further include comparing the body image with the X-ray image data when the X-ray image data for the examinee is newly acquired; And refining the comparison result by reflecting it on the learning model.

According to another embodiment, the skeleton evaluation index calculating step may include calculating a skeleton evaluation index for each body part by learning a skeleton image and a body shape image for each of a plurality of body parts included in the X-ray image data, do.

According to another embodiment of the present invention, the method further includes the step of calculating correction posture information based on the skeletal state and providing the corrected posture information to the user client.

The skeleton state prediction program based on the body contour based on the X-ray image according to another embodiment of the present invention executes the skeleton state prediction method based on the body contour based on the deep learning based X-ray image combined with the hardware computer And stored in the medium.

According to the present invention as described above, it is possible to predict and guide the skeletal state of the patient in a state in which the hospital is not desired. In other words, since the skeletal state is predicted based on the body contour affected by the skeletal state, the user is informed of the abnormality of the skeletal state of the examinee, .

FIG. 1 is a flowchart of a skeletal state predicting method based on a body contour in an X-ray image according to an exemplary embodiment of the present invention.
2 is an exemplary view showing X-ray image data and a body image of a normal person and a scoliosis patient according to an embodiment of the present invention.
FIG. 3 is an exemplary diagram comparing skeleton images based on X-ray image data of a normal person and a swinging patient according to an embodiment of the present invention.
4 is a view showing a body image obtained from a user client and a body contour image extracted from a body image according to an embodiment of the present invention.
5 is a flowchart illustrating a skeletal state predicting method based on body contours in an X-ray image, which further includes a step of refining a learning model based on new X-ray image data of a subject to be examined .
FIG. 6 is a flowchart illustrating a skeleton state predicting method based on a body contour in an X-ray image, which further includes a process of providing posture correction information based on a skeletal state, according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating a skeleton state predicting method based on a body contour in an X-ray image, which further includes a step of constructing learning object data by a computer according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element.

The term " computer " as used herein includes various devices capable of performing arithmetic processing to visually present results to a user. For example, the computer may be a smart phone, a tablet PC, a cellular phone, a personal communication service phone (PCS phone), a synchronous / asynchronous A mobile terminal of IMT-2000 (International Mobile Telecommunication-2000), a Palm Personal Computer (PC), a personal digital assistant (PDA), and the like. In addition, the computer may be a server computer connected to various client computers. Further, the computer may be a combination of a plurality of computing devices.

In the present specification, 'X-ray image data' means image data obtained through X-ray imaging.

In the present specification, 'skeleton image (10)' refers to an image obtained by extracting a skeleton part in an X-ray image. The 'skeleton image 10' may be an image generated by extracting only a skeleton area, or may be X-ray image data in which a skeleton boundary is identified.

In the present specification, the 'body image (20)' is a body contour included in the X-ray image.

In the present specification, the 'body image 30' refers to an image of a body of a person to be inspected using a user client.

As used herein, the term " user client " refers to any device capable of capturing an image through a camera and transmitting an image captured by a computer.

In this specification, the term "subject to be photographed" means a patient who has taken an X-ray image.

In the present specification, the term 'subject to be examined' refers to a subject to which a body image is photographed.

Below. A detailed description of a skeletal state predicting method based on body contour lines in an X-ray image according to embodiments of the present invention will be described.

FIG. 1 is a flowchart of a skeletal state predicting method based on a body contour in an X-ray image according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a skeleton state predicting method based on a body contour line in an X-ray image according to an embodiment of the present invention is a method in which a computer learns big data based on a plurality of X- A skeleton evaluation index step (S600; skeleton evaluation index calculation step) for calculating an index; A step (S800) of receiving a body imaging image (30) for a specific examinee from a user client; And calculating a skeletal state by applying the skeleton evaluation index to the body image (S1000). Hereinafter, a detailed description of each step will be described.

The computer learns big data based on a plurality of X-ray image data, and calculates a skeleton evaluation index according to the body image (S600; skeleton evaluation index calculation step). The big data is constructed by storing a skeleton image and a body image extracted from a plurality of X-ray image data. 2, the X-ray image data includes a skeleton image 10 of a subject to be photographed and a body image 20 which is a body contour line generated in the X-ray image as an X-ray is transmitted through the body, The big data is obtained by matching the skeleton image 10 and the body image 20 extracted from the X-ray image data. The big data (that is, the data to be learned) can be constructed by the computer directly based on the X-ray image data, or the big data based on the X-ray image data constructed in the separate server.

In one embodiment of the process of constructing the big data based on a plurality of X-ray image data, the computer extracts body contour lines in the X-ray image data through image processing, and the skeleton image 10 included in the X- (I.e., learning target data). The computer constructs a data set by matching the skeleton image 10 and the body image 20 with a plurality of X-ray image data to form learning target data for learning.

Specifically, the body image 20, which is the body contour of a patient with a normal vertebral skeleton (see Fig. 2 (a)), is symmetric in both, but a patient with scoliosis (particularly a pediatric patient) Are curved without being symmetrical on both sides of the body image 20 as in the bent vertebra direction. Accordingly, since the body image 20 follows the skeletal state, the computer acquires the skeleton image 10 and the body image 20 as analysis target data for skeletal state prediction based on the body contour. In addition, as shown in FIG. 3, since the body shape varies depending on the skeletal state, the computer extracts the body image 20 and the skeleton image 10 from the X-ray image data obtained by photographing the body, .

In one embodiment of a method of forming data to be learned, the computer separately generates a body image 20 based on the body contour in the X-ray image data, extracts the other part from the X- A skeleton image 10 can be separately generated. In another embodiment of the method of forming learning data, the computer displays a body contour in the X-ray image data, and extracts and displays a skeleton (e.g., vertebrae) outline in the X-ray image data. Thus, the computer can generate analysis target data including both the body image 20 information and the skeleton image 10 information in the X-ray image data.

Further, in another embodiment, the big data (i.e., the data to be learned) includes the body condition of the imaging subject matched with the X-ray image data. The physical condition may be a height, a body weight, a body fat mass, and the like of the subject to be photographed. Even if the patient has the same skeletal state, the body may differ in appearance due to the amount of body fat or the like. Therefore, the computer performs learning of the learning target data including the body condition data together with the X-ray image data.

The skeletal evaluation index means an index for evaluating the skeletal state through the body image 20 by calculating the correlation between the body image 20 and the skeleton image 10. [ The skeleton evaluation index according to the body image 20 is used for evaluation of the body image 30 as described later.

In one embodiment, the computer learns a skeleton image 10 and a body image 20 for each of a plurality of body parts included in the X-ray image data, and calculates a skeleton evaluation index for each body part. Since the criterion for evaluating the skeletal state according to the body image 20 may differ depending on the body parts, the computer learns the skeleton image 10 and the body image 20 for each body part.

The computer learns the image to be analyzed using various machine learning algorithms. In one embodiment, the skeleton evaluation index calculation step S600 performs analysis of the big data on the basis of a deep learning algorithm using a depth neural network. The Deep Neural Network (DNN) refers to a system or network in which one or more layers are built in a computer to perform a judgment based on a plurality of data.

For example, a neural network may be implemented with a set of layers including a Convolutional Pooling Layer, a locally-connected layer and a fully-connected layer. The convolutional pulling layer or the local connection layer may be configured to extract features in the image. The complete link layer can determine the correlation between image features. In some embodiments, the overall structure of the depth-of-field neural network may be of the form that the local connection layer is followed by the convolutional pulling layer and the full connection layer is in the local connection layer. The in-depth neural network may include various criteria (i.e., parameters) and may add new criteria (i.e., parameters) through input image analysis.

In one embodiment of the present invention, the depth neural network is a structure called convolutional neural network suitable for acquisition of learning data. The feature extraction layer (Self-Organizing Layer) that learns a feature having the greatest discriminative power from given image data, And a prediction layer (Prediction Layer) for learning a prediction model so as to have the highest prediction performance based on extracted features.

The Feature Extraction layer consists of a Convolution Layer that creates a Feature Map by applying a plurality of filters to each region of the image, and features that are invariant to changes in position or rotation by spatially integrating feature maps. And can be formed in a structure in which an integral layer (a pooling layer) is alternately repeated several times. Through this, it is possible to extract various levels of features from low-level features such as points, lines, and surfaces to complex and meaningful high-level features.

The convolution layer obtains a feature map by taking the nonlinear activation function of the inner product of the filter and the local receptive field for each patch of the input image, CNN is characterized by the use of filters with sparse connectivity and shared weights. This connection structure reduces the number of parameters to be learned and makes learning through the backpropagation algorithm efficient, resulting in improved prediction performance.

The integration layer (Pooling Layer or Sub-sampling Layer) creates a new feature map by utilizing the local information of the feature map obtained from the previous convolution layer. Generally, the feature map newly generated by the integration layer is reduced to a size smaller than the original feature map. As representative integration methods, Max Pooling which selects the maximum value of the corresponding region in the feature map, And average pooling to obtain the average value of the area. The feature map of the integrated layer generally has less influence on the position of any structure or pattern existing in the input image than the feature map of the previous layer. That is, the integrated layer can extract more robust features in the local changes such as noise or distortion in the input image or the previous feature map, and this feature can play an important role in the classification performance. The role of the other integrated layer is to reflect the characteristics of a wider area as it goes from the deep structure to the upper learning layer. As the feature extraction layer accumulates, the lower layer reflects local characteristics and climbs up to the upper layer It is possible to reflect more abstract features of the whole image.

In this way, the feature finally extracted through repetition of the convolution layer and the integration layer is that the classification model such as Multi-layer Perception (MLP) or Support Vector Machine (SVM) -connected layer) to be used for classification model learning and prediction.

However, the structure of the depth-based neural network described in the embodiment of the present invention is not limited to this, and may be formed of various structures of neural networks.

In another embodiment, in the skeleton evaluation index calculation step (S600), the computer calculates a skeleton evaluation index by learning the body condition together with the body image (20) and the skeleton image (10). That is, the computer learns the skeleton image 10, the body image 20, and the data set matched with the new body image by the user to generate a skeleton evaluation index reflecting the physical condition of the person to be photographed.

The computer receives the body image 30 of the specific examinee from the user client (S800: receiving the body image 30). The body image 30 is photographed by a camera provided in the user client. That is, a user having a user client (for example, a smart phone) photographs a body image 30 of a test subject (e.g., a child) using a camera of a user client, And receives the photographed image 30. It is preferable that the body image 30 is photographed in a state in which the body part of the examinee is exposed (i.e., the body).

In one embodiment, the user client displays a body photographing standard for applying the skeleton evaluation index on a screen as a user interface. For example, the user client may display on the screen a guideline on which the body contour of the examinee should correspond, so as to acquire the body photographed image 30 in which the body is photographed suitable for applying the skeletal evaluation index . In addition, since the body image 30 may be distorted according to the placement state of the user client when the user captures the body of the user using the user client, the user client may use the motion sensor or camera included therein to analyze It is possible to confirm the arrangement state of the client and guide the user to acquire the undistorted body image 30.

Thereafter, the computer calculates the skeletal state by applying the skeletal evaluation index to the body image 30 (S1000). As shown in FIG. 4, if the contour is extracted from the body image 30, the computer can generate the same body contour extracted from the X-ray image. Accordingly, the skeletal evaluation index is applied to the body image 30 corresponding to the body image 20 generated by extracting the body contour line from the X-ray image, and the computer calculates the skeletal state The result can be predicted.

In addition, when the skeletal evaluation index is generated by reflecting the physical condition of the person to be photographed, the computer calculates the skeletal state by applying the skeletal evaluation standard to the physical condition of the person to be examined and the X-ray image data.

Further, in another embodiment, the step of correcting the body photographed image 30 so as to be applicable to the skeletal evaluation index. For example, when the body imaging image 30 is photographed in a tilted state (for example, when the upper body is inclined at a specific angle when the upper body of the examinee is photographed), the computer performs correction such as image rotation So as to approximate to the X-ray image data.

5, when the X-ray image data for the examinee is newly acquired, comparing the body image 30 with the X-ray image data (S1200); And refining the comparison result to the learning model (S1300). After receiving the skeletal status, the subject can visit the hospital to take X-ray images. The computer can compare the newly acquired X-ray image data with the body image (30) photographed by the user client to determine the accuracy of the skeleton state calculated based on the body image (30). Thereafter, the computer refines the algorithm by reflecting the result of comparing the skeletal image 10 obtained through the body image 30 with the skeletal image 10 obtained through the X-ray image data to the skeletal state learning model to which the machine learning algorithm is applied And performs a process of increasing the skeletal state calculation accuracy.

In another embodiment, the method further includes calculating (S1400) the correction posture information based on the skeletal state and providing the corrected posture information to the user client as shown in FIG. 6 (S1400). The computer stores attitude information for becoming a normal skeleton according to the skeletal state, and can provide an appropriate correction attitude to the user client based on the skeletal state calculated based on the body image 30.

In another embodiment, as shown in FIG. 7, the computer acquires a plurality of X-ray image data (S200); And a step (S400) in which the computer stores the skeleton image 10 and the body image 20 extracted from the X-ray image data to construct big data (S400). That is, the computer can perform a process of building big data (i.e., learning target data) based on a plurality of X-ray image data.

The computer acquires a plurality of X-ray image data (S200). The computer can receive a plurality of X-ray image data from the medical image server in which the medical image data is stored in the hospital. Since the X-ray image data stored in the medical image server is classified with respect to the photographed body part, the taken direction, and the like, the computer can acquire the X-ray image data in the desired photographing direction from the medical image server.

The computer may receive data on a person to be photographed at the time of receiving the X-ray image data, and may match with the X-ray image data. For example, the computer can receive the age, physical condition, etc. of the subject to be photographed from the hospital server. Even if the patient has the same skeletal state, the external appearance of the body may be different depending on the body fat amount, etc. Therefore, the computer can receive the body condition data to be used together with the X-ray image data from the hospital server.

The computer stores the skeleton image 10 and the body image 20 extracted from the X-ray image data to construct big data (S400). As shown in FIG. 2, the X-ray image data includes a skeleton image 10 of a subject to be photographed and a body image 20 which is a body contour line generated as an X-ray passes through the body. Accordingly, the computer extracts the body contour lines in the X-ray image data through the image processing and extracts the skeleton image 10 contained in the X-ray image data. The computer forms the learning object data for learning by matching the skeleton image 10 and the body image 20. A computer forms a data set (Dataset) by forming a process of forming learning target data with respect to a plurality of X-ray image data.

The above-described method of predicting skeletal state based on body contour in an X-ray image according to an embodiment of the present invention may be implemented as a program (or application) to be executed in combination with a hardware computer and stored in a medium.

The above-described program may be stored in a computer-readable medium such as C, C ++, JAVA, machine language, or the like that can be read by the processor (CPU) of the computer through the device interface of the computer, And may include a code encoded in a computer language of the computer. Such code may include a functional code related to a function or the like that defines necessary functions for executing the above methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions in a predetermined procedure can do. Further, such code may further include memory reference related code as to whether the additional information or media needed to cause the processor of the computer to execute the functions should be referred to at any location (address) of the internal or external memory of the computer have. Also, when the processor of the computer needs to communicate with any other computer or server that is remote to execute the functions, the code may be communicated to any other computer or server remotely using the communication module of the computer A communication-related code for determining whether to communicate, what information or media should be transmitted or received during communication, and the like.

The medium to be stored is not a medium for storing data for a short time such as a register, a cache, a memory, etc., but means a medium that semi-permanently stores data and is capable of being read by a device. Specifically, examples of the medium to be stored include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, but are not limited thereto. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed to a network-connected computer system so that computer-readable codes may be stored in a distributed manner.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

10: Skeleton image 20: Body image
30: Body imaging video

Claims (9)

A processor of a computer acquiring a plurality of X-ray image data;
A big data construction step in which the processor constructs big data by matching and storing a skeleton image and a body image extracted from X-ray image data;
A skeleton evaluation index calculation step of the processor calculating a skeleton evaluation index according to a body image by learning large data based on a plurality of X-ray image data, wherein the body image is a body contour included in the X-ray image;
A body imaging image receiving step of receiving a body imaging image of a specific examinee from a communication module of a user client, the body imaging image being photographed by a camera provided in the user client;
Wherein the processor calculates the skeletal state by applying the skeletal evaluation index to the body image,
Wherein the big data construction step comprises:
The processor extracts a body contour line in the X-ray image data through image processing, and extracts a skeleton image included in the X-ray image data to construct big data. The skeleton image and the skeleton image of a plurality of X- A data set is constructed from data to be learned by matching the body image,
The step of calculating the skeletal state includes:
Wherein a skeleton state corresponding to a body image of a body photographing image photographed by a camera provided in the user client is calculated using the skeleton evaluation index,
A method for predicting skeletal state based on body contour in X - ray image. The method according to claim 1,
The skeleton evaluation index calculating step may include:
A method for predicting a skeletal state based on body contours in an X-ray image, characterized in that the analysis of the big data is performed based on a deep learning algorithm using a depth neural network. The method according to claim 1,
And correcting the body imaging image so that the processor can apply the skeleton evaluation index to the skeleton evaluation index. The method according to claim 1,
The user client,
Wherein the body imaging standard for applying the skeletal evaluation index is displayed on a screen on a user interface. The method according to claim 1,
The big data includes:
Ray image data by matching the physical condition of the subject to be photographed with the X-ray image data,
The skeleton evaluation index calculating step may include:
Calculating a skeleton evaluation index by learning the body condition together with the body image and the skeleton image,
The body imaging image receiving step may include:
Wherein the communication module of the computer receives the physical condition of the user from the communication module of the user client,
The skeleton state calculating step may include:
Wherein the skeletal evaluation index is applied to the physical condition and the X-ray image data. The method according to claim 1,
Comparing the body image with the X-ray image data when the processor acquires X-ray image data for the examinee; And
Further comprising the step of the processor refining the comparison result to the learning model to refine the skeleton state based on the body contour in the X-ray image. The method according to claim 1,
The skeleton evaluation index calculating step may include:
Wherein the skeleton image is generated by learning a skeleton image and a body image for each of a plurality of body parts included in the X-ray image data, and calculating a skeleton evaluation index for each body part. The method according to claim 1,
Wherein the processor is further configured to calculate calibration attitude information based on the skeletal state and to provide the calibration attitude information calculated by the communication module of the computer to the user client, Method of predicting skeletal status. 9. A skeleton state prediction program based on body contour lines in an X-ray image, which is stored in a medium in combination with a computer which is hardware, in order to execute the method of any one of claims 1 to 8.

Images