KR20230168954A - Device and method for analyzing functional modalities - Google Patents

Abstract

Embodiments provide a functional analysis device comprising: an acquisition module that acquires information from one or more modalities; A functional analysis module that generates association information between different modalities or between different basic information within a single modality; and an analysis result providing module that provides a functional analysis result including at least one of one or more related information and integrated information associated with the basic information. By providing related and integrated information between different modalities or modalities, the accuracy of diagnosis can be increased, and it can help with diagnosis subdivision, treatment selection, and cause determination, and can help reduce risk found in multiple modalities. Confirmation of findings, extensive evaluation, and rapid emergency treatment can improve patient safety.

Description

Functional analysis device and method {Device and method for analyzing functional modalities}

This specification relates to a functional analysis device, which assists the interpretation results of chest radiology images by providing related and integrated information between different modalities or the same modality, such as chest radiology images and electrocardiograms/pulmonary function tests. do.

Chest X-ray is the most commonly used image in medical history, and it is inexpensive and allows you to easily obtain information about the thoracic structure. Because changes that occur in the heart or lungs are often reflected in chest radiography images, various morphological information can be provided. However, because only morphological information can be obtained when taking a chest radiography image, there are limitations in the performance and interpretation of the chest radiology image analysis algorithm.

In order to solve the above problems, electrocardiography (ECG) can be used to assist in the interpretation of chest radiography images.

An electrocardiogram is a record of the action currents generated in the myocardium as the heart beats. An electrocardiogram is a very simple test, the results can be checked right away, and it is also useful when continuously monitoring a patient.

Recently, it has been reported that functional evaluation of the heart can be performed using an electrocardiogram. This functional information is provided together with the chest radiology image analysis results to enable more accurate evaluation, and can be used as a functional label when developing artificial intelligence based on chest radiography.

In one example, when it is necessary to distinguish between pulmonary edema, which is fluid in the lungs, and pneumonia, which is inflammation of the lungs, accuracy in disease determination can be improved by using not only chest radiography images but also electrocardiograms.

In one example, when it is necessary to diagnose pulmonary embolism, which causes blood clots in the lungs, the accuracy in determining the disease can be further improved by analyzing the decline in right ventricular function using an electrocardiogram. In addition, when diagnosing cardiac tamponade or ARDS disease, not only chest radiography images but also electrocardiograms can be used to provide further information necessary for causal relationships and treatment decisions.

In one example, the results of functional analysis using an electrocardiogram can be used as a training label for chest radiography artificial intelligence to predict functional evaluation results using only chest radiography.

Embodiments of the present application include an acquisition module that acquires information from one or more modalities; A functional analysis module that generates at least one of association information between different modalities or between different basic information within a single modality; and an analysis result providing module that provides a functional analysis result including at least one of one or more related information and integrated information associated with the basic information.

In one example, the one or more modalities include a chest radiology modality and an electrocardiogram modality, and the different modalities include a chest radiology modality; and one or more of the electrocardiogram modality and pulmonary function test modality; The basic information may be information about morphological characteristics provided from chest radiation modality, and the related information may be functional abnormality information related to the basic information based on the morphological characteristics.

In one example, the integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and a task of diagnosing or predicting a specific disease, The integrated information can be used as basic information and related information for new items.

In one example, the information about the functional abnormality may include one or more of the presence, probability, and severity of the abnormality related to the morphological characteristic.

In one example, the basic information is medical information about pulmonary edema, cardiac hypertrophy, coronary artery disease/calcification, and valve disease/calcification, and the related information is the presence and probability of heart failure, predicted ejection fraction, and heart valve. abnormalities, myocardial infarction including ST segment elevation myocardial infarction (STEMI), probability of acute coronary syndrome and cardiomyopathy, blood Troponin, ProBNP, pH, Lactate, TCO2, Serum Creatinine, C-reactive protein (C -reactive protein) results, may be medical information about echocardiography summary.

In one example, the basic information is medical information about lung infectious diseases including pneumonia, pulmonary tuberculosis, and lung abscess, and pneumonia including cirrhosis, infiltration, ground glass, and atelectasis, and the related information is shock, respiratory failure, risk of death, Presence and probability of heart failure, predicted ejection fraction results, blood Troponin T/I, ProBNP, pH, Lactate, TCO2, Serum Creatinine, WBC, ESR, Eosinophil, C-reactive protein results It may be medical information about.

In one example, the basic information is medical information about pulmonary embolism, and the related information is about right ventricular failure, pulmonary hypertension, blood D-dimer, Troponin I, ProBNP results, echocardiography, and cardiac arrhythmia ultrasound. This may be medical information.

In one example, the basic information may be medical information about pericardial effusion, and the related information may be medical information about pericardial effusion, cardiac tamponade, right ventricular failure risk, shock, respiratory failure and death risk, blood CRP, and IGRA test.

In one example, the basic information is medical information about ARDS, and the related information includes shock, respiratory failure, risk of death, right ventricular failure, pulmonary hypertension, SpO2, Troponin I, ProBNP, SpO2, and FIO2. It may be medical information about vital signs.

In one example, the basic information is medical information about emphysema, bronchitis, and interstitial lung disease, and the related information is left ventricular failure, right ventricular failure, pulmonary hypertension, CRP of blood, It may be medical information about at least one of Troponin I, ProBNP, FVC, FEV1, FEV1/FVC values and interpretation, summary of pulmonary function test results, and raw data and processed products of pulmonary function tests.

In one example, the acquisition module includes first basic information that is chest radiology data; Second basic information, which is electrocardiogram data; Obtain additional basic information from at least one of a blood test result, a pulmonary function test result, and a sample test result, and the related information includes functional abnormalities associated between the first basic information, the second basic information, and the additional basic information. It may include information about.

In one example, the chest radiation data is chest radiation image data, the electrocardiogram data is electrocardiogram image data or time series data, the blood test result data is one or more selected from text, numerical values, and images, and the pulmonary function test result. The data is one or more selected from time series data, image data, text data, and numerical data, and the sample test result data may be one or more of text or numerical data.

In one example, the chest radiology image data is an original chest radiology digital image, a converted image of the original, captured image data or screen capture image data, and an image scanned after translating an expert's opinion on the chest radiology digital image into writing. , the time series data is time series data of a wave-shaped signal or an image showing it on a two-dimensional plane, the electrocardiogram image data is photographed image data, screen capture image data, or image data obtained from an electronic medical record system, and the blood test The result data is user input data, data obtained from an electronic medical record system, or OCR (artificial intelligence document recognition) data of a test result, and the pulmonary function test result data is FVL (Flow-Volume Loop) or DLCO (Diffusing capacity of It may be the original time series data of the lung for CO) curve, data visualized as a diagram on a two-dimensional plane, image data taken from this, or a reading of the lung function test results, or OCR processing data of the image taken.

In one example, the one or more integrated information combines evaluation values for the same item between different basic information, and the evaluation values may be calculated by at least one of a single probability value, a distribution value, and a distribution equation.

In one example, the analysis result providing module includes the one or more related information and integrated information, thereby providing improved risk values, probabilities, and distribution formulas thereof compared to those that do not include the one or more related information and integrated information. At least one of these can be provided.

In one example, the analysis result providing module outputs at least one of basic information, related information, and integrated information about the morphological lesion observed on the chest radiology image, and considers the anatomical location where the morphological lesion is located, basic information, At least one of related information and integrated information may be displayed, or a separate reading may be provided.

In one example, the first basic information relates to findings and diseases that can be suspected or diagnosed from chest radiology modality, such as consolidation, infiltration, ground glass opacity, and adult respiratory failure ( ARDS), Pneumonia, Abscess, Aspiration Pneumonia, Atypical Pneumonia, Active Tuberculosis, Non-Tuberculous Mycobacteria, Chronic Obstructive Pulmonary Disease (COPD), Interstitial Lung Disease, Bronchiectasis, Sarcoidosis, Lung Nodule, Lung Mass, Lung Cancer, Lung Metastasis Metastasis, Aortic Dissection, Aortic Aneurysm, Pleural Effusion, Empyema, Pneumothorax, Pneumoperitoneum, Pneumopericardium, Pneumomediastinum , Subcutaneous Emphysema, Coronary Artery Calcification, Cardiomegaly, Pulmonary Edema, Pericardial Effusion, Pulmonary Embolism, Chamber (LA, LV) , RA, RV) Enlargements, Valvular: Aortic, Mitral, Tricuspid, Pulmonic Valve Calcification/Stenosis/Regur- gitation) and hypertrophic cardiomyopathy.

In one example, the second basic information is numerical information reflecting the risk of heart rhythm, various emergency diseases, and various cardiac dysfunction from the electrocardiogram modality, such as atrial fibrillation, LV dysfunction, and RV dysfunction. ), Pulmonary Hypertension, Pericardial Effusion, Acute Coronary Syndrome, Myocardial Infarction, Pulmonary Edema, Chamber: LA, LV, RA, RV, Enlargements, Valvular: Aortic, Mitral, Tricuspid, Pulmonic Valve Stenosis/Regurgitation, Hypertrophic Cardiomyopathy, It may include at least one of Left Ventricular Ejection Fraction (LVEF), Global Longitudinal Strain (GLS), Respiratory Failure, Shock, and Cardiac Arrest.

In other exemplary embodiments of the present application, a first neural network that generates first basic information about one or more task items based on main input data, which is a numerical vector extracted from a first single modality; a second neural network that generates second basic information about one or more task items based on main input data, which is a numerical vector extracted from a second single modality; It may further include a functional analysis device, wherein the functional analysis module generates one or more related information and integrated information associated with the first basic information and the second basic information.

In one example, the first neural network includes a first encoder unit located at a data input end to output a numerical vector including feature information of a first single modality; And a first task unit that generates first basic information for one or more task items by using the numerical vector output from the first encoder unit as an input, wherein the second neural network is located on the data input side and generates a second single a second encoder unit that outputs a numeric vector containing feature information of the modality; and a second task unit that generates second basic information for one or more task items by using the numerical vector output from the second encoder unit as an input.

In one example, at least one of the first task unit and the second task unit includes additional information including at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease. 2. It further includes an additional neural network input in addition to the numerical vector output from the encoder unit, wherein the additional neural network generates additional task information related to first basic information or additional task information related to second basic information, and the first basic information Additional task information related to the subject's age, gender, anatomical characteristics, pathological characteristics not included in the first basic information, shooting environment, shooting method, image pre-processing method, and related to the second basic information. Additional task information may include information about the subject's age, gender, height, weight, pulse rate, electrocardiogram section duration, and electric axis including P wave, QRS wave, and T wave.

In one example, at least one of the first task unit, the second task unit, and the additional neural network includes one or more common layers; And it may include one or more task-specific layers to which the output vector of the common layer is input.

In one example, the functional analysis module may generate the one or more related information from the first basic information and the second basic information by one or more selected from regression analysis, machine learning, or set criteria.

In one example, the machine learning is learned by further including an additional data set, and the additional data set may be first basic information and second basic information acquired by the same person within a predetermined time frame.

In one example, all or part of the training label is assigned to a first single modality by using the related information and integrated information generated by the functional analysis module as a training label, and the assigned first single modality is used as training data. The first neural network can be trained.

In other exemplary embodiments of the present application, a first neural network that generates first basic information about one or more task items based on main input data, which is a numerical vector extracted from a first single modality; and a third neural network that generates association information for one or more task items based on the first basic information and the numerical vector extracted from the second single modality. The second single modality includes a first single modality that is performed concurrently - a first single modality that is performed along with the first single modality that is performed within a set time frame on blood testers for whom the second single modality test was performed. It is possible to provide a functional analysis device characterized in that - exists.

In one example, additional information including at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease may be further input to the third neural network.

In other exemplary embodiments of the present application, A fourth neural network that generates association information for task items related to the first single modality and task items related to the second single modality based on the numerical vector extracted from the first single modality and the numerical vector extracted from the second single modality. and that the first single modality and the second single modality are concurrent execution modalities - the first single modality and the second single modality are performed together within a certain time frame by the same subject. An analysis device may be provided.

In one example, the fourth neural network may further input at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease.

In one example, the apparatus includes a first auxiliary task neural network that generates first basic information about a task item related to a first single modality based on a numerical vector extracted from the first single modality; and/or a second auxiliary task neural network that generates second basic information about a task item related to the second single modality based on the numerical vector extracted from the second single modality; It may further include.

In other exemplary embodiments of the present application, 1. A functional analysis method performed by a processor, comprising: obtaining information from one or more modalities; A functional analysis step of generating at least one of association information between different modalities or between different basic information within a single modality; and providing a functional analysis result including at least one of one or more related information and integrated information associated with the basic information.

In one example, the one or more modalities include a chest radiology modality and an electrocardiogram modality, and the different modalities include a chest radiology modality; and one or more of the electrocardiogram modality and pulmonary function test modality; The basic information may be information about morphological characteristics provided from chest radiation modality, and the related information may be functional abnormality information related to the basic information based on the morphological characteristics.

In one example, the integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and a task of diagnosing or predicting a specific disease, The integrated information can be used as basic information and related information for new items.

In one example, the information about the functional abnormality may include one or more of the presence, probability, and severity of the abnormality related to the morphological characteristic.

In one example, obtaining information from one or more modalities includes first basic information, which is chest radiology data; Second basic information, which is electrocardiogram data; Obtain additional basic information from at least one of a blood test result, a pulmonary function test result, and a sample test result, and the related information includes functional abnormalities associated between the first basic information, the second basic information, and the additional basic information. It may include information about.

In one example, the one or more integrated information combines evaluation values for the same item between different basic information, and the evaluation values may be calculated by at least one of a single probability value, a distribution value, and a distribution equation.

In one example, the step of providing the functional analysis result includes improved risk values, probabilities, and the like by including the one or more related information and integrated information compared to not including the one or more related information and integrated information. At least one of the distribution equations may be provided.

In one example, the step of providing the functional analysis result outputs at least one of basic information, association information, and integrated information about the morphological lesion observed on the chest radiography image, and considers the anatomical location where the morphological lesion is located. At least one of basic information, related information, and integrated information may be displayed, or a separate reading may be provided.

In other exemplary embodiments of the present application, a computer-readable recording medium is readable by a computer and stores program instructions operable by the computer, wherein when the program instructions are executed by a processor of the computer, the computer-readable recording medium stores program instructions operable by the computer. A computer-readable recording medium that allows a processor to perform a functional analysis method is provided.

In one aspect, according to the embodiments of the present application, diagnosis is made by providing related information and integrated information between different modalities or within one modality, such as chest radiography images and electrocardiograms/pulmonary function tests. Accuracy can be increased. It can help with diagnosis segmentation, treatment selection, and cause determination. In particular, by confirming disease findings in multiple modalities, medical staff can provide emergency treatment with greater confidence in the diagnosis, thereby improving patient safety.

In one example, when pulmonary edema is diagnosed on a chest radiograph and a functional abnormality such as left heart failure is diagnosed on an electrocardiogram, it is possible to help determine the cause of pulmonary edema due to left heart failure. Additionally, if pulmonary edema is diagnosed on a chest radiograph and a myocardial infarction is diagnosed on an electrocardiogram, it can ultimately help determine the cause of pulmonary edema due to myocardial infarction.

In one example, if pulmonary edema is diagnosed on a chest radiograph and a functional abnormality such as heart failure is diagnosed on an electrocardiogram, diuretics may be prescribed to help select prompt treatment.

The effects of the present application are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly describe the exemplary embodiments of the present application, drawings necessary in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for illustrative purposes only and not for limiting purposes of the embodiments of the present specification. Additionally, for clarity of explanation, some elements may be shown in the drawings below with various modifications, such as exaggeration or omission.
1 is a schematic diagram showing the configuration of a functional analysis device according to one aspect of the present application.
FIG. 2A is a schematic diagram showing an artificial neural network structure that generates first basic information by taking the chest radiation modality of the functional analysis device as an input, in an embodiment of the present application, and FIG. 2B is a schematic diagram showing the structure of an artificial neural network that generates first basic information by taking the electrocardiogram modality as an input. This is a schematic diagram showing the structure of an artificial neural network that generates information.
FIG. 3A is a schematic diagram showing the structure of the first neural network and the second neural network in FIGS. 2A and 2B, and FIG. 3B is a schematic diagram showing the structure of the additional neural network in FIGS. 2A and 2B, in an embodiment of the present application.
4 shows, in an embodiment of the present application, a first neural network that generates first basic information using chest radiation modality as input and a second neural network that generates second basic information using electrocardiogram modality as input are independently trained. This is a schematic diagram showing the ordering process.
Figure 5 is a schematic diagram showing a process of generating integrated information by receiving the first basic information and the second basic information generated in Figure 4, in an embodiment of the present application.
6 shows that, in one embodiment of the present application, the second neural network is first trained, the parameter weights in the neural network are fixed, and then the first neural network and the third neural network are trained using the first single modality input and the second neural network output. This is a schematic diagram showing the process of generating integrated information for new integrated input.
Figure 7 shows, in an embodiment of the present application, a new integration by training the first neural network, the second neural network, and the fourth neural network using the first single modality and second single modality data sets performed by a plurality of identical subjects. This is a schematic diagram showing the process of generating integrated information about input.
Figure 8a is an example in which, in an embodiment of the present application, the analysis result providing module outputs the functional analysis result including related information as a separate readout, and Figure 8b is an example in which the analysis result providing module outputs the functional analysis result including related information as a separate readout in an embodiment of the present application. This is an example of how to output functional analysis results including related information on the morphological lesion of the input chest radiography image.
Figure 9 is an example in which, in an embodiment of the present application, the analysis result providing module provides chest radiology image analysis findings and related information through electrocardiogram analysis.
Figure 10 is a flow diagram of a functional analysis method, in one embodiment of the present application.

Hereinafter, some embodiments of the present application will be described in detail with reference to the exemplary drawings. In adding reference numerals to components in each drawing, identical components may have the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present technical idea, the detailed description may be omitted.

Term Definition

When “comprises,” “has,” “consists of,” etc. mentioned in the specification are used, other parts may be added unless “only” is used. When a component is expressed in the singular, it can also include the plural, unless specifically stated otherwise.

Additionally, in describing the components of the present application, terms such as first, second, A, B, (a), and (b) may be used. Unless otherwise specified, these terms are only used to distinguish the component from other components, and the nature, sequence, order, or number of the components are not limited by the term.

In this specification, 'learning' or 'learning' is a term that refers to performing machine learning through procedural computing.

In this specification, terms such as “unit,” “module,” “device,” or “system” are intended to refer to a combination of not only hardware but also software driven by the hardware. For example, the hardware may be a data processing device that includes a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or other processor. Additionally, software may refer to a running process, object, executable, thread of execution, program, etc.

In this specification, modality refers to, for example, time series data of waveform signals including electrocardiograms or images depicting them on a two-dimensional plane, chest radiography images, or images scanned after translating an expert's opinions on chest radiology images into writing, and blood tests. It can refer to text, numbers and images related to the results, time series or single measurement data related to flow rate, flow rate, pressure and volume measured during pulmonary function tests and charts that visualize them, numerical data or text related to test results.

In this specification, basic information is information about morphological characteristics provided from chest radiation modality, and related information may be information about functional abnormalities related to the basic information based on the morphological characteristics. In another example, the first basic information relates to all findings and diseases that can be suspected or diagnosed using chest radiography, and the second basic information is a value reflecting the risk of heart rhythm, various emergency diseases, and various cardiac dysfunction using electrocardiogram. It could be information.

In one example, the integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and the task of diagnosing or predicting a specific disease, Integrated information can be used as basic information and related information for new items.

As used herein, neural network refers to a machine learning algorithm or network. The first neural network generates a numeric vector or first basic information related to morphological features provided from the chest radiology modality, and the second neural network generates a numeric vector or second basic information provided from the electrocardiogram modality, and in some cases, each The detailed structure of a neural network may vary. The specific neural network structures located on the data input side of the first neural network and the second neural network can be separately distinguished and designated as a first encoder unit and a second encoder unit, respectively. The specific neural network structures located at the output ends of the first neural network and the second neural network can be separately distinguished and designated as a first task unit and a second task unit, respectively. Each encoder unit corresponds to a part of each neural network, but can be trained and managed separately from the task unit, and each encoder unit outputs a numerical vector providing feature information of the input data.

In one example, in addition to the main task of generating the first basic information, additional task information related to the first basic information may be generated. The additional task is not included in the first basic information, but includes all biological and It corresponds to non-biological information, and additional task information related to the first basic information includes the subject's age, gender, anatomical characteristics, pathological characteristics (not included in the first basic information), shooting environment, shooting method, image preprocessing method, etc. may include.

In another example, the second or additional neural network may generate additional task information in addition to the main task. In addition to the main task of generating the second basic information, additional task information related to the second basic information can be created. The additional task is not included in the second basic information but corresponds to all information that can be measured with an electrocardiogram, and the second basic information Additional task information related to basic information includes the subject's age, gender, height, weight, pulse rate, ECG section duration (PR-section, QRS complex, QT-section), electric axis (P wave, QRS wave, T wave), etc. may include.

Description of Exemplary Implementations

1 is a schematic diagram showing the configuration of a functional analysis device according to one aspect of the present application.

Referring to FIG. 1, the functional analysis device 1 includes an acquisition module 10 that acquires information from one or more modalities, and a functional analysis device that generates association information and integrated information between different modalities or between different basic information within a single modality. It may include an analysis module 20 and an analysis result provision module 30 that provides a functional analysis result including one or more related information and integrated information related to the basic information.

The one or more modalities include a chest radiation modality and an electrocardiogram modality, and the different modalities include a chest radiation modality; and one or more of the electrocardiogram modality and pulmonary function test modality. The acquisition module 10 may acquire information from at least one of chest radiation modality, electrocardiogram modality, and pulmonary function test modality.

The basic information is a risk value or probability used for the task of diagnosing or predicting a specific disease or informing the presence or absence of a specific radiological finding using morphological features provided from the chest radiology modality, or a distribution function that defines their distribution. It corresponds to a parameter and can be a numeric vector with one or more elements, especially when performing this task multiple times. The tasks may correspond to at least one of binary classification, multinomial classification, and regression.

The related information is functional abnormality information related to abnormal findings discovered by the functional analysis module 20 using basic information based on the morphological characteristics, depending on the presence or absence of medical, statistical, causal, or practical correlation. This corresponds to another selected basic information. The information regarding the functional abnormality may include one or more of the presence/absence, probability, or severity of the abnormality associated with basic information based on morphological characteristics.

The integrated information is new basic information created by integrated analysis of basic information in different modalities or within the same modality by the functional analysis module 20, and is used to improve existing basic information and related information or to create new items. It can be used as basic information and related information. The integrated information may be calculated as at least one of the parameters of a distribution function that defines a risk value or probability or the distribution of the risk values or probabilities used for the task of informing the presence or absence of a specific finding or diagnosing or predicting a specific disease.

The analysis result providing module 30 includes the one or more related information and integrated information, so that the user can evaluate and understand specific abnormal findings in various ways, compared to the case that does not include the one or more related information and integrated information. It is possible to deliver more accurate information by providing users with improved risk figures or probabilities and their distribution estimates.

In one embodiment, the analysis result providing module 30 outputs one or more of basic information about morphological lesions observed in radiological images, integrated information that can improve them, and related information related to them, These output positions can be displayed in consideration of the anatomical location of the morphological lesion (e.g., on or near the lesion) to improve the user's understanding of the lesion, or can be displayed separately as a reading text outside the radiological image to improve user readability. there is.

FIG. 2A is a schematic diagram showing the structure of an artificial neural network model that generates first basic information by taking the chest radiation modality of a functional analysis device as an input, and FIG. 2B is a schematic diagram showing the structure of an artificial neural network model that generates first basic information by taking the electrocardiogram modality as an input, in an embodiment of the present application. This is a schematic diagram showing the structure of an artificial neural network model that generates basic information.

Referring to FIG. 2A, the artificial neural network model for generating the first basic information may input chest radiology image data, and the chest radiology image data is the original chest radiology digital image and DICOM format image, and these are converted into different formats. It may be image data obtained by converting converted images, outputting them as images, shooting them with a camera, or capturing a screen.

The input chest radiology image data is output as a numerical vector through the first encoder unit of the first neural network including MLP, CNN, transformer, etc., and the numerical vector may be input to the first task unit of the first neural network or an additional neural network. there is. The first neural network inputs only information related to the chest radiology image data, and the additional neural network inputs not only information related to the chest radiology image data, but also the subject's age, gender, chief symptom (CC), blood test result data, and pulmonary function test result data. , the normalized or embedded value of at least one of vital signs (VS) and underlying disease is input. The blood test result data may be user input data, data obtained from an electronic medical record system, or OCR (artificial intelligence document recognition) data of a test result sheet. The pulmonary function test result data is the original time series data of the FVL (Flow-Volume Loop) or DLCO (Diffusing capacity of the lung for CO) curve, data visualized in a diagram on a two-dimensional plane, and image data taken from this, or lung This may be a readout of the functional test result or OCR processing data of the image taken.

If additional information is not used as input, only the first neural network is used for training and inference learning. If additional information is used as input, both the first neural network and the additional neural network are used for training, and the additional neural network is used for inference. The type, size, and structure of the artificial neural network, first neural network, and additional neural network may vary depending on the characteristics and amount of training data. In one example, the artificial neural network may have a CNN or transformer structure, and the first neural network and additional neural networks may have an MLP structure.

The encoder part of the first neural network is shared with an additional neural network to extract features from the first modality input, and the task part of the first neural network is a neural network trained to use only chest radiology image features, unlike the additional neural network, and determines whether additional information is used. Accordingly, one of the first neural network and the additional neural network can be appropriately selected and used. The first neural network or additional neural network is trained to perform various tasks to increase rich feature extraction ability and generalization ability.

The first neural network or the additional neural network may generate first basic information, which is the main task. The first basic information refers to all findings and diseases that can be suspected or diagnosed using chest radiography, including consolidation, infiltration, ground glass opacity, adult respiratory failure syndrome (ARDS), and pneumonia ( Pneumonia, Abscess, Aspiration Pneumonia, Atypical Pneumonia, Active Tuberculosis, Non-Tuberculous Mycobacteria, Chronic Obstructive Pulmonary Disease (COPD), Epilepsy Interstitial Lung Disease, Bronchiectasis, Sarcoidosis, Lung Nodule, Lung Mass, Lung Cancer, Lung Metastasis, Aortic Dissection (Aortic Dissection), Aortic Aneurysm, Pleural Effusion, Empyema, Pneumothorax, Pneumoperitoneum, Pneumopericardium, Pneumomediastinum, Subcutaneous Emphysema Emphysema, Coronary Artery Calcification, Cardiomegaly, Pulmonary Edema, Pericardial Effusion, Pulmonary Embolism, Chamber (LA, LV, RA, RV) Enlargements, Valvular: Aortic, Mitral, Tricuspid, Pulmonic Valve Calcification/Stenosis/Regur-gitation, Hypertrophic Cardiomyopathy It may include at least one of (Hypertrophic Cardiomyopathy). The first neural network or additional neural network may generate additional task information related to the first basic information in addition to the main task. The additional task is not included in the first basic information, but corresponds to all biological and non-biological information that can be confirmed through chest radiography. In one example, the additional task information related to the first basic information includes the subject's age, gender, and anatomy. It may include characteristics, pathological characteristics (not included in the first basic information), shooting environment, shooting method, image pre-processing method, etc.

The first basic information is based on morphological characteristics provided from the chest radiology modality, and defines the risk value or probability or their distribution used for the task of informing the presence or absence of specific radiological findings or diagnosing or predicting a specific disease. It corresponds to a parameter of a distribution function, and may be a numerical vector with one or more elements, especially when performing this task multiple times. The tasks may correspond to at least one of binary classification, multinomial classification, and regression.

When using binary classification, the sigmoid function can be used to express the probability of disease occurring for each task as a probability between 0 and 1. When using multiple classifications, the softmax function can be used to represent the probability distribution of the number of two or more cases. For the regression value, a method of using the output value as is without probability conversion as described above can be adopted. Examples of such regression include Coronary Calcium score, Cardiothoracic (CT) ratio, etc.

Referring to FIG. 2B, the artificial neural network model for generating second basic information may input electrocardiogram data, and the electrocardiogram data may be, for example, at least one of waveform time series data and image data. may include. The ECG image data may be ECG image data, screen capture image data, or image data obtained from an electronic medical record system. The ECG data may include at least one of an ECG signal that is a one-dimensional single-channel or multi-channel signal and an ECG image in which the ECG signal is depicted on a two-dimensional plane. In one example, the signal is in the form of a two-dimensional array, for example, C C that includes an included image or a patch image for each lead channel cropped for each lead channel, and has a black and white image of one or more lead channels, or three channels of R (Red), G (Green), and B (Blue) It may be in the form of a three-dimensional array of × W × H (number of channels × number of horizontal pixels × number of vertical pixels).

Meanwhile, not limited to the electrocardiogram data, all biodata that is other patient-derived information can be used, and various time-series data such as electroencephalogram (EEG), electromyogram (EMG), electrooculography (EOG), etc. Bio signals may be included. In addition, it can have waves, images, sounds, and any other form of modality, and is not limited to a specific modality.

The input ECG data is output as a numerical vector through an artificial neural network (second encoder unit) including MLP, CNN, RNN, transformer, etc., and the numerical vector is input to the second neural network (task unit) or an additional neural network. You can. The second neural network inputs only information related to the electrocardiogram data, and the additional neural network inputs not only information related to the electrocardiogram data, but also the subject's age, gender, main symptoms, blood test result data, pulmonary function test result data, vital signs, and underlying diseases. The normalized or embedded value of at least one piece of additional information is input. The blood test result data may be user-entered data, data obtained from an electronic medical record system, or OCR-processed data of a test result sheet. The pulmonary function test result data may be the original time series data of the FVL or DLCO curve, data visualized as a diagram on a two-dimensional plane, image data taken from this, or a readout of the pulmonary function test result or OCR processing data of the image taken. there is.

If additional information is not used as input, only the second neural network is used for training and inference learning. If additional information is used as input, both the second neural network and the additional neural network are used for training, and the additional neural network is used for inference. The type, size, and structure of the artificial neural network, secondary neural network, and additional neural network may vary depending on the characteristics and amount of training data. In one example, the artificial neural network may have a CNN, RNN, Transformer, etc. structure, and the second neural network and the additional neural network may have an MLP structure.

The encoder part of the second neural network is shared with an additional neural network to extract features from the second modality input, and the task part of the second neural network is a neural network trained to use only chest radiology image features, unlike the additional neural network, and determines whether additional information is used. Accordingly, one of the two (second neural network and additional neural network) can be appropriately selected and used. Secondary neural networks, or additive neural networks, are trained to perform a variety of tasks to increase the ability to extract rich features and generalize.

The second neural network or additional neural network may generate second basic information, which is the main task. The second basic information is numerical information that reflects the risk of heart rhythm, various emergency diseases, and various cardiac dysfunction, including atrial fibrillation, LV dysfunction, RV dysfunction, pulmonary hypertension, Pericardial Effusion, Acute Coronary Syndrome, Myocardial Infarction, Pulmonary Edema, Chamber: LA, LV, RA, RV, Enlargements, Valve ( Valvular: Aortic, Mitral, Tricuspid, Pulmonic Valve Stenosis/Regurgitation, Hypertrophic Cardiomyopathy, Left Ventricular Ejection Fraction (LVEF) ), Global Longitudinal Strain (GLS), Respiratory Failure, Shock, and Cardiac Arrest. The second neural network or additional neural network may generate additional task information in addition to the main task. The additional task is not included in the second basic information, but corresponds to all information that can be measured with an electrocardiogram. In one example, the additional task information related to the second basic information includes the subject's age, gender, height, weight, pulse rate, and electrocardiogram. It can include interval duration (PR-interval, QRS complex, QT-interval), electric axis (P wave, QRS complex, T wave), etc.

The first basic information is based on morphological characteristics provided from the chest radiology modality, and defines the risk value or probability or their distribution used for the task of informing the presence or absence of specific radiological findings or diagnosing or predicting a specific disease. It corresponds to the parameter of the distribution function, and especially when performing this task multiple times, it may be a numeric vector with one or more elements of the same number. The tasks may correspond to at least one of binary classification, multinomial classification, and regression.

The second basic information is based on waveform features that can be extracted from the electrocardiogram modality, and is a distribution that defines the risk value or probability or the distribution thereof used for the task of informing of specific electrocardiogram abnormalities or diagnosing or predicting a specific disease. It corresponds to a parameter of a function and can be a numeric vector with one or more elements, especially when performing this task multiple times. The tasks may correspond to at least one of binary classification, multinomial classification, and regression.

When using binary classification, the sigmoid function can be used to express the probability of disease occurring for each task as a probability between 0 and 1. When using multiple classifications, the softmax function can be used to represent the probability distribution of the number of two or more cases. For the regression value, a method of using the output value as is without probability conversion as described above can be adopted. Examples of such regression include Left Ventricular Ejection Fraction.

FIG. 3A is a schematic diagram showing the structure of the first neural network and the second neural network in FIGS. 2A and 2B, and FIG. 3B is a schematic diagram showing the structure of the additional neural network in FIGS. 2A and 2B, in an embodiment of the present application.

Referring to FIG. 3A, the task units of the first neural network and the second neural network receive input from each encoder unit and include one or more common layers; and one or more task-specific layers to which the output vector of the common layer is input.

Referring to Figure 3b, the additional neural network not only provides information related to chest radiography data and electrocardiogram data, but also adds information related to at least one of the subject's age, gender, main symptom, blood test result data, pulmonary function test result data, vital signs, and underlying disease. Information is also input. The additional information may be a normalized or embedded value. Additional neural networks, like the first neural network and the second neural network, also have one or more common layers; and one or more task-specific layers to which the output vector of the common layer is input. In one example, each layer is a fully-connected layer, and before and after each layer, depending on the use and purpose, a dropout layer, a non-linear activation function, and a normalization layer ( A normalization layer may be included.

Figure 4 is a schematic diagram showing a first neural network that generates first basic information by taking chest radiation modality as input and a second neural network that generates second basic information by taking electrocardiogram modality as input, in an embodiment of the present application. .

Referring to FIG. 4, the first neural network generates first basic information for one or more task items from chest radiology single modality input. The second neural network generates second basic information for one or more task items from input starting from a single modality of the electrocardiogram. The first neural network and the second neural network can be trained independently using a plurality of training samples, and the functional analysis module 30 provides the first basic information generated by the first neural network and the second neural network generated by the second neural network. One or more integrated information can be created from basic information.

In one embodiment, when integrating evaluation values for the same item between different modalities [e.g., in the case of Figure 4, Pulmonary Edema and Pericardial Effusion (Pulmonary Edema) as the same item between the first basic information and the second basic information ( [There is Pericardial Effusion], the evaluation value corresponds to at least one of the parameters of the distribution function that defines the risk value or probability or the distribution of these, which is used for the task of informing the presence or absence of a specific finding or diagnosing or predicting a specific disease. can do. And the tasks may correspond to at least one of binary classification, multinomial classification, and regression.

If two or more evaluation values to be integrated all correspond to risk values or probability estimates, they can be integrated by using an arithmetic average, or by obtaining a weighted average that applies appropriate weights. Here, the weight can be determined based on the performance of each evaluation value (e.g., Accuracy, AUC ROC, F1-score, PR-AUC, or similar accuracy measures).

[Equation 1]

Final single probability value = w_A * p_A + w_B * p_B (w_A + w_B = 1, w_A≥0, w_B≥0)

The w_A is a weight for the basic information on chest radiation, and p_A is an evaluation value corresponding to the basic information on chest radiation. The w_B is a weight for the basic ECG information, and p_B is an evaluation value corresponding to the basic ECG information.

If two or more evaluation values to be integrated all correspond to the parameters of a predetermined distribution function equation, what is integrated corresponds to the probability distribution defined by these distribution functions and their parameters, and in order to integrate two or more distribution functions There is a parametric method that integrates distribution functions into a formula, and a sampling method that estimates the integrated distribution from all samples obtained by applying random sampling to each probability distribution.

A method of integrating two or more distribution functions using the parametric method may be to obtain a weighted average function that applies appropriate weights. Here, the weight is determined based on the performance of each evaluation value (e.g., Accuracy, AUC, F1-score, PR-AUC, or similar accuracy measures). In this process, an additional normalization process may be required to integrate the entire section of the new function created and divide the function.

[Equation 2]

Final distribution equation = (w_A * D_A + w_B * D_B)/integral value (w_A + w_B = 1, w_A≥0, w_B≥0)

Above, w_A is the weight for chest radiation basic information, and D_A is the probability distribution equation of the evaluation value corresponding to chest radiation basic information. The w_B is the weight for the basic ECG information, and D_B is the probability distribution equation of the evaluation value corresponding to the ECG basic information. The integral value corresponds to the integral value for the entire section of w_A * D_A + w_B * D_B.

The sampling method can be applied even when the integrated evaluation value is not a parameter of the distribution function equation, that is, even when the evaluation value is a risk value or probability (estimate). This applies dropout, dropconnect, or similar methods in the inference calculation process to randomly remove parts of each unit and connection between elements of the artificial neural network to perform multiple evaluations. How to obtain a number, how to obtain several evaluation values through sampling through a Bayesian layer in the inference calculation process, and how to use input data such as test time augmentation in the inference calculation process. It is implemented through methods of obtaining multiple evaluation values by applying a random data augmentation algorithm, and methods of obtaining multiple evaluation values from multiple different models trained for the same purpose.

Figure 5 is a schematic diagram showing a method of generating integrated information based on the first basic information and second basic information generated in Figure 4 using an artificial intelligence algorithm or expert standards, in an embodiment of the present application.

In one embodiment, the functional analysis module 20 may generate the one or more integrated information using an artificial intelligence algorithm. Artificial intelligence algorithms may include logistic regression, random forest, gradient boost, ElasticNet, support vector machine, ANN, decision tree, etc. there is. Standardization, normalization, and variable selection corresponding to general preprocessing of input data can be applied to the evaluation values corresponding to the basic information used in this process. In addition, each evaluation value corresponding to basic information or a specific subset can undergo a transformation process by passing a non-linear function. In one example, a step function can be used. If the probability value for a specific task is below a set standard, the probability can be ignored by converting it to 0. In one example, if the probability of disease occurrence in the task for the left heart failure item is 30%, the value may be ignored and treated as 0.

The artificial intelligence algorithms are learned by further including an additional data set, and the additional data set may be first basic information and second basic information acquired by the same person within a specified time frame. The additional data set is a separate data set for training the artificial intelligence algorithm.

In one embodiment, the functional analysis module 20 may generate integrated information using expert standards. In the first method, expert standards allow the selection and output of the more appropriate modality result when a specific finding comes from both modalities. In one example, in the case of pulmonary edema, information about morphological lesions in chest radiography images is more important, so results can be selected and output based on information generated from chest radiology modality. The second method is to define a new diagnosis by combining different types of basic information. This means that when a combination of different basic information between different modalities or within the same modality appears that meets the criteria predetermined by an expert, the presence or absence of a specific finding or diagnosis or a level of risk is assigned to that combination. In one example, if the right ventricular failure risk score is high but the pulmonary hypertension risk score is low, it may be concluded that acute pulmonary embolism is included in the differential diagnosis, or the risk score for acute pulmonary embolism may be set to any score in the high range. When applying this combination, it is possible to create more accurate integrated information by determining the usage modality.

By including the one or more integrated information, the analysis result providing module 30 can provide one or more of a single risk value, probability, and distribution equation that are improved compared to those that do not include the one or more integrated information. In one example, if Pericardial Effusion is 50% on the chest radiograph and 70% on the ECG, the functional analysis module takes the average of these and replaces it with the final probability value of Pericardial Effusion of 60% and outputs it. Alternatively, an improved single probability value can be output together.

The analysis result providing module 30 can help the user interpret the results by including one or more integrated information. For example, if left ventricular dysfunction (LV dysfunction) > 50% on the electrocardiogram and pulmonary edema (Pulmonary Edema) > 50% on the chest radiograph, the functional analysis module 20 combines the two information based on the above expert standards to determine psychogenic Pulmonary edema can be included in the differential diagnosis or the risk score for cardiogenic pulmonary edema can be set high, and the analysis result provision module 30 can output “pulmonary edema due to left heart failure is suspected.” In this way, when the functional analysis module 20 calculates the average of evaluation values, applies artificial intelligence algorithms, and uses expert standards, not only does it improve the reading results of existing artificial intelligence that generates information from a single modality, but it also improves the reading results of existing artificial intelligence that generates information from a single modality. It can provide new diagnostic information that was previously unknown.

Figure 6 shows that, in an embodiment of the present application, after first training the second neural network and fixing the weights, the first neural network and the first neural network are trained using the first modality data and the evaluation value output corresponding to the basic information of the second neural network This is a schematic diagram showing the process of training a third neural network and generating integrated information from two modality inputs through the entire neural network.

Referring to FIG. 6, a second neural network that generates second basic information for one or more task items from the electrocardiogram modality and one or more numerical vectors extracted through a first encoder from the second basic information and the chest radiology modality. It may include a third neural network that generates integrated information about task items.

The first neural network is trained using the chest radiology modality along with the accompanying trial electrocardiogram modality. And even at the time of inference, the first neural network makes inferences using the accompanying ECG modality. However, since only the electrocardiogram modality is used for training of the second neural network, it is not necessary to have chest radiography performed along with it to construct training data for training of the second neural network.

The first neural network and the second neural network can be trained independently using only the input data corresponding to their respective modalities. Therefore, in the process of constructing each training data, it is not necessary to have the modality data of chest radiography and electrocardiogram performed together, but the paired modality data of each person performed together is required when training the third neural network. If the first neural network and the third neural network are trained together, the first neural network is trained using chest radiography data with accompanying ECG modality data. At the time of inference, in all cases, inference is made using the accompanying ECG modality.

In this process, the first encoder and the second encoder, which are the encoder parts of the first neural network and the second neural network, can be trained and managed separately.

The third neural network can expand its structure to receive additional information including at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease. This additional information is information measured close to the time when the chest radiography examination was performed, and the degree of proximity is determined by the purpose of using artificial intelligence, applied patient characteristics, and characteristics of each additional information. For example, if it is an acute disease or the patient's condition is poor, the time range must be close because the patient's condition changes quickly. Also, unlike other information, vital signs change quickly, so the time range must be close. The additional information may be normalized and/or embedded values.

Additionally, the third neural network directly generates integrated information through this process or generates basic information to be used in the previously described methodology for generating various integrated information.

7 is based on numerical vectors extracted from the first encoder and the second encoder trained using the first single modality and second single modality data sets performed by multiple identical subjects, in an embodiment of the present application. This is a schematic diagram showing the process of generating integrated information through the fourth neural network.

Referring to FIG. 7, the chest radiation modality and the ECG modality are a concurrent execution modality, and the accompanying execution modality is one in which the chest radiation modality and the ECG modality are performed together within a certain time frame by the same subject.

The fourth neural network may further input at least one additional information of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease. The additional information may be normalized and/or embedded values.

The fourth neural network can generate integrated information corresponding to the first basic information and the second basic information based on the numerical vectors output from the first neural network and the second neural network. Additionally, integrated information corresponding to new conditions and diseases that do not correspond to the first basic information and the second basic information can be generated.

The functional analysis device includes a first auxiliary task neural network that generates first basic information about a task item related to the chest radiation modality based on a numerical vector extracted from the chest radiation modality; And/or a second auxiliary task neural network that generates second basic information about the task item related to the ECG modality based on the numerical vector extracted from the ECG modality. No additional information is input to the first auxiliary task neural network and the second auxiliary task neural network. The reason for training the auxiliary task neural networks together is to train the first and second neural networks more effectively, and they are not used after training is completed.

FIG. 8A is an example in which the analysis result providing module 30 outputs a functional analysis result including related information under the input chest radiography image in an embodiment of the present application, and FIG. 8B is an example in an embodiment of the present application. , This is an example of how the analysis result provision module outputs functional analysis results including related information on top of the morphological lesion observed in the input chest radiography image.

Referring to FIG. 8A, the analysis result providing module 30 bundles the functional analysis results including information on the presence and probability of morphological lesions as well as related information (and integrated information) and outputs them as a separate readout. This is an example.

Referring to FIG. 8B, the analysis result providing module 30 of the present application outputs information about morphological lesions on the chest radiography image and provides one or more related information (and integrated information) to the corresponding location where the morphological lesion is located. Displays functional analysis results, including: Furthermore, by including one or more related information (and integrated information), the analysis result providing module 30 can provide more information about improvements compared to those that do not include the one or more related information. .

Figure 9 is an example in which, in an embodiment of the present application, the analysis result providing module provides chest radiology image analysis findings and related information through electrocardiogram analysis.

Referring to FIG. 9, the chest radiology AI findings as basic information are medical information on pulmonary edema, cardiomegaly, coronary artery disease/calcification, and valve disease/calcification, and the related information as accompanying display items is the presence, probability, and prediction of heart failure. Ejection fraction, heart valve abnormalities, myocardial infarction including ST segment elevation myocardial infarction (STEMI), probability of acute coronary syndrome and cardiomyopathy, blood Troponin, ProBNP, pH, Lactate, TCO2, Serum Creatinine, This may be medical information about C-reactive protein results and echocardiogram summary.

Basic information is medical information about lung infectious diseases including pneumonia, pulmonary tuberculosis, and lung abscess, and pneumonia including cirrhosis, infiltration, ground glass, and atelectasis, and related information is shock, respiratory failure, risk of death, presence and probability of heart failure, Predicted ejection fraction results, blood Troponin T/I, ProBNP, pH, Lactate, TCO2, Serum Creatinine, WBC, ESR, Eosinophil, and C-reactive protein results. It may be medical information about

Basic information is medical information about pulmonary embolism, and related information may be medical information about right ventricular failure, pulmonary hypertension, blood D-dimer, Troponin I, ProBNP results, echocardiography, and cardiac arrhythmia ultrasound. .

Basic information is medical information about pericardial effusion, and related information may be medical information about pericardial effusion, cardiac tamponade, right ventricular failure risk, shock, respiratory failure and death risk, blood CRP, and IGRA test.

The basic information is medical information about ARDS, and the related information may be medical information about vital signs including shock, respiratory failure, risk of death, right ventricular failure, pulmonary hypertension, SpO2, Troponin I, ProBNP, SpO2, and FIO2.

Basic information is medical information about emphysema, bronchitis, and interstitial lung disease, and related information is left ventricular failure, right ventricular failure, pulmonary hypertension, blood CRP, Troponin I, ProBNP, and FVC. , FEV1, FEV1/FVC values and interpretation, summary of pulmonary function test results, and raw data and artifacts of pulmonary function tests.

The functional analysis method according to another aspect of the present application is performed by a computing device including a processor. The computing device including the processor is, for example, performed by the functional analysis device 1 or some components (e.g., the acquisition module 10, the functional analysis module 20, the analysis result providing module 30), or , or may be performed by another computing device. Hereinafter, for clarity of explanation, the present application will be described in more detail with embodiments in which the functional analysis method is performed by the functional analysis device 1.

Figure 10 is a flowchart of a functional analysis method according to an embodiment of the present application.

Referring to Figure 10, the functional analysis method includes: acquiring information from one or more modalities (e.g., by acquisition module 10) (S10); A functional analysis step (S20) of generating at least one of association information between different modalities or between different basic information within a single modality (for example, by the functional analysis module 20); and providing (for example, by the analysis result providing module 30) a functional analysis result including at least one of one or more related information and integrated information associated with the basic information (S30); may include.

The one or more modalities include a chest radiation modality and an electrocardiogram modality, and the different modalities are one or more of a chest radiation modality, an electrocardiogram modality, and a pulmonary function test modality, and the basic information is based on morphological features provided from the chest radiation modality. This information is related to the information, and the related information may be functional abnormality information related to the basic information based on the morphological characteristics.

The integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and the task of diagnosing or predicting a specific disease, and the integrated information It can be used as basic information and related information for new items.

The information regarding the functional abnormality may include one or more of the presence/absence, probability, and severity of the abnormality related to the morphological characteristic.

The step (S10) of acquiring information from one or more modalities includes first basic information, which is chest radiation data; Second basic information, which is electrocardiogram data; Obtain additional basic information from at least one of a blood test result, a pulmonary function test result, and a sample test result, and the related information includes functional abnormalities associated between the first basic information, the second basic information, and the additional basic information. It may include information about.

The one or more integrated information combines evaluation values for the same item between different basic information, and the evaluation values can be calculated by at least one of a single probability value, a distribution value, and a distribution equation.

The step of providing the functional analysis result (S30) includes the one or more related information and integrated information, thereby providing improved risk values, probabilities, and their values compared to those that do not include the one or more related information and integrated information. At least one of the distribution equations can be provided.

The step of providing the functional analysis results (S30) outputs at least one of basic information, association information, and integrated information about the morphological lesion observed on the chest radiography image, and takes into account the anatomical location where the morphological lesion is located. At least one of information, related information, and integrated information may be displayed, or a separate reading may be provided.

According to such functional analysis devices and methods, the accuracy of diagnosis can be increased by providing correlation information between different modalities or modalities, such as chest radiography images and electrocardiograms/pulmonary function tests. It can help with diagnosis refinement, treatment selection, and cause determination, and can improve patient safety through extensive evaluation and quick emergency treatment by identifying dangerous findings found in multiple modalities.

Operations by the devices according to the embodiments described above may be implemented, at least in part, as a method or computer program that can be implemented by a computer, and may be recorded on a computer-readable recording medium. For example, implemented with a program product comprised of a computer-readable medium containing program code, which can be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be a computing device, such as a desktop computer, laptop computer, notebook, smart phone, or the like, or any device that may be integrated. A computer is a device that has one or more alternative, special-purpose processors, memory, storage, and networking components (either wireless or wired). The computer may run an operating system such as, for example, Microsoft's Windows-compatible operating system, Apple's OS X or iOS, a Linux distribution, or Google's Android OS.

The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present application reviewed above has been described with reference to the embodiments shown in the drawings, but these are merely examples and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered to be within the technical protection scope of this application. Therefore, the true technical protection scope of this application should be determined by the technical spirit of the attached patent claims.

Claims (39)

an acquisition module that acquires information from one or more modalities;
A functional analysis module that generates at least one of association information between different modalities or between different basic information within a single modality; and
A functional analysis device comprising: an analysis result providing module that provides a functional analysis result including at least one of one or more related information and integrated information associated with the basic information.
According to claim 1,
The one or more modalities include a chest radiology modality and an electrocardiogram modality,
The different modalities include chest radiation modality; and one or more of the electrocardiogram modality and pulmonary function test modality; and
The basic information is information about morphological characteristics provided from the chest radiation modality,
A functional analysis device, characterized in that the relevant information is functional abnormality information related to basic information based on the morphological characteristics.
According to claim 1,
The integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and the task of diagnosing or predicting a specific disease,
A functional analysis device characterized in that the integrated information is used as basic information and related information for new items.
According to claim 2,
A functional analysis device, wherein the information regarding the functional abnormality includes one or more of the presence/absence, probability, and severity of the abnormality associated with the morphological characteristic.
According to claim 2,
Basic information is medical information about pulmonary edema, cardiomegaly, coronary artery disease/calcification, and valve disease/calcification;
Related information includes presence and probability of heart failure, predicted ejection fraction, heart valve abnormalities, myocardial infarction including ST-segment elevation myocardial infarction (STEMI), probability of acute coronary syndrome and cardiomyopathy, and blood troponin. ), ProBNP, pH, Lactate, TCO2, Serum Creatinine, C-reactive protein (C-reactive protein) results, functional analysis device characterized in that it is medical information about echocardiography test summary.
According to claim 2,
Basic information is medical information about lung infectious diseases, including pneumonia, pulmonary tuberculosis, and lung abscess, and pneumonia, including cirrhosis, infiltrates, ground glass, and atelectasis;
Related information includes shock, respiratory failure, risk of death, presence and probability of heart failure, predicted ejection fraction results, blood Troponin T/I, ProBNP, pH, Lactate, TCO2, Serum Creatinine, WBC, ESR, Eosinophil, A functional analysis device characterized by medical information about C-reactive protein (C-reactive protein) results.
According to claim 2,
Basic information is medical information about pulmonary embolism,
A functional analysis device characterized in that the relevant information is medical information about right ventricular failure, pulmonary hypertension, blood D-dimer, Troponin I, ProBNP results, echocardiography, and cardiac arrhythmia ultrasound.
According to claim 2,
Basic information is medical information about pericardial effusion,
A functional analysis device characterized in that the relevant information is medical information about pericardial effusion, cardiac tamponade, right ventricular failure risk, shock, respiratory failure and death risk, blood CRP, and IGRA test.
According to claim 2,
Basic information is medical information about ARDS,
A functional analysis device characterized in that the relevant information is medical information on vital signs including shock, respiratory failure, risk of death, right ventricular failure, pulmonary hypertension, SpO2, Troponin I, ProBNP, SpO2, and FIO2.
According to claim 2,
Basic information is medical information about emphysema, bronchitis, and interstitial lung disease.
Related information includes left ventricular failure, right ventricular failure, pulmonary hypertension, blood CRP, Troponin I, ProBNP, FVC, FEV1, FEV1/FVC values and interpretation, summary of pulmonary function test results, and raw data of pulmonary function tests. A functional analysis device comprising medical information about at least one of the artifacts.
According to claim 2,
The acquisition module includes first basic information that is chest radiation data; Second basic information, which is electrocardiogram data; Obtain additional basic information from at least one of blood test results, pulmonary function test results, and specimen test results,
The related information includes information on related functional abnormalities between the first basic information, the second basic information, and the additional basic information.
According to claim 11,
The chest radiation data is chest radiation image data,
The electrocardiogram data is electrocardiogram image data or time series data,
The blood test result data is one or more selected from text, numerical values, and images, and the pulmonary function test result data is one or more selected from time series data, image data, text data, and numerical data,
A functional analysis device, characterized in that the sample test result data is one or more of text or numerical data.
According to claim 12,
The chest radiology image data is the original chest radiology digital image, a converted image of the original, captured image data or screen capture image data, and an image scanned after translating an expert's opinion on the chest radiology digital image into writing,
The time series data is time series data of a wave-shaped signal or an image depicting it on a two-dimensional plane, and the electrocardiogram image data is photographed image data, screen capture image data, or image data obtained from an electronic medical record system,
The blood test result data is user input data, data obtained from an electronic medical record system, or OCR (artificial intelligence document recognition) data of the test result sheet,
The pulmonary function test result data is the original time series data of the FVL (Flow-Volume Loop) or DLCO (Diffusing capacity of the lung for CO) curve, or the data visualized in a diagram on a two-dimensional plane, and the image data taken or pulmonary function. A functional analysis device characterized in that the inspection result readout and OCR processing data of the image taken.
According to claim 2,
The one or more integrated information combines evaluation values for the same item among different basic information,
A functional analysis device, characterized in that the evaluation value is calculated by at least one of a single probability value, a distribution value, and a distribution equation.
According to claim 14,
The analysis result providing module includes the one or more related information and integrated information, and at least one of improved risk values, probabilities, and distribution formulas thereof compared to those that do not include the one or more related information and integrated information. A functional analysis device characterized in that it provides.
According to claim 2,
The analysis result providing module outputs at least one of basic information, related information, and integrated information about the morphological lesion observed on the chest radiology image, and takes into account the anatomical location where the morphological lesion is located. A functional analysis device that displays at least one piece of information or provides a separate readout.
According to claim 11,
The first basic information relates to findings and diseases that can be suspected or diagnosed from chest radiology modality,
Consolidation, Infiltration, Ground Glass Opacification, ARDS, Pneumonia, Abscess, Aspiration Pneumonia, Atypical Pneumonia , Active Tuberculosis, Non-Tuberculous Mycobacteria, Chronic Obstructive Pulmonary Disease (COPD), Interstitial Lung Disease, Bronchiectasis, Sarcoidosis, Lung Nodule, Lung Mass, Lung Cancer, Lung Metastasis, Aortic Dissection, Aortic Aneurysm, Pleural Effusion, Empyema , Pneumothorax, Pneumoperitoneum, Pneumopericardium, Pneumomediastinum, Subcutaneous Emphysema, Coronary Artery Calcification, Cardiomegaly, Pulmonary Edema ), Pericardial Effusion, Pulmonary Embolism, Chamber (LA, LV, RA, RV) Enlargements, Valvular: Aortic, Mitral, Tricuspid Valve ( A functional analysis device characterized by including at least one of Tricuspid, Pulmonic Valve Calcification/Stenosis/Regurgitation, and Hypertrophic Cardiomyopathy.
According to claim 11,
The second basic information is numerical information that reflects the risk of heart rhythm, various emergency diseases, and various cardiac dysfunction from the electrocardiogram modality,
Atrial Fibrillation, LV dysfunction, RV dysfunction, Pulmonary Hypertension, Pericardial Effusion, Acute Coronary Syndrome, Myocardial Infarction ), Pulmonary Edema, Chamber: LA, LV, RA, RV, Enlargements, Valvular: Aortic, Mitral, Tricuspid, Pulmonic Valve Stenosis/Regurgitation, Hypertrophic Cardiomyopathy, Left Ventricular Ejection Fraction (LVEF), Global Longitudinal Strain (GLS), Respiratory Failure, Shock ( A functional analysis device, characterized in that it includes at least one of Shock) and Cardiac Arrest.
a first neural network that generates first basic information about one or more task items based on main input data, which is a numerical vector extracted from a first single modality; Including,
a second neural network that generates second basic information about one or more task items based on main input data, which is a numerical vector extracted from a second single modality; and
A functional analysis device further comprising a functional analysis module that generates one or more related information and integrated information associated with the first basic information and the second basic information.
According to claim 19,
The first neural network includes a first encoder unit located at a data input side and outputting a numeric vector containing characteristic information of a first single modality; And a first task unit that generates first basic information for one or more task items by using the numerical vector output from the first encoder unit as input,
The second neural network includes a second encoder unit located at the data input side and outputting a numeric vector containing feature information of a second single modality; and a second task unit that generates second basic information for one or more task items by using the numerical vector output from the second encoder unit as input.
According to claim 20,
The first or second encoder unit outputs additional information, including at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease, to at least one of the first and second task units. It further includes an additional neural network that is input in addition to one numerical vector,
The additional neural network generates additional task information related to first basic information or additional task information related to second basic information,
Additional task information related to the first basic information includes information about the subject's age, gender, anatomical characteristics, pathological characteristics not included in the first basic information, imaging environment, imaging method, and image pre-processing method,
The additional task information related to the second basic information includes information about the subject's age, gender, height, weight, pulse rate, electrocardiogram section duration, and electric axis including P wave, QRS wave, and T wave. A functional analysis device that performs
According to claim 21,
At least one of the first task unit, the second task unit, and the additional neural network includes one or more common layers; and one or more task-specific layers to which the output vector of the common layer is input.
According to claim 19,
The functional analysis module is a functional analysis device characterized in that it generates the one or more related information from the first basic information and the second basic information by one or more selected from regression analysis, machine learning, or set criteria.
According to claim 23,
The machine learning is learned by further including additional data sets,
A functional analysis device, characterized in that the additional data set is first basic information and second basic information acquired by the same person within a specified time frame.
According to claim 19,
Using the related information and integrated information generated by the functional analysis module as training labels, all or part of the training labels are assigned to the first single modality, and the assigned first single modality is used as training data to create a first neural network. A functional analysis device characterized by training.
a first neural network that generates first basic information about one or more task items based on main input data, which is a numerical vector extracted from a first single modality; and
a third neural network that generates association information for one or more task items based on the first basic information and a numerical vector extracted from the second single modality; It includes,
In the second single modality, there is a accompanying first single modality - the first single modality performed within a set time frame on the blood tester for whom the second single modality test was performed is referred to as the accompanying first single modality. A functional analysis device characterized in that.
According to claim 26,
A functional analysis device, wherein additional information including at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease is further input to the third neural network.
A fourth neural network that generates association information for task items related to the first single modality and task items related to the second single modality based on the numerical vector extracted from the first single modality and the numerical vector extracted from the second single modality. It includes,
The first single modality and the second single modality are concurrent execution modalities - the first single modality and the second single modality are performed together within a certain time frame by the same subject.
According to clause 28,
The fourth neural network is a functional analysis device characterized in that it further inputs at least one of the subject's age, gender, main symptom, blood test results, vital signs, and underlying disease.
29. The apparatus of claim 28, wherein the apparatus includes: a first auxiliary task neural network that generates first basic information about a task item related to a first single modality based on a numerical vector extracted from the first single modality; and/or
a second auxiliary task neural network that generates second basic information about a task item related to a second single modality based on a numerical vector extracted from the second single modality; A functional analysis device further comprising:
A functional analysis method performed by a processor, comprising:
Obtaining information from one or more modalities;
A functional analysis step of generating at least one of association information between different modalities or between different basic information within a single modality; and
A functional analysis method comprising: providing a functional analysis result including at least one of one or more related information and integrated information associated with the basic information.
According to claim 31,
The one or more modalities include a chest radiology modality and an electrocardiogram modality,
The different modalities include chest radiation modality; and one or more of the electrocardiogram modality and pulmonary function test modality; and
The basic information is information about morphological characteristics provided from the chest radiation modality,
A functional analysis method, wherein the relevant information is functional abnormality information related to basic information based on the morphological characteristics.
According to claim 31,
The integrated information is information improved from the basic information and the related information, and relates to the presence or absence of specific findings that could not be provided from the basic information and the related information, and the task of diagnosing or predicting a specific disease,
A functional analysis method characterized in that the integrated information is used as basic information and related information for new items.
According to claim 32,
A functional analysis method, wherein the information regarding the functional abnormality includes one or more of the presence/absence, probability, and severity of the abnormality associated with the morphological characteristic.
According to claim 32,
Obtaining information from one or more modalities includes first basic information, which is chest radiography data; Second basic information, which is electrocardiogram data; Obtain additional basic information from at least one of blood test results, pulmonary function test results, and specimen test results,
The functional analysis method characterized in that the association information includes information on associated functional abnormalities between the first basic information, the second basic information, and the additional basic information.
According to claim 32,
The one or more integrated information combines evaluation values for the same item among different basic information,
A functional analysis method, characterized in that the evaluation value is calculated by at least one of a single probability value, a distribution value, and a distribution equation.
According to claim 36,
The step of providing the functional analysis result includes improved risk values, probabilities, and distribution formulas thereof compared to those that do not include the one or more related information and integrated information by including the one or more related information and integrated information. A functional analysis method characterized by providing at least one or more.
According to claim 32,
The step of providing the functional analysis results includes outputting at least one of basic information, related information, and integrated information about the morphological lesion observed on the chest radiography image, and considering the anatomical location where the morphological lesion is located. A functional analysis method characterized by displaying at least one of information and integrated information or providing a separate readout.
A computer-readable recording medium that is readable by a computer and stores program instructions operable by the computer, wherein when the program instructions are executed by a processor of the computer, the processor performs any of claims 31 to 38. A computer-readable recording medium that allows performing a functional analysis method according to one clause.

Images