KR20240043488A - A multimodal deep learning model for predicting future visual field in glaucoma patients - Google Patents

Abstract

An embodiment of the present invention relates to a method for predicting the future visual field of a glaucoma patient using an artificial intelligence model, comprising the steps of extracting features included in OCT data to obtain OCT feature data, visual field test data, the OCT feature data, and time information. Including entering a prediction model, using the output result of the prediction model to obtain a final visual field prediction result based on recent visual field test data and last time information corresponding to the recent visual field test data. Disclosed is a method and device for predicting the future visual field of a glaucoma patient.

Description

Method for predicting future vision of glaucoma patients using a multimodal deep learning model {A MULTIMODAL DEEP LEARNING MODEL FOR PREDICTING FUTURE VISUAL FIELD IN GLAUCOMA PATIENTS}

The present invention relates to a method for predicting the future visual field of a glaucoma patient using a multimodal deep learning model.

Glaucoma is a progressive optic neuropathy that can lead to blindness. It is common to use visual field testing and optical coherence tomography (OCT) images to monitor glaucoma. Meanwhile, in the conventional glaucoma detection method using artificial intelligence, the visual field test (VF test) is generally performed based on the initial defect pattern, so previous visual field test data is generally required to predict future visual field.

Additionally, as another method, VAE (variational autoencoder) and RNN (Recurrent Neural Network) models were used to manage time series data. Meanwhile, the above methods required a long period of time and had difficulties in obtaining VF data, such as obtaining inconsistent results depending on the patient's condition.

The present invention was devised to solve the above problems, and aims to disclose a method that utilizes optical coherence tomography (OCT) data, which provides structural information about optic nerve head damage related to glaucoma, in addition to visual field testing. do.

The purpose of the present invention is to develop a multimodal deep learning model that predicts future visual field damage patterns for glaucoma disease, which can worsen into blindness.

According to an embodiment of the present invention, in a method for predicting the future visual field of a glaucoma patient using an artificial intelligence model, extracting features included in OCT data to obtain OCT feature data, visual field test data, the OCT feature data, and time Inputting information into a prediction model, using the output result of the prediction model to obtain a final visual field prediction result based on recent visual field test data and latest time information (Last time info) corresponding to the recent visual field test data. Disclosed is a method for predicting the future visual field of a glaucoma patient, including a method.

In addition, the step of inputting the visual field test data, the OCT feature data, and the time information into a prediction model includes generating a multidimensional vector by connecting the OCT feature data, the visual field test data, and the time information, and generating the multidimensional vector. It may include inputting to the LSTM network.

In addition, if the OCT feature data is missing, it further includes generating virtual OCT data using mask information, and inputting the visual field test data, the OCT feature data, and time information into the prediction model, It may include inputting visual field test data, the virtual OCT data, and time information into a prediction model.

Additionally, the OCT feature data may include a thickness map feature that provides information on the patient's entire field of view and vertical and horizontal tomographic map features that provide information on a specific area of the patient's field of view.

Additionally, the artificial intelligence model can be trained using visual field test data samples with re-weighted values based on the degree of noise.

Additionally, the loss function of the artificial intelligence model includes a weighted MSE, and the weighted MSE may include Equation 1.

[Equation 1]

( is the rebalanced weight, is the predicted visual field test data, is the correct answer value)

A future vision prediction device according to an embodiment of the present invention may include the following configuration. In the device for predicting the future visual field of a glaucoma patient using an artificial intelligence model, an input unit where OCT data and visual field test data are input, an output unit that outputs the visual field prediction results, and features included in the OCT data are extracted to obtain OCT feature data. , visual field test data, the OCT feature data, and time information are input into a prediction model, and the output results of the prediction model are used to enter the latest visual field test data and the latest time information (Last time info) corresponding to the recent visual field test data. A device for predicting the future visual field of a glaucoma patient, including a control unit that obtains the final visual field prediction result based on the visual field prediction result.

In addition, the operation of the control unit inputting the visual field test data, the OCT feature data, and the time information into a prediction model includes the steps of connecting the OCT feature data, the visual field test data, and the time information to generate a multidimensional vector, and It may include the operation of inputting a multidimensional vector into an LSTM network.

Additionally, if the OCT feature data is missing, the control unit may generate virtual OCT data using mask information and input the visual field test data, virtual OCT data, and time information into a prediction model.

Additionally, the OCT feature data may include a thickness map feature that provides information on the patient's entire field of view and vertical and horizontal tomographic map features that provide information on a specific area of the patient's field of view.

Additionally, the artificial intelligence model can be trained using visual field test data samples with re-weighted values based on the degree of noise.

Additionally, the loss function of the artificial intelligence model includes a weighted MSE, and the weighted MSE may include Equation 1.

[Equation 1]

( is the rebalanced weight, is the predicted visual field test data, is the correct answer value).

The present invention develops a multimodal deep learning model among AI (artificial intelligence) techniques to predict future visual field damage patterns for glaucoma disease, which can worsen into blindness, and quantifies the progression of the disease based on the current state of glaucoma patients. /Visualization enables individualized treatment.

In glaucoma patients, customized treatment can be performed by predicting future visual field damage progression patterns, identifying patients with potential blindness affecting the central area or both upper and lower visual fields in advance, and providing more active treatment.

Figure 1 shows an AI device 100 according to an embodiment of the present invention.
Figure 2 shows a flowchart of a field of view prediction method according to an embodiment of the present invention.
Figure 3 shows a field of view prediction model according to an embodiment of the present invention.
Figure 4 shows a method of learning a visual field prediction model according to an embodiment of the present invention.
Figure 5 shows a method of learning a visual field prediction model according to an embodiment of the present invention.

The technology described below may be subject to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be named a second component without departing from the scope of the technology described below, and similarly, the second component may also be named a first component. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

In terms used in this specification, singular expressions should be understood to include plural expressions, unless clearly interpreted differently from the context, and terms such as “including” refer to the described features, numbers, steps, operations, and components. , it means the existence of parts or a combination thereof, but should be understood as not excluding the possibility of the presence or addition of one or more other features, numbers, step operation components, parts, or combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions. In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by .

In addition, when performing a method or operation method, each process forming the method may occur in a different order from the specified order unless a specific order is clearly stated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

First, the definitions of terms used in the following description are as follows.

Tomography data may refer to image data acquired using optical coherence tomography (OCT) (hereinafter used interchangeably with OCT data).

Optical coherence tomography (OCT) can refer to tests that provide different types of data about the optic nerve head, such as thickness maps and tomography (transverse and longitudinal).

For example, OCT's thickness maps provide information about the retinal nerve fiber layer (RNFL) thickness around the optic disc, which is generally directly related to glaucomatous damage, while tomography's horizontal tomograms and It includes vertical tomograms and can provide information about the structural features of the optic disc.

Visual field test data may refer to data on the results of a patient's visual field test. According to an embodiment of the present invention, the visual field test per patient can be performed multiple times at time intervals and stored in correspondence with each timeline.

Hereinafter, the visual field prediction model of the present invention will be described.

The visual field prediction model disclosed by the present invention discloses a method of operating a deep learning model capable of predicting future visual field patterns based on the shape of the optic nerve using the above data (perimetry data, OCT data, etc.).

Specifically, the visual field prediction model corresponds to the development of a glaucoma prognosis prediction model using image information of glaucoma patients, and the future visual field defect pattern of glaucoma patients using existing visual field test data, thickness maps, and OCT image data such as tomography. It may correspond to a deep learning model that predicts.

Through this, it is possible to ultimately predict the likelihood of glaucoma progression, speed of progression, and visual field progression pattern in glaucoma patients.

The field of view prediction model according to an embodiment of the present invention may be composed of a combination of CNN and RNN to extract features from image data and serial data.

The vision prediction model can apply an efficient training mechanism with weighted loss to manage noisy data. The training mechanism will be explained later.

According to an embodiment of the present invention, a visual field prediction model with multiple inputs (OCT data and perimetry data) can help to better learn structure-function relationships, and the performance of the proposed model based on quantitative results can be improved.

Hereinafter, the field of view prediction device will be described as being equipped with the field of view prediction model. The visual field prediction device is a device that consistently processes input data and performs the calculations necessary to predict the visual field pattern of a glaucoma patient according to a specific model or algorithm. For example, the field of view prediction device can be implemented in the form of a PC, a server on a network, a smart device, or a chipset with an embedded design program.

Figure 1 is an example of a configuration diagram of a field of view prediction device 100 according to an embodiment of the present invention.

The field of view prediction device 100 may be implemented as a mobile device, such as a mobile phone, smartphone, personal digital assistants (PDA), portable multimedia player (PMP), navigation, tablet PC, wearable device, or vehicle.

Referring to FIG. 1, the field of view prediction device 100 may include a communication unit 110, an input unit 120, a control unit 130, an interface unit 140, an output unit 160, and a memory 150. .

The communication unit 110 can transmit and receive data with external devices such as other electronic devices or servers using wired or wireless communication technology. For example, the communication unit 110 may transmit and receive sensor information, user input, learning models, and control signals with external devices.

The input unit 120 can acquire various types of data.

At this time, the input unit 120 may include a camera for inputting video signals, a microphone for receiving audio signals, and a user input unit for receiving information from the user. Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

The output unit 160 may generate output related to vision, hearing, or tactile sensation.

At this time, the output unit 160 may include a display unit that outputs visual information, a speaker that outputs auditory information, and a haptic module that outputs tactile information.

The memory 150 can store data supporting various functions. For example, the memory 150 may store input data obtained from the input unit 120 and various data obtained from a server or connected device.

The control unit 130 may determine at least one executable operation of the field of view prediction device 100. Additionally, the control unit 130 may control the components of the viewing pattern prediction device 100 to perform the determined operation.

Figure 2 shows a flowchart of a field of view prediction method according to an embodiment of the present invention.

Figure 3 shows a field of view prediction model according to an embodiment of the present invention.

First, referring to FIG. 3, the visual field prediction model according to an embodiment of the present invention includes a preprocessing unit 310 that performs preprocessing to extract features of OCT data and a prediction unit that processes a series of images and visual field test data ( 320) may be included. It may also include a result generating unit 330 that derives the final prediction result.

This will be described in detail below.

First, the visual field prediction device 100 according to an embodiment of the present invention may perform image preprocessing before feeding (inputting) the OCT data to the visual field prediction model (S210).

Specifically, the visual field prediction model may use input data to perform a data preprocessing process to predict the patient's future visual field pattern corresponding to the data.

For example, a visual field prediction model can extract features through preprocessing of OCT data. OCT data may include thickness maps, horizontal tomograms, and vertical tomograms.

Specifically, in the above image preprocessing, all images can be normalized (pixel values from -1 to 1) and resized to 224 Х 224.

According to an embodiment of the present invention, a convolution neural network (CNN) model may be used to analyze image data when preprocessing a visual field prediction model.

Referring to FIG. 3, according to an embodiment of the present invention, each of the thickness map, horizontal tomography, and vertical tomography images among the OCT data can be input to a predetermined preprocessing model 310, and each feature (Features from image) data) can be extracted. The feature may be defined as OCT feature data.

For example, a vision pattern prediction device can use 'ResNet-50', which is pre-trained as a preprocessing model, as a feature extractor.

The feature extractor may provide a feature vector for each image as input. Specifically, the OCT feature data may include a thickness map feature that provides information about the patient's entire field of view, and vertical and horizontal tomography map features that provide information about a specific area of the patient's field of view.

According to an embodiment of the present invention, the visual field prediction model can generate a feature vector that places emphasis on thickness map features among OCT data during preprocessing.

For example, the output vectors of the thickness map, vertical tomography, and horizontal tomography may consist of a 512-d vector, a 128-d vector, and a 128-d vector, respectively.

The visual field prediction model according to an embodiment of the present invention can generate OCT feature data based on OCT data through the above preprocessing process.

Meanwhile, according to an embodiment of the present invention, several general data augmentation techniques (movement, resizing, rotation) may be applied to the dataset to avoid overfitting.

In the visual field prediction method according to an embodiment of the present invention, visual field test data, OCT feature data, and time information can be input into the prediction model (S220). The prediction model may be the prediction unit 320.

At this time, the visual field test data input to the prediction model may be the patient's previous visual field test data (Previous VFs test data), and the visual field test data can be reconstructed as a vector and normalized to have an average of 0 and a standard deviation of 1. .

The time information may include two types of information: timeline and time interval. The timeline information may refer to the time between the selected image and the first image. Additionally, the interval information may mean the time between the selected image and the previous image.

The visual field prediction model according to an embodiment of the present invention can generate a multidimensional vector by connecting OCT feature data, visual field test data, and time information for each time step.

The multidimensional vector can be used as an input to a Long Short-Term Memory (LSTM) network. The output of the LSTM may be a 128-d vector.

Meanwhile, according to an embodiment of the present invention, the vision prediction model ideally inputs pair-data (visual field test data and OCT data), but if any one of the pair-data is missing, mask information is used to Missing data can be replaced.

The mask information may mean a vector indicating whether the OCT data is missing or available. For example, a mask could define 1 as available and 0 as missing.

Specifically, when OCT data is missing, virtual OCT data can be generated to generate the missing OCT image. At this time, the virtual OCT data may be an image filled with black.

According to an embodiment of the present invention, when OCT data is missing and mask information indicates '0', the OCT data may be replaced with virtual OCT data and input into the visual field prediction model.

The visual field prediction model according to an embodiment of the present invention can obtain LSTM output results (S230).

Afterwards, the visual field prediction model according to an embodiment of the present invention produces a final visual field prediction result based on the LSTM output result, the latest visual field test data (Last VF), and the latest time information (Last time info) corresponding to the recent visual field test data. Can be obtained (S240).

Meanwhile, a structure-function relationship exists between the field of view and the thickness map, and the structure-function relationship may be different for each person. Additionally, although glaucoma occurs in one of several general patterns, each patient has unique characteristics, so customization was necessary. Unlike conventional artificial intelligence models, the present invention can solve the above problem by using OCT data and visual field test data together.

Specifically, to solve the above customization problem, previous visual field test data (only the last visual field test data) can be used as reference visual field test data to predict future visual field.

Prediction accuracy can be greatly improved based on the LSTM output and the reference visual field test data.

As described above, the visual field prediction model according to an embodiment of the present invention can predict the future visual field by combining LSTM output, previous visual field test data, and time information, and medical staff can diagnose the degree of progression of a glaucoma patient based on this.

Meanwhile, the future visual field may be generated in the form of visual field test data to be used as reference data for future visual field tests.

Hereinafter, a training method for a visual pattern prediction model according to an embodiment of the present invention will be described.

4 to 5 are diagrams showing a method of training a visual field prediction model according to an embodiment of the present invention.

Referring to FIG. 4, in a visual field test, noisy visual field test data may be generated depending on the patient's current condition, so there is difficulty in obtaining stable data.

Therefore, in order to reflect the above situation, the field of view prediction model according to an embodiment of the present invention re-assigned weights to samples to process noisy data, and the field of view prediction model has a field of view with re-weighted values based on the degree of noise. Can be trained by inspection data samples.

Meanwhile, compared to visual field test data, OCT data has consistent and reliable thickness maps, so the above process can be omitted.

The visual field prediction model according to an embodiment of the present invention may be composed of a regression model, and the regression model may predict visual field inspection data based on the thickness map. The input value of the regression model may be a thickness map, and the output may be visual field inspection data corresponding to the thickness map. The backbone of the regression model may be the pre-trained ResNet-50 [7].

The loss function is the mean square error (MSE loss), and data augmentation can be applied like the main model.

The visual field prediction model may learn the structure-function relationship using the regression model. After training, the model can predict common glaucoma patterns based on the thickness map.

Hereinafter, reassignment of weights of visual field inspection data samples will be described based on the algorithm progress procedure of FIG. 5.

According to an embodiment of the present invention, first, the mean absolute error (MAE) between ground truth and predicted visual field test data can be calculated for each patient. Since the calculated mean absolute error value () is different for each patient, it is normalized ( can do.

According to an embodiment of the present invention, to identify noisy samples, samples with errors where the normalized error 'is less than the threshold (TH) may be considered good samples, and the rest may be considered noisy.

At this time, the weight of the good sample may be set higher than the weight of the noisy sample.

Referring to FIG. 4, for example, the weight of a good sample may be 1, and the weight of a noisy sample may be set based on an exponential function. According to an embodiment of the present invention, after obtaining all weights, the field of view prediction model can be learned using training data reflecting weighted loss to reduce the influence of noise data, and the loss function used to train the field of view prediction model is Weighted MSE can be used as follows:

( is the rebalanced weight, is the predicted visual field test data, is the correct answer value)

The future visual field prediction model according to an embodiment of the present invention discloses a deep learning model that predicts future visual field inspection data based on multiple inputs using previous visual field inspection data and OCT data (thickness map, vertical and horizontal tomographic images). . A CNN-RNN model is launched to analyze both series data (previous visual field test data) and image data (OCT data). Additionally, a mechanism to detect samples and re-weight them was introduced to handle noisy data.

Using the above model, the re-weighting method can improve performance with or without OCT images.

Those skilled in the art will understand that various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be used in electronic hardware, (for convenience) It will be understood that it may be implemented by various forms of program or design code (referred to herein as software) or a combination of both.

The above-described present invention can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is.

Claims (12)

In a method of predicting the future visual field of a glaucoma patient using an artificial intelligence model,
Obtaining OCT feature data by extracting features included in the OCT data;
Inputting visual field test data, the OCT feature data, and time information into a prediction model;
Comprising the step of using the output result of the prediction model to obtain a final visual field prediction result based on recent visual field test data and last time information corresponding to the recent visual field test data,
A method for predicting future visual field in glaucoma patients. According to clause 1,
The step of inputting the visual field test data, the OCT feature data, and time information into the prediction model,
Comprising the step of generating a multidimensional vector by connecting the OCT feature data, the visual field test data, and the time information, and inputting the multidimensional vector into an LSTM network,
A method for predicting future visual field in glaucoma patients. According to clause 1,
If the OCT feature data is missing, it further includes generating virtual OCT data using mask information,
The step of inputting the visual field test data, the OCT feature data, and time information into the prediction model,
Including inputting the visual field test data, the virtual OCT data, and time information into a prediction model,
A method for predicting future visual field in glaucoma patients. According to clause 1,
The OCT feature data includes a thickness map feature that provides information about the patient's entire field of view and vertical and horizontal tomographic map features that provide information about a specific area of the patient's field of view.
A method for predicting future visual field in glaucoma patients. According to clause 1,
The artificial intelligence model is
Trained on perimetry data samples re-weighted based on the degree of noise,
A method for predicting future visual field in glaucoma patients. In paragraph 1
The loss function of the artificial intelligence model includes a weighted MSE, and the weighted MSE includes Equation 1,
[Equation 1]

( is the rebalanced weight, is the predicted visual field test data, is the correct answer value)
A method for predicting future visual field in glaucoma patients. In a device for predicting the future vision of glaucoma patients using an artificial intelligence model,
An input unit where OCT data and visual field test data are input;
An output unit that outputs the field of view prediction results and
Obtain OCT feature data by extracting features included in the OCT data, input the visual field test data, the OCT feature data, and time information into a prediction model, and use the output result of the prediction model to obtain the most recent visual field test data and the most recent visual field test data. Comprising a control unit that obtains the final visual field prediction result based on last time information corresponding to the visual field test data,
A device for predicting future vision in glaucoma patients. According to clause 7,
The operation of the control unit inputting the visual field test data, the OCT feature data, and time information into the prediction model is,
Comprising the step of generating a multidimensional vector by connecting the OCT feature data, the visual field test data, and the time information, and inputting the multidimensional vector into an LSTM network,
A device for predicting future vision in glaucoma patients. According to clause 7,
When the OCT feature data is missing, the control unit generates virtual OCT data using mask information, and inputs the visual field test data, the virtual OCT data, and time information into a prediction model,
A device for predicting future vision in glaucoma patients. According to clause 7,
The OCT feature data includes a thickness map feature that provides information about the patient's entire field of view and vertical and horizontal tomographic map features that provide information about a specific area of the patient's field of view.
A device for predicting future vision in glaucoma patients. According to clause 7,
The artificial intelligence model is
Trained on perimetry data samples re-weighted based on the degree of noise,
A device for predicting future vision in glaucoma patients. In paragraph 7
The loss function of the artificial intelligence model includes a weighted MSE, and the weighted MSE includes Equation 1,
[Equation 1]

( is the rebalanced weight, is the predicted visual field test data, is the correct answer value)
A device for predicting future vision in glaucoma patients.