JP7388525B2 - Ophthalmology image processing device and ophthalmology image processing program - Google Patents

Description

The present disclosure relates to an ophthalmologic image processing device that processes an ophthalmologic image of an eye to be examined, and an ophthalmologic image processing program that is executed in the ophthalmologic image processing device.

In recent years, techniques for acquiring various medical information using mathematical models trained by machine learning algorithms have been proposed. For example, in the ophthalmological device described in Patent Document 1, IOL-related information of the eye to be examined (e.g., predicted predicate anterior chamber depth) is acquired by inputting eye shape parameters to a mathematical model trained by a machine learning algorithm. Ru. IOL power is calculated based on the acquired IOL related information.

It is also conceivable to obtain high-resolution images from low-resolution images using mathematical models trained by machine learning algorithms. In this case, a low-resolution image is input to the mathematical model, and a high-resolution image is output.

JP2018-51223A

One aspect of the technology in this disclosure will be described. When acquiring a high-resolution image from a low-resolution image using a mathematical model trained by a machine learning algorithm, the mathematical model is must be trained in advance. In other words, a worker constructing a mathematical model must prepare a large number of sets of low-resolution and high-resolution images of the same body part and train the mathematical model. Preparing a large number of sets of images requires a lot of effort.

Explain other aspects. In order to properly train a mathematical model, the correspondence between low-resolution images and high-resolution images used for training needs to be appropriate. It is difficult to obtain good high-resolution images using a mathematical model if the mathematical model is trained using a set of images with inappropriate correspondences.

The present disclosure aims to solve at least one of the above multiple aspects. That is, a typical object of the present disclosure is to provide an ophthalmologic image processing device and an ophthalmologic image processing program for appropriately acquiring a high-resolution image from a low-resolution image.

An ophthalmologic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of an eye to be examined, and the control unit of the ophthalmologic image processing apparatus is configured to control the ophthalmologic image processing apparatus. By acquiring a base image, which is an ophthalmological image taken by an imaging device, and inputting the base image to a mathematical model trained by a machine learning algorithm, a target image whose image resolution is higher than that of the base image is obtained. The mathematical model is a low-resolution training image that is at least one ophthalmological image from a set of a plurality of ophthalmological images that target the same tissue site and that are taken by the same ophthalmological image capturing device. is used as input training data, and a high-resolution training image, which is an ophthalmological image having a higher image resolution than the low-resolution training image, is used as output training data. The image to be displayed can be set in advance according to the timing when the target image can be displayed, and when the timing when the target image can be displayed has arrived, the target image can be set in advance. And among the base images, an image that is preset according to the timing is displayed on the display device .

An ophthalmologic image processing program provided by a typical embodiment of the present disclosure is an ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of a subject's eye, When the program is executed by the control unit of the ophthalmological image processing device, a base image acquisition step of acquiring a base image, which is an ophthalmologic image taken by the ophthalmology image capturing device, and a mathematical model trained by a machine learning algorithm are performed. By inputting the base image, the ophthalmological image processing apparatus executes a target image acquisition step of acquiring a target image whose image resolution is higher than that of the base image, and the mathematical model A low-resolution training image that is at least one ophthalmological image from a set of a plurality of ophthalmological images taken by the same ophthalmological image capturing device is used as input training data, and the low-resolution training A high-resolution training image, which is an ophthalmological image having a higher image resolution than the image, is trained as output training data, and among the target image and the base image, the image to be displayed on the display device is the target image. It is possible to set in advance according to the timing when the target image can be displayed, and when the timing when the target image can be displayed has arrived, the target image and the base image can be set according to the timing The ophthalmologic image processing apparatus is caused to execute a step of displaying a preset image on the display device.

According to the ophthalmologic image processing device and the ophthalmologic image processing program according to the present disclosure, a high-resolution image is appropriately acquired from a low-resolution image.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image photographing devices 11A and 11B. 3 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1. FIG. FIG. 3 is an explanatory diagram for explaining a training data set 30 including a high-resolution training image 31 and a low-resolution training image 32. FIG. It is an explanatory diagram for explaining the contents of training of the mathematical model in this embodiment. It is a flowchart of ophthalmological image processing performed by the ophthalmological image processing device 21. 4 is a diagram showing an example of a composite image 40 of a target image 42 and an attention area image 44. FIG.

<Summary>
The control unit of the ophthalmologic image processing device exemplified in this disclosure acquires a base image that is an ophthalmologic image captured by the ophthalmologic image capturing device. The control unit obtains a target image having an image resolution higher than that of the base image by inputting the base image to a mathematical model trained by a machine learning algorithm. The mathematical model is trained by a training data set, which is a set of input training data and output training data. The training data set includes a plurality of ophthalmologic images that target the same tissue site and are captured by the same ophthalmologic image capture device. The input training data uses a low-resolution training image that is at least one ophthalmological image in the training data set. Moreover, among the training data sets, high-resolution training images, which are ophthalmological images having a higher image resolution than the low-resolution training images, are used as the output training data.

That is, a mathematical model construction device that constructs a mathematical model performs an image set acquisition step and a training step. In the image set acquisition step, a set of a plurality of ophthalmological images, each of which targets the same region of the tissue of the eye to be examined and is photographed by the same ophthalmological image photographing device, is acquired. In the training step, at least one low-resolution training image, which is an ophthalmological image, of the acquired set is used as input training data, and at least one high-resolution ophthalmic image, which is an ophthalmological image with higher image resolution than the low-resolution training image, is used as input training data. A mathematical model is trained using the resolution training images as output training data.

In this case, both the low-resolution training images and the high-resolution training images included in the training data set are captured by the same ophthalmological image capturing device. Therefore, the training data set is acquired more efficiently than when the low-resolution training images and the high-resolution training images are taken by different ophthalmological imaging devices. Furthermore, the correspondence between the low-resolution training images and the high-resolution training images included in the training data set is likely to be appropriate compared to the case where the low-resolution training images and the high-resolution training images are captured by different ophthalmological imaging devices. (For example, the color tone of the image, the amount of noise contained in the image, artifacts, optical resolution, imaging angle of view, tissue condition at the time of imaging, etc. are likely to be similar between low-resolution training images and high-resolution training images. ) Therefore, by using the trained mathematical model, a better target image can be obtained.

Note that "image resolution" in the present disclosure means the fineness of depicting the object to be photographed (that is, the fineness of the image). In other words, "image resolution" in the present disclosure is the number of pixels of an image corresponding to a unit area on the tissue to be imaged.

Further, the low-resolution training images and high-resolution training images included in the training data set are not limited to the ophthalmological images themselves (that is, original images) taken by the same ophthalmological image capturing device. For example, at least one of the low-resolution training image and the high-resolution training image may be an image generated based on an original image captured by an ophthalmologic image capturing device. In other words, if at least one of the low-resolution training images and high-resolution training images has been processed, the original images on which the low-resolution training images and the high-resolution training images are based were taken by the same ophthalmological imaging device. It would be fine if it had been done.

The base image, the low-resolution training image, and the high-resolution training image may be OCT angio images of the fundus of the subject's eye taken by an OCT device. The OCT angio image shows the fundus blood vessels of the eye to be examined. Therefore, by using the OCT angio image, a high-resolution target image showing fundus blood vessels can be appropriately acquired based on the base image.

Note that the OCT angio image may be a two-dimensional front image of the fundus viewed from the front (that is, the line of sight direction of the eye to be examined). The OCT angio image may be, for example, a motion contrast image obtained by processing at least two OCT signals obtained at different times regarding the same position.

However, it is also possible to use images other than the OCT angio image of the fundus as the ophthalmological image used as the training data set and the base image input to the mathematical model. For example, at least one of a tomographic image (two-dimensional tomographic image or three-dimensional tomographic image) taken by an OCT device, an image taken by a fundus camera, an image taken by a laser scanning ophthalmoscope (SLO), etc. It may be used as a training dataset and base image. Further, the training data set and the base image may be images of tissues other than the fundus of the eye to be examined (for example, the anterior segment of the eye to be examined).

The high-resolution training image used in at least one of the plurality of training datasets is a panoramic image generated by aligning a plurality of ophthalmological images taken in each of a plurality of imaging regions that do not completely overlap. You can. That is, the ophthalmologic image capturing apparatus may capture an ophthalmologic image (hereinafter referred to as a "unit image") for each of a plurality of capturing regions that do not completely overlap. A panoramic image may be generated by positioning a plurality of captured unit images. The image data acquisition step performed by the mathematical model construction device may include a panoramic image acquisition step of acquiring the generated panoramic image as a high-resolution training image.

In this case, by capturing a unit image with a high image resolution, a high-resolution image capturing a wide range of tissue is used as a high-resolution training image of the training data set. Therefore, high-resolution training images of the training dataset are better obtained.

The low-resolution training image used in at least one of the plurality of training data sets may be an image generated by lowering the image resolution of a panoramic image that is a high-resolution training image. That is, the image data acquisition step performed by the mathematical model construction device may include a low-resolution training image generation step of generating a low-resolution training image by lowering the image resolution of the panoramic image.

In this case, in at least one of the training datasets, the low-resolution training image and the high-resolution training image have the same imaging range, imaging position, and tissue state at the time of imaging. Therefore, by using a trained mathematical model, a better target image is obtained.

Note that when training a mathematical model, multiple training datasets are used for training. Here, when panoramic images are used as high-resolution training images in the training dataset, all of the multiple high-resolution training images do not need to be panoramic images, and at least some of the multiple high-resolution training images are panoramic images. Good to have. Note that when the proportion of panoramic images in the plurality of high-resolution training images is large (preferably 50% or more, more preferably 70% or more), high-resolution training images are more easily acquired. Similarly, when an image obtained by reducing the image resolution of a panoramic image (hereinafter referred to as a "low-resolution panoramic image") is used for training, it is not necessary that all of the plurality of low-resolution training images are low-resolution panoramic images. Note that when the proportion of low-resolution panoramic images in the plurality of low-resolution training images is large (preferably 50% or more, more preferably 70% or more), the quality of the target image is further improved.

However, it is also possible to change the specific method for generating the low-resolution training images and high-resolution training images of the training dataset. For example, ophthalmological images that are not panoramic images may be used as high resolution training images in the training dataset. Additionally, low-resolution training images of the training dataset may be generated by reducing the image resolution of ophthalmological images that are not panoramic images. Further, a plurality of ophthalmological images generated by photographing the same tissue multiple times at different image resolutions may be used as the low-resolution training image and the high-resolution training image of the training data set. Furthermore, when a panoramic image is used as a high-resolution training image, an image obtained by photographing a shooting range equivalent to the panoramic image at low resolution may be used as the low-resolution training image of the training dataset.

A low-resolution training image used for at least one of the plurality of training data sets may have undergone deterioration processing that degrades image quality on at least a portion of the image region. That is, the mathematical model construction device may perform a deterioration step of degrading the image quality on at least a portion of the image region of the low-resolution training image.

In this case, even if the image quality of at least part of the base image is poor, the mathematical model is trained to output a target image in which the influence of the poor image quality is reduced. Therefore, a more appropriate target image is obtained.

The specific details of the deterioration process can be selected as appropriate. For example, the control unit of the mathematical model construction device may perform at least one of a noise adding process that adds noise and a blurring process that adds blur.

The control unit of the ophthalmological image processing device may execute at least one of simultaneous display processing and switching display processing. In the simultaneous display process, the control unit displays the base image and the target image simultaneously on the display device. In the switching display process, the control unit switches between the base image and the target image and displays them on the display device according to an instruction input by the user. In this case, the user can easily confirm both the target image with high image resolution and the base image (original image) on which the target image is based. Therefore, the user can perform diagnosis etc. more appropriately. Note that the base image and the target image may be displayed on the same display device, or may be displayed on separate display devices.

The control unit may sequentially acquire base images that are continuously captured by the ophthalmologic image capturing device. The control unit may successively acquire the target images by sequentially inputting the acquired base images to the mathematical model. The control unit may generate moving image data of the tissue based on the plurality of acquired target images. In this case, the user can check the moving image of the tissue displayed at high image resolution. Therefore, the user can perform diagnosis etc. more appropriately.

In particular, when generating a high-resolution panoramic image, the panoramic image is generated from a plurality of unit images, so when displaying a moving image of tissue using the panoramic image, the frame rate tends to drop significantly. In particular, when the shooting position changes over time, it is difficult to generate a good moving image using a panoramic image. In contrast, the ophthalmological image processing device of the present disclosure is capable of generating a video that captures a wide range with high resolution while suppressing a decrease in frame rate.

Note that the control unit may cause the display device to display a high-resolution moving image of the tissue being imaged by the ophthalmologic image capturing device in real time in parallel with the image capturing. Further, the control unit may cause the display device to simultaneously display the moving image of the base image and the moving image of the target image. Further, the control unit may switch between the base image video and the target image video and display them on the display device according to an instruction input by the user. In this case, the user can easily check both the moving image of the target image and the moving image of the base image.

The control unit of the ophthalmological image processing apparatus may execute the attention area image acquisition step and the compositing step. In the attention area image acquisition step, an attention area image photographed by an ophthalmologic image capturing device is acquired. The attention area image is an image in which the attention area in the photography area of the base image is photographed at an image resolution higher than the image resolution of the base image. In the compositing step, the original image of the region of interest image is composited with the region of interest in the target image. In this case, in the combined image, the region of interest is formed by the original image with high image resolution, and the regions other than the region of interest are formed by the target image. Therefore, the user can confirm the region of interest in the original image, and can also confirm the entire image at high resolution.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The constructed mathematical model outputs a target image having a higher image resolution than the base image, based on the input base image. The ophthalmological image processing device 21 uses a mathematical model to obtain a target image from a base image. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images that are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using the ophthalmologic images acquired from the ophthalmologic image capturing device 11A. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic image capturing device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmological image capturing apparatus 11A) may cooperate to construct the mathematical model.

Further, a PC is used as the ophthalmological image processing apparatus 21 of this embodiment. However, the device that can function as the ophthalmological image processing device 21 is not limited to a PC either. For example, the ophthalmologic image capturing device 11B, a server, or the like may function as the ophthalmologic image processing device 21. When the ophthalmologic image capturing device 11B (OCT device in this embodiment) functions as the ophthalmologic image processing device 21, the ophthalmologic image capturing device 11B captures an ophthalmologic image while also capturing an ophthalmologic image with higher quality than the captured ophthalmologic image. can be obtained appropriately. A mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmological image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmological image capturing apparatus 11B) may cooperate to perform various processes.

Further, in this embodiment, a case will be described in which a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be used as the controller to speed up the processing.

The mathematical model construction device 1 will be explained. The mathematical model construction device 1 is placed, for example, at a manufacturer that provides the ophthalmologic image processing device 21 or the ophthalmologic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller that controls, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a mathematical model building process (see FIG. 2), which will be described later. Further, the communication I/F 5 connects the mathematical model construction device 1 to other devices (for example, the ophthalmologic image capturing device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8. The operation unit 7 is operated by the user in order for the user to input various instructions to the mathematical model construction device 1. As the operation unit 7, for example, at least one of a keyboard, a mouse, a touch panel, etc. can be used. Note that a microphone or the like for inputting various instructions may be used together with or in place of the operation section 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that "image" in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as "ophthalmologic image") from the ophthalmologic image capturing device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic image capturing device 11A through at least one of wired communication, wireless communication, a removable storage medium (for example, a USB memory), and the like.

The ophthalmologic image processing device 21 will be explained. The ophthalmological image processing device 21 is placed, for example, in a facility (for example, a hospital or a medical examination facility) that performs diagnosis or examination of subjects. The ophthalmologic image processing device 21 includes a control unit 22 that performs various control processes, and a communication I/F 25. The control unit 22 includes a CPU 23 that is a controller that controls control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing ophthalmologic image processing (see FIG. 5), which will be described later. The ophthalmologic image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 1. The communication I/F 25 connects the ophthalmologic image processing device 21 to other devices (for example, the ophthalmologic image capturing device 11B and the mathematical model construction device 1).

The ophthalmological image processing device 21 is connected to an operation section 27 and a display device 28. As with the operation section 7 and display device 8 described above, various devices can be used for the operation section 27 and the display device 28.

The ophthalmologic image processing device 21 can acquire ophthalmologic images from the ophthalmologic image capturing device 11B. The ophthalmological image processing device 21 may acquire ophthalmological images from the ophthalmological image capturing device 11B, for example, by wired communication, wireless communication, a removable storage medium (for example, a USB memory), or the like. Further, the ophthalmological image processing device 21 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

The ophthalmologic image capturing devices 11A and 11B will be explained. As an example, in this embodiment, a case will be described in which an ophthalmologic image capturing device 11A that provides ophthalmologic images to the mathematical model construction device 1 and an ophthalmologic image capturing device 11B that provides ophthalmologic images to the ophthalmologic image processing device 21 are used. . However, the number of ophthalmological imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmology image processing device 21 may acquire ophthalmology images from a plurality of ophthalmology image capturing devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic image capturing device. Note that the two ophthalmologic image capturing apparatuses 11A and 11B illustrated in this embodiment have the same configuration. Therefore, below, the two ophthalmological image capturing apparatuses 11A and 11B will be collectively described.

Further, in this embodiment, an OCT device capable of capturing an OCT angio image of the fundus is used as the ophthalmologic image capturing device 11 (11A, 11B). The OCT angio image shows the fundus blood vessels of the eye to be examined. Therefore, the ophthalmological image processing device 21 can acquire a high-resolution target image in which the fundus blood vessels are displayed. Note that the OCT angio image of this embodiment is a two-dimensional front image when the fundus of the eye is viewed from the front (that is, the line of sight direction of the subject's eye).

The ophthalmologic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic image capturing section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that controls control, and a storage device 14 (14A, 14B) that can store programs, data, and the like.

The ophthalmological image photographing unit 16 includes various configurations necessary for photographing an ophthalmological image of the eye to be examined. The ophthalmologic image capturing section 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into a measurement light and a reference light, a scanning section for scanning the measurement light, and a scanning section for scanning the measurement light. It includes an optical system for irradiating the eye to be examined, a light receiving element that receives the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic image capturing device 11 captures (generates) a motion contrast image, which is a type of OCT angio image. Specifically, the CPU 13 acquires at least two frames of OCT signals (interference signals) with different measurement times by scanning the same position in the tissue multiple times with OCT light. The CPU 13 acquires motion contrast image data by performing arithmetic processing on the acquired OCT signal. Methods for obtaining motion contrast data by processing OCT signals include, for example, a method of calculating a phase difference of a complex OCT signal, a method of calculating a vector difference of a complex OCT signal, a method of decorrelation, a ratio, a difference, a variance, etc. At least one of a method of calculating a change in amplitude or intensity, a method of multiplying a plurality of calculation results, etc. may be adopted.

Further, the ophthalmologic image capturing device 11 can capture (generate) a panoramic image of a tissue (in this embodiment, the fundus). A panoramic image is generated by aligning a plurality of images (hereinafter referred to as unit images) photographed in each of a plurality of photographing regions that do not completely overlap with each other. The photographing area of each unit image is narrower than the photographing area of the entire panoramic image. Therefore, the ophthalmologic image capturing device 11 can capture each unit image with high image resolution. Therefore, the panoramic image is a high-resolution image that captures a wide range of tissues. Note that an example of a method for acquiring a panoramic image of an OCT angio image is described in, for example, Japanese Patent Application Publication No. 2017-47113.

However, it is also possible to use an ophthalmologic imaging device other than the OCT device. For example, a fundus camera, a laser scanning ophthalmoscope (SLO), a corneal endothelial cell imaging device (CEM), or the like may be used as the ophthalmological imaging device. Further, the OCT device may provide a two-dimensional tomographic image or a three-dimensional tomographic image to the mathematical model construction device 1 or the ophthalmological image processing device 21.

(mathematical model construction process)
The mathematical model construction process executed by the mathematical model construction device 1 will be described with reference to FIGS. 2 to 4. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

In the mathematical model construction process, a mathematical model is trained using a training data set, thereby constructing a mathematical model for outputting a target image from a base image. Although details will be described later, the training data set includes input training data and output training data. In this embodiment, low-resolution training image data is used as input training data. Further, as the output training data, data of a high-resolution training image whose image resolution is higher than that of the low-resolution training image is used. The low-resolution training image and the high-resolution training image target the same tissue site. Further, the low-resolution training image and the high-resolution training image are images based on images captured by the same ophthalmologic image capturing device 11.

As shown in FIG. 2, the CPU 3 acquires a high-resolution panoramic image (in this embodiment, a panoramic image of an OCT angio image) as a high-resolution training image 30 (S1). In this embodiment, the CPU 3 acquires a panoramic image generated by the ophthalmological image capturing device 11A via communication or the like. However, the CPU 3 may acquire the panoramic image by acquiring data (for example, motion contrast data) that is the basis for generating the panoramic image, and generating the panoramic image based on the acquired data.

FIG. 3 shows an example of the high-resolution training image 31 used in this embodiment. As shown in FIG. 3, the high-resolution training image 31 of this embodiment is a panoramic image of an OCT angio image, and is generated by aligning four unit images 311, 312, 313, and 314. As an example, in this embodiment, each of the four unit images 311, 312, 313, and 314 is an image of a square area of 3 mm on a side on the tissue. The resolution of each of the four unit images 311, 312, 313, and 314 is 256 pt×256 pt. The photographing areas of the four unit images 311, 312, 313, and 314 are in contact with each other without gaps in a grid pattern. Therefore, in the high-resolution training image 31, which is a panoramic image, the imaging area on the tissue is a square area of 6 mm on a side, and the resolution is 512 pt x 512 pt.

Next, the CPU 3 generates a low-resolution training image 32 by lowering the image resolution of the high-resolution training image 31, which is a panoramic image (S2). As an example, the CPU 3 of the present embodiment generates the low-resolution training image 32 by thinning out a plurality of pixels constituting the high-resolution training image 31 regularly (in this embodiment, one by one). As a result, in the low-resolution training image 32, the imaging region on the tissue becomes a square region of 6 mm on a side, similar to the imaging region in the high-resolution training image 31. On the other hand, the resolution of the low resolution training image 32 is reduced to 256pt x 256pt. As a result, the image resolution of the low-resolution training image 32 (the number of pixels of the image per unit area on the tissue) is lower than the image resolution of the high-resolution training image 31.

As described above, the CPU 3 obtains the training data set 30 including the high-resolution training image 31 and the low-resolution training image 32 by reducing the image resolution of the high-resolution training image 31 and generating the low-resolution training image 32. do. In this case, the high-resolution training image 31 and the low-resolution training image 32 have the same imaging area, imaging position, and tissue state at the time of imaging. Therefore, training of a mathematical model, which will be described later, is performed more appropriately.

Additionally, multiple training data sets are used to train the mathematical model. In this embodiment, among a plurality of training data sets, training data includes a high-resolution training image 31 that is a panoramic image and a low-resolution training image 32 that is generated by lowering the image resolution of the high-resolution training image 31. The proportion of set 30 is 50% or more (preferably 70% or more). As a result, the quality of the target image output by the mathematical model is further improved.

Note that among the plurality of training data sets, a method for acquiring a training data set that does not include the above-described high-resolution training images 31 and low-resolution training images 32 can be selected as appropriate. For example, the CPU 3 may acquire an ophthalmological image that is not a panoramic image as a high-resolution training image. The CPU 3 may generate a low-resolution training image by lowering the image resolution of a high-resolution training image that is not a panoramic image. Furthermore, a plurality of ophthalmological images generated by photographing the same tissue multiple times at different image resolutions may be used as the training data set. Note that the shooting area of the high-resolution training image and the shooting area of the low-resolution training image do not need to completely match. For example, the imaging area of the high-resolution training image and the imaging area of the low-resolution training image may partially overlap.

Next, the CPU 3 randomly adds noise to the low-resolution training image 32 (S3). Specifically, the CPU 3 randomly selects a low-resolution training image 32 to which noise is added from among the plurality of low-resolution training images 32. Further, when performing the noise adding process, the CPU 3 randomly determines the amount of noise to be applied, the area to which the noise is applied, and the like.

Next, the CPU 3 randomly adds blur to the low-resolution training image 32 (S4). Specifically, the CPU 3 randomly selects a low-resolution training image 32 to be blurred from among the plurality of low-resolution training images 32. Further, when performing the blurring process, the CPU 3 randomly determines the degree of blurring, the area to be blurred, and the like. Note that the noise adding process (S3) and the blurring process (S4) are examples of deterioration processes that degrade at least a portion of the low-resolution training images 32. It is also possible to change the specific method of deterioration treatment.

Next, the CPU 3 executes training of a mathematical model using the training data set using a machine learning algorithm (S5). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

Neural networks are a method that imitates the behavior of biological nerve cell networks. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and stochastic neural networks. There are various types of neural networks (Boltzmann machines, Bassian networks, etc.).

Random forest is a method of generating a large number of decision trees by performing learning based on randomly sampled training data. When using random forest, branches of multiple decision trees trained in advance as a classifier are followed, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. For example, the SVM learns the parameters of a linear input element based on the criterion (hyperplane separation theorem) of finding a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by being trained using a training dataset. The training data set is a set of input training data and output training data. A mathematical model is trained such that when certain input training data is input, corresponding output training data is output. For example, training updates correlation data (eg, weights) between each input and output.

As an example, in this embodiment, a generative adversarial network (GAN) that uses two competing neural networks is employed as a machine learning algorithm. However, other machine learning algorithms may be used. For example, a multilayer neural network (eg, a convolutional neural network (CNN)) may be used as the machine learning algorithm.

With reference to FIG. 4, a method for training a mathematical model using a machine learning algorithm (GAN) employed in this embodiment will be described. As mentioned above, the GAN network includes two subnetworks (generator and discriminator) that compete with each other. In this embodiment, first, an image 32E is generated by enlarging the low-resolution training image 32, which is input training data. As a result, the enlarged low-resolution training image 32E and the high-resolution training image 31 match in size. Note that the low-resolution training image 32 may be used as training input data without being enlarged, and the enlargement process may be performed by the generator. The generator then outputs a fake image 32F that matches the high resolution training image 31 as closely as possible based on the low resolution training image 32E. The discriminator attempts to distinguish between the high-resolution training image 31, which is a real image, and the fake image 32F. The determination result is reflected in both the generator and the discriminator. By repeating this training, the generator attempts to generate a fake image 32F that is closer to the high-resolution training image 31, and the discriminator attempts to improve the accuracy of identification. The mathematical model (generator model) constructed as a result of training generates an ophthalmological image with a higher image resolution than the input image (e.g., A target image (described later) can be generated.

As described above, in this embodiment, the high-resolution images and low-resolution images included in the training data set are images captured by the same ophthalmological image capturing apparatus 11A (more specifically, images captured by the same ophthalmic image capturing apparatus 11A). 11A). Therefore, it is easy to approximate the high-resolution image and the low-resolution image in terms of color tone, amount of noise contained in the image, artifacts, optical resolution, photographing angle of view, state of tissue at the time of photographing, and the like. Thus, the mathematical model is properly trained.

Furthermore, in the present embodiment, the low-resolution training image used for at least one of the plurality of training datasets has undergone deterioration processing on at least a portion of the image region. Therefore, even if the image quality of at least part of the ophthalmological image (for example, a base image described below) input to the constructed mathematical model is not good, an image in which the influence of the poor image quality is reduced is output.

Returning to the explanation of FIG. 2. The processes of S1 to S5 are repeated until construction of the mathematical model is completed (S6: NO). When construction of the mathematical model is completed (S6: YES), the mathematical model construction process ends. A program and data for realizing the constructed mathematical model are incorporated into the ophthalmological image processing device 21.

(Ophthalmology image processing)
With reference to FIG. 5, the ophthalmologic image processing performed by the ophthalmologic image processing device 21 will be described. The ophthalmological image processing is executed by the CPU 23 according to the ophthalmological image processing program stored in the storage device 24.

First, the CPU 23 acquires a base image that is an ophthalmological image photographed by the ophthalmological image photographing device 11B (S11). Note that in this embodiment, the ophthalmologic image capturing device 11B captures the base images continuously (that is, as a moving image). The CPU 23 sequentially acquires the continuously photographed base images.

The base image in this embodiment is an OCT angio image of the fundus of the eye to be examined, similar to the high-resolution training image 31 and low-resolution training image 32 described above. Further, the base image in this embodiment is an image having a lower image resolution than the high-resolution training image 31 described above. As an example, the ophthalmologic imaging apparatus 11B of the present embodiment captures an imaging area of the same size as the high-resolution training image 31 (for example, a imaging area of 6 mm x 6 mm) with a resolution lower than that of the high-resolution training image 31 (for example, an imaging area of 6 mm x 6 mm). For example, data of the base image is generated by photographing at 256 pt x 256 pt). Therefore, the photographing time when photographing the base image is shorter than the photographing time when photographing the high-resolution training image 31. Therefore, when capturing the base image, the influence of the movement of the subject's eye (for example, visual fixation and blinking), the burden of tracking operations to follow the movement of the subject's eye, etc. compared to when shooting. Note that when the target image is acquired, the ophthalmologic image capturing apparatus 11B automatically sets the capturing area of the base image to the same capturing area as the high-resolution training image 31. As a result, the base image and the target image are acquired more smoothly.

Next, the CPU 23 inputs the base image into the above-described mathematical model to obtain a target image having a higher image resolution than the base image (S12). Note that in this embodiment, the CPU 23 sequentially inputs continuously acquired base images into the mathematical model, thereby continuously acquiring target images.

Next, the CPU 23 determines whether an instruction to display the target image on the display device 28 has been input (S14). In this embodiment, the user instructs the ophthalmological image processing device 21 by operating the operation unit 27 as to which of the target image, the base image, or both the target image and the base image should be displayed on the display device 28. can be entered. If an instruction to display the target image has not been input (S14: NO), the process directly advances to S17. If an instruction to display the target image has been input (S14: YES), the CPU 23 displays the target image on the display device 28 (S15). Specifically, in S15 of the present embodiment, a moving image of the tissue is displayed by sequentially displaying the plurality of target images sequentially acquired in S12 on the display device 28. That is, a high-resolution video of the tissue being photographed is displayed on the display device 28 in real time based on the video of the base image (original image) currently photographed by the ophthalmologic image photographing device 11B. Note that the CPU 23 may generate data of a moving image of the target image based on the plurality of target images sequentially acquired in S12, and store the data in the storage device.

Note that the ophthalmological image processing device 21 can input an instruction as to which of the target image, the base image, or both the target image and the base image should be displayed on the display device 28 using various methods. For example, the CPU 23 may cause the display device 28 to display an icon for inputting a display instruction for the target image, an icon for inputting a display instruction for the base image, and an icon for inputting a display instruction for both images. The user may input a display instruction to the ophthalmological image processing device 21 by selecting a desired icon.

Next, the CPU 23 determines whether an instruction to display the base image on the display device 28 has been input (S17). If it has not been input (S17: NO), the process moves directly to S20. If an instruction to display the base image has been input (S17: YES), the CPU 23 displays the base image on the display device 28 (S18). Specifically, in S18 of the present embodiment, a moving image of the tissue is displayed by sequentially displaying the plurality of base images sequentially acquired in S11 on the display device 28. As described above, the CPU 23 can switch between the target image and the base image and display them on the display device 28 according to instructions input by the user. Therefore, the user can easily check both the target image and the base image.

Next, the CPU 23 determines whether an instruction to simultaneously display the target image and the base image on the display device 28 has been input (S20). If it has not been input (S20: NO), the process moves directly to S23. If a simultaneous display instruction has been input (S20: YES), the CPU 23 causes the display device 28 to simultaneously display the target image and the base image. Specifically, in S20 of this embodiment, both the moving image of the target image and the moving image of the base image are displayed on the display device 28.

Note that in S15, S18, and S21, still images may be displayed instead of moving images. The CPU 23 may switch between displaying moving images and displaying still images in accordance with instructions input by the user. Further, the CPU 23 may cause the storage device 24 to store data of at least one of the moving image and the still image without displaying the moving image and the still image on the display device 28.

Next, the CPU 23 determines whether an instruction to end the display has been input (S23). If it has not been input (S23: NO), the process returns to S11, and the processes from S11 to S23 are repeated. When an instruction to end the display is input (S23: YES), the CPU 23 executes various analysis processes on the target image (in the present embodiment, at least one of the plurality of target images that are sequentially acquired). S24). As mentioned above, the image resolution of the target image is higher than the image resolution of the base image. Therefore, by performing the analysis process on the target image, the analysis accuracy is improved compared to the case where the analysis process is performed on the base image. Note that the specific content of the analysis process can be selected as appropriate. As an example, in S24 of this embodiment, the blood vessel density of the fundus (for example, the blood vessel density around the foveal non-perfused area) is analyzed.

Furthermore, the timing for displaying the target image includes, for example, the timing for displaying the target image in real time while the ophthalmological image capturing device 11B is capturing images, the timing for making the user check whether the captured image is good, and the timing for displaying the analysis results after capturing. Or there are multiple timings, such as when to display it with a report. The CPU 23 separately instructs which of the target image, the base image, and both the target image and the base image should be displayed on the display device 28 at each of a plurality of timings at which the target image can be displayed. can be entered. In other words, the user sets in advance which of the target image, the base image, or both the target image and the base image is to be displayed on the display device 28, depending on the timing at which the target image can be displayed. You can leave it there. As a result, user convenience is further improved.

(Synthetic image generation processing)
With reference to FIG. 6, the composite image generation process executed by the CPU 23 of the ophthalmological image processing device 21 will be described. As shown in FIG. 6, the composite image 40 includes a target image 42 and a region of interest image 44. The target image is a high-resolution ophthalmological image obtained by inputting the base image to the mathematical model, as described in S12 of FIG. The region of interest image 44 is an image in which a part of the region of interest of the base image is photographed by the ophthalmological image photographing device 11B. The attention area image 44 is photographed at an image resolution higher than that of the base image. Further, the attention area image 44 is the image itself (that is, the original image) photographed by the ophthalmologic image photographing device 11B. The method of setting the position, size, shape, etc. of the region of interest relative to the imaging region of the base image can be selected as appropriate. For example, the ophthalmologic image capturing apparatus 11B may set a region of interest within the capturing region of the base image according to an instruction input by the user. In addition, the ophthalmologic image capturing device 11B performs image processing or the like on the base image to detect a characteristic region (for example, at least one of the macula, optic disc, fundus blood vessel, etc.) in the base image, and The region of interest may be automatically set based on.

The CPU 23 of the ophthalmological image processing device 21 acquires a base image and a region of interest image 44. The CPU 23 inputs the base image into a mathematical model to obtain a target image 42 having a higher image resolution than the base image. The CPU 23 generates a composite image 40 by combining the region of interest image 44 with the region of interest in the target image 42 . According to the composite image 40, the user can confirm the region of interest in the original image, and can also confirm the entire image at high resolution.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to execute only some of the techniques exemplified in the above embodiments. For example, instead of adopting a technique of using a panoramic image as a high-resolution training image, a technique of performing at least one of a noise adding process and a blurring process on a low-resolution training image may be adopted.

Note that the process of acquiring the base image in S11 of FIG. 5 is an example of a "base image acquisition step." The process of acquiring the target image in S12 of FIG. 5 is an example of a "target image acquisition step." The process of simultaneously displaying the target image and the base image in S21 of FIG. 5 is an example of a "simultaneous display process" and a "simultaneous display step." The process of switching and displaying the base image and the target image in S14 to S18 in FIG. 5 is an example of a "switching display process" and a "switching display step." The process of acquiring a training data set in S1 and S2 of FIG. 2 is an example of an "image set acquisition step." The process of constructing a mathematical model in S5 of FIG. 2 is an example of a "training step." The process of acquiring a panoramic image in S1 of FIG. 2 is an example of a "panoramic image acquisition step." The process of generating the low-resolution training image 32 in S2 of FIG. 2 is an example of a "low-resolution training image generation step." The process of adding noise to the low-resolution training image 32 in S3 of FIG. 2 and the process of adding blur to the low-resolution training image 32 in S4 of FIG. 2 are examples of "degradation processing" and "degradation step". .

1 Mathematical model construction device 3 CPU
4 Storage device 11 Ophthalmology image capturing device 21 Ophthalmology image processing device 23 CPU
24 Storage device 27 Operation unit 28 Display device 30 Training data set 31 High-resolution training image 32 Low-resolution training image 40 Composite image 42 Target image 44 Attention area image

Claims (4)

An ophthalmological image processing device that processes an ophthalmological image that is an image of a tissue of an eye to be examined,
The control unit of the ophthalmological image processing device includes:
Obtaining a base image that is an ophthalmological image taken by an ophthalmological image capturing device,
obtaining a target image whose image resolution is higher than that of the base image by inputting the base image to a mathematical model trained by a machine learning algorithm;
The mathematical model is inputted with a low-resolution training image that is at least one ophthalmological image from a set of a plurality of ophthalmological images that target the same tissue site and are taken by the same ophthalmological image capturing device. training data, and a high-resolution training image that is an ophthalmological image having a higher image resolution than the low-resolution training image is trained as output training data,
Of the target image and the base image, an image to be displayed on a display device can be set in advance according to a timing at which the target image can be displayed,
When a timing at which the target image can be displayed has arrived, an image that is preset according to the timing among the target image and the base image is displayed on the display device. Ophthalmology image processing device. The ophthalmological image processing device according to claim 1,
An ophthalmological image processing device, wherein the base image, the low-resolution training image, and the high-resolution training image are OCT angio images of the eye to be examined taken by an OCT device. The ophthalmological image processing device according to claim 1 or 2,
The control unit includes:
An ophthalmological image processing apparatus characterized in that an image to be displayed on the display device among the target image and the base image is set in advance according to an instruction input by a user. An ophthalmologic image processing program executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmology image processing program is executed by the control unit of the ophthalmology image processing device,
a base image acquisition step of acquiring a base image that is an ophthalmologic image captured by an ophthalmologic image capturing device;
a target image acquisition step of acquiring a target image having an image resolution higher than that of the base image by inputting the base image to a mathematical model trained by a machine learning algorithm;
causing the ophthalmological image processing device to execute
The mathematical model is inputted with a low-resolution training image that is at least one ophthalmological image from a set of a plurality of ophthalmological images that target the same tissue site and are taken by the same ophthalmological image capturing device. training data, and a high-resolution training image that is an ophthalmological image having a higher image resolution than the low-resolution training image is trained as output training data,
Of the target image and the base image, an image to be displayed on a display device can be set in advance according to a timing at which the target image can be displayed,
The ophthalmological image includes a step of displaying, on the display device, an image that is preset according to the timing, among the target image and the base image, when a timing at which the target image can be displayed has arrived. An ophthalmological image processing program that is executed by a processing device.

Images