JP2023115058A - Ophthalmologic image processing device and ophthalmologic image processing program - Google Patents

Abstract

To provide an ophthalmologic image processing device and an ophthalmologic image processing program capable of assisting a user's work more appropriately.SOLUTION: A control unit of an ophthalmologic image processing device acquires an ophthalmologic image taken by an ophthalmologic imaging device. The control unit acquires an analysis result on at least one of a specific disease in an eye to be examined and a specific structure in the eye to be examined by inputting the ophthalmologic image in a mathematical model trained by a machine learning algorithm. The control unit acquires information on weight distribution on the analysis by the mathematical model as supplemental distribution information with an image area of the ophthalmologic image input in the mathematical model as a variable. The control unit sets an area of attention to a part of an image area of the ophthalmologic image based on the supplemental distribution information. The control unit acquires an image of a tissue including the area of attention of the tissue of the eye to be examined and causes it to be displayed in a display unit.SELECTED DRAWING: Figure 9

Description

The present disclosure relates to an ophthalmic image processing apparatus that processes an ophthalmic image of an eye to be examined, and an ophthalmic image processing program executed in the ophthalmic image processing apparatus.

In recent years, techniques have been proposed for obtaining various medical information using mathematical models trained by machine learning algorithms. For example, in the ophthalmologic apparatus described in Patent Document 1, eye shape parameters are input to a mathematical model trained by a machine learning algorithm to acquire IOL-related information (e.g., expected postoperative anterior chamber depth) of the eye to be examined. be done. The IOL power is calculated based on the acquired IOL-related information.

JP 2018-51223 A

It is also conceivable to obtain analytical results for diseases and/or structures in the subject's eye by inputting the ophthalmic images into a mathematical model trained by a machine learning algorithm. Here, since the presence or absence of disease, the degree of disease, structural features, etc. in the eye to be examined differ depending on the eye to be examined, the certainty of the analysis results may vary. Therefore, simply presenting analysis results output by a mathematical model to a user (for example, a doctor) may not be able to adequately assist the user's work such as diagnosis.

A typical object of the present disclosure is to provide an ophthalmic image processing apparatus and an ophthalmic image processing program capable of more appropriately assisting a user's work.

An ophthalmic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of a tissue of an eye to be examined, wherein a control unit of the ophthalmic image processing apparatus includes: Acquiring an ophthalmologic image captured by an imaging device and inputting the ophthalmologic image into a mathematical model trained by a machine learning algorithm to obtain an analysis result for a specific structure in the eye to be examined and inputting it into the mathematical model. Obtained as supplemental distribution information is weight distribution information relating to the analysis by the mathematical model using the image region of the obtained ophthalmologic image as a variable, and information indicating the distribution of the confidence of the analysis by the mathematical model. and setting a region of interest in an image region of the ophthalmologic image in which the degree of certainty indicated by the supplemental distribution information is equal to or less than a threshold , and displaying the region of interest superimposed on the front image of the tissue of the subject's eye.

An ophthalmic image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of tissue of an eye to be examined, the ophthalmic image processing A program is executed by the control unit of the ophthalmic image processing device to obtain an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device, and inputting the ophthalmic image to a mathematical model trained by a machine learning algorithm. By doing so, an analysis result acquisition step of acquiring an analysis result for a specific structure in the eye to be inspected, and weighting related to the analysis by the mathematical model using the image region of the ophthalmologic image input to the mathematical model as a variable. a supplementary distribution information acquisition step of acquiring, as supplementary distribution information, information that is distribution information and indicates the distribution of the degree of certainty of the analysis by the mathematical model; and a display step of superimposing and displaying the attention area on a front image of the tissue of the eye to be examined. Let

According to the ophthalmic image processing apparatus and the ophthalmic image processing program according to the present disclosure, the user's work is more appropriately assisted.

1 is a block diagram showing schematic configurations of a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B; FIG. FIG. 1 shows an example of a frontal image of a fundus with age-related macular degeneration disease. FIG. 1 shows an example of a frontal image of a fundus with diabetic retinopathy disease. 3 is a diagram showing an example of a supplementary map 40A when automatic analysis is performed on the ophthalmologic image 30A shown in FIG. 2; FIG. 4 is a diagram showing an example of a supplementary map 40B when automatic analysis is performed on the ophthalmologic image 30B shown in FIG. 3; FIG. 4 is a flowchart of mathematical model building processing executed by the mathematical model building device 1; 4 is a flowchart of confirmation image display processing executed by the ophthalmologic image processing apparatus 21. FIG. FIG. 11 is an explanatory diagram for explaining an example of a method of setting areas 46A, 46B, and 48 based on a supplemental map 40B; FIG. 10 is a diagram showing an example of a display screen of a display device 28 displaying an ophthalmologic image 30B, a supplemental map 40B, and a confirmation image 50; 4 is a flowchart of integrated map display processing executed by the ophthalmologic image processing apparatus 21. FIG. FIG. 4 is an explanatory diagram for explaining an example of a method of integrating two supplementary maps 40A and 40B to generate an integrated map 60; 6 is a diagram showing an example of a state in which an integrated map 60 is superimposed on an ophthalmologic image 30. FIG.

<Overview>
A control unit of an ophthalmic image processing apparatus exemplified in the present disclosure acquires an ophthalmic image captured by an ophthalmic image capturing apparatus. The control unit inputs ophthalmic images into a mathematical model trained by a machine learning algorithm to obtain analysis results for a specific disease and/or specific structure in the eye to be examined. The control unit obtains weight distribution information related to analysis by the mathematical model, with the image area (that is, coordinates in the area) of the ophthalmologic image input to the mathematical model as variables, as supplementary distribution information. The control unit sets a region of interest in a part of the image region of the ophthalmologic image based on the supplemental distribution information. The control unit acquires an image of tissue including the region of interest among the tissues of the eye to be examined, and causes the display unit to display the image.

In this case, the user (for example, a doctor) can appropriately check the tissue of the region of interest set based on the supplemental distribution information among the tissues of the subject's eye based on the displayed image. For example, when an image of tissue in a region with a large weight for analysis is displayed, the user can easily confirm an image of an important region with a large weight. In addition, when an image of tissue in a region with a small weight is displayed, the user can easily confirm, for example, an image of a site with low certainty of analysis. Therefore, operations such as diagnosis by the user are appropriately assisted.

The ophthalmic imaging device that captures the ophthalmologic image to be input to the mathematical model and the ophthalmic imaging device that captures the image of the tissue including the region of interest may be the same device or different devices. good.

Also, the number of ophthalmologic images input to the mathematical model may be one or plural. When multiple ophthalmic images are input to the mathematical model, the multiple ophthalmic images may be images captured by the same device or may be images captured by different devices. Also, the dimension of the ophthalmologic image input to the mathematical model is not particularly limited. The ophthalmologic image may be a still image or a moving image.

One or more attention areas may be set. When multiple attention areas are set, images of all attention areas may be displayed, or images of some of the multiple attention areas (for example, an attention area selected by the user, etc.) may be displayed. may

The supplemental distribution information may include information (sometimes referred to as an "attention map") indicating the distribution of the degree of influence (degree of attention) that the mathematical model has influenced when outputting the analysis result. In this case, the region of interest is appropriately set according to the degree of influence on the automatic analysis by the mathematical model.

The supplemental distribution information may also include information indicating the distribution of the confidence of the analysis by the mathematical model (hereinafter sometimes referred to as "confidence map"). In this case, the attention area is appropriately set according to the certainty of the analysis by the mathematical model.

The “certainty degree” may be the high degree of certainty of the analysis, or may be the reciprocal of the low degree of certainty (uncertainty degree). Further, for example, when the uncertainty is expressed by x%, the certainty may be a value expressed by (100-x)%. "Confidence" indicates the degree of confidence in the mathematical model's analysis. Confidence and accuracy of analysis results are not always proportional.

The specific contents of the confidence map can be selected as appropriate. For example, the confidence factor may include the entropy (average amount of information) of the probability distribution in the automatic analysis by the mathematical model. Entropy represents the degree of uncertainty, clutter, and disorder. The entropy of the probability distribution is 0 when the confidence in the automatic analysis is the maximum value. Also, the lower the confidence, the higher the entropy. Therefore, by using the entropy of the probability distribution as the confidence, the confidence map is appropriately generated. Also, a value other than entropy may be adopted as the certainty factor. For example, at least one of the standard deviation, the coefficient of variation, the variance, etc., which indicates the degree of dispersion of the probability distribution in the automatic analysis, may be used as the degree of certainty. KL divergence or the like, which is a measure of the difference between probability distributions, may be used as the degree of certainty. The maximum value of the probability distribution may be used as the certainty. In addition, when ranking multiple diseases by automatic analysis, the probability of first place, or the probability of first place and the probability of other ranks (for example, second place, or multiple ) may be used as the confidence factor. In addition, variation in output between a plurality of mathematical models with mutually different data, conditions, etc. used for learning may be used as the degree of certainty. In this case, it can be applied even if the output of the mathematical model is not a probability distribution.

When a confidence map is used as the supplemental distribution information, the control unit assigns a region of interest to a region in the image region of the ophthalmologic image in which the confidence of automatic analysis to the effect that it is a specific disease or structure is equal to or less than a threshold. may be set. In this case, an image of a part of the tissue of the subject's eye that has a low degree of certainty in a specific analysis (for example, an analysis indicating that there is a specific disease) (that is, a part with a low degree of certainty in the analysis) is displayed on the display unit. be. Therefore, the user can check the state of the part with low confidence in the analysis by himself/herself by the displayed image. Therefore, the user's work such as diagnosis is more appropriately assisted.

The control unit selects an area including a position having the largest weight (for example, at least one of the degree of influence and the degree of certainty) indicated by the supplementary distribution information, or an area having a weight equal to or greater than a threshold, within the image area of the ophthalmologic image. , a region of interest may be set. In this case, an image of a portion of the tissue of the subject's eye that is weighted heavily in the automatic analysis is displayed on the display unit. Therefore, the user can check the state of the important part with a large weight by himself/herself by the displayed image. Therefore, the user's work such as diagnosis is more appropriately assisted.

Note that it is also possible to change the method of setting the attention area. For example, the control unit may set a region of interest in a region of the image region of the ophthalmologic image whose weight is relatively large compared to other regions. Even in this case, the user can appropriately confirm the state of the important part with a large weight.

Further, the control unit may cause the display unit to display an image of a supplemental map showing the weight distribution. The control unit may set the region of interest to a region corresponding to a region specified by the user on the displayed supplemental map, among the image regions of the ophthalmologic image input to the mathematical model. The control unit may acquire an image of the tissue including the region of interest and display it on the display unit. In this case, the user can easily cause the ophthalmologic image processing apparatus to display the image of the region that the user wants to check according to the weight distribution.

The control unit may acquire a tomographic image of a portion of the tissue of the subject's eye that passes through the set region of interest, and display the image on the display unit. In this case, the user can appropriately confirm the state of the tissue in the depth direction at the site passing through the region of interest based on the displayed image. Therefore, the user can perform tasks such as diagnosis more appropriately. Note that the tomographic image may be a two-dimensional tomographic image or a three-dimensional tomographic image. A tomographic image may be captured by, for example, an OCT apparatus.

The control unit may extract an image region including the region of interest from an ophthalmologic image of the tissue of the subject's eye and cause the display unit to display the extracted image region. In this case, the user can more appropriately check the state of the tissue in the region of interest from the image in which the region including the region of interest is extracted. The image from which the image region is extracted may be the same image as the ophthalmologic image input to the mathematical model, or may be a different image.

Note that it is also possible to change the mode of the image displayed on the display unit based on the attention area. For example, the control unit adjusts the image quality (for example, at least one of contrast, resolution, transparency, etc.) of an image region including a region of interest in an ophthalmologic image obtained by photographing the tissue of an eye to be examined, and the image region other than the region of interest. The image may be displayed on the display section with a higher image quality. Further, the control unit may enlarge the image area including the attention area and display it on the display unit. Even in this case, the user can appropriately confirm the state of the tissue in the region of interest. In addition, the control unit may mask a region other than the region of interest in the ophthalmologic image of the tissue of the subject's eye and cause the display unit to display the masked region.

The control unit may switch display and non-display of the image of the tissue including the region of interest on the display unit in accordance with an instruction input by the user. In this case, the user can confirm the state of the tissue including the region of interest at an appropriate timing.

Even if the control unit acquires the analysis result based on the ophthalmologic image without using the mathematical model trained by the machine learning algorithm, it is possible to acquire the supplemental distribution information accompanying the analysis result. . In this case, the ophthalmologic image processing apparatus can also be expressed as follows. An ophthalmic image processing device for processing an ophthalmic image that is an image of tissue of an eye to be examined, wherein a control unit of the ophthalmic image processing device acquires an ophthalmic image captured by an ophthalmic imaging device, and identifies obtaining an analysis result for at least one of the disease and the specific structure based on the ophthalmologic image, and using the image region of the ophthalmologic image as a variable, weight distribution information related to the analysis is obtained as supplemental distribution information. setting a region of interest in a part of the image region of the ophthalmologic image based on the supplemental distribution information, acquiring an image of a tissue including the region of interest among tissues of the eye to be examined, and displaying the image on a display unit display.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B are used. The mathematical model building device 1 builds a mathematical model by training the mathematical model with a machine learning algorithm. The constructed mathematical model outputs analysis results for at least one of a specific disease and/or a specific structure in the subject's eye based on the input ophthalmologic image. The ophthalmologic image processing apparatus 21 acquires analysis results using a mathematical model, and also calculates a weight distribution (for example, a Supplementary distribution information (details will be described later) indicating the distribution of influence and/or the distribution of confidence of automatic analysis is acquired. Based on the supplemental distribution information, the ophthalmologic image processing apparatus 21 generates various types of information for assisting the work of a user (for example, a doctor) and presents it to the user. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images, which are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as “PC”) is used for the mathematical model building device 1 of this embodiment. Although the details will be described later, the mathematical model construction device 1 includes data of an ophthalmologic image of the subject's eye (hereinafter referred to as a "training ophthalmologic image") acquired from the ophthalmologic imaging device 11A, and Data indicative of the disease and/or structure of the optometry are used to train the mathematical model. As a result, a mathematical model is built. However, a device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic imaging device 11A may function as the mathematical model building device 1 . Also, the controllers of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may work together to construct the mathematical model.

A PC is used for the ophthalmologic image processing apparatus 21 of this embodiment. However, a device that can function as the ophthalmologic image processing apparatus 21 is not limited to a PC either. For example, the ophthalmologic image capturing device 11B, a server, or the like may function as the ophthalmologic image processing device 21 . When the ophthalmic image capturing device 11B functions as the ophthalmic image processing device 21, the ophthalmic image capturing device 11B can acquire analysis results and supplemental distribution information from the captured ophthalmic image while capturing the ophthalmic image. Further, the ophthalmologic image capturing device 11B can also capture an appropriate part of the subject's eye based on the acquired supplementary distribution information. Also, a portable terminal such as a tablet terminal or a smartphone may function as the ophthalmologic image processing apparatus 21 . The controllers of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmologic imaging apparatus 11B) may work together to perform various processes.

Also, in this embodiment, a case where a CPU is used as an example of a controller that performs various processes will be described. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be employed as a controller to speed up processing.

The mathematical model construction device 1 will be described. The mathematical model building device 1 is located, for example, in a manufacturer or the like that provides the ophthalmic image processing device 21 or an ophthalmic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5 . The control unit 2 includes a CPU 3, which is a controller for control, and a storage device 4 capable of storing programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a later-described mathematical model building process (see FIG. 2). Also, the communication I/F 5 connects the mathematical model construction device 1 with other devices (for example, the ophthalmologic imaging device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8 . The operation unit 7 is operated by the user to input various instructions to the mathematical model construction device 1 . For example, at least one of a keyboard, a mouse, a touch panel, and the like can be used as the operation unit 7 . A microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7 . The display device 8 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used as the display device 8 . It should be noted that the “image” in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as “ophthalmologic image”) from the ophthalmologic imaging device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic imaging device 11A, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory).

The ophthalmologic image processing apparatus 21 will be described. The ophthalmologic image processing apparatus 21 is installed, for example, in a facility (for example, a hospital, a health checkup facility, or the like) for diagnosing or examining a subject. The ophthalmologic image processing apparatus 21 includes a control unit 22 that performs various control processes, and a communication I/F 25 . The control unit 22 includes a CPU 23 which is a controller for control, and a storage device 24 capable of storing programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing later-described ophthalmologic image processing (confirmation image display processing shown in FIG. 7 and integrated map display processing shown in FIG. 10). The ophthalmologic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1 . The communication I/F 25 connects the ophthalmologic image processing apparatus 21 with other devices (for example, the ophthalmic imaging apparatus 11B, the mathematical model construction apparatus 1, etc.).

The ophthalmologic image processing apparatus 21 is connected to an operation section 27 and a display device 28 . Various devices can be used for the operation unit 27 and the display device 28, similarly to the operation unit 7 and the display device 8 described above.

The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmologic image processing apparatus 21 may acquire an ophthalmologic image from the ophthalmologic imaging apparatus 11B, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory). Further, the ophthalmologic image processing apparatus 21 may acquire, via communication or the like, a program or the like for realizing the mathematical model constructed by the mathematical model construction apparatus 1 .

The ophthalmologic imaging devices 11A and 11B will be described. As an example, in this embodiment, an ophthalmic imaging device 11A that provides an ophthalmic image to the mathematical model building device 1 and an ophthalmic imaging device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used. . However, the number of ophthalmic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from a plurality of ophthalmologic imaging devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic imaging device.

Further, in this embodiment, an OCT apparatus is exemplified as the ophthalmic imaging apparatus 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning optometric device (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

The ophthalmologic imaging apparatus 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic imaging section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B), which is a controller for control, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like.

The ophthalmologic image capturing unit 16 has various configurations necessary for capturing an ophthalmologic image of the subject's eye. The ophthalmologic image capturing unit 16 of the present embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and a scanning unit for scanning the measurement light. It includes an optical system for irradiating an eye to be examined, a light receiving element for receiving the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic image capturing apparatus 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the subject's eye. Specifically, the CPU 13 captures a two-dimensional tomographic image of a cross section that intersects the scan lines by scanning OCT light (measurement light) along the scan lines. The two-dimensional tomographic image may be an averaging image generated by averaging a plurality of tomographic images of the same region. Further, the CPU 13 can capture a three-dimensional tomographic image of the tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by causing measurement light to scan each of a plurality of scan lines at different positions in a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 acquires a three-dimensional tomographic image by combining a plurality of captured two-dimensional tomographic images. Furthermore, the ophthalmologic image capturing apparatus 11 of the present embodiment can also capture a two-dimensional front image when the fundus of the subject's eye is viewed from the front (in the Z direction along the optical axis of the measurement light). The data of the two-dimensional frontal image includes, for example, integrated image data in which luminance values are integrated in the depth direction at each position in the XY directions intersecting the Z direction, integrated values of spectral data at each position in the XY directions, , brightness data at each position in the XY directions in any layer of the retina (for example, the surface layer of the retina), or the like.

(automatic analysis)
An example of automatic analysis performed by the ophthalmologic image processing apparatus 21 will be described with reference to FIGS. 2 and 3. FIG. As described above, the ophthalmic image processor 21 performs automatic analysis of the disease and/or structure of the subject's eye using a mathematical model. As an example, in the present embodiment, a case of executing automatic analysis (that is, automatic diagnosis) of a disease of an eye to be examined will be described. The type of disease targeted for automatic analysis can be selected as appropriate. The ophthalmologic image processing apparatus 21 of this embodiment can automatically analyze the presence or absence of each of a plurality of diseases including age-related macular degeneration and diabetic retinopathy by inputting an ophthalmologic image into a mathematical model.

FIG. 2 shows an example of an en face image of a fundus with age-related macular degeneration disease. In the ophthalmologic image 30A illustrated in FIG. 2, a lesion 35 due to age-related macular degeneration appears in the vicinity of the macula 32 in addition to the optic papilla 31, the macula 32, and the fundus blood vessel 33 in the fundus. The mathematical model is based on the data (input training data) of the ophthalmologic image 30A illustrated in FIG. , is pre-trained. Therefore, when an ophthalmologic image similar to that of FIG. 2 is input to the mathematical model, an automatic analysis result indicating that the subject's eye is likely to have age-related macular degeneration is output.

FIG. 3 shows an example of an en face image of a fundus with diabetic retinopathy disease. In the ophthalmologic image 30B illustrated in FIG. 3, in addition to the optic papilla 31, the macula 32, and the fundus blood vessel 33 in the fundus, a lesion 36 due to diabetic retinopathy appears in the vicinity of the fundus blood vessel 33. The mathematical model is based on the data (input training data) of the ophthalmologic image 30B illustrated in FIG. pre-trained. Therefore, when an ophthalmologic image similar to that of FIG. 3 is input to the mathematical model, an automatic analysis result indicating that the subject's eye is likely to have diabetic retinopathy is output.

Note that, according to the mathematical model of this embodiment, automatic analysis of a plurality of diseases is performed on one ophthalmologic image. Therefore, in some cases, multiple diseases (for example, age-related macular degeneration and diabetic retinopathy) are automatically analyzed as diseases with a high possibility of developing.

Note that the ophthalmologic image processing apparatus 21, instead of or in conjunction with the automatic analysis of the disease, analyzes the structure of the eye to be examined (for example, at least one of the fundus layer, the macula, the optic papilla, and the fundus blood vessel). Automatic analysis may be performed. Specifically, the ophthalmologic image processing apparatus 21 inputs a tomographic image of the fundus (at least one of a two-dimensional tomographic image and a three-dimensional tomographic image) to a mathematical model, thereby identifying a specific layer or a specific layer in the fundus of the subject's eye. layer boundaries may be automatically analyzed. Further, the ophthalmologic image processing apparatus 21 automatically analyzes the structure of the tissue of the subject's eye (for example, at least one of the macula and the optic papilla) by inputting a front image or a tomographic image of the fundus into the mathematical model. may Further, the ophthalmologic image processing apparatus 21 may automatically analyze the fundus blood vessels of the subject's eye by inputting a front image or a tomographic image of the fundus into a mathematical model. In this case, the ophthalmologic image processing apparatus 21 may automatically analyze arteries and veins in the fundus of the subject's eye.

(Supplemental distribution information)
An example of supplemental distribution information will be described with reference to FIGS. 4 and 5. FIG. In this embodiment, the ophthalmologic image input to the mathematical model is a two-dimensional image. Therefore, the supplementary distribution information also becomes two-dimensional information (map). However, for example, a three-dimensional ophthalmic image may also be input to the mathematical model. In this case, the complementary distribution information is three-dimensional information.

In this embodiment, at least one of an attention map and a confidence map is used as supplemental distribution information (supplemental map).

The attention map is the distribution in the image area of the degree of influence (degree of attention) for each position that the mathematical model has influenced when outputting the automatic analysis result. Regions with a high degree of influence have a stronger influence on the automatic analysis results than regions with a low degree of influence. An example of the attention map is described in, for example, the following papers. "Ramprasaath R. Selvaraju, et al. "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" Proceedings of the IEEE International Conf. erence on Computer Vision, 2017-Oct, pp. 618-626"

A confidence map is the distribution, in an image region, of the autoanalysis confidence at each pixel when the mathematical model performs the autoanalysis. The degree of certainty may be the high certainty of the automatic analysis, or the reciprocal of the low certainty (uncertainty). Also, for example, when the uncertainty is expressed by x%, the certainty may be a value expressed by (100-x)%. As an example, in this embodiment, the entropy (average amount of information) of the probability distribution in the automatic analysis is used as the certainty factor. The entropy of the probability distribution is 0 when the confidence factor in the automatic analysis is the maximum value. Also, the lower the confidence, the higher the entropy. Therefore, for example, by using the reciprocal of the entropy of the probability distribution or the like as the confidence, the confidence map is appropriately generated. However, information other than the entropy of the probability distribution (eg, standard deviation, coefficient of variation, variance, etc. of the probability distribution) may be used to generate the confidence map. When the probability distribution in the automatic analysis is used as the degree of certainty, the probability distribution may be the probability distribution of the automatic analysis for each pixel, or the probability distribution with one or more dimensional coordinates as random variables. . In addition, when performing some ranking by automatic analysis, the probability of 1st place, or the probability of 1st place and the probability of other rankings (for example, 2nd place, or the sum of multiple probabilities of 2nd place and below, etc.) , etc. may be used as the degree of certainty.

FIG. 4 is a diagram showing an example of a supplementary map 40A when automatic analysis based on a mathematical model is performed on the ophthalmologic image 30A shown in FIG. In FIG. 4, in order to facilitate comparison between the image area of the ophthalmologic image 30A (see FIG. 2) and the area of the supplemental map 40A, tissues such as the fundus blood vessel 33 in the ophthalmologic image 30A are schematically indicated by dotted lines. . In the supplemental map 40A shown in FIG. 4, the magnitude of the weight (in this embodiment, the degree of influence or the degree of certainty) at each position regarding the automatic analysis result indicating that age-related macular degeneration is highly likely is determined by the color density. is represented by In other words, dark-colored portions have a greater degree of influence or degree of certainty on the automatic analysis results than light-colored portions. (In FIGS. 4 and 5, the depth of color is expressed by the thickness of the line of the figure for convenience.) In the example shown in FIG. It's getting bigger. In addition, the influence or confidence of the center within the lesion 35 is greater than the influence or confidence of the periphery within the lesion 35 .

FIG. 5 is a diagram showing an example of a supplementary map 40B when automatic analysis based on a mathematical model is performed on the ophthalmologic image 30B shown in FIG. In FIG. 5, as in FIG. 4, tissues such as fundus blood vessels in the ophthalmologic image 30B (see FIG. 3) are schematically shown by dotted lines. In the supplemental map 40B shown in FIG. 5, the degree of influence or degree of confidence at each location regarding the automatic analysis result indicating the high possibility of diabetic retinopathy is represented by the intensity of color. Areas with darker colors have a greater degree of influence or degree of certainty on the automatic analysis results. In the example shown in FIG. 5 as well, the degree of influence or degree of certainty is high in the lesion 36 (see FIG. 3).

In the supplementary maps 40A and 40B of the present embodiment, the display mode of the degree of influence or degree of certainty is changed according to the disease or structure targeted for automatic analysis. In this embodiment, the color indicating the degree of impact or degree of confidence is changed according to the disease or structure targeted for automatic analysis. However, it is also possible to change the specific aspect of the supplemental map. For example, the magnitude of influence or confidence may be indicated by generating contour lines by connecting locations with equal magnitudes of influence or confidence. In this case, at least one of the color, line type, line thickness, etc. of the contour lines may be changed according to the disease or structure to be automatically analyzed.

In addition, in this embodiment, when an ophthalmologic image is input to the mathematical model, the mathematical model outputs both the automatic analysis result for the ophthalmologic image and supplemental distribution information (supplementary map in this embodiment) that accompanies the automatic analysis. . However, it is also possible to vary the manner in which the supplemental distribution information is generated. For example, the CPU 23 of the ophthalmologic image processing apparatus 21 may generate supplemental distribution information based on the automatic analysis results output by the mathematical model.

Moreover, in this embodiment, the case of automatically analyzing the disease of the eye to be examined is illustrated. However, it is also possible to generate supplemental distribution information similar to the above-described supplemental distribution information when automatically analyzing the structure of the subject's eye. For example, when automatically analyzing the arteries and veins in the fundus of the subject's eye, supplementary distribution information for the automatic analysis results indicating arteries and supplementary distribution information for the automatic analysis results indicating veins are generated. good too. In this case, supplemental distribution information may be generated in a display form indicating that the blood vessels with a low degree of certainty in the automatic analysis of arteries and veins among the fundus blood vessels.

(mathematical model construction processing)
A mathematical model building process executed by the mathematical model building device 1 will be described with reference to FIG. The mathematical model building process is executed by CPU 3 according to a mathematical model building program stored in storage device 4 . In the mathematical model building process, the mathematical model is trained with the training data set to build the mathematical model for automatic analysis of the disease or structure of the eye to be examined. The training data set includes data on the input side (training data for input) and data on the output side (training data for output).

As shown in FIG. 6, the CPU 3 acquires training ophthalmologic image data, which is an ophthalmologic image captured by the ophthalmologic imaging device 11A, as input training data (S1). In this embodiment, the training ophthalmologic image data is acquired by the mathematical model construction device 1 after being generated by the ophthalmologic imaging device 11A. However, the CPU 3 acquires a signal (for example, an OCT signal) that is the basis for generating a training ophthalmic image from the ophthalmologic imaging device 11A, and generates a training ophthalmic image based on the acquired signal. Image data may be obtained.

In S1 of the present embodiment, a two-dimensional front image of the fundus (so-called Enface image) captured by the ophthalmologic imaging apparatus 11A, which is an OCT apparatus, is acquired as a training ophthalmologic image. However, the training ophthalmologic images may be captured by devices other than the OCT device (for example, at least one of an SLO device, a fundus camera, an infrared camera, and a corneal endothelial cell imaging device). Also, the training ophthalmologic image is not limited to the two-dimensional frontal image of the fundus. For example, a two-dimensional tomographic image, a three-dimensional tomographic image, or the like may be acquired as a training ophthalmologic image. A moving image may be acquired as the training ophthalmologic image.

Next, the CPU 3 acquires, as output training data, data indicating at least one of the disease and structure (diseases in this embodiment) of the subject's eye for which the ophthalmologic training image was captured (S2). As an example, in the present embodiment, an operator (for example, a doctor) checks training ophthalmological images to diagnose a disease. By inputting 1, training data for output is generated. The training data for output may include data indicating the position of the lesion in addition to data regarding the presence or absence of disease and the type of disease.

It is also possible to change the output training data. For example, when performing automatic analysis of the structure of the subject's eye using a mathematical model, at least the position of a specific structure (for example, the position of a layer, the position of a boundary, the position of a specific tissue, etc.) in training ophthalmologic images Either) may be used as output training data.

CPU 3 then performs training of the mathematical model using the training data set by a machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

A neural network is a technique that imitates the behavior of a neural network of living organisms. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probability neural networks (Boltzmann machine, Baysian network, etc.), etc.

Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When a random forest is used, branches of a plurality of decision trees learned in advance as discriminators are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining a plurality of weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. The SVM learns the parameters of the linear input element, for example, based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance to each data point from the training data (hyperplane separation theorem).

The mathematical model, for example, predicts the relationship between input data (in this embodiment, data of two-dimensional frontal images similar to training ophthalmological images) and output data (in this embodiment, data of automatic analysis results regarding diseases). A data structure for A mathematical model is built by being trained using a training data set. As described above, the training data set is a set of input training data and output training data. For example, training updates the correlation data (eg, weights) for each input and output. The mathematical model of the present embodiment is also trained to output supplemental distribution information (in the present embodiment, supplemental maps) that accompanies the automatic analysis along with the results of the automatic analysis.

In this embodiment, a multilayer neural network is used as the machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data of automatic analysis results to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used.

Note that other machine learning algorithms may be used. For example, Generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as machine learning algorithms.

The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S4: NO). That is, the mathematical model is iteratively trained with multiple training data sets, including a training data set of diseased eyes (see, for example, FIGS. 2 and 3) and a training data set of non-diseased eyes. . When construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. Programs and data for realizing the constructed mathematical model are installed in the ophthalmologic image processing apparatus 21 .

(Confirmation image display processing)
Confirmation image display processing executed by the ophthalmologic image processing apparatus 21 will be described with reference to FIGS. 7 to 9 . The confirmation image is an image that is highly likely to be effective for direct confirmation by the user. In this embodiment, the confirmation image is automatically acquired based on the supplemental distribution information. The confirmation image display process is executed by the CPU 23 according to the ophthalmologic image processing program stored in the storage device 21 .

First, the CPU 23 acquires an ophthalmologic image of the subject's eye (S11). The type of ophthalmologic image acquired in S11 is desirably the same as the ophthalmologic image used as input training data in the above-described mathematical model building process (see FIG. 6). As an example, in S11 of the present embodiment, a two-dimensional front image of the fundus of the subject's eye is obtained. Note that the CPU 23 may acquire a signal (for example, an OCT signal) from which an ophthalmologic image is generated from the ophthalmologic imaging apparatus 11B and generate an ophthalmologic image based on the acquired signal.

The CPU 23 inputs the obtained ophthalmologic image to the mathematical model to obtain the automatic analysis result output by the mathematical model (S12). As described above, in this embodiment, automatic analysis results for each of a plurality of specific diseases are output by the mathematical model.

The CPU 23 acquires supplemental distribution information accompanying the automatic analysis in S12 (S13). As described above, in this embodiment, at least one of the attention map and the confidence map is acquired as supplemental distribution information (supplemental map). Also, in this embodiment, the mathematical model outputs both automatic analysis results and supplemental distribution information. However, the CPU 23 may generate supplemental distribution information based on the automatic analysis results.

Next, the CPU 23 sets a region of interest in a part of the image region of the ophthalmologic image acquired in S11 based on the supplemental distribution information (S16). More specifically, the ophthalmologic image processing apparatus 21 of the present embodiment, when a certainty map is acquired as supplemental distribution information, detects a region with a low certainty (that is, a specific disease in the present embodiment). It is possible to display the image of the part where the certainty of the automatic analysis is considered to be low on the display device 28 . In this case, the user can personally check the state of the part where the certainty of the specific automatic analysis is low by the displayed confirmation image. Further, the ophthalmologic image processing apparatus 21 of the present embodiment can cause the display device 28 to display an image of a region having a large weight (at least one of the degree of influence and the degree of certainty) indicated by the supplemental distribution information. In this case, the user can appropriately check the state of the important part with a large weight (that is, the state of the part determined to be highly likely to have a specific disease in this embodiment) using the confirmation image. can be done. By operating the operation unit 27 , the user can input to the ophthalmologic image processing apparatus 21 either an instruction to display a region with a low degree of confidence or an instruction to display a region with a high degree of influence or confidence.

When an instruction to display a part with a low confidence factor is input, the CPU 23 sets an attention area to an area where the confidence factor indicated by the supplemental distribution information (specifically, the confidence map) is equal to or less than the threshold (S16). In other words, the CPU 23 sets the region of interest to a region in which the degree of certainty of automatic analysis indicating a specific disease or structure (a specific disease in this embodiment) is equal to or less than a threshold.

Specific processing contents for setting the attention area can be selected as appropriate. As an example, as shown in FIG. 8, the CPU 23 of the present embodiment separately extracts continuous regions in which the certainty of automatic analysis indicating a specific disease or structure is greater than or equal to a first threshold. In the example shown in FIG. 8, two regions 46A and 46B whose certainty is greater than or equal to the first threshold are extracted. Next, the CPU 23 sets a region of interest to a region in which the maximum value of certainty in the extracted region is equal to or less than the second threshold (second threshold>first threshold). In the example shown in FIG. 8, in the region 46A, since the maximum value of the degree of certainty is larger than the second threshold, it is considered that the certainty of the automatic analysis of the specific disease is high. On the other hand, in the area 46B, the maximum value of the degree of certainty is equal to or less than the second threshold, and the certainty of the automatic analysis of the specific disease is considered to be low. Therefore, in the example shown in FIG. 8, of the two areas 46A and 46B, the area of interest 48 is set to the area 46B in which the maximum value of the degree of certainty is equal to or less than the second threshold. In this case, of the regions 46A and 46B that have been automatically analyzed as having the possibility of a specific disease, only those regions with low automatic analysis certainty are set as regions of interest.

Note that it is also possible to change the processing content for setting the attention area. For example, the CPU 23 may set all the regions whose degrees of certainty are equal to or more than the first threshold value and equal to or less than the second threshold value as regions of interest. As long as the first threshold is a value greater than 0, it can be appropriately set according to various conditions.

Further, when an instruction to display a part with a large weight (influence or confidence in this embodiment) is input, the CPU 23 determines that the supplemental distribution information (more specifically, at least one of the attention map and the confidence map) is A region of interest is set to a region including the position with the largest weight indicated (S16). For example, in the example shown in FIG. 8, the position with the highest degree of influence or degree of certainty is within region 46A. Therefore, the CPU 23 sets the area 46A as the attention area. It should be noted that it is possible to change the processing content in S16 even when an instruction to display a part with a large weight is input. For example, the CPU 23 may set a region of interest to a region whose weight (degree of influence or degree of certainty) is equal to or greater than a threshold. Also, the CPU 23 may set the attention area to an area whose weight is relatively larger than other areas. Also, a region of interest having a specific size may be set so that the cumulative value of the weight within the region is the highest. Two or more attention areas may be set.

Next, the CPU 23 acquires an image of the tissue including the region of interest among the tissues of the subject's eye as a confirmation image (S18). As an example, the CPU 23 of the present embodiment acquires, as a confirmation image, a tomographic image of a portion of the tissue of the eye to be examined that passes through the region of interest.

FIG. 9 is a diagram showing an example of the display screen of the display device 28 displaying the ophthalmologic image 30B, the supplemental map 40B, and the confirmation image 50. As shown in FIG. In the example shown in FIG. 9, a supplementary map 40B is displayed superimposed on the ophthalmologic image 30B input to the mathematical model in a state where the areas are matched. Therefore, the user can appropriately confirm the distribution of weights associated with the automatic analysis on the ophthalmologic image 30B.

In the example shown in FIG. 9, the CPU 23 sets a line 49 indicating the acquisition position of the tomographic image on the two-dimensional attention area 48 and causes the display device 28 to display it. In the example shown in FIG. 9 , one straight line 49 is set across the attention area 48 . However, the line need not be straight. Also, multiple lines may be set. Also, instead of the line, a two-dimensional area indicating the acquisition position of the tomographic image may be set. The CPU 23 acquires a two-dimensional tomographic image 50 of a cross section crossing the set line in the depth direction of the tissue, and causes the display device 28 to display it. Further, when a two-dimensional region indicating the acquisition position of the tomographic image is set, the CPU 23 acquires a three-dimensional tomographic image extending in the depth direction of the tissue from the set two-dimensional region, and displays it on the display device 28. display. Note that the CPU 23 may acquire a tomographic image corresponding to the set line or region from a two-dimensional tomographic image or a three-dimensional tomographic image captured in advance by the ophthalmologic imaging apparatus 11B. Further, the CPU 23 may output to the ophthalmologic image capturing apparatus 11B an instruction to capture a tomographic image corresponding to the set line or area.

Note that in S16 of the present embodiment, the attention area is automatically set based on the supplemental distribution information. However, the attention area may be set based on an instruction input by the user. For example, as shown in FIG. 9, the CPU 23 may cause the user to specify the attention area on the supplemental map 40B while displaying the supplemental map 40B on the display device 28 . The CPU 23 may set the region of interest to the region corresponding to the region designated by the user on the supplemental map 40B, among the image regions of the ophthalmologic image 30B.

In addition, the CPU 23 of this embodiment can display an image other than a tomographic image as a confirmation image. For example, the CPU 23 can extract an image region including the region of interest from an ophthalmologic image of the tissue of the subject's eye as a confirmation image. In this case, the ophthalmologic image from which the confirmation image is extracted may be the same image as the ophthalmologic images 30A and 30B input to the mathematical model, or may be a different image. The ophthalmologic image from which the confirmation image is extracted may be a two-dimensional image or a three-dimensional image.

In addition, the CPU 23 can also cause the display device 28 to display an ophthalmologic image obtained by photographing the tissue of the eye to be inspected by setting the image quality of the image area including the attention area higher than the image quality of other areas. Further, the CPU 23 can enlarge the image area including the attention area in the ophthalmologic image of the tissue of the subject's eye and display it on the display device 28 . Further, the CPU 23 may cause the display device 28 to display the ophthalmologic image with areas other than the attention area masked.

Next, the CPU 23 determines whether or not the user has input an instruction to display a confirmation image (S20). In this embodiment, the user can input an instruction through the operation unit 27 to select display or non-display of the confirmation image on the display device 28 . If a display instruction has been input (S20: YES), the CPU 23 causes the display device 28 to display the confirmation image. On the other hand, when a non-display instruction is input (S20: NO), the CPU 23 hides the confirmation image (S22). The processes of S20 to S23 are repeated until an instruction to end the process is input (S23: NO). When the end instruction is input (S23: YES), the confirmation image display process ends.

8 and 9 illustrate the case where only the automatic analysis result of diabetic retinopathy, which is one of a plurality of diseases and structures, is output. However, as described above, according to the mathematical model of the present embodiment, automatic analysis of multiple diseases is performed on one ophthalmic image. When automatic analysis results for a plurality of diseases or structures are output, the process of displaying confirmation images may be executed for one automatic analysis result or for each of a plurality of automatic analysis results. may be Further, a process of displaying a confirmation image may be executed for an automatic analysis result selected by the user from among a plurality of automatic analysis results.

(Integrated map display processing)
Integrated map display processing executed by the ophthalmologic image processing apparatus 21 will be described with reference to FIGS. 10 to 12 . An integrated map is a map generated by integrating a plurality of supplementary maps attached to each of a plurality of automatic analysis results. According to the integrated map, the distribution of weights (impact or confidence in this embodiment) for each of the multiple automatic analysis results can be easily grasped. The integrated map display process is executed by the CPU 23 according to the ophthalmologic image processing program stored in the storage device 21 .

First, the CPU 23 acquires an ophthalmologic image of the subject's eye (S31). For the process of S31, the same process as the process of S11 described above (see FIG. 7) can be adopted. Next, the CPU 23 obtains a plurality of automatic analysis results for each of a plurality of diseases and structures by inputting the obtained ophthalmologic images into the mathematical model (S32). As described above, in this embodiment, automatic analysis results for each of a plurality of specific diseases are output by the mathematical model.

The CPU 23 acquires a plurality of supplementary maps accompanying each of the plurality of automatic analyzes in S32 (S33). As described above, in this embodiment, at least one of the attention map and the confidence map is acquired as a supplemental map.

The CPU 23 generates an integrated map by integrating the plurality of supplementary maps acquired in S33 within the same area (S34). FIG. 11 is an explanatory diagram illustrating an example of a method of generating an integrated map 60 by integrating two supplementary maps 40A and 40B (see FIGS. 4 and 5). The CPU 23 generates one integrated map 60 by integrating a plurality of supplementary maps (two supplementary maps 40A and 40B in the example shown in FIG. 11) within the same area. Specifically, the CPU 23 of this embodiment changes the display mode for each of the supplementary maps 40A and 40B and integrates the plurality of supplementary maps 40A and 40B. In the example shown in FIG. 11, the CPU 23 changes the color indicating the degree of influence or the degree of certainty for each of the supplementary maps 40A and 40B. However, the CPU 23 may change the display mode other than the color for each supplementary map.

Next, the CPU 23 determines whether or not an instruction to superimpose the integrated map 60 on the ophthalmologic image 30 has been input (S36). In this embodiment, the user can input an instruction via the operation unit 27 to select whether or not to superimpose the integrated map 60 on the ophthalmologic image 30 . If no superimposed display instruction has been input (S36: NO), the CPU 23 hides the superimposed display of the integrated map 60 on the ophthalmologic image 30 (S37). If the superimposed display instruction has been input (S36: YES), the CPU 23 superimposes and displays the integrated map 60 on the ophthalmologic image 30 of the subject's eye as shown in FIG. 12 (S38). Therefore, the user can easily compare the distribution of multiple types of influence or confidence with the tissue distribution of the subject's eye.

Next, the CPU 23 determines whether or not an instruction to change the transparency of the integrated map 60 superimposed on the ophthalmologic image 30 has been input (S39). In this embodiment, the user can specify the transparency of the integrated map 60 by operating the operation unit 27 . If an instruction to change the transparency has been input (S39: YES), the CPU 23 changes the transparency of the integrated map 60 superimposed and displayed on the ophthalmologic image 30 to the instructed transparency (S40). Therefore, the user can more appropriately compare the distribution of multiple types of weights (degrees of influence or degrees of confidence) with the distribution of tissues of the subject's eye. The processes of S36 to S42 are repeated until an instruction to end the process is input (S42: NO). When an end instruction is input (S42: YES), the integrated map display process ends.

The technology disclosed in the above embodiment is merely an example. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. For example, in the above embodiment, the case of executing automatic analysis for each of a plurality of diseases is illustrated. However, even when performing automatic analysis for each of a plurality of structures in the subject's eye, it is possible to perform processing similar to the processing described above. For example, when automatically analyzing the arteries and veins in the fundus of the subject's eye, a supplementary map relating to the automatic analysis results indicating arteries and a supplementary map relating to the automatic analysis results indicating veins are integrated. You can generate a map. In this case, the high-certainty and low-certainty portions of the artery and vein analysis are easily grasped by the integrated map.

Also, in the above embodiment, the two-dimensional supplemental map and integrated map are displayed on the display device 28 . However, the CPU 23 may acquire a three-dimensional supplemental map and cause the display device 28 to display it. Further, the CPU 23 may integrate a plurality of three-dimensional supplementary maps to generate a three-dimensional integrated map and display it on the display device 28 .

Also, in the above embodiments, mathematical models trained by machine learning algorithms are utilized to obtain analytical results for specific diseases and/or specific structures. However, even if the CPU 23 automatically acquires analysis results based on ophthalmologic images without using a mathematical model trained by a machine learning algorithm, it is possible to acquire supplementary distribution information accompanying automatic analysis. is. Also in this case, the CPU 23 may set a region of interest based on the supplemental distribution information and cause the display device 28 to display an image of the tissue including the region of interest, as in the above embodiment. Further, when acquiring a plurality of automatic analysis results without using a machine learning algorithm, the CPU 23 acquires a plurality of supplementary maps accompanying each of the plurality of automatic analyses, and integrates the plurality of supplementary maps. You can generate a map.

It should be noted that the process of acquiring an ophthalmologic image in S11 of FIG. 7 and S31 of FIG. 10 is an example of the "image acquisition step". The process of acquiring the automatic analysis result in S12 of FIG. 7 and S32 of FIG. 10 is an example of the "analysis result acquisition step". The process of acquiring the supplementary distribution information (supplementary map) in S13 of FIG. 7 and S33 of FIG. 10 is an example of the supplementary distribution information acquisition step and the supplementary map acquisition step. The process of setting the attention area in S16 of FIG. 7 is an example of the "attention area setting step". The process of displaying the confirmation image in S21 of FIG. 7 is an example of the "display step". The process of generating and displaying the integrated map in S34 and S38 of FIG. 10 is an example of the "integrated map display step."

1 mathematical model building device 11 ophthalmic imaging device 21 ophthalmic image processing device 23 CPU
24 storage device 28 display device 30A, 30B ophthalmic image 40A, 40B supplemental map 48 region of interest

Claims (5)

An ophthalmic image processing apparatus for processing an ophthalmic image that is an image of tissue of an eye to be examined,
The control unit of the ophthalmologic image processing apparatus includes:
Acquiring an ophthalmic image captured by an ophthalmic imaging device,
Obtaining an analysis result for a specific structure in the eye to be examined by inputting the ophthalmologic image into a mathematical model trained by a machine learning algorithm,
Weight distribution information related to the analysis by the mathematical model with the image region of the ophthalmologic image input to the mathematical model as a variable, and information indicating the distribution of confidence of the analysis by the mathematical model. obtained as distribution information,
setting a region of interest in an image region of the ophthalmologic image in which the degree of certainty indicated by the supplemental distribution information is equal to or less than a threshold ;
An ophthalmologic image processing apparatus, wherein the attention area is superimposed on a front image of the tissue of the eye to be examined. The ophthalmic image processing apparatus according to claim 1 ,
The control unit inputs the ophthalmologic image, which is a tomographic image of the fundus, into the mathematical model, thereby obtaining an analysis result of a specific layer or a boundary of a specific layer in the fundus of the subject's eye. ophthalmic image processing device. The ophthalmic image processing apparatus according to claim 1 or 2 ,
An ophthalmologic image processing apparatus, wherein the control unit acquires a tomographic image of a portion of the tissue of the eye to be examined that passes through the region of interest , and displays the image on a display unit. The ophthalmologic image processing apparatus according to any one of claims 1 to 3,
An ophthalmologic image processing apparatus, wherein the control unit extracts an image region including the region of interest from an ophthalmologic image of the tissue of the eye to be examined, and displays the extracted image region on a display unit. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of an eye to be examined,
By executing the ophthalmic image processing program by the control unit of the ophthalmic image processing apparatus,
an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic imaging device;
an analysis result obtaining step of obtaining analysis results for a specific structure in the eye to be examined by inputting the ophthalmologic image into a mathematical model trained by a machine learning algorithm;
Weight distribution information related to the analysis by the mathematical model with the image region of the ophthalmologic image input to the mathematical model as a variable, and supplementary information indicating the distribution of confidence of the analysis by the mathematical model. a supplemental distribution information acquisition step for acquiring as distribution information;
a region-of-interest setting step of setting a region-of-interest in an image region of the ophthalmologic image in which the degree of certainty indicated by the supplemental distribution information is equal to or less than a threshold ;
a display step of superimposing and displaying the region of interest on a front image of the tissue of the eye to be inspected;
is executed by the ophthalmologic image processing apparatus.

Images