JP2024040372A - Ophthalmologic image processing program and ophthalmologic image processing device - Google Patents

Abstract

To provide an ophthalmologic image processing program and an ophthalmologic image processing device capable of appropriately presenting medical information desired by a user to the user.SOLUTION: A control unit of an ophthalmologic image processing device acquires an ophthalmologic image captured by an OCT device (S11). The control unit acquires a high quality image obtained by improving the quality of a base image as medical information by inputting the base image to a mathematical model trained by a machine learning algorithm with a partial image, which is a part of the acquired ophthalmologic image, or an image generated on the basis of the partial image as the base image. When an instruction to select the partial image is received, the control unit causes medical information acquired on the selected partial image to be displayed in a display unit (S28). The control unit executes preliminary acquisition processing for preliminarily acquiring and storing medical information on a partial image yet to be selected of the ophthalmologic image (S22).SELECTED DRAWING: Figure 5

Description

The present disclosure relates to an ophthalmological image processing program and an ophthalmological image processing apparatus used to process an ophthalmological image of an eye to be examined.

Techniques have been proposed for acquiring various medical information using mathematical models trained by machine learning algorithms. For example, in the ophthalmological device described in Patent Document 1, IOL-related information (e.g., predicted postoperative anterior chamber depth) of the eye to be examined is acquired by inputting eye shape parameters to a mathematical model trained by a machine learning algorithm. be done. IOL power is calculated based on the acquired IOL related information.

It is also conceivable to obtain high-quality images from low-quality images using mathematical models trained by machine learning algorithms. In this case, a low-quality image is input to the mathematical model, and a high-quality image is output.

JP 2018-51223 A

The user may select a part of the image of interest (hereinafter referred to as a "partial image") out of all the captured images, and may wish to check medical information (for example, high-quality images, etc.) regarding the selected partial image. . In this case, each time a partial image is selected by the user, the image processing device may acquire medical information regarding the selected partial image using a mathematical model, and present the acquired medical information to the user. However, the process of acquiring medical information using a mathematical model takes time, so it is difficult to shorten the user's work time with a method that acquires and presents medical information every time a partial image is selected. It is. Therefore, it is desirable to be able to appropriately present medical information desired by the user to the user.

A typical object of the present disclosure is to provide an ophthalmological image processing program and an ophthalmic image processing apparatus that can appropriately present medical information desired by the user to the user.

An ophthalmologic image processing program provided by a typical embodiment of the present disclosure is an ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of an eye to be examined, The program is executed by the control unit of the ophthalmological image processing device, and includes an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the eye to be examined photographed by the OCT device; Based on a partial image that is a three-dimensional tomographic image within a part of the area sandwiched between two boundary surfaces of the original tomographic image , the partial image is viewed from a direction along the optical axis of the measurement light of the OCT device. By generating an Enface image when a step of acquiring medical information to be stored; an instruction receiving step of receiving an input of an instruction to select a partial image in the three-dimensional tomographic image acquired in the image acquiring step; and when the instruction to select the partial image is received; A medical information display step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction is executed by the ophthalmological image processing apparatus, and in the medical information acquisition step, the three-dimensional tomographic image is displayed. Among them, high-quality images related to partial images that have not yet been selected by an instruction to select a partial image are obtained in advance before input of an instruction to select the partial image is actually accepted in the instruction receiving step. A pre-acquisition process is executed to store the partial image, and the pre-acquisition process includes the instruction receiving step of accepting an input of an instruction to select a partial image, and the step of receiving an instruction to select a partial image. The medical information display step is executed in parallel with the medical information display step of displaying the high quality image on the display unit.

An ophthalmologic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of an eye to be examined, and a control unit of the ophthalmologic image processing apparatus includes an OCT apparatus. an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye taken by the image acquisition step , and a part of the area sandwiched between two boundary surfaces of the three-dimensional tomographic image acquired in the image acquisition step Based on a partial image that is a three-dimensional tomographic image of A medical information acquisition step of inputting the Enface image as a base image to obtain a high-quality image with improved image quality of the base image and storing it in a storage device; and a partial image in the acquired three-dimensional tomographic image. an instruction receiving step for receiving an input of an instruction to select a partial image; and a medical information display for displaying, on a display unit, a high-quality image acquired for the partial image selected by the instruction when the instruction to select a partial image is received. In the medical information acquisition step, the control unit selects a high-quality image related to a partial image that has not yet been selected according to an instruction to select a partial image from among the three-dimensional tomographic images. A pre-acquisition process is executed to acquire and store the input of the instruction to select the partial image in advance before the input of the instruction to select the partial image is actually accepted in the instruction receiving step, and the pre-acquisition process includes input of the instruction to select the partial image. This step is executed in parallel with the instruction receiving step of accepting an input and the medical information display step of displaying on the display section a high-quality image acquired for the partial image selected by the instruction to select a partial image.

According to the ophthalmologic image processing program and the ophthalmologic image processing apparatus according to the present disclosure, medical information desired by the user is appropriately presented to the user.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image photographing devices 11A and 11B. FIG. 6 is a diagram showing an example of input training data and output training data when outputting a high-quality two-dimensional tomographic image to a mathematical model. 5 is a diagram showing an example of a three-dimensional tomographic image 50 and an Enface image 55 generated based on a partial image included in the three-dimensional tomographic image 50. FIG. 3 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1. FIG. It is a flowchart of ophthalmological image processing performed by the ophthalmological image processing device 21. This is an example of a display screen 60 displayed on the display device 28 during execution of ophthalmological image processing. 6 is a diagram showing an example of a partial image 66 selected by a user and a high-quality image 67 acquired from the partial image 66. FIG. FIG. 3 is a diagram illustrating an example of a setting screen displayed on a display device when a user sets a pre-acquisition pattern. FIG. 6 is an explanatory diagram for explaining an example of a method of setting priorities for a plurality of partial images 66 using a reference image BL as a reference. FIG. 7 is an explanatory diagram for explaining an example of a method of setting priorities for a plurality of partial images 66 along the direction in which partial images 66 selected by the user are switched. It is a flowchart of the preliminary acquisition process which the ophthalmology image processing device 21 performs. It is an example of the display screen 80 displayed on the display device 28 during execution of ophthalmological image processing in the first modification example.

<Summary>
The control unit of the ophthalmologic image processing apparatus exemplified in this disclosure executes an image acquisition step, a medical information acquisition step, an instruction reception step, and a medical information display step. In the image acquisition step, the control unit acquires an ophthalmological image of the tissue of the subject's eye taken by the OCT device. In the medical information acquisition step, the control unit uses a partial image of the acquired ophthalmological images or an image generated based on the partial image as a base image, and uses a mathematical model trained by a machine learning algorithm as a base image. By inputting a base image, a high-quality image with improved image quality of the base image is acquired as medical information and stored in a storage device. In the instruction receiving step, the control unit receives an input of an instruction to select a partial image in the acquired ophthalmological image. In the medical information display step, when an instruction to select a partial image is received, the control section causes the display section to display medical information acquired regarding the partial image selected by the instruction. In the medical information acquisition step, the control unit executes a pre-acquisition process in which medical information regarding a partial image of the ophthalmological image that has not yet been selected according to an instruction is acquired and stored in advance.

In the ophthalmological image processing apparatus exemplified in the present disclosure, medical information (high-quality images) regarding partial images that have not yet been selected by the user is acquired and stored in advance through preliminary acquisition processing. Therefore, if the medical information regarding the actually selected partial image is already stored in the storage device, the control section can display the medical information desired by the user on the display section in a short time. Therefore, the user can appropriately confirm desired medical information.

Various ophthalmological images can be employed as the ophthalmological images processed by the ophthalmological image processing device. For example, the ophthalmological image acquired in the image acquisition step may be a three-dimensional tomographic image including a plurality of two-dimensional tomographic images. In this case, the partial image may be one or more of a plurality of two-dimensional tomographic images included in the three-dimensional tomographic image. The control unit may use a two-dimensional tomographic image, which is a partial image, as a base image, and obtain an image with improved image quality of the base image as a high-quality image.

In addition, when the ophthalmological image is a three-dimensional tomographic image, a partial image is a part of the three-dimensional tomographic image included in the entire image area of the three-dimensional tomographic image (for example, a three-dimensional tomographic image in an area sandwiched between two boundary surfaces). (original tomographic image, etc.). The control unit may use, as the base image, an Enface image (OCT front image) when the three-dimensional tomographic image, which is the partial image, is viewed from a direction (front direction) along the optical axis of the measurement light of the OCT apparatus. The Enface image may be, for example, an integrated image in which pixel values of partial images are integrated in the depth direction (Z direction) at each position in the XY direction intersecting the optical axis. Further, the Enface image may be generated by an integrated value of spectral data of partial images at each position in the XY directions. The control unit may acquire a high-quality image by inputting a base image, which is an Enface image, to a mathematical model. Furthermore, the three-dimensional tomographic image itself, which is a partial image, may be used as the base image. Further, the base image may be an OCT angio image. The OCT angio image may be a two-dimensional front image of the fundus viewed from the front (that is, the line of sight direction of the subject's eye). The OCT angio image may be, for example, a motion contrast image obtained by processing at least two OCT signals obtained at different times regarding the same position.

Further, the ophthalmological image acquired in the image acquisition step may be a moving image including a plurality of still images acquired by intermittently photographing the same tissue multiple times in time series. In this case, the partial image may be one or more of a plurality of still images that make up the moving image. The control unit may use a still image that is a partial image as the base image.

Moreover, the ophthalmological image acquired in the image acquisition step may be a two-dimensional image that is a still image. In this case, the partial image may be an image of a part of the area included in the area of the two-dimensional image. The control unit may use an image of a part of the region, which is a partial image, as the base image.

In the pre-acquisition process, the control unit may sequentially execute the process of acquiring medical information regarding each partial image in order of the priority set for the plurality of partial images in the ophthalmological image. In this case, medical information is sequentially acquired in an appropriate order for each of the plurality of partial images.

In the pre-acquisition process, the control unit may set one or more of the plurality of partial images in the ophthalmological image as a reference image, and set a higher priority for each of the plurality of partial images as it is closer to the reference image. . In this case, medical information is preferentially acquired from partial images close to the reference image. Therefore, by appropriately setting the reference image, medical information can be sequentially acquired in a more appropriate order. Therefore, the user's working time can be further reduced.

Note that various types of "closeness" can be adopted as the "closeness" of a partial image to a reference image, which is a reference when setting priorities. For example, when a two-dimensional tomographic image constituting a three-dimensional tomographic image is used as a partial image, "closeness" of the distance to the two-dimensional tomographic image used as a reference image may be used as a reference.

In addition, when a part of the 3D tomographic image included in the image area of the 3D tomographic image is used as a partial image, the "closeness" of the distance to the 3D tomographic image used as the reference image, the "closeness" of the range, At least one of the "closeness" of the overlap ratio, etc. may be used as a criterion. Furthermore, when a three-dimensional tomographic image in a region sandwiched between two boundary surfaces is used as a partial image, the "closeness" of the distance of the boundary surface of the partial image to the boundary surface of the reference image may be used as a reference. .

Furthermore, when a still image forming a moving image is used as a partial image, "closeness" of the shooting time to the still image used as the reference image may be used as a reference. Furthermore, when an image of a part of a region within a two-dimensional image is used as a partial image, "closeness" of at least one of distance, range, and overlap ratio with respect to a reference image may be used as a reference.

In the pre-acquisition process, the control unit may acquire a specific position of the tissue appearing in the ophthalmological image, and may use a partial image at the acquired specific position as the reference image. In this case, medical information is preferentially acquired from a specific location in the tissue. Therefore, medical information at a specific location can be appropriately grasped by the user.

Note that the method by which the control unit acquires the specific position can be selected as appropriate. For example, the control unit may detect the specific position of the tissue by performing well-known image processing on the ophthalmological image. Further, the control unit may acquire the specific position by inputting the acquired ophthalmologic image to a mathematical model that inputs the ophthalmologic image and outputs the specific position.

Further, the type of specific position to be acquired by the control unit can also be selected as appropriate. For example, when the ophthalmological image is a fundus image of the subject's eye, the specific position may be at least one of the fovea, optic disc, lesion site, fundus blood vessels, etc. in the fundus. Furthermore, when the fundus image is a three-dimensional tomographic image of the fundus, the specific position may be the position of a specific layer in the fundus.

Moreover, the specific position may be acquired by using various ophthalmological examination devices. As an example, the specific position may be a position determined by a perimeter, which is a type of ophthalmological examination device (for example, a position on the fundus of the eye with low sensitivity). In this case, the control unit may acquire the specific position based on the image of the inspection result obtained by the perimeter.

However, the reference image may be set regardless of the specific position. For example, the reference image may be a partial image that includes the center of the image area of the ophthalmological image.

When the instruction to select the partial image is received, the control unit may set the partial image selected by the instruction as the reference image. When a user selects a partial image, the user often wants to also check medical information regarding a partial image close to the selected partial image. By setting the partial image selected by the user as the reference image, the control unit can acquire and store medical information about a partial image close to the selected partial image through preliminary acquisition processing. As a result, medical information of a partial image close to the selected partial image is displayed on the display section in a short time. Therefore, the user can confirm desired medical information more appropriately.

Note that, if the medical information regarding the partial image selected by the instruction has not been acquired yet, the control unit acquires the medical information regarding the selected partial image with the highest priority in the medical information acquisition step, and then performs the medical information display step. It may also be displayed on the display section. Furthermore, if the medical information regarding the partial image selected by the instruction has been acquired in advance through preliminary acquisition processing, the control unit causes the already acquired medical information to be displayed on the display unit in the medical information display step. It's okay. As a result, the medical information desired by the user is smoothly displayed on the display unit.

When a partial image selected by a user's instruction is changed, the control unit sets priorities for the plurality of partial images along the direction of the partial image after the change with respect to the partial image before the change. They may be set in order. A user often switches partial images sequentially along the same direction. The control unit acquires in advance the medical information of the partial images that the user is likely to select later by setting priorities for the plurality of partial images in order along the direction in which the partial images are switched. It can be stored in memory. Therefore, the user can confirm desired medical information more appropriately.

The control unit may cause the storage device to store the position of the reference image in the ophthalmologic image when the partial image selected by the user is set as the reference image. When processing an ophthalmological image in which the position of the reference image is stored in the past and an ophthalmological image in which the photographing subject is the same, the control unit sets a partial image of the ophthalmological image at the position stored in the past as the reference image. You may. When performing a diagnosis or the like by taking ophthalmological images of the same eye tissue at different times, the user often confirms medical information at the same position in the ophthalmological images. By setting a partial image of a position stored in the past as a reference image, the control unit can efficiently acquire in advance medical information of a position that is likely to be checked by the user.

When the reference image is newly set, the control unit may suspend the pre-acquisition process that was being executed and restart the pre-acquisition process in accordance with the priority order based on the newly set reference image. In this case, compared to the case where pre-acquisition processing based on a newly set reference image is executed after the previous pre-acquisition processing is completed, the medical information of the partial image close to the newly set reference image will be more accurate. Retrieved in a short time. Therefore, the user can confirm desired medical information more appropriately.

Note that the control unit may restart the interrupted preliminary acquisition process when the preliminary acquisition process based on the newly set reference image is completed. Furthermore, when the control unit sequentially executes the pre-acquisition process for each of the plurality of partial images in accordance with the priority order, the control unit may omit the pre-acquisition process for the partial images for which medical information has already been acquired. In this case, medical information regarding each of the plurality of partial images can be acquired more efficiently.

Among a plurality of partial images in an ophthalmological image, a pattern of partial images from which medical information is acquired in advance may be provided in advance. In the pre-acquisition process, the control unit may execute a process of acquiring medical information regarding the partial image corresponding to the pattern. In this case, medical information regarding the partial image corresponding to the pattern is obtained in advance. Therefore, the user can appropriately grasp the medical information corresponding to the pattern.

The specific form of the pattern can be selected as appropriate. For example, when two-dimensional tomographic images constituting a three-dimensional tomographic image of the fundus are used as partial images, the arrangement of the plurality of partial images when the fundus is viewed from the front may be cross-shaped, radial, or The pattern may be provided in a concentric circular pattern, a raster (parallel) pattern, or a combination thereof.

In addition, when a three-dimensional tomographic image within a region sandwiched between two boundary surfaces in a three-dimensional image of the fundus is used as a partial image, a pattern of the partial image in which the position of at least one of the two boundary surfaces is set in advance. (For example, a pattern in which the layer between the two interfaces is at least one of the retinal surface layer, the deep retinal layer, the vitreous body, the outer retinal layer, the ORCC, the choriocapillaris, the choroid, etc.) may be provided. In particular, when a three-dimensional image within a region between two boundary surfaces is used as a partial image, there are many (in some cases infinite) combinations of the two boundary surfaces. Therefore, it is extremely difficult to obtain medical information for all partial images included in a three-dimensional image in advance before displaying the medical information. In contrast, in the ophthalmological image processing apparatus of the present disclosure, medical information is acquired by pre-acquisition processing with respect to partial images corresponding to one or more patterns provided in advance. Therefore, medical information useful to the user is presented to the user more efficiently.

Further, when the still images forming the moving image are taken as partial images, a pattern of a plurality of partial images may be provided in which the time intervals of the images are set at fixed time intervals.

The control unit may acquire medical information corresponding to a pattern selected by the user from among a plurality of predetermined patterns through a pre-acquisition process.

Further, the control unit may automatically acquire medical information corresponding to a default pattern (for example, a two-dimensional tomographic image passing through the center of a three-dimensional tomographic image) through a pre-acquisition process. In this case, when starting to display medical information about the acquired ophthalmological image, the control unit may first display medical information corresponding to the default pattern on the display unit. In this case, by setting a partial image that is likely to be selected by the user as the default pattern, user convenience is further improved. Further, the control unit may perform a pre-acquisition process of medical information corresponding to the pattern before displaying a display screen for displaying medical information about the acquired ophthalmological image on the display unit. In this case, when displaying the display screen on the display unit, the medical information corresponding to the pattern is displayed smoothly.

The device that executes the image acquisition step, medical information acquisition step, instruction reception step, and medical information display step can be selected as appropriate. For example, a control unit of a personal computer (hereinafter referred to as "PC") may execute all of the image acquisition step, medical information acquisition step, instruction reception step, and medical information display step. In other words, the control unit of the PC may acquire an ophthalmologic image from the OCT device and acquire medical information based on the acquired ophthalmologic image. Further, the control unit of the OCT apparatus may execute all of the image acquisition step, medical information acquisition step, instruction reception step, and medical information display step. Further, control units of a plurality of devices (for example, an OCT apparatus, a PC, etc.) may cooperate to execute the image acquisition step, the medical information acquisition step, the instruction reception step, and the medical information display step.

Note that medical information obtained from a base image by using a mathematical model is not limited to a high-quality image obtained by improving the image quality of the base image. For example, the mathematical model may output an analysis result of the structure of the tissue of the eye to be examined by inputting the base image. Specifically, the medical information may be analysis results (segmentation results) of tissue layers and boundaries. Further, the medical information may be an analysis result regarding a disease of the eye to be examined.

Furthermore, ophthalmological images are not limited to ophthalmological images taken by an OCT device. For example, at least some of the techniques exemplified in the present disclosure can be applied to ophthalmological images taken by a laser scanning ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, a corneal endothelial imaging device (CEM), etc. .

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. A program for realizing the constructed mathematical model is stored in the storage device 24 of the ophthalmological image processing device 21. The ophthalmological image processing device 21 acquires medical information by inputting an image (base image) into a mathematical model. As an example, in this embodiment, a high-quality image in which the image quality of the base image is improved (for example, the influence of noise on the base image is suppressed) is acquired as medical information. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images that are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although the details will be described later, the mathematical model construction device 1 uses images acquired from the ophthalmological image capturing device 11A (hereinafter referred to as "ophthalmic images for training") and medical information corresponding to the ophthalmological images for training to perform mathematical calculations. Build a mathematical model by training the model. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic image capturing device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmological image capturing apparatus 11A) may cooperate to construct the mathematical model.

Further, a PC is used as the ophthalmological image processing apparatus 21 of this embodiment. However, the device that can function as the ophthalmological image processing device 21 is not limited to a PC either. For example, the ophthalmologic image capturing device 11B, a server, or the like may function as the ophthalmologic image processing device 21. When the ophthalmologic image capturing device 11B (in this embodiment, the OCT device) functions as the ophthalmologic image processing device 21, the ophthalmologic image capturing device 11B captures the ophthalmologic image while appropriately controlling the partial images included in the captured ophthalmologic image. can obtain medical information. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmological image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmological image capturing apparatus 11B) may cooperate to perform various processes.

Further, in this embodiment, a case will be described in which a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be used as the controller to speed up the processing.

The mathematical model construction device 1 will be explained. The mathematical model construction device 1 is placed, for example, at a manufacturer that provides the ophthalmologic image processing device 21 or the ophthalmologic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller that controls, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a mathematical model building process (see FIG. 4), which will be described later. Further, the communication I/F 5 connects the mathematical model construction device 1 to other devices (for example, the ophthalmologic image capturing device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8. The operation unit 7 is operated by the user in order for the user to input various instructions to the mathematical model construction device 1. As the operation unit 7, for example, at least one of a keyboard, a mouse, a touch panel, etc. can be used. Note that a microphone or the like for inputting various instructions may be used together with or in place of the operation section 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that "image" in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as "ophthalmologic image") from the ophthalmologic image capturing device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic image capturing device 11A through at least one of wired communication, wireless communication, a removable storage medium (for example, a USB memory), and the like.

The ophthalmologic image processing device 21 will be explained. The ophthalmological image processing device 21 is placed, for example, in a facility (for example, a hospital or a medical examination facility) that performs diagnosis or examination of subjects. The ophthalmologic image processing device 21 includes a control unit 22 that performs various control processes, and a communication I/F 25. The control unit 22 includes a CPU 23 that is a controller that controls control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing ophthalmologic image processing (see FIG. 5), which will be described later. The ophthalmologic image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 1. The communication I/F 25 connects the ophthalmologic image processing device 21 to other devices (for example, the ophthalmologic image capturing device 11B and the mathematical model construction device 1).

The ophthalmological image processing device 21 is connected to an operation section 27 and a display device 28. As with the operation section 7 and display device 8 described above, various devices can be used for the operation section 27 and the display device 28.

The ophthalmologic image processing device 21 can acquire ophthalmologic images from the ophthalmologic image capturing device 11B. The ophthalmological image processing device 21 may acquire ophthalmological images from the ophthalmological image capturing device 11B, for example, by wired communication, wireless communication, a removable storage medium (for example, a USB memory), or the like. Further, the ophthalmological image processing device 21 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

The ophthalmological image capturing devices 11A and 11B will be explained. As an example, in this embodiment, a case will be described in which an ophthalmologic image capturing device 11A that provides ophthalmologic images to the mathematical model construction device 1 and an ophthalmologic image capturing device 11B that provides ophthalmologic images to the ophthalmologic image processing device 21 are used. . However, the number of ophthalmological imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmology image processing device 21 may acquire ophthalmology images from a plurality of ophthalmology image capturing devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic image capturing device.

Moreover, in this embodiment, an OCT device is illustrated as the ophthalmologic image capturing device 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

The ophthalmologic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic image capturing section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that controls control, and a storage device 14 (14A, 14B) that can store programs, data, and the like. When the ophthalmological image capturing device 11 executes at least a part of the ophthalmological image processing (see FIG. 5) to be described later, at least a part of the ophthalmological image processing program for executing the ophthalmological image processing is stored in the storage device 14. Needless to say.

The ophthalmological image photographing unit 16 includes various configurations necessary for photographing an ophthalmological image of the eye to be examined. The ophthalmologic image capturing section 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into a measurement light and a reference light, a scanning section for scanning the measurement light, and a scanning section for scanning the measurement light. It includes an optical system for irradiating the eye to be examined, a light receiving element that receives the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic image capturing device 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the eye to be examined. Specifically, the CPU 13 scans the scan line with OCT light (measuring light) to capture a two-dimensional tomographic image (see FIG. 2, etc.) of a cross section intersecting the scan line. Further, the CPU 13 can capture a three-dimensional tomographic image of the tissue (see FIG. 3, etc.) by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by scanning measurement light on each of a plurality of scan lines at different positions within a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 obtains a three-dimensional tomographic image by combining the plurality of captured two-dimensional tomographic images.

Furthermore, the CPU 13 is also capable of capturing a plurality of ophthalmological images of the same site by scanning the measurement light multiple times over the same site on the tissue (in this embodiment, on the same scan line). The CPU 13 can obtain an average image in which the influence of speckle noise is suppressed by performing averaging processing on a plurality of ophthalmological images of the same site. By performing averaging processing on a plurality of two-dimensional tomographic images of the same region, the image quality of the two-dimensional tomographic images can be improved. Furthermore, it is also possible to improve the image quality of the three-dimensional tomographic image by averaging each of the plurality of two-dimensional tomographic images that constitute the three-dimensional tomographic image. The averaging process may be performed, for example, by averaging pixel values of pixels at the same position among a plurality of ophthalmological images. The larger the number of images subjected to the averaging process, the more likely it is to suppress the influence of speckle noise, but the longer the imaging time will be. Note that while the ophthalmologic image capturing apparatus 11 captures a plurality of ophthalmologic images of the same site, it executes a tracking process that causes the scanning position of the OCT light to follow the movement of the subject's eye.

(mathematical model construction process)
The mathematical model construction process executed by the mathematical model construction device 1 will be described with reference to FIGS. 2 to 4. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

In the mathematical model construction process, a mathematical model is trained using a training dataset to construct a mathematical model that outputs medical information based on ophthalmological images. The training data set includes input side data (input training data) and output side data (output training data). The mathematical model can output various medical information. The type of training data set used for training the mathematical model is determined depending on the type of medical information to be output to the mathematical model. An example of the relationship between the type of medical information output to the mathematical model and the training data set used for training the mathematical model will be described below.

First, a case will be described in which a two-dimensional tomographic image is input into a mathematical model as a base image, and a two-dimensional tomographic image (high-quality image) with improved image quality of the base image is output to the mathematical model as medical information. In this case, in this embodiment, a two-dimensional tomographic image of the tissue of the eye to be examined is used as input training data, and a two-dimensional tomographic image of the same region with higher image quality than the input training data is used as output training data. A mathematical model is trained. Note that the high-quality image refers to, for example, at least one of an image in which the noise of the input base image is reduced, an image in which the resolution of the original image is increased, an image in which the visibility of the original image is improved, and the like.

FIG. 2 shows an example of input training data and output training data when outputting a high-quality two-dimensional tomographic image to a mathematical model. In the example shown in FIG. 2, the CPU 3 acquires a set 40 of a plurality of two-dimensional tomographic images 400A to 400X taken of the same part of the tissue. The CPU 3 uses a portion of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 (the number of images is smaller than the number used for averaging the output training data to be described later) as input training data. Further, the CPU 3 acquires an average image 41 of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 as training data for output. When a mathematical model is trained using the input training data and output training data illustrated in Figure 2, the influence of speckle noise is suppressed by inputting the two-dimensional tomographic image as the base image to the trained mathematical model. A high-quality image is output.

Next, a case will be described in which an Enface image is input to a mathematical model as a base image, and an Enface image (high quality image) with improved image quality of the base image is outputted to the mathematical model. In this case, in this embodiment, the mathematical model is trained by using an Enface image of the tissue of the eye to be examined as input training data, and an Enface image of the same region with higher image quality than the input training data as output training data. be done.

An example of the Enface image 55 will be described with reference to FIG. 3. The Enface image 55 is a part of the three-dimensional tomographic image (partial image) included in the entire image area of the three-dimensional tomographic image 50 in a direction along the optical axis of the measurement light of the ophthalmologic imaging device (OCT device) 11 (front view). This is a two-dimensional image when viewed from the direction). The Enface image 55 may be, for example, an integrated image in which pixel values of partial images are integrated in the depth direction (Z direction) at each position in the XY direction intersecting the optical axis. Further, the Enface image 55 may be generated by an integrated value of spectral data of partial images at each position in the X and Y directions. In this embodiment, in the image area of the three-dimensional tomographic image 50, a three-dimensional tomographic image sandwiched between two boundary surfaces (slabs) 52A and 52B is taken as a partial image, and an Enface image 55 of the partial image is generated. . In the example shown in FIG. 3, two boundary surfaces 52A and 52B are formed on one two-dimensional tomographic image 51 (for example, a two-dimensional tomographic image passing through the center of the three-dimensional tomographic image 50) constituting the three-dimensional tomographic image 50. Locations are illustrated. When the positions of the boundary surfaces 52A and 52B are changed, the partial images and the Enface image 55 generated from the partial images are also changed.

An example of input training data and output training data when outputting a high-quality Enface image 55 to a mathematical model will be described. The CPU 3 acquires a plurality of three-dimensional tomographic images taken of the same tissue (in this embodiment, the fundus of the same eye to be examined) by the fundus image photographing device 11A. The plurality of three-dimensional tomographic images include a low-quality three-dimensional tomographic image and a high-quality three-dimensional tomographic image. As described above, a high-quality three-dimensional tomographic image may be obtained by averaging each of the plurality of two-dimensional tomographic images forming the image. Next, the CPU 3 extracts some three-dimensional tomographic images as partial images from the low-quality three-dimensional tomographic images, and uses the Enface image generated from the extracted partial images as input training data. The CPU 3 extracts a three-dimensional tomographic image in the same range as the input training data as a partial image from the high-quality three-dimensional tomographic image, and uses the Enface image generated from the extracted partial image as the output training data. When a mathematical model is trained using input training data and output training data, by inputting the Enface image as a base image to the trained mathematical model, a high-quality image with improved image quality of the base image is output. Ru.

Note that it is also possible to change the method of generating high-quality two-dimensional tomographic images or three-dimensional tomographic images. For example, image quality may be improved by processing other than averaging processing. Furthermore, the input training data is not limited to tomographic images of tissues.

The mathematical model may also be trained to output medical information other than high quality images. For example, by inputting an ophthalmological image into a mathematical model, it is also possible to cause the mathematical model to output an analysis result of the structure of the tissue of the eye to be examined as medical information. In this case, a mathematical model is trained using an ophthalmological image of the tissue of the eye to be examined as input training data, and medical data showing the structure of the tissue shown in the input training data (ophthalmological image) as output training data. Ru. As an example, a two-dimensional tomographic image of fundus tissue is used as input training data, and data indicating the position of the tissue structure (for example, the position of at least one of the layers and boundaries of the tissue) shown in the input training data is output. A mathematical model may be trained as training data. In this case, when a two-dimensional tomographic image is input to the trained mathematical model, an analysis result of the tissue structure (for example, the position of at least one of layers and boundaries) is output. Furthermore, it is also possible to output the analysis results of at least one of the fundus blood vessels, fovea, optic disc, etc. of the eye to be examined to the mathematical model. The input training data does not need to be a two-dimensional tomographic image, and may be a two-dimensional frontal image or the like.

Furthermore, by inputting an ophthalmological image into the mathematical model, it is also possible to cause the mathematical model to output an automatic analysis result indicating whether or not the subject's eye has some kind of disease as medical information. In this case, an ophthalmological image of the tissue of the eye to be examined (for example, at least one of a two-dimensional tomographic image, a three-dimensional tomographic image, an Enface image, etc.) is used as the input training data, and the imaging target shown in the input training data A mathematical model may be trained using data indicating at least one of the presence or absence of a disease and the type of disease as output training data.

The mathematical model construction process will be described with reference to FIG. 4. The CPU 3 acquires at least a portion of the ophthalmological image photographed by the ophthalmological image photographing device 11A as input training data (S1). In this embodiment, the data of the ophthalmologic image is generated by the ophthalmologic image capturing device 11A, and then acquired by the mathematical model construction device 1. However, the CPU 3 acquires the data of the ophthalmological image by acquiring a signal (for example, an OCT signal) that is the basis for generating the ophthalmological image from the ophthalmological image capturing device 11A, and generating the ophthalmological image based on the acquired signal. It's okay.

Next, the CPU 3 acquires output training data corresponding to the input training data acquired in S1 (S3). An example of the correspondence between input training data and output training data is as described above.

Next, the CPU 3 executes training of a mathematical model using the training data set using a machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

Neural networks are a method that imitates the behavior of biological nerve cell networks. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and stochastic neural networks. There are various types of neural networks (Boltzmann machines, Bassian networks, etc.).

Random forest is a method of generating a large number of decision trees by performing learning based on randomly sampled training data. When using random forest, branches of multiple decision trees trained in advance as a classifier are followed, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. For example, the SVM learns the parameters of a linear input element based on the criterion (hyperplane separation theorem) of finding a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by being trained using a training dataset. As described above, the training data set is a set of input training data and output training data. For example, training updates correlation data (eg, weights) between each input and output.

In this embodiment, a multilayer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used. However, other machine learning algorithms may be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as the machine learning algorithm.

The processes of S1 to S3 are repeated until construction of the mathematical model is completed (S5: NO). When construction of the mathematical model is completed (S5: YES), the mathematical model construction process ends. A program and data for realizing the constructed mathematical model are incorporated into the ophthalmological image processing device 21.

(Ophthalmology image processing)
An example of ophthalmological image processing performed by the ophthalmological image processing device 21 will be described with reference to FIGS. 5 to 11. 5 to 11 illustrate cases in which a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image of the fundus are used as partial images, and high-quality images with improved image quality of each partial image are acquired and displayed. . The ophthalmological image processing illustrated in FIG. 5 is executed by the CPU 23 according to the ophthalmological image processing program stored in the storage device 24.

As shown in FIG. 5, the CPU 23 acquires an ophthalmologic image of the tissue of the eye to be examined, which is captured by the ophthalmologic image capturing device (OCT device in this embodiment) 11B (S11). In S11 of this embodiment, a three-dimensional tomographic image (for example, see the three-dimensional tomographic image 50 shown in FIG. 3) of the fundus tissue of the eye to be examined is acquired. As described above, a three-dimensional tomographic image is generated by combining (arranging) a plurality of two-dimensional tomographic images.

Next, the CPU 23 displays the partial image selection image 61 on the display device 28 (S12). FIG. 6 is an example of a display screen 60 displayed on the display device 28 during execution of ophthalmological image processing. On the display screen 60, a partial image selection image 61 and a selected image display field 65 are displayed.

The partial image selection image 61 is displayed for the user to select at least one of the plurality of partial images 66 (two-dimensional tomographic images in this embodiment) included in the three-dimensional tomographic image. In the example shown in FIG. 6, the partial image selection image 61 includes a two-dimensional front image when a three-dimensional tomographic image is viewed from the optical axis direction of the OCT measurement light. A selection range 62 from which a partial image 66 can be selected is displayed within the image area of the two-dimensional front image. A selection line 63 is displayed within the selection range 62, through which the selected partial image 66 passes. The user inputs an instruction to select the partial image 66 to the ophthalmological image processing device 21 by operating the operation unit 27 (for example, a mouse) and setting the position of the selection line 63 to a desired position (for example, moving it). can do. The CPU 23 sets the partial image 66 at the position through which the selection line 63 passes as the partial image 66 selected by the user.

The selected image display field 65 displays at least one of a partial image 66 selected by the user and a high-quality image 67 (see FIG. 7) regarding the selected partial image 66. In the example shown in FIG. 6, the partial image 66 itself is displayed because inference processing is in progress to obtain a high-quality image 67 regarding the partial image 66. When the acquisition of the high-quality image 67 is completed, the high-quality image 67 is displayed instead of the partial image 66. In addition, if a high-quality image 67 related to the selected partial image 66 has already been acquired by a pre-acquisition process (see FIG. 11) described later, the selected image display field 65 will display the selected partial image 66. At this point, a high quality image 67 for the selected partial image 66 is displayed. Therefore, the user's working time is appropriately shortened.

Further, the display screen 60 (in this embodiment, the selected image display column 65 within the display screen 60) includes partial image switching buttons 68F and 68B. The partial image switching button 68F causes the partial image 66 to be selected to be adjacent to the currently selected partial image 66 in the forward direction (for example, in the example shown in FIG. 6, toward the bottom of the screen in the selection range 62). The partial image 66 is operated to change (switch) to the partial image 66 shown in FIG. The partial image switching button 68B also allows the partial image 66 to be selected to be adjacent to the partial image 66 selected at that time in the opposite direction (for example, in the example shown in FIG. 6, toward the top of the screen in the selection range 62). The partial image 66 is operated to change to the partial image 66 shown in FIG. The user can sequentially switch the partial images 66 in a predetermined direction by continuously operating the partial image switching button 68F or the partial image switching button 68B.

Furthermore, an image type display section 69 is displayed on the display screen 60. The image type display section 69 displays whether the image currently displayed in the selected image display field 65 is a partial image 66 selected by the user or a high-quality image 67 related to the partial image 66. be done. Therefore, the user can easily understand whether the image displayed in the selected image display field 65 is a high-quality image 67 or not.

In the present embodiment, the selection range 62 in the partial image selection image 61 includes two thickness distributions of a specific layer in the fundus tissue (in the example shown in FIG. 6, the layer between the ILM and RPE/BM). A dimensional thickness map is displayed in a superimposed manner. Therefore, the user can select the partial image 66 after considering the thickness distribution of a specific layer in the fundus.

Further, when starting the display on the display screen 60 in S12, the CPU 23 generates a high-quality image 67 regarding the partial image 66 of the default pattern (for example, the partial image 66 passing through the center of the image area of the two-dimensional front image). , the selected image is automatically displayed in the selected image display field 65 first. Here, before displaying the display screen 60, the CPU 23 inputs the partial image 66 of the default pattern into the mathematical model to obtain a high-quality image 67 regarding the partial image 66 of the default pattern in advance. put. The process of pre-obtaining the high-quality image 67 corresponding to the default pattern is also an example of the pre-obtaining process. When displaying the display screen 60, the CPU 23 causes the high-quality image 67 corresponding to the default pattern to be displayed in the selected image display column 65 from the beginning. Therefore, the high quality image 67 corresponding to the default pattern is displayed smoothly.

Returning to the explanation of FIG. 5. The CPU 23 determines whether a pre-acquisition pattern has been set by the user (S13). FIG. 8 shows an example of a setting screen displayed on the display device 28 when the user sets a pre-acquisition pattern. A pre-acquisition pattern is a pattern of a partial image 66 in which medical information (high-quality image 67 in this embodiment) is acquired in advance among a plurality of partial images 66 in an ophthalmological image (in this embodiment, a three-dimensional tomographic image). It is.

As shown in FIG. 8, when the user inputs an instruction to set a pre-obtained pattern, the CPU 23 displays a pattern selection field 70, a position designation field 71, an angle designation field 72, and a size designation field 73 on the display device 28. to be displayed. The pattern selection column 70 displays a plurality of types of pre-obtained patterns that differ from each other in at least one of the arrangement, number, and shape of lines through which the partial image 66 passes. The user can arbitrarily select one or more of the plurality of pre-acquired patterns.

In the position specification field 71, the position on the partial image selection image 61 (see FIG. 6) where the selected pre-acquisition pattern is to be set is specified. For example, when "macula center" is specified, the position of the pre-acquired pattern is set so that the center of the selected pre-acquired pattern matches the macula (for example, the fovea, which is the center of the macula). The positions of the macula, papilla, abnormal region, etc. in the ophthalmological image may be detected by performing image processing on the ophthalmological image. Furthermore, the position of the macula or the like may be acquired by using a mathematical model trained by a machine learning algorithm. When "manual setting" is specified, the selected pre-obtained pattern is set at the position specified by the user's instruction. In the angle specification column 72, the angle of the selected pre-obtained pattern on the ophthalmological image is specified. In the size specification field 90, the size of the selected pre-obtained pattern on the ophthalmological image is set.

Returning to the explanation of FIG. 5. If the pre-acquisition pattern is set (S13: YES), the CPU 23 inputs the partial image 66 corresponding to the pre-acquisition pattern that has been set into the mathematical model, thereby improving the image quality of the input partial image 66. The improved high-quality image 67 is acquired as medical information. The process of acquiring the high-quality image 67 corresponding to the pre-acquisition pattern is an example of the pre-acquisition process. The CPU 23 causes the acquired high-quality image 67 to be stored in the storage device 24 and displayed on the display device 28 (S14). Therefore, the user can appropriately confirm the high-quality image 67 corresponding to the desired pattern.

Note that the high-quality image 67 corresponding to the pre-obtained pattern may be previously executed before the process (S12) for starting display of the display screen 60. In this case, in S12, the high-quality image 67 corresponding to the pre-acquired pattern may be displayed in the selected image display field 65 from the beginning. As a result, the high-quality image 67 corresponding to the pre-acquired pattern is displayed more smoothly on the display device 28.

Next, the CPU 23 determines whether the follow-up position information is stored in the storage device 24 (S16). Follow-up position information refers to the position of the partial image 66 selected by the user (in other words, the position of the reference image to be described later) when ophthalmological image processing was performed on ophthalmological images of the same photographic subject in the past. This is the information shown. If the follow-up position information is stored (S16: YES), the CPU 23 sets the partial image 66 at the follow-up position as the reference image. The reference image will be described later. Note that if a plurality of follow-up positions are stored, any one of the follow-up positions (for example, the position of the partial image 66 first selected by the user in past processing) may be adopted. Furthermore, when follow-up position information is stored, the partial image 66 at the follow-up position may be set as the default partial image 66 that causes the high-quality image 67 to be displayed on the display screen 60 first.

If the follow-up position information is not stored (S16: NO), the CPU 23 acquires the specific position of the tissue shown in the ophthalmological image (S18). The specific position may be, for example, at least one of the macula, optic disc, lesion site, fundus blood vessels, etc. in the fundus of the eye. For example, the CPU 23 may detect the specific position by performing image processing on the ophthalmological image. Further, the CPU 23 may obtain the specific position using a mathematical model trained by a machine learning algorithm. The CPU 23 sets the partial image 66 at the specific position as a reference image to be described later (S19). However, the specific position may be determined in advance. For example, the position of the partial image 66 of the default pattern described above may be set as the specific position.

Next, the CPU 23 sets a priority order for each of the plurality of partial images 66 included in the ophthalmological image, based on the reference image, when performing a pre-acquisition process (see FIG. 9) to be described later (S21). The priority indicates the order in which the preliminary acquisition process is executed for each of the plurality of partial images 66. That is, the process of acquiring medical information (high quality image 66 in this embodiment) for each partial image 66 is performed in order according to the priority order. The reference image is a partial image 66 that serves as a reference for setting priorities for a plurality of partial images 66.

In S21 of this embodiment, the CPU 23 sets the priority order of the plurality of partial images 66 such that the closer the image is to the reference image, the higher the priority order. Specifically, in S21 of this embodiment, the CPU 23 sets the priority order of the plurality of partial images 66 such that the closer the distance to the reference image, the higher the priority order.

FIG. 9 is an explanatory diagram for explaining an example of a method of setting priorities for a plurality of partial images 66 using the reference image BL as a reference. FIG. 9 shows the line through which each of the plurality of partial images 66 passes, viewed from the direction along the optical axis of the OCT measurement light. As shown in FIG. 9, in S21 of the present embodiment, the CPU 23 prioritizes two partial images 66 that are adjacent to the reference image BL in the forward direction (lower direction in the figure) and reverse direction (upper direction in the figure). The ranking is set to P1, which is the highest. Further, the CPU 23 sets the priority order in both the forward direction and the reverse direction such that the greater the distance from the reference image BL, the lower the priority order is in the order of P2 and P3. As a result, medical information regarding the partial image 66 close to the reference image BL is preferentially acquired.

Returning to the explanation of FIG. 5. The CPU 23 starts pre-acquisition processing (S22). In the preliminary acquisition process, medical information regarding the partial image 66 that has not yet been selected by the user is acquired in advance and stored. Details of the advance acquisition process will be described later with reference to FIG. 11.

Next, the CPU 23 determines whether the user has input an instruction to select the partial image 66 (S24). If no instruction has been input (S24: NO), the process directly advances to S29. When the CPU 23 receives the input of the instruction to select the partial image 66 (S24: YES), the CPU 23 sets the selected partial image 66 as a new reference image (S25). The CPU 23 stores information on the position of the newly set reference image in the storage device 24 as follow-up position information to be used when performing ophthalmologic image processing on an ophthalmologic image of the same eye to be examined in the future. S26). The CPU 23 resets the priority order of the plurality of partial images 66 in the order of proximity to the newly set reference image (S27).

In S27, when the partial image 66 selected by the user is changed, the CPU 23 selects a plurality of partial images 66 along the direction of the partial image 66 after being changed relative to the partial image 66 before being changed. Priority can be set in order. For example, when the partial image 66 to be selected is switched by operating the partial image switching button 66F or the partial image switching button 66B (see FIG. 6), the user can switch the partial images 66 in the same direction (forward or reverse direction). In many cases, the partial images 66 to be selected are successively switched along. Therefore, when the partial images 66 are switched by the partial image switching button 66F or the partial image switching button 66B, the CPU 23 of this embodiment resets the priority order along the direction in which the partial images 66 are switched.

FIG. 10 is an explanatory diagram for explaining an example of a method for setting priorities for a plurality of partial images 66 along the direction in which the partial images 66 selected by the user are switched. In the example shown in FIG. 10, the partial image 66 selected by the user is switched from BL1 to BL2. In this case, the CPU 23 changes the reference image from BL1 to BL2 (S25), and also changes the direction of the partial image 66 (BL2) after being changed with respect to the partial image 66 (BL1) before being changed (as shown in FIG. In the example shown, the priority order is reset in the forward direction (downward in the drawing) (S27). That is, regarding the changed direction, the priority order is set such that the greater the distance from the reference image BL2, the lower the priority order is in the order of P1, P2, and P3.

Next, the CPU 23 causes the display device 28 (in the present embodiment, the selected image display field 65 shown in FIG. 6) to display medical information regarding the partial image 66 selected in S24 (in the present embodiment, the high-quality image 67). (S28). Specifically, if medical information regarding the selected partial image 66 has not been acquired yet, the CPU 23 inputs the selected partial image 66 into a mathematical model to obtain medical information regarding the input partial image 66. Obtain and display. Note that, as described above, the selected partial image 66 is displayed in the selected image display field 65 while the process of acquiring medical information is being executed. If the medical information regarding the selected partial image 66 has already been acquired and stored in the storage device 24, the CPU 23 causes the display device 24 to display the stored medical information.

The processes of S24 to S29 are repeated until an instruction to end the process is input (S29: NO). When the termination instruction is input (S29: YES), the ophthalmological image processing is terminated.

An example of the pre-acquisition process executed by the ophthalmological image processing device 21 will be described with reference to FIG. 11. As described above, in the pre-acquisition process, medical information regarding the partial image 66 that has not yet been selected by the user is obtained and stored in advance. The medical information acquisition process regarding each partial image 66 is executed in order according to the set priorities. The preliminary acquisition process illustrated in FIG. 11 is executed by the CPU 23 according to the ophthalmological image processing program stored in the storage device 24. The pre-acquisition process is executed in the background in parallel with the ophthalmological image processing illustrated in FIG.

First, the CPU 23 sets the value of the counter N for counting the priority order to "1" (S31). The CPU 23 determines whether medical information regarding the partial image 66 with the Nth priority has already been acquired (S32). If the medical information regarding the Nth partial image 66 is already stored in the storage device 24 (S32: YES), the value of the counter N is set to "1" in order to avoid duplication of processing and shorten the processing time. ” is added (S33). The process returns to S32. That is, if the medical information regarding the Nth partial image 66 has already been acquired, the process of acquiring the medical information is omitted.

If the medical information regarding the N-th partial image 66 has not yet been acquired (S32: NO), the CPU 23 uses the N-th partial image 66 as a base image based on the mathematical model trained by the machine learning algorithm. Enter image. As a result, inference processing using a mathematical model is started to output a high-quality image 67 (see FIG. 7), which is an improved image quality of the base image, as medical information (S34).

In the ophthalmological image processing (see FIG. 5), the CPU 23 determines whether a new reference image has been set (S35). If the reference image has not been newly set (S35: NO), it is determined whether the medical information acquisition process regarding the Nth partial image 66 (that is, the inference process started in S34) has been completed ( S36). If the medical information acquisition process is not completed (S36: NO), the process returns to S35, and the processes of S35 and S36 are repeated.

When the process for acquiring medical information regarding the Nth partial image 66 is completed (S36: YES), the CPU 23 stores the acquired medical information in the storage device 24 (S38). The CPU 23 adds "1" to the value of the counter N for counting the priority order (S39). The CPU 24 executes a series of medical information acquisition processing (that is, a group of partial images that were executed using the current counter It is determined whether the medical information acquisition process for the image 66 has been completed (S40). If it has not been completed (S40: NO), the process returns to S32, and the process is performed on the partial image 40 to which the next priority is set (S32 to S38).

If a new reference image is set in the ophthalmology image processing (see FIG. 5) before the medical information acquisition process regarding the Nth partial image 66 is completed (S36: NO), the medical information It is desirable to change the priority order for acquiring information to the priority order based on the newly set reference image. Therefore, the CPU 23 interrupts the pre-acquisition process (inference process) that is being executed (S46). The priorities reset in S27 are set again based on the newly set reference image (S47). The process returns to S31, and the pre-acquisition process (S31 to S40) is executed according to the newly set priority order. In other words, the CPU 23 interrupts the processing on the group of partial images 66 that was being executed and restarts the processing on a new group of partial images 66.

When the medical information acquisition process for the group of partial images 66 that was being executed using the current counter N is completed (S40: YES), the CPU 23 performs the process for the other group of partial images 66 that was interrupted in S46. It is determined whether or not there is (S41). If there is a suspended process (S41: YES), the CPU 23 resets the priority order and counter N of the suspended process (S42), and the process returns to S32. As a result, the interrupted process is appropriately resumed.

If there is no interrupted processing (S41: NO), it is determined whether the medical information acquisition processing for all partial images 66 included in the ophthalmological image has been completed (S44). If it has not been completed (S44: NO), a new priority order is set to execute the process for the partial image 66 for which medical information has not yet been acquired (S49), and the process returns to S31. When medical information regarding all partial images 66 is acquired (S44: YES), the preliminary acquisition process ends.

(First transformation example)
A first modification of the above embodiment will be described with reference to FIG. 12. FIG. 12 is an example of a display screen 80 displayed on the display device 28 during execution of ophthalmological image processing in the first modification example. The ophthalmological image processing apparatus 1 according to the first modification example uses a part of the three-dimensional tomographic image included in the entire image area of the three-dimensional tomographic image as a partial image, and generates an Enface image 55 from the partial image. The ophthalmological image processing apparatus 1 acquires high-quality images 85 (85A, 85B, 85C), which are obtained by improving the image quality of the generated Enface image 55, as medical information. Note that at least a part of the ophthalmological image processing (see FIG. 5) and the preliminary acquisition processing (see FIG. 11) illustrated in the above embodiment can be similarly adopted in the following first modification example. Therefore, descriptions of processes that can employ processes similar to those of the above embodiments will be omitted or simplified.

In the ophthalmological image processing according to the first modification example (see FIG. 5), first, a three-dimensional tomographic image (for example, see the three-dimensional tomographic image 50 shown in FIG. 3) of the fundus tissue of the eye to be examined is acquired (S11). Next, the CPU 23 displays the partial image selection image 81 (see FIG. 12) on the display device 28 (S12). As shown in FIG. 12, during execution of ophthalmological image processing, a partial image selection image 81 and high-quality images 85A, 85B, and 85C are displayed on the display screen 80. One two-dimensional tomographic image forming the three-dimensional tomographic image acquired in S11 is used as the partial image selection image 81 of the first modification example. The partial image selection image 81 displays the positions of two boundary surfaces (slabs) 82A and 82B for defining the range of the partial image. A three-dimensional tomographic image within a region sandwiched between the two boundary surfaces 82A and 82B becomes a partial image in the first modification example.

Internally, for each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image acquired in S11, at least one of the tissue layers and layer boundaries (hereinafter simply referred to as "layer/boundary") reflected in the image is determined. ) has been obtained. The position of the layer/boundary may be obtained by image processing on a two-dimensional tomographic image, or may be obtained by using a machine learning algorithm. Alternatively, the position may be acquired by the user specifying the position of the layer/boundary.

The user operates the operation unit 27 to set (for example, move) the position of at least one of the two displayed boundary surfaces 82A and 82B to a desired position. As a result, the positions of the boundary surfaces 82A and 82B that define the range of the partial images are set. That is, the user can select a partial image in the ophthalmological image acquired in S11 by setting the positions of the boundary surfaces 82A and 82B. As an example, the CPU 23 selects a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image based on the distances and directions of the boundary surfaces 82A and 82B set by the user with respect to the positions of the layers and boundaries of the tissue that have already been acquired. For each image, two boundary surfaces are established. The CPU 23 sets the three-dimensional tomographic image of the area sandwiched between the two set boundary surfaces as a partial image.

Returning to the explanation of FIG. 5. The CPU 23 determines whether a pre-acquisition pattern has been set by the user (S13). The pre-acquired pattern in the first modification example is provided in advance with a pattern in which a partial image is a specific layer that is likely to attract the attention of the user. In the example shown in FIG. 12, a pattern in which the vitreous interface is a partial image and a pattern in which the inner layer of the retina is a partial image are set. The CPU 23 generates an Enface image 55 (see FIG. 3) based on the partial image corresponding to the set pre-acquisition pattern. The CPU 23 inputs the generated Enface image 55 as a base image into the mathematical model to obtain high-quality images 85A and 85B. The process of acquiring high-quality images 85A and 85B corresponding to the pre-acquisition pattern is an example of pre-acquisition processing. The CPU 23 causes the acquired high-quality images 85A and 85B to be stored in the storage device 24 and displayed on the display device 28 (S14).

As described above, the high-quality images 85A and 85B corresponding to the pre-acquired pattern may be executed in advance before the process (S12) of starting display on the display screen 80. In this case, in S12, the high-quality images 85A and 85B corresponding to the pre-acquired pattern may be displayed on the display screen 80 from the beginning. As a result, the high-quality images 85A and 85B corresponding to the pre-acquired pattern are displayed more smoothly on the display device 28.

Next, the CPU 23 determines whether the follow-up position information is stored in the storage device 24 (S16). The follow-up position may be set based on the positions of the two boundary surfaces 82A and 82B selected by the user in past processing. Further, the specific position used in the processes of S18 and S19 may be the position of a specific layer/boundary.

In S21 in the first modification example, the priority order of the partial images is set based on at least one of the proximity of the distance, the proximity of the range, the proximity of the overlap ratio, etc. to the three-dimensional tomographic image used as the reference image. Ru. For example, when the distance between the two boundary surfaces 82A, 82B is kept constant and the positions of the two boundary surfaces 82A, 82B are repeatedly moved by a unit distance, the CPU 23 controls the boundary surfaces 82A, 82B at the respective positions. The plurality of sandwiched partial images may be prioritized in order of their proximity to the reference image. Further, when repeatedly moving the position of one of the two boundary surfaces 81A, 82B by a unit distance, the CPU 23 gives priority to a plurality of partial images sandwiched between the boundary surfaces 82A, 82B at each position in the order of proximity to the reference image. You may also set a ranking.

Further, in S27 in the first modification example, the priority order may be set according to the direction in which the partial image is changed (that is, the direction in which the boundary surfaces 82A and 82B are changed). Further, in S28 in the first modification example, as shown in FIG. 12, a high-quality image 85C of the Enface image 55 based on the selected partial image is displayed on the display screen 28.

Further, in S34 (see FIG. 11) in the first modification example, the CPU 23 generates the Enface image 55 based on the N-th priority partial image. The CPU 23 inputs the generated Enface image 55 into a mathematical model to obtain a high-quality image 85C.

(Second transformation example)
A second modification of the above embodiment will be described. The ophthalmologic image processing apparatus 1 of the embodiment and the first modification obtains a high-quality image obtained by improving the image quality of the base image as medical information. However, the medical information acquired by the ophthalmologic image processing apparatus 1 is not limited to high-quality images. The ophthalmological image processing device 1 of the second modified example acquires the analysis result of the structure of the tissue of the subject's eye as medical information by inputting a partial image or an image generated based on the partial image into a mathematical model. Specifically, the medical information may be analysis results (segmentation results) of tissue layers and boundaries. Further, the medical information may be an analysis result of a tissue lesion site, a fundus blood vessel, or the like. In this case, the training method described above can be used as the training method for the mathematical model. Further, the medical information may be an analysis result regarding a disease of the eye to be examined.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to execute only some of the techniques exemplified in the above embodiments. For example, it is also possible to omit at least one of S13 to S14, S16 to S17, and S18 to S19 in ophthalmological image processing (see FIG. 5). Further, instead of executing the processing in S18 and S19, the CPU 23 sets a predetermined partial image (for example, a partial image including the center of the image area of the ophthalmological image acquired in S11) as the reference image. It's okay. Further, the process of starting the pre-acquisition process (S22) may be executed after the partial image selection instruction is received.

Note that the process of acquiring an ophthalmologic image in S11 of FIG. 5 is an example of an "image acquisition step." The process of acquiring medical information in S14 and S28 of FIG. 5 and S34 to S38 of FIG. 11 is an example of a "medical information acquisition step." The process of accepting a partial image selection instruction in S24 of FIG. 5 is an example of an "instruction receiving step." The process of displaying medical information on the display device 28 in S14 and S28 of FIG. 5 is an example of a "medical information display step."

11A, 11B Ophthalmological image photographing device 21 Ophthalmological image processing device 23 CPU
24 Storage device 27 Operation unit 28 Display device 50 Three-dimensional tomographic image 51 Two-dimensional tomographic image 55 Enface image 66 Partial image 67 High-quality images 82A, 82B Boundary surfaces 85A, 85B, 85C High-quality image

Claims (7)

An ophthalmologic image processing program executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmological image processing program is executed by the control unit of the ophthalmological image processing apparatus,
an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the eye to be examined taken by the OCT device;
Of the three-dimensional tomographic images acquired in the image acquisition step , the partial images are processed by the OCT apparatus based on a partial image that is a three-dimensional tomographic image in a part of the area sandwiched between two boundary surfaces. The image quality of the base image was improved by generating an Enface image when viewed from the direction along the optical axis of the measurement light and inputting the Enface image as a base image to a mathematical model trained by a machine learning algorithm. a medical information acquisition step of acquiring high-quality images and storing them in a storage device;
an instruction receiving step of receiving an input of an instruction to select a partial image in the three-dimensional tomographic image obtained in the image obtaining step;
When an instruction to select a partial image is accepted, a medical information display step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction;
is executed by the ophthalmological image processing device,
In the medical information acquisition step, a high-quality image related to a partial image that has not yet been selected by an instruction to select a partial image from among the three-dimensional tomographic images is inputted when the instruction to select the partial image is input. A pre-acquisition process is executed to acquire and store the information in advance before it is actually accepted in the step.
The pre-acquisition process includes the instruction receiving step of receiving an input of an instruction to select a partial image, and the step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction to select a partial image. An ophthalmologic image processing program that is executed in parallel with a medical information display step. The ophthalmological image processing program according to claim 1,
In the instruction receiving step, the control unit inputs an instruction to set the position of at least one of the two boundary surfaces defining a range of the partial image as an instruction to select the partial image. Processing program. The ophthalmological image processing program according to claim 2,
In the instruction receiving step, the control unit causes a two-dimensional tomographic image constituting the three-dimensional tomographic image acquired in the image acquisition step to display the positions of the two boundary surfaces defining a range of the partial image. An ophthalmological image processing program characterized in that an instruction to move the position of at least one of the two displayed boundary surfaces is input as an instruction to select a partial image in a state where the partial image is selected. The ophthalmological image processing program according to claim 2 or 3,
In the pre-acquisition process, the control unit sets a plurality of partial images in which at least one of two boundary surfaces has a different position among the three-dimensional tomographic images acquired in the image acquisition step. An ophthalmological image processing program characterized by sequentially executing high-quality image acquisition processing for each partial image in descending order of priority. The ophthalmological image processing program according to claim 4,
In the pre-acquisition process, when the partial image selected by the instruction is changed, the control unit controls the boundary surface of the partial image after being changed with respect to the boundary surface of the partial image before being changed. An ophthalmological image processing program characterized in that the priority order for the plurality of partial images is set in order along a boundary surface direction. The ophthalmological image processing program according to any one of claims 1 to 5,
The ophthalmologic image processing apparatus is characterized in that the Enfece image is a motion contrast image obtained by processing at least two OCT signals obtained at different times at the same position. An ophthalmological image processing device that processes an ophthalmological image that is an image of a tissue of an eye to be examined,
The control unit of the ophthalmological image processing device includes:
an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye taken by the OCT device;
Of the three-dimensional tomographic images acquired in the image acquisition step , the partial images are processed by the OCT apparatus based on a partial image that is a three-dimensional tomographic image within a part of the area sandwiched between two boundary surfaces. The image quality of the base image was improved by generating an Enface image when viewed from the direction along the optical axis of the measurement light and inputting the Enface image as a base image to a mathematical model trained by a machine learning algorithm. a medical information acquisition step of acquiring high-quality images and storing them in a storage device;
an instruction receiving step of accepting an input of an instruction to select a partial image in the acquired three-dimensional tomographic image;
When an instruction to select a partial image is accepted, a medical information display step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction;
Run
In the medical information acquisition step, the control unit selects a high-quality image related to a partial image that has not yet been selected according to an instruction to select a partial image from among the three-dimensional tomographic images, according to an instruction to select the partial image. Executing a pre-acquisition process to acquire and store the input in advance before the input is actually accepted in the instruction receiving step;
The preliminary acquisition process includes the instruction receiving step of receiving an input of an instruction to select a partial image, and the step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction to select a partial image. An ophthalmological image processing device characterized in that the device is executed in parallel with a medical information display step.

Images