JP7130043B2 - MEDICAL IMAGE PROCESSING APPARATUS, ENDOSCOPE SYSTEM, AND METHOD OF OPERATION OF MEDICAL IMAGE PROCESSING APPARATUS - Google Patents

Description

The present invention relates to a medical image processing apparatus and an endoscope system for detecting a region of interest such as a lesion, and a method of operating the medical image processing apparatus.

In the medical field, medical images such as endoscopic images, X-ray images, CT (Computed Tomography) images, and MR (Magnetic Resonance) images are used to perform diagnostic imaging, such as diagnosing patient conditions and observing their progress. ing. A doctor or the like determines a treatment policy based on such image diagnosis.

In recent years, in image diagnosis using medical images, medical images are analyzed by a medical image processing apparatus to automatically detect attention areas such as lesions and benign tumors in organs that should be carefully observed. be. In particular, by performing machine learning such as deep learning, the accuracy of detecting a region of interest is dramatically improved.

Patent Literature 1 describes a medical image processing apparatus that performs image processing based on detection information when a region of interest such as a lesion is detected from a medical image. In the medical image processing apparatus described in Patent Document 1, when an attention area is detected, an elapsed time for changing the display mode is set, and alert information for the attention area is displayed until the set elapsed time elapses. An added or superimposed display image is generated and displayed on the display unit.

JP 2011-160848 A

However, in the medical image processing apparatus described in Patent Document 1, when the time for displaying alert information is set short, the attention area may be overlooked without noticing the alert information. On the other hand, if the alert information is displayed for a long time, the alert information will continue to be displayed until a certain period of time has elapsed even if the region of interest no longer exists in the medical image. is searched for a region of interest that does not already exist in the medical image, and the alert information may interfere with observation by the doctor.

The present invention provides a medical image processing apparatus, an endoscope system, and a medical image processing apparatus capable of preventing the observation of a medical image from being obstructed by the display based on the detection of the attention area and preventing the attention area from being overlooked. The object is to provide a method of operation.

The medical image processing apparatus of the present invention includes a medical image acquisition section, an attention area detection section, and a display control section. A detection time marker is displayed on the detection time display section during real-time display . The medical image acquisition unit acquires a medical image by imaging an observation target with an imaging unit. The region-of-interest detection unit detects a region of interest within the observation target in the medical image acquired by the medical image acquisition unit. The display control unit acquires a medical image by the medical image acquisition unit and displays it on the medical image display unit, and displays the detection time when the attention area is detected on the detection time display unit as a detection time marker. The imaging section is provided at the distal end of the insertion section of the endoscope.

Preferably, the display control section displays a time bar corresponding to the elapsed time on the detection time display section, and displays a detection time marker on the time bar.

It is preferable that the display control section changes the time bar in the detection time display section as the elapsed time increases.

Preferably, the display control section extends the length of the time bar from one end side toward the other end side in the detection time display section as the elapsed time increases.

When the length of the time bar reaches a predetermined length as the elapsed time increases, the display control section preferably scrolls the time bar within the detection time display section.

It is preferable that the time bar is elongated strip-shaped or arc-shaped.

Preferably, the display controller displays past medical images corresponding to the past elapsed time on the time bar when an input is made to any position on the time bar.

It is preferable that, when an input is made to the position of the detection time marker, the display control unit displays a past image related to the detection time marker that has been input.

It is preferable that, when an input is performed at the position of the detection time marker, the display control unit displays a moving image including a plurality of past images related to the input detection time marker.

Preferably, the display control unit displays a detection graph in which the elapsed time from the start of acquisition of the medical image is the value on one axis and the index related to the detection of the region of interest is the value on the other axis in the detection time display unit. .

The index is preferably calculated from a medical image and is a degree of reliability indicating the likelihood of being the region of interest.

The region-of-interest detection unit preferably detects the region of interest from the medical image by neural network, deep learning, Adaboost, or random forest.

The region of interest is preferably a lesion.

An endoscope system according to the present invention includes a light source device, an endoscope, a medical image acquisition unit, an attention area detection unit, a display control unit, and a display unit . The detection time marker is displayed on the detection time display unit during real-time display in which the medical images sequentially acquired by the unit are displayed on the medical image display unit . The light source device emits illumination light for illuminating an observation target. An endoscope has an imaging unit that captures an image of an observation target illuminated with illumination light. The imaging section is provided at the distal end of the insertion section. The medical image acquisition unit acquires a medical image obtained by imaging an observation target with the imaging unit. The region-of-interest detection unit detects a region of interest within the observation target in the medical image acquired by the medical image acquisition unit. The display control unit acquires a medical image by the medical image acquisition unit and displays it on the medical image display unit, and displays the detection time when the attention area is detected on the detection time display unit as a detection time marker. The display displays the medical image and the detection time marker.

A method of operating a medical image processing apparatus according to the present invention includes steps in which a medical image acquisition unit acquires a medical image by imaging an observation target with an imaging unit provided at the distal end of an insertion section of an endoscope; a detection unit detecting a region of interest within an observation target in a medical image acquired by the medical image acquisition unit; and a display control unit acquiring the medical image by the medical image acquisition unit and displaying it on the medical image display unit. and displaying the detection time at which the region of interest was detected as a detection time marker on the detection time display unit, wherein the medical images sequentially acquired by the medical image acquisition unit are displayed on the medical image display unit in real time. and displaying the detection time marker on the detection time display .

According to the present invention, the display based on the detection of the attention area does not interfere with the observation of the medical image, and it is possible to prevent the attention area from being overlooked.

1 is an external view of an endoscope system; FIG. 1 is a block diagram showing functions of an endoscope system according to a first embodiment having multiple LED light sources; FIG. 4 is a graph showing spectral spectra of violet light V, blue light B, blue light Bx, green light G, and red light R; 4 is a graph showing the spectral spectrum of normal light according to the first embodiment; 4 is a graph showing the spectral spectrum of special light according to the first embodiment; 3 is a block diagram showing functions of an attention area detection mode image processing unit and a display control unit; FIG. 8A and 8B are explanatory diagrams showing an example of a display screen before (A) and after (B) the display control unit causes the detection time marker to be displayed; FIG. 4 is a flow chart showing a series of flows in a region-of-interest detection mode; 13A and 13B are explanatory diagrams illustrating an example of a display screen before (A) and after (B) the display control unit displays the detection time marker in the second embodiment; FIG. FIG. 11 is an explanatory diagram showing an example of a display screen before (A) and after (B) the attention area is detected when the display control unit according to the third embodiment causes the detection graph to be displayed; . It is an explanatory view of a display screen showing a first modification of the third embodiment. FIG. 11 is an explanatory diagram of a display screen showing a second modified example of the third embodiment; It is explanatory drawing of the display screen which shows the 3rd modification of 3rd Embodiment. FIG. 11 is an explanatory diagram showing an example of a display screen when the time bar reaches the right end of the detection time display portion in the fourth embodiment (A) and when the time bar scrolls within the range of the detection time display portion (B); be. FIG. 14 is an explanatory diagram showing an example of a display screen when the time bar reaches the right end of the detection time display portion (A) and when the entire time bar is reduced (B) in the modified example of the fifth embodiment; FIG. 21 is an explanatory diagram showing an example of a display screen when a time bar is displayed in an arc shape according to the fifth embodiment; FIG. 21 is a block diagram showing functions of an attention area detection mode image processing unit and a display control unit in the sixth embodiment; FIG. 21 is an explanatory diagram showing an example of a display screen before (A) displaying a thumbnail image and after (B) displaying a thumbnail image in the sixth embodiment; It is explanatory drawing of the display screen which shows the modification of 6th Embodiment.

[First embodiment]

As shown in FIG. 1 , the endoscope system 10 has an endoscope 12 , a light source device 14 , a processor device 16 , a monitor 18 (display section), and a console 19 . The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 16 . The endoscope 12 has an insertion section 12a to be inserted into the subject, an operation section 12b provided at the proximal end portion of the insertion section 12a, and a bending section 12c and a distal end section 12d provided at the distal end side of the insertion section 12a. is doing. By operating the angle knob 13a of the operation portion 12b, the bending portion 12c is bent. This bending action directs the tip 12d in a desired direction.

The distal end portion 12d has an illumination window, an observation window, an air/water feeding nozzle, and a forceps outlet on the distal end surface (all not shown). The illumination window is for irradiating the observation site with illumination light. The observation window is for taking in light from the observation site. The air/water nozzle is for cleaning the illumination window and observation window. The forceps outlet is for performing various treatments using forceps and a treatment tool such as an electric scalpel.

In addition to the angle knob 13a, the operation unit 12b includes a still image acquisition unit 13b used for still image acquisition operations, a mode switching unit 13c used for observation mode switching operations, and a zoom operation unit 13d used for zoom magnification change operations. is provided. The still image acquiring unit 13b can perform a freeze operation for displaying a still image of an observation target on the monitor 18 and a release operation for saving the still image in the storage.

The endoscope system 10 has a normal mode, a special mode, and an attention area detection mode as observation modes. When the observation mode is the normal mode, the normal light obtained by combining light of multiple colors at the light amount ratio Lc for the normal mode is emitted. Further, when the observation mode is the special mode, the special light is emitted by combining light of a plurality of colors at the light amount ratio Ls for the special mode.

When the observation mode is the attention area detection mode, the attention area detection mode illumination light is emitted. In this embodiment, normal light is emitted as illumination light for the attention area detection mode, but special light may be emitted.

Processor unit 16 is electrically connected to monitor 18 and console 19 . The monitor 18 outputs and displays an image to be observed, information incidental to the image, and the like. The console 19 functions as a user interface that receives input operations such as specifying a region of interest (ROI) and setting functions.

As shown in FIG. 2 , the light source device 14 includes a light source section 20 that emits illumination light used to illuminate an observation target, and a light source control section 22 that controls the light source section 20 . The light source unit 20 is a semiconductor light source such as a multi-color LED (Light Emitting Diode). The light source control unit 22 controls the light emission amount of the illumination light by turning on/off the LED or the like and adjusting the drive current or drive voltage of the LED or the like. Also, the light source control unit 22 controls the wavelength band of the illumination light by changing an optical filter or the like.

In the first embodiment, the light source unit 20 includes a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20 d with four color LEDs and a wavelength cut filter 23 . As shown in FIG. 3, the V-LED 20a emits violet light V with a wavelength band of 380 nm to 420 nm.

The B-LED 20b emits blue light B with a wavelength band of 420 nm to 500 nm. Blue light B emitted from the B-LED 23b is cut by the wavelength cut filter 23 at least at the longer wavelength side than the peak wavelength of 450 nm. As a result, the blue light Bx after passing through the wavelength cut filter 23 has a wavelength range of 420 to 460 nm. In this way, the light in the wavelength region longer than 460 nm is cut because the light in the wavelength region longer than 460 nm lowers the blood vessel contrast of the blood vessel to be observed. Because there is Note that the wavelength cut filter 23 may reduce light in the wavelength region longer than 460 nm instead of cutting light in the wavelength region longer than 460 nm.

The G-LED 20c emits green light G with a wavelength band ranging from 480 nm to 600 nm. The R-LED 20d emits red light R with a wavelength band of 600 nm to 650 nm. The center wavelength and peak wavelength of the light emitted from each of the LEDs 20a to 20d may be the same or different.

The light source control unit 22 independently controls the lighting and extinguishing of each of the LEDs 20a to 20d, and the amount of light emitted when the LEDs 20a to 20d are lit, thereby adjusting the light emission timing, light emission period, light amount, and spectral spectrum of the illumination light. Control of lighting and extinguishing by the light source control unit 22 differs for each observation mode. The reference brightness can be set by the brightness setting unit of the light source device 14, the console 19, or the like.

In the normal mode or the attention area detection mode, the light source control unit 22 lights all of the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. At that time, as shown in FIG. 4, the light intensity ratio Lc between the violet light V, the blue light B, the green light G, and the red light R is such that the peak of the light intensity of the blue light Bx is the violet light V, the green light G , and red light R. As a result, in the normal mode or the attention area detection mode, the multicolor light for the normal mode or the attention area detection mode including the violet light V, the blue light Bx, the green light G, and the red light R from the light source device 14 is normally is emitted as light. Ordinary light has a certain intensity or more from the blue band to the red band, so it is almost white.

In the special mode, the light source control unit 22 turns on all of the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. At that time, as shown in FIG. 5, the light amount ratio Ls between the violet light V, the blue light B, the green light G, and the red light R is such that the peak of the light intensity of the violet light V is the blue light Bx, the green light G , and red light R. Also, the peaks of the light intensity of the green light G and the red light R are set to be smaller than the peaks of the light intensity of the violet light V and the blue light Bx. As a result, in the special mode, the light source device 14 emits special mode polychromatic light including violet light V, blue light Bx, green light G, and red light R as special light. The special light has a bluish tinge because the violet light V accounts for a large proportion of the special light. Note that the special light does not have to include light of all four colors, and may include light from at least one of the LEDs 20a to 20d of the four colors. Also, the special light preferably has a main wavelength range of 450 nm or less, for example, a peak wavelength or a central wavelength.

As shown in FIG. 2, the illumination light emitted by the light source unit 20 enters the light guide 24 inserted through the insertion portion 12a via an optical path coupling portion (not shown) formed of mirrors, lenses, and the like. The light guide 24 is built in the endoscope 12 and the universal cord, and propagates the illumination light to the distal end portion 12d of the endoscope 12. As shown in FIG. A universal cord is a cord that connects the endoscope 12 , the light source device 14 and the processor device 16 . As the light guide 24, a multimode fiber can be used. As an example, the light guide 24 can be a thin fiber cable with a core diameter of 105 μm, a clad diameter of 125 μm, and a diameter of φ0.3 mm to φ0.5 mm including the outer protective layer.

The distal end portion 12d of the endoscope 12 is provided with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30 a has an illumination lens 32 . The illumination light propagated through the light guide 24 illuminates the object to be observed through the illumination lens 32 . The imaging optical system 30b has an objective lens 34, an enlarging optical system 36, and an imaging sensor 38 (corresponding to the "imaging section" of the present invention). Various kinds of light such as reflected light, scattered light, and fluorescence from the observation target enter the imaging sensor 38 via the objective lens 34 and the magnifying optical system 36 . As a result, an image of the observation target is formed on the imaging sensor 38 .

The magnifying optical system 36 includes a zoom lens 36a that magnifies an observation target, and a lens driving section 36b that moves the zoom lens 36a in the optical axis direction CL. The zoom lens 36a is freely moved between the telephoto end and the wide end according to the zoom control by the lens driving section 36b, thereby enlarging or reducing the observation target imaged on the imaging sensor 38. FIG.

The imaging sensor 38 is a color imaging sensor that captures an image of an observation target irradiated with illumination light. Each pixel of the imaging sensor 38 is provided with either an R (red) color filter, a G (green) color filter, or a B (blue) color filter. The imaging sensor 38 receives violet to blue light with B pixels provided with B color filters, receives green light with G pixels provided with G color filters, and receives green light with R color filters. Red light is received by the R pixels in the region. Then, image signals of RGB colors are output from the pixels of each color. The imaging sensor 38 transmits the output image signal to the CDS circuit 40 .

In the normal mode or the attention area detection mode, the imaging sensor 38 outputs Bc image signals from B pixels, outputs Gc image signals from G pixels, and outputs R An Rc image signal is output from the pixel. In the special mode, the imaging sensor 38 captures an image of an observation target illuminated with special light, thereby outputting a Bs image signal from the B pixels, a Gs image signal from the G pixels, and an Rs image signal from the R pixels. Outputs image signals.

As the imaging sensor 38, a CCD (Charge Coupled Device) imaging sensor, a CMOS (Complementary Metal-Oxide Semiconductor) imaging sensor, or the like can be used. Further, instead of the imaging sensor 38 provided with RGB primary color filters, a complementary color imaging sensor provided with C (cyan), M (magenta), Y (yellow) and G (green) complementary color filters may be used. good. When a complementary color imaging sensor is used, image signals of four colors of CMYG are output. Therefore, by converting the CMYG four-color image signals into RGB three-color image signals by complementary color-primary color conversion, RGB image signals similar to those of the image sensor 38 can be obtained. Also, instead of the imaging sensor 38, a monochrome sensor without a color filter may be used.

The CDS circuit 40 performs correlated double sampling (CDS) on the analog image signal received from the imaging sensor 38 . The image signal that has passed through the CDS circuit 40 is input to the AGC circuit 42 . The AGC circuit 40 performs automatic gain control (AGC) on the input image signal. An A/D (Analog to Digital) conversion circuit 44 converts the analog image signal that has passed through the AGC circuit 42 into a digital image signal. The A/D conversion circuit 44 inputs the digital image signal after A/D conversion to the processor device 16 .

As shown in FIG. 2, the processor device 16 includes an image signal acquisition unit 50 (corresponding to the "medical image acquisition unit" of the present invention), a DSP (Digital Signal Processor) 52, a noise reduction unit 54, and image processing. 56 and a display control unit 58 .

The image signal acquisition unit 50 acquires a digital image signal corresponding to the observation mode from the endoscope 12. FIG. In the normal mode or the attention area detection mode, the Bc image signal, the Gc image signal, and the Rc image signal are obtained. In the special mode, the Bs image signal, Gs image signal, and Rs image signal are acquired. In the case of the region-of-interest detection mode, one frame of Bc image signal, Gc image signal, and Rc image signal is acquired during normal light illumination, and one frame of Bs image signal, Gs image signal, and Acquire the Rs image signal.

The DSP 52 performs various signal processing such as defect correction processing, offset processing, DSP gain correction processing, linear matrix processing, gamma conversion processing, and demosaicing processing on the image signal acquired by the image signal acquisition section 50 . The defect correction process corrects signals of defective pixels of the imaging sensor 38 . The offset processing removes the dark current component from the defect-corrected image signal and sets an accurate zero level. The DSP gain correction process adjusts the signal level by multiplying the offset-processed image signal by a specific DSP gain.

Linear matrix processing enhances the color reproducibility of image signals that have undergone DSP gain correction processing. The gamma conversion process adjusts the brightness and saturation of the linear matrix processed image signal. The gamma-converted image signal is subjected to demosaic processing (also referred to as isotropic processing or simultaneous processing) to generate signals of missing colors in each pixel by interpolation. This demosaicing process causes all pixels to have RGB signals. The noise reduction unit 54 performs noise reduction processing using, for example, a moving average method, a median filter method, or the like on the image signal that has undergone demosaicing processing or the like in the DSP 52, thereby reducing noise. The image signal after noise reduction is input to the image processing section 56 .

The image processing section 56 includes a normal mode image processing section 60 , a special mode image processing section 62 , and an attention area detection mode image processing section 64 . The normal mode image processing unit 60 operates when the normal mode is set, and performs color conversion processing, color enhancement processing, and structure enhancement processing on the received Bc image signal, Gc image signal, and Rc image signal. conduct. In the color conversion processing, color conversion processing is performed on RGB image signals by 3×3 matrix processing, gradation conversion processing, three-dimensional LUT (Look Up Table) processing, and the like.

Color enhancement processing is performed on RGB image signals that have undergone color conversion processing. Structure enhancement processing is processing for enhancing the structure of an observation target, and is performed on RGB image signals after color enhancement processing. A normal image can be obtained by performing various types of image processing as described above. Since the normal image is an image obtained based on normal light in which the violet light V, the blue light Bx, the green light G, and the red light R are emitted in a well-balanced manner, the image has natural colors. A normal image is input to the display control unit 58 .

The special mode image processing section 62 operates when the special mode is set. The special mode image processing unit 62 performs color conversion processing, color enhancement processing, and structure enhancement processing on the received Bs image signal, Gs image signal, and Rs image signal. The processing contents of the color conversion processing, color enhancement processing, and structure enhancement processing are the same as those of the normal mode image processing section 60 . A special image is obtained by performing various types of image processing as described above. The special image is an image obtained based on special light in which violet light V, which has a high absorption coefficient of hemoglobin in blood vessels, emits a greater amount of light than blue light Bx, green light G, and red light R of other colors. Therefore, the resolution of vascular structures and ductal structures is higher than that of other structures. A special image is input to the display control unit 58 .

The attention area detection mode image processing unit 64 operates when the attention area detection mode is set. As shown in FIG. 6 , the attention area detection mode image processing section 64 has a detection image processing section 70 , an attention area detection section 71 and a time measurement section 72 . The detection image processing unit 70 sequentially obtains endoscopic images from the received Bc image signal, Gc image signal, and Rc image signal by performing image processing similar to that of the normal mode image processing unit 60, such as color conversion processing.

The region-of-interest detection unit 71 analyzes the endoscopic image and performs region-of-interest detection processing for detecting the region of interest within the observation target. In this embodiment, the region-of-interest detection unit 71 detects a lesion (for example, a tumor or inflammation) in the observation target as the region of interest. In this case, the region-of-interest detection unit 71 first divides the endoscopic image into a plurality of small regions, for example, square regions each having several pixels. Next, an image feature amount is calculated from the divided endoscopic images. Subsequently, recognition processing is performed to determine whether or not each small region is a lesion based on the calculated feature amount. Such recognition processing is preferably a machine learning algorithm such as a convolutional neural network, deep learning, Adaboost, or random forest.

Further, the feature amount calculated from the endoscopic image by the region-of-interest detection unit 71 is preferably the shape and color of a predetermined portion of the observation target, or an index value obtained from the shape and color. For example, the feature values include blood vessel density, blood vessel shape, number of blood vessel branches, blood vessel thickness, blood vessel length, blood vessel tortuosity, blood vessel invasion depth, duct shape, duct opening shape, duct It is preferably at least one of the length of the duct, the meandering degree of the duct, and the color information, or a value obtained by combining two or more of them.

Finally, a cluster of small regions identified as the same type is extracted as one lesion. The region-of-interest detection unit 71 associates information such as the position, size, and type of the extracted lesion with the endoscopic image as detection information.

On the other hand, the time measurement unit 72 measures the elapsed time after the image signal acquisition unit 50 starts acquisition of the endoscopic image. The time measuring unit 72 measures the elapsed time from the start of acquisition of the endoscopic image using, for example, a counter. The initial value of the counter is 0, and the counter value is incremented by one (counter value +1) each time a clock signal having a predetermined cycle is input from the start of endoscopic image acquisition. Advancing the counter value by one means measuring the elapsed time for each cycle of the counter.

In this case, the attention area detection mode image processing unit 64 outputs the endoscopic image 75 associated with the detection information and the elapsed time T (counter value) from the acquisition start of the endoscopic image 75 to the display control unit 58. do. When starting to acquire the endoscopic image 75, the time measuring unit 72 outputs an initial value T0 (the counter value is 0) as the elapsed time T. FIG. While the endoscopic image 75 is being output, the time measuring section 72 continues outputting the elapsed time T to the display control section 58 .

The display control unit 58 performs display control for displaying images and data from the image processing unit 56 on the monitor 18 . When the normal mode is set, the display control unit 58 performs control to display a normal image on the monitor 18 . When the special mode is set, the display control section 58 performs control to display the special image on the monitor 18 .

When the attention area detection mode is set, the display control unit 58 displays the endoscopic images sequentially acquired from the attention area detection mode image processing unit 64 in real time, and detects the attention area from the endoscopic images. The detected time is displayed on the monitor 18 as a detection time marker.

When displaying the detection time marker on the monitor 18, the display control unit 58 first displays a time bar having a length corresponding to the elapsed time T output from the time measurement unit 72 described above. Then, the display control unit 58 acquires the endoscope image 75 to which the detection information is added, and the detection time at which the attention area is detected at the position corresponding to the elapsed time T output from the time measurement unit 72. Display the detection time marker at the position on the time bar as .

As shown in FIG. 7A, when the attention area detection mode is set, the display control unit 58 controls the endoscope image captured by the imaging sensor 38 and image-processed by the attention area detection mode image processing unit 64. 75 (images similar to the normal image) are sequentially acquired and displayed in real time on the display screen 76 of the monitor 18, and a time bar 77 having a length corresponding to the elapsed time T output from the time measurement unit 72 is displayed.

The endoscopic image 75 is displayed on the medical image display portion 76A within the display screen 76, and the time bar 77 is displayed on the detection time display portion 76B (the range indicated by the two-dot chain line) located outside the medical image display portion 76A. Is displayed. The medical image display section 76A is formed in a shape in which a part of a circle is notched according to the imaging range imaged by the imaging sensor 38, and the detection time display section 76B is provided below the medical image display section 76A. To position. Note that the two-dot chain line indicating the detection time display portion 76B is a virtual line and does not have to be displayed in practice.

The time bar 77 has a long strip shape extending in the X-axis direction (horizontal direction) of the display screen 76 . In the present embodiment, the time bar 77 has an initial value T0 of the elapsed time T near the left end of the detection time display section 76B, and as the elapsed time T increases, the time bar 77 increases from the left end to the right end of the detection time display section 76B. gradually increasing in length. Note that the state shown in FIG. 7A is after real-time display of the endoscopic image 75 has started, and the region of interest within the observation target has not yet been detected. Therefore, the detection time marker indicating the detection time of the attention area is not yet displayed on the time bar 77 .

After the state shown in FIG. 7(A), when a lesion 78 is detected as an attention area in the observation object, detection time markers 79A and 79B are displayed on the time bar 77 as shown in FIG. 7(B). Append. The detection time markers 79A and 79B are triangles pointing downward, and correspond to the elapsed time T output from the time measurement unit 72 when the display control unit 58 acquires the endoscopic image 75 associated with the detection information. attached to the position.

In the state shown in FIG. 7B, the length of the time bar 77 corresponding to the elapsed time T is longer than in the state shown in FIG. 7A. Detection time markers 79A and 79B are displayed on the time bar 77 at positions corresponding to the elapsed times T1, T2, T3, T4 and T5. In FIG. 7B, for convenience of drawing, the difference in color between the detection time markers 79A and 79B is represented by the presence or absence of hatching applied to the detection time marker 81. As shown in FIG. Further, the shape of the detection time markers 79A and 79B is not limited to a downward-pointing triangle, and may be any shape that can indicate the time when the lesion 78 is detected on the time bar 77, such as a polygon other than a triangle or an arrow. Just do it.

Detection time markers 79A (hatched) indicate positions corresponding to the elapsed times T1, T3, and T5 at which detection of the lesion 78 is started, and detection time markers 79B (not hatched) indicate the end of detection of the lesion 78. The positions corresponding to the elapsed times T2 and T4 are shown. Note that detection time markers may be added for all times when the lesion 78 is detected without being limited to this.

Next, a series of flows in the region-of-interest detection mode will be described with reference to the flowchart shown in FIG. A doctor who is a user operates the mode switching unit 13c to switch to the region-of-interest detection mode (S11). Thereby, the observation target in the body cavity is illuminated with the attention area detection mode illumination light. The imaging sensor 38 captures an image of the observation target illuminated by the attention area detection mode illumination light to obtain an endoscopic image 75 . When switched to the region-of-interest detection mode, the display control unit 58 displays the endoscopic image 75 on the display screen 76 of the monitor 18 in real time and also displays the time bar 77 (S12). The time bar 77 increases in length as the elapsed time T increases.

During real-time display in the attention area detection mode, the attention area detection unit 71 performs attention area detection processing for detecting an attention area in the observation target on the acquired endoscopic image 75 (S13). When the lesion portion 78 as the attention area is detected (Y in S14), the detection information is associated with the endoscopic image 75 and output.

Then, the display control unit 58 displays a detection time marker 79A indicating the start of detection of the lesion 78 on the time bar 77 based on the detection information associated with the endoscopic image 75 and the elapsed time T (S15). The display control unit 58 monitors the detection information and the elapsed time T, and when the detection information associated with the endoscopic image 75 is lost, that is, when the attention area is no longer detected (Y in S16), the lesion part 78 is displayed (S17). Thereafter, until the attention area detection mode ends (N at S18), attention area detection processing is performed (S13), and when an attention area is detected (Y at S14), detection time markers 79A and 79B are displayed ( S15-S17).

As described above, when a region of interest is detected, the detection time markers 79A and 79B are displayed on the time bar 77 positioned outside the endoscopic image 75. Therefore, the doctor, who is the user, can display the detection time markers Since the line of sight can be kept focused on the endoscopic image 75 without being distracted by 79A and 79B, it is possible to prevent the attention area from being overlooked. Moreover, when the detection of the attention area is completed, there is no display of the detection result of the attention area in the endoscopic image 75, so the observation by the doctor is not hindered.

[Second embodiment]

In the above-described first embodiment, the detection time marker is added to the time bar to display the detection time when the attention area is detected. You can change the color of the bar and display it. In this case, as shown in FIGS. 9A and 9B, the time bar 80 is displayed in the detection time display section 76B as in the first embodiment. Further, as in the first embodiment, the time bar 80 has the initial value T0 of the elapsed time T located near the left end of the detection time display portion 76B, and as the elapsed time T increases, the left end of the detection time display portion 76B increases. The length gradually extends from the side toward the right end. The state shown in FIG. 9A is after real-time display of the endoscopic image 75 has started and the region of interest within the observation target has not yet been detected. Therefore, the detection time marker indicating the detection time of the attention area is not yet displayed on the time bar 80 .

After the state shown in FIG. 9(A), when a lesion 78 is detected as an attention area in the observation object, a detection time marker 81 is displayed on the time bar 80 as shown in FIG. 9(B). . The detection time marker 81 is displayed at a position corresponding to the elapsed time T output from the time measurement unit 72 when the endoscopic image 75 associated with the detection information is obtained. In the state shown in FIG. 9B, the length of the time bar 80 corresponding to the elapsed time T is longer than in the state shown in FIG. 9A. Detection time markers 81 are displayed on the time bar 80 at positions from the elapsed time T1 to the elapsed time T2 and from the elapsed time T3 to the elapsed time T4. For convenience of drawing, the difference in color between the detection time marker 81 and the time bar 80 excluding the detection time marker 81 is represented by the presence or absence of hatching.

As in the first embodiment, the attention area detection mode image processing unit 64 outputs detection information in association with the endoscope image 75 when the lesion 78 as the attention area is detected. The left end of the detection time marker 81 indicates the positions corresponding to the elapsed times T1 and T3 when the detection of the lesion 78 is started, and the right end of the detection time marker 81 indicates the elapsed times T2 and T4 when the detection of the lesion 78 is finished. It shows the position according to Since the detection time marker 81 is displayed on the time bar 80 in the same manner as in the first embodiment, it is possible to prevent the attention area in the endoscopic image from being overlooked and hinder observation.

[Third embodiment]

In the first and second embodiments, the detection time marker is displayed only to indicate the time when the attention area is detected. , a detection graph 83 may be displayed as a detection time marker. In the present embodiment, the detection graph 83 is a curve line graph in which the elapsed time T is the value on the X axis (one axis) and the index related to the detection of the attention area is the value on the Y axis (the other axis).

In this embodiment, the index related to the detection of the attention area is the reliability calculated from the detection information of the attention area. The region-of-interest detection mode image processing unit 64 uses the image information of the region of interest detected from the endoscopic image in the same manner as in the first and second embodiments, for example, the area, position, pixel value, etc. of the region of interest, A reliability, which is an index indicating the probability of being a lesion (probability of being a region of interest), is calculated. When the image information of the calculated attention area exceeds a predetermined threshold value, the attention area is evaluated as having high reliability, and it is determined that the attention area has been detected. On the other hand, if the calculated image information of the attention area is equal to or less than the predetermined threshold value, the attention area is evaluated as having a low reliability, and it is determined that the attention area has not been detected. As a method for calculating reliability, it is preferable to use artificial intelligence (AI), deep learning, convolutional neural network, template matching, texture analysis, frequency analysis, or the like.

The state shown in FIG. 10A is after real-time display of the endoscopic image 75 has started, and a time bar 82 indicating the elapsed time T and a detection graph 83 located on the time bar 82 are displayed. there is In this state, in the detection graph 83, the Y-axis value indicating the reliability is small and is equal to or less than the threshold value (broken line 83A indicates the threshold value). can do. Note that the time bar 82 and detection graph 83 are displayed in the detection time display section 76B.

After the state shown in FIG. 10(A), the state shown in FIG. 10(B) shows a case where a lesion portion 78 as an attention area is detected in the observation object. In this case, the detection graph 83 displayed on the time bar 82 has a larger Y-axis value indicating reliability than the state shown in FIG. It can be determined that portion 78 has been detected.

As a modification of the third embodiment, as shown in FIG. 11, the time bar 82 may not be displayed in the detection time display section 76B, and only the detection graph 83 may be displayed. In this case, even if the time bar 82 is not displayed, the doctor can visually recognize that the detection graph 83 extends in the X-axis direction with the passage of time, so the doctor can know the elapsed time T.

As another modification of the third embodiment, a plurality of bar graphs 84 may be displayed as the detection graph instead of the line graph as shown in FIG. In the example shown in FIG. 12, bar graphs 84 as detection graphs are sequentially displayed on a time bar 82 as the elapsed time T increases. Note that the time bar 82 and detection graph 83 are displayed in the detection time display section 76B.

The Y-axis value of the bar graph 84 is the reliability for determining the detection of the attention area, as in the third embodiment. A dashed line 84A indicates a reliability threshold, like the dashed line 83A of the third embodiment. The position where each bar graph 84 is arranged corresponds to the elapsed time T when the endoscopic image from which the reliability was calculated was acquired. As a result, the endoscopic image 75 at the elapsed time T and the bar graph 84 indicating the reliability corresponding to this endoscopic image 75 are sequentially displayed.

Furthermore, as another embodiment, as shown in FIG. 13, the time bar 82 may not be displayed in the detection time display section 76B, and only a plurality of bar graphs 84 may be displayed. In this case, even if the time bar 82 is not displayed, the doctor can recognize the elapsed time T because the number of bar graphs 84 increases with the passage of time.

The detection time marker as in the first and second embodiments may be displayed on the time bar, and the detection graph 83 may be displayed as in the third embodiment. Further, in the third embodiment and the modified example, the reliability threshold is indicated by a dashed line, but the present invention is not limited to this. , the color of the detection graph may be changed depending on whether the threshold is exceeded or not.

[Fourth embodiment]

In the first to third embodiments, the length of the time bar gradually extends from the left end to the right end within the range of the detection time display section to indicate the elapsed time. It is not mentioned after the length of the bar has been stretched to reach the predetermined length. In this fourth embodiment, when the length of the time bar reaches a predetermined length, the time bar scrolls within the range of the detection time display portion to indicate the elapsed time. Further, in the present embodiment, the predetermined length reached by the length of the time bar is the length from one end to the other end of the detection time display portion 76B.

In this case, as shown in FIG. 14A, the length of the time bar 85 extends from the left end (initial value T0) toward the right end until it reaches the right end of the detection time display section 76B. The length of the time bar 85 indicates the elapsed time T as in the first to third embodiments. Note that detection time markers 79A and 79B are added to the time bar 85 to display the detection time when the attention area is detected, as in the first embodiment. The detection time marker on the time bar 85 is not limited to this, and the color of the time bar 85 may be changed and displayed as the detection time marker as in the second embodiment.

After the time bar 85 reaches the right end of the detection time display portion 76B, the time bar 85 horizontally scrolls within the range of the detection time display portion 76B as shown in FIG. 14(B). That is, when the length of the time bar 85 continues to increase with the length corresponding to the elapsed time T, it exceeds the range of the detection time display section 76B. Therefore, according to the elapsed time T, the entire time bar 85 is moved in the opposite direction, that is, from the right side to the left side by the length of the time bar 85 exceeding the range of the detection time display portion 76B. Furthermore, the portion protruding from the left end of the detection time display portion 76B is sequentially deleted by the amount of movement of the entire time bar 85 . As a result, the right end of the detection time display section 76B always indicates the time TP at which the currently displayed endoscopic image 75 was acquired. Further, the detection time markers 79A and 79B are also moved as the entire time bar 85 moves.

As described above, even after the length of the time bar 85 reaches the right end of the detection time display section 76B, as in the first to third embodiments, based on the endoscopic image and the detection information of the attention area, can be displayed. Further, since the entire time bar 85 is moving, the doctor can know that time has passed (acquisition of endoscopic images is continuing).

Note that the display method of the time bar is not limited to this. As shown in FIG. , the entire length of the time bar 85 may be reduced and displayed. By reducing the length of the entire time bar 85, the doctor can know that time has passed (acquisition of endoscopic images is continuing). In this case, the positions and widths of the detection time markers 79A and 79B are also changed according to the reduction of the time bar 85 as a whole. If the length of the time bar 85 reaches the right end of the detection time display portion after the length of the entire time bar 85 has been reduced, the length of the entire time bar 85 may be reduced again.

[Fifth embodiment]

In the first to fourth embodiments, the time bar indicating the elapsed time from the start of acquisition of the endoscopic image is displayed in the form of a long band. The bar 87 may be displayed in an arc shape. In the example shown in FIG. 16, the time bar 87 is displayed in the detection time display section 76B positioned to the side of the medical image display section 76A. The arc length L of the time bar 87 increases according to the elapsed time T. FIG.

The arc of the time bar 87 is provided with detection time markers 88A and 88B similar to those of the first embodiment. The detection time markers 88A and 88B, like the detection time markers 79A and 79B of the first embodiment, indicate the start and end of detection of the lesion as the region of interest, and the detection time marker 88A (hatched) is , indicates the elapsed time when the detection of the lesion 78 is started, and the detection time marker 88B (not hatched) indicates the elapsed time when the detection of the lesion 78 is finished. The detection time marker on the time bar 87 is not limited to this. As in the second embodiment, the time bar 87 may be displayed in a different color as the detection time marker. Similarly, a detection graph may be displayed around the time bar 87 .

[Sixth embodiment]

In the first to fifth embodiments, endoscopic images to be displayed on the display screen are sequentially acquired and displayed in real time on the display screen, but not limited to this, past images may also be displayed. good too. In the sixth embodiment described below, when any one of the detection time markers is selected, thumbnail images (past endoscopic images) associated with the detection time marker are displayed. As shown in FIG. 17 , the display control section 58 has an image storage section 89 . The image storage unit 89 is for temporarily storing past endoscope images acquired from the attention area detection mode image processing unit 64 and displayed by the display control unit 58 .

As shown in FIG. 18A, when neither of the detection time markers 79A and 79B is selected, past endoscopic images are not displayed. As in the first embodiment, detection time markers 79A and 79B are added to the time bar 77 to indicate the start and end of detection of the lesion as the region of interest. Moreover, the detection time marker is not limited to this. As in the second embodiment, the time bar 77 may be displayed in a different color as the detection time marker. A detection graph may be displayed around the . Further, the cursor 90 indicates a position when an object within the display screen 76 is selected by an input operation on the console 19, and nothing is selected in the state shown in FIG. 18(A).

When one of the detection time markers 79A and 79B is selected, the example shown in FIG. 18B shows a state in which the detection time marker 79A indicating the position of the elapsed time T1 is selected. In this embodiment, the input operation for selecting one of the detection time markers 79A and 79B is a so-called mouse-over operation, and input is performed by placing the cursor 90 over the detection time markers 79A and 79B. is done. In this case, a window 91 appears at the position of the selected (inputted) detection time marker 79A. A thumbnail image 92 is displayed inside this window 91 . The thumbnail image 92 is based on the endoscopic image related to the detection time marker 79A, that is, the past endoscopic image acquired during the elapsed time T1 and read from the image storage unit 89, and the number of pixels and the image quality. is dropped and displayed.

As described above, by performing an input at the position of the detection time marker, the past image in which the region of interest was detected is displayed. Never miss an area. Moreover, when the user wants to see the attention area again, it is possible to display the past image by performing an input at the position of the detection time marker, thereby further preventing oversight.

As a modification of the sixth embodiment, by performing an input at the position of the detection time marker, not only one image (still image) may be displayed, but also a moving image consisting of a plurality of consecutive images may be displayed. . In this case, the display control unit 58 selects from a plurality of endoscopic images acquired from the start of detection of the attention area to the end of detection, that is, a plurality of endoscopic images (still images) associated with the detection information. One moving image file is created and stored in the image storage unit 89 .

When one of the detection time markers 79A and 79B, for example, the detection time marker 79A indicating the position of the elapsed time T1 is selected as shown in FIG. b) A window 91 appears at the position of the detection time marker 79A. Inside this window 91, a moving image related to the detection time marker 79A, that is, a moving image file created from a plurality of endoscopic images acquired between the elapsed time T1 of detection start and the elapsed time T2 of detection end. is read from the image storage unit 89 and displayed.

As another modification of the sixth embodiment, instead of selecting any one of the detection time markers, as shown in FIG. You may display a past image in . In the modified example shown in FIG. 19 as well, the input operation for inputting to any position on the time bar 77 is a mouse-over operation, and by placing the cursor 90 on any position on the time bar 77, Input is made. In this case, the display control unit 58 stores all endoscopic images displayed on the medical image display unit 76A from the initial value T0 of the elapsed time T to the end of observation, that is, from the start of acquisition of endoscopic images to the end of acquisition. let me When the cursor 90 is moved to any position on the time bar 77 and input is performed, a window 91 appears at the position on the time bar 77 where the input is performed. Inside this window 91, a past thumbnail image 92 read out from the image storage unit 89 corresponding to the past elapsed time on the time bar 77 is displayed. The thumbnail image 92 is based on the past endoscopic image read out from the image storage unit 89 and displayed with reduced pixel count and image quality, as in the sixth embodiment.

Note that the input operation in the sixth embodiment and the modified example is a mouse-over operation, but is not limited to this, and an input can be made at any position on the detection time marker or the time bar. Any possible operation may be used, such as a so-called click operation, or a touch operation if the input device is a touch panel.

In each of the above embodiments, the elapsed time T is measured by using a counter to measure the elapsed time from the start of acquisition of the endoscopic image, but the configuration for measuring the elapsed time is limited to this. Any value that indicates the time series from the start of endoscopic image acquisition is sufficient. may be used to measure the elapsed time T. That is, in the case of the first frame of the endoscopic image, the elapsed time T is the initial value (0), and thereafter a value corresponding to the order in which the endoscopic images are acquired is output. Then, the display control unit 58 displays the detection time marker, the detection graph, etc., using the values according to this order.

Moreover, the start time of the elapsed time T is not limited to the start of acquisition of the endoscopic image, and may be the time when the attention area detection unit 71 starts detection of the attention area. In this case, the detection of the attention area is preferably started at the time when the automatic detection by the attention area detection unit 71 is started. The time at which the detection of is turned on may be the time when the elapsed time T starts. The time when the attention area is first detected by the attention area detection unit 71, or the time when the display of the time bar can be switched on/off, and the time when the display of the time bar is turned on is defined as the start of the elapsed time T. You may

In each of the embodiments described above, the four-color LEDs 20a to 20d are used to illuminate the observation target, but the observation target may be illuminated using a laser light source and a phosphor. In each of the above-described embodiments, the four-color LEDs 20a to 20d are used to illuminate the observation target, but the observation target may be illuminated using a white light source such as a xenon lamp and a rotating filter. Further, instead of the color image sensor 38, a monochrome image sensor may be used to capture an image of the observation target.

In the above embodiment, the medical image processing apparatus of the present invention is applied to an endoscope system that acquires endoscopic images as medical images. Acquisition of other medical images such as X-ray images, CT images, MR images, ultrasound images, pathological images, PET (Positron Emission Tomography) images, etc. can be applied to the mirror system. It is also possible to apply the medical image processing apparatus of the present invention to various medical image apparatuses for

In the above embodiment, the hardware structure of a processing unit that executes various processes such as the image processing unit 56 is various processors as described below. Various processors include CPU (Central Processing Unit), GPU (Graphical Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that run software (programs) and function as various processing units. Programmable Logic Devices (PLDs), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electric circuits, which are processors with circuit configurations specially designed to perform various types of processing, etc. .

One processing unit may be configured with one of these various processors, or a combination of two or more processors of the same or different type (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, or a CPU and a GPU). Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units in one processor, first, as represented by computers such as clients and servers, one processor is configured by combining one or more CPUs and software, There is a form in which this processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured by using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electrical circuit in the form of a combination of circuit elements such as semiconductor elements.

10 Endoscope system

12 Endoscope

12a insert

12b Operation unit

12c bend

12d tip

13a angle knob

13b still image acquisition unit

13c mode switching unit

13d Zoom operation part

14 Light source device

16 processor device

18 monitor

19 console

20 light source

20a V-LED

20b B-LEDs

20c G-LED

20d R-LED

22 light source controller

23 wavelength cut filter

24 light guide

30a illumination optical system

30b imaging optical system

32 illumination lens

34 objective lens

36 magnifying optical system

36a zoom lens

36b lens driver

38 imaging sensor

40 CDS circuit

42 AGC circuit

44 A/D conversion circuit

50 image signal acquisition unit

52 DSPs

54 noise reduction unit

56 Image processing unit

58 Display control unit

60 normal mode image processing unit

62 Special mode image processor

64 Attention area detection mode image processing unit

70 Image processing unit for detection

71 attention area detection unit

72 Time measurement unit

75 Endoscopic Images

76 display screen

76A medical image display unit

76B detection time display

77 time bar

78 Lesions

79A detection time marker

79B detection time marker

80 time bar

81 detection time marker

82 time bar

83 detection graph

84 bar graph

85 time bar

87 time bar

88A detection time marker

88B detection time marker

89 image memory

90 Cursor

91 windows

92 thumbnail images

L Length T Elapsed time

Claims (15)

a medical image acquisition unit that acquires a medical image by imaging an observation target with an imaging unit provided at the distal end of the insertion section of the endoscope ;
a region-of-interest detection unit that detects a region of interest within the observation target in the medical image acquired by the medical image acquisition unit;
a display control unit that acquires the medical image by a medical image acquisition unit and displays it on a medical image display unit, and displays a detection time at which the attention area is detected on the detection time display unit as a detection time marker ;
The display control unit displays the detection time marker on the detection time display unit during real-time display in which the medical images sequentially acquired by the medical image acquisition unit are displayed on the medical image display unit. 2. The medical image processing apparatus according to claim 1, wherein the display control unit displays a time bar corresponding to the elapsed time on the detection time display unit, and displays the detection time marker on the time bar. 3. The medical image processing apparatus according to claim 2, wherein the display control section changes the time bar in the detection time display section as the elapsed time increases. 4. The medical image processing apparatus according to claim 3, wherein the display control section extends the length of the time bar from one end side toward the other end side in the detection time display section as the elapsed time increases. 5. The display control unit scrolls the entire time bar within the detection time display unit when the length of the time bar reaches a predetermined length as the elapsed time increases. medical image processing equipment. 6. The medical image processing apparatus according to any one of claims 2 to 5, wherein said time bar has an elongated strip shape or an arc shape. 7. The display control unit according to any one of claims 2 to 6, wherein when an input is made to any position on the time bar, the display control unit displays the past medical image corresponding to the past elapsed time on the time bar. The medical image processing device described. 7. The medical image according to any one of claims 1 to 6, wherein, when an input is performed at the position of the detection time marker, the display control unit displays a past image related to the input detection time marker. processing equipment. 7. The display control unit according to any one of claims 1 to 6, wherein, when an input is performed at the position of the detection time marker, the display control unit displays a moving image including a plurality of past images related to the input detection time marker. The medical image processing apparatus according to claim 1. The display control unit displays, on the detection time display unit, a detection graph having an elapsed time from the start of acquisition of the medical image as a value on one axis and an index related to detection of the attention area as a value on the other axis. The medical image processing apparatus according to any one of claims 1 to 9. 11. The medical image processing apparatus according to claim 10, wherein the index is calculated from the medical image and is a degree of reliability indicating the likelihood of being the attention area. The medical image processing apparatus according to any one of claims 1 to 11, wherein the attention area detection unit detects the attention area from the medical image by neural network, deep learning, Adaboost, or random forest. 13. The medical image processing apparatus according to any one of claims 1 to 12, wherein the region of interest is a lesion. a light source device that emits illumination light for illuminating an observation target;
an endoscope provided at the distal end of the insertion section and having an imaging section configured to capture an image of an observation target illuminated by the illumination light;
a medical image acquisition unit that acquires a medical image obtained by imaging the observation target with the imaging unit;
a region-of-interest detection unit that detects a region of interest within the observation target in the medical image acquired by the medical image acquisition unit;
a display control unit that acquires the medical image by a medical image acquisition unit and displays it on a medical image display unit, and displays a detection time when the attention area is detected on the detection time display unit as a detection time marker;
a display unit that displays the medical image and the detection time marker ;
The endoscope system , wherein the display control unit displays the detection time marker on the detection time display unit during real-time display in which the medical images sequentially acquired by the medical image acquisition unit are displayed on the medical image display unit . a step in which the medical image acquisition unit acquires a medical image by imaging the observation target with an imaging unit provided at the distal end of the insertion section of the endoscope ;
a step in which a region-of-interest detection unit detects a region of interest within the observation target in the medical image acquired by the medical image acquisition unit;
a step in which the display control unit acquires the medical image by the medical image acquisition unit and displays it on the medical image display unit, and displays the detection time at which the attention area is detected on the detection time display unit as a detection time marker ; and displaying the detection time marker on the detection time display unit during real-time display of the medical images sequentially acquired by the medical image acquisition unit on the medical image display unit. method of operation.

Images