JPWO2020008834A1 - Image processing equipment, methods and endoscopic systems - Google Patents

Abstract

Provided are an image processing device, a method, and an endoscope system capable of improving recognition accuracy based on a plurality of images and presenting good observation images and recognition results. A recognizer (15) that accepts an image set consisting of a plurality of images sequentially acquired using a plurality of different observation lights and outputs a recognition result for the image set, and a part or a plurality of images of the plurality of images are used. The display control unit (66) of the endoscope processor (13) for displaying the calculated observation image and the recognition result on the display unit (14) is provided. The recognizer (15) receives image sets sequentially acquired using a plurality of different observation lights, and acquires a recognition result for the image set, based on one image acquired by one observation light. The recognition accuracy can be improved as compared with the case of recognizing the image.

Description

The present invention relates to image processing devices, methods and endoscopic systems, and particularly to techniques available to assist physicians in endoscopy.

In the medical field, examinations using an endoscopic device are performed. In recent years, it has been known that image analysis recognizes the position of a lesion and the type of lesion included in an endoscopic image, and notifies the recognition result to support an examination.

In image analysis for recognition, machine learning of images such as deep learning is widely used.

In Patent Document 1, a acquisition unit that acquires a plurality of images of cells taken in chronological order and a predetermined one or more evaluation items for each of the acquired plurality of images are arranged in chronological order. An information processing apparatus including an imparting unit for imparting an evaluation value according to the time series and an evaluation unit for evaluating cells based on a time change of the evaluation value according to the given time series has been proposed. Here, the evaluation unit assigns evaluation values along the time series to a plurality of images of cells taken along the time series according to a machine learning algorithm, and based on the time change of the given evaluation values. We are evaluating the cells to be observed. This makes it possible to comprehensively consider the time-series evaluation of cells.

Further, in Patent Document 2, time-series input data which is a data string in a moving image is acquired, and a plurality of input values corresponding to the input data at one time in the time-series input data correspond to the time-series input data. Multiple nodes supplied to multiple nodes of the trained model (models that make up the Boltzmann machine), corresponding to the input data series before the prediction target time in the time series input data and the input data in the input data series in the model. Based on the weighting parameters between each of the input values of and each of the plurality of nodes, the conditional probability of each input value corresponding to the prediction target time point under the condition where the input data series is generated is calculated and predicted. A processing device has been proposed that calculates the conditional probability that the next input data will be a predetermined value under the condition that the time series input data is generated, based on the conditional probability of each input value corresponding to the target time point. ing.

As an example, this processing device can predict one image data arranged at the next time based on T-1 image data arranged in a time series, and generate a moving image including a total of T images. it can.

Japanese Unexamined Patent Publication No. 2018-22216 Japanese Unexamined Patent Publication No. 2016-71697

The information processing apparatus described in Patent Document 1 evaluates various changes in the culture process of cells (fertilized eggs) to be imaged by a plurality of images captured in time series, and a plurality of images. The image was taken under the same imaging conditions. This is because changes in fertilized eggs cannot be evaluated from a plurality of acquired images unless images are taken under the same imaging conditions. That is, the plurality of images are not images sequentially acquired using different observation lights.

The processing apparatus described in Patent Document 2 enables prediction of image data at the next time by a trained model that inputs time-series input data, and the time-series input data is imaged under the same imaging conditions. It is a thing. This is because the image data at the next time cannot be predicted from the input time-series input data unless the image is taken under the same imaging conditions. That is, the time-series input data is not input data sequentially acquired using different observation lights.

Further, in each of the inventions described in Patent Documents 1 and 2, a plurality of time-series images are input in order to predict an object (cell, future moving image) that changes with time, and the recognition accuracy by the recognizer can be improved. It is not intended to input multiple images for the purpose of improvement.

The present invention has been made in view of such circumstances, and is an image processing apparatus capable of improving recognition accuracy based on a plurality of images and presenting a good observation image and recognition result. It is an object of the present invention to provide a method and an endoscopic system.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention accepts an image set consisting of a plurality of images sequentially acquired using a plurality of different observation lights, and outputs a recognition result for the image set. It includes a recognizer and a display control unit that displays an observation image calculated using a part or a plurality of images of the plurality of images and a recognition result on the display unit.

According to one aspect of the present invention, one image acquired by one observation light is used to input image sets sequentially acquired using a plurality of different observation lights and acquire a recognition result for the image set. Based on the above, the recognition accuracy can be improved as compared with the case where the image is recognized. Further, by displaying the recognition result on the display unit together with the observation image obtained from the plurality of images, the recognition result can be appropriately presented.

In the image processing apparatus according to another aspect of the present invention, the recognizer has a learned model learned by setting a plurality of images for learning and correct answer data as a set, and each time a plurality of images for recognition are received. It is preferable to output the recognition result based on the trained model.

In the image processing apparatus according to still another aspect of the present invention, the trained model is preferably composed of a convolutional neural network. Convolutional neural networks are excellent at recognizing images.

In the image processing apparatus according to still another aspect of the present invention, the plurality of images are the first endoscopic image and the second endoscopic image acquired by using an observation light different from the first endoscopic image. It is preferable to include it. In endoscopy, a plurality of images may be acquired using a plurality of different observation lights, and this can be applied to endoscopy in this case.

In the image processing apparatus according to still another aspect of the present invention, the first endoscopic image is a normal light image captured by normal light, and the second endoscopic image is a special light imaged by special light. It is preferably an image. Generally, a normal optical image is used as an observation image, and a special optical image is used when it is desired to observe a surface structure.

In the image processing apparatus according to still another aspect of the present invention, the special light image includes two or more special light images captured by two or more different special lights. Two or more special optical images can be captured depending on the purpose of observation, such as when the depth of the surface structure to be observed is different.

In the image processing apparatus according to still another aspect of the present invention, the first endoscopic image is a first special light image captured by the first special light, and the second endoscopic image is the first special light. This is a second special light image captured by a second special light different from the above. That is, the plurality of endoscopic images may not include a normal optical image.

In the image processing apparatus according to still another aspect of the present invention, it is preferable that the display control unit displays an observation image calculated by using a part or a plurality of images of the plurality of images on the display unit as a moving image. .. This enables real-time inspection while viewing the observation image displayed as a moving image and the recognition result.

In the image processing apparatus according to still another aspect of the present invention, the recognizer recognizes a region of interest included in a plurality of images, and the display control unit displays an index indicating the recognized region of interest on the display unit. It is preferable to superimpose and display it on the image. As a result, it is possible to support the inspection so that the region of interest in the observation image is not overlooked.

In the image processing apparatus according to still another aspect of the present invention, the recognizer recognizes a region of interest included in a plurality of images, and the display control unit displays information indicating the presence or absence of the region of interest on the display unit. It is preferable to display it so that it does not overlap with. As a result, it is possible to notify that the region of interest exists in the observation image, and it is possible to prevent the observation of the observation image from being hindered by the information displayed on the display unit.

In the image processing apparatus according to still another aspect of the present invention, the recognizer executes the discrimination related to the lesion based on a plurality of images and outputs the discrimination result, and the display control unit displays the discrimination result on the display unit. Is preferable. This makes it possible to visually inspect the observation image while referring to the discrimination result by the recognizer.

The endoscope system according to still another aspect of the present invention is a light source device that sequentially generates a first observation light and a second observation light different from the first observation light, and sequentially by the first observation light and the second observation light. The endoscope includes an endoscope scope that captures a plurality of images by sequentially capturing an illuminated observation target, a display unit, and the above-mentioned image processing device, and the recognizer includes a plurality of images captured by the endoscope scope. Accepts an image set consisting of images.

The endoscope system according to still another aspect of the present invention includes an endoscope processor that accepts a plurality of images captured by an endoscope scope and performs image processing of the plurality of images, and the recognizer is an endoscope. It is preferable to accept a plurality of images after image processing by the processor. The endoscope processor has a function of image processing a plurality of images captured by the endoscope, and the recognizer can detect and distinguish the lesion area using the plurality of images after the image processing. it can. The recognizer may be a separate body from the endoscope processor, or may be built in the endoscope processor.

In the image processing method according to still another aspect of the present invention, the first step of accepting an image set consisting of a plurality of images acquired by using a plurality of different observation lights, and the recognizer outputs the recognition result for the image set. The first step includes a second step of displaying an observation image and a recognition result calculated by the display control unit using a part or a plurality of images of the plurality of images on the display unit. The process of the third step is repeatedly executed from.

In the image processing method according to still another aspect of the present invention, the second step is to learn each time a recognizer having a learned model learned by the image set for learning and the correct answer data receives the image set for recognition. It is preferable to output the recognition result based on the completed model.

In the image processing method according to still another aspect of the present invention, the trained model is preferably composed of a convolutional neural network.

In the image processing method according to still another aspect of the present invention, the plurality of images are the first endoscopic image and the second endoscopic image acquired by using an observation light different from the first endoscopic image. It is preferable to include it.

In the image processing method according to still another aspect of the present invention, the first endoscopic image is a normal light image captured by normal light, and the second endoscopic image is a special light imaged by special light. It is preferably an image.

According to the present invention, since the image set is recognized based on the image set consisting of a plurality of images sequentially acquired by using a plurality of different observation lights, the recognition accuracy can be improved. Further, by displaying the recognition result on the display unit together with the observation image obtained from the plurality of images, the recognition result can be appropriately presented.

FIG. 1 is a perspective view showing the appearance of the endoscope system 10 according to the present invention. FIG. 2 is a block diagram showing an electrical configuration of the endoscope system 10. FIG. 3 is a diagram showing an example of a multi-frame image and an image set mainly captured in the multi-frame shooting mode. FIG. 4 is a schematic diagram showing a typical configuration example of a convolutional neural network, which is one of the learning models constituting the recognizer 15. FIG. 5 is a schematic view showing a configuration example of the intermediate layer 15B of the CNN 15 shown in FIG. FIG. 6 is a block diagram showing a main configuration used for explaining the operation of the endoscope system 10 according to the present invention. FIG. 7 is a diagram showing an example of an R image, a G image, a B image, and a V image and an image set captured in a surface sequence. FIG. 8 is a flowchart showing an embodiment of the image processing method according to the present invention.

Hereinafter, preferred embodiments of the image processing apparatus, method, and endoscopic system according to the present invention will be described with reference to the accompanying drawings.

[Overall configuration of the endoscope system]
FIG. 1 is a perspective view showing the appearance of the endoscope system 10 according to the present invention.

As shown in FIG. 1, the endoscope system 10 mainly includes an endoscope scope (here, a flexible endoscope) 11 for imaging an observation target in a subject, a light source device 12, an endoscope processor 13, and an endoscope processor 13. It is composed of a display unit (display) 14 such as a liquid crystal monitor and a recognizer 15.

The light source device 12 supplies various observation lights such as white light for capturing a normal light image and light in a specific wavelength band for capturing a special light image to the endoscope scope 11.

The endoscope processor 13 has an image processing function and a light source that generate image data of a normal light image for display / recording, a special light image, or an observation image based on an image signal obtained by the endoscope scope 11. It has a function of controlling the device 12, a normal image or an image for observation, a function of displaying the recognition result by the recognizer 15 on the display 14, and the like. Although the details of the recognizer 15 will be described later, the endoscope processor 13 receives the endoscopic image, and the position detection and lesion of the region of interest (lesion, surgical scar, treatment scar, treatment tool, etc.) with respect to the endoscopic image It is a part that recognizes the type of discrimination.

The display 14 displays a normal image, a special light image or an observation image, and a recognition result by the recognizer 15 based on the display image data input from the endoscope processor 13.

The endoscope scope 11 is connected to a flexible insertion portion 16 to be inserted into the subject and a base end portion of the insertion portion 16, and is used for gripping the endoscope scope 11 and operating the insertion portion 16. The hand operation unit 17 is provided, and a universal cord 18 for connecting the hand operation unit 17 to the light source device 12 and the endoscope processor 13 is provided.

An illumination lens 42, an objective lens 44, an image pickup element 45, and the like are built in the insertion portion tip portion 16a, which is the tip portion of the insertion portion 16 (see FIG. 2). A bendable portion 16b is continuously provided at the rear end of the insertion portion tip portion 16a. Further, a flexible tube portion 16c having flexibility is continuously provided at the rear end of the curved portion 16b.

The hand operation unit 17 is provided with an angle knob 21, an operation button 22, a forceps inlet 23, and the like. The angle knob 21 is rotated when adjusting the bending direction and bending amount of the bending portion 16b. The operation button 22 is used for various operations such as air supply / water supply and suction. The forceps inlet 23 communicates with the forceps channel in the insertion portion 16. Further, the hand operation unit 17 is provided with an endoscope operation unit 46 (see FIG. 2) and the like for making various settings.

The universal cord 18 incorporates an air supply / water supply channel, a signal cable, a light guide, and the like. At the tip of the universal cord 18, a connector portion 25a connected to the light source device 12 and a connector portion 25b connected to the endoscope processor 13 are provided. As a result, observation light is supplied from the light source device 12 to the endoscope scope 11 via the connector portion 25a, and the image signal obtained by the endoscope scope 11 is input to the endoscope processor 13 via the connector portion 25b. Will be done.

The light source device 12 is provided with a light source operation unit 12a such as a power button, a lighting button for lighting the light source, and a brightness adjustment button, and the endoscope processor 13 is provided with a power button, a mouse (not shown), and the like. A processor operation unit 13a including an input unit that receives an input from the pointing device of the above is provided. Although the endoscope processor 13 and the light source device 12 of this example are separate types, the endoscope processor may be a built-in type of the light source device.

[Electrical configuration of endoscopic system]
FIG. 2 is a block diagram showing an electrical configuration of the endoscope system 10.

As shown in FIG. 2, the endoscope scope 11 is roughly classified into a light guide 40, an illumination lens 42, an objective lens 44, an image pickup element 45, an endoscope operation unit 46, and an endoscope control unit 47. , ROM (Read Only Memory) 48.

As the light guide 40, a large-diameter optical fiber, a bundle fiber, or the like is used. The incident end of the light guide 40 is inserted into the light source device 12 via the connector portion 25a, and the emitting end thereof passes through the insertion portion 16 and faces the illumination lens 42 provided in the insertion portion tip portion 16a. ing. The illumination light supplied from the light source device 12 to the light guide 40 is applied to the observation target through the illumination lens 42. Then, the illumination light reflected and / or scattered by the observation target is incident on the objective lens 44.

The objective lens 44 forms an image of the reflected light or scattered light (that is, an optical image to be observed) of the incident illumination light on the image pickup surface of the image pickup element 45.

The image pickup device 45 is a CMOS (complementary metal oxide semiconductor) type or CCD (charge coupled device) type image pickup device, and is positioned and fixed relative to the objective lens 44 at a position behind the objective lens 44. On the image pickup surface of the image pickup device 45, a plurality of pixels composed of a plurality of photoelectric conversion elements (photodiodes) that photoelectrically convert an optical image are two-dimensionally arranged. Further, red (R), green (G), and blue (B) color filters are arranged for each pixel on the incident surface side of the plurality of pixels of the image pickup element 45 of this example, whereby the R pixel and the G pixel are arranged. , B pixel is configured. The filter array of the RGB color filter is generally a Bayer array, but is not limited to this.

The image sensor 45 converts the optical image imaged by the objective lens 44 into an electrical image signal and outputs it to the endoscope processor 13.

When the image sensor 45 is a CMOS type, an A / D (Analog / Digital) converter is built in, and a digital image signal is directly output from the image sensor 45 to the endoscope processor 13. To. When the image sensor 45 is a CCD type, the image signal output from the image sensor 45 is converted into a digital image signal by an A / D converter or the like (not shown), and then transferred to the endoscope processor 13. It is output.

The endoscope operation unit 46 has a still image image pickup button (not shown), a shooting mode setting unit for setting one of a normal light image shooting mode, a special light image shooting mode, and a multi-frame shooting mode. ing. The photographing mode setting unit may be provided in the processor operation unit 13a of the endoscope processor 13.

The endoscope control unit 47 sequentially executes various programs and data read from the ROM 48 and the like in response to the operation of the endoscope operation unit 46, and mainly controls the drive of the image sensor 45. For example, in the normal optical image pickup mode, the endoscope control unit 47 controls the image pickup element 45 so as to read the signals of the R pixel, the G pixel, and the B pixel of the image pickup element 45, and the special optical image pickup mode or the multi In the frame shooting mode, when purple light is emitted from the V-LED32a as observation light to acquire a specific special light image, or when blue light is emitted from the B-LED32b, these purple colors are used. Only the signal of the B pixel of the image sensor 45 having spectral sensitivity in the wavelength band of light and blue color light is read out, or any one of the three color pixels of the R pixel, the G pixel and the B pixel, or 2 The image sensor 45 is controlled so as to read out one color pixel.

Further, the endoscope control unit 47 communicates with the processor control unit 61 of the endoscope processor 13, and the operation information in the endoscope operation unit 46 and the endoscope scope 11 stored in the ROM 48 are stored. Identification information and the like for identifying the type of the endoscope are transmitted to the endoscope processor 13.

The light source device 12 has a light source control unit 31 and a light source unit 32. The light source control unit 31 communicates between the control of the light source unit 32 and the processor control unit 61 of the endoscope processor 13 to exchange various information.

The light source unit 32 has, for example, a plurality of semiconductor light sources. In the present embodiment, the light source unit 32 includes a V-LED (Violet Light Emitting Diode) 32a, a B-LED (Blue Light Emitting Diode) 32b, a G-LED (Green Light Emitting Diode) 32c, and an R-LED (Red Light). Emitting Diode) 32d with 4 color LEDs. The V-LED32a, B-LED32b, G-LED32c, and R-LED32d are observation lights having peak wavelengths at 410 nm, 450 nm, 530 nm, and 615 nm, respectively, and are purple (V) light, blue (B) light, and so on. It is a semiconductor light source that emits green (G) light and red (R) light.

The light source control unit 31 individually controls the lighting and extinguishing of the four LEDs of the light source unit 32, the amount of light emitted at the time of lighting, and the like according to the shooting mode set by the shooting mode setting unit. In the normal optical image capturing mode, the light source control unit 31 lights all the V-LED32a, B-LED32b, G-LED32c, and R-LED32d. Therefore, in the normal optical image capturing mode, white light including V light, B light, G light, and R light is used as observation light.

On the other hand, in the special light image capturing mode, the light source control unit 31 lights any one of the V-LED32a, B-LED32b, G-LED32c, and R-LED32d, or a plurality of light sources that are appropriately combined. When the light source is turned on or a plurality of light sources are turned on, the light emission amount (light amount ratio) of each light source is controlled, which enables imaging of a plurality of layers having different depths of the subject.

The multi-frame shooting mode is a shooting mode in which a normal light image and one or more special light images are switched for each frame and shot, or two or more special light images are switched for each frame and shot. In the shooting mode, the light source control unit 31 emits different observation light from the light source unit 32 for each frame.

The light of each color emitted by each of the LEDs 32a to 32d is incident on the light guide 40 inserted into the endoscope scope 11 via an optical path coupling portion formed by a dichroic mirror, a lens, or the like, and an aperture mechanism (not shown). Will be done.

The observation light of the light source device 12 is white light (light in a white wavelength band or light in a plurality of wavelength bands), light having a peak in one or a plurality of specific wavelength bands (special light), or these. Light in various wavelength bands is selected according to the purpose of observation such as combination.

A first example of a particular wavelength band is, for example, the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the first example has a peak wavelength in the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. ..

A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the second example has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. ..

The third example of a specific wavelength band includes a wavelength band in which the absorption coefficient of hemoglobin oxide and reduced hemoglobin are different, and the light of the third example has a peak wavelength in a wavelength band in which the absorption coefficient of hemoglobin oxide and reduced hemoglobin are different. Has. The wavelength band of the third example includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and the light of the third example is 400 ± 10 nm, 440 ± 10 nm, 470. It has a peak wavelength in the wavelength band of ± 10 nm or 600 nm or more and 750 nm or less.

The fourth example of the specific wavelength band is the wavelength band (390 nm to 470 nm) of the excitation light used for observing the fluorescence emitted by the fluorescent substance in the living body (fluorescence observation) and exciting the fluorescent substance.

A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less.

The endoscope processor 13 includes a processor operation unit 13a, a processor control unit 61, a ROM 62, a digital signal processor (DSP: Digital Signal Processor) 63, an image processing unit 65, a display control unit 66, a storage unit 67, and the like. ing.

The processor operation unit 13a includes a power button, an input unit that accepts inputs such as a coordinate position indicated on the screen of the display 14 by a mouse and a click (execution instruction), and the like.

The processor control unit 61 reads necessary programs and data from the ROM 62 according to the operation information in the processor operation unit 13a and the operation information in the endoscope operation unit 46 received via the endoscope control unit 47. The sequential processing controls each part of the endoscope processor 13 and also controls the light source device 12. The processor control unit 61 may receive necessary instruction input from another external device such as a keyboard connected via an interface (not shown).

The DSP 63, which functions as a form of the image acquisition unit that acquires the image data of each frame of the moving image output from the endoscope scope 11 (imaging element 45), is the endoscope scope 11 under the control of the processor control unit 61. Various signal processing such as defect correction processing, offset processing, white balance correction, gamma correction, and demosaic processing (also referred to as "simultaneous processing") is performed on the image data for one frame of the moving image input from. Generates image data for one frame.

The image processing unit 65 inputs image data from the DSP 63, performs image processing such as color conversion processing, color enhancement processing, and structure enhancement processing on the input image data as necessary, and the observation target is captured. Generate image data showing an endoscopic image. The color conversion process is a process of converting image data by a 3 × 3 matrix process, a gradation conversion process, a three-dimensional look-up table process, or the like. The color enhancement process is a process of emphasizing the color of the image data that has undergone the color conversion process, for example, in a direction that makes a difference in the color tone between the blood vessel and the mucous membrane. The structure enhancement process is a process for emphasizing a specific tissue or structure included in an observation target such as a blood vessel or a pit pattern, and is performed on the image data after the color enhancement process.

When there is an instruction to shoot a still image or a moving image, the image data of each frame of the moving image processed by the image processing unit 65 is recorded in the storage unit 67 as a still image or a moving image instructed to be taken.

The display control unit 66 generates display data for displaying a normal light image or a special light image on the display 14 based on the image data input from the image processing unit 65, and displays the generated display data on the display 14. The display image (such as a moving image captured by the endoscope scope 11) is displayed on the display 14.

In the multi-frame shooting mode, if a plurality of images sequentially acquired using different observation lights are sequentially displayed as they are, the appearance changes and flickers. Therefore, the display control unit 66 selects one of the plurality of images. The image (a part of the image) is displayed on the display 14, or the observation image calculated by the image processing unit 65 using the plurality of images is displayed on the display 14.

Further, the display control unit 66 causes the display 14 to display the recognition result input from the recognizer 15 via the image processing unit 65 or the recognition result input from the recognizer 15.

When the area of interest is detected by the recognizer 15, the display control unit 66 superimposes and displays an index indicating the area of interest on the image displayed on the display 14. For example, highlighting such as changing the color of the region of interest in the display image, display of a marker, and display of a bounding box can be considered as indicators.

Further, the display control unit 66 can display the information indicating the presence / absence of the attention region based on the detection result of the attention region by the recognizer 15 so as not to overlap with the image displayed on the display 14. For information indicating the presence or absence of the region of interest, for example, the color of the frame of the endoscopic image may be changed depending on whether the region of interest is detected or not, or the text "There is an region of interest!" Is included. It is conceivable that the image is displayed in a display area different from that of the endoscopic image.

Further, when the recognition device 15 executes the discrimination regarding the lesion, the display control unit 66 causes the display device 14 to display the discrimination result. As a method of displaying the discrimination result, for example, display of a text representing the detection result on the display image of the display 14 can be considered. The display of the text does not have to be on the display image, and is not particularly limited as long as the correspondence with the display image is known.

[Recognizer 15]
Next, the recognizer 15 according to the present invention will be described.

The recognizer 15 receives an image after image processing by the endoscope processor 13. First, a recognition image received by the recognizer 15 will be described.

The recognizer 15 of this example is applied when the multi-frame shooting mode is set.

When the multi-frame shooting mode is set, the light source device 12 uses white light including purple light, blue light, green light, and red light, and V-LED32a, B-LED32b, G-LED32c, and R-LED32d. Light (special light) in one or a plurality of specific wavelength bands whose lighting is controlled is sequentially generated, and the endoscope processor 13 sequentially generates light under white light (normal light image) from the endoscope scope 11 and special light. Images under light (special light images) are acquired in sequence.

In the multi-frame shooting mode of this example, as shown in FIG. 3, two types of special light images, a normal light image (WL (White Light) image) which is a first endoscopic image and a second endoscopic image, are used. (BLI (Blue Light Imaging or Blue LASER Imaging) image) and LCI (Linked Color Imaging) image) are sequentially switched for each frame and repeatedly acquired.

Here, the BLI image and the LCI image are images captured by the observation light for BLI and the observation light for LCI, respectively.

The observation light for BLI is an observation light in which the ratio of V light having a high absorption rate in the surface blood vessels is high and the ratio of G light having a high absorption rate in the middle layer blood vessels is suppressed. It is suitable for generating an image (BLI image) suitable for enhancing the structure.

Further, the observation light for LCI has a higher ratio of V light than the observation light for WL, and is suitable for capturing minute color tone changes as compared with the observation light for WL. This is an image in which color enhancement processing is performed so that a reddish color becomes redder and a whitish color becomes whiter, centering on the color near the mucous membrane by using the signal of the R component.

The recognizer 15 accepts an image set Sa including a plurality of images (in this example, a WL image, a BLI image, and an LCI image) sequentially acquired by the endoscope processor 13 as images for recognition.

Since the WL image, the BLI image, and the LCI image are color images, respectively, they have an R image, a G image, and a B image (three color channels). Therefore, the image set Sa input by the recognizer 15 is an image having 9 channels (= 3 × 3).

Further, the recognizer 15 sequentially inputs the image set Sa, but since the image set Sa is composed of three consecutive frames (WL image, BLI image, and LCI image) in chronological order, each image set Sa is formed. The time interval for inputting the image corresponds to the time for three frames of each frame imaged in the multi-frame shooting mode. _{That is, the time interval between the image set Sa at the time t n} received by the recognizer 15 and the image set Sa at the previous time t _n-1 is three frames of each frame captured in the multi-frame shooting mode. Corresponds to time.

FIG. 4 is a schematic diagram showing a typical configuration example of a convolutional neural network (CNN), which is one of the learning models constituting the recognizer 15.

CNN15 is, for example, a learning model for detecting the position of a region of interest (lesion, surgical scar, treatment scar, treatment tool, etc.) shown in an endoscopic image and distinguishing the type of lesion, and has a plurality of layer structures. , Holds multiple weight parameters. The CNN 15 becomes a trained model and functions as a recognizer by setting the weight parameter to the optimum value.

As shown in FIG. 4, the CNN 15 includes an input layer 15A, an intermediate layer 15B having a plurality of convolution layers and a plurality of pooling layers, and an output layer 15C, and each layer has a plurality of "nodes" connected by "edges". It has a structure that can be used.

The CNN15 of this example is a learning model that performs segmentation to recognize the position of the region of interest in the endoscopic image, and a full-layer convolutional network (FCN), which is a type of CNN, is applied to the inside. The position of the region of interest in the endoscopic image can be grasped at the pixel level.

An image set Sa (FIG. 3) for recognition is input to the input layer 15A.

The intermediate layer 15B is a portion for extracting features from the image set Sa input from the input layer 15A. The convolution layer in the intermediate layer 15B filters the nearby nodes in the image set Sa and the previous layer (performs a convolution operation using the filter) to acquire a "feature map". The pooling layer reduces (or enlarges) the feature map output from the convolution layer to obtain a new feature map. The "convolution layer" plays a role of feature extraction such as edge extraction from an image, and the "pooling layer" plays a role of imparting robustness so that the extracted features are not affected by translation or the like. The intermediate layer 15B is not limited to the case where the convolution layer and the pooling layer are set as one set, and may include the case where the convolution layers are continuous and the normalization layer.

The output layer 15C is a portion that outputs a recognition result for detecting the position of the region of interest in the endoscopic image and classifying (distinguishing) the type of lesion based on the features extracted by the intermediate layer 15B.

Further, this CNN 15 is learned by a large number of sets of an image set Sa for learning and correct answer data for the image set Sa, and the coefficient and offset value of the filter applied to each convolution layer of the CNN 15 are learned. It is set to the optimum value by the data set for. Here, the correct answer data is preferably a region of interest or a discrimination result designated by a doctor with respect to the endoscopic image (in this example, at least one image of the image set Sa).

FIG. 5 is a schematic view showing a configuration example of the intermediate layer 15B of the CNN 15 shown in FIG.

The convolution layer of the first (1st) of the region of interest, and the image set Sa for recognition, the convolution operation of the filter F ₁ is performed. Here, the image set Sa is an N image (N channel) having an image size of H in the vertical direction and W in the horizontal direction. In this example, as shown in FIG. 3, the image set Sa is a 9-channel image.

_{Since the image set S is N channels (N sheets) in the filter F 1 that} is convolved with the image set Sa, for example, in the case of a size 5 filter, the filter size is 5 × 5 × N.

By the convolution operation using this filter F ₁ , one channel (one sheet) of "feature map" is generated for one _{filter F 1.} In the example shown in FIG. 5, the use of the M filter F _1, "feature map" of M channels is generated.

Filter F ₁ used in the second convolution layer, for example, in the case of size 3 filter, the filter size will filter 3 × 3 × M.

The size of the "feature map" in the nth convolution layer is smaller than the size of the "feature map" in the second convolution layer because it is downscaled by the convolution layers up to the previous stage.

The convolution layer in the first half of the intermediate layer 15B is responsible for extracting the features, and the convolution layer in the second half is responsible for the segmentation of the object (area of interest). The convolution layer in the latter half is upscaled, and in the last convolution layer, one "feature map" having the same size as the input image set Sa is obtained. The output layer 15C (FIG. 4) of the CNN 15 grasps the position of the region of interest in the image of the image set Sa at the pixel level by the "feature map" obtained from the intermediate layer 15B. That is, it is possible to detect whether or not each pixel of the endoscopic image belongs to the region of interest and output the detection result.

According to the present embodiment, a plurality of images (WL) sequentially acquired in the multi-frame shooting mode are compared with the case where the image is recognized by any one (one type) of the WL image, the BLI image, and the LCI image. Since recognition is performed using an image (an image set of an image, a BLI image, and an LCI image), the recognition accuracy can be improved.

Further, the CNN 15 of this example recognizes the position of the region of interest in the endoscopic image, but the recognizer (CNN) according to the present invention is not limited to this, and discriminates the lesion. The discrimination result may be output. For example, the recognizer classifies endoscopic images into three categories: "neoplastic," "non-neoplastic," and "other," and the discrimination results are "neoplastic," "non-neoplastic," and "other." It may be output as three scores corresponding to (the total of the three scores is 100%), or if it can be clearly classified from the three scores, the classification result may be output. Further, in the case of a CNN that outputs such a discrimination result, it is preferable that the CNN has a fully connected layer as the last one or a plurality of layers of the intermediate layer instead of the full-layer convolutional network (FCN).

[Action of endoscopic system]
FIG. 6 is a block diagram showing a main configuration used for explaining the operation of the endoscope system 10 according to the present invention.

From the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d of the light source unit 32, observation lights (V light, B light, G light, and R light) having different peak wavelengths are emitted from the light guide 40. The subject 20 is irradiated with the light. Since the V light, the B light, the G light, and the R light each reach a plurality of layers having different depths of the subject 20, it is possible to capture images having different depths of the subject 20 by these observation lights.

As described with reference to FIG. 3, in the multi-frame shooting mode, WL is performed by a plurality of different observation lights (for example, the first observation light for WL, the second observation light for BLI, and the third observation light for LCI). Images, BLI images, and LCI images are acquired in sequence, but the observation lights for WL, BLI, and LCI have different light intensity ratios of V light, B light, G light, and R light as described above. Is.

In the endoscope scope 11, a WL image, a BLI image, and an LCI image are sequentially and repeatedly imaged by irradiation with a plurality of different observation lights. Since the WL image, the BLI image, and the LCI image are color images, the endoscope processor 13 generates RGB 3-channel WL images, BLI images, and LCI images.

The recognizer 15 accepts an image set Sa (a total of 9 channels of images) including a WL image, a BLI image, and an LCI image as an image for recognition.

The recognizer 15 detects the position of the region of interest (in this example, the lesion region) shown in the endoscopic image, and outputs the position information (recognition result) indicating the lesion region to the endoscope processor 13.

The image processing unit 65 of the endoscope processor 13 generates a WL image, a BLI image, and an LCI image from the image signal input from the endoscope scope 11, and also generates an observation image. As the observation image, a part of a plurality of images (for example, a WL image among a WL image, a BLI image, and an LCI image) may be used as an observation image, or an image calculated using the plurality of images (WL image). , A composite image of two or more images of a BLI image and an LCI image) may be used as an observation image. If a plurality of images sequentially acquired using different observation lights are sequentially displayed as observation images as they are, the appearance changes and flickers. Therefore, the observation image is preferably one type of image.

The display control unit 66 inputs an observation image from the image processing unit 65, inputs position information indicating a lesion area from the recognizer 15, and displays the observation image and the recognition result on the display 14.

In this example, the display control unit 66 displays the observation image 26 on the display 14 and performs an enhancement process for emphasizing the recognized region of interest (lesion region). The highlighting process by the display control unit 66 highlights the lesion area by superimposing the index 28 indicating the lesion area on the observation image 26 displayed on the display 14. Here, as the display of the index 28, in addition to the highlighting such as changing the color of the lesion area and the display of the boundary line indicating the outline of the lesion area, the display of the marker indicating the lesion area and the display of the bounding box can be considered.

In this way, by superimposing the index 28 indicating the region of interest on the observation image 26 displayed on the display 14, it is possible to support the inspection so that the region of interest is not overlooked.

The recognizer 15 of this example recognizes the position of the region of interest in the endoscopic image, but is not limited to this, and may be one that executes discrimination related to the lesion and outputs the discrimination result. .. As a method of displaying the discrimination result, for example, a method of displaying a text representing the discrimination result on the image of the display 14 can be considered. The display position of the text does not have to be on the image, and may be a window different from the image as long as the correspondence with the image is known, and is not particularly limited.

[Other embodiments of multi-frame photography]
When a color endoscopic image is acquired by an endoscope scope equipped with a monochrome image sensor that does not have a color filter instead of the image sensor 45 (color image sensor), the subject is subjected to observation light of a different color. Illuminate sequentially, and image images are taken for each observation light (images are taken in surface order).

For example, by sequentially emitting observation light (R light, G light, B light, and V light) of different colors from the light source unit 32, the monochrome imaging element converts the observation light into R light, G light, B light, and V light. The R image, G image, B image, and V image of the corresponding colors are imaged surface-sequentially.

FIG. 7 is a diagram showing an example of an R image, a G image, a B image, and a V image and an image set captured in a surface-sequential manner.

The endoscope processor 13 sequentially acquires a plurality of images (R image, G image, B image, and V image) using a plurality of different observation lights (R light, G light, B light, and V light). It is possible to generate an observation image such as a WL image, a BLI image, and an LCI image based on the above. These observation images can be generated by adjusting the composition ratio of the R image, the G image, the B image, and the V image.

Further, an image obtained by multiplying at least two images of the R image, the G image, the B image, and the V image by a preset coefficient (four-rule calculation) may be included in the image set. For example, an image obtained by dividing each pixel by an image (V image) having a central wavelength of 410 nm and an image (B image) having a central wavelength of 450 nm, or an image (V image) having a central wavelength of 410 nm and an image having a central wavelength of 450 nm (B). An image obtained by multiplying each pixel by (image) may be used.

The recognizer 15 can accept the WL image, the BLI image, and the LCI image generated by the endoscope processor 13 as an image set Sb, and return the recognition result for the endoscope image to the endoscope processor 13.

The recognizer 15 of this example accepts an image set Sa (total, 9-channel image) composed of a WL image, a BLI image, and an LCI image as an image for recognition, but is not limited to this, for example, the above R image. An image set including a G image, a B image, and a V image may be accepted and the recognition result for the endoscopic image may be output.

[Image processing method]
FIG. 8 is a flowchart showing an embodiment of the image processing method according to the present invention, and shows the processing procedure of each part of the endoscope system 10 shown in FIG.

In FIG. 8, the multi-frame shooting mode is set, and the endoscope scope 11 sequentially captures multi-frame images using a plurality of different observation lights (step S10).

The endoscope processor 13 acquires an image set constituting a multi-frame image captured by the endoscope scope 11 (step S12, first step).

The image set is a WL image, a BLI image and an LCI image captured by observation light for WL, BLI, and LCI captured by the endoscope scope 11, or an R image or G image captured in a surface sequence. , B image, and WL image, BLI image, and LCI image generated from the V image can be considered, but the special light image may be any one of the BLI image and the LCI image, and the other. It may be a special light image captured by the special light of. Further, the image set does not include a WL image (normal light image), and includes two or more special light images including a first special light image captured by the first special light and a second special light image captured by the second special light. It may be an optical image. In short, any image set may be used as long as it is an image set consisting of a plurality of images sequentially acquired using a plurality of different observation lights.

The image processing unit 65 of the endoscope processor 13 generates an observation image based on the acquired image set (step S14). The observation image is a part of a plurality of images (for example, a WL image, a BLI image, and a WL image among LCI images) or an image calculated by using the plurality of images.

On the other hand, the recognizer 15 detects the position of the region of interest in the endoscopic image based on the image set received via the endoscope processor 13, discriminates the type of lesion, and outputs the recognition result. (Step S16, second step).

Then, the display control unit 66 displays the generated observation image and the recognition result by the recognizer 15 on the display 14 (step S18, third step).

Subsequently, it is determined whether or not to end the imaging of the multi-frame image (step S20), and if the imaging of the multi-frame image is continued (in the case of "No"), the process proceeds to step S10 and step S10. The process of step S20 is repeated. As a result, the observation image is displayed as a moving image, and the recognition result of the recognizer 15 is also continuously displayed.

When the imaging of the multi-frame image is completed (in the case of "Yes"), this process is terminated.

[Other]
In the present embodiment, the endoscope system 10 including the endoscope scope 11 and the like has been described, but the present invention is not limited to the endoscope system 10, but includes an endoscope processor 13 and a recognizer 15. An image processing device may be used. In this case, the endoscope processor 13 and the recognizer 15 may be integrated or separate.

Further, the different observation lights are not limited to those emitted from the four-color LEDs. For example, a blue laser diode that emits a blue laser light having a center wavelength of 445 nm and a blue-purple laser diode that emits a blue-purple laser light having a center wavelength of 405 nm. A YAG (Yttrium Aluminum Garnet) -based phosphor may be irradiated with the laser light of these blue laser diodes and the blue-violet laser diodes to emit light. By irradiating this phosphor with blue laser light, the phosphor is excited to emit wide-band fluorescence, and some of the blue laser light passes through the phosphor as it is. The bluish-purple laser beam is transmitted without exciting the phosphor. Therefore, by adjusting the intensity of the blue laser light and the blue-violet laser light, it is possible to irradiate the observation light for WL, the observation light for BLI, and the observation light for LCI, and the blue-purple laser light. When only the light is emitted, it is possible to irradiate the observation light having a center wavelength of 405 nm.

Further, the observation image according to the present invention is not limited to a moving image, but may be a still image stored in a storage unit 67 or the like, and the recognizer may output a recognition result based on an image set of the still image.

Further, the recognizer is not limited to CNN, and may be a machine learning model other than CNN such as DBN (Deep Belief Network) and SVM (Support Vector Machine).

The hardware structure of the endoscope processor 13 and / or the recognizer 15 is various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various control units. Includes a dedicated electric circuit, which is a processor having a circuit configuration specially designed to execute a specific process such as a programmable logic device (PLD) and an ASIC (Application Specific Integrated Circuit). Is done.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which the processor functions as a plurality of control units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of control units with one IC (Integrated Circuit) chip is used. is there. As described above, the various control units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

10 Endoscope system 11 Endoscope scope 12 Light source device 12a Light source operation unit 13 Endoscope processor 13a Processor operation unit 14 Display 15 Recognizer (CNN)
15A Input layer 15B Intermediate layer 15C Output layer 16 Insertion part 16a Insertion part tip 16b Curved part 16c Flexible tube part 17 Hand operation part 18 Universal cord 20 Subject 21 Angle knob 22 Operation button 23 Forceps inlet 25a Connector part 25b Connector part 26 Observation image 28 Index 31 Light source control unit 32 Light source unit 32a V-LED
32b B-LED
32c G-LED
32d R-LED
40 Light guide 42 Illumination lens 44 Objective lens 45 Image sensor 46 Endoscope operation unit 47 Endoscope control unit 48, 62 ROM
61 Processor control unit 65 Image processing unit 66 Display control unit 67 Storage unit F1 Filter S Image set S10 Step S12 Step S14 Step S16 Step S18 Step S20 Step

Claims (18)

A recognizer that accepts an image set consisting of a plurality of images sequentially acquired using a plurality of different observation lights and outputs a recognition result for the image set.
An observation image calculated by using a part of the plurality of images or the plurality of images, and a display control unit for displaying the recognition result on the display unit.
Image processing device equipped with. The recognizer has a learned model learned by setting the plurality of images for learning and correct answer data as a set, and each time the plurality of images for recognition are received, the recognition result is based on the learned model. The image processing apparatus according to claim 1. The image processing apparatus according to claim 2, wherein the trained model is composed of a convolutional neural network. The plurality of images according to any one of claims 1 to 3, which include a first endoscopic image and a second endoscopic image acquired by using an observation light different from the first endoscopic image. The image processing apparatus described. The image processing apparatus according to claim 4, wherein the first endoscope image is a normal light image captured by normal light, and the second endoscope image is a special light image captured by special light. .. The image processing apparatus according to claim 5, wherein the special light image includes two or more special light images captured by two or more different special lights. The first endoscopic image is a first special light image captured by the first special light, and the second endoscopic image is captured by a second special light different from the first special light. The image processing apparatus according to claim 4, which is a second special optical image. The item according to any one of claims 1 to 6, wherein the display control unit displays a part of the plurality of images or an observation image calculated by using the plurality of images on the display unit as a moving image. Image processing device. The recognizer recognizes a region of interest contained in the plurality of images and recognizes the region of interest.
The image processing device according to any one of claims 1 to 8, wherein the display control unit superimposes and displays the recognized index indicating the area of interest on the image displayed on the display unit. The recognizer recognizes a region of interest contained in the plurality of images and recognizes the region of interest.
The image processing device according to any one of claims 1 to 8, wherein the display control unit displays information indicating the presence or absence of the region of interest so as not to overlap the image displayed on the display unit. The recognizer executes the discrimination regarding the lesion based on the plurality of images and outputs the discrimination result.
The image processing device according to any one of claims 1 to 10, wherein the display control unit displays the discrimination result on the display unit. A light source device that sequentially generates the first observation light and the second observation light different from the first observation light,
An endoscope scope that captures a plurality of images by sequentially capturing an observation object that is sequentially illuminated by the first observation light and the second observation light.
With the display unit
The image processing apparatus according to any one of claims 1 to 11 is provided.
The recognizer is an endoscope system that accepts the image set including the plurality of images captured by the endoscope. An endoscope processor that receives the plurality of images captured by the endoscope and performs image processing on the plurality of images is provided.
The endoscope system according to claim 12, wherein the recognizer receives the plurality of images after image processing by the endoscope processor. The first step of accepting an image set consisting of a plurality of images acquired using a plurality of different observation lights, and
The second step in which the recognizer outputs the recognition result for the image set, and
The display control unit includes a part of the plurality of images or a third step of displaying the observation image calculated by using the plurality of images and the recognition result on the display unit.
An image processing method in which the processes of the first step to the third step are repeatedly executed. In the second step, each time the recognizer having the trained model learned by the image set for training and the correct answer data receives the image set for recognition, the recognition result is obtained based on the trained model. The image processing method according to claim 14, which is to be output. The image processing method according to claim 15, wherein the trained model is composed of a convolutional neural network. The plurality of images according to any one of claims 14 to 16, including the first endoscopic image and the second endoscopic image acquired by using an observation light different from the first endoscopic image. The image processing method described. The image processing method according to claim 17, wherein the first endoscope image is a normal light image captured by normal light, and the second endoscope image is a special light image captured by special light. ..

Images