WO2024195100A1

WO2024195100A1 - Endoscopy assistance device, endoscopy assistance method, and recording medium

Info

Publication number: WO2024195100A1
Application number: PCT/JP2023/011446
Authority: WO
Inventors: 弘泰齊賀
Original assignee: 日本電気株式会社
Priority date: 2023-03-23
Filing date: 2023-03-23
Publication date: 2024-09-26

Abstract

In this endoscopy assistance device, an image acquisition means acquires an endoscopic image taken by an endoscopic camera. A detection means detects a polyp from the endoscopic image. A three-dimensional restoration means restores a three-dimensional point cloud, for an imaged site, from the endoscopic image. A major axis calculation means calculates the major axis of the polyp on the basis of the three-dimensional point cloud and the polyp. A display control means causes a display device to display a display image including the value of the major axis.

Description

Endoscopic examination support device, endoscopic examination support method, and recording medium

This disclosure relates to technology that supports endoscopic examinations.

In endoscopic examinations, determining the size of polyps found is important in order to determine whether or not they need to be removed, the method of removal, and surveillance. Patent Document 1 discloses a method of displaying a cross line on an image captured by an endoscope and displaying a scale indicating the actual size on the cross line.

International Publication No. WO2019/017019

However, Patent Document 1 merely displays intersecting lines with scale marks near polyps, etc., and requires a doctor or other examiner to visually determine the size of the polyp.

One objective of this disclosure is to estimate and display the size of polyps from endoscopic images.

According to one aspect of the present disclosure, an endoscopic examination support apparatus includes:
an image acquisition means for acquiring an endoscopic image captured by an endoscopic camera;
a detection means for detecting a polyp from the endoscopic image;
a three-dimensional reconstruction means for reconstructing a three-dimensional point cloud of an imaging region from the endoscopic image;
a major axis calculation means for calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
and a display control means for causing a display image including the value of the major axis to be displayed on a display device.

In another aspect of the present disclosure, a method for assisting endoscopic examination is provided, the method being executed by a computer and comprising:
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
A display image including the value of the major axis is displayed on a display device.

According to yet another aspect of the present disclosure, a recording medium includes:
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
A program for causing a computer to execute a process for displaying a display image including the value of the major axis on a display device is recorded.

According to this disclosure, it is possible to estimate and display the size of polyps from endoscopic images.

1 is a block diagram showing a schematic configuration of an endoscopic inspection system. 2 is a block diagram showing a hardware configuration of the endoscopic examination support device. FIG. 2 is a block diagram showing the functional configuration of the endoscopic examination support device. FIG. 13 is a flowchart of a polyp information display process. 1 shows a first display example of a display image. 11 shows a second display example of a display image. 13 shows a third display example of a display image. 13 shows a fourth display example of a display image. 13 shows a fifth display example of a display image. 13 shows a sixth display example of a display image. 13 shows a seventh display example of a display image. FIG. 11 is a block diagram showing the functional configuration of an endoscopic examination support device according to a second embodiment. 10 is a flowchart of a process performed by the endoscopic examination support device of the second embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
First Embodiment
[System configuration]
Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 estimates and displays the size of a polyp from an endoscopic image during examination (including treatment) using an endoscope.

As shown in FIG. 1, the endoscopic examination system 100 mainly comprises an endoscopic examination support device 1, a display device 2, and an endoscope scope 3 connected to the endoscopic examination support device 1.

The endoscopic examination support device 1 acquires from the endoscope scope 3 images (i.e., video images; hereinafter, also referred to as "endoscopic images Ic") captured by the endoscope scope 3 during an endoscopic examination, and displays on the display device 2 display data for confirmation by the examiner performing the endoscopic examination. Specifically, the endoscopic examination support device 1 acquires video images of the large intestine captured by the endoscope scope 3 during an endoscopic examination as the endoscopic images Ic. The endoscopic examination support device 1 extracts frame images from the endoscopic images Ic, estimates the size of the polyps based on the frame images, and displays information regarding the size of the polyps on the display device 2 together with the endoscopic images.

The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the endoscopic examination support device 1.

The endoscope scope 3 mainly comprises an operation section 36 that allows the examiner to input instructions such as air supply, water supply, angle adjustment, and imaging instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip section 38 that incorporates an imaging section such as a miniature imaging element, and a connection section 39 for connecting to the endoscopic examination support device 1. The imaging section provided in the endoscope scope 3 will be referred to as the "endoscopic camera" below.

[Hardware configuration]
2 shows a hardware configuration of the endoscopic examination support device 1. The endoscopic examination support device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter, referred to as "DB") 17. These elements are connected via a data bus 19.

The processor 11 executes a predetermined process by executing a program stored in the memory 12. The processor 11 may be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these. The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for the processing of the endoscopic examination support device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs that allow the endoscopic examination support device 1 to execute each process in this embodiment.

In addition, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope scope 3 during an endoscopic examination.

The interface 13 performs interface operations between the endoscopic examination support device 1 and an external device. For example, the interface 13 supplies the display data Id generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope scope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ic supplied from the endoscope scope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

The input unit 14 generates an input signal based on the examiner's operation. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

DB17 stores endoscopic images and the like acquired during past endoscopic examinations of the subject. DB17 may include an external storage device such as a hard disk connected to or built into the endoscopic examination support device 1, or may include a removable storage medium such as a flash memory. Instead of providing DB17 within the endoscopic examination system 100, DB17 may be provided on an external server, etc., and related information may be acquired from the server via communication.

The endoscopic examination support device 1 may also be equipped with a sensor capable of measuring the rotation and translation of the endoscopic camera, such as a magnetic sensor.

[Functional configuration]
3 is a block diagram showing the functional configuration of the endoscopic examination support device 1. The endoscopic examination support device 1 functionally includes an interface 13, a depth estimation unit 21, a three-dimensional reconstruction unit 22, a lesion detection unit 23, a three-dimensional reconstruction unit 24, a polyp region mapping unit 25, a polyp long axis calculation unit 26, and a display image generation unit 27.

The endoscopic examination support device 1 receives an endoscopic video Ic from the endoscope scope 3. The endoscopic video Ic is input to the interface 13. The interface 13 extracts frame images (hereinafter also referred to as "endoscopic images") from the input endoscopic video Ic, and outputs them to the depth estimation unit 21, the lesion detection unit 23, and the display image generation unit 27.

The depth estimation unit 21 receives an endoscopic image from the interface 13. The depth estimation unit 21 estimates the depth from the endoscopic camera (i.e., the tip of the endoscopic scope 3) corresponding to each pixel contained in the endoscopic image from the endoscopic image, and generates depth information. "Depth" is the distance from the endoscopic camera to each pixel in the endoscopic image, and "depth information" is information indicating the depth for each pixel in the endoscopic image. Note that the depth information corresponding to one endoscopic image is also called a "depth image."

For example, image processing techniques such as Structure from Motion (hereinafter, referred to as "SfM") or machine learning techniques such as deep learning can be applied to estimate depth. As a machine learning technique, a monocular depth estimation technique that estimates depth on a pixel-by-pixel basis for one frame image may be used. In the machine learning technique, a depth estimation model that has been trained in advance is used. In this case, as a technique for training the depth estimation model, a supervised learning technique in which correct depth data is given, or a self-supervised learning technique in which learning is performed using video data as input can be used. The depth estimation unit 21 outputs the generated depth information to the 3D restoration units 22 and 24.

The 3D restoration unit 22 generates a 3D point cloud using the depth information input from the depth estimation unit 21 and the internal parameters of the endoscopic camera provided in the endoscope scope 3. That is, the 3D restoration unit 22 restores the 2D endoscopic image to a 3D point cloud in a 3D coordinate system. The endoscopic image is a 3D shape of the photographed area in the large intestine that has been converted into a 2D shape using the internal parameters of the endoscopic camera. Therefore, the 3D restoration unit 22 restores the endoscopic image to its original 3D shape (3D point cloud) using the internal parameters of the camera and the depth information of the endoscopic image. At this time, the internal parameters of the camera may be treated as known, or may be estimated from the endoscopic image.

The three-dimensional restoration unit 22 may perform three-dimensional restoration using multiple endoscopic images (frame images). In this case, the three-dimensional restoration unit 22 calculates the movement of the camera position in other frame images relative to the camera position in the reference frame image (transformation of the camera position and orientation: referred to as "external parameters"). Then, the three-dimensional restoration unit 22 transforms each frame image into the coordinate system of the reference frame image using the external parameters, and integrates them in the coordinate system of the reference frame image. At this time, the three-dimensional restoration unit 22 may use an optimization method to improve the accuracy of the three-dimensional restoration. In one example of the optimization method, the three-dimensional restoration unit 22 may perform bundle adjustment to minimize reprojection errors based on corresponding feature points between the frame images. In another example of the optimization method, the three-dimensional restoration unit 22 may use NeRF (Neural Radiance Field) to estimate the distance between the object and the camera, and the brightness of the object obtained from the distance. The 3D reconstruction unit 22 outputs a 3D point cloud of the imaging site in the endoscopic image to the polyp region mapping unit 25.

The lesion detection unit 23 receives an endoscopic image from the interface 13. The lesion detection unit 23 detects lesion (also called "polyp") areas from the endoscopic image using a pre-prepared image recognition model, etc., and generates a polyp image including the polyp area. The polyp area in the polyp image may be a rectangle surrounding the polyp, or may be a pixel-by-pixel area detected by segmentation. The lesion detection unit 23 outputs the polyp image to the 3D restoration unit 24 and the display image generation unit 27.

The three-dimensional restoration unit 24 three-dimensionally restores the polyp region included in the polyp image, and generates a three-dimensional point cloud of the polyp region in a three-dimensional coordinate system. The three-dimensional restoration unit 24 basically needs to generate a three-dimensional point cloud of the polyp region using a method similar to that used by the three-dimensional restoration unit 22, and in practice the three-dimensional restoration units 22 and 24 may be configured using a common three-dimensional restoration unit. The three-dimensional restoration unit 24 outputs the three-dimensional point cloud of the polyp region to the polyp region mapping unit 25.

The polyp region mapping unit 25 maps the three-dimensional point cloud of the polyp region input from the three-dimensional reconstruction unit 24 onto the three-dimensional point cloud of the imaged area input from the three-dimensional reconstruction unit 22. This makes it possible to identify the polyp region on the three-dimensional point cloud of the imaged area. The polyp region mapping unit 25 outputs the three-dimensional point cloud onto which the polyp region is mapped to the polyp long diameter calculation unit 26.

The polyp long diameter calculation unit 26 calculates the long diameter of the polyp using the three-dimensional point cloud onto which the polyp region is mapped. Polyps come in a variety of shapes, but the polyp long diameter calculation unit 26 determines the longest part of the polyp as the long diameter. Specifically, the polyp long diameter calculation unit 26 calculates the distance between the two farthest points in the three-dimensional point cloud of the polyp indicated by the polyp region as the long diameter. The polyp long diameter calculation unit 26 outputs the calculated long diameter to the display image generation unit 27.

The display image generating unit 27 generates a display image using the endoscopic image input from the interface 13, the polyp image input from the lesion detection unit 23, and the polyp long diameter input from the polyp long diameter calculation unit 26. Specifically, the display image generating unit 27 generates a display image including an image showing a polyp region on the endoscopic image and the value of the polyp long diameter. An example of the display image will be described later. The display image generating unit 27 outputs the display data Id of the generated display image to the display device 2 and causes it to be displayed on the display device 2.

In the above configuration, the interface 13 is an example of an image acquisition means, the three-dimensional reconstruction unit 22 is an example of a three-dimensional reconstruction means, the lesion detection unit 23 is an example of a detection means, the polyp long diameter calculation unit 26 is an example of a long diameter calculation means, and the display image generation unit 27 is an example of a display control means.

[Polyp information display processing]
Next, a description will be given of the polyp information display process executed by the endoscopic examination support device 1. Fig. 4 is a flowchart of the polyp information display process. This process is realized by the processor 11 shown in Fig. 2 executing a program prepared in advance and operating as each element shown in Fig. 3.

First, the endoscopic video Ic is input from the endoscope scope 3 to the interface 13. The interface 13 acquires an endoscopic image from the input endoscopic video Ic (step S11). Next, the depth estimation unit 21 estimates the depth from the endoscopic image and generates depth information (step S12). Next, the three-dimensional reconstruction unit 22 uses the depth information and the internal parameters of the endoscopic camera to reconstruct a three-dimensional point cloud of the captured area in the endoscopic image (step S13).

The lesion detection unit 23 also detects polyps from the endoscopic image (step S14). Next, the three-dimensional reconstruction unit 24 generates a three-dimensional point cloud of the polyp region from the polyp image (step S15).

Next, the polyp region mapping unit 25 maps the three-dimensional point cloud of the polyp region onto the three-dimensional point cloud of the imaging site of the endoscopic image (step S16). Next, the polyp long diameter calculation unit 26 calculates the long diameter of the polyp region using the three-dimensional point cloud onto which the polyp region is mapped (step S17). Next, the display image generation unit 27 generates a display image using the endoscopic image, the polyp image, and the long diameter value of the polyp, and displays it on the display device 2 (step S18).

[Modification]
In the functional configuration of FIG. 3, the three-dimensional reconstruction units 22 and 24 perform three-dimensional reconstruction based on the internal parameters and depth information of the camera. Instead, the three-dimensional reconstruction units 22 and 24 may perform three-dimensional reconstruction by applying image processing to the endoscopic image without performing depth estimation. Specifically, the three-dimensional reconstruction units 22 and 24 may use SfM, which estimates the camera posture and three-dimensional coordinates of feature points by performing feature point matching. In addition, since SfM can only perform three-dimensional reconstruction of sparse point clouds, a dense point cloud may be restored using MVS (Multi-View Stereo). In this case, instead of the depth estimation process, three-dimensional reconstruction is performed from the endoscopic image using the following processing procedure.
(1) Estimation of camera position using SfM and 3D reconstruction of feature points (2) Reconstruction of dense 3D point cloud using MVS

[Display example]
Next, an example of a display image generated by the display image generating unit 27 will be described.
(First display example)
Fig. 5 shows a first display example of a display image displayed on the display device 2. As shown in Fig. 5(A), a display image 40 of the first display example mainly includes a video area 41 and an analysis result area 42. The video area 41 displays an endoscopic image input from the endoscope 3. The analysis result area 42 displays the analysis results of the endoscopic image, etc. In the example of Fig. 5(A), a polyp 43 is detected in the endoscopic image, and a rectangle (frame) 44 surrounding the polyp 43 is displayed superimposed on the endoscopic image displayed in the video area 41. A scale indicating the length is displayed on one side of the rectangle 44.

5B is an enlarged view of the rectangle 44. In FIG. 5B, the top side 44a of the rectangle 44 has a scale 44b indicating the length. In this example, the distance between two adjacent scales 44b is 1 mm. The length indicated by the scale 44b is not the length on the displayed image, but the actual length of the imaged part in the endoscopic image, and is calculated based on the three-dimensional point cloud of the imaged part described above. By simultaneously looking at the polyp 43 in the endoscopic image and the scale 44b on one side of the rectangle 44, the examiner can estimate the size of the polyp 43 from the endoscopic image displayed in the video area 41. Note that in the example of FIG. 5B, the scale 44b is on the top side of the rectangle 44, but the scale 44b may be on any one of the four sides of the rectangle.

In FIG. 5(A), the analysis result area 42 displays an analysis result image 45 and a polyp's long diameter value 46. When a polyp is detected, the analysis result image 45 is basically the same as the endoscopic image displayed in the video area 41, and a rectangle 45x indicating the position of the polyp is superimposed. The polyp's long diameter value 46 is the long diameter value calculated by the polyp long diameter calculation unit 26 based on the polyp region on the three-dimensional point cloud. When a polyp is not detected, the analysis result image 45 and the polyp's long diameter value 46 are not displayed. In this way, the polyp's long diameter value 46 calculated from the endoscopic image is displayed in the displayed image, so the examiner can consider treatment for the detected polyp by referring to the polyp's long diameter value calculated by the endoscopic examination support device 1.

(Second display example)
Fig. 6 shows a second display example of the display image displayed on the display device 2. As shown in Fig. 6(A), a display image 40x of the second display example includes a video area 41 and an analysis result area 42, similar to the first display example. The second display example is different from the first display example in the rectangle 47 surrounding the polyp, but is otherwise similar to the first display example.

FIG. 6(B) is an enlarged view of rectangle 47. Rectangle 47 in the second display example has scale marks 47b on its upper side 47a. However, in the second display example, scale marks 47b are provided every 5 mm. Also, a numerical value 47c indicating the interval between scale marks 47b (in this example, "5" indicating 5 mm) is displayed near scale marks 47b. This allows the examiner to estimate the size of the polyp on the assumption that the interval between scale marks 47b is 5 mm. Note that in the second display example as well, polyp long diameter value 46 is displayed in analysis result area 42.

(Third display example)
7 shows a third display example of a display image displayed on the display device 2. As shown in Fig. 7(A), a display image 40y of the third display example includes a video area 41 and an analysis result area 42, similar to the first display example. The third display example is different from the first display example in the area within a rectangle 48 surrounding a polyp, but is otherwise similar to the first display example.

7B is an enlarged view of the rectangle 48. In the first and second display examples, a scale is provided on one side of the rectangle. In contrast, in the rectangle 48 of the third display example, a line segment (axis) 48a indicating the long diameter of the polyp is provided inside the rectangle 48, and a scale 48b is provided on the line segment 48a. The line segment 48a is displayed so as to extend in the direction of the long diameter used in the calculation by the polyp long diameter calculation unit 26. In the example of FIG. 7B, the scale 48b is provided at intervals of 5 mm, as in the second display example. Note that the polyp long diameter value 46 is also displayed in the analysis result area 42 in the third display example. In the third display example, the line segment 48a indicating the long diameter of the polyp is displayed, so the examiner can know which part of the polyp 43 the polyp long diameter value 46 displayed in the analysis result area 42 is measured, and can judge the reliability of the displayed long diameter value 46.

(Fourth display example)
FIG. 8 shows a fourth display example of the display image displayed on the display device 2. As shown in FIG. 8, the display image 40z of the fourth display example includes a video area 41 and an analysis result area 42, similar to the first display example. In the fourth display example, instead of displaying a rectangle surrounding the polyp, a contour 49 of the polyp 43 is superimposed on the endoscopic image. The fourth display example can be used when the lesion detection unit 23 shown in FIG. 3 detects the area of the polyp in pixel units by segmentation. Other than this, the fourth display example is the same as the third display example. That is, in the fourth display example, a line segment 48a indicating the long axis of the polyp is displayed, and a scale is displayed on the line segment 48a, similar to the third display example. Therefore, the examiner can know which part of the polyp 43 is measured by the long axis value 46 of the polyp displayed in the analysis result area 42.

(Fifth display example)
Fig. 9 shows a fifth display example of a display image displayed on the display device 2. As shown in Fig. 9, a display image 50 in the fifth display example includes only an image area 51. An endoscopic image is displayed in the image area 51. In the example of Fig. 9, a polyp 53 is detected in the endoscopic image, and a rectangle (frame) 54 surrounding the polyp 53 is displayed superimposed on the endoscopic image displayed in the image area 51. The rectangle 54 is basically the same as in the first display example, and a scale indicating the length is displayed on the upper side of the rectangle 54.

In addition, in the fifth display example, a box 55 indicating the long diameter value of the polyp is displayed near the rectangle 54 (in this example, below). By looking at the box 55, the examiner can know the long diameter value of the polyp calculated by the endoscopic examination support device 1.

(Sixth display example)
10 shows a sixth display example of a display image displayed on the display device 2. A display image 50x in the sixth display example includes only an image area 51. The sixth display example is obtained by using a rectangle 56 similar to the rectangle 47 in the second display example in the fifth display example. That is, the rectangle 56 has scale marks at 5 mm intervals on its upper side. Apart from this, the sixth display example is the same as the fifth display example.

(Seventh display example)
11 shows a seventh display example of a display image displayed on the display device 2. A display image 50y in the seventh display example includes only an image area 51. The seventh display example uses a rectangle 57 similar to the rectangle 48 in the third display example in the fifth display example. That is, a line segment 54d indicating the major axis of the polyp is displayed inside the rectangle 57, and a scale is displayed on the line segment 54d. Apart from this, the seventh display example is the same as the fifth display example.

Second Embodiment
12 is a block diagram showing the functional configuration of an endoscopic examination support device according to the second embodiment. The endoscopic examination support device 70 includes an image acquisition unit 71, a detection unit 72, a three-dimensional reconstruction unit 73, a major axis calculation unit 74, and a display control unit 75.

FIG. 13 is a flowchart of processing by the endoscopic examination support device of the second embodiment. The image acquisition means 71 acquires an endoscopic image captured by an endoscopic camera (step S71). The detection means 72 detects polyps from the endoscopic image (step S72). The three-dimensional reconstruction means 73 reconstructs a three-dimensional point cloud of the photographed area from the endoscopic image (step S73). The major axis calculation means 74 calculates the major axis of the polyp based on the three-dimensional point cloud and the polyp (step S74). The display control means 75 causes a display image including the major axis value to be displayed on the display device (step S75).

The second embodiment of the endoscopic examination support device 70 makes it possible to estimate and display the size of polyps from endoscopic images.

Some or all of the above embodiments can be described as follows, but are not limited to the following:

(Appendix 1)
an image acquisition means for acquiring an endoscopic image captured by an endoscopic camera;
a detection means for detecting a polyp from the endoscopic image;
a three-dimensional reconstruction means for reconstructing a three-dimensional point cloud of an imaging region from the endoscopic image;
a major axis calculation means for calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
a display control means for causing a display image including the value of the major axis to be displayed on a display device;
An endoscopic examination support device comprising:

(Appendix 2)
a depth estimation means for estimating a depth of the endoscopic image,
The endoscopic examination support device according to claim 1, wherein the three-dimensional reconstruction means reconstructs a three-dimensional point cloud of the imaging area based on the depth and parameters of the endoscopic camera.

(Appendix 3)
a mapping means for mapping the polyp onto the three-dimensional point cloud;
The endoscopic examination support device according to claim 1, wherein the long diameter calculation means calculates the long diameter based on a region of the polyp on the three-dimensional point cloud.

(Appendix 4)
The endoscopic examination support device of claim 1, wherein the display image includes the endoscopic image, a scale displayed at the position of the polyp on the endoscopic image, and a numerical value indicating the value of the major diameter.

(Appendix 5)
An endoscopic examination support device as described in Appendix 4, wherein the display image includes a rectangle displayed at the position of the polyp on the endoscopic image, and the scale is displayed on one side of the rectangle.

(Appendix 6)
The endoscopic examination support device according to claim 4, wherein the display image includes a line segment indicating the long diameter of the polyp on the endoscopic image, and the scale is displayed on the line segment.

(Appendix 7)
the display image has a first area for displaying the endoscopic image and a second area different from the first area,
An endoscopic examination support device as described in Appendix 4, wherein the scale is displayed in the first area and a numerical value indicating the value of the major diameter is displayed in the second area.

(Appendix 8)
An endoscopic examination support device as described in Appendix 4, wherein a numerical value indicating the value of the major axis is displayed on the endoscopic image.

(Appendix 9)
Executed by a computer,
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
An endoscopic examination support method for displaying a display image including the value of the major axis on a display device.

(Appendix 10)
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
A recording medium having recorded thereon a program for causing a computer to execute a process for displaying, on a display device, a display image including the value of the major axis.

The present disclosure has been described above with reference to embodiments and examples, but the present disclosure is not limited to the above embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

REFERENCE SIGNS LIST 1 Endoscopic examination support device 2 Display device 3 Endoscope scope 11 Processor 12 Memory 13 Interface 21 Depth estimation unit 22, 24 Three-dimensional reconstruction unit 23 Lesion detection unit 25 Polyp region mapping unit 26 Polyp long diameter calculation unit 27 Display image generation unit 100 Endoscopic examination system

Claims

an image acquisition means for acquiring an endoscopic image captured by an endoscopic camera;
a detection means for detecting a polyp from the endoscopic image;
a three-dimensional reconstruction means for reconstructing a three-dimensional point cloud of an imaging region from the endoscopic image;
a major axis calculation means for calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
a display control means for causing a display image including the value of the major axis to be displayed on a display device;
An endoscopic examination support device comprising:
a depth estimation means for estimating a depth of the endoscopic image,
The endoscopic examination support device according to claim 1 , wherein the three-dimensional reconstruction means reconstructs a three-dimensional point cloud of the imaging region based on the depth and parameters of the endoscope camera.
a mapping means for mapping the polyp onto the three-dimensional point cloud;
2. The endoscopic examination support device according to claim 1, wherein the major axis calculation means calculates the major axis based on a region of the polyp on the three-dimensional point cloud.
The endoscopic examination support device according to claim 1, wherein the display image includes the endoscopic image, a scale displayed at the position of the polyp on the endoscopic image, and a numerical value indicating the value of the major axis.
The endoscopic examination support device according to claim 4, wherein the display image includes a rectangle displayed at the position of the polyp on the endoscopic image, and the scale is displayed on one side of the rectangle.
The endoscopic examination support device according to claim 4, wherein the display image includes a line segment indicating the major axis of the polyp on the endoscopic image, and the scale is displayed on the line segment.
the display image has a first area for displaying the endoscopic image and a second area different from the first area,
The endoscopic examination support device according to claim 4 , wherein the scale is displayed in the first area, and a numerical value indicating the value of the major axis is displayed in the second area.
The endoscopic examination support device according to claim 4, wherein a numerical value indicating the value of the major axis is displayed on the endoscopic image.
Executed by a computer,
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
An endoscopic examination support method for displaying a display image including the value of the major axis on a display device.
Acquire an endoscopic image captured by an endoscopic camera;
Detecting polyps from the endoscopic image;
Reconstructing a three-dimensional point cloud of the imaging region from the endoscopic image;
Calculating a major axis of the polyp based on the three-dimensional point cloud and the polyp;
A recording medium having recorded thereon a program for causing a computer to execute a process for displaying, on a display device, a display image including the value of the major axis.