KR20230133701A - Method for automatic photographing during endoscope procedure based on landmarks and device for the same - Google Patents

Abstract

Disclosed is an automatic photographing device including an endoscope device and an information processing device connected to the endoscope device to obtain image information photographed by the endoscope device. The information processing device includes a processing unit and a storage unit. The processing unit is configured to obtain a plurality of photographed images including a landmark from the endoscope device according to a predetermined schedule when the predetermined landmark is detected in a photographed image obtained from the endoscope device. And, one final selection image among the plurality of photographed images is determined and stored in the storage unit.

Description

Method for automatic endoscopic photography using landmarks and device therefor {Method for automatic photography during endoscope procedure based on landmarks and device for the same}

The present invention relates to technology for automatically taking and organizing photos during the endoscope imaging process.

According to existing technology, medical staff operating an endoscope make medical decisions by checking images output in real time while trying to avoid damaging the body of the patient inserting the endoscope, and, if necessary, directly press the button on the endoscope device. You have to manipulate it by hand to save some of the images being shot. Among these, if the process of directly manipulating the buttons can be automated, medical treatment using an endoscope will become easier.

Korean registered patents KR 2210806 (2021.01.27.) and KR 2168485 (2020.10.15.) disclose endoscopic devices that diagnose gastric lesions based on gastroscopy images.

However, these prior technologies disclose a lesion diagnosis function that performs gastric lesion diagnosis through an artificial neural network system and a constructed artificial neural network through learning that outputs items related to the lesion diagnosis result, but the manipulation of the medical staff operating the endoscope The purpose of providing convenience and its structure are not disclosed.

In order to increase the operating convenience of medical staff operating an endoscope, the present invention seeks to provide technology for automatically capturing images during endoscopic diagnosis and technology for automatically organizing and presenting imaged information.

The endoscope device includes a curved or non-bent insertion tube, a lighting device that illuminates the object being examined, an imaging device that photographs the object, and a control device that controls the operation of various electrical/electronic components included in the endoscope device. A part of the endoscope device is designed to enter the inside of the patient's body, and the other part, an exposed part, can be placed outside the patient's body, and the operator of the endoscope device operates the hand grip and various operation buttons installed on the exposed part to control the endoscope device. Can be operated and manipulated.

The endoscope device may include an image output port for transmitting information about the captured image to another computing device. In addition, the imaging device of the endoscope device encodes the photographed information according to a predetermined standard standard (standard video standard/standard image standard) to output the image/video output by the photographing device through the video output port. You can have more wealth.

The endoscope device may further include an image signal output unit that outputs the encoded image/video through the image output port. The video output port may be provided in the exposed portion.

The automatic photographing device provided according to one aspect of the present invention may include the endoscope device and an information processing device that can be connected to an image output port of the endoscope device.

The information processing device includes an image input port capable of receiving an image signal generated according to the standard standard, a processing unit that decodes, analyzes and processes the image signal, a storage unit that stores the processed data, and a processing unit that stores the processed data. It may be a computing device including a display device that outputs output.

The video output port and the video input port may be connected with a data cable.

According to one aspect of the present invention, an automatic photographing device including an information processing device connected to the endoscope device to obtain image information captured by the endoscope device may be provided. The information processing device includes a processing unit and a storage unit. The processing unit is configured to acquire a plurality of captured images including the landmark from the endoscope device according to a predetermined schedule when a predetermined landmark is detected in the captured image obtained from the endoscope device, and the plurality of captured images are The final selected image of one of the captured images is determined and stored in the storage unit.

At this time, the storage unit may include a landmark detection command code that detects the landmark in the image obtained from the endoscope device, and an automatic shooting execution command code that automatically takes the image when the landmark is detected. . The processing unit may be configured to execute a landmark detection process by executing the landmark detection command code, and to execute an automatic photography execution process by executing the automatic photography execution command code. The landmark detection process includes inputting any captured image obtained from the endoscope device to a landmark detection network; starting the automatic shooting execution process when the landmark detection network first outputs information that the random captured image includes the landmark; and terminating the automatic shooting execution process when the landmark detection network no longer outputs information indicating that the landmark is included in the arbitrary captured image.

At this time, the automatic photographing process includes acquiring a plurality of photographed images including the landmark from the endoscope device according to the predetermined schedule; may include.

At this time, the automatic photographing process includes, before acquiring the plurality of photographed images, setting photographing conditions of the photographing device of the endoscope device and transmitting a photographing command according to the set photographing conditions to the photographing device. It may further include, and the photographing device may be configured to photograph according to the photographing command.

At this time, the storage unit further includes a captured image analysis command code configured to evaluate the plurality of captured images, and the processing unit may be configured to execute the captured image analysis process by executing the captured image analysis command code. . The captured image analysis process includes calculating a score for each of the acquired plurality of captured images based on dynamic range, color, and contrast of each of the acquired plurality of captured images; and selecting at least one final image from among the obtained plurality of captured images based on the score.

At this time, the landmark detection process includes: starting the captured image analysis process when the landmark detection network first outputs information that the landmark is included in the arbitrary captured image; and terminating the captured image analysis process when the landmark detection network no longer outputs information indicating that the landmark is included in the arbitrary captured image.

At this time, the storage unit may further include a captured image output command code configured to output the final image, and the processing unit may be configured to execute the captured image output process by executing the captured image output command code. The captured image output process may include storing pair information associating the final image with the detected landmark in the storage unit.

At this time, the landmark detection process transmits information that detection of the landmark has begun to the captured image output process when the landmark detection network first outputs information that the landmark is included in the random captured image. and transmitting the identifier of the detected landmark; And when the landmark detection network no longer outputs information that the landmark is included in the random captured image, transmitting information that detection of the predetermined landmark has ended to the captured image output process; More may be included.

At this time, the information processing device may further include a display device. The captured image output process includes associating the acquired plurality of captured images with the detected landmark and displaying them on the display device in the order of acquisition; and It may include displaying pair information associating the final image with the detected predetermined landmark on the display device.

At this time, the information processing device may further include a speaker. The storage unit may further include a photography guide command code, and the processing unit may be configured to execute the photography guide process by executing the photography guide command code. The shooting guide process is, It may include outputting a beep sound through the speaker every time a plurality of captured images including the landmark are acquired from the endoscope device through the automatic photographing process.

At this time, the landmark detection process includes starting the shooting guide process when the landmark detection network first outputs information that the landmark is included in the arbitrary captured image; and terminating the photographing guide process when the landmark detection network no longer outputs information indicating that the landmark is included in the arbitrary photographed image.

At this time, the landmark detection network includes a neural network, and the neural network may be supervised learning using a plurality of input images for learning that include or do not include the predetermined landmark. When each learning input image includes the predetermined landmark, the label assigned to each learning input image may have a first value. When each learning input image does not include the predetermined landmark, the label assigned to each learning input image may have a second value.

At this time, the landmark detection network includes a plurality of neural networks including a first neural network and a second neural network, and the first neural network determines whether the input image input to the neural network includes a first landmark. It may be supervised learning. The second neural network may be supervised to determine whether the input image input to the neural network includes a second landmark.

In this case, the landmark detection process stores the current output value currently output by the landmark detection network in a predetermined first register in order to determine whether the information that the random captured image includes the landmark is output for the first time. Save to; determining whether the value of the first register is the same as the value of a predetermined second register; When it is determined that the value of the first register is not the same as the value of the predetermined second register and the current output value represents a specific landmark, information that the landmark is included in the random captured image is provided. A step of determining that it has been printed for the first time; and updating the value of the second register by replacing it with the value of the first register.

At this time, the landmark detection process is to determine whether the landmark detection network no longer outputs information that the landmark is included in the random captured image, and the landmark detection network currently outputs storing the current output value in a predetermined first register; determining whether the value of the first register is the same as the value of a predetermined second register; When it is determined that the value of the first register is not the same as the value of the predetermined second register and the current output value indicates that the random captured image does not include a landmark, the random captured image determining that information indicating that the landmark is included is no longer output; and updating the value of the second register by replacing it with the value of the first register.

According to the present invention, in order to increase the operating convenience of medical staff operating an endoscope, it is possible to provide a technology for automatically capturing images during endoscopic diagnosis and a technology for automatically organizing and presenting imaged information.

1 is a diagram showing the configuration of an automatic photographing device provided according to an embodiment of the present invention.
Figure 2 is a diagram showing the structure of an endoscope device according to an embodiment.
Figure 3 is a diagram showing the configuration of a medical treatment system that performs medical treatment using an endoscopic device according to an embodiment.
Figure 4 is a diagram showing the specific configuration of an information processing device provided according to an embodiment of the present invention.
Figure 5 is a flowchart showing several processes provided according to an embodiment of the present invention.
Figure 6 shows the concept and learning method of a landmark detection network provided according to an embodiment of the present invention.
Figure 7 shows an example of a process that a user undergoes when taking endoscopic photographs using an automatic photographing device provided according to an embodiment of the present invention.
Figure 8 shows an example of information output to a display device according to an embodiment of the present invention.
Figure 9 shows the internal execution sequence of the landmark detection process provided according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. The terms used in this specification are intended to aid understanding of the embodiments and are not intended to limit the scope of the present invention. Additionally, as used herein, singular forms include plural forms unless phrases clearly indicate the contrary.

1 is a diagram showing the configuration of an automatic photographing device provided according to an embodiment of the present invention.

The automatic photographing device 1 may include an endoscope device 100, an information processing device 200, and a cable 300, which is a data cable connecting the endoscope device 100 and the information processing device 200.

Figure 2 is a diagram showing the structure of an endoscope device according to an embodiment.

The endoscope device 100 includes an insertion tube 110 inserted into the human body, a lighting device 120 that illuminates the subject being examined with light, an imaging device 130 that photographs the subject, and an image output port 141. It may include a device interface 140 that controls the operation of various electrical/electronic components included in the endoscope device, including the above-mentioned parts, and a control device 150 that controls the operation.

A portion of the lighting device 120 and a portion of the imaging device 130 may be coupled to the insertion tube 110.

The insertion tube 110 may be configured to enter the inside of the patient's body, and the device interface 140 may be a portion disposed and exposed outside the patient's body.

When the operator of the endoscope device 100 inserts the insertion tube 110 into the human body, the images output by the imaging device 130 can be transmitted to the outside in real time through the image output port 141. You can.

Figure 3 is a diagram showing the configuration of a medical treatment system that performs medical treatment using an endoscopic device according to an embodiment.

The endoscope device 100 may be connected to a predetermined recording device 400 with a cable.

Captured image data output in real time through the video output port 141 can be transmitted to the recording device 400 in real time through the cable 300.

The recording device 400 is a device including a user interface 410 that receives user input, and an image input port 441 that can be connected to the cable 300 and receives information transmitted through the cable 300. An interface 440, a display device 420 that visually outputs images transmitted through the cable 300 in real time, a storage device 430 that stores images designated by the user of the recording device 400, and It may include a processing unit 450 that controls the operation of various electrical/electronic devices included in the recording device 400.

In one embodiment, the operator of the endoscopic device 100 and the user of the recording device 400 may be the same person.

The user interface 410 may include a photographing button that the user can press at a desired time. When the user presses the photographing button while the insertion tube 110 is inserted into the patient's body, the image captured by the photographing device 130 at the time the photographing button is pressed is stored in the storage device 430. It can be saved.

In one embodiment, the photographing conditions (shutter speed, exposure) of the photographing device 130 in the time period in which the photographing button 411 is not pressed are the photographing device at the time of pressing the photographing button 411 ( 130) may be set differently from the above shooting conditions. To this end, information indicating that the photographing button 411 has been pressed may need to be transmitted to the photographing device 130 or the control device 150. For this purpose, not only are the real-time images transmitted from the endoscope device 100 to the recording device 400 through the cable 300, but also a message indicating that the photographing button 411 has been pressed through the cable 300 Information may be transmitted from the recording device 400 to the endoscope device 100. Alternatively, for this purpose, the endoscope device 100 and the recording device 400 may be designed and provided as virtually one integrated device.

The operator or the user must decide when to press the shooting button 411. However, in the process of actually diagnosing a patient, the act of pressing the photographing button 411 may be repeated several to dozens of times, and the image captured at each photographing time may not be clear. In this case, the image stored in the storage device 430 also becomes unclear, which may be a problem when the medical staff rechecks the patient's lesion imaging results in the future. However, while taking pictures using the endoscope device 100, the operator or user must make various other decisions, including medical judgment, in addition to taking pictures, so the display device 420 determines whether there are any abnormalities in the taken images. There is a problem that it is difficult to check the images displayed one by one and make a decision.

Figure 4 is a diagram showing the specific configuration of an information processing device provided according to an embodiment of the present invention.

The information processing device 200 may be a computing device that includes a processing unit 510, a storage device 520, a display device 530, a device interface 540, and a speaker 550. The processing unit 510 may be a processor such as a CPU, the storage device 520 may be a storage device such as an SSD or USB memory, and the display device 530 may be a display device such as an LCD, and The device interface 540 may be a physical/logical port including an image input port 541 corresponding to the image output port 141.

The image input port 541 may be connected to the image output port 141 of the endoscope device 100.

The processing unit 510 may execute a landmark detection process 511, a shooting guide process 512, an automatic shooting execution process 513, a captured image analysis process 514, and a captured image output process 515.

In order to execute the landmark detection process 511, the shooting guide process 512, the automatic shooting execution process 513, the captured image analysis process 514, and the captured image output process 515, the storage device 520 Command codes corresponding to each of these processes include a landmark detection command code (521), a shooting guide command code (522), an automatic shooting execution command code (523), a captured image analysis command code (524), and a captured image output command code. It may include (525).

Hereinafter, Figures 5A to 5E are collectively referred to as Figure 5.

Figure 5a shows the internal execution sequence of the landmark detection process 511.

Referring to FIG. 5A , in step S110, the currently captured image currently input through a cable can be input to a landmark detection network to be described later.

Step S110 can be executed only once when a new captured image different from the previous one is input through the cable.

The landmark detection network is a learnable network including a neural network such as CNN, and the landmark detection network used in step S110 may have completed learning. The learning method of the landmark detection network will be described later.

In step S120, it is determined whether the current output value output by the landmark detection network in response to the currently captured image represents a specific landmark. For example, determining whether the current output value represents the upper cardia among predefined landmarks, the upper pylorus among predefined landmarks, or does not correspond to any of the predefined landmarks. can do.

If the landmark detection network is well learned, it will output a specific landmark among predefined landmarks as an output value corresponding to the currently captured image, or output information that no landmark has been detected.

If it is determined that the current output value represents a specific landmark, step S121 can be performed. If it is determined that the current output value does not represent a specific landmark, that is, if it is determined that the landmark has not been detected, step S122 can be performed.

A plurality of images may be continuously transmitted to the information processing device 200 and the processing unit 510 through the cable. Therefore, even just before acquiring the current captured image, a different captured image, the previous captured image, may have been received. In addition, the landmark detection process 511 may generate the previous output value, which is the result of the landmark detection network, in response to the input of the immediately previous captured image and temporarily store it.

In step S121, the immediately preceding output value output by the landmark detection network for the immediately preceding captured image acquired immediately before the processing unit 510 acquires the current captured image is the current output value output by the landmark detection network. You can determine if they are the same. If they are the same, you can return to step S110, and if they are not the same, you can proceed to step S130.

Here, the fact that the previous output value output by the landmark detection network is not the same as the current output value in step S120 means that the specific landmark confirmed in step S120 was photographed for the first time, and therefore, the specific landmark was captured for the first time. This may mean that you need to start taking sophisticated automatic shots of the mark.

In contrast, the fact that the immediately previous output value output by the landmark detection network is the same as the current output value in step S120 means that the specific landmark confirmed in step S120 has already become the subject of elaborate photography, and automatic photography This may mean that this is in progress.

In step S130, the shooting guide process 512, the automatic shooting execution process 513, and the captured image analysis process 514 can be started. Then, a notification of the start of landmark detection and a notification of the identifier of the detected landmark can be sent to the captured image output process 515 that is being executed.

When step S130 is completed or when step S130 is not completed, the process may return to step S110.

Returning to step S110, when another image in the next order, other than the currently captured image, is acquired, the other image is input to the landmark detection network.

Going back to step S120, if it is determined that the current output value does not represent a specific landmark, that is, if it is determined that the landmark is not detected, step S122 can be performed.

In step S122, it may be determined whether the previous output value output by the landmark detection network for the immediately captured image is the same as the current output value in step S120. If they are not the same, you can proceed to step S140, and if they are the same, you can return to step S110. Returning to step S110, when another image in the next order, other than the currently captured image, is acquired, the other image is input to the landmark detection network.

At this time, the fact that the immediately previous output value output by the landmark detection network is the same as the current output value in the step (S120) means that the current landmark has not been detected, and the immediately preceding output value is also the same as the previous output value. This means that the landmark was not detected. Therefore, since the situation in which a landmark for precise photography has not been found continues, the process returns to step S110.

In contrast, the fact that the immediately preceding output value output by the landmark detection network is not the same as the current output value in the step (S120) means that the current landmark has not been detected, whereas the immediately preceding output value means that a specific landmark was detected just before. Therefore, it means that the landmark has been detected but is no longer detected. Therefore, it can be seen that the shooting guide process 512, the automatic shooting execution process 513, the captured image analysis process 514, and the captured image output process 515 are being executed until just before the current point.

Accordingly, in step S140, the currently executing shooting guide process 512, automatic shooting execution process 513, and captured image analysis process 514 may be terminated. Additionally, the captured image output process 515 may be notified that detection of a specific landmark has ended.

While the operator or user moves the endoscopic device in the patient's body, no landmarks may be imaged during some time periods, and landmarks may be imaged during other portions of time.

Figure 5b shows the internal execution sequence of the shooting guide process 512.

Referring to FIG. 5B, in step S210, information indicating the name of the landmark discovered by the landmark detection network is output to the speaker, and a shooting beep sound is output to the speaker at each point when automatic image capturing is performed. You can.

The discovered landmark may be one of a plurality of predefined landmarks (eg, the upper cardia and the upper pylorus).

For example, the speaker can output “Ear cardia detected! Start recording. Slow down the endoscope operation speed during recording!”, or simply output “Ear cardia detected! Start recording!”, limited by the examples above. It doesn't work. After that, even if the operator or user does not manually press a separate shooting button, shooting can be done automatically at regular intervals, and beep sounds can be output every time shooting is done so that the operator/user is aware of the fact. .

The automatic image capturing can be performed by an automatic capturing execution process described later.

Figure 5c shows the internal execution sequence of the automatic shooting execution process 513.

The automatic shooting execution process 513 can be implemented in two ways.

The upper part of Figure 5c is a flowchart for implementation using the first method, and the lower part is a flowchart for implementation using the second method.

Referring to the first method shown in the upper part of FIG. 5C, in step S310, the endoscope device 100 is repeatedly installed according to a predetermined schedule according to the type of landmark discovered by the landmark detection network. ), the shooting conditions of the photographing device 130 can be set, and a photographing command according to the set shooting conditions can be transmitted to the photographing device 130.

In one embodiment, different shooting conditions may be assigned to different landmarks. For example, when a first landmark is detected, a photo may be taken N1 times, whereas if a second landmark is detected, a photo may be taken N2 times. Alternatively, the shooting aperture value when the first landmark is detected may be different from the shooting aperture value when the second landmark is detected.

In step S320, captured images input through the video input port 541 can be selected according to a predetermined schedule and stored in the storage device 520. At this time, the photographed image may have been photographed according to the photographing command transmitted in step S310.

Referring to the second part shown at the bottom of FIG. 5C, in step S311, the captured image is input through the image input port 541 according to the type of landmark discovered by the landmark detection network. can be selected according to a predetermined schedule and stored in the storage device 520.

At this time, the processing unit 510 does not set the imaging conditions for the imaging device 130 of the endoscope device 100, and the processing unit 510 does not transmit a separate imaging command to the imaging device 130. However, the photographing device 130 may capture images captured and transmitted according to a self-set shooting schedule and store them in the storage device 520 according to a predetermined schedule.

Figure 5d shows the internal execution sequence of the captured image analysis process 514.

Referring to FIG. 5D , in step S410, the current image selected by the automatic photography execution process 513 may be acquired.

In step S420, the auto capture execution process 513 may evaluate the quality of the current image based on the selected dynamic range, color, and contrast of the current image. The results of the above evaluation can be converted into numbers.

The automatic shooting execution process 513 may have selected another image (immediately before image) just before selecting the current image.

In step S430, if the quality of the current image is better than the quality of the immediately preceding image selected by the auto capture execution process 513, the current image is determined as the final selection image, otherwise, the previous image is determined as the final selection image. can be decided.

That is, when a new landmark is detected in the landmark detection process 511, images containing the detected landmark are captured multiple times, and one of them with the best image quality is selected as the final selection image. You can take it. The one final selected image may be stored in association with the new landmark.

Alternatively, in a modified embodiment, when a new landmark is detected in the landmark detection process 511, images containing the detected landmark are photographed multiple times, and among them, K images with the best image quality are selected. It can be used as K final selected images. The K final selected images may be stored in association with the new landmark (K is a natural number of 2 or more).

Figure 5e shows the internal execution sequence of the captured image output process 515.

Referring to FIG. 5E , in step S510, upon receiving notification of the start of landmark detection and notification of the identifier of the detected landmark from the landmark detection process 511, the captured image analysis process 514 selects the final selected image being generated. can be stored in association with the detected landmark.

In step S520, when notification of the end of landmark detection is received from the landmark detection process 511, the identifier of the detected landmark and the final selection image associated therewith can be displayed on the display device and stored in the storage device 520. there is.

Figures 6a to 6c show a landmark detection network and its learning method provided according to an embodiment of the present invention.

Figures 6A to 6C may be collectively referred to as Figure 6.

The structure and learning method of the landmark detection network shown in Figure 6 is an example, and may be designed in a different way.

When described with reference to FIG. 6A, the landmark detection network 600 may be composed of a plurality of sub-networks 601 and 602. Different subnetworks may be trained to detect different landmarks. The sub-network may also be referred to as a neural network.

For example, the landmark detection network 600 includes a first sub-network 601 that detects a first landmark corresponding to the cardia above, and a second sub-network 602 that detects a second landmark corresponding to the pylorus above. ), and a post-processing unit 610 that outputs a final result based on the output values of the subnetworks. A total of two sub-networks are shown in Figure 6a, but this is for convenience of explanation. If a total of M landmarks must be detected, the landmark detection network 600 can be configured with a total of M sub-networks. .

Input data input to the landmark detection network 600 may be an image captured by an imaging device of an endoscope device. The input data may be input to the first subnetwork 601 and the second subnetwork 602, respectively. Each of the sub-networks 601 and 602 outputs a value close to 1 if the landmark to be detected is included in the input data, and otherwise outputs a value close to 0.

Figure 6b is for explaining a method of learning the first sub-network.

This learning method may be supervised learning.

The first sub-network 601 may include a neural network such as CNN. The first sub-network 601 may be a network that must determine whether a specific first landmark among a plurality of landmarks exists in the input image.

Learning input data 711 to be input to the first sub-network 601 may be prepared. And a learning input data label corresponding to the learning input data 711 can be prepared. If the learning input data 711 includes the first landmark, the learning input data label of the learning input data 711 is given as '1', otherwise, the learning input data label is given as '0'. You can. The learning input label of the learning input data 711 can be determined by a clinical expert by directly viewing the image of the learning input data 711. The image of the learning input data 711 may be, for example, the above image taken separately.

When the learning input data 711 is input to the first sub-network 601, the first sub-network 601 can output a corresponding output value. The error calculation unit 611 may calculate an error value that is a difference value between the output value and the learning input data label. The network update unit 612 may update parameter values within the first subnetwork 601 to further reduce the error value.

Ideally, the learned first sub-network 601 outputs the same value as the learning input data label as the output value of the first sub-network 601 if the learning input data 711 includes the first landmark. Can be printed. On the other hand, if the first landmark is not included in the learning input data 711, the ideally learned first sub-network 601 uses the learning input data label as the output value of the first sub-network 601. A value different from that, for example, 0, can be output.

Figure 6b is for explaining a method of learning the second sub-network.

This learning method may be supervised learning.

The second sub-network 602 may include a neural network such as CNN. The second sub-network 602 may be a network that must determine whether a specific second landmark among a plurality of landmarks exists in the input image.

Learning input data 712 to be input to the second sub-network 602 may be prepared. And a learning input data label corresponding to the learning input data 712 can be prepared. If the learning input data 712 includes the second landmark, the learning input data label of the learning input data 712 is given as '1', otherwise, the learning input data label is given as '0'. You can. The learning input label of the learning input data 712 can be determined by a clinical expert by directly viewing the image of the learning input data 712.

When the learning input data 712 is input to the second sub-network 602, the second sub-network 602 can output a corresponding output value. The error calculation unit 612 may calculate an error value that is a difference value between the output value and the learning input data label. The network update unit 622 may update parameter values within the second subnetwork 602 to further reduce the error value.

The ideally learned second sub-network 602 outputs the same value as the learning input data label as the output value of the second sub-network 602 if the learning input data 712 includes the second landmark. Can be printed. On the other hand, if the second landmark is not included in the learning input data 712, the ideally learned second sub-network 602 uses the learning input data label as the output value of the second sub-network 602. A value different from that, for example, 0, can be output.

The first sub-network 601 and the second sub-network 602 can each be learned repeatedly using various training input data and corresponding training input data labels.

Figure 6 shows an example of the landmark detection network 600, and may be designed and used differently.

Figure 7 shows an example of a process that a user undergoes when taking endoscopic photographs using an automatic photographing device provided according to an embodiment of the present invention.

The user of the automatic imaging device 1 can insert the insertion tube 110 of the endoscope device 100 into the patient's body while visually checking the video provided from the endoscope device 100 and displayed on the display device 530. (S610).

When the information processing device 200 discovers a new landmark and outputs “Fungus detected! Start shooting!” (one time) and a shooting beep sound (continuous output), the user uses the display device 530 according to the shooting beep sounds. The insertion speed of the insertion tube 110 of the endoscopic device 100 can be adjusted while checking the captured images displayed in ) and the final selection image selected among them (S620). For example, the insertion speed of the insertion tube 110 can be slowed down or stopped for a while while the automatic imaging process is in progress.

When the user further moves the insertion tube 110 of the endoscope device 100 and the landmark is no longer included in the video displayed on the display device 530, the shooting beep sound is no longer output, and the operator If necessary, the insertion tube 110 can be moved in a different direction as needed (S620).

The user can adjust the position of the insertion tube 110 while visually checking the photos automatically taken by the information processing device 200 and displayed on the display device, and can separately provide user input by hand to save the photos. no need.

Figure 8 shows an example of information output to a display device according to an embodiment of the present invention.

Figure 8 is an illustration to aid understanding of the present invention, and may be modified differently in actual implementation.

Figure 8 shows the screen 531 of the display device 530.

In a portion of the screen (left side), sequentially captured images (images 1 to 6) of a landmark (ex: landmark #3) discovered by the automatic photographing device 1 and currently being photographed may be listed. The image with the highest quality among the sequentially captured images may be marked separately. In Figure 8, a black square icon is added to image 4, indicating that image 4 is the best quality image.

On the tab (right side) of the screen, identification information of landmarks that should be included in the part to be photographed using the automatic photographing device 1 may be listed. In Figure 8, identification information of a total of six landmarks (landmark #1 to landmark #6) is illustrated. Around each landmark identification information, the highest quality captured image (finally selected image) of the captured area including the landmark is displayed.

In a portion of the screen (left side), the portion including landmark #3 may be continuously photographed and displayed, and the continuous photographing of the portion including landmark #3 may end at some point. At this time, when the highest quality image (ex: image 4) is determined among all captured images, the determined image is displayed around landmark #3 on the tab (right side) of the screen.

When automatic photography using the automatic photographing device 1 is completed, the information displayed on the other side (right side) of the screen is stored in memory and can be used in the future or printed.

Figure 9 shows the internal execution sequence of the landmark detection process.

Figure 9 is a more detailed embodiment of the process of Figure 5A.

9, when the landmark detection process starts, in step S105, Reg. The value of 2 can be initialized to a value representing 'none' (for example, the number '0'). Here, 'none' may mean that no landmark was detected in the input captured image.

Reg.1 and Reg.2 shown in FIG. 9 are accessible to the processing unit 510 and are the first and second registers provided to the information processing device 200, respectively. The first register and the second register may be hardware memory provided in the CPU constituting the processing unit or hardware memory provided in the storage device 520. Reg. 1 stores the current output value currently detected and output by the landmark detection network, and Reg. 2 may be to store the previous output value detected and output by the landmark detection network immediately before outputting the current output value.

In step S110, the currently captured image currently input through the cable can be input to the landmark detection network.

In step S1101, the current output value output by the landmark detection network is converted to the Reg. It can be saved at 1.

In step S1201, the Reg. Determine whether the value stored in 1 represents a specific landmark.

Above Reg. If it is determined that the value stored in 1 represents a specific landmark, the process proceeds to step S1211. If it is determined that it does not, that is, if it is determined that the landmark is not detected, the process proceeds to step S1221.

At step S1211, Reg. The value stored in 2 is Reg. If it is the same as the value stored in 1, you can proceed to step S123, and if it is not the same, you can proceed to step S130.

In step S123, the Reg. Set the value of 1 to Reg. It can be copied with a value of 2. That is, the above Reg. The current output value of the landmark detection network stored in 1 is stored in Reg. By copying to 2, Reg. The value of the previous output value stored in 2 can be updated.

After executing step S123, you can return to step S110. Returning to step S110, when another image in the next order, other than the currently captured image, is acquired, the other image is input to the landmark detection network.

Step S130 is the same as step S130 described in FIG. 5A. Either when step S130 is completed or when step S130 is not completed, the process can proceed to step S123.

Again, going back to step S1201, the Reg. If the value stored in 1 does not represent a specific landmark, that is, if the landmark has not been detected, the process can proceed to step S1221.

At step S1221, Reg. The value stored in 2 is Reg. If it is the same as the value stored in 1, you can proceed to step S124, and if it is not the same, you can proceed to step S140.

In step S124, the Reg. Set the value of 1 to Reg. It can be copied with a value of 2. In other words, the current output value of the landmark detection network can be saved as the previous output value.

After executing step S124, you can return to step S110. Returning to step S110, when another image in the next order, other than the currently captured image, is acquired, the other image is input to the landmark detection network.

At step S1221, Reg. The value stored in 2 is Reg. That it is not the same as the value stored in 1 means that Reg. A value of 1 means that no landmark is currently detected, while the Reg. A value of 2 means that a specific landmark was detected just before. Therefore, it means that the landmark has been detected but is no longer detected. Therefore, it can be seen that the shooting guide process 512, the automatic shooting execution process 513, the captured image analysis process 514, and the captured image output process 515 are being executed right up to the current point.

Step S140 is the same as step S140 shown in FIG. 5A. Either when step S140 is completed or when step S140 is not completed, the process can proceed to step S124.

By using the above-described embodiments of the present invention, those in the technical field of the present invention will be able to easily make various changes and modifications without departing from the essential characteristics of the present invention. The contents of each claim in the patent claims can be combined with other claims without reference within the scope that can be understood through this specification.

Claims (15)

an information processing device connected to the endoscope device to obtain image information captured by the endoscope device;
Includes,
The information processing device includes a processing unit and a storage unit,
The processing unit,
When a predetermined landmark is detected in a captured image obtained from the endoscope device, a plurality of captured images including the landmark are acquired from the endoscope device according to a predetermined schedule, and
configured to determine one final selected image among the plurality of captured images and store it in the storage unit,
Automatic filming device. According to paragraph 1,
The storage unit includes a landmark detection command code that detects the landmark in the image obtained from the endoscope device, and an automatic shooting execution command code that automatically takes a photo when the landmark is detected,
The processing unit is configured to execute the landmark detection process by executing the landmark detection command code, and to execute the automatic shooting execution process by executing the automatic shooting execution command code,
The landmark detection process is,
Inputting any captured image acquired from the endoscope device to a landmark detection network;
starting the automatic shooting execution process when the landmark detection network first outputs information that the random captured image includes the landmark; and
terminating the automatic shooting execution process when the landmark detection network no longer outputs information indicating that the landmark is included in the random captured image;
Includes,
Automatic filming device. According to paragraph 2,
The automatic photographing process includes acquiring a plurality of photographed images including the landmark from the endoscope device according to the predetermined schedule; Including,
Automatic filming device. According to paragraph 3,
The automatic photographing process further includes, prior to acquiring the plurality of photographed images, setting photographing conditions for the photographing device of the endoscope device and transmitting a photographing command according to the set photographing conditions to the photographing device. and
The photographing device is configured to photograph according to the photographing command,
Automatic filming device. According to paragraph 2,
The storage unit further includes a captured image analysis command code configured to evaluate the plurality of acquired images,
The processing unit is configured to execute the captured image analysis process by executing the captured image analysis command code,
The captured image analysis process is,
calculating a score for each of the acquired plurality of captured images based on the dynamic range, color, and contrast of each of the acquired plurality of captured images; and
selecting at least one final image from among the obtained plurality of captured images based on the score;
Including,
Automatic filming device. According to clause 5,
The landmark detection process is,
starting the captured image analysis process when the landmark detection network first outputs information that the random captured image includes the landmark; and
terminating the captured image analysis process when the landmark detection network no longer outputs information indicating that the landmark is included in the random captured image;
Containing more,
Automatic filming device. According to clause 5,
The storage unit further includes a captured image output command code configured to output the final image,
The processing unit is configured to execute the captured image output process by executing the captured image output command code,
The captured image output process is,
storing pair information associating the final image with the detected landmark in the storage unit;
Including,
Automatic filming device. In paragraph 7
The landmark detection process is,
When the landmark detection network first outputs information that the random captured image includes the landmark, it transmits information that detection of the landmark has started to the captured image output process, and detects the predetermined landmark. passing an identifier; and
When the landmark detection network no longer outputs information indicating that the landmark is included in the random captured image, transmitting information that detection of the predetermined landmark has ended to the captured image output process;
Containing more,
Automatic filming device. In clause 7,
The information processing device further includes a display device,
The captured image output process is,
associating the acquired plurality of captured images with the detected landmark and displaying them on the display device in the order of acquisition; and
displaying pair information associating the final image with the detected landmark on the display device;
Including,
Automatic filming device. According to paragraph 2,
The information processing device further includes a speaker,
The storage unit further includes a shooting guide command code,
The processing unit is configured to execute the shooting guide process by executing the shooting guide command code,
The shooting guide process is,
outputting a beep sound through the speaker each time a plurality of captured images including the landmark are acquired from the endoscope device through the automatic capturing process;
Including,
Automatic filming device. According to clause 10,
The landmark detection process is,
starting the shooting guide process when the landmark detection network first outputs information that the landmark is included in the random captured image; and
terminating the shooting guide process when the landmark detection network no longer outputs information indicating that the landmark is included in the random shooting image;
Containing more,
Automatic filming device. According to paragraph 2,
The landmark detection network includes a neural network,
The neural network is supervised learning using a plurality of input images for learning that include or do not include the predetermined landmark,
When each learning input image includes the predetermined landmark, the label assigned to each learning input image has a first value,
When each learning input image does not include the predetermined landmark, the label assigned to each learning input image has a second value,
Automatic filming device. According to paragraph 2,
The landmark detection network includes a plurality of neural networks including a first neural network and a second neural network,
The first neural network is supervised to determine whether the first landmark is included in the input image input to the neural network,
The second neural network is supervised to determine whether the input image input to the neural network includes a second landmark,
Automatic filming device. According to paragraph 2,
The landmark detection process is to determine whether information that the landmark is included in the random captured image is output for the first time,
storing the current output value currently output by the landmark detection network in a predetermined first register;
determining whether the value of the first register is the same as the value of a predetermined second register;
When it is determined that the value of the first register is not the same as the value of the predetermined second register and the current output value represents a specific landmark, information that the landmark is included in the random captured image is provided. A step of determining that it has been printed for the first time; and
updating the value of the second register by replacing the value of the first register;
Containing more,
Automatic filming device. According to paragraph 2,
The landmark detection process is to determine whether the landmark detection network no longer outputs information that the landmark is included in the random captured image,
storing the current output value currently output by the landmark detection network in a predetermined first register;
determining whether the value of the first register is the same as the value of a predetermined second register;
When it is determined that the value of the first register is not the same as the value of the predetermined second register and the current output value indicates that the random captured image does not include a landmark, the random captured image determining that information indicating that the landmark is included is no longer output; and
updating the value of the second register by replacing the value of the first register;
Containing more,
Automatic filming device.

Images