WO2011040769A2

WO2011040769A2 - Surgical image processing device, image-processing method, laparoscopic manipulation method, surgical robot system and an operation-limiting method therefor

Info

Publication number: WO2011040769A2
Application number: PCT/KR2010/006662
Authority: WO
Inventors: 최승욱; 이민규; 원종석; 민동명
Original assignee: 주식회사 이턴
Priority date: 2009-10-01
Filing date: 2010-09-30
Publication date: 2011-04-07
Also published as: WO2011040769A3

Abstract

Disclosed are a surgical image processing device and a method therefor. The surgical image processing device comprises an image input unit for receiving input in the form of endoscopic images provided by a surgical endoscope, an image storage unit for storing modelling images relating to a surgical implement performing surgery on a surgical object being photographed by the surgical endoscope, an image matching unit for matching the endoscopic images and the modelling images with each other and generating output images, and a screen display unit for outputting output images comprising the endoscopic images and the modelling images; and the surgical image processing device makes smooth surgery possible by providing actual images and modelling images together during surgery.

Description

Surgical image processing apparatus, image processing method, laparoscopic operation method, surgical robot system and its motion limiting method

The present invention relates to a surgical image processing apparatus and its method, a surgical robot system and its operation limiting method, a surgical robot system and a laparoscopic operation method, a tangible surgical image processing apparatus and method.

Surgical robot refers to a robot having a function that can replace a surgical operation performed by a surgeon. Such a surgical robot has the advantage of being capable of accurate and precise operation and remote surgery compared to humans.

Surgical robots currently being developed worldwide include bone surgery robots, laparoscope surgery robots, and stereotactic surgery robots. The laparoscopic surgical robot is a robot that performs minimally invasive surgery using a laparoscope and a small surgical tool.

Laparoscopic surgery is an advanced surgical technique that involves surgery after inserting a laparoscope, an endoscope that looks into the belly and drills a hole about 1 cm in the navel area.

Recent laparoscopic is equipped with a computer chip to obtain a clearer and enlarged image than the naked eye, and has been developed so that any operation can be performed using specially designed laparoscopic surgical instruments while viewing the screen through the monitor.

Moreover, laparoscopic surgery has the same range of surgery as laparotomy, but has fewer complications than laparotomy, and can start treatment much faster after the procedure, and it has the advantage of maintaining the stamina or immune function of the patient. have. As a result, laparoscopic surgery is increasingly recognized as a standard surgery for treating colorectal cancer in the US and Europe.

The surgical robot system generally consists of a master robot and a slave robot. When the operator manipulates a manipulator (for example, a handle) provided in the master robot, a surgical tool coupled to the robot arm of the slave robot or held by the robot arm is operated to perform surgery. The master robot and the slave robot are combined through a communication network to perform network communication.

At the time of laparoscopic surgery, the surgical image taken by the laparoscope is output to the operator, and the operator sees this, but there is a problem that the actual feeling is somewhat lower than that of the surgery performed by direct surgery. This problem is that even when the laparoscope moves and rotates in the abdominal cavity to illuminate other parts of the body, the image seen by the user is output from the monitor of the same position and size, so that the relative distance and movement between the manipulator and the image as described above may be reduced. This can be caused by a difference in the relative distance and movement between the surgical tool and the organ.

In addition, since the surgical image taken by the laparoscopic includes only some shapes of the surgical tool, the operator may be difficult to operate when they collide or overlap each other, or may not be able to proceed with the operation smoothly due to obscured vision.

On the other hand, the surgical robot system is equipped with a plurality of surgical instruments, each instrument performs the operation required for the operation in accordance with the operator's operation on the surgical robot system. Instruments operated by the operator during the operation are controlled by the operator, but are not operated by the operator, especially those that are not visible on the user's operation screen (i.e., located in an out-of-visible area). However, there is a possibility that unnecessary operations may be performed by operations other than the intention of the operator.

As such, by operating the instrument in an unintended manner, the operator may collide with the robot body or an adjacent instrument, and the possibility of causing damage or injury to blood vessels or tissues of the patient under surgery cannot be excluded.

In addition, the conventional surgical robot system requires a separate user operation for the operator to move the laparoscope to a desired position or to adjust the image input angle in order to obtain an image of the surgical site. That is, the operator must input a manipulation for the control of the laparoscope separately using the hands or feet during the surgical procedure.

However, this may be a cause of weakening the operator's concentration in the course of the surgery to maintain a high level of concentration, and incomplete surgery due to the weakened concentration has a problem that can cause serious sequelae to the surgical patient.

The background art described above is technical information possessed by the inventors for the derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily a publicly known technique disclosed to the general public before the application of the present invention.

The present invention is to provide a surgical image processing apparatus and method for enabling a smooth operation by providing a real image and a modeling image at the time of surgery.

In addition, the present invention provides a surgical image processing apparatus and a method for allowing the operator to operate by referring to the image of the neighboring site and the external environment of the patient as well as the surgical site away from the conventional method of operating by looking only at the surgical site. It is for.

In addition, the present invention is to provide a surgical image processing apparatus and method that can perform the operation smoothly by detecting the collision of the surgical instruments in advance, so that the operator can recognize the collision in advance and avoid it.

In addition, the present invention is to provide a surgical image processing apparatus and the method that the operator can use the surgical image more conveniently by implementing an interface that the operator can select the type of the image to be displayed on the monitor for convenience.

In addition, the present invention is to provide a surgical robot system and a method of limiting its operation to allow the instrument to be controlled only in the manner that the operator intends to perform a normal operation.

It is another object of the present invention to provide a surgical robot system and a method of limiting its operation, which can promote safety of a patient under surgery by fundamentally preventing misoperation of a robot arm and / or an instrument.

In addition, the present invention is to provide a surgical robot system and a laparoscopy manipulation method that allows the operator to control the position and image input angle of the laparoscope only by the action to see the desired surgical site.

In another aspect, the present invention is to provide a surgical robot system and a laparoscopy manipulation method that allows the operator to concentrate only on the surgical operation is not necessary because the operator does not need a separate operation for the operation of the laparoscope.

In addition, the present invention changes the output position of the endoscope image output to the monitor to the user in response to the viewpoint of the endoscope changes according to the movement of the surgical endoscope, so that the user can feel the actual surgical situation more realistic It is to provide a tangible surgical image processing apparatus and method.

In addition, the present invention, by extracting the previously stored endoscope image at the present time point and outputs to the screen display unit along with the current endoscope image, haptic surgical image that can inform the user about the change in the endoscope image It is to provide a processing apparatus and method.

In addition, the present invention by modifying the image, such as to match the image or adjust the size of each of the modeling image stored in the image storage and generated in advance for the endoscopic image and the surgical instrument actually taken using the endoscope during surgery It is to provide a tangible surgical image processing apparatus and method that can output to an observable monitor.

In addition, the present invention is to provide a tangible surgical image processing apparatus and method that can make the user feel more vivid to the operation by rotating and moving the monitor in accordance with the viewpoint of varying endoscopes.

Technical problems other than the present invention will be easily understood through the following description.

According to an aspect of the present invention, the image input unit for receiving an endoscope image provided from the surgical endoscope, an image storage unit for storing a modeling image of a surgical tool for operating a surgical target taken by the surgical endoscope, and an endoscope image Surgical image processing apparatus including an image matching unit for generating an output image by matching the modeling image with each other, and a screen display unit for outputting the output image including the endoscope image and the modeling image.

Here, the surgical endoscope may be at least one of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac.

The image matching unit may generate an output image by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.

Here, the image matching unit may include a characteristic value calculating unit calculating a characteristic value using at least one of endoscope images and position coordinate information of an actual surgical tool coupled to at least one robot arm, and a characteristic value calculated by the characteristic value calculating unit. The apparatus may further include a modeling image implementation unit for implementing a modeling image.

In addition, the present embodiment may further include an overlapping image processor for removing an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

Here, the position of the modeling surgical tool image output to the modeling image may be set using the operation information of the surgical tool, the screen display unit outputs the endoscope image to any region of the output image, the modeling image of the output image Output to the surrounding area.

In addition, the modeling image may be an image of a surgical tool photographed at a specific time point before the start of the operation, the embodiment may further include a camera to take a picture of the outside of the surgical target during surgery to generate a camera image.

Here, the image matching unit may generate an output image by matching the endoscope image, the modeling image, the camera image, and a combination image thereof, and at least one of the endoscope image, the modeling image, and the camera image may be a 3D image.

In addition, the present embodiment may further include a mode selection unit for selecting any combination of two or more of the endoscope image, the modeling image and the camera image, the collision detection unit for detecting whether the modeling surgery tool images included in the modeling image collide with each other. In this case, the collision detection unit may further include a warning information output unit for outputting warning information when detecting a collision between the modeling surgical tool image and generating a collision detection signal.

In addition, the collision detection unit may perform any one or more of a force feedback process and an operation limitation of the arm operation unit for controlling the robot arm on which the surgical tool is mounted when detecting a collision between the modeling surgical tool images.

In addition, the surgical image processing apparatus may be included in an interface mounted on a master robot that controls a slave robot including a robot arm, and the modeling image may include CT, MR, It may further include an image modeling an image obtained from any one or more imaging equipment of PET, SPECT, US.

According to another aspect of the invention, in the surgical image processing apparatus for processing an image during surgery, generating and storing a modeling image for the surgical instrument for operating the surgical target, the endoscope image provided from the surgical endoscope There is provided a surgical image processing method comprising the step of receiving an input, generating an output image by matching the endoscope image and the modeling image with each other and outputting the output image including the endoscope image and the modeling image.

Here, the surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, the output image generation step, the actual surgical tool included in the endoscope image The output image may be generated by matching the modeling surgical tool image included in the image and the modeling image with each other.

The generating of the output image may include calculating a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm, and modeling the characteristic value calculated by the characteristic value calculator. The method may further include implementing an image.

The output image generating step may further include removing an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image. The output image generating step may further include operation information of the surgical tool. The position of the modeling surgical tool image output to the modeling image may be set using the image.

Here, in the output image output step, the endoscope image may be output to an arbitrary region of the output image, and the modeling image may be output to a peripheral region of the output image.

In the generating and storing of the modeling image, the modeling image may be generated from an image of a surgical tool photographed at a specific time point before the start of surgery.

In addition, the present embodiment may further comprise the step of generating a camera image by the camera photographing the outside of the operation target during the operation, in this case, the output image generation step, endoscope image, modeling image, camera image and a combination image thereof Are matched with each other to generate an output image.

Here, at least one of the endoscope image, the modeling image, and the camera image may be a 3D image.

The present invention may further include selecting any combination of two or more of an endoscope image, a modeling image, and a camera image, and further comprising detecting whether a modeling surgery tool image included in the modeling image collides with each other. In this case, the collision detection may further include outputting warning information when detecting a collision between the modeling surgical tool image and generating a collision detection signal.

In addition, the collision detection step, if detecting a collision between the modeling surgical tool image, performing any one or more of the process of force feedback processing and the operation control of the arm control unit for controlling the robot arm on which the surgical tool is mounted It may further include.

The generating and storing of the modeling image may further include modeling an image obtained from at least one imaging device of CT, MR, PET, SPECT, and US for the surgical target.

According to still another aspect of the present invention, a program is recorded in which a program of instructions that can be executed by a digital processing apparatus is tangibly implemented to perform the above-described surgical image processing method, and records a program that can be read by the digital processing apparatus. A medium is provided.

According to another aspect of the present invention, as a surgical robot, a restricted area setting unit for receiving the restricted area setting information for the area where the operation of the controlled object is restricted and generates and stores the restricted area coordinate information, and a controlled object An arm operation unit for receiving operation information for the operation of the control unit, and an operation determination unit for determining whether the outer surface of the control object is in contact with the coordinate information of the restricted area by referring to the displacement information of the control object according to the operation information, Wherein the object to be controlled is provided with a surgical robot, characterized in that at least one of the robot arm, the instrument and the endoscope.

When the outer surface of the object to be controlled is in contact with the coordinate information of the restricted area, the controller may further control to control the object to be controlled, and control to output the response information accordingly. Here, the reaction information may be one or more of tactile information, visual information, and auditory information.

The controller may determine whether to control the control object when the outer surface of the control object is in contact with the coordinate information of the restricted area, and then control the control object to be restricted only when the control object is determined to be restricted. Can be.

In addition, when the input unit for receiving the command to ignore the restricted area setting is further included, the controller may process to allow the manipulation of the object to be controlled when the restricted area setting ignore command is input.

The restricted area coordinate information may be coordinate range information corresponding to a closed curve input using a touch-sensitive screen on which a video image is displayed.

The image image may be one or more of an image image obtained by an endoscope, a CT image image, an MRI image image, and a human modeling image image.

The restricted area coordinate information may be coordinate range information of an area that is not displayed in the video image obtained by the endoscope.

The endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac.

The restricted area coordinate information may be coordinate range information of a figure formed by positions designated as restricted area setting information among positions at which the object to be controlled is manipulated by operation information of the arm operating unit.

The object to be controlled is manipulated in three-dimensional space, and the designated positions may be points designated to form an outline or an outer surface of a figure in three-dimensional space, respectively.

The surgical robot may further include an arm manipulation setting unit configured to generate and store manipulation setting information on whether to allow manipulation of each control object by manipulation of the arm manipulation unit. Here, the manipulation determination unit may further determine whether the manipulation information for any manipulation object is valid manipulation information.

According to another aspect of the present invention, as a method of limiting the operation of a surgical robot, receiving the restricted area setting information for the area where the operation of the controlled object is restricted, generating and storing the restricted area coordinate information, and the control object Receiving operation information for manipulating an object, determining whether an outer surface of the controlled object is in contact with coordinate information of a restricted area by referring to displacement information of the controlled object according to the manipulation information, and controlling the object And controlling the controlled object to be restricted in operation when the outer surface of the contact surface is in contact with the coordinate information of the restricted area, wherein the controlled object is one or more of a robot arm, an instrument, and an endoscope. This is provided.

The controlling may include determining whether to control the control object when the external surface of the control object is in contact with the restricted area coordinate information, and when the control object is determined to be restricted to control. Controlling to include.

In the determining step, it may be determined whether a command to ignore the restricted area setting is input or not to control the controlled object.

The controlling may include controlling the response information to be output when the outer surface of the object to be controlled is in contact with the coordinate information of the restricted area.

The operation limiting method of the surgical robot includes generating and storing operation setting information on whether to allow operation of each control target object by operation of the arm operating unit, and whether the operation information for any operation target object is valid operation information. The method may further include determining whether or not.

If the manipulation information is invalid manipulation information, the method may further include controlling the reaction information to be output.

According to another aspect of the invention, as a surgical robot, a display unit for displaying an image image obtained and provided by the endoscope, an arm operation unit for receiving the operation information for the operation of the controlled object, the control object is moved It includes a control unit for controlling to display the area image corresponding to the area including the plurality of points of the point input command inputted in the outline to the image image is overlaid on the image image, the object to be controlled includes: robot arm, instrument and endoscope There is provided a surgical robot, characterized in that at least one.

By inputting a point designation command and a segmentation command for two or more positions defining a boundary line connected to segment the region image, the region image can be divided into two or more separate regions.

By inputting a point designation command and a folding command to two or more locations that define a boundary line that is connected to segment the area image, the area image is rotated and overlapped with respect to the other one or more individual areas by the input of a point designation command and a folding command. Can be displayed.

The region image may be displayed by rotating and shifting the overlapping individual regions returned to their original positions by input of the restoration command.

One or more individual areas may be erased by inputting a point designation command and a delete command for two or more locations that define a boundary line connected to divide the area image.

The outline and the outer surface of the area image are recognized as a series of points, and by the point designation and movement command for any one point, the area image can be transformed.

The coordinate range extracted to correspond to the area image may be set as a restricted area or allowed area.

If the coordinate range extracted to correspond to the area image is set as the restricted area, an operation determination unit for determining whether the outer surface of the control object is in contact with the coordinate range by referring to the displacement information of the control object according to the operation information is further included. Can be.

When the external surface of the object to be controlled is in contact with the coordinate range, the controller may control the object to be controlled to be restricted, and may control the response information to be output accordingly.

The arm operation setting unit may further include an arm operation setting unit for generating and storing operation setting information on whether to allow operation of each control target object by an operation of the arm operation unit, and the operation determination unit may determine whether the operation information for any control target object is valid operation information. It can be further judged.

According to still another aspect of the present invention, there is provided a method for setting a region of a surgical robot, the method comprising: displaying an image image obtained and provided by an endoscope; receiving operation information for manipulation of a controlled object; And controlling a region image corresponding to a region including a plurality of positions where a point designation command is input among the positions to which the object is moved is overlaid on the video image to be displayed, wherein the object to be controlled includes a robot arm and an instrument. And at least one of endoscopes, and a coordinate range extracted to correspond to an area image is set to a restricted area or an allowable area.

According to another aspect of the present invention, a surgical robot, a display unit for displaying an image image obtained and provided by the endoscope, a storage unit for storing the human body modeling information and the corresponding modeling data, and obtained one or more organs And a control unit configured to extract modeling data corresponding to organ selection information for and control the display so that an area image corresponding to the modeling data is overlaid on the image image, wherein the modeling data includes at least one of color, shape, and size of the organ. There is provided a surgical robot comprising a.

Human body modeling information and modeling data may be generated using a reference image that is one or more of a CT image and an MRI image.

The organ selection information may be selectively designated as an organ corresponding to one or more of the shape and color of the organ recognized by the image recognition of the image image among the modeling data.

When the arm operation unit further receives operation information for manipulation of the controlled object including one or more of the robot arm, the instrument, and the endoscope, when one or more of the position change or shape change of the organ is generated by the operation information, The region image may be deformed to correspond to the outline of the organ.

According to still another aspect of the present invention, there is provided a method for setting a region of a surgical robot, the method comprising: storing human modeling information and corresponding modeling data, displaying an image image obtained and provided by an endoscope, and Extracting modeling data corresponding to organ selection information for one or more organs, and controlling the area image corresponding to the modeling data to be overlaid and displayed on the image image, wherein the modeling data includes the color, shape, and size of the organs. There is provided a method for setting a region of a surgical robot comprising one or more pieces of information.

According to another aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, the flow in the direction and size corresponding to the movement direction and size of the contacted operator's face ( A motion detector for outputting sensing information corresponding to the contacting part that is moved, the direction and size of the contacting part, and generating and outputting an operation command for at least one of the position and the image input angle of the vision part using the sensing information. There is provided a surgical robot including an operation command generation unit.

If the operation command relates to one or more of straight line operation and rotational operation of the vision unit, the operation handle direction of the surgical robot can be changed and operated accordingly.

The contact portion may be formed as part of a console panel of the surgical robot.

The support may be formed to protrude at one or more points of the contact portion to fix the position of the operator's face.

The eyepiece may be perforated in the contacting portion such that the image obtained by the vision portion is shown as visual information.

The contact portion may be formed of a material of light transmitting material so that the image obtained by the vision portion is shown as visual information.

The surgical robot includes a touch sensing unit for detecting whether the operator's face is in contact with the contact portion or the support, and a reference state in which the contact portion is designated as a default position and state when the contact release is recognized by the sensing of the contact sensing unit. It may further include an original state recovery unit for processing to return to.

The original state restoration unit may process the contact portion to return to the reference state by reverse manipulation of the contact portion flow direction and size according to the sensing information.

The surgical robot may further include an eye tracker unit configured to compare the sequentially generated image data in order of time to generate analysis information for analyzing one or more of a change in the position of the eye, a change in the shape of the eye, and a gaze direction. The camera unit may further include a camera unit for generating image data by capturing toward the contact portion inside the surgical robot, and a storage unit for storing the generated image data.

The operation command generation unit may determine whether the analysis information satisfies a preset change as an arbitrary operation command, and output a corresponding operation command when the operation information is satisfied.

The vision portion may be any one of a microscope and an endoscope, and the endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal and cardiac.

The contact portion is formed on the front surface of the console panel to be supported by the elastic body, the elastic body may provide a restoring force so that the contact portion is returned to its original position when the external force for the movement of the contact portion is removed.

According to another aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, the eyepiece for providing the image obtained by the vision unit as visual information; The eye tracker unit for generating analysis information analyzing one or more of a change in the position of the pupil, a change in the shape of the eye, and a viewing direction seen through the eyepiece; and whether the analysis information satisfies a preset change as an arbitrary operation command. There is provided a surgical robot comprising an operation command generation unit for determining and outputting an operation command for operating the vision unit when satisfied.

The eye tracker unit includes a camera unit for imaging the eyepiece from the inside of the surgical robot to generate image data, a storage unit for storing the generated image data, and sequentially comparing the image data generated sequentially to determine the position of the pupil. It may include an eye tracker unit for generating analysis information that interprets one or more of a change, an eye shape change, and a viewing direction.

The eyepiece may be perforated in a contact portion formed as part of a console panel of the surgical robot.

According to another aspect of the invention, the surgical robot for controlling one or more of the position of the vision portion and the image input angle, the step of outputting the sensing information corresponding to the direction and the size of the contact portion flows; And generating and outputting an operation command for at least one of the position and the image input angle of the vision unit using the sensing information, wherein the contact portion is formed as part of the console panel of the surgical robot, Provided is a method for operating a vision unit, which is configured to flow in a direction and a size corresponding to the direction and size of movement.

The vision unit manipulation method may further include determining whether the operator's face is in contact with the contact portion, and if the contact is in contact with the operator, controlling to start output of sensing information.

The operation method of the vision unit includes determining whether the contact portion exists as a reference state which is a position and state designated as default when the contact is released, and when the contact state is not present, returning to the reference state. It may further comprise the step of processing.

The return to the reference state may be performed by inverse manipulation of the contact portion flow direction and size according to the sensing information.

Vision operating method includes the steps of generating and storing the image data captured from the inside of the surgical robot toward the contact portion, and comparing the stored image data in chronological order to determine the position of the pupil, the change of the eye shape and the direction of attention The method may further include generating interpretation information of one or more interpretations.

The vision unit manipulation method may further include determining whether the analysis information satisfies a preset change as an arbitrary manipulation command, and outputting a corresponding manipulation command that is preset accordingly if satisfied.

According to another aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, a contact for providing the image obtained by the vision unit as visual information An analysis processor for generating analysis information for analyzing the face part, the movement of the face seen through the contact part, and determining whether the analysis information satisfies a preset change as an arbitrary operation command, and if so, to operate the vision part. There is provided a surgical robot including an operation command generation unit for outputting an operation command.

The analysis processing unit includes a camera unit for generating image data by imaging toward the contact portion inside the surgical robot, a storage unit for storing the generated image data, and a change in position of a predetermined feature point in the sequentially generated image data. It may include an analysis unit for generating the analysis information on the movement of the face by comparing and determining in chronological order.

The contact portion may be formed as part of a console panel of the surgical robot, and the contact portion may be formed of a material of light transmitting material so that the image obtained by the vision part is viewed as visual information. Can be.

According to another aspect of the invention, the surgical robot is a control method for controlling one or more of the position and the image input angle of the vision portion, one of the position change, eye shape change and the direction of eye viewing through the eyepiece Generating analysis information that interprets the above, determining whether the analysis information satisfies a preset change as an arbitrary operation command, and if satisfactory, manipulating at least one of the position of the vision unit and the image input angle. A vision operation method is provided that includes generating and outputting a command.

The generating of the analysis information includes generating and storing image data photographed toward the eyepiece from the inside of the surgical robot, and comparing the stored image data in chronological order to determine the position of the pupil, the change of the eye shape, and the gaze direction. It may include generating analysis information that interprets one or more of the.

According to another aspect of the present invention, an image input unit for receiving an endoscope image provided from the surgical endoscope, a screen display unit for outputting the endoscope image in a specific region, and a screen for outputting the endoscope image corresponding to the viewpoint of the surgical endoscope There is provided an immersive surgical image processing apparatus including a screen display control unit for changing a specific area of a display unit.

Here, the surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, cystoscopic, rectal, duodenum, mediastinal, cardiac, or may be a stereoscopic endoscope.

Here, the screen display control unit, the endoscope perspective tracking unit for tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, and the image to extract the movement information of the endoscope image using the perspective information of the surgical endoscope The apparatus may include a movement information extracting unit and an image position setting unit for setting a specific region of the screen display unit on which the endoscope image is output using the movement information.

In addition, the screen display controller may move the center point of the endoscope image corresponding to the coordinate change value of the viewpoint of the surgical endoscope.

According to another aspect of the invention, the image input unit for receiving the first endoscope image and the second endoscope image provided at different time points from the surgical endoscope, and outputting the first endoscope image and the second endoscope image to different areas A screen display unit, an image storage unit for storing the first endoscope image and the second endoscope image, and a screen display unit for outputting the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope There is provided an immersive surgical image processing apparatus including a screen display control unit for controlling the same.

The image input unit may receive the first endoscope image before the second endoscope image, and the screen display unit may differently output one or more of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. Can be.

The screen display controller may further include a storage image display unit which extracts the first endoscope image stored in the image storage unit and outputs the first endoscope image stored in the image storage unit while the screen display unit outputs the second endoscope image input in real time.

According to another aspect of the invention, the image input unit for receiving an endoscope image provided from the surgical endoscope, a screen display unit for outputting the endoscope image to a specific area, and a surgical tool for operating a surgical target taken by the surgical endoscope Change a specific area of an image storage unit for storing a modeling image, an image matching unit for generating an output image by matching an endoscope image and a modeling image, and a screen display unit for outputting an endoscope image in accordance with a surgical endoscope In addition, there is provided a tangible surgical image processing apparatus including a screen display control unit for outputting the matched endoscope image and modeling image to the screen display unit.

Here, the image matching unit may generate an output image by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.

The image matching unit may further include a characteristic value calculator configured to calculate a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm, and a characteristic value calculated by the characteristic value calculator. The apparatus may further include a modeling image implementation unit for implementing a modeling image.

Here, the image matching unit may further include an overlapping image processor which removes an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image, and the position of the modeling surgical tool image output on the modeling image is It can be set using the operation information of the surgical instrument.

According to another aspect of the invention, the image input unit for receiving the endoscope image provided from the surgical endoscope, the screen display unit for outputting the endoscope image, the screen drive unit for rotating and moving the screen display, and the endoscope for surgery Correspondingly, there is provided a immersive surgical image processing apparatus including a screen driving control unit which controls the screen driving unit so that the screen driving unit rotates and moves the screen display unit.

Here, the screen driving control unit, the endoscope perspective tracking unit for tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, and the image to extract the movement information of the endoscope image using the perspective information of the surgical endoscope The apparatus may include a movement information extracting unit and a driving information generating unit generating screen driving information of the screen display unit using the movement information.

Here, the screen display unit may include a dome-shaped screen and a projector that projects an endoscope image on the dome-shaped screen.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image to a specific area of the screen display unit and There is provided a tangible surgical image processing method comprising the step of changing a specific area of the screen display unit for outputting the endoscope image corresponding to the viewpoint of the surgical endoscope.

In addition, the step of changing a specific area of the screen display unit, the step of tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, extracting the movement information of the endoscope image using the perspective information of the surgical endoscope And setting a specific area of the screen display unit on which the endoscope image is output using the movement information.

Here, the changing of the specific area of the screen display unit may include moving the center point of the endoscope image corresponding to the coordinate change value of the viewpoint of the surgical endoscope.

According to another aspect of the present invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving the first endoscope image and the second endoscope image provided at different time points from the surgical endoscope, the first Outputting the endoscope image and the second endoscope image to different areas of the screen display unit, storing the first endoscope image and the second endoscope image, and the first endoscope image and the second corresponding to different perspectives of the surgical endoscope; There is provided a tangible surgical image processing method comprising controlling the screen display unit to output an endoscope image to different areas.

Here, the receiving of the endoscope image may include receiving the first endoscope image before the second endoscope image, and the outputting step may include any one of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. You can print one or more differently.

The controlling of the screen display unit may further include extracting and outputting the first endoscope image stored in the image storage unit to the screen display unit while the screen display unit outputs the second endoscope image input in real time.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image to a specific area of the screen display unit, Storing a modeling image of a surgical tool for operating a surgical target photographed by the surgical endoscope; generating an output image by matching the endoscope image and the modeling image; and outputting the endoscope image corresponding to the perspective of the surgical endoscope There is provided a tangible surgical image processing method including changing a specific region of a screen display unit, and outputting a matched endoscope image and a modeling image to a screen display unit.

Here, in the generating of the output image, the output image may be generated by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.

The generating of the output image may include calculating a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm, and generating a modeling image corresponding to the calculated characteristic value. It may further comprise the step of implementing.

The generating of the output image may further include removing an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

In addition, the position of the modeling surgical tool image output to the modeling image may be set using the operation information of the surgical tool.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image on the screen display and the endoscope for surgery According to the aspect of the present invention, there is provided a haptic surgical image processing method comprising the step of rotating and moving the screen display.

Here, the rotating and moving the screen display unit, tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, extracting the movement information of the endoscope image using the perspective information of the surgical endoscope The method may include generating motion information of the screen display unit using the movement information.

According to another aspect of the present invention, a program of instructions that can be executed by a digital processing apparatus is tangibly embodied in order to perform the above-described immersive surgical image processing method, and a program that can be read by a digital processing apparatus. Recorded recording medium is provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

Surgical image processing apparatus and method according to the present invention has the effect of enabling a smooth operation by providing a real image and a modeling image together during surgery.

In addition, the surgical image processing device and the method according to the present invention has the effect of allowing the operator to operate by referring to the image of the neighboring site and the external environment of the patient as well as the surgical site apart from the conventional method of operating by looking only at the surgical site There is.

In addition, since the surgical image processing apparatus and the method according to the present invention can detect and warn the collision of the surgical tools in advance, the operator has an effect that can perform the operation smoothly by recognizing the collision state in advance and avoiding it.

In addition, the surgical image processing apparatus and method according to the present invention has the effect that the operator can use the surgical image more conveniently by implementing an interface that the operator can select the type of the image to be displayed on the monitor for convenience.

In addition, according to an embodiment of the present invention, there is an effect that the instrument is controlled only in the manner that the operator intends to perform normal surgery.

In addition, the erroneous manipulation of the robot arm and / or instrument is fundamentally prevented, so that the safety of the patient under surgery can be improved.

In addition, according to an embodiment of the present invention, there is an effect that the position of the laparoscope and the image input angle can be controlled only by the operator to see the desired surgical site.

In addition, the operator does not need a separate operation for the operation of the laparoscopic has the effect of allowing the operator to focus on the operation only.

In addition, the bodily-type surgical image processing apparatus and method according to the present invention, by changing the output position of the endoscope image output to the monitor in accordance with the viewpoint of the endoscope changes according to the movement of the surgical endoscope, The effect is to make the actual surgical situation feel more realistic.

In addition, the apparatus and method for immersive surgical image processing according to the present invention extracts a previously input and stored endoscope image at the present time point and outputs the information on the change of the endoscope image by outputting it to the screen display together with the current endoscope image. There is an effect that can inform the user.

In addition, the bodily-type surgical image processing apparatus and method according to the present invention match each or each of the modeling images that are actually generated by using an endoscope during surgery and modeling images previously generated for a surgical tool and stored in an image storage unit or the same. It is effective to modify the image such as adjusting the size and output it to the monitor that the user can observe.

In addition, the bodily-type surgical image processing apparatus and method according to the present invention has an effect that allows the user to feel more realistic about the surgery by rotating and moving the monitor in accordance with the viewpoint of various changes in the endoscope.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.

Figure 2 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

Figure 3 is a block diagram of a surgical robot according to an embodiment of the present invention.

Figure 4 is a block diagram of a surgical image processing apparatus according to an embodiment of the present invention.

5 is a flow chart of a surgical image processing method according to an embodiment of the present invention.

6 is a block diagram of an output image according to the surgical image processing method according to an embodiment of the present invention.

Figure 7 is a block diagram of a surgical robot according to an embodiment of the present invention.

8 is a block diagram of a surgical image processing apparatus according to an embodiment of the present invention.

9 is a flow chart of a surgical image processing method according to an embodiment of the present invention.

10 is a block diagram of an output image according to the surgical image processing method according to an embodiment of the present invention.

Figure 11 is a block diagram of a surgical robot according to an embodiment of the present invention.

12 is a flow chart of a surgical image processing method according to an embodiment of the present invention.

Figure 13 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.

14 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

15 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.

16 to 27 illustrate a method for designating a region according to embodiments of the present invention.

28 is a flowchart illustrating a method for setting a restricted area according to an embodiment of the present invention.

29 and 30 are flow charts showing the operation limiting method of the surgical robot system according to an embodiment of the present invention.

31 is an exemplary view of a screen display for explaining an operation limiting method according to an embodiment of the present invention.

32 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.

33 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

34 to 37 are views illustrating the movement form of the contact portion according to the embodiment of the present invention.

FIG. 38 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention. FIG.

39 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

40 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention.

41 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

FIG. 42 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention. FIG.

43 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

44 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

45 is a diagram illustrating an image display form by a telescopic display unit according to an embodiment of the present invention.

46 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

47 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.

48 is a conceptual diagram illustrating a master interface of a surgical robot according to an embodiment of the present invention.

49 is a block diagram of a surgical robot according to an embodiment of the present invention.

50 is a block diagram of a bodily-type surgical image processing apparatus according to an embodiment of the present invention.

51 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention.

52 is a block diagram of an output image according to the tangible surgical image processing method according to the embodiment of the present invention.

53 is a block diagram of a surgical robot according to an embodiment of the present invention.

54 is a block diagram illustrating an apparatus for immersive surgical image processing according to an embodiment of the present invention.

55 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention.

56 is a block diagram of an output image according to the tangible surgical image processing method according to the embodiment of the present invention.

57 is a block diagram of a surgical robot according to an embodiment of the present invention.

58 is a block diagram of an apparatus for processing immersive surgical images according to an embodiment of the present invention.

59 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention.

60 is a block diagram of an output image according to the tangible surgical image processing method according to an embodiment of the present invention.

61 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

62 is a block diagram of a surgical robot according to an embodiment of the present invention.

FIG. 63 is a block diagram illustrating an apparatus for immersive surgical image processing according to an embodiment of the present invention. FIG.

64 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention.

65 is a conceptual diagram illustrating a master interface of a surgical robot according to an embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

In addition, the terms "... unit", "... group", "module" and the like that can be described in the specification means a unit for processing at least one function or operation, which means hardware or software or hardware and It can be implemented in a combination of software.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals and redundant description thereof will be omitted. Shall be.

In addition, in describing various embodiments of the present invention, each embodiment should not be interpreted or implemented independently, and the feature elements and / or technical ideas described in each embodiment may be combined with other embodiments separately described. It is to be understood that it may be interpreted or practiced.

In addition, through the following description it will be apparent that the present invention is a technical idea that can be used universally for surgery or experiments in which vision parts such as endoscopes and microscopes are used. In addition, the endoscope may be diversified into laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, and the like. However, hereinafter, the case of the vision unit is a kind of endoscope, that is, laparoscope for the convenience of explanation and understanding will be described as an example.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention, Figure 2 is a conceptual diagram showing a master interface of a surgical robot according to an embodiment of the present invention.

In this embodiment, the endoscope image actually taken by using the endoscope during surgery and the modeling image generated and stored in advance for the surgical instrument are modified or matched with each other, or the size thereof is modified and output to a monitor that can be observed by the operator. There are features that can be done. In addition, the present embodiment outputs a camera image generated by photographing a surgical target (patient) lying on the operating table with a endoscopic image and / or modeling image outputted in combination with the endoscope image and / or modeling image, the surgeon is operating from the conventional operation while looking only at the surgical site In addition to the site, there is a feature that allows the operation by referring to the image of the adjacent area and the external environment of the patient.

The endoscope according to the present embodiment may be not only a laparoscope but also various kinds of tools used as an imaging tool during surgery, such as thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, etc. In addition, the surgical image processing apparatus according to the present embodiment is not necessarily implemented to be limited to the surgical robot system as shown, and may be applied to a system for outputting an endoscope image during surgery and operating using a surgical tool. Hereinafter, the case where the surgical image processing apparatus according to the present embodiment is applied to the surgical robot system will be described.

1 and 2, the surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 remotely controlling the slave robot 2. . The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor part 6 and a master controller, and the slave robot 2 comprises a robot arm 3 and an instrument 8. The instrument 8 is a surgical tool such as an endoscope, such as a laparoscope, or a surgical instrument for directly manipulating the affected part. The master interface 4 may further include a button for selecting an image, which will be described later. The image selection button may be implemented in the form of a clutch button or a pedal (not shown), and the implementation of the image selection button is not limited thereto. For example, a function menu or a mode selection displayed through the monitor unit 6 may be selected. It may also be implemented as a menu.

The master interface 4 is provided with a master controller so that the operator can be gripped and manipulated by both hands. As illustrated in FIGS. 1 and 2, the master controller may be implemented with two handles 10. An operation signal according to the manipulation of the operator's handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be moved. Controlled. By the operation of the operator's handle 10, the position movement, rotation, cutting, etc. of the robot arm 3 and / or the instrument 8 may be performed.

For example, the handle 10 may be composed of a main handle and a sub handle. It is also possible to operate the slave robot arm 3, the instrument 8, etc. with only one handle, or to operate a plurality of surgical equipment in real time by adding a sub handle. The main handle and the sub handle may have various mechanical configurations depending on the operation method thereof. For example, the robot arm 3 and / or other surgery of the slave robot 2, such as a joystick type, a keypad, a trackball, and a touch screen, may be used. Various input means for operating the equipment can be used.

The master controller is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

The monitor 6 of the master interface 4 displays an endoscope image, a camera image, and a modeling image input by the instrument 8 as an image image. In addition, the information displayed on the monitor unit 6 may vary according to the type of the selected image.

The monitor unit 6 may be composed of one or more monitors, and may display information necessary for surgery on each monitor separately. 1 and 2 illustrate a case in which the monitor unit 6 includes three monitors, the quantity of the monitors may be variously determined according to the type or type of information requiring display.

The slave robot 2 and the master robot 1 may be coupled to each other through a wired communication network or a wireless communication network so that an operation signal and an endoscope image input through the instrument 8 may be transmitted to the counterpart. If two operation signals provided by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the instrument 8 need to be transmitted at the same time and / or at a similar time point, Each operation signal may be independently transmitted to the slave robot 2. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical tool inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical tool in the yaw direction according to a surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical tool, a transfer drive for moving the surgical tool in the longitudinal direction, a rotation drive for rotating the surgical tool, the surgical tool drive is installed on the end of the surgical tool to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. In addition, the actual control process, such as the operator rotates the robot arm 3 in the corresponding direction by operating the handle 10 is somewhat distanced from the subject matter of the present invention, so a detailed description thereof will be omitted.

One or more slave robots 2 may be used to operate the patient, and the instrument 8 for causing the surgical site to be displayed as an image image through the monitor unit 6 may be implemented as an independent slave robot 2. In addition, the master robot 1 may also be implemented integrally with the slave robot 2.

3 is a block diagram schematically showing the configuration of a surgical robot according to an embodiment of the present invention. Referring to FIG. 3, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the image matching unit 350, the image storage unit 360, and the control unit 370. A master robot 1 comprising a and a robot arm 3, a slave robot 2 including a laparoscope 5 are shown.

Surgical image processing apparatus according to the present embodiment may be implemented as a module including an image input unit 310, the screen display unit 320, the image matching unit 350, the image storage unit 360, of course, such a module The control unit may include an operation signal generator 340, an image matcher 350, and a controller 370.

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network. Laparoscope 5 may also be one type of surgical instrument according to the present embodiment, and the number thereof may be one or more.

The screen display unit 320 outputs an image image corresponding to the image received through the image input unit 310 as visual information. The screen display unit 320 may output the endoscope image as it is or zoom in / zoom out, or may match the endoscope image and the modeled image with each other, or may output each as a separate image.

In addition, the screen display unit 320 is an image that reflects the endoscope image and the entire surgical situation, for example, as described later, the camera image generated by the camera photographing the outside of the operation target and / or matched with each other and outputs the surgery It can also make the situation easier to grasp. In addition, the screen display unit 320 outputs a reduced image of the entire image (endoscopic image, modeling image, camera image, etc.) in a portion of the output image or a window generated on a separate screen, and the operator described above When selecting or rotating a specific point among the reduced images output using the master controller, the entire output image may be moved or rotated, so-called bird's eye view function of the CAD program. Functions such as zooming in / out, moving, and rotating the image output to the screen display unit 320 as described above may be controlled by the controller 370 according to the operation of the master controller.

The screen display unit 320 may be implemented in the form of a monitor unit 6. An image processing process for outputting a received image as an image image through the screen display unit 320 may include a control unit 360 and an image matching unit. 350 or by a separate image processor (not shown). Alternatively, the screen display unit 320 may be a terminal for outputting an image to the monitor unit 6. In this case, the present embodiment does not include a display like the monitor unit 6, and the screen unit 320 may be used. ) May be any means (hardware or software) for transmitting the image signal to the monitor unit 6.

The arm manipulation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2. The arm manipulation unit 330 may be formed in the shape of the handle 10 as illustrated in FIG. 2, but the shape is not limited thereto and may be modified in various shapes for achieving the same purpose. Further, for example, some may be formed in the shape of a handle, others may be formed in other shapes such as a clutch button, and a finger cannula or insertion may be inserted to fix the operator's finger to facilitate manipulation of the surgical tool. More rings may be formed.

The operation signal generator 340 generates a corresponding operation signal when the operator manipulates the arm operation unit 330 for the movement of the robot arm 3 and / or the laparoscope 5 or the operation for surgery. Transfer to the robot (2). As described above, the manipulation signal may be transmitted and received through a wired or wireless communication network.

The image matching unit 350 generates an output image by matching the endoscope image received through the image input unit 310 with the modeling image of the surgical tool stored in the image storage unit 360, and outputs the output image to the screen display unit 320. do. The endoscope image is an image of the inside of the patient's body using the endoscope. Since the image is obtained by capturing only a limited area, the endoscope image includes an image of a part of the surgical instrument.

The modeling image is an image generated by realizing the shape of the entire surgical tool as a 2D or 3D image. The modeling image may be an image of a surgical tool photographed at a specific time point before the start of surgery, for example, an initial setting state. Since the modeling image is an image generated by a computer simulation technique of a surgical tool, the image matching unit 350 may match and output the surgical tool and the modeling image shown in the actual endoscope image. Since a technique of obtaining an image by modeling a real object has a little distance from the gist of the present invention, a detailed description thereof will be omitted. In addition, specific functions, various detailed configurations, and the like of the image matching unit 350 will be described in detail with reference to the accompanying drawings.

The controller 360 controls the operation of each component so that the above-described function can be performed. The controller 360 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320. In addition, the controller 360 controls the image matching unit 350 to output the modeling image through the screen display unit 320 in response to the manipulation information according to the manipulation of the arm manipulation unit 330.

The actual surgical tool included in the endoscope image is a surgical tool included in the image inputted by the laparoscope 5 and transmitted to the master robot 1, and is a surgical tool that applies a surgical operation directly to the patient's body. In contrast, the modeling surgical tool included in the modeling image is mathematically modeled with respect to the entire surgical tool in advance and stored in the image storage unit 360 as a 2D or 3D image. Surgical tools and modeling images of the endoscope image Surgical tools are controlled by the operation information (that is, information about the movement, rotation, etc. of the surgical tool) recognized by the master robot 1 as the operator operates the cancer operation unit 330 Can be. In actual surgical tools and modeling surgical tools, their position and manipulation shape may be determined by the manipulation information.

The operation signal generator 340 generates an operation signal by using the operation information according to the operation of the operator's arm operation unit 340, and transmits the generated operation signal to the slave robot 2. To be manipulated accordingly. In addition, the position and operation shape of the actual surgical instrument operated by the operation signal can be confirmed by the operator by the image input by the laparoscope (5).

In addition, the modeling image may include an image reconstructed by modeling not only the surgical instrument but also the organ of the patient. In other words, the modeled images are CT (Computer Tomography), MR (Magnetic Resonance), PET (Positron Emission Tomography), SPECT (Single Photon Emission Computed Tomography), single photon emission tomography ), Which may include 2D or 3D images of the organ surface of the patient, reconstructed with reference to images acquired from imaging equipment such as US (Ultrasonography), in which case the actual endoscope image and the computer modeling image are matched. It is more effective to provide the operator with a full image including the surgical site.

Figure 4 is a block diagram of a surgical image processing apparatus according to an embodiment of the present invention. Referring to FIG. 4, the image matcher 350 may include a feature value calculator 351, a modeled image implementer 353, and an overlapped image processor 355.

The characteristic value calculator 351 uses the characteristic value by using the image inputted by the laparoscope 5 of the slave robot 2 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The actual position of the surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2, the information on the position may be provided from the slave robot 2 to the master robot (1). .

The characteristic value calculator 351 may use, for example, a field of view (FOV), an enlargement ratio, a viewpoint (for example, a viewing direction), a viewing depth, etc. of the laparoscope 5 using an image of the laparoscope 5. In addition, it is possible to calculate characteristic values such as the type, direction, depth, and degree of bending of the actual surgical instrument. When the characteristic value is calculated using the image of the laparoscope 5, an image recognition technique for recognizing the outline of the subject included in the image, shape recognition, tilt angle, or the like may be used. In addition, the type of the actual surgical tool may be input in advance in the process of coupling the corresponding surgical tool to the robot arm (3).

The modeling image implementer 353 implements a modeling image corresponding to the feature value calculated by the feature value calculator 351. Data related to the modeled image may be extracted from the image storage unit 360. That is, the modeling image implementer 353 may determine the characteristic values of the laparoscope 5 (field of view (FOV), magnification, perspective, viewing depth, etc., type, direction, depth, degree of bending, etc. of the actual surgical tool). The modeling image is implemented to extract the modeling image data of the corresponding surgical tool and the like to match the surgical tool of the endoscope image.

Various methods may be implemented by the modeling image implementer 353 to extract an image corresponding to the feature value calculated by the feature value calculator 351. For example, the modeling image implementer 353 may extract a modeling image corresponding to the characteristic value of the laparoscope 5 directly. That is, the modeling image implementer 353 may extract the 2D or 3D modeling surgical tool image corresponding to the laparoscopic 5 and the data, such as the angle of view and the magnification of the laparoscope 5, and match it with the endoscope image. Here, the characteristic values such as the angle of view and the magnification may be calculated by comparing and analyzing the images of the laparoscope 5 which are calculated or sequentially generated through comparison with the reference image according to the initial set value.

In addition, according to another exemplary embodiment, the modeling image implementer 353 may extract the modeling image by using manipulation information for determining the position and the manipulation shape of the laparoscope 5 and the robot arm 3. That is, as described above, since the surgical tool of the endoscope image may be controlled by the operation information recognized by the master robot 1 as the operator manipulates the cancer operation unit 330, the modeling surgery corresponding to the characteristic value of the endoscope image is performed. The position and manipulation shape of the tool can be determined by the manipulation information.

Such manipulation information may be stored in a separate database according to the temporal order, and the modeling image implementer 353 may recognize the characteristic value of the actual surgical tool by referring to the database, and correspondingly, the information about the modeling image. Can be extracted. That is, the position of the surgical tool output on the modeling image may be set using cumulative data of the position change signal of the surgical tool. For example, if the operation information for the surgical instrument includes information that rotates 90 degrees clockwise and 1 cm in the extension direction, the modeling image implementation unit 353 is included in the modeling image corresponding to the operation information. The image of the surgical instrument can be converted and extracted.

Here, the surgical instrument is mounted to the front end of the surgical robot arm is provided with an actuator, the driving wheel (not shown) provided in the drive unit (not shown) by receiving the driving force from the actuator is operated, connected to the drive wheel and surgery The operator 150 inserted into the patient's body performs surgery by performing a predetermined operation. The driving wheel is formed in a disc shape, and may be clutched to the actuator to receive the driving force. In addition, the number of driving wheels may be determined corresponding to the number of objects to be controlled, and the description of such driving wheels will be apparent to those skilled in the art related to surgical instruments, and thus detailed description thereof will be omitted.

The superimposed image processor 355 outputs a partial image of the modeled image so that the actually captured endoscope image and the modeled image do not overlap. That is, when the endoscope image includes some shape of the surgical tool and the modeling image implementer 353 outputs the corresponding modeling surgical tool, the superimposed image processor 355 may perform the actual surgical tool image and the modeling surgical tool image of the endoscope image. By checking the overlapping region of, and deleting the overlapping portion from the modeling surgical tool image, the two images can be matched with each other. The overlapping image processor 355 may process the overlapping region by removing the overlapping region of the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

For example, the total length of an actual surgical instrument is 20 cm, and characteristics values (field of view (FOV), magnification, perspective, depth of view, etc., type, direction, depth, degree of bending, etc. of the actual surgical instrument) are considered. When the length of the actual surgical tool image of the endoscope image is 3cm, the superimposed image processor 355 outputs the modeling surgical tool image which is not output to the endoscope image by using the characteristic value in the modeling image.

In operation S510, a modeling image is generated and stored in advance with respect to the surgical target and / or the surgical tool. The modeled image may be modeled by computer simulation, and the embodiment may generate a modeled image by using a separate modeling image generating apparatus.

In step S520, the endoscope is used to generate an endoscope image adjacent to the affected part of the patient. Endoscopic images are images of organs and surgical instruments of a patient, and include some images thereof.

In operation S530, the characteristic value calculator 351 calculates a characteristic value of the endoscope image. As described above, the characteristic value calculator 351 may input an image provided by the laparoscope 5 of the slave robot 2 and / or coordinate information about the position of the actual surgical tool coupled to the robot arm 3. The characteristic value is calculated using the field of view (FOV, field of view), magnification, perspective (for example, viewing direction), viewing depth, and the type, direction, and depth of the surgical instrument. , Bend, and so on.

In operation S540, the image matching unit 350 extracts the modeling image corresponding to the endoscope image, processes the overlapping region, and matches the two images to each other and outputs the image.

6 is an exemplary view of an output image according to a surgical image processing method according to an embodiment of the present invention. Referring to FIG. 6, a modeling image 610, a first modeling surgical tool 612, a second modeling surgical tool 614, an endoscope image 620, and an actual surgical tool 622 are shown.

The endoscope image 620 is located in the center region of the modeling image 610 and the modeling image 610 is output to the peripheral region of the endoscope image 620, the endoscope image 620 is not particularly limited to the position, For example, the screen display unit 320 may be output at various positions such as the upper right side, the upper left side, the lower right side, and the lower left side. The endoscope image 620 may be output as a full screen of the screen display unit 320 or as a zoomed out image so as to have an effect spaced apart from the screen as shown. In the latter case, the modeling image 610 may be matched with the endoscope image 620 and output together.

The first modeling surgical tool 612 is an image that is not output to the endoscope image 620 but is a surgical tool positioned adjacent to the actual surgical site. The second modeling surgical tool 614 is a modeling image in which the actual surgical tool 622 included in the endoscope image 620 is extended. The actual surgical tool 622 and the second modeling surgical tool 614 may be shown using characteristic values as described above to match each other and not overlap each other.

In addition, such an image may be output for the operator's convenience through functions such as zoom in, zoom out, rotation, refresh, and focus shift. For example, when the total size of the endoscope image 620 is reduced, the size of the output to the full screen display 320 is determined. Therefore, the size of the first modeling surgical tool 612 and the second modeling surgical tool 614 and The exposure image may be determined and output. Also, the endoscope image 620 and the modeling image 610 may be implemented in 3D so that when the operator rotates the image in a predetermined direction, the modeling image may be converted and output according to the rotation direction.

Although the first modeling surgical tool 612 and the second modeling surgical tool 614 are shown as a surgical instrument having an operation unit capable of manipulating the affected part at one end, the present invention is not limited thereto. The first modeling surgical tool 612 and / or the second modeling surgical tool 614 may be a surgical tool such as a laparoscope.

7 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 7, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the image matching unit 350, the image storage unit 360, and the control unit 370. A slave robot 2 including a master robot 1 and a robot arm 3, a camera 7, and a laparoscope 5 are shown. The differences from the above will be explained mainly.

The present embodiment combines the image of the patient taken from the outside with the above-described endoscope image and modeling image to provide the operator with the image not only inside the patient but also outside the patient to perform the operation more effectively. There is this.

The camera 7 is provided to the outside of the patient, for example, the operating room ceiling, one side of the operating table and the like to photograph the operation site and generate an image. The image input unit 310 receives a camera image generated by the camera 7 and is displayed on the screen display 320.

As described above, the image matching unit 350 registers two or more images of the endoscope image, the modeling image, and the camera image and displays them on the screen display unit 320. Accordingly, the images output together according to the present embodiment may be various combination images, for example, a combination of an endoscope image and a modeling image, a combination of an endoscope image and a camera image, a combination of a modeling image and a camera image, and the like. Can be.

8 is a block diagram of a surgical image processing apparatus according to an embodiment of the present invention. Referring to FIG. 8, the image matching unit 350, the characteristic value calculating unit 351, the modeling image implementing unit 353, the overlapping image processing unit 355, the collision detecting unit 357, and the warning information output unit 359 may be used. Shown. The differences from the above will be explained mainly.

This embodiment has a feature that can alert the operator by detecting if there is a risk of collision of the above-described surgical instruments. That is, the surgical tools may collide with each other not only within the endoscope image but also outside thereof, and the present embodiment may detect and warn the collision of these surgical tools in advance, so that the operator recognizes the collision state in advance and avoids the surgery. There is an advantage that can be performed smoothly.

The collision detector 357 detects whether the modeling surgical tool images included in the modeling image collide with each other. Surgical instruments such as surgical instruments and laparoscopics 5 are determined in position and operation shape in accordance with the above-described operation information, so that when the operator operates the arm operation unit 330, the collision detection unit 357 receives the operation information. The analysis determines whether the surgical tools collide with each other, and generates a collision detection signal if the surgical tools are manipulated to the colliding position by the operation information.

The warning information output unit 359 may receive a collision detection signal from the collision detection unit 357 and output predetermined warning information. The warning information may be information that can be recognized by the operator, such as sound information, color information, and vibration information. For example, an alarm sound transmitted from a predetermined speaker and text, an icon, and a border displayed on the screen display 320 may be used. It may be a color, background color change information, a shake signal of the arm manipulation unit 330, and the like. The friction sound generated when the actual surgical tool collides may be stored in advance, and then the friction sound may be sent as a warning sound when the collision is detected.

In addition, when the collision detection unit 357 determines that the surgical tools are in contact, for example, the arm operation unit 330 is no longer manipulated or the force feedback (force feedback) through the arm operation unit (330) Can be processed to occur. Form a virtual wall in the direction of the collision for force feedback, and detect when surgical tools approach each other's virtual walls when the operator manipulates the master controller and the surgical tools move along the virtual wall or By controlling to move in the other direction, it is possible to avoid the actual collision with each other. A separate component for processing force feedback may be included as a component of the master robot 1.

In step S510, a modeling image is generated and stored in advance with respect to the surgical object and / or the surgical tool, and in step S520, an endoscope image adjacent to the affected part of the patient is generated using the endoscope.

In operation S530, the characteristic value calculator 351 calculates a characteristic value of the endoscope image. In operation S540, the image matcher 350 extracts a modeling image corresponding to the endoscope image, and processes the overlapped region to generate two images. Match each other and output.

In step S550, the collision detection unit 357 analyzes and detects the operation information whether the surgical tool is in the collision position, and then exists in the collision position in step S560, the warning information output unit 359, as described above, the sound information, Output warning information such as color information, vibration information, etc.

Here, although the present embodiment has been described in the order of determining whether the surgical tools collide after extracting and matching the modeling image corresponding to the endoscope image, the present invention is not limited thereto. For example, the step of determining whether the surgical instruments are collided may be immediately performed whenever the above-described manipulation information is generated, or may be immediately performed whenever the endoscope image is generated.

10 is a block diagram of an output image according to the surgical image processing method according to an embodiment of the present invention. Referring to FIG. 10, the first robot arm 121, the second robot arm 122, the first surgical tool 202, the second surgical tool 204, the first driver 231, and the second driver 232. ), The modeling image 610, the first modeling surgical tool 612, the second modeling surgical tool 614, the endoscope image 620, the actual surgical tool 622, and the camera image 630 are shown. The differences from the above will be explained mainly.

The camera image 630 is an image generated by the camera 7 installed in the external environment of the patient as described above. The camera image 630 may include an image of an operating table, a patient's appearance, and the like. The first driving unit 231 and the second driving unit 232 are driving means for driving the first surgical tool 202 and the second surgical tool 204, respectively, and the description thereof is somewhat distance from the gist of the present invention. Detailed description thereof will be omitted.

Referring to FIG. 10, an endoscope image 620, a modeling image 610, and a camera image 630 are output while occupying a specific position of the screen display 320. As described above, the first modeling surgical tool 612, the first surgical tool 202, the second modeling surgical tool 614, and the second surgical tool 204 are computed, matched, and output. According to another embodiment, the camera image 630 may be entirely replaced by the modeling image 610. That is, the modeling image 610 may include a first robot arm 121, a second robot arm 122, a first surgical tool 202, a second surgical tool 204, a first driver 231, and a second driver. The image may be implemented by modeling the entire surgical system such as 232, which may be combined with and / or matched with the endoscope image 620 and output.

11 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 11, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generator 340, the image matching unit 350, the image storage unit 360, and the control unit 370. The master robot 1 including the image selecting unit 380 and the slave robot 2 including the robot arm 3, the camera 7, and the laparoscope 5 are shown. The differences from the above will be explained mainly.

This embodiment has a feature that the operator can use the surgical image more conveniently by implementing an interface that the operator can select the type of the image to be displayed on the screen display unit 320 for convenience. That is, according to the present embodiment, one or more images of the endoscope image, the modeling image, and the camera image as described above may be output by the operator's selection.

The image selector 380 is a means for selecting an image to be matched with each other and output from the above-described endoscope image, modeling image, and camera image. The image selection unit 380 may be an image selection button implemented in the form of a clutch button or a pedal (not shown), as described above, and a function menu or mode selection displayed through the monitor unit 6. It may also be implemented as a menu.

In addition, the image selecting unit 380 functions to zoom in, zoom out, rotate, refresh, horizontally move, 2D / 3D conversion, hide / display a specific image, and change viewpoints with respect to the above-described endoscope images, modeling images, and camera images. It may also include. That is, the image selector 380 may be an interface provided to express the image output by the operator at various angles.

In operation S525, any one of the screen display unit 320, the image matching unit 350, and the control unit 370 receives an image selection signal to be output from the image selection unit 380 and outputs an image from among an endoscope image, a modeling image, and a camera image. Select the video to be.

In operation S530, the characteristic value calculator 351 calculates a characteristic value of the endoscope image, and in operation S545, the image matcher 350 processes the overlapped area corresponding to the selected image to match and output the two images.

Other common platform technologies such as specific embedded systems, O / S, and communication protocols, interface standardization technologies such as I / O interfaces, and components such as actuators, batteries, cameras, sensors, etc. Detailed description of the standardization technology, etc. will be omitted since it is obvious to those skilled in the art.

Surgical image processing method according to the present invention can be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. In other words, the recording medium may be a computer readable recording medium having recorded thereon a program for causing the computer to execute the above steps.

The computer readable medium may include a program command, a data file, a data structure, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. -Magneto-Optical Media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

In the above, the surgical image processing apparatus according to an embodiment of the present invention described the configuration of a surgical tool and a robotic surgical system according to an embodiment, but is not necessarily limited thereto, the present invention is not a robot surgery Even if applied to a surgical image processing system or modeling and outputting a variety of other surgical instruments, such other configurations may be included in the scope of the present invention if there is no difference in the overall operation and effect.

13 is a plan view showing the overall structure of the surgical robot according to an embodiment of the present invention, Figure 14 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

Referring to FIGS. 13 and 14, the laparoscopic surgical robot system includes a slave robot 2 that actually performs surgery on a patient lying on an operating table, and a master robot 1 that remotely controls the slave robot 2. Include. 13 illustrates a case in which the master robot 1 and the slave robot 2 are separately implemented, the master robot 1 and the slave robot 2 are not necessarily separated into physically independent separate devices. It can be integrated into one and configured in one piece, in which case the master interface 4 can correspond to, for example, the interface part of the one-piece robot.

The master interface 4 of the master robot 1 comprises a monitor 6 and a master manipulator, and the slave robot 2 comprises a robot arm 3 and a laparoscope 5. The illustrated laparoscopic 5 is one embodiment of a surgical endoscope, and may be replaced with one or more of thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, etc.

The master manipulator of the master interface 4 is embodied so that the operator can grip and manipulate each hand. As illustrated in FIGS. 13 and 14, the master controller may be implemented with two handles 10 or more handles 10, and an operation signal according to the operator's manipulation of the handle 10 may be controlled by the slave robot 2. Is transmitted to control the robot arm 3 and / or the instrument (not shown). By operation of the operator's handle 10, for example, a surgical operation such as position movement, rotation, and cutting of the robot arm 3 may be performed. The master controller is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

For example, the handle 10 may be configured to include a main handle and a sub handle. The operator may operate the slave robot arm 3 or the laparoscope 5 or the like only by the main handle, or may operate the sub handles to simultaneously operate a plurality of surgical equipments in real time. The main handle and the sub handle may have various mechanical configurations depending on the operation method thereof. For example, the robot arm 3 and / or other surgery of the slave robot 2, such as a joystick type, a keypad, a trackball, and a touch screen, may be used. Various input means for operating the equipment can be used.

An image input by the laparoscope 5 may be displayed as an image image on the monitor unit 6 of the master interface 4. Of course, the manner in which the image input by the laparoscope 5 is displayed may be various in addition to the manner in which the image is displayed through the monitor 6, which is omitted from the description because it is somewhat distance from the gist of the present invention.

The monitor unit 6 may be composed of one or more monitors, and may display information necessary for surgery on each monitor separately. 13 and 14 illustrate the case where the monitor unit 6 includes three monitors, the quantity of the monitors may be variously determined according to the type or type of information requiring display.

The monitor unit 6 may further output one or more biometric information about the patient. In this case, one or more indicators indicating a patient's condition, for example, body temperature, pulse rate, respiration, and blood pressure, may be output through one or more monitors of the monitor unit 6. It may be output. In order to provide such biometric information to the master robot 1, the slave robot 2 includes a biometric information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like. It may include. The biometric information measured by each module may be transmitted from the slave robot 2 to the master robot 1 in the form of an analog signal or a digital signal, and the master robot 1 monitors the received biometric information. Can be displayed via

The slave robot 2 and the master robot 1 may be coupled to each other through a wired communication network or a wireless communication network so that an operation signal and a laparoscope image input through the laparoscope 5 may be transmitted to the counterpart. If two operation signals by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the laparoscope 5 need to be transmitted at the same time and / or at a similar point in time. Each operation signal may be independently transmitted to the slave robot 2. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical instrument inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical instrument in the yaw direction according to the surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical instruments, a transfer drive for moving the surgical instruments in the longitudinal direction, a rotation drive for rotating the surgical instruments, a surgical instrument drive unit installed on the end of the surgical instruments to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention.

One or more slave robots 2 may be used to operate the patient, and the laparoscope 5 for displaying the surgical site as an image image through the monitor unit 6 is implemented as an independent slave robot 2. May be In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

15 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention, Figures 16 to 27 is a view showing a region designation method according to embodiments of the present invention. As described above, the master robot 1 and the slave robot 2 may be integrally implemented.

Referring to FIG. 15, the master robot 1 includes an image input unit 310, a screen display unit 320, an arm operation setting unit 331, a storage unit 341, a restricted area setting unit 1350, and an arm operation unit 330. ), An operation determination unit 371, a reaction information processing unit 1380, and a control unit 370. Although not shown, the master robot 1 includes an input unit for the operator to input a control command, a speaker unit for outputting the reaction information as auditory information (for example, a warning sound or a warning voice message), and visualizes the response information. It may further include one or more of the LED unit for outputting the information.

The slave robot 2 may comprise a robot arm 3 and a laparoscope 5. In addition, the robot arm 3 may be interpreted as a concept including an instrument unless explicitly limited herein. Although not shown, the slave robot 2 further includes a location information providing unit for providing location information of the robot arm 3 and the laparoscope 5, a biometric information measuring unit for measuring and providing biometric information about the patient. It may include. The position information providing unit masters, for example, information about how much the robot arm 3 or the laparoscope 5 has moved at an angle from the basic position (for example, the rotation angle of a driving motor such as a robot arm). It may be configured to provide to the robot 1.

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network.

The screen display unit 320 outputs an image image corresponding to the image received through the image input unit 310 as visual information. In addition, the screen display unit 320 may output the response information as visual information (for example, screen flickering), or may further output corresponding information when biometric information is input from the slave robot 2. The screen display unit 320 may be implemented in the form of, for example, the monitor unit 6.

The arm manipulation setting unit 331 receives the arm manipulation setting from the operator or the setter and generates manipulation setting information on whether the robot arm 3 and / or the instrument is to be manipulated according to the manipulation instruction of the operator. . The arm manipulation setting may be, for example, a setting of whether or not the robot arm and / or the instrument located in the out-of-visible region not visible in the video image acquired by the laparoscope 5 is to be manipulated.

The manner in which the operator performs cancer manipulation settings can vary. Only a few of these examples are presented below.

As a first embodiment, the operator sets only the robot arm and / or instrument shown on the screen with reference to the image image input through the laparoscope 5 to be operable using the identification information, and other robots. Arms and instruments can be set inoperable.

As a second embodiment, after the master robot 1 receives the position information of each robot arm and / or instrument, the position of the laparoscope 5 and the image input angle from the slave robot 2, After calculating whether or not the image image of the coordinate range is input, it can be set to be operable only the robot arm and the instrument positioned to include some or all within the coordinate range. The laparoscope 5 may further include a distance sensing sensor for sensing a distance to the surface of the surgical site to calculate a coordinate range of the surgical site represented by the image image obtained by the laparoscope 5. Of course, the robot arm outside the coordinate range for the replacement of the robot arm or instrument to be used during the operation can be set to be manipulated.

The storage unit 341 stores the operation setting information and the restricted area coordinate information set by the restricted area setting unit 1350 in the robot arm 3 and / or the instrument generated by the arm operating setting unit 331. In addition, when the response information processing method is designated by the operator, information on the response information processing type (eg, one or more of tactile information output method, auditory information output method, visual information output method, etc.) may be further stored.

The control unit 370, when the operation information for any robot arm (3) or the instrument is input by the operator, when the operation using the operator's arm operation unit 330 falls within the specified restricted area for the response information processing type With reference to the information, the reaction information processor 1380 may control to perform a corresponding process. The control unit 370 may include an operation determination unit 371 described below.

The restricted area setting unit 1350 may access the robot arm 3 or the instrument to prevent damage to a part of the patient's body (for example, blood vessels or organs) due to a misoperation or the like. Create and store coordinate information for restricted areas that cannot be manipulated. Here, the restricted area may be designated or changed by the operator before or during the operation.

There may be a variety of ways for the operator to designate the restricted area and to generate corresponding restricted area coordinate information. In the present specification, a description will be given of a method of setting a restricted area by a method such as designating an arbitrary area by an operator, but only an area designated by an operator is set as an allowable area, and other areas are set as restricted areas. Naturally, the method can be applied.

The following is only a few examples of various embodiments in which an operator designates a restricted area. The area set as follows may be displayed overlaid on the image image obtained by the laparoscope 5.

As a first embodiment, when the operator wants to perform surgery on a limited organ (eg, liver, etc.), when the operator selects the organ using a menu item displayed on the monitor 6, the master robot 1 ) Analyzes the image image acquired by the laparoscope 5 (for example, the presence or absence of the selected organ by the color analysis of the surgical site included in the image image and the interpretation of its coordinate region, etc.). Coordinate information (eg, restricted area coordinate information or allowed area coordinate information) may be generated.

As a second embodiment, the allowable area may be set only for the coordinate range corresponding to the image image obtained and displayed by the laparoscope 5, and all other areas may be set as the restricted area. At this time, only the robot arm 3 or the instrument, which is partially or entirely located in the allowable area, will be operated by the operator's arm operation or the like.

In a third embodiment, 3 corresponding to an area (ie, a restricted area or an allowable area) set using a master controller and a controlled object moving according to its manipulation (eg, at least one of a robot arm, an instrument, and an endoscope). Dimensional coordinate values can be set. Hereinafter, a case in which the corresponding area is set as the restricted area will be described in detail with reference to the accompanying drawings.

For example, as shown in FIG. 16, the end of the instrument is recognized as a three-dimensional mouse cursor, and the operator performs an operation of taking a point in space (that is, specifying a point) while viewing an image image obtained by the laparoscope 5. This allows the dots to be connected to be set as a restricted area.

That is, the operator sets an area to include S1 in the area on the image image obtained and displayed by the laparoscope 5 (in the image image, organs, blood vessels such as S1, S2, and S3 are included). After the end is positioned at the first vertex, after inputting a point designation command, the user moves to the second vertex and additionally inputs a point designation command. After repeating the vertex input operation such that an area to be set is defined, and inputting an area partition command, an area to which vertices previously set by the point designation command are connected is set. Each vertex may be connected in a straight line, but may be connected in a curved form, and the corresponding area may be set as a planar region (for example, a triangle or a circle) or a three-dimensional region (for example, a polyhedron shape). The aforementioned point designation command, area section command, and the like may be input using a master controller provided in the master interface 4 or the like.

As shown in FIG. 17, the restricted area setting method may be extended by designating a vertex on a three-dimensional space to three-dimensionally set the restricted area.

By the above-described method, not only the three-dimensional restriction zone can be set in a manner of designating a vertex or the like on the three-dimensional space, but also a planar restriction zone such as a wall can be set.

In addition, the region set by the operator may be deformed by locating the end of the instrument at one point of the outline of the region and then moving the point out of the region or inward. Here, the outline and the outer surface of the region formed by the designated vertices may be interpreted and stored in advance as being the connection of the points.

That is, in order for the operator to further include S2 as well as S1, the operator places the instrument end at the vertex P4 and performs point movement as shown in FIG. 18. In addition, the end of the instrument is located on the line connecting the vertices P3 and P4 and then the point movement is performed. By such point movements, an area set to include S2 in the corresponding area may be extended.

The above-described deformation of the set area is not limited to be made only on a plane, and as shown in FIG. 19, it is natural that the same can be performed even when the three-dimensional area is set. In addition, although the case in which the region is deformed by a point movement method using vertices is illustrated in FIG. 16, it is natural that the region may be deformed by pulling a surface, a corner, or the like in a three-dimensional space.

In addition, the area set by the operator may be divided into a plurality of areas and separated into individual areas. That is, as shown in FIGS. 20 and 21, the operator may separate a region set to include S1 and S2 into a first individual region including only S1 and a second individual region including only S2. To this end, the operator may designate two or more points (eg, Q1 and Q2) at a point to be separated, and then input a division command to separate the corresponding area into two or more separate areas.

In addition, after the area set by the operator is divided into a plurality of areas, unnecessary areas may be removed. That is, as illustrated in FIGS. 22 and 23, the operator may allow the region set to include S1 and S2 to be reduced to the region including only S2. To do this, the operator assigns two or more points (e.g., Q1 and Q2) to the boundary point to be removed, allows the instrument end to be located in the area to be removed, and then enters a delete command. In this case, the separated area can be deleted.

In addition, the area set by the operator may be manipulated to be folded around the boundary point partitioning into a plurality of areas or to be restored to its original position. That is, as shown in FIGS. 24 and 25, the operator divides a region set to include S1 and S2 into a first individual region including only S1 and a second individual region including only S2, and then each individual region is Folding operations can be performed to act as a restricted or permitted area. For reference, FIG. 24 assumes a case where the restricted area is planarly deformed by folding or restoring a planarly set area flatly, and FIG. 25 assumes a case where the restricted area is folded into a three-dimensional limited area. .

Referring to FIG. 24, when an operator specifies two or more points (for example, Q1 and Q2) to function as boundary points and inputs a folding command, an individual area including a point where an instrument end is located is centered on the boundary point. Rotation is performed. As a result, when the corresponding area is set as the restricted area, the S2 is able to operate the instrument in a state in which it is released from the restricted area. Thereafter, when the areas folded by the restore command are restored to their original state, S2 is also reset to the restricted area.

Similarly, as shown in FIG. 25, the operator specifies a plurality of points (e.g., Q1-Q2 and Q3-Q4) that serve as boundary points and enters a fold command, whereby one direction (e.g., The rotation and folding commands may be executed in the direction corresponding to the position so that the restricted area may be set in the shape of a three-dimensional figure. Of course, it may be set in advance as to what angle the individual regions at which positions are to be rotated around the boundary point. In addition, the three-dimensional shape of the area image shown in plan by the folding command may be various.

The method of setting and modifying the above-described area is one of arbitrary shapes (for example, circles, triangles, squares, spheres, hexahedrons, tetrahedrons, etc.) designated as a template, in addition to the method in which the operator precedes the area setting using an instrument. After the shape of the organ or the organ is shown in space, it is natural that the three-dimensional mouse (or the end of the instrument) can be deformed or separated. For example, as shown in FIG. 26, when the heart region is to be set as a restricted area, a template corresponding to the heart is selected to be shown, and then scaled or drawn by drawing an outline or an outline. The template can be adapted to suit the characteristics of the surgical patient. In this case, the control may be controlled so that the set restricted area is deformed correspondingly by tracking the change of the actual organ (for example, position shift, shape change, etc.) that is deformed by the operator's control object manipulation. To this end, for example, an image analysis technique such as an edge detection may be applied to an image of an actual organ.

As a fourth embodiment, a reference image (eg, an image image obtained by the laparoscope 5, an MRI image of a patient, etc.) of a surgical patient displayed through a monitor implemented as a touch-sensitive input device is generated and generalized. In the human modeling image, when the operator shows a certain range in the shape of a closed curve with a finger, the corresponding region may be set as a restricted area. In this case, the coordinates of the range shown by the operator are coordinated to correspond to the position of organs or blood vessels inside the body of the surgical patient based on an arbitrary reference point (for example, the position of the characteristic organs, the position of the laparoscope 5, etc.). The information is mapped and the coordinate information mapping may be processed by a position recognition method by conventional image analysis, and thus description thereof will be omitted.

As a fifth embodiment, modeling of each organ or blood vessel of a human model image generated using a reference image (eg, CT image, MRI image, etc.) of a surgical patient or a human model image generated of a general human body You can also set the area automatically using the data.

Referring to FIG. 27, a human modeling image is generated using a reference image (eg, CT image, MRI image, etc.) of a surgical patient in step P450, and modeling data corresponding to the human modeling image (for example, in step P455). For example, the color, shape, size, etc. of each organ and blood vessel) are generated and stored. When the human modeling image and the modeling data generated for the general human body are used, steps P450 and P455 may be omitted.

In step P460, long-term selection information is obtained. The organ selection information is obtained by using an image analysis technique (for example, pixel-by-pixel color analysis, organ outline analysis, etc.) to determine whether organs or blood vessels in the image image obtained and displayed by the laparoscope 5 are displayed, for example. After analysis, it can be automatically recognized against pre-stored modeling data. Of course, a method in which an operator selects an arbitrary organ from an organ list including a drop down menu or a human modeling image displayed through the monitor unit 6 may be used.

Referring to modeling data in step P465, the corresponding area is partitioned and displayed as an area image (see FIGS. 16 to 26) for the operator to recognize. The master robot 1 recognizes the position of the organ corresponding to the organ selection information in consideration of the height of the patient, the lying down shape and the general position of the organ in the human body, and then the area of the shape defined by the modeling data. This is set so as to be overlaid and displayed in the image image obtained by the monitor unit 6 and / or the laparoscope 5. For example, when the heart is selected as the organ selection information, modeling data is extracted from the storage 341 or the like to display an area image corresponding to the shape of the heart, and the area image corresponding to the extracted modeling data is the actual organ. The image may be processed to be overlaid and displayed on the video image.

In operation P470, the operator checks whether the area image overlaid on the image image corresponds to the area of the actual organ, and then modifies the area image overlaid (see FIGS. 19 to 26) when the area image does not match. The region image deformation of step P470 may be processed by the operator as described above, but if the organ is present in the image image acquired and displayed by the laparoscope 5, the shape (size) of the organ is determined using an image analysis technique. And so on, the image may be automatically transformed to match the shape of the organ.

In addition, a method of setting and using a restricted area in advance may also be applied. For example, a method of using a three-dimensional image obtained by reconstructing a CT, MRI image, etc., in which a restriction zone is set in advance in a three-dimensional image, and the image image obtained by the laparoscope 5 (for example, Endoscope images). Of course, the matching of the 3D image and the real-time image image may be omitted, and a method of matching only a few points required may be applied.

The arm operation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2. The arm manipulation unit 331 may be formed in the shape of the handle 10 as illustrated in FIG. 14, but the shape is not limited thereto and may be modified in various shapes for achieving the same purpose. Further, for example, some may be formed in the shape of a handle, others may be formed in a different shape, such as a clutch button, finger insertion tube or insertion to enable the operator's fingers can be inserted and fixed to facilitate the operation of the surgical tool More rings may be formed. In addition, if the laparoscope 5 for receiving an image is not fixed at a specific position, and the position and / or the image input angle can be moved or changed by the operator's adjustment, the clutch button 14 or the like is used as the laparoscope 5. It may be set to function for the adjustment of the position and / or the image input angle of.

The operation determination unit 371 determines whether the operation information by the operator's operation of the arm operation unit 330 is valid operation information by referring to at least one of operation setting information and restricted area coordinate information stored in the storage unit 341.

For example, the manipulation determination unit 371 may determine that the manipulation information is invalid when the manipulation information according to the manipulation of the arm manipulation unit 330 is set to be inoperable with reference to the manipulation setting information.

In addition, the operation determination unit 371 may determine that the operation information according to the operation of the arm operation unit 330 is invalid when the robot arm 3 contacts the restricted area with reference to the restricted area coordinate information. . To this end, the operation determination unit 371 may determine displacement information (for example, the robot arm 3) at which angle and at which angle the robot arm 3 or the instrument received from the position information providing unit has moved from the basic position. Rotation angle of a driving motor, etc.).

The response information processing unit 1380 performs the processing of the response information specified by the operator and / or the setter when the operation determination unit 371 determines that the operator's operation information is invalid operation information. The processing of the response information includes visual information such as tactile information output for the operator to recognize force by applying force feedback to the handle 10 and outputting a warning message corresponding to the image image obtained by the blinking LED or the laparoscope 5. May be processed in one or more ways, such as output of auditory information, such as output of warning sounds.

The controller 370 controls the operation of each component so that the above-described function can be performed. The controller 370 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320.

In the present embodiment described below, the generation and storage of the arm operation setting information and the generation and storage of the restricted area setting information are illustrated as sequential steps, but the order of each step may be changed or performed simultaneously.

Referring to FIG. 28, in step P510, the master robot 1 receives an arm manipulation setting on whether to operate the robot arm 3 and / or the instrument located in the out-of-visible region from the operator or setter.

In step P520, the master robot 1 generates and stores arm operation setting information corresponding to the input arm operation setting.

In step P530, the master robot 1 inputs a restricted area setting for designating a zone that is restricted from entering the process of operating the robot arm 3 and / or the instrument located in the visible and non-visible areas from the operator or setter. Receive.

In step P540, the master robot 1 generates and stores restricted zone setting information corresponding to the entered restricted zone setting.

As such, the operator or / and setter may be displayed by pre-acquired image information (e.g., one or more of patient images such as CT, MRI, etc., virtual images modeled to be generalized, etc.) or / and clicked on a drop down menu. Long-term items can be used to establish restricted areas before or during surgery. The area set as the restricted area is restricted so that the robot arm 3 and / or the instrument does not operate in spite of the user's operation, and the access or contact to the restricted area is fed back to the operator as response information, thereby being damaged during the surgery. The occurrence of medical accidents due to malfunctions can be prevented for important areas that should not be allowed. The response information may be output to the operator in one or more of a tactile information type, a visual information type, and an auditory information type.

For example, when a surgeon designates a specific area using an external input device such as a touch screen or a mouse device in an image image obtained and viewed by the laparoscope 5, the designated area is set as a restricted area. can do.

Alternatively, the patient image information such as CT, MRI, or the like taken in advance or a reconstructed image from the patient image information is displayed on the operator's screen, and if the operator designates a specific region in the image displayed on the screen, the designated region becomes a restricted area. It can also be set.

In this case, as described above, a process of recognizing the coordinate range of the designated specific region may be performed by image analysis or the like.

As such, when the restricted area is designated, the operator may designate the restricted area by referring to the screen on which the corresponding image is displayed, may be specified in advance in the photographed image, and in addition, the restricted area may be specified in various ways.

29 and 30 are flowcharts illustrating an operation limiting method of a surgical robot system according to an exemplary embodiment of the present invention, and FIG. 31 is an exemplary view of a screen display for explaining an operation limiting method according to an exemplary embodiment of the present invention.

Referring to FIG. 29, in operation P610, the master robot 1 receives arm manipulation information according to manipulation of the arm manipulation unit 330 from an operator.

The master robot 1 determines whether the arm operation information input in step P520 is for the robot arm 3 and / or the instrument located in the out-of-visible region. For example, the robot arm positioned in the non-visible region may be manipulated by a misoperation of an operator or when the robot arm 3 positioned in the visible region is to be replaced with the robot arm 3 positioned in the non-visible region. (3) The operation may be commanded. As mentioned above, the robot arm 3 may be interpreted as a concept including an instrument, unless expressly limited herein.

As a result of the determination in step P620, if it is not an operation command for the robot arm 3 or the like located in the non-visible region (i.e., the visible region), the flow advances to step P630, whereby the master robot 1 generates an arm manipulation instruction. Output After performing step P630, step P710 shown in FIG. 30 is followed. Referring to FIG. 31, robot arms or instruments positioned in the visible region may be referred to as 830a and 830b, and robot arms or instruments positioned in the non-visible region may be referred to as 850.

However, as a result of the determination in step P620, in the case of an operation instruction for the robot arm 3 or the like located in the out-of-visible region, the flow advances to step P640, where the master robot 1 moves to the robot arm 3 or the like in the out-of-visible region. It is determined whether the operation setting information for allowing the operation to be permitted is stored.

As a result of the determination in step P640, if operation setting information for allowing operation on the robot arm 3 and the like located in the out-of-visible region is stored, the process proceeds to step P630 and the master robot 1 generates and outputs an arm operation command. . In this case, the robot arm 3 or the like located in the visible region or the non-visible region may be moved or manipulated in the same manner by the arm manipulation command. An arm operation instruction may be generated to allow the robot arm 3 or the like positioned in the visible area to be processed slower than the operation speed or the operation speed.

However, as a result of the determination in step P640, if operation setting information is stored so that the operation on the robot arm 3, etc. located in the out-of-visible region is not allowed, the process proceeds to step P650 and the master robot 1 performs output processing of the reaction information. To allow the operator to recognize it.

Referring to FIG. 30, in step P710, the master robot 1 receives arm operation state information from the slave robot 2. The arm operation state information may include, for example, information about how much the robot arm 3 or the instrument has moved at an angle from the basic position (for example, the rotation angle of a driving motor such as the robot arm), The corresponding information may be provided from the location information providing unit.

In step P720, the master robot 1 determines whether the robot arm 3 or the instrument is in contact with the preset restricted area with reference to the arm operation state information.

As described above, the restricted area may be designated by the operator with reference to the screen on which the corresponding image is displayed, or may be specified in advance in the captured image, or may be set by various methods.

In the restricted area set by the above-described process, the augmented reality technique is applied to the real image acquired by the laparoscope 5, etc. (see 840 of FIG. 31), a line treatment method of a specific color, and an opacity process. It can be handled variously in one or more ways. As a result, the operator may continuously inform that the area is a restricted area. Naturally, the restricted zone display method may be processed in conjunction with the movement or enlargement of the display screen.

As a result of the determination in step P720, if the robot arm 3 or the instrument has not contacted the restricted area, the process proceeds to step P610 again.

However, if the robot arm 3 or the instrument is in contact with the restricted area as a result of the determination in step P720, the process proceeds to step P730 where the master robot 1 performs the operation limitation control of the robot arm, and also performs reaction information output processing. This allows the operator to recognize it.

Operation restriction control of the robot arm 3 in the restricted zone can, for example, allow the set restricted zone to be recognized as if it were a virtual wall. In other words, when the end of the instrument or part of the intermediate shaft tries to enter the restricted area during the robotic operation, the operator restricts the operation of the instrument as if the instrument is not operated by the virtual wall and recognizes it through the output of the response information. Do it.

Of course, the contact with the restricted area is not necessarily limited to not operate the instrument, it is also possible to determine whether the operation by the operator's choice. For example, if bleeding occurs in a region (eg, a blood vessel) set as a restricted area irrespective of an operator's intention, rapid bleeding may be possible if the instrument is set to operate. Here, the command for ignoring the restricted area setting for allowing an instrument or the like to be operated in the restricted area when the contacted with the restricted area may be input by, for example, an operator operating a predetermined button or pedal. will be.

Further, the restricted area is not necessarily set only within the area currently displayed on the screen (ie, the visible area 810), and is not displayed on the screen (ie, the non-visible area 820) as illustrated in FIG. 31. Of course, the restricted area 840 may be designated and set. That is, even when the restricted area 840 is set in the out-of-visible region 820, if the instrument is in contact with the restricted area may cause a serious risk to the safety of the surgical patient, the movement of the instrument is limited to prevent this Can be controlled.

As a control method for limiting the operation of the instrument, for example, a method of locking the instrument so as not to operate may be used, and further, by allowing reaction force to be fed back to the operation handle of the user who operates the locked instrument, The user may be informed that the operation is limited. In addition, the manner in which the reaction information is output may vary as described above.

The operation limiting method of the above-described surgical robot system may be implemented by a software program or the like. Codes and code segments constituting a program can be easily inferred by a computer programmer in the art. The program is also stored in a computer readable media, and read and executed by a computer to implement the method. The information storage medium includes a magnetic recording medium, an optical recording medium and a carrier wave medium.

32 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention, Figure 33 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention, Figures 34 to 37 is the present invention Figure is a view illustrating a movement form of the contact portion according to the embodiment.

32 and 33, the surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 remotely controlling the slave robot 2. The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor 6, a telescopic display 20 and a master manipulator, and the slave robot 2 comprises a robot arm 3 and a laparoscope 5.

The monitor unit 6 of the master interface 4 may be composed of one or more monitors, and each monitor may be individually displayed information necessary for the operation. 32 and 33 illustrate a case in which the monitor unit 6 is included on each side of the telescopic display unit 20 one by one, but the quantity of the monitor may vary depending on the type or type of information requiring display. have.

The monitor unit 6 may output one or more biometric information about the patient, for example. In this case, at least one of indicators indicating the patient's condition, for example, biometric information such as body temperature, pulse rate, respiration and blood pressure may be output through at least one monitor of the monitor unit 6, and a plurality of pieces of information may be output. In this case, each piece of information may be divided and output by area. In order to provide such biometric information to the master robot 1, the slave robot 2 includes a biometric information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like. It may include. The biometric information measured by each module may be transmitted from the slave robot 2 to the master robot 1 in the form of an analog signal or a digital signal, and the master robot 1 monitors the received biometric information. Can be displayed via

The telescopic display unit 20 of the master interface 4 provides the operator with an image input through the laparoscope 5 as a surgical site. The operator views the image through the eyepiece 220 formed on the contact portion 210 of the telescopic display portion 20, and manipulates the robot arm 3 and the end effector by manipulating the master controller to operate on the surgical site. Proceed. 33 illustrates a case in which a panel portion 210 is implemented as an example of the folding portion 210, the folding portion 210 may be formed to be recessed to face the inside of the master interface 4. 33 illustrates an example in which the eyepiece 220 for the operator to view an image acquired by the laparoscope 5 is formed on the contacting part 210, but the contacting part 210 shows the image of the backside being transmitted. When formed of a losing material, the formation of the eyepiece 220 may be omitted. For example, the folding unit 210 may be formed of a transparent material, coated with a polarizing film, or used to watch a 3D IMAX film so that an image of the rear surface of the folding unit 210 may be transmitted to the operator. It may be formed by a light transmissive material.

The telescopic display unit 20 functions not only as a display device for the operator to check the image of the laparoscope 5 through the eyepiece 220, but also as a control command input unit for controlling the position and image input angle of the laparoscope 5. It is configured to have together.

The operator's face is in contact with or close to the contact portion 210 of the telescopic display unit 20, and a plurality of

supports

230 and 240 are formed to protrude so that the operator's face movement can be recognized. For example, the support 230 formed at the top may be used to contact the operator's forehead to fix the forehead position, and the support 240 formed at the side may be located at an area under the operator's eye (for example, the cheekbone area). It can be used to contact and fix the face position. The position and quantity of the support illustrated in FIG. 33 are exemplary, and the position or the shape of the support may be varied, for example, a jaw fixing support, a face left and right support 290, or the like. . In the case of the left and right face support, for example, when the left or right side of the face moves, the contact portion 210 may be formed in the form of a rod or a wall to support the movement in the corresponding direction.

The position of the operator's face is fixed by the

support parts

230 and 240 formed as described above, and when the operator turns the face in an arbitrary direction while viewing the image by the laparoscope 5 through the eyepiece 220, the facial movement accordingly This can be detected and used as input information for adjusting the position of the laparoscope 5 and / or the image input angle. For example, if the operator wants to check the area on the left side (i.e., the area on the left side of the display screen) rather than the surgical area displayed on the current image, the operator can turn his head so that his face is relatively left. The laparoscope 5 may be manipulated so that the image of the corresponding area is output.

In other words, the contact portion 210 of the telescopic display unit 20 is coupled to the master interface 4 so that the position and / or the angle is changed in accordance with the operator's face movement. To this end, the master interface 4 and the contact portion 210 of the telescopic display portion 20 may be coupled to each other by the flow portion 240. The flow unit 250 may be formed of, for example, an elastic body so as to easily change the position and / or angle of the telescopic display unit 20 and restore the original state when the external force caused by the operator's face movement is removed. In addition, even when the flow unit 250 is formed of an inelastic material, the telescopic display unit 20 may control the original state restoration unit (see FIG. 40) so that the telescopic display unit 20 may be restored to the original state.

The contact part 210 is moved by the flow part 250 based on a virtual center point and coordinates in a three-dimensional space formed on the XYZ axis, or is operated in any direction (eg, clockwise, counterclockwise, etc.). Rotational movement). Here, the virtual center point may be any one point or axis in the contact portion 210, for example, the center point of the contact portion 210.

34 to 37 illustrate the movement of the contact portion 210.

When the operator's face movement direction is parallel to the X, Y or Z axis, the contact portion 210 is moved in the direction in which the force due to the face movement is applied as illustrated in FIG.

When the operator's face movement direction rotates on the X-Y plane, the contact portion 210 is rotated in a direction in which a force caused by the face movement is applied as illustrated in FIG. 35. At this time, the contact portion 210 may be rotated in a clockwise or counterclockwise direction depending on the direction in which the force is applied.

When the operator's face movement is rotated about the X, Y or Z axis, the contact portion 210 is rotated in the direction in which the force by the face movement is applied about the reference axis as illustrated in FIG. 36. . In this case, the contact portion 210 may be rotated in a clockwise or counterclockwise direction according to the direction in which the force is applied.

When the force according to the operator's face movement is applied to two of the X, Y and Z axes, the contact portion 210 is based on the virtual center point and the two axes to which the force is applied as illustrated in FIG. Rotation is moved.

As described above, the vertical and horizontal movements of the contact portion 210 and the rotational movement are determined by the direction of the force applied by the face movement, and one or more types of movements described above may be combined.

A method and a configuration in which the telescopic display unit 20 detects an operator's face movement and generates an operation command according to the present invention will be described in detail with reference to the accompanying drawings.

As illustrated in FIGS. 32 and 33, the master interface 4 is provided with a master manipulator so that the operator can be gripped and manipulated by both hands. The master controller may be implemented by two handles 10 or more handles 10, and an operation signal according to the operator's manipulation of the handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be moved. Controlled. By operating the handle 10 of the operator, a surgical operation such as a position movement, rotation, and cutting operation of the robot arm 3 may be performed.

The master manipulator is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

The master robot 1 and the slave robot 2 may be coupled to each other through a wired communication network or a wireless communication network so that an operation signal and a laparoscope image input through the laparoscope 5 may be transmitted to the counterpart. If a plurality of operation signals by the plurality of handles 10 provided in the master interface 4 and / or an operation signal for adjusting the laparoscope 5 need to be transmitted at the same time and / or at a similar time point, each The operation signal may be transmitted to the slave robot 2 independently of each other. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical instrument inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical instrument in the yaw direction according to the surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical instruments, a transfer drive for moving the surgical instruments in the longitudinal direction, a rotation drive for rotating the surgical instruments, a surgical instrument drive unit installed on the end of the surgical instruments to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. In addition, the actual control process, such as the operator rotates the robot arm 3 in the corresponding direction by operating the handle 10 is somewhat distanced from the subject matter of the present invention, so a detailed description thereof will be omitted.

One or more slave robots 2 may be used to operate the patient, and the laparoscope 5 for allowing the surgical site to be displayed as an image (that is, an image image) that can be seen through the eyepiece 220 is an independent slave. It may be implemented by the robot (2). In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

FIG. 38 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention, and FIG. 39 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

Referring to FIG. 38, the telescopic display unit 20 includes a motion detector 311, an operation command generator 321, and a transmitter 332. In addition, the telescopic display unit 20 may further include a component that allows the operator to visually recognize the image of the surgical site input through the laparoscope 5 through the eyepiece 220, but this is the gist of the present invention. Since there is a little distance from the description thereof will be omitted.

The motion detector 311 outputs sensing information by detecting in which direction the operator moves the face while the operator contacts the face 230 and / or 240 of the contact portion 210. The motion detector 311 may include sensing means for detecting a direction and a size (eg, a distance) of the face moving. The sensing means is sufficient as sensing means capable of detecting how much the contact portion 210 has moved in which direction, for example, in which direction the flow portion 250 having an elastic force supporting the contact portion 210 is directed. It may be a sensor that detects to what extent the tension (引張), or a sensor provided to the inside of the master robot 1 to detect how close or / and rotated feature points formed on the inner surface of the contact portion 210, and the like. .

The manipulation command generation unit 321 analyzes the operator's face movement direction and size using the sensing information received from the motion detection unit 311, and controls the position and image input angle of the laparoscope 5 according to the analyzed result. Create an operation command to

The transmission unit 332 transmits the operation command generated by the operation command generation unit 321 to the slave robot 2 so that the position and image input angle of the laparoscope 5 are manipulated, and an image is provided accordingly. . The transmission unit 332 may be a transmission unit already provided in the master robot 1 to transmit an operation command for the operation of the robot arm 3.

39 shows a method of transmitting a laparoscope manipulation command according to an embodiment of the present invention.

Referring to FIG. 39, the telescopic display unit 20 detects the operator's face movement in step 410, and proceeds to step 420 to manipulate the laparoscopic 5 using the sensing information generated by the detection of the face movement. Create a command.

Subsequently, in step 430, the manipulation command generated by step 420 is transmitted to the slave robot 2 for manipulation of the laparoscope 5.

Here, the operation command generated for the operation of the laparoscope 5 may be functioned so that a specific operation is performed also on the master robot 1. For example, when the face is rotated to detect the rotation of the laparoscope 5, a manipulation command for the rotation is transmitted to the slave robot 2 and the direction of the manipulation handle of the master robot 1 is correspondingly changed. By doing so, it is possible to maintain the intuition and ease of operation of the operator. For example, when the rotation signal is detected by the contact unit 210, the laparoscope 5 is rotated by the generated operation signal, and the image displayed on the screen and the position of the surgical tool shown in the image are currently Since it may not coincide with the position of the hand of the manipulation handle, an operation of matching the position of the surgical tool displayed on the screen by moving the position of the manipulation handle may be performed. The control of the operation handle direction is a case where the position / direction of the surgical tool displayed on the screen and the actual operation handle position / direction are not only in the case of the rotary motion of the contact portion 210 but also in the case of the linear motion. The same may apply.

40 is a block diagram schematically illustrating the configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention, and FIG. 41 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention.

Referring to FIG. 40, the telescopic display unit 20 may include a motion detector 311, an operation command generator 321, a transmitter 332, a contact detector 510, and an original state restorer 520. have.

Since the motion detector 311, the operation command generator 321, and the transmitter 332 are described with reference to FIG. 38, description thereof will be omitted. However, the motion detector 311 may perform an operation while it is recognized that the operator's face is in contact with the support 230 or / and 240 as the sensing information by the touch detector 510.

The touch detector 510 detects whether the operator's face is in contact with the support 230 or / and 240 and outputs sensing information. To this end, for example, a touch sensor may be provided at the end of the support, and in addition, various sensing schemes may be applied to detect whether a face is in contact.

The original state restoring unit 520 controls the motor driver 530 when the face of the operator is sensed to be in contact with the supporting unit 230 or / and 240 as the sensing information by the contact detecting unit 510. 210 is returned to its original state. The original state restorer 520 may include a motor driver 530 to be described below.

In FIG. 40, the motor driving unit 530 using the motor is illustrated as an operation means for returning the contact portion 210 to its original state, but it is obvious that the operation means for achieving the same purpose is not limited thereto. For example, the contact portion 210 may be treated to return to its original state by various methods such as pneumatic or hydraulic pressure.

The original state restoring unit 520 controls the motor driving unit 530 using, for example, information on the reference state (ie, position and / or angle) of the contacting unit 210, or the operation command generation unit 321. The motor driver 530 may be controlled to be manipulated in the reverse direction and size using the face movement direction and size analyzed by the face movement, so that the contact portion 210 may be returned to its original position.

For example, the operator turns his face in the corresponding direction (by which the contact portion 210 is also moved or rotated) in order to check a region different from the surgical region displayed in the current image or to take an action on the region. ) Detects that the face contact with the contacting unit 210 has ended in a state where it is operated accordingly, the original state restoring unit 520 may return the motor unit to return to the reference state designated by the contacting unit 210 as the default. 530 may be controlled.

The motor driving unit 530 may include a motor that rotates under the control of the original state restoring unit 520, and the motor (eg, position and / or angle) of the contacting unit 210 is adjusted by the rotation of the motor. The driving unit 530 and the contacting unit 210 are coupled to each other. The motor driver 530 may be formed to be received inside the master interface 4. The motor included in the motor driving unit 530 may be, for example, a spherical motor for allowing a degree of freedom movement, and the support structure of the spherical motor may be a spherical bearing in order to remove the limitation of the inclination angle. It may be composed of a circular rotor or a frame structure having three degrees of freedom for supporting the circular rotor.

Even if the contact unit 210 is restored to its original state by the operation of the above-described components, the operation command generation unit 321 does not generate and transmit the operation command for the image, and thus is input and output by the laparoscope 5. The image does not change. Therefore, after the operator checks the laparoscopic (5) image through the eyepiece 220 may be consistent in the operation.

Although the case where the original state return of the contact portion 210 of the telescopic display unit 20 is performed by the operation of the motor driving unit 530 has been described with reference to FIG. 40, the operation is performed by the flow unit 250 of the elastic material having elastic force. Naturally, if the external force caused by the movement of the face of the child is removed, it may be returned to its original state. Even when the contact portion 210 is returned to its original state by the elastic force, an operation signal for manipulating the laparoscope 5 will not be generated.

41 shows a method of transmitting a laparoscope manipulation command according to an embodiment of the present invention.

Referring to FIG. 41, the telescopic display unit 20 detects the operator's face movement in step 410, and proceeds to step 420 to manipulate the laparoscopic 5 using the sensing information generated by the detection of the face movement. Create a command. Thereafter, in step 430, the manipulation command generated by step 420 is transmitted to the slave robot 2 for the manipulation of the laparoscope 5.

Subsequently, the telescopic display unit 20 determines whether the operator releases contact with the contacting unit 210 in step Q610. If the contact is maintained, the process proceeds to step 410 again. If the contact is released, the process proceeds to step Q620 to control the contact unit 210 to return to its original position.

42 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention.

Referring to FIG. 42, the telescopic display unit 20 includes a contact detector 510, a camera unit 710, a storage unit 720, an eye tracker unit 730, an operation command generation unit 321, and a transmission unit 332. And the controller 740.

The touch detector 510 detects whether the operator's face is in contact with the support 230 or / and 240 formed to protrude from the contact portion 210 and outputs sensing information.

When the camera unit 710 detects that the operator's face is in contact with the contacting unit 210 by sensing information of the contact sensor 510, the camera unit 710 photographs an image of the operator's eyes in real time. The camera unit 710 is arranged to photograph the operator's eye seen through the eyepiece 220 inside the master interface 4. The image of the operator's eye photographed by the camera unit 710 is stored in the storage unit 720 for the eye tracking process of the eye tracker unit 730.

The image photographed by the camera unit 710 is sufficient that the eye tracking process of the eye tracker unit 730 is possible. May be stored. Since the image generating method and the generated image type for the eye tracking process will be apparent to those skilled in the art, a description thereof will be omitted.

The eye tracker unit 730 analyzes the images stored in the storage unit 720 in real time or at predetermined intervals in chronological order, and analyzes the change of the pupil position of the operator and the gaze direction by the operator and outputs the analysis information. In addition, the eye tracker unit 730 may further analyze the shape of the pupil (for example, blinking eyes, etc.) and output analysis information thereof.

The operation command generation unit 321 refers to the analysis information by the eye tracker unit 730 when the operator's gaze direction is changed according to the operation command for controlling the position and / or image input angle of the laparoscope 5 accordingly. Create In addition, the operation command generation unit 321 may generate an operation command for this if the change in the shape of the pupil is for inputting a predetermined command. For example, the designation command according to the change in the shape of the pupil is, for example, the laparoscopic (5) approach to the surgical site in the case of two consecutive blinks of the right eye, and the clockwise rotation in the case of two consecutive blinks of the left eye. It can be specified in advance.

The controller 740 controls each of the above components to perform a specified operation.

Up to now, referring to FIG. 42, the telescopic processor 20 for recognizing and processing eye movements using eye tracking technology has been described. However, the present invention is not limited thereto, and the telescopic processor 20 may be implemented in a manner of detecting, recognizing, and processing a movement of the operator's face itself. For example, the camera unit 710 captures a face image, and the analysis processing unit replacing the eye tracker unit 730 captures a feature point (for example, a position of two eyes, a position of a nose, a position of a person, etc.). If the position and change of one or more of the) is analyzed, the operation command generation unit 321 may generate a corresponding operation command.

Referring to FIG. 43, when the contact of the operator's face is detected by the contact sensor 510 in step Q810, the telescopic display unit 20 activates the camera unit 710 to the eye of the operator visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In step Q820, the telescopic display unit 20 compares and determines digital image data stored in the storage unit 720 in real time or at predetermined intervals, and generates analysis information about changes in the eye position and the gaze direction of the operator. In the comparison determination, the telescopic display unit 20 may allow a certain error so that a change in the position information of a certain range may be recognized as not changing the position of the pupil.

In step Q830, the telescopic display unit 20 determines whether the gaze direction of the operator changed over the preset threshold time is maintained.

If the changed gaze direction is maintained for more than a threshold time, in step Q840, the telescopic display unit 20 manipulates (e.g., moves or / or changes the image input angle) the laparoscope 5 to receive an image of the corresponding position. An operation command is generated and transmitted to the slave robot 2. Here, the threshold time may be set to a time for preventing the laparoscopic 5 from being manipulated by the movement of the pupil for movement of the operator's pupils or general overview of the surgical site, and the time value is set experimentally and statistically. Or set by an operator or the like.

However, if the changed gaze direction is not maintained for more than the threshold time, the flow returns to step Q810 again.

44 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention, and FIG. 45 is a view illustrating an image display form by a telescopic display unit according to an embodiment of the present invention.

Referring to FIG. 44, when the contact of the operator's face is detected by the contact sensor 510 in step Q810, the telescopic display unit 20 activates the camera unit 710 to the eye of the operator visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In step Q820, the telescopic display unit 20 compares and determines digital image data stored in the storage unit 720 in real time or at predetermined intervals, and generates analysis information about changes in the eye position and the gaze direction of the operator.

In step Q910, the telescopic display unit 20 determines whether the gaze position of the operator is a predetermined setting.

45 illustrates an image display form by the telescopic display unit 20.

As illustrated in FIG. 45, through the eyepiece 220, an operator may check an image image 1010 provided through the laparoscope 5, and the image image may include a surgical part and an instrument 1020. . In addition, the image by the telescopic display unit 20 may be displayed by overlapping the gaze position 1030 of the operator, and the setting positions may be displayed together.

The setting position may include one or more of an outer edge 1040, a first rotation instruction position 1050, a second rotation instruction position 1060, and the like. For example, when the operator watches the side positioned in any direction of the outer edge 1040 or more for a threshold time, the laparoscope 5 may be controlled to move in the corresponding direction. That is, when the left side of the outer edge 1040 is watched for more than a threshold time, the laparoscope 5 may be controlled to be moved to the left to photograph the left side of the current display position. In addition, when the operator watches the first rotational instruction position 1050 for a threshold time or more, the laparoscope is controlled to rotate in a counterclockwise direction, and when the operator watches the second rotational instruction position 1060 for more than the threshold time, the laparoscope is in a clockwise direction. It may be controlled to rotate.

Referring back to FIG. 44, if the operator has a gaze position other than the above-described setting position, the process proceeds to step Q810 again.

However, referring back to FIG. 44, when the operator has the gaze position as described above, the flow advances to step Q920 to determine whether the gaze of the operator is maintained for more than a predetermined threshold time.

If the operator's attention to the set position is maintained for more than the threshold time, the telescopic display unit 20 generates an operation command to operate the laparoscope 5 according to the command specified for the set position in step Q930, so that the slave robot 2 To send.

However, if the operator's attention to the set position is not maintained for more than the threshold time, the process proceeds to step Q810 again.

Referring to FIG. 46, when the contact of the operator's face is detected by the contact sensor 510 in step Q810, the telescopic display unit 20 activates the camera unit 710 to the eye of the operator visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In operation 1110, the telescopic display unit 20 compares and determines image information stored in the storage unit 720 in real time or at a predetermined period to generate analysis information about a change in the shape of the driver's eyes. For example, the interpretation information may be information about how many times the operator's eyes blinked during a certain time, and which eyes blinked when the operator blinked.

In operation 1120, the telescopic display unit 20 determines whether analysis information regarding the change in eye shape satisfies a predetermined predetermined condition. For example, the designated condition according to the change of the eye shape may be set in advance, for example, whether the right eye blinks twice in a predetermined time or whether the left eye blinks twice in a predetermined time.

If the analysis information on the change in eye shape satisfies a predetermined condition, the flow advances to step 1130 and generates an operation command for manipulating the laparoscope 5 as a designated command in the case of satisfying the condition, thereby generating the slave robot 2. To send. For example, the designation command according to the change of the eye shape may be, for example, the laparoscopic (5) approach to the surgical site in the case of two consecutive blinks of the right eye, or the clockwise rotation in the case of two consecutive blinks of the left eye. It may be specified in advance.

However, if the analysis information for the eye shape change does not satisfy the predetermined condition, the flow advances to step Q910.

The above-described laparoscopic manipulation method may be implemented by a software program or the like. Codes and code segments constituting a program can be easily inferred by a computer programmer in the art. The program is also stored in a computer readable media, and read and executed by a computer to implement the method. The information storage medium includes a magnetic recording medium, an optical recording medium and a carrier wave medium.

47 is a plan view showing the overall structure of the surgical robot according to an embodiment of the present invention, Figure 48 is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.

According to the present embodiment, the output position of the endoscope image 9 output to the monitor viewed by the user corresponds to the viewpoint of the endoscope changing according to the movement of the surgical endoscope, so that the user can feel the actual surgical situation more realistically. There is a characteristic to make. That is, since the viewpoint of the endoscope may coincide with the viewpoint of the user performing the surgery, the present embodiment may match the view of the endoscope in the abdominal cavity with the position and output direction of the monitor outputting the endoscope image 9 at the external surgery site. In addition, the motion of the system located at the external surgery site reflects the motion of the endoscope moving inside the actual patient.

Surgical endoscopes according to the present embodiment may be a variety of tools used as an imaging tool during surgery, such as laparoscopic as well as thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinal, cardiac. In addition, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope. That is, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope for generating stereoscopic image information, and the stereoscopic image information generating method may be implemented by various techniques. For example, the surgical endoscope according to the present embodiment includes a plurality of cameras to acquire a plurality of images having stereoscopic information, or acquire a plurality of images using a single camera. An image can be obtained. In addition to this method, the surgical endoscope according to the present embodiment may of course generate a stereoscopic image by various other methods.

In addition, the bodily-type surgical image processing apparatus according to the present embodiment is not necessarily implemented to be limited to the surgical robot system as shown, and if the system outputs an endoscope image 9 during surgery and operates using a surgical tool. Applicable. Hereinafter, the case where the surgical image processing apparatus according to the present embodiment is applied to the surgical robot system will be described.

Referring to FIGS. 47 and 48, the surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 remotely controlling the slave robot 2. . The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor part 6 and a master controller, and the slave robot 2 comprises a robot arm 3 and an instrument 8. The instrument 8 is a surgical tool such as an endoscope, such as a laparoscope, or a surgical instrument for directly manipulating the affected part.

The master interface 4 is provided with a master controller so that the operator can be gripped and manipulated by both hands. The master controller may be implemented with two handles 10 as illustrated in FIGS. 47 and 48. The operation signal according to the manipulation of the operator's handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be operated. This is controlled. By the operation of the operator's handle 10, the position movement, rotation, cutting, etc. of the robot arm 3 and / or the instrument 8 may be performed.

The monitor 6 of the master interface 4 displays an endoscope image 9, a camera image, and a modeling image input by the instrument 8 as an image image. In addition, the information displayed on the monitor unit 6 may vary according to the type of the selected image.

The monitor unit 6 may be composed of one or more monitors, and may display information necessary for surgery on each monitor separately. 47 and 48 illustrate the case in which the monitor unit 6 includes three monitors, the quantity of monitors may be variously determined according to the type or type of information requiring display. In addition, when the monitor unit 6 includes a plurality of monitors, the screen may be extended in cooperation with each other. That is, the endoscope image 9 may freely move each monitor, such as a window output to one monitor, and the entire image may be output by outputting some images connected to each monitor.

The slave robot 2 and the master robot 1 are coupled to each other via wired or wireless, so that the master robot 1 transmits an operation signal to the slave robot 2, and the slave robot 2 is the master robot 1. The endoscope image 9 input through the instrument 8 may be transmitted to the. If two operation signals provided by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the instrument 8 need to be transmitted at the same time and / or at a similar time point, Each operation signal may be independently transmitted to the slave robot 2. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

49 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 49, a master robot 1 including an image input unit 310, a screen display unit 320, an arm operation unit 330, an operation signal generation unit 340, a screen display control unit 3350, and a control unit 370. And a slave robot 2 comprising a robot arm 3, an endoscope 5.

The haptic surgical image processing apparatus according to the present embodiment may be implemented as a module including an image input unit 310, a screen display unit 320, and a screen display control unit 3350. Of course, such a module may be an arm manipulation unit 330. ), The operation signal generator 340 and the controller 370 may be included.

The image input unit 310 receives an image input through the endoscope 5 of the slave robot 2 through wired or wireless transmission. Endoscope 5 may also be one type of surgical tool according to the present embodiment, the number may be one or more.

The screen display unit 320 outputs an image image corresponding to the image received through the image input unit 310 as visual information. The screen display 320 may output the endoscope image as it is or zoom in / zoom out, or may match the endoscope image and a modeling image to be described later, or output each as a separate image.

In addition, the screen display unit 320 outputs the endoscope image and the image reflecting the entire surgical situation, for example, a camera image generated by the camera photographing the outside of the surgical target simultaneously and / or matched with each other and outputted to identify the surgical situation. It may be easy.

In addition, the screen display unit 320 outputs a reduced image of the entire image (endoscopic image, modeling image, camera image, etc.) in a portion of the output image or a window generated on a separate screen, and the operator described above When selecting or rotating a specific point among the reduced images output using the master controller, the entire output image may be moved or rotated, so-called bird's eye view function of the CAD program. Functions such as zooming in / out, moving, and rotating the image output to the screen display unit 320 as described above may be controlled by the controller 370 according to the operation of the master controller.

The screen display unit 320 may be implemented in the form of a monitor unit 6. An image processing process for outputting a received image as an image image through the screen display unit 320 may include a control unit 370 and a screen display control unit. 3350 or a separate image processor (not shown). The screen display unit 320 according to the present embodiment may be a displayer implemented by various technologies, and may be, for example, an ultra high resolution monitor such as a UHDTV (7380x4320). In addition, the screen display 320 according to the present embodiment may be a 3D display. For example, the screen display 320 according to the present exemplary embodiment may allow the user to separately recognize the left eye and right eye images by using the principle of binocular disparity. Such a 3D image implementation method may be implemented in various ways such as glasses (for example, red blue glasses (anaglyph), polarized glasses (passive glass), shutter glasses (active glass), etc.), lenticular method, barrier method. have.

The screen display unit 320 outputs the input endoscope image to a specific region. Here, the specific area may be an area on the screen having a predetermined size and location. This particular area may be determined in correspondence with the change of viewpoint of the endoscope 5 as described above.

The screen display controller 3350 may set this specific area in accordance with the viewpoint of the endoscope 5. That is, the screen display control unit 3350 tracks the point of view corresponding to the motion, such as rotation and movement of the endoscope 5, and sets the specific area for outputting the endoscope image on the screen display unit 320 by reflecting the viewpoint.

The arm manipulation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2. The arm manipulation unit 330 may be formed in the shape of the handle 10 as illustrated in FIG. 48, but the shape is not limited thereto and may be modified in various shapes for achieving the same purpose. Further, for example, some may be formed in the shape of a handle, others may be formed in other shapes such as a clutch button, and a finger cannula or insertion may be inserted to fix the operator's finger to facilitate manipulation of the surgical tool. More rings may be formed.

The operation signal generator 340 generates a corresponding operation signal when the operator manipulates the arm operation unit 330 for the movement of the robot arm 3 and / or the endoscope 5 or the operation for surgery. Transfer to the robot (2). The operation signal may be transmitted and received via a wired or wireless communication network.

The operation signal generator 340 generates an operation signal by using the operation information according to the operation of the operator's arm operation unit 340, and transmits the generated operation signal to the slave robot 2. To be manipulated accordingly. In addition, the position and operation shape of the actual surgical instrument operated by the operation signal can be confirmed by the operator by the image input by the endoscope (5).

50 is a block diagram illustrating an apparatus for immersive surgical image processing according to an embodiment of the present invention. Referring to FIG. 50, the screen display controller 3350 may include an endoscope perspective tracker 1351, an image movement information extractor 1353, and an image position setter 1355.

The endoscope perspective tracking unit 1351 tracks the perspective information of the endoscope 5 corresponding to the movement and rotation of the endoscope 5. Here, the view point information refers to a view point viewed by the endoscope 5, and the view point information may be extracted from a signal for manipulating the endoscope 5 in the above-described surgical robot system. That is, the viewpoint information can be specified by signals for manipulating the movement and rotational movement of the endoscope 5. Since the endoscope 5 manipulation signal is generated in the surgical robot system and transmitted to the robot arm 3 for manipulating the endoscope 5, the signal used to track the endoscope 5 can be tracked.

The image movement information extractor 1353 extracts movement information of the endoscope image using the viewpoint information of the endoscope 5. That is, the viewpoint information of the endoscope 5 may include information on the position change amount of the target object of the acquired endoscope image, and the movement information of the endoscope image may be extracted from this information.

The image position setting unit 1355 sets a specific area of the screen display unit 320 on which the endoscope image is output using the extracted movement information. For example, if the viewpoint information of the endoscope 5 has been changed by a predetermined vector A, the endoscope image of the patient's internal organs has its movement information corresponding to the corresponding vector, and the screen using the movement information A specific area of the display unit 320 is set. If the endoscope image is changed by a predetermined vector B, a specific area in which the endoscope image is actually output to the screen display unit 320 may be set using this information and the size, shape, and resolution of the screen display unit 320.

51 is a flowchart illustrating a tangible surgical image processing method according to an embodiment of the present invention. Each step to be performed below may be executed by the screen display control unit 3350 as a subject, and the steps need not be executed in time series in the order described.

In step R511, the viewpoint information of the endoscope 5, which is information about the viewpoint viewed by the endoscope 5, is tracked corresponding to the movement and rotation of the endoscope 5. The viewpoint information is specified by signals for manipulating the movement and rotational movement of the endoscope 5 so that the direction that the endoscope 5 faces can be tracked.

In step R513, the movement information of the endoscope image corresponding to the change amount of the position of the object to be captured of the endoscope image is extracted using the viewpoint information of the endoscope 5.

In step R515, a specific area of the screen display unit 320 for outputting the endoscope image is set using the extracted movement information. That is, when the viewpoint information of the endoscope 5 and the movement information of the endoscope image are specified as described above, a specific area for outputting the endoscope image on the screen display unit 320 is set using the movement information.

In step R517, the endoscope image is output to the specific area set by the screen display unit 320.

52 is a block diagram of an output image according to the tangible surgical image processing method according to the embodiment of the present invention. The screen display unit 320 may be a full screen, and the endoscope image 620 acquired from the endoscope 5 may include the endoscope image 620 at a specific position of the screen display unit 320, for example, coordinates (X, Y). ) Can be output at the centered position. Coordinates (X, Y) can be set corresponding to the amount of change in the viewpoint of the endoscope (5). For example, when the viewpoint information of the endoscope 5 and the movement amount of the endoscope image change by +1 to the left and -1 to the vertical, the center point of the endoscope image 620 is moved to the position of coordinates (X + 1, Y-1). I can move it.

53 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 53, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the screen display control unit 3350, the image storage unit 360, and the control unit 370. A master robot 1 and a robot arm 3 comprising a slave robot 2 including an endoscope 5 are shown. The differences from the above will be explained mainly.

The present embodiment has a feature of extracting a previously input and stored endoscope image at the present time point and outputting the endoscope image together with the current endoscope image to inform the user of the change of the endoscope image. .

The image input unit 310 receives a first endoscope image and a second endoscope image provided at different time points from the surgical endoscope. Here, the ordinal numbers, such as the first and the second, may be identifiers for distinguishing different endoscope images, and the first endoscope image and the second endoscope image may be images captured by the endoscope 5 from different viewpoints and viewpoints. Can be. In addition, the image input unit 310 may receive the first endoscope image before the second endoscope image.

The image storage unit 360 stores the first endoscope image and the second endoscope image. The image storage unit 360 stores not only image information which is actual image content of the first endoscope image and the second endoscope image, but also information on a specific region to be output to the screen display unit 320.

The screen display unit 320 outputs the first endoscope image and the second endoscope image to different areas, and the screen display control unit 3350 corresponds to the first endoscope image and the second endoscope image according to different viewpoints of the endoscope 5. The screen display 320 may be controlled to output the data to different areas.

Here, the screen display unit 320 may differently output one or more of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. For example, the screen display unit 320 may output a second endoscope image currently input as a color image, and output a first endoscope image, which is a past image, as a black and white image, so that the user may distinguish the images from each other. Referring to FIG. 56, a second endoscope image 623 which is a currently input image is output as a color image at coordinates X1 and Y1, and a first endoscope image 621 which is a previously input image is a screen pattern, that is, An example in which hatched patterns are formed and output at coordinates X2 and Y2 is shown.

In addition, the first endoscope image, which is a previous image, may be output only for a continuous or preset time. In the latter case, the past image is output to the screen display unit 320 only for a predetermined time, so that the new endoscope image may be continuously updated on the screen display unit 320.

54 is a block diagram illustrating an apparatus for immersive surgical image processing according to an embodiment of the present invention. Referring to FIG. 54, the screen display controller 3350 may include an endoscope perspective tracker 1351, an image movement information extractor 1353, an image position setting unit 1355, and a stored image display unit 1357.

The endoscope perspective tracking unit 1351 tracks the perspective information of the endoscope 5 in correspondence with the movement and rotation of the endoscope 5, and the image movement information extracting unit 1353 uses the perspective information of the endoscope 5 in detail. As described above, the movement information of the endoscope image is extracted.

The image position setting unit 1355 sets a specific area of the screen display unit 320 on which the endoscope image is output using the extracted movement information.

The stored image display unit 1357 extracts the first endoscope image stored in the image storage unit 360 and outputs the image to the screen display unit 320 while the screen display unit 320 outputs the second endoscope image input in real time. Since the output region and the image information of the first endoscope image and the second endoscope image are different from each other, the storage image display unit 1357 may extract the information from the image storage unit 360 to store the first endoscope image, which is a past image. Output to the screen display unit 320.

55 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention. Each step to be described below may be executed by the screen display control unit 3350 as a main subject, and may be classified into a step of outputting a first endoscope image and a step of outputting a second endoscope image. The first endoscope image may be output together with the second endoscope image.

In step R511, the viewpoint information of the endoscope 5, which is information about the viewpoint viewed by the endoscope 5, is tracked corresponding to the first movement and rotation information of the endoscope 5.

In step R513, the movement information of the first endoscope image is extracted. In step R515, a specific area of the screen display unit 320 on which the endoscope image is output is set using the extracted movement information. The first endoscope image is output.

In operation R519, the information about the outputted first endoscope image and the first screen position are stored in the image storage unit 360.

In step R521, the viewpoint information of the endoscope 5, which is information about the viewpoint viewed by the endoscope 5, is tracked corresponding to the second movement and rotation information of the endoscope 5.

In step R522, movement information of the second endoscope image is extracted. In step R523, a specific area of the screen display unit 320 for outputting the endoscope image is set using the extracted movement information, and in step R524, The second endoscope image is output.

In operation R525, the information about the outputted second endoscope image and the second screen position are stored in the image storage unit 360. In operation R526, the first endoscope image stored in the image storage unit 360 together with the second endoscope image is output to the first screen position. Here, the first endoscope image may be output by differently performing any one or more of saturation, brightness, color, and screen pattern from the second endoscope image.

57 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 57, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generator 340, the screen display control unit 3350, the control unit 370, and the image matching unit 450. A master robot 1 and a robot arm 3 comprising a slave robot 2 including an endoscope 5 are shown. The differences from the above will be explained mainly.

In this embodiment, the endoscope image actually photographed using an endoscope during surgery and a modeling image generated in advance for a surgical tool and stored in the image storage unit 360 are matched with each other, or the size thereof is corrected. There is a feature that can be output to the screen display unit 320 that the user can observe.

The image matching unit 450 generates an output image by matching the endoscope image received through the image input unit 310 with the modeling image of the surgical tool stored in the image storage unit 360 and generating the output image. Output to. The endoscope image is an image of the inside of the patient's body using the endoscope. Since the image is obtained by capturing only a limited area, the endoscope image includes an image of a part of the surgical instrument.

The modeling image is an image generated by realizing the shape of the entire surgical tool as a 2D or 3D image. The modeling image may be an image of a surgical tool photographed at a specific time point before the start of surgery, for example, an initial setting state. Since the modeling image is an image generated by a computer simulation technique of the surgical tool, the image matching unit 450 may output the registered surgical tool and the modeling image shown in the actual endoscope image. Since a technique of obtaining an image by modeling a real object has a little distance from the gist of the present invention, a detailed description thereof will be omitted. In addition, specific functions, various detailed configurations, and the like of the image matching unit 450 will be described in detail with reference to the accompanying drawings.

The controller 370 controls the operation of each component so that the above-described function can be performed. The controller 370 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320. In addition, the controller 360 controls the image matching unit 450 to output the modeling image through the screen display unit 320 in response to the manipulation information according to the manipulation of the arm manipulation unit 330.

The actual surgical tool included in the endoscope image is a surgical tool included in the image inputted by the endoscope 5 and transmitted to the master robot 1 and is a surgical tool that applies a surgical operation directly to the patient's body. In contrast, the modeling surgical tool included in the modeling image is mathematically modeled with respect to the entire surgical tool in advance and stored in the image storage unit 360 as a 2D or 3D image. Surgical tools and modeling images of the endoscope image Surgical tools are controlled by the operation information (that is, information about the movement, rotation, etc. of the surgical tool) recognized by the master robot 1 as the operator operates the cancer operation unit 330 Can be. In actual surgical tools and modeling surgical tools, their position and manipulation shape may be determined by the manipulation information. Referring to FIG. 60, the endoscope image 620 is matched with the modeling image 610 and output to the coordinates (X, Y) of the screen display unit 320.

58 is a block diagram illustrating an apparatus for immersive surgical image processing according to an embodiment of the present invention. Referring to FIG. 58, the image matcher 450 may include a feature value calculator 451, a modeled image implementer 453, and an overlapped image processor 455.

The characteristic value calculator 451 uses the characteristic value by using the image inputted by the laparoscope 5 of the slave robot 2 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The actual position of the surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2, the information on the position may be provided from the slave robot 2 to the master robot (1). .

The characteristic value calculator 451 may use, for example, a field of view (FOV), an enlargement ratio, a viewpoint (for example, a viewing direction), a viewing depth, etc. of the laparoscope 5 using an image of the laparoscope 5. In addition, it is possible to calculate characteristic values such as type, direction, depth, and degree of bending of the actual surgical instrument. When the characteristic value is calculated using the image of the laparoscope 5, an image recognition technique for recognizing the outline of the subject included in the image, shape recognition, tilt angle, or the like may be used. In addition, the type of the actual surgical tool may be input in advance in the process of coupling the corresponding surgical tool to the robot arm (3).

The modeling image implementer 453 implements a modeling image corresponding to the feature value calculated by the feature value calculator 451. Data related to the modeled image may be extracted from the image storage unit 360. That is, the modeling image implementer 453 may determine the characteristic values of the laparoscope 5 (field of view (FOV), magnification, perspective, viewing depth, etc., type, direction, depth, degree of bending, etc. of the actual surgical instrument). The modeling image is implemented to extract the modeling image data of the corresponding surgical tool and the like to match the surgical tool of the endoscope image.

The modeling image implementer 453 may extract various images according to the feature values calculated by the feature value calculator 451. For example, the modeling image implementer 453 may extract a modeling image corresponding to the characteristic value of the laparoscope 5 directly. That is, the modeling image implementer 453 may extract the 2D or 3D modeling surgical tool image corresponding to the above-described data such as the angle of view and the magnification of the laparoscope 5 and match it with the endoscope image. Here, the characteristic values such as the angle of view and the magnification may be calculated by comparing and analyzing the images of the laparoscope 5 which are calculated or sequentially generated through comparison with the reference image according to the initial set value.

In addition, according to another exemplary embodiment, the modeling image implementation unit 453 may extract the modeling image by using the manipulation information for determining the position and the manipulation shape of the laparoscope 5 and the robot arm 3. That is, as described above, since the surgical tool of the endoscope image may be controlled by the operation information recognized by the master robot 1 as the operator manipulates the cancer operation unit 330, the modeling surgery corresponding to the characteristic value of the endoscope image is performed. The position and manipulation shape of the tool can be determined by the manipulation information.

Such manipulation information may be stored in a separate database according to the temporal order, and the modeling image implementer 453 may recognize the characteristic values of the actual surgical tool by referring to the database, and correspondingly, the information about the modeling image. Can be extracted. That is, the position of the surgical tool output on the modeling image may be set using cumulative data of the position change signal of the surgical tool. For example, if the operation information for the surgical instrument, which is one of the surgical instruments, includes the information that it is rotated 90 degrees clockwise and 1 cm in the extending direction, the modeling image implementer 453 may correspond to the operation information. An image of a surgical instrument included in the modeling image may be converted and extracted.

Here, the surgical instrument is mounted to the front end of the surgical robot arm is provided with an actuator, the driving wheel (not shown) provided in the drive unit (not shown) by receiving the driving force from the actuator is operated, connected to the drive wheel and surgery The operator inserted into the patient's body performs the operation by predetermined operation. The driving wheel is formed in a disc shape, and may be clutched to the actuator to receive the driving force. In addition, the number of driving wheels may be determined corresponding to the number of objects to be controlled, and the description of such driving wheels will be apparent to those skilled in the art related to surgical instruments, and thus detailed description thereof will be omitted.

The superimposed image processor 455 outputs a partial image of the modeled image so that the actually captured endoscope image and the modeled image do not overlap. That is, when the endoscope image includes some shape of the surgical tool and the modeling image implementer 453 outputs the corresponding modeling surgical tool, the superimposed image processor 455 may perform the actual surgical tool image and the modeling surgical tool image of the endoscope image. By checking the overlapping region of, and deleting the overlapping portion from the modeling surgical tool image, the two images can be matched with each other. The overlapping image processor 455 may process the overlapping region by removing the overlapping region of the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

For example, the total length of an actual surgical instrument is 20 cm, and characteristics values (field of view (FOV), magnification, perspective, depth of view, etc., type, direction, depth, degree of bending, etc. of the actual surgical instrument) are considered. When the length of the actual surgical tool image of the endoscope image is 3cm, the superimposed image processor 455 outputs a modeling surgical tool image which is not output to the endoscope image by using the characteristic value.

59 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention. The differences from the above will be explained mainly.

In step R131, a modeling image is generated and stored in advance with respect to the surgical target and / or the surgical tool. The modeled image may be modeled by computer simulation, and the embodiment may generate a modeled image by using a separate modeling image generating apparatus.

In operation R132, the characteristic value calculator 1351 calculates a characteristic value of the endoscope image. As described above, the characteristic value calculating unit 1351 receives coordinates of an image provided by the laparoscope 5 of the slave robot 2 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3. The characteristic value is calculated using the field of view (FOV, field of view), magnification, perspective (for example, viewing direction), viewing depth, and the type, direction, and depth of the surgical instrument. , Bend, and so on.

In operation R133, the image matching unit 450 extracts the modeling image corresponding to the endoscope image, processes the overlapping region, and matches the two images to be output to the screen display unit 320. Here, the output time point may be variously set such that the endoscope image and the modeling image are initially output at the same time point or the endoscope image is output and the modeling image is also output together.

61 is a conceptual diagram illustrating a master interface of a surgical robot according to an embodiment of the present invention. Referring to FIG. 61, the master interface 4 may include a monitor unit 6, a handle 10, a monitor driving unit 12, and a moving groove 13. The differences from the above will be explained mainly.

The present embodiment can rotate and move the monitor unit 6 of the master interface 4 in accordance with the viewpoint of the endoscope 5, which is variously changed as described above, thereby making the user feel more realistic about the surgery. There is a characteristic.

One end of the monitor driving means 12 is coupled to the monitor portion 6, and the other end thereof is coupled to the main body portion of the master interface 4 to rotate the monitor portion 6 by applying a driving force to the monitor portion 6. Move it. Here, the rotation may include rotation about various axes (X, Y, Z), that is, rotation by a pitch, roll, yaw axis. Referring to FIG. 61, rotation A by the yaw axis is shown.

In addition, the monitor driving means 12 moves (B direction) along the moving groove 13 formed in the main body of the master interface 4 located at the lower end of the monitor 6 to endoscope 5. ) Can be moved according to the point of view. The moving groove 13 may have a concave direction toward the user so that the front surface of the monitor 6 always faces the user when the monitor 6 moves along the moving groove 13.

62 is a block diagram of a surgical robot according to an embodiment of the present invention. Referring to FIG. 62, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the control unit 370, the screen driving control unit 2380, and the screen driving unit 390 may be used. A master robot 1 and robot arm 3 comprising, a slave robot 2 comprising an endoscope 5 is shown. The differences from the above will be explained mainly.

The screen driver 390 may be a means for rotating and moving the screen display 320, and may include, for example, a motor and a monitor 6 supporting means. The screen driving controller 2380 may control the screen driving unit 390 so that the screen driving unit 390 rotates and moves the screen display unit 320 according to the viewpoint of the endoscope 5.

Referring to FIG. 63, the screen driving controller 2380 may include an endoscope perspective tracker 381 and a perspective view of the endoscope 5, which track perspective information of the endoscope 5 in correspondence with the movement and rotation of the endoscope 5. An image movement information extraction unit 383 for extracting movement information of the endoscope image using the information, and a driving information generation unit 385 for generating motion information (screen driving information) of the screen display unit 320 using the movement information. It may include. The screen driver 390 may drive the screen display 320 as described above using the motion information of the screen display 320 generated by the drive information generator 385.

According to another embodiment, the screen driver 390 may be driven by a user's command. For example, the screen driving control unit 2380 may be replaced with a user interface, for example, a switch (eg, a step switch) that is operable by a user. In this case, the screen driving unit 390 may be rotated and moved by a user's operation. This may be controlled.

The motion of the screen driver 390 can also be controlled by the touch screen. For example, the screen display unit 320 is implemented as a touch screen, and when the user touches the screen display unit 320 using a finger or the like and drags in a predetermined direction, the screen display unit 320 rotates accordingly. You can also move. In addition, the motion of the screen display 320 may be controlled by using the rotation / movement signal generated according to the user's eyes or the rotation / movement signal generated according to the moving direction of the face contact unit or the voice command.

64 is a flowchart of a tangible surgical image processing method according to an embodiment of the present invention. Each step to be performed below may be performed by the screen driving controller 2380 as a main agent.

In step R181, the viewpoint information of the endoscope 5, which is information about the viewpoint viewed by the endoscope 5, is tracked corresponding to the movement and rotation of the endoscope 5.

In step R182, the movement information of the endoscope image corresponding to the amount of change in the position of the image capturing object of the endoscope image is extracted using the viewpoint information of the endoscope 5.

In step R183, the above-described screen driving information is generated using the viewpoint information of the endoscope 5 and / or the extracted movement information. That is, when the viewpoint information of the endoscope 5 and the movement information of the endoscope image are specified as described above, information for moving and rotating the screen display unit 320 is generated using the movement information.

In step R184, the screen display unit 320 is moved and rotated according to the screen driving information.

65 is a conceptual diagram illustrating a master interface of a surgical robot according to an embodiment of the present invention. Referring to FIG. 65, a dome screen 191, a projector 192, a work bench 193, a first endoscope image 621, and a second endoscope image 623 are illustrated.

The present embodiment implements a function of outputting an endoscope image to a specific region of the screen display unit 320 using the dome screen 191 and the projector 192 as described above, so that the user can quickly operate the surgery on a wide screen. There is a characteristic which can be confirmed conveniently.

The projector 192 projects the endoscope image onto the dome screen 191. The endoscope image may be a spherical image having a spherical shape in front of the projected image. Here, the spherical shape does not mean mathematically strictly a shape having a spherical shape, and may include various shapes such as an ellipse, a curved shape of a cross section, and some spherical shapes.

The dome screen 191 has an open front end and a hemispherical inner dome surface that reflects the image projected from the projector 192. The size of the dome screen 191 is easy to see the user, for example, the diameter may be about 1m ~ 2m. The inner dome surface of the dome screen 191 may be faceted or have a hemispherical shape for each block. In addition, the dome screen 191 may be axially symmetrically formed about a central axis thereof, and the line of sight of the user may be located at the central axis of the dome screen 191.

The projector 192 may be located between the user performing the surgery and the dome screen 191 to prevent the image projected by the user from being blocked. In addition, the projector 192 may be attached to the bottom surface of the work bench 530 in order to secure the space of the work table without covering the projected image during the user's work. The inner dome surface may be formed of or coated with a material with high reflectivity.

When the dome screen 191 and the projector 192 are used, the first endoscope image 621 and the second endoscope image (i.e., the first endoscope image 621 and the second endoscope image) in a specific area of the dome screen 191 may correspond to various viewpoints of the endoscope 5 as described above. 623) can be projected.

In addition, specific standards for immersive surgical image processing apparatus according to an embodiment of the present invention, common platform technology such as embedded system, O / S, communication protocol, interface standardization technology such as I / O interface, actuator, battery, camera Details of parts standardization technology such as sensors and the like will be omitted since it is obvious to those skilled in the art.

The immersive surgical image processing method according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. In other words, the recording medium may be a computer-readable recording medium having recorded thereon a program for causing the computer to execute the above-described steps.

The medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention.

Those skilled in the art will appreciate that various modifications and changes can be made in the present invention without departing from the spirit and scope of the invention as set forth in the claims below.

Claims

An image input unit for receiving an endoscope image provided from a surgical endoscope;

An image storage unit for storing a modeling image of a surgical tool for operating a surgical target photographed by the surgical endoscope;

An image matching unit which matches the endoscope image and the modeling image to generate an output image;

Surgical image processing apparatus comprising a screen display for outputting an output image including the endoscope image and the modeling image.
The method of claim 1,

The surgical endoscope is a surgical image processing device, characterized in that at least one of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac.
The method of claim 1,

And the image matching unit generates the output image by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.
The method of claim 3,

The image matching unit,

A characteristic value calculator for calculating characteristic values using at least one of the endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm;

And a modeling image implementer configured to implement a modeling image corresponding to the characteristic value calculated by the characteristic value calculator.
The method of claim 3,

And an overlapping image processor configured to remove an overlapping area between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.
The method of claim 3,

And a position of the modeling surgical tool image output to the modeling image is set using operation information of the surgical tool.
The method of claim 1,

And the screen display unit outputs the endoscope image to an arbitrary region of the output image, and outputs the modeling image to a peripheral region of the output image.
The method of claim 1,

The modeling image is a surgical image processing apparatus, characterized in that the image of the surgical tool taken at a specific time before the start of the operation.
The method of claim 1,

Surgical image processing apparatus further comprises a camera for generating a camera image by photographing the outside of the surgical target during the operation.
The method of claim 9,

And the image matching unit generates the output image by matching the endoscope image, the modeling image, the camera image, and a combination thereof.