STEREOSCOPIC PROJECTION-BASED DISPLAY FOR SURGICAL ROBOT
BACKGROUND OF THE INVENTION
1. TECHNICAL FIELD
Generally, the present invention relates to surgical robots. More specifically, the present invention relates to a video display for -use in conjunction with surgical robots.
2. BACKGROUND ART
Minimally invasive medical techniques are aimed at reducing the amount of extraneous tissue that may be damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and deleterious side effects. Many surgeries are performed each year in the United States. A significant amount of these surgeries can potentially be performed in a minimally invasive manner. However, only a relatively small percentage of surgeries currently use minimally invasive techniques due to limitations of minimally invasive surgical instruments and techniques currently used, the difficulty experienced in performing surgeries, and the difficulty in training surgeons using such traditional instruments and techniques. Advances in minimally invasive surgical technology could dramatically increase the number of surgeries performed in a minimally invasive manner. The average length of a hospital stay for a standard
surgery is significantly longer than the average length for the equivalent surgery performed in a minimally invasive surgical manner. Thus, expansion in the use of minimally invasive techniques could save millions of hospital days, and consequently millions of dollars annually, in hospital residency costs alone. Patient recovery times, patient discomfort, surgical" side effects, and time away from work can also be reduced by expanding the use of minimally invasive surgery.
There are many disadvantages relating to such traditional minimally' invasive surgical (MIS) techniques. For example, existing MIS instruments deny the surgeon the flexibility of tool placement found in open surgery. Difficulty is experienced in approaching the surgical site with straight instruments through the incisions on the abdominal wall. The length and construction of many endoscopic instruments reduces the surgeon's ability to feel forces exerted by tissues and organs on the end effector of the associated instrument. Furthermore, coordination of the movement of the end effector of the instrument as viewed in the image on the television monitor with actual end effector movement is particularly difficult, since the movement as perceived in the image normally does not correspond intuitively with the actual end effector movement. Accordingly, lack of intuitive response to surgical instrument movement input is often experienced. Such a lack of intuitiveness, dexterity and sensitivity of endoscopic tools has been found to be an impediment to the expansion of the use and training of minimally invasive surgery. Additionally, the 2D image viewed by the assisting surgeon limits the effectiveness of training.
Minimally invasive telesurgical systems for use in surgery have been and are still being developed to increase a surgeon's dexterity as well as to permit a surgeon to operate on a patient in an intuitive manner. Telesurgery is a general term for surgical systems where the surgeon uses some form of remote control, e.g., a servomechanism, or the like, to manipulate surgical instrument movements, rather than directly holding and
moving the tools by hand. In such a telesurgery system, the surgeon is typically provided with an image of the surgical site on a visual display at a location remote from the patient. The surgeon can typically perform the surgical procedure at the location remote from the patient whilst viewing the end effector movement during the surgical procedure on the visual display. While viewing typically a three-dimensional image of the surgical site on the visual display, the surgeon performs the surgical procedures on the patient by manipulating master control devices at the remote location, which master control .devices control motion of the remotely controlled instruments.
Typically, such a telesurgery system can be provided with at least two master control devices (one for each of the surgeon's hands), which are normally operatively associated with two robotic arms on each of which a surgical instrument is' mounted. Operative communication between master control devices and associated robotic arm and instrument assemblies is typically achieved through a control system. The control system typically includes at least one processor which relays input commands from the master control devices to the associated robotic arm and instrument assemblies and from the arm and instrument assemblies to the associated master control devices in the case of, e.g., force feedback, or the like.
The currently available systems for telesurgery typically use special endoscopes to provide a three dimensional (3D) or stereoscopic visualization to the operating surgeon through an arrangement of television monitor and mirror system. These systems rely on traditional 2D imagery for the assistants and other personnel observing the surgery. This limitation made assisting, training and mentoring difficult due to the different visual perspective between the surgeon (in 3D) and the others (in 2D).
Intuitive surgical of Sunnyvale, California provides a system under trademark of daVinci surgical system that allows a surgeon to perform minimally invasive procedure like coronary bypass grafting and robotic radical prostatectomy. The procedures are performed with instruments that are inserted through small incisions in the patient's chest or abdomen. The instruments are controlled by robotic arms. Movement of the robotic arms and actuation of instrument end effectors are controlled by the surgeon through a pair of masters and foot pedal that are coupled to an electronic controller. The master arms also control movement of an endoscopic camera. The surgeon views the internal view on a display system made of two video monitors and mirror system. The masters and the display system are typically integrated into a console that is operated by the surgeon to control the various robotic arms and medical instruments. Utilizing a robotic system to perform surgery requires a certain amount of training. It would be desirable to provide a system that would either allow a second surgeon to assist another surgeon in controlling a robotic medical system or to train other surgeons. The second surgeon could both teach and assist a surgeon learning to perform a medical procedure with a daVinci system. The primary problem with achieving this goal is that the existing equipment does not provide another surgeon with a three dimensional view. Instead the second surgeon is able to two- dimensionally view the surgery, but is not able to three-dimensionally view the surgery.
Other training devices and surgical devices disclosed in the prior art are described below. The primary problem with the disclosed devices is the lack of 3D viewing capabilities for the surgeon assisting or being trained.
For example, U.S. Patent No. 5,217,003 to Wilk discloses a surgical system that allows a surgeon to remotely operate robotically controlled
medical instruments through a telecommunication link. The system allows one surgeon to operate the robotic arms at a given time. It does not disclose or contemplate a system that allows two different surgeons to operate or three-dimensionally view the surgery.
U.S. Patent No. 5,149,270 to McKeown discloses an apparatus for practicing surgical endoscopic procedures. Simulators incorporate features to simulate visual and manipulation surgical conditions for training surgeons in surgical procedures such as laparoscopy and hysteroscopy. The apparatus has a cavity in which an object simulating a human organ is mounted for performing the practice procedure. The cavity is closeable to outside view or access, thus forcing the individual to use and manipulate the instruments under conditions that mimic real life operating and diagnostic conditions. There is no disclosure of a system enabling multiple individuals to three-dimensionally view an operation from the surgeon's perspective.
U.S. Patent No. 5,273,038 to Beavin discloses a computer system receiving two dimensional slice data of a heart or other organ to be simulated in three dimensions. The three dimensional image data and chemical composition data of the heart or other organ are stored in the computer memory. Then a Voxel View or three-dimensional volume- rendering program forms images of the organ to be studied. Diagnostic data obtained from a patient with electrical measurement signals including an electro-cardiogram, electro-myogram, electro-encephalogram, and other diagnostic measured electrical signals obtained from a patient are fed into the system and are placed in computer memory. Physiological data of the patient, including the strength, weakness, and other parameters of the organ is also considered diagnostic data and is supplied into the system. The data may be fed in black and white or in color to a device that shows the organ for visualization, operation simulation, or training. The Beavin patent does not disclose a system that enables a second surgeon
to view a surgical procedure three-dimensionally.
U.S. -Patent No, 5,385,474 to Brindle discloses a method for simulating anesthesiology and operating room conditions including the following six steps: displaying initial patient simulated vital sign information from a memory to signify an initial patient condition; randomly modifying the displayed patient vital, sign information according to a script matrix in a manner analogous to that in which a patient's vital signs would be effectedin the operating room by drugs or other external effects, thereby indicating a deteriorating condition; displaying user options; evaluating the timeliness and appropriateness of user input selections from the options in response to the changes in* patient vital sign information to improve its initial state or deteriorate to a critical state in accordance with the successive script blocks, in the script matrix depending upon the user's response and timeliness. While all of these conditions are beneficial for training purposes, the system does not enable the trainee to simultaneously, three- dimensionally view a surgical procedure that the trainee is not performing.
U.S. Patent No. 5,261 ,404 to Mick et al. discloses a three- dimensional mammal anatomy imaging system that provides images of the internal anatomy of a mammal. However, the system does not enable a second surgeon, or trainee, can experience the same three-dimensional view as that of the surgeon performing the surgery. U.S. Patent No. 4,331 ,422 10, 1994 to Loftin et al. discloses a training system for use in a wide variety of training tasks and environments. Artificial intelligence is used to provide computer-aided training.
U.S. Patent No. 5,130,794 to Ritchey discloses a panoramic image based virtual reality display system. However, there is still no disclosure for a system that enables a second surgeon, or trainee, can experience the same three-dimensional view as that of the surgeon performing the
surgery.
It would therefore be useful to develop a surgical and/or training system that enables a second surgeon, or trainee, can experience the same three-dimensional view as that of the surgeon performing the surgery.
SUMMARY OF THE INVENTION According to the present invention, there is provided a robotic system for broadcasting or telecasting a three-dimensional image from a surgical robot to a screen including at least one video input operatively connected to a surgical robot, at least one projector for receiving and projecting the 3D image from the video input, and a screen for displaying the 3D image from the projector. A method of training surgeons to use a surgical robot by performing a surgical procedure using a surgical robot, transmitting the 3D images seen by the surgeon during surgery to a screen, and viewing the 3D images, thereby training surgeons regarding surgery using the 3D imagery is provided. A method of transmitting a three dimensional image from a surgical robot to a screen by obtaining a 3D image from a surgical robot, transmitting the 3D image to projector, and displaying the 3D image on a screen is also provided.
BRIEF DESCRIPTION OF THE DRAWINGS
Other advantages of the present invention are readily appreciated as the same becomes better understood by reference to the following detailed description, when considered in connection with the accompanying drawings wherein:
Figure 1 is a diagram of a rear projection system of the present invention;
Figure 2 is a diagram of the projection system, including the screen, of the present invention ;ι and Figure 3 is a photograph of an operating room containing the system of the present invention.
DESCRIPTION OF THE INVENTION Generally, the present invention provides a device that creates a three-dimensional (3D) image that can be viewed by individuals in the operating room or telecast to a geologically .different location in 3D. Additionally, the system of the present invention can be used for training individuals. More specifically, the present invention has two components a local viewing mode and a recording/telecasting mode. The device of the present invention can be used in connection with any surgical robot or other telesurgical device.
The local viewing mode is a local live display. The display is a device that creates a 3D image viewable by individuals that are not able to view the surgeon's console of the telesurgical system but are present in the geographically same location of the surgical robot.
The recording/telecasting mode of the present invention enables individuals at geologically different location to view, in 3D, a recorded display or live display. Thus, individuals at a remote location can view a surgery as though they were present at the site of the procedure.
The term "minimally invasive surgery" is intended to include endoscopy. One of the more common forms of endoscopy is laparoscopy, which is minimally invasive inspection or surgery within the abdominal cavity. Other endoscopic techniques include, but are not limited to,
arthroscopy, retroperitoneoscopy, pelviscopy, nephroscopy, cystoscopy, cistemoscopy, sinoscopy, hysteroscopy, urethroscopy, and other similar procedures. The system of the present invention can be used for local-live 3D display or for telecasting. The system includes video inputs, projectors, and a screen. The video inputs are connected to a surgical robot. The robot includes video streams that are displayed for the surgeon. The device of the present invention in connected to the video streams and channels the video stream to a projector. The projector then displays the image that is seen by the surgeon to a screen, thus enabling individuals in the room to view the exact same image as that seen by the surgeon.
The vided inputs that are utilized can be any video input devices known to those of skill in the art. For example, the inputs can include, but are not limited to, S-video cables, BNC-RCA composite cables, KVM switches, component video inputs, coaxial cables, twisted pair cables, and fiber optic cables. The projectors of the present invention are preferably projectors that are compatible with the video stream data. An example of such a projector includes, but is not limited to, a DLP or LCD projector. Examples, of such projectors include, but are not limited to, BenQ PB2220, HP mp3130, InFocus LP 70+, Mitsubishi XD50U Mini-Mits, NEC LT10, NEC LT170, Philips bCool XG1 , Olympus VP1 , Toshiba TLP-S10U and TLP-S41 U, Sharp Notevision PG-B10S, NEC VT465, Dell 3200MP, Epson PowerLite 73c, HP vp6120, Mitsubishi XD300U, NEC MT1065, NEC VT46, NEC VT460, Optoma EzPro 737, Panasonic PT-LC76U & PT-LC80U, PLUS V3- 131 , Sharp XR-1X, Toshiba TDP-P5-US, and Toshiba TLP-T501. Preferably, the system of the present invention utilizes two projectors. The projectors are preferably mounted above one another on a frame with each projector including a polarizing filter. A dual projection system is preferably
used, as this enables the 3D image to be projected. Alternatively, the projectors can project the 3D image without requiring the polarization filters or use an alternative method for creating the 3D image. The screen of the present invention can be any screen capable to displaying the images. The screen is preferably capable of maintaining the integrity of the polarization of the image. Individuals of skill in the art are familiar with such screens. Examples of such screens are a Silver matte screen from Dalite or Film screen 150 from Stewart.
The robot, or surgical device, for use in conjunction with the device of the present invention can be any surgical system with capability of binocular vision can be connected to the device of the present invention. Examples of such robots include, but are not limited to, the da Vinci™ surgical robot, EndoVia robot, surgical robot hand, computer assisted surgery, and other known surgical or surgery-related robots known to be useful in performing minimally invasive surgery.
The system of the present invention improves telesurgery display systems by providing similar 3D visual perspectives to the operating surgeon on the computer console and others in the operating room. In this manner the surgical procedure can typically be performed, learned and taught with greater confidence, safety, efficacy and in some cases greater accuracy. The system of the present invention also provides means to telecast the telesurgical procedure live in 3D, to a geologically different location or to record, edit, and display the procedure later in 3D. The system improves the training of surgeons as they can watch the educational videos of the surgery performed by the telesurgical system in 3D.
The individuals that view the projected images preferably wear special eyewear in order to visualize the projection to get a 3D perception.
The glasses can be either wired or wireless. Examples of 3D eyeware include, but are not limited to, Polarized 3D glasses, Pulfrich 3D glasses, and anaglyph 3D glasses. The glasses can be clip-on lenses or can be self-contained glasses. Preferably, the glasses are wireless as this makes use significantly easier. The glasses can be made by any known maker of 3D eyeware. Some well-known companies include, but are not limited to, Razor3D, Rainbow Symphony, Inc., eDimensional, and l-glasses. Alternatively, eyeglasses are not required when the projectors do not utilize the polarizing filters.
The system of the present invention can be used for recording or telecasting of the 3D images from the surgical robot. The two video streams from surgical systems camera units can be synchronized using commercially available frame synchronizer. A frame synchronizer synchronizes outputs from multiple video sources so that they are in similar phase of their frame cycle.
The FA-145 is a frame synchronizer and time base corrector in a single system. The FA-145 includes 10-bit 4:2:2 digital component processing and a Y/C decoder. The FA-145 also supports SD-SDI video or analog composite video signals in and two-way SD-SDI video output signals. The analog composite output connections can also be used for signal monitoring. A full frame of memory is provided to enable reliable and frame accurate synchronization, and a variety of genlock functions are provided. Individuals of skill in the art are familiar with other such synchronizers that can be used in the system of the present invention.
The synchronized signals thus obtained are multiplexed with a video multiplexer. The multiplexers combine two genlocked video streams and output a sigle multiplexed 3D video stream. Such multiplexers are commercially available like Mux1/Mux2/Mux3 from VrRex or Model 100/200/ 400/500 from 3DTV.
The multiplexed signal can be recorded and edited on videotape or a digital nonlinear editing system. Non-linear editing (NLE) is editing using random-access video storage, so that there is no wait for the tape to shuttle to see a scene at the other end of the reel. Examples of such editing software include, but are not limited to, VideoRaid RT3X, VideoRaid RT3, Premiere Pro 1.5, XSreve Raid, XPress DV v3.5, Media Studio Pro 7, and DV Storm made by companies such as Avid (Media Composers of various flavors, models, qualities, and capabilities), Accom (formerly Scitex, formerly Immix) with its "sphere" products (descended from the VideoCube and TurboCube), Quantel (Harry, Henry, Harriet, EditBox, iQ, etc.), and Medial 00 (844/X, iFinish, etc.).
The signal can also be transmitted by microwave, cables, satellite, infrared beams, telephone (ISDN and PSDN), fibreoptic channels or any other standard means used by the tele-communication companies for transmitting video signals. The transmitted or recorded multiplexed signal is converted into two separate signals one each for right and left eye by commercially available decoders like SpaceBox 1/2 from 3DTV or XPO stereo or VR Video converter from VRex and displayed through the projection and screen system described above.
The system of the present invention can also be incorporated as part of an operating room. For example, such an operating room can include a 3D display wall including the system of the present invention. The wall enables a larger image to be displayed. Diagrams of such operating rooms are shown in Figures 1 and 2. Figure 3 is a photograph of an actual operating room. The present invention provides a projection based display system for the surgical robots. The surgical robots have two optical channels one for the right eye and another for the left eye. The video signals from the
robot are fed into a two DLP or LCD projectors. The output from the projectors polarized1 and is projected on special silver or rear projection screens. The projected image is viewed through special polarized glasses to give a 3D perception. The two video signals can be field multiplexed and converted into a single signal. The multiplexed signal can be recorded, edited or transmitted on standard video equipments. The multiplexed signal can be de-multiplexed into two separate signals for each eye to be displayed by the projection system described above. The robot has a 3-D display for the surgeon and 2D CRT display for assistants and other personnel in the operating room. The manufacturer of the robot does not provide a solution for 3D display for the assistants and other trainees in the operating room. This brings a disparity in the visual perspectives of the operating fields of the surgeon, his assistants and the trainees. Currently available surgical robots do not have provision for recording, editing and transmitting their 3D displays. These limitations have made assisting, training and mentoring difficult.
Surgical robots have a telescope with two separate optical channels one for right eyes and another for the left eye. They provide two streams of video signals. The two video streams are then synchronized and displayed into the surgeon console in 3D by two cathode ray monitors and an arrangement of minors. Any surgical robot providing two separate video outputs for each eye can be used. The daVinci™ surgical system is preferred. The video streams were tapped from the synchronizers in the vision cart. S-Video cables and BNC- RCA composite cables were used for connection between the vision cart and the display system.
Any projector capable of projecting an electronic video signal can be used. Two DLP projectors were mounted one above another on a frame.
The video signals from the da Vinci™ vision cart are fed in the projectors through S-Video or RCA composite inputs. However other forms of standard connectors can also be used. Each of the two projectors is designated to project image of either right or left eye. In front of the lenses of the two projectors, two polarizing filters are mounted. CRT and DLP projectors can use linear or circular polarizers. The LCD projectors can use only circular polarizers. Linear polarizers are arranged in a 45° and 135° orientation. Circular polarizers are arranged in a clockwise and counterclockwise orientation.
The polarized images are projected on a special screen. This type of screen surface maintains the integrity of the polarization of the image. The preferred screens are silver matte (Dalite), Datex 1.8(Dalite) and Flimscreen 150(Stewart). A rear projection system with reflectors assembly to decease the projection throw distance can also be used.
Special eyewear with 45° and 135° linear polarizing filters is used to visualize the projection to get a 3D perception. If the projectors have circular polarizers then eyeware used has circular polarizers arranged in a clockwise and counter-clockwise orientation. The special eyeware permits the isolation of image seen by each eye.
The two video signals for each eye are multiplexed into a single stream. Any commercially available field multiplexer can be used. Preferably, the Mux-3 from Vrex Inc is used. The multiplexed signal is then recorded on videotape or as DV format on a hard drive. The recording thus obtained can be edited on commercially available non-linear editing software using native DV codecs. A codec is a compresser/decompresser, a bit of software or hardware that takes raw video and compresses it, and can take the compressed video and decompress it back to raw video.
Codecs exist for all kinds of compressed video, including DV, motion- JPEG, MPEG, Indeo, Cinepak, Sorensen, wavelet, fractal, RealVideo,
vXtreme, MPEG2 codec and many others. Preferably, the codecs have a bit rate of 4.5MBPS and above. Final Cut Pro 4 or Ulead Video Studio 4 is preferably used for the editing. The multiplexed signal can be transmitted to geographically different locations using microwave, Radio frequency, satellite, fiber-optic channel or any other means of transmitting video signal in analogue or uncompressed digital form. For display the multiplexed signal is separated into two separate video streams by commercially available decoders. A Video Converter box from Vrex Inc is preferably used. The two-separated videos are displayed through a setup similar to that described above.
The present invention will have immediate effect in easing the training and mentorship for the future robotic surgeons. The learning curves of two consecutive surgeons were analyzed. Surgeon-A was trained without and surgeon-B was trained and mentored with the stereoscopic projection based display. First 20 cases of both surgeons are analyzed (n=40). The mean operative time during mentoring for surgeon-B was 45% less than the mean operative time of the surgeon-A, during the same phase of training (p < 0001).
Throughout this application, author and year, and patents, by number, reference various publications, including United States patents. Full citations for the publications are listed below. The disclosures of these publications and patents in their entireties are hereby incorporated by reference into this application in order to more fully describe the state of the art to which this invention pertains.
The invention has been described in an illustrative manner, and it is to be understood that the terminology that has been used is intended to be in the nature of words of description rather than of limitation.
Obviously, many modifications and variations of the present invention are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the described invention, the invention may be practiced otherwise than as specifically described.