Plenoptic Cameras in Surgical Robotics: Calibration, Registration, and Evaluation

A. System Description

The core components of the system are a light-weight 7-DOF robot arm (LWR 4+, KUKA Robotics Corp., Augsburg, Germany), a surgical tool (Endo360, EndoEvolution, Raynham, MA), a plenoptic camera and software for quantitative 3D measurement applications (R12, Raytrix GmbH, Schauenburgerstrasse, Germany) with a custom field of view (FOV) of 70 × 65 × 30 mm [25], and custom software developed in Open RObot COntrol Software (OROCOS) [26] and Robot Operating System (ROS) [27] as shown in Fig. 1. The KUKA Robot Controller (KRC) computer is in charge of the kinematics, dynamics, control and generating motion trajectories. A simple program written in KUKA Robot Language (KRL) enables the communication between the KRC and the OROCOS components and ROS packages via Ethernet using the Fast Research Interface (FRI) [28]. The Raytrix camera has an Application Programming Interface (API) with proprietary binary libraries and some read-only parameters. The API provides a virtual depth image and a focused image which are captured on a dedicated Windows machine with an NVidia GeForce GTX Titan graphics card. Virtual 3D depth is processed at 10 frames per second (fps) at a pixel resolution of 2008 × 1508.

B. Plenoptic Camera Model

Plenoptic cameras have a main lens and an additional micro lens array (MLA). Image features are matched between different nearby microlens images from which virtual depth is calculated by parallax. A simplified plenoptic camera model based on [20] is shown in Fig. 2. The distance b between the image sensor plane and the MLA and distance a between the MLA and the virtual image plane are used to define virtual depth ν = a / b. Parameter b is proprietary and provided by the manufacturer. In Fig. 2, the image feature seen in two microlens images corresponds to I₁ and I₂. The difference between matched points Δ_I = |I₂ – I₁|, and the distance between microlens centers D are used to calculate the virtual depth following the intercept theorem: ν = D / Δ_I. The MLA makes it possible to evaluate the depth of an image feature seen in more than one micro lens image (Fig. 3a and 3b). The camera manufacturer provides software to find the centers of the micro lenses, from which distance D can be found.

The software also enables the generation of an extended depth-of-field image, where the entire image is focused provided the subject is inside the depth-of-field range (Fig. 3c). The camera provides a grayscale image with additional depth information at each pixel coordinate (Fig. 3d). The intrinsic camera parameters include the focal length, the focus distance, the distance between the MLA and image sensor, and pixel size. It is also necessary to include distortion coefficients to rectify the image for metric depth measurement. Through our calibration and custom scripts, this information was translated to a metric point cloud and used to position the arm.

C. Command Interface

The system performance is evaluated on phantom suturing pads, which are used to train surgeons. We use the system as a point-and-click positioner for a robotic arm. The user must have an easy to use command interface to select the target tool position and informing the robot. A graphical user interface is developed based on the visualization component RViz of the Robot Operating System (ROS) framework and ROS messages and topic.

An operator clicks on desired suture location P_c within the visualized point cloud on the RViz display:

where the coordinates are expressed in the camera frame after metric calibration. This point is then expressed in the robot base frame to generate Cartesian motions:

where rigid-body transformation ^CT₀ takes the camera frame to the robot base frame and is often called the hand-eye calibration. Fig. 4 shows the command interface and the transformations. In Section II.E, we describe how the hand-eye transformation is found.

D. Metric Depth Calibration

Metric depth calibration will ensure that the values reported by our imager are at the correct scale and location with respect to the camera coordinate system. After intrinsic parameters are calculated, the camera natively reports the virtual depth, which is the distance between the virtual image point and the microlens array, as shown in [20]. As we will see, this virtual depth does not correlate linearly with rectified metric coordinates.

The camera data is streamed over Ethernet to a custom ROS node developed in C++, where subsequent rectification and conversion to metric coordinates is performed (Fig. 1, Raytrix node).

We adopt a pinhole camera model for the total-focus grayscale image and find the camera intrinsic parameters using Zhang’s well-known camera calibration algorithm [29] to generate a rectified extended depth-of-field image. Using a checkerboard pattern and the intrinsic parameters, the reference coordinate frame of the checkerboard is expressed in the camera coordinate frame. For metric calibration of virtual depth, we rigidly attach a flat calibration pattern (Fig. 5a) to the robot arm and manually move the robot to align the checkerboard to the camera coordinate system XY plane. Verification of alignment is possible by a ROS node for pose estimation (Fig. 5b), which takes as input the extended depth-of-field image of the pattern, intrinsic camera parameters, and known size of the checkerboard. A separate gradient detection module is run on the rectified image to find the corners of the checkerboard on the rectified 2D extended depth-of-field image, which are then overlaid on the corresponding point cloud points as seen in Fig. 5c. This is possible because the camera API provides a one-to-one matching between the extended depth-of-field image pixels and the depth image.

While moving the checkerboard at regular fixed intervals in Z, we observe a non-linear depth distortion related to distance from the sensor. This can be seen in Fig. 6a and 6b. As the reference moves away from the camera, its size in the XY plane also increases disproportionately, when it should remain constant. There is also a distortion of depth values along the Z axis [20]. We evaluate the reported virtual depth in a FOV larger than the one selected for the surgical task. In fact, we find that it is possible to approximate the distortion in our chosen FOV with linear regressions.

The best-fit 3D lines that go through the four corners intersect at a vanishing point in front of the camera in the least squares sense. The vanishing point and the four corner points approximate a pyramid (Fig. 6c). We moved the planar calibration pattern at fixed displacements of 10 mm along the z-axis. This helps to find the z scale using the faces of the pyramid and the principle of similar triangles. Once the z scale is found, and because we used a calibration pattern with known x and y dimensions, the x and y scales can be calculated as a function of actual depth. In essence, we rectify and scale the pyramid to create a homogeneous rectangular box, where a constant size for the calibration checkerboard can be reported. This box is the calibrated FOV, which translates the reported virtual depth and XY locations to metric 3D measurements.

E. Robot-Camera (Hand-Eye) Calibration

Once we are satisfied that the camera is reporting correct metric measurements, we must know the camera position and orientation with respect to the robot base frame, also known as hand-eye calibration. To accomplish this, we observe a calibration object with the camera and also with the robot. The object is a checkerboard pattern seen previously in Fig. 5b. Using the rectified plenoptic camera image and the pose estimator explained in the previous section, we are able to segment the checkerboard boundaries and determine its position and orientation with respect to the camera coordinate system. The checkerboard pose with respect to the robot is then determined.

We cannot use a reference feature or camera directly attached to the robot arm, as is standard practice [31], [32], because long surgical tool lengths put our workspace and camera view out of reach of the robot arm and necessitate a pointer rod. A precisely machined calibration pointer rod is attached to the robot arm. The pointer rod has been designed to provide less than 0.2 mm of runout, meaning 0.1 mm of positioning error in the XY plane of the tool. This ensures that the tool model closely matches reality, and allows us to know the location of the pointer rod tip very accurately. The rod is then used to touch the four corners of the planar calibration checkerboard and quantify their positions in the robot coordinate system. Once we have at least 4 points, we can solve for the rigid transformation between the two coordinate frames. In practice we use two sets of four-point correspondences, measured at different depths with respect to the camera to quickly estimate the transformation from non-coplanar points. This registers the camera to the robot, and allows position commands from the plenoptic point cloud to be executed in the robot base coordinate frame.

F. Phantom Targeting

To evaluate our total system accuracy, we target 10 pre-marked locations on a soft-tissue suture pad (3-Dmed, Franklin, OH). These locations have been distributed randomly throughout the FOV and at different heights. The distance between the predetermined and targeted points will count as the error. The error is a 3D dimensional norm which is derived from knowing the error in the XY plane of the suture pad as well as the error in the depth or bite size of the suture. These will be measured as shown in Fig. 7.

The XY error is the distance from the target dot to the midpoint of the suture entry and exit points. The depth error is the distance from the desired to achieved bite depth. These are combined using Pythagoras’ theorem to find the overall 3D targeting error (Fig. 7a). To evaluate the precision and accuracy of the placement of sutures in a curvilinear track, we will also measure the consistency of suture spacing (the distance between consecutive stitches) after completing 6 stitches across a curvilinear track with desired spacing of 3 mm ((Fig. 7b). Both tests are repeated 3 times at different locations within the camera FOV. These tests combine the error of the camera, robot, and user clicking on target dots into one metric. We measure the XY accuracy, depth, and spacing consistency using a pair of calipers (Products Engineering Corporation, Torrance, CA).

A. Accuracy of Metric Calibration of Plenoptic Camera

The camera was evaluated previously [23] in terms of accuracy and precision. An accurate metric calibration will rectify the distorted pyramid and report a checkerboard of uniform size at any depth in the calibrated FOV. We observe the checkerboard again, detecting a grid of 110 points at 5 depths, and quantify the deviation from this ideal rectangle across a volume of 50 × 45 × 20 mm. The accuracy was found to be 1.14 ± 0.80 mm on average, and errors were concentrated at the edges of the FOV as seen in Fig. 8. To test precision, the camera viewed a flat plane at 6 different locations within the camera FOV. The resulting 6 point clouds were split into 15 sections and a plane was fit to each section. The deviation of each point from its local plane is the precision at that location. The average precision was found to be 0.97 ± 0.77 mm for this FOV. In comparison, 3D reconstruction error with a calibrated stereo endoscope can range from 1 mm at 0.25 Hz [32], to 3 mm at a comparable framerate of 1.77 Hz, to 10 mm at 4.9 Hz [33]. These accuracy and precision tests give us confidence that our metric calibration was successful, and we can proceed to apply this technology in a demanding surgical task.

B. Accuracy of Hand-eye Calibration

To evaluate the success of the hand-eye calibration, we compare the position of the tool tip of the calibration rod, measured in the robot base frame as it touched the corners of the calibration pattern, with the coordinates of the same point from the point cloud in the camera frame transformed to the robot frame using the calibrated ^CT₀. The calibration pattern was moved within the FOV of the plenoptic camera 5 times, each time recording and comparing 4 corner points. The average reprojection error from 20 locations is 1.57 mm with a standard deviation of 0.9 mm. The significance of this result is the low standard deviation of 0.90 mm, which is similar to the metric calibration standard deviation of 0.80 mm. This means that addition of long length of surgical tool does not grossly affect accuracy if care is taken in designing the tool with small runout. This result gives us confidence that the registration will allow a precise task to be informed by the camera and executed by the robot.

It should be emphasized that these results are obtained within the calibrated FOV of the camera. Because we do not model the radial distortion of the plenoptic camera, the metric calibration breaks outside the calibrated FOV, where the linearity assumption is no longer true. In these circumstances, we have observed reprojection errors of 4 mm or higher.

C. Phantom Targeting

As a robust evaluation of our system performance in a relevant setting, we target various locations on the pad seen in Fig 6a. After the pad is held firmly in place, the user clicks on 30 desired suture locations and allows the robot to move and place stitches. Results from this experiment can be seen in Tables I and II.

The clinical performance of the camera can be judged by the targeting error and the consistency of suture spacing, which were measured on the suture pad as seen in Fig. 9. The average 3D norm targeting error for all 30 targets across both phantoms was 1.8 mm with a standard deviation of 0.43 mm. The average gap between suture placements for all 18 targets on the curvilinear phantom was 2.94 mm with a standard deviation of 0.11 mm over 6 stitches. The low standard deviation is evidence that the camera performed well as a positioning tool, and enabled easy and consistent targeting for a surgical task.