Needle Tip Force Estimation Using an OCT Fiber and a Fused convGRU-CNN Architecture

Needle insertion is widely used in minimally invasive procedures, e.g., for biopsies or brachytherapy. Automated needle insertion is challenging and includes aspects like image guidance, needle steering, and force measurement [14]. Precise estimation of the forces acting on the needle tip is particularly interesting, e.g., for monitoring the needle-tissue interaction and detecting tissue ruptures, or to generate feedback during an intervention [10]. One approach is to measure the forces at the needle shaft. While this allows for simple integration of conventional force-torque sensors, the measurements do not reflect the actual forces acting on the needle tip. As large frictional forces act on the needle shaft during insertion, force sensors need to be decoupled or located close to the tip in order to obtain accurate needle tip force estimates [6]. The small diameter of the needles complicates the integration of sensors [12], which is particularly challenging for conventional mechatronic force sensors [4, 6]. In contrast, fiber optical force sensors are small, largely biocompatible, and not affected by electromagnetic interference, i.e., they are MRI-compatible [2]. Therefore, sensors based on Fabry-Pérot interferometry [2] or Fiber Bragg Gratings [8] have been proposed. Although these approaches have shown promising calibration results, they are rarely validated with tissue experiments [9] and manufacturing and signal processing can be difficult, e.g., when the fibers are subjected to varying temperatures or lateral forces. Yet another approach is based on optical coherence tomography (OCT), where A-scan images of a cylindric instrument tip have been used to estimate the deformation and hence the strain acting on a translucent silicone layer [7].

We consider force sensing using OCT and a sharp needle with a cone tip mounted on a needle using epoxy resin. The axial force acting on the tip is inferred from the epoxy layer’s deformation observed in a series of A-scans. We can tailor our method to specific application scenarios by using softer epoxy resin for higher sensitivity, as required for microsurgery, or stiffer epoxy resin for larger forces, e.g., as occurring during biopsies [2]. Generally, our approach is flexible and easy to manufacture as the epoxy material is interchangeable, the cone shape can be varied, and no accurate fiber placement is required. However, this imposes some challenges for calibration and force estimation, namely the robust identification of the deformation of the epoxy layer and a non-linear mapping of the measured deformations to forces. To this end we propose a force estimation method based on a novel convolutional gated recurrent unit-convolutional neural network (convGRU-CNN) architecture. Considering the high temporal sampling rate of OCT, we use a sequence of subsequent A-scans as an input to our model. In this way, we can take advantage of a rich spatio-temporal signal space for precise force estimates.

First, we present a detailed description of the force sensing needle and our convGRU-CNN architecture. Second, we describe our setups for calibration and evaluation of the needles. Third, we study the repeatability of force estimation for three different needles with different epoxy resin types. Finally, we present results for needle insertion into actual ex-vivo prostate tissue illustrating the feasibility of the approach and the importance of measurements at the needle tip.

Schematic drawing of the needle and the experimental setup. Not to scale. The needle contains an OCT fiber that images a deformable epoxy layer at the needle tip. Forces are measured by the force sensor at the base. A shielding tube is decoupled from the needle and the force sensor and prevents shaft friction measurements for tissue insertion experiments. The setup is moved with a linear stage.

Full size image

2.1 Needle Design and Experimental Setup

A schematic drawing of our needle and the needle driver is shown in Fig. 1. The needle has a diameter of 1.25 mm. An epoxy resin layer of approximately 0.5 mm connects the cone shaped tip to the needle shaft. An optical fiber is embedded into the shaft and glued to the epoxy. The fiber is connected to an OCT device (Thorlabs Telesto I). A linear motion stage is used to drive the needle along its axial direction. For calibration and evaluation, a force sensor (ATI Nano43) is mounted between the needle shaft and the motion stage. To study how the sensitivity of the sensor can be varied by using different epoxy resin, the resin was mixed with Norland Optical Adhesive (NOA) 1625 in different concentrations. The layer and the needle tip on top are attached using NOA 63. For calibration, the needle was driven against a metal plate. A large set of data was acquired by deforming the tip with random magnitude and velocity. For evaluation, the needle was inserted into a freshly resected human prostate at constant velocity. As the force sensor at the base measures the total force including friction, we consider a second setup using a shielding tube decoupled from the needle and the force sensor. In this way, we can measure the actual axial tip forces. Note, that this would be impractical for actual application as the tube is not flexible and would increase trauma. A photograph of the experimental setup is shown in Fig. 2. We provide a video and ultrasound recording of two insertion procedures with pork liver and a human ex-vivo prostate in our supplementary material.

Photograph of the experimental setup for the prostate insertion experiment. Note, that for calibration the needle is driven against a metal plate.

Full size image

2.2 Model Architecture

For our model input, we consider a series of A-scans prior to the current observation, as the current force estimate likely depends on prior deformation [1]. Furthermore, we do not extract the epoxy surface as an explicit deformation feature but instead let our model learn relevant features. In this way, we avoid inconsistencies when extracting features for different materials and tips and we exploit information captured in the deformed epoxy layer itself. Prior approaches for spatio-temporal data used CNNs to extract features from image data which are fed into a recurrent model [3]. Alternatively, the temporal dimension can also be handled by a convolution operation [13]. Also, convolutional long-short term memory (convLSTM) cells have been introduced which allow for temporal processing of high dimensional structured data [15]. Based on these approaches, we propose a novel convGRU-CNN architecture, as shown in Fig. 3. First, convGRU units take care of the temporal processing which results in a set of 1D feature maps. Then, a ResNet inspired [5] 1D CNN takes care of spatial processing. Compared to LSTMs, GRUs merge the input and forget gate and they merge the cell and hidden state for higher efficiency. We use a combination of convLSTMs and GRUs by replacing the matrix multiplications with convolutions. The proposed architecture is compared to other approaches that have been introduced for spatio-temporal data processing. We consider a 2D CNN that processes both the temporal and the spatial dimension with convolutions. Its structure is the same as the CNN part in the convGRU-CNN model. Moreover, we consider a CNN-GRU model where the 1D CNN first processes the A-scans at each time step separately. Then, the CNN feature vector is fed into two standard GRUs. Next, we consider a pure 1D CNN that does not consider prior A-scans for the current force prediction. Last, we consider a pure 3-layer GRU model. All networks are trained end-to-end in a single optimization run. We use the Adam algorithm for mini-batch gradient descent with a batch size of \(B=100\). We implement our models using the Tensorflow environment.

The convGRU-CNN model we employ. A series \(t_s\) of cropped A-scans of size \(d_c\) is fed into the model. The metal tip’s lower surface cannot be penetrated by infrared light which is why this signal part is considered noise. The first block in a series of ResBlocks uses a stride of 2 for the convolutions with kernel \(3^1\) and increases the number of feature maps. Subsequent blocks have a stride of 1 and keep the same feature map amount. The change in the number of feature maps is denoted in each group of ResBlocks. \(F_j\) denotes the number of feature maps of ResBlock j.

Full size image

2.3 Data Acquisition and Datasets

OCT is an interferometry-based image modality using near infrared light to create 1D depth scans (A-scans) of up to 3 mm reflecting the inner structure of materials. We acquired A-scans at a rate of 5500 Hz and force measurements at 500 Hz. We match the two data streams with nearest-neighbor interpolation, using the streams’ timestamps. Our dataset consists of sequences of \(t_s\) subsequent A-scans, each labled with a force measurement. Given that we do not need the full imaging depth of the OCT, image data beyond the maximum depth of the cone tip surface is cropped.

We consider calibration datasets for three needles with different epoxy resin types for evaluation with our convGRU-CNN model. Each dataset contains approximately 90000 sequences of A-scans, each labeled with an axial force. By default we use a window of \(t_s = 50\) with a crop size \(d_c = 70\) pixels. We use \({80}{\%}\) of the data for training and validation sets, which we use to optimize hyperparamters, e.g., \(t_s\), \(d_c\), \(l_r\), and network depth. The remaining \({20}{\%}\) of the data are used for testing. Sequences from the three sets are non-overlapping. Furthermore, one of the needles was evaluated in an ex-vivo experiment in a human prostate. We evaluate our proposed architecture by comparing it to the models mentioned in Sect. 2.2, reporting the mean absolute error (MAE), relative MAE (rMAE) and correlation coefficient (CC) between predictions and targets. All errors are given for the test set.

First, we report the results for different needles with a different stiffness of the epoxy layer. The results with the corresponding force magnitudes are shown in Table 1. The absolute error values vary, as the corresponding force ranges differ, however, the relative measures rMAE and CC show that models perform overall similar. Next, we present results for alternative model architectures. The results are shown in Table 2. The results show, that models that take prior A-scans into account perform better. Moreover, our proposed model outperforms previously introduced approaches for our application. Last, we show results for a needle insertion experiment for two different scenarios in Fig. 4. One experiment was conducted with the shielding tube and one without. With the tube, predicted values closely match the measured values. Without, there are large deviations as the sensor also measures friction forces.

Predicted and measured force values are shown for an insertion with the shielding tube (left) and without (right). Note, that differences between predictions and targets are caused by friction during target measurements, not inaccurate model calibration. Needle 2 and the model convGRU-CNN are used for this experiment.

Full size image

We introduce a novel technique for needle tip force estimation using an OCT fiber that images the deformation of an epoxy layer. As OCT has been used for needle-based tissue analysis [11], it may become more widely available in clinical settings. Our method comes with the typical advantages of optical methods, such as MRI-compatibility and bio-compatibility, while also being flexible and easy to manufacture. This is highlighted by the results for three different needles with epoxy layers of different stiffness. All needles show similar relative calibration errors with a CC in the range of 0.9996 to 0.9989, indicating that our method generalizes well for different epoxy resins. This allows for easy adaptation of our method to different scenarios with different requirements for force sensitivity and range.

Moreover, we propose a novel method for processing the spatio-temporal OCT data. Previously, time series of A-scans have been processed using recurrent architectures [11]. The approach shares parameters over time, however, it lacks effective spatial exploitation with parameter sharing over space through convolutions. As our convGRU model takes care of efficient processing of both temporal and spatial information, it outperforms the pure temporal GRU and pure spatial 1D CNN with an MAE of \(1.76\,\pm \,1.5\) compared to an MAE of \(3.24\,\pm \,4.0\) and \(3.22\,\pm \,4.1\), respectively. Furthermore, we adopted a CNN-GRU and 2D CNN model from non-medical domains for comparison of spatio-temporal processing architectures [3, 15]. Compared to the other models, the convGRU units in our model enable temporal processing first while keeping the data structure intact for subsequent CNN processing. This leads to superior performance for the problem at hand.

Lastly, we tested our needle in an ex-vivo experiment with a human prostate. Several other needle tip force estimation methods have been proposed, however, they often lack validation in tissue experiments [9]. One of the reasons for this is the difficulty to measure pure tip forces inside tissue as large friction forces will also be captured by external force sensors [6]. Therefore, we use a shielding tube that decouples friction forces from the needle. Although the decoupling is not perfect due to deformations, we can show that our method accurately captures events such as ruptures. Without the mechanism, in-tissue evaluation is not possible as frictional forces overlap with tip forces. The results indicate that our method is usable for actual force estimation in soft tissue.

We propose a novel method for needle tip force estimation. Our approach uses an OCT fiber imaging the deformation of an epoxy layer to infer the force that acted on the needle tip. The concept is easy to realize and allows flexibility by using different materials for different force sensitivity and maximum range requirements. In order to process the spatio-temporal OCT data we propose a novel convGRU-CNN architecture. For our problem, the method outperforms prior approaches for similar problems and also methods from other domains. Experimental results for force estimation in human prostate tissue underline the method’s potential for practical application.