Use of tactile feedback to control exploratory movements to characterize object compliance

Introduction

Humans interact with compliant objects to judge ripeness of fruits, the air pressure in bicycle tires or the quality of a mattress. Expert bakers judge the quality of flour by evaluating physical firmness or toughness of dough (Katz, 1937). During breast or prostate examinations, healthcare practitioners use their hands to locate and characterize a hard lump in soft tissue. Unlike visual features such as size and shape, compliance can only be appreciated via active or passive touch.

It is essential for social and personal assistive robots and prosthetic hands (a form of telerobot) to be able to perceive material properties such as compliance to handle household objects. The ability to interact with fragile objects is necessary particularly if such systems are designed to interact physically with humans. Compliance perception could also be beneficial to robot-assisted, minimally invasive surgeries by detecting a hidden tumor in an organ or a calcified artery in heart tissue (Yamamoto et al., 2009). A variety of tactile sensors have been designed to solve the tactile sensing problems in robotic manipulation and medicine (Webster, 1988), but their practical use is limited by the hostile environments to which robotic and prosthetic hands are typically exposed. The BioTac^® is a robust and easy to repair tactile sensor that is capable of detecting point of contact, normal/tangential contact forces, and object spatial properties with impedance sensing electrodes (Wettels et al., 2008a; Wettels and Loeb, 2011), micro-vibrations associated with slip and textures through a hydro-acoustic pressure sensor (Fishel et al., 2008), and thermal fluxes with a thermistor (Lin et al., 2009).

Previous studies of compliance discrimination by robots used a combination of tactile and force sensors. (Takamuku et al., 2007) built a tendon-driven robot hand covered with strain gauges and a piezoelectric polyvinylidene fluoride (PVDF) skin. By performing squeezing and tapping over objects with different material properties, the strain gauges in this tactile sensor enabled the discrimination of hardness of different materials. Campos and Bajcsy (1991) proposed a robotic haptic system architecture that performed haptic exploratory procedures based on Lederman and Klatzky (1987) psychophysical studies of human performance. Hardness of objects were determined by measuring the force required to produce a given displacement. Both studies focused on measuring contact force and indentation displacement to discriminate object hardness or compliance. An adaptive force/position control algorithm was tested on an industrial robot to maintain force along the normal direction to the surface while moving in tangential directions on a rubber ball with 10 cm radius and 5000 N/m stiffness (Villani et al., 2000). In this paper, we present the results of using information about distributed deformation of the elastic skin of our tactile senor to discriminate compliance, a strategy that appears to be similar to that used by humans. This is made possible by using sensory feedback from a cluster of impedance sensing electrodes in the BioTac that are responsive to distributed forces. With these electrodes we were able to maintain a consistent orientation while applying normal forces to the surface of the object.

Subjective hardness/softness discrimination has been studied in psychophysical studies. Srinivasan and LaMotte (1995) showed that humans are efficient at discriminating subtle differences in softness under both active touch and passive touch with only cutaneous sensation but they are unable to discriminate even large differences during local cutaneous anesthesia. This suggests that tactile sensory information independent of proprioceptive information is necessary for discriminating softness of objects with deformable surface. Their studies also show that randomizing maximum force levels and indentation velocity in passive touch does not seem to affect sensitivity. This indicates that compliance discrimination can be done without fine control of these movements. Instead, we propose that spatial distribution of skin could be the cue for compliance discrimination. Peine (1999) developed a taxonomy that classifies the surgeons' finger motion during palpation procedures. They found that surgeons apply various normal force with no lateral motion to sense the stiffness of body tissues. Lateral motion after applying heavy pressure was found to enhance the ability to detect hard lumps in soft tissue.

To acquire information about object properties, humans tend to perform stereotyped exploratory movements Lederman and Klatzky, 1987. The exploratory movements to detect hardness are pressing and squeezing (Lederman and Klatzky, 1990). We have developed a haptic robot platform with a Barrett hand-wrist-arm system whose three fingers have been equipped with novel BioTac^® multimodal tactile sensors. In this paper, we present algorithms for the control of human-like exploratory movements for pressing on and characterizing objects with various hardnesses (durometer values). When robot gradually presses its fingertip into rubber samples with compliant surfaces, it uses the sensory feedback from the tactile sensor (BioTac) to control both normal and tangential contact forces and to adjust the orientation of its fingertip to account for the potentially unknown orientation of contact surfaces and internal discontinuities such as buried lumps. The distributed deformation sensed by the BioTac can be used to estimate the compliance of the contact surface.

Materials and Methods

We present data from initial experiments with flat objects made from materials with varying hardness to demonstrate the simultaneous use of multimodal tactile sensor data to control exploratory movements and to interpret their results.

Experiment Setup

Overview of the biomimetic tactile sensor (BioTac)

The BioTac (Figure 1A) consists of a rigid core housing all electronics and sensory components surrounded by an elastic skin that is inflated with an incompressible and conductive fluid. When the skin contacts an object, this fluid is displaced, resulting in distributed impedance changes in the electrode array on the surface of the rigid core. The impedance of each electrode tends to be dominated by the thickness of the fluid layer between the electrode and the immediately overlying skin. The skin has a pattern of asperities on its inner layer that gradually compress with increasing normal force, preventing object saturation (Wettels et al., 2008b). A MEMS pressure transducer measures hydrostatic pressure, which increases depending on the distribution of deformation in the elastic skin.

Similar to the human fingertip, the BioTac sensors are sensitive to tangential as well as normal forces. When performing a compliance movement it is desirable to apply forces normally and symmetrically to the object. For the haptic robot, this means servoing its end-effectors in the pitch and roll directions to orient a flat portion of the core of the BioTac that defines a local coordinate frame (Figure 1B). The sensory feedback is provided by four adjacent electrodes on this flat region whose impedance depends on compression of the skin against the electrode surface. These four adjacent electrodes are labeled electrode 7, 8, 9, and 10 on the electrode array map (Figure 1C). The pair of electrodes along the x-direction (8 and 9) and the pair of electrodes along the y-direction (7 and 10) are used for servocontrol of the pitch and roll, respectively, of the robotic fingertip. When the tactile sensor detects differences between these pairs of electrodes the error is corrected by adjusting the pitch or roll of the fingertip with the robot. The total contact force during indentation is estimated from the sum of impedance changes on all four electrodes. When pressing into a compliant object, the object has a tendency to wrap around the finger and the resulting forces can be measured by lateral electrodes not on the flat surface (such as 17). Comparing this change with the relative magnitude of impedance changes in the central four electrodes can yield substantial information about the compliance of the object. Additional information from the fluid pressure can also be used to characterize these changes. The sensor signals that provide information about compliance depend also on the curvature of the surface of the object, which must be estimated simultaneously from the complete temporal profiles of all sensor signals (Wettels and Loeb, 2011).

Testing materials

The levels of compliance for objects used in this experiment are classified by their durometers. The durometer is measured by the indentation depth into a material created by a given force on a standardized indenter with specific diameter. There are several scales of durometer depending on the diameter and configuration of the indenter, the spring forces applied on the tested materials. The samples in this experiment were all one inch thick and made from Neoprene rubber (50 Shore A) and polyurethane rubber (30 Shore A, 50 Shore 00, and 30 Shore OO), going from hard to soft.

Experimental procedure

The experiments were conducted on the seven DOF Barrett WAM robot arm and four DOF Barrett Hand BH-280 equipped with the BioTac. In each trial, the robot pressed one digit against a rubber sample in an unknown orientation and position. The robot controller had no prior knowledge of the orientation of the surface; instead it used tactile sensory feedback to identify a contact surface and adjust its finger orientation while pressing onto the compliant surface (Figure 2A).

Robot Exploratory Movements

The exploratory movement can be divided into three phases: (1) Reach to an object surface by controlling position in Cartesian coordinate system with smooth path movement. The desired position is either provided a priori or estimated by machine vision. (2) Maintain normal contact and orientation with the center of the fingertip by maintaining a symmetrical distribution of force on a cluster of tactile sensors. (3) Controlling the exploratory movement, which consists of pressing the fingertip gradually into the contact surface while maintaining normal orientation of the fingertip in the pitch and roll directions.

Online orientation control using tactile sensor feedback

In order to maintain the orientation of the flat portion of the sensor while gradually pressing into a compliant surface, the desired orientation trajectory is generated by feedback signals on the two pairs of electrodes. These differential signals are used to incrementally increase or decrease current pitch and roll angles (β_c, γ_c) with very small increments (Δβ, Δγ) in the finger local coordinates, respectively. From the new local roll-pitch angles (β, γ) in the finger local coordinate frame (shown in Figure 1B), the corresponding finger local rotation matrix R_local can be derived and translated into rotation matrix in the robot base coordinates ${}_B{}^{F}R{}_{finger}$ which is the forward kinematic from fingertip to robot base. Instead of using roll-pitch-yaw angles for orientation control directly, a unit quaternion representation of orientation [η,ϵ₁,ϵ₂,ϵ₃] is derived from the new rotation matrix, because of its singularities-free property (Yuan, 1988). This online orientation generation algorithm is shown in a pseudo-code, (Table 1).

The scalar η part and the vector part ϵ₁,ϵ₂,ϵ₃) in a unit quaternion representation fulfill ${\text{η}}^2+{ϵ}_1^2+{ϵ}_2^2+{ϵ}_3^2=1$.

A velocity-based orientation control with quaternion feedback is written below:

where ω_d is the desired angular velocities, K_o is diagonal gain matrix and e_o is the orientation error which is formulated using the unit quaternion (Yuan, 1988) as:

where $[{ϵ}_d\times ]=\left[\begin{array}{ccc}0& -{ϵ}_{\text{3}d}& {ϵ}_{\text{2}d}\\ {ϵ}_{\text{3}d}& 0& -{ϵ}_{\text{1}d}\\ -{ϵ}_{\text{2}d}& {ϵ}_{\text{1}d}& 0\end{array}\right]$ and $[\text{η},{ϵ}_{\text{1}},{ϵ}_{\text{2}},{ϵ}_{\text{3}}]$ is the current orientation and $[\text{η},{ϵ}_{\text{1}d},{ϵ}_{\text{2}d},{ϵ}_{\text{3}d}]$ is the desired orientation.

Robot position control

The desired positions and orientation generated by the online trajectory generation using tactile sensory feedback is achieved by a velocity-based operational space controller together with an inverse dynamic law and PD feedback error compensation in joint space (Nakanishi et al., 2008). Inverse dynamics control enables low PD feedback gains for compliant control while ensuring high tracking performances. The control law is written as:

where τ_{arm, p} is computed vector of torques to track desired joint angles q_d with measured current joint angles q, M is rigid-body inertia matrix of the arm, ${\stackrel{.}{\text{q}}}_d$ is the vector of desired joint velocity shown as:

where x and x_d are the measured and desired finger position and orientation, h is the vector of Coriolis, centrifugal, and gravitational forces, K_p, K_d, K_x, and K_post are diagonal gain matrices. J is the end-effector Jacobian, J⁺ denotes the pseudo-inverse of Jacobian and q_post is the vector of default posture optimized in the nullspace of the end-effector motion. The desired joint acceleration ${\stackrel{\mathrm{..}}{\text{q}}}_{\text{d}}$ and desired joint position q_d are obtained by numerical differentiation and integration of the desired velocity ${\stackrel{.}{\text{q}}}_d$.

Robot force control

Because the robot will press its end-effector onto compliant surfaces, external contact forces need to be taken into account. The external contact forces are obtained from the three force vectors on the BioTac extracted from impedance changes. They are used to compute torques in the joint space to account for the external contact forces by premultiplying them with Jacobian transpose, shown in Equation 5. The tracking of desired contact forces is achieved with a PI controller (Pastor et al., 2011)

where F_{arm_des} is desired forces at the end-effector, F_arm is the measured forces interpreted from BioTac, K_I is a diagonal positive definite gain matrix and Δt is the time-window during which the force error is integrated. The integral controller will compensate for steady-state errors during contact. An overview of the presented control architecture is shown in Figure 2B.

Normal and Tangential Force Extraction

During contact with an object, external forces deform the skin and fluid path around the impedance sensing electrodes. This deformation results in a distributed pattern of impedance changes on the electrodes. Previous studies have shown that both normal and tangential forces can be characterized from the impedance changes on the electrodes using machine learning techniques (Wettels et al., 2009; Wettels and Loeb, 2011). Here we present a simpler and more robust analytical algorithm to estimate normal and tangential forces.

The BioTac contains an array of 19 impedance sensing electrodes distributed over the surface of the core, which has a coordinate frame aligned with its long axis (Figure 3). Each impedance sensing electrode was determined to have the highest sensitivity to forces applied normally to its surface. The normal vectors to each of these electrodes in 3-axis coordinate space can be weighted with the change in impedance of these electrodes to determine an estimate of tri-axial force. We calculate the x, y and z force vectors from these electrodes with the following equation:

where [F_x, F_y, F_z] is the three dimensional force vectors on the BioTac, [E₁ ⋯ E₁₉] and [E_{1, rest} ⋯ E_{19, rest}] are the measured impedance changes and resting impedance values on the BioTac, $\left[\begin{array}{ccc}{\text{N}}_{1,x}& \cdots & {\text{N}}_{19,x}\\ {\text{N}}_{1,y}& \cdots & {\text{N}}_{19,y}\\ {\text{N}}_{1,z}& \cdots & {\text{N}}_{19,z}\end{array}\right]$ is a matrix in which each column is calculated normal vector for each impedance sensing electrode surface from the geometry of the rigid core, the S_x, S_y and S_z are scaling values for x-y-z three dimensional vectors to transform these arbitrary units into engineering units (N).

Results

Force Extraction

The scaling factor for each of the three dimensional estimated force vectors from the BioTac were calibrated on a 6-axis force plate (HE6x6-16, ATMI). We found that using the above-mentioned normal/tangential force calibration method was computationally efficient and achieved a low root-mean-squared (rms) error that exceeded performance of the neural network and machine learning techniques described in (Wettels et al., 2009; Wettels and Loeb, 2011). Figure 4 shows the actual forces (blue) measured from force plate and the predicted force vectors (red) extracted from BioTac by manually pressing and sliding the BioTac on the force plate. While pressing and sliding the BioTac, the flat portion of the BioTac was kept parallel with the surface of the force plate, similar to the orientation goal of the servocontroller for the exploratory poking movements. The rms errors for these sample movements were less than 10% of the applied forces in each axis.

Pressing with Orientation Uncertainty

Typical behavior of the system poking a surface with unknown orientation is illustrated in Figure 5. The top two plots show the impedances of the pairs of electrodes along x and y direction (E8 vs. E9, E7 vs. E10) on the flat portion of the core of the BioTac, which is the desired center of contact. The differential signals between those two pairs of electrodes are also displayed in the two middle plots in Figure 5. After the initial contact (around 0.5–1 s), the robot gradually pressed the BioTac into the compliant surface. A small asymmetry in the x-direction pair triggered the small correction to the roll angle that occurred at about 2–2.5 s and a larger correction at 4.5–5 s, shown in the bottom left plot. A larger asymmetry in the y-direction pair triggered a large pitch angle correction at 4–6 s, shown in the bottom right plot. The correction to pitch angle was relatively slow because it involved most of the proximal joints of the Barrett arm and it actually resulted in the second correction to the roll angle.

While the robot performed a pressing behavior, its contact force on the compliant surface was controlled by using tangential and normal force feedback extracted from the impedance electrode array on the BioTac. Figure 6 shows that the robot pressed 10 N on the compliant surface and kept its lateral tangential force (X Force) close to zero. The axial tangential force of 4 N is what would be expected given the 30° tilt of the flat portion of the fingertip with respect to the long axis of the BioTac. Stretch between the BioTac elastic skin and the compliant rubber sample caused by two rolling movements (around 2–2.5 s and 4.5–5 s in Figure 5) created a positive tangential force on the sensor, but the force controller gradually decreased the tangential force to close to zero by the end of the movements shown in Figure 6.

Compliance Discrimination

Force and displacement

Previous experiments showed that force and indentation displacement can be used in compliance discrimination when actively palpating with a tool (LaMotte, 2000). Thus, the ratio between force and indentation displacement can also provide information for the perception of compliance, especially for compliant objects covered with non-deformable surfaces, such as piano keys. During our experiment, the robot is controlled to apply 10 N in the normal direction onto five objects, consisting of an aluminum plate and four progressively softer rubber samples with durometer Shore 50A, Shore 30A, Shore 50OO, and Shore 30OO. Shore 50A is as hard as a pencil eraser and shore 30OO is a little bit softer than a racquet ball. When the robot was actively pressing the BioTac onto five objects with various hardnesses, normal forces and indentation displacement were measured from BioTac and robot joint encoders, respectively, as shown in Figures 7 and 8. The initial rates of rise of force were similar for all materials, but it took the robot longer to reach 10 N on soft materials 50OO and 30OO because it needed to constantly adjust its fingertip orientation to keep its fingertip orthogonal to the surface of soft materials. In Figure 8, we observe that the softer materials have more indentation displacement than the harder materials. The indentation displacement was measured by the position sensors in the robot actuators. It reflects the sum of indentation of the skin of the BioTac plus indentation of the object being probed plus stretching of the fine stainless-steel cables that link the motors to the joints. Figure 9 shows that indentation displacement trajectories were similar for all materials up to about 6 mm and 1 N, which were due to displacement of the elastic skin on the BioTac and the initial stretching on the finger joints, which explain a nearly linear relationship between force and displacement. From 6 mm to 12 mm, indentation displacement trajectories diverged as a result of the deformation of rubber samples with different compliance properties and stretching of cables in the robot. At about 15 mm and 10 N, they reconverged because they were then dominated by the stretching of the cables in the robot wrist subjected to large external forces from the compressed samples.

Deformation

In light of the complex combination of factors that contributes to apparent indentation displacement of the fingertip, it would be desirable to use temporal variations of average pressure and spatio-temporal variations of distributed skin deformation, as proposed for human discrimination of hardness by Srinivasan and LaMotte (1995). Both types of tactile information are available from the BioTac. The MEMS pressure transducer measures the average pressure of the fluid inside the space between the elastic skin and rigid core. The spatio-temporal variations of distributed deformation are provided by the impedance electrode array, especially from lateral electrodes adjacent to the central four electrodes used for controlling the applied force.

When the BioTac was pressed against hard materials (e.g., the aluminum plate and shore A 50 and shore A 30 rubber samples), fluid pressure plateaued or actually declined after normal force reached 2.87 N (around 2 s in Figure 10). This saturation is caused by the rigid object pushing the elastic skin against the rigid core on BioTac. The first part of the increasing fluid pressure reflects the compliance of the BioTac skin and fluid pressure, which grows nonlinearly after the skin contacts the core. The curves diverge before that occurs if and when the object compliance exceeds the BioTac compliance. As shown in Figure 9 (1–2 s), harder objects created a higher rate of average pressure changes in the BioTac. When BioTac pressed objects with softer surface, the soft surface not only pushed the elastic skin against the rigid core more gradually, but also progressively enveloped the side of the BioTac fingertip. This created higher saturation pressures for softer surfaces (around 2–3 s in Figure 10).

The tendency of soft surfaces to envelop the fingertip as they are deformed can be seen also in the impedances of more lateral electrodes such as #17 (Figure 11). The BioTac actually measures the current admitted into the electrode from a test pulse applied to various reference electrodes distributed in the fingertip, so a decrease in measured voltage from an initial value reflects an increase in electrode impedance. For the lateral electrode #17, the impedance initially increased similarly for all materials as increasing force was applied at the fingertip and the skin deformed, but the curves diverged as the more compliant materials deformed and enveloped the skin further from the centroid of contact. The reorientation movement that the robot made to correct pitch to maintain normal force (4.5–5 s in Figure 5) resulted in the transients in lateral electrode impedance at that time (Figure 11), which were particularly pronounced for the hard materials. After the robot corrected its orientation and reached its maximum contact force, the resting voltage on the lateral electrode reflected the compliance of the object.

Discussion

The tactile sensors available in the BioTac have properties similar to those in human fingertips and can be used to measure compliance of objects, but only if there is accurate control of the exploratory movement. Those same sensors can be used to control the exploratory movements, using tactile feedback control that may also be similar to what humans use when deciding how to palpate an unknown object. The preliminary results presented here are a first step in designing algorithms that can enable robots to produce the range of exploratory movements and the percepts that humans achieve thereby.

In this paper, the BioTac was controlled to explore flat compliant objects. Compliant objects that have curved surfaces or inhomogeneities in material properties will generate different responses in the sensors, whose interpretation may require additional exploratory movements. The tactile-based control of exploratory movements presented here should enable systematic exploration of such unknown objects regardless of their location or orientation with respect to the robot hand.

Systematic datasets need to be generated by poking the BioTac into objects with various curvatures and various compliances to develop a more complete perceptual algorithm. In previous studies, the impedance sensing electrodes of the BioTac could be used to make coarse determinations of radius of curvature of rigid objects (Wettels and Loeb, 2011). Humans tend to follow the contour of objects to perceive their precise shapes (Lederman and Klatzky, 1990). Palpation of hard objects buried in soft tissues probably reflects a combination of tactile-driven movements to determine the orientation of hard surfaces and kinesthesia to keep track of the location and size of those surfaces (Peine, 1999). In the future, we will combine pressing and contour-following exploratory movements to facilitate the perception of both compliance and shape of objects. Eventually, tactile information from exploratory movements must be fused with machine vision to permit location, characterization, identification, and dexterous manipulation of objects in the environment.

Conflict of Interest Statement

Jeremy A. Fishel and Gerald E. Loeb are equity partners in SynTouch, LLC, which manufactures and sells the BioTac sensors described in this article. Zhe Su and Tomonori Yamamoto have no commercial or financial relationships that could be construed as a potential conflict of interest.

References