Sensing Ultrasonic Mid-Air Haptics with a Biomimetic Tactile Fingertip

Ultrasonic phased arrays can generate haptic sensations in mid-air. They focus acoustic radiation pressure in space, which deflects the skin to induce tactile sensation  [1]. To evaluate whether the array is producing the desired haptic sensations, we need to understand how focal points of pressure interact with compliant skin to cause it to deform. In this paper, we propose a method for sensing mid-air haptics with a biomimetic tactile fingertip inspired by the human sense of touch. Using the data obtained from the sensor, we are able to visualize the different patterns produced by the haptic array.

Efforts to measure the haptic output from a phased ultrasonic array range from quantitative to qualitative. Quantitative methods include microphones to measure the sound pressure level of the generated focal points  [1, 8], directly measuring the ultrasonic output of the system without considering its interaction with other material. On the other hand, to quantitatively consider the interaction of the sensations with skin-like materials, Laser Doppler Vibrometry (LDV), a tool commonly used for non-contact vibration measurement, can give insight on how the haptic stimuli would interact with human skin at high frequencies  [2]. Alternatively, qualitative methods include pulsed schlieren imaging, which was used to visualize the pressure field produced by a focal point as it interacts with external materials  [5]. Additionally, by projecting the focal points onto the surface of an oil bath, it can be used to visualize the patterns generated by the haptic array  [6]. New research has used a microphone-based tactile sensor array to evaluate the vibrations of its surface due to ultrasonic haptic sensations  [9], highlighting the potential of tactile sensors for testing the output of a haptic system.

In this work, we propose a method to sense and evaluate mid-air haptics using the TacTip, a biomimetic tactile fingertip. The TacTip is biologically inspired by glabrous (hairless) human skin, which has an intricate morphology of layers, microstructures, and sensory receptors that contribute to its functions  [3, 10]. We present a method for analyzing mid-air haptic sensations with a tactile sensor, allowing us to quantitatively test ultrasonic arrays with a method inspired by the human sense of touch.

This work aims to develop a method for testing mid-air haptics with a biomimetic tactile sensor. We carried out experiments with the tactile fingertip mounted on a robot arm and an ultrasonic array (Fig. 1).

The TacTip, a biomimetic tactile sensor (left): the flat-tipped model used in this study; the skin of the TacTip with 127 inner nodular pins (middle); and the experimental setup with the tactile sensor mounted on a robot arm to collect data over the ultrasonic phased array (right).

Full size image

2.1 Biomimetic Tactile Sensor

The TacTip (Fig. 1, left panel) is a biomimetic tactile sensor developed at the Bristol Robotics Laboratory  [3, 10], based on the structure of glabrous skin. The human fingertip has dermal papillae where the dermis interdigitates with intermediate ridges in the epidermis. These ridges and papillae focus strain from the skin surface down to mechanoreceptors within the dermis. The TacTip mimics this structure with an outer rubber-like skin which connects to inner nodular pins (Fig. 1, middle panel). As the soft sensor interacts with objects, its skin deforms and the nodular pins transmit surface strain into inner mechanical movements, similar to human skin. An internal camera tracks the movement of its artificial papillae, making it possible to detect the shear deformation of the skin. The sensor has been used in many tasks in robot touch such as object exploration and slip detection [10]. The TacTip is manufactured using dual-material 3D printing, which prints both the sensor’s plastic base and the soft rubber-like material for the skin. This allows for low-cost and rapid prototyping of different designs as well as its integration with robotic grippers and hands. Additionally, the design of the TacTip is modular, allowing for different tips to be used, such as varying the shape or texture of the skin or varying the layout of the nodular pins  [10]. The tip of the sensor can be filled with gel to affect its compliance or be left unfilled. Since this is the first time the TacTip has been used to detect small forces on the order of millinewtons, we needed a more compliant tip; after testing tips with these variations, we found the flat-tipped TacTip without gel (Fig. 1, left panel) to be more sensitive, and thus suitable for this work.

2.2 Ultrasound Phased Array

To generate the mid-air haptic stimuli for the experiments, we used the Ultrahaptics Evaluation Kit (UHEV1) from Ultraleap. The array has a 16 by 16 grid of ultrasonic transducers which operate at 40 kHz to generate focal points in mid-air, with an update rate of 16 kHz. The device is accompanied by software which allows us to modulate these focal points so that they can be felt by users  [1] and to generate various shapes and textures  [6].

2.3 Experiment

We used a 6-DOF robotic arm (ABB IRB120) to move the tactile sensor over the haptic array. The robot arm moved the sensor in 10 mm increments over an 80 mm by 80 mm grid at a height of 200 mm above the haptic array. At each position, 30 frames were captured from the camera to image the TacTip’s inner nodular pins at 30 fps. This was done for a focal point generated by the array, as well as two shapes (a line and a circle). The shapes were generated by the array using Amplitude Modulation (AM) and Spatiotemporal Modulation (STM), to see whether the sensor distinguishes between these two standard modulation techniques. AM generates focal points in the path of the desired pattern and modulates their intensity over time, while STM generates one focal point and moves it rapidly along the path.

Analysis method. We capture an image from the tactile sensor as it interacts with the mid-air haptic stimulus (1) and extract each pin position (2). Voronoi tessellation is generated with pin positions as the center point for each cell (3); the change of area of each cell compared to an unstimulated sensor, \(\varDelta A\), is used as a measure of the stimulus intensity (4). This is repeated for readings over a grid (5). Gaussian Process Regression combines the data sets to produce detailed visualizations (6).

Full size image

2.4 Analysis

In this work, we developed an analysis method to sense mid-air haptics with a biomimetic tactile fingertip (Fig. 2). The images captured from the tactile sensor as it interacts with the mid-air haptic stimulus were processed to find the positions of the nodular pins at each time step. Then we used the pin positions to generate a bounded Voronoi tessellation, shown by Cramphorn et al. to transduce a third dimension to the sensor data  [4]. Voronoi tessellation partitions a plane based on the distance between points on that plane; each point along an edge is equidistant from two points, and each vertex is equidistant from at least three points. The areas of the cells give us information for tactile perception; increasing areas indicate a compression of the skin. Thus, the areas of each cell in the Voronoi tessellation were compared with a data set in which the sensor was not stimulated, and the difference between the two areas, \(\varDelta A\), was used as a measure of the intensity of the stimulus as felt by the sensor. This was done for every time step, and then averaged over the 30 frames of data. The process was repeated for readings in a grid over the haptic display to populate a two-dimensional plane. Then we trained a Gaussian process regression (GPR) model for the measured intensity, represented by \(\varDelta A\) (using the MATLAB function fitrgp with the default squared exponential covariance function). The output values were then scaled between zero and one, to represent the relative intensity of the stimulus as felt by the tactile sensor.

The mid-air haptic shapes, line and circle, as felt by the biomimetic tactile sensor. The Amplitude Modulation (AM) cases use four focal points modulated with a 200 Hz sine wave. The Spatiotemporal Modulation (STM) cases use one focal point moving at a repeat frequency of 100 Hz along the path of the shape.

Full size image

In this work, we used a biomimetic tactile fingertip to sense mid-air haptics, to develop a method for testing the output of a haptic display. We measured the response of the tactile sensor to a focal point of pressure generated by an ultrasonic haptic array as well as two haptic shapes, each generated by Amplitude Modulation (AM) and Spatiotemporal Modulation (STM).

3.1 Sensing Mid-Air Haptics on a Two-Dimensional Grid

The experiments showed that the biomimetic tactile fingertip used in this study, the TacTip, is able to sense the mid-air haptic stimuli produced by the ultrasonic phased array using our developed method (Fig. 2). The focal points of pressure generated by the ultrasound caused the skin-like surface to deform, expanding the areas of the cells in the Voronoi tessellation, allowing us to identify the location and intensity of contact. Voronoi tessellation was a valuable tool in transducing a third dimension in the data, which allowed us to visualize the sensations produced by the ultrasonic array.

Our analysis methods enables us to produce detailed visualizations of the mid-air haptic sensations (Fig. 3), allowing us to distinguish between different patterns produced by the ultrasonic array. Additionally, the variation in the strength of the focal point can be clearly seen. The visual representation of the focal point shows that it creates a localized region of increased displacement (Fig. 2, lower right panel). The point is much stronger in the center, and then decreases in intensity as you move radially outwards. This is similar for the other shapes; the center path of the shape has increased intensity, which decreases as you move away (Fig. 3).

The visualizations produced by our method allow us to compare the shapes generated by the ultrasonic array using different modulation techniques. We see that the tactile sensor is able to distinguish the four focal points that make up the amplitude modulated line (Fig. 3, top left panel). A user of the ultrasonic array would not distinguish the points as the distance between them is small  [1], and so it creates the illusion of a continuous line. The sensor can discriminate between the points because our analysis method is measuring the deformation of the tactile sensor’s surface, which would correspond to the deformation of the user’s skin rather than their perception of the sensation. On the other hand, the spatiotemporally modulated line (Fig. 3, lower left panel) is felt as a continuous line by the sensor. The focal point used to generate the line is moved along its path at very small increments. The distance between the points in this case is too small to be distinguished by the sensor, making the output more similar to how a user would sense the stimulus.

3.2 Sensing a Focal Point in Three-Dimensional Space

In the previous section, we presented our results when sensing various shapes over a two-dimensional grid. In this section, we extend our method to sense mid-air haptic stimuli in three dimensions. When a person interacts with the haptic array to sense the shapes it generates, they naturally move their hand around the display surface, which includes moving their hands up and down as they process the sensations they feel. Thus, understanding how the generated sensations vary with height is important to determine whether the desired effect is being produced and to check that there are no undesired artefacts. We repeated the data collection process described earlier for the point stimulus at different heights over the array, at 10 mm increments. This results in a three-dimensional grid on which we applied our presented analysis method.

The three-dimensional view of a focal point as felt by the biomimetic tactile sensor presented as cross sections at \(z = 0\) (left), which corresponds to 200 mm above the array; \(y = 0\) (middle); and \(x = 0\) (right).

Full size image

This experiment allows us to see how the shape is sensed over a three-dimensional surface, looking at how the shape varies by height. The focal point is generated by the array at a specific height in space; however, there are still sensations at other points due to the interaction of the ultrasonic waves  [8]. The point stimulus is sensed by the tactile fingertip as an elongated spheroid, with a localized region of increased intensity (Fig. 4). It appears the lower the intensity of the stimulus felt, the more elongated it is. As the sensor moves away from the center height, the stimulus becomes fainter.

Tactile sensors can further our understanding of the human sense of touch. Our experiments have shown that we can use a biomimetic tactile fingertip to sense the mid-air haptic stimuli produced by an ultrasonic phased array, providing insights on the deformation of the skin-like material of the TacTip due to ultrasonic mid-air haptic sensations. This allows us to produce detailed visualizations of the sensations produced by the device. Using our analysis methods, we were able to see the difference between shapes that are amplitude modulated versus spatiotemporally modulated by the ultrasonic array. Additionally, we were able to sense and visualize a focal point in three-dimensional space, providing insights on how the focal point varies by height.

The visualizations of the stimuli produced in our study are similar to those in other works which use alternative methods. For example, Laser Doppler Vibrometry was used to measure the deformation of skin-like material due to ultrasonic mid-air haptics  [2]; it measured the high frequency vibrations (50 Hz and above) of the skin surface, and the root mean square (RMS) of the deformation was used to visualize the sensations. While we do not measure the high frequency vibrations, we get similar results. This could indicate that the data we collect is similar to the RMS of the skin deformation. Additionally, our three-dimensional measurements of the focal point look very similar to simulations of the same stimulus  [8]. The elongated spheroid felt by the sensor looks like the higher values of acoustic field pressure in the simulation, suggesting that the tactile fingertip is able to sense the ultrasound when it crosses a threshold pressure.

This work has provided insights into the measurement of haptic stimuli, but it has areas for improvement. At this point, we have measured the intensity of the stimulus without relating it to a specific physical value. Further work is planned to determine the relationship of the measured stimulus intensity to the skin deformation, which would allow us to compare our results with other quantitative experiments. Additionally, we do not measure the skin deformation at high frequencies. While the results we get are similar to those which use vibrometry, studying the vibrations of the artificial skin could determine whether the sensor does behave similarly to human skin. One approach is to modify the sensor with a higher frame rate camera which could allow us to see the high-frequency deformations of the skin; another approach would be to add another high-frequency tactile sensing modality to the TacTip  [7].

Our work has shown promising results for sensing mid-air haptics with a biomimetic tactile fingertip. The developed approach could be used as an investigative tool for evaluating and improving the capabilities of haptic displays. The insights gained from this work could also be used to investigate human perception. In the future, we could apply our methods to intelligent exploration of the haptic stimuli. This could allow us to develop an autonomous robotic system that is able to feel and interact with the sensations similar to how a person would explore mid-air haptic stimuli.