Keywords

1 Introduction

1.1 Research Background

In recent years, virtual reality (VR) technology has attracted significant attention. In 2016, head mounted displays (HMD) such as Oculus Rift [1], HTC Vive [2] and PlayStation VR [3] were released allowing the use of VR devices on a daily basis.

The term “virtual reality” originated in 1989, and since the 1980s, the emergence of VR research has emerged in various disciplines such as computer interfaces, simulation and robotics [4].

VR technology requires the satisfaction of three elements: (1) “three-dimensional spatiality”, (2) “real-time interactivity” and (3) “self-projectability.” Self-projectability, allows the user to obtain a “state without inconsistency between sensory modalities [4].”

Sensory modality is an integrated perception, consisting of visual and auditory sensations, such as somatosensory and vestibular. In other words, it is essential that both the user’s perception of arm position and acceleration be recognized by a sense of self-acceptance, and that both are consistent with accompanying audiovisual information.

HMDs such as the Oculus Rift and HTC Vive are wearable, visual displays, both worn and used on the head. Images are displayed in such a manner to cover the user’s field of vision, presenting, to the user, an immersive experience with high three-dimensional spatiality. Recent HMDs also have self-position estimation functions, allowing the possibility to increase self-projectability via the combination of movement in real space with movement in the projection.

HMD, a technology for visual perception, is currently on the market, but technologies that let the user experience sensations such as smell, taste, acceleration and force sense are currently in production.

Among these sensations, the sensation of force and tactility is thought to be an important ability in human interaction. Force/tactile sensations will play an important role in many areas of VR exploitation – e.g., medical, manufacturing, art, education and entertainment. For example, force/tactile sensations have a number of applications in medical fields, especially in surgical simulations, with the aim of increasing surgical success rates of surgery.

Devices such as the Phantom [5] and Falcon [6] simulate this sense of force. Although these devices are able to obtain feedback on force, the user is not able to use the device while mobile. A device that solves this problem is the SPIDAR-W [7].

SPIDAR - W is a device worn on the body, allowing users to obtain force feedback while walking. With SPIDAR-W, force measurements are sent directly to the end effector, held in hand by the user. This mechanism is capable of simulating the forces involved with hitting an object, but is incapable of simulating the grasp force, i.e., while the user grasps an object.

If, in the future, SPIDAR – W technology is able to simulate this grasp force, there are numerous applications to various fields. For example, the self-projectability of VR sports training can be further enhanced. In the immersive social network service or the teleexistance technology which can be interacted as if it is on the spot with the acquaintance of a remote place, user will be able to get close communication.

1.2 Research Objectives

This paper documents the attempt to add a new grasp force function to the SPIDAR-W, a wearable, force, display device. In order to increase the “self-projectability of grasp operation”, a device has been developed, which provides a correct calculation result to the user’s grasp operation.

1.3 Research Outline

First, a new end effector for grasp operation was developed for the SPIDAR-W. Next, the performance and points of improvement of the manufactured end effector were evaluated by a number of experiments. In the experiment, the self-projectability of the grasp operation was evaluated by measurement. Finally, experimental results were analyzed and the performance of the device as well as future issues were examined.

2 End Effector for the SPIDAR-W

2.1 SPIDAR-W

SPIDAR-W is a wire drive type 6 DoF force display device that is controlled by two end effectors, each with eight motor units, enabling a dual-hand force feedback. A thread is used for the force feedback, and a force of 6 DoF is provided by the resultant force of the tension generated by the motor. By moving the end effector in the SPIDAR-W, both hands can be manipulated in VR space with the corresponding force in VR space transferred to both hands of the user. SPIDAR-W consists of an aluminum frame, motor unit and SPIDAR circuit board. Figure 1 displays the SPIDAR-W installation scheme and Fig. 2 displays the outline of SPIDAR-W.

Fig. 1.
figure 1

A landscape view with the SPIDAR-W.

Full size image
Fig. 2.
figure 2

Outline of the SPIDAR-W.

Full size image

The motor unit is composed of a DC motor (MABUCHI), a pulley with slit and an encoder. The SPIDAR circuit board is connected to the PC via USB cable.

The HMD (HTC Vive) is attached to the user’s head and the VR space is projected. The HMD, in Fig. 1, has a stereo camera but was not used for experiments performed here.

The HMD can perform 6 DoF tracking routines in the space surrounded by the two sensors. However, the coordinate system of the HMD and the coordinate system of the SPIDAR-W are independent.

2.2 Current End Effector

The end effector is a device that links the hand of the user to the hand simulated in VR space. With the current end effector, it is possible to grab or release objects in VR space using buttons. The current end effector is composed of four end points for the fixation of the thread, a grip and two buttons. Dimensions of the end effector are 15.5 cm in height and 19.5 cm in width. The four end points are placed upon the apex of the regular tetrahedron and are connected by an aluminum pipe. Due to the sensitivity of the grip, it is based on the expansion controller for the Nintendo Wii video game system. The grip has the two buttons, each used for calibration and grasp operation. The current end effector is shown in Fig. 3.

Fig. 3.
figure 3

The current end effector.

Full size image

2.3 Grasp Operation End Effector

Since the current end effector always needs to be held, the user perceives that (s)he is continuously gripping the object. To solve this problem, a new end effector was designed, capable of achieving both a “grasp” state while grasping the object and an “open” state while the hand is open. An “open” state was achieved using velcro tape.

The use of a pressure sensor, instead of a button, permitted a more intuitive grasping operation. By using the pressure sensor, not only the ON/OFF of the grasp operation but also the grasping strength can be reflected in VR space.

Similar to the current end effector, the grasp operation end effector is composed of four end points to fix the thread. Figure 4 displays the new end effector for grasp operation.

Fig. 4.
figure 4

The new end effector for grasp operation.

Full size image

The semi-cylindrical end effector body was constructed using a Value 3D MagiX MF-1100 (MUTOH) 3D printer, with an ABS HG 1.75 mm filament. The size and arrangement of the endpoints were identical to that of the current end effector. The end effector was fixed with Velcro tape for easy attachment to the palm of the hand, with the pressure sensor designed to touch the middle phalanx of the finger. Figure 5 displays both the mounted configuration and performance of the grasp operation for the end effector.

Fig. 5.
figure 5

Performing a grasp operation.

Full size image

Pressure sensor FSR - 406 (INTERLINK ELECTRONICS) was used as the input sensor. In this pressure sensor, resistance values change in accordance with the contact area, allowing the detection of continuous pressure changes. The pressure is digitally converted to 1024 steps through Arduino Uno so that it can reflect the strength of grasping in the simulation.

Subsequently, in order to evaluate the performance of the manufactured end effector, the experiment of the next chapter was conducted.

3 Experimental Methods

3.1 Outline of Experiment

Experiments were conducted to evaluate the performance of the new end effector. In the experiment, the subjects tried to grasp a bottle of ketchup and draw the figure of an omelet in the VR environment. In accordance with the input of pressure from the end effector, the experimenter distinguished between three states: (1) not gripping the ketchup, (2) only lifting the ketchup and (3) squeezing the ketchup bottle. In this experiment, the degree of coincidence between operations performed by the subjects and the operations simulated in VR was evaluated. We found a high degree of coincidence between grasp operations performed by the subject and grasp operations reproduced in the VR environment. That is to say that the self-projectability between grasp operations was evaluated to be higher.

In a second experiment (Fig. 6), a “rectangle”, “star”, and “character (TUS)” were used as guidelines (Fig. 6), and each trial was performed twice. In each trial, the subject lifted the ketchup bottle, performed a grasp operation while moving the ketchup bottle in order to trace the guideline. For the simplicity of the simulation, spilt ketchup was represented by individual drops falling according to the grasping strength. At this time, the subject was instructed to “operate as quickly and accurately as possible, without making gaps, do not turn back, without letting go”. A sufficient amount of time to practice the task was given to each user in order to get used to the grasping operation, and only afterward was the task started.

Fig. 6.
figure 6

Example of results.

Full size image

Software developing tools, Unity 2017.3 and Arduino IDE were used for simulation development.

3.2 System Configuration

Subjects were asked to perform experiments with the HMD (HTC Vive) and SPIDAR-W. The HMD and the end effector each track 6 DoF, but since their respective coordinate systems are independent, we allowed subjects to work facing forward.

The control board of the SPIDAR-W, HMD, and Arduino Uno, with the pressure sensor, were connected to the PC, respectively, and each device was controlled with an application developed by Unity. Serial communication was used for communication between Arduino Uno and Unity. Figure 7 shows a simulation screen of the experiment projected onto the subject’s HMD.

Fig. 7.
figure 7

Work space presented in the HMD.

Full size image

A conventional end effector was used for the left hand of the SPIDAR-W while the new end effector was used for the right hand. Calibration of end effector position and bottle position in VR space as well as the end of the measurement working time were performed by button input for the end effector on the left hand. The input value of the pressure sensor was handled by using 1024 A/D converted values. This was converted to a scale of 0 to 1: (1) a state of releasing from 0 to 0.05, (2) a state of grasping from 0.05 to 0.5 and (3) a state of squeezing from 0.5 to 1.

3.3 Measurement Item

In each experimental trial, a guideline was displayed first, the end effector was brought to the home position and a calibration was performed. After the bottle was lifted, the operation started, the subjects traced the guidelines as instructed and the distance of each drop was measured.

Items measured were the distance between each droplet and the guideline and the number of times the bottle was dropped.

The distance between the drop and the guideline was measured with an accuracy of half of the drop radius, and the distance was recorded for each trial.

From each data measurement, voluntary operation was confirmed with bottle error as bottle error and drop error of an unintentional drop as drop error.

4 Results and Discussion

4.1 Bottle Error

When performing the task, the user was instructed to do “without releasing the bottle.” Therefore, dropping the bottle during the task, i.e., the input was “not gripped,” is evaluated as an unintended input by the user.

The number of times the subject dropped the bottle is shown in Table 1 below.

Table 1. Bottle error count for each trial
Full size table

Table 1 shows that bottle removal is occurring throughout the experiment. Since the maximum number of dropouts is one per trial, the user may be accustomed to the sensation of dropping, or they may be cautious about sensor handling. Unintended input could be made with regard to bottle error, since a maximum of 3 dropouts per person occurred in 6 trials.

Bottle error occurs between the states of “non-grasping” and “grasping” at the input of the pressure sensor and it is therefore necessary to review the threshold value. By establishing a threshold value, it is possible to determine a method to measure the actual gripping strength.

4.2 Drop Error

When judging the accuracy of the traced guidelines, if the path drawn by the user deviates from the guideline task has not been completed. However, it is not possible to judge whether or not the subject failed due to an unintended input or some other reason. Contrarily, when a droplet is significantly deviated from the guideline, we can determined that grip input corresponds to an action or timing not intended by the user. It is due to the fact that the subject is required to trace only the guideline composed of lines.

For each trial, the drop distance furthest from the subject’s trajectory is summarized in Table 2 in units of measurement accuracy.

Table 2. Drop error distance for each trial.
Full size table

The larger the value of the drop error, the more drops that are placed either singly or multiply, indicating that they were dropped at a position distant from the trajectory drawn by the subject.

An example of when a large drop error is occurring is shown in Fig. 8.

Fig. 8.
figure 8

Drop error example

Full size image

In Table 2, drop errors as large as a drop diameter of 2 or 3 are recorded, enough to conclude that drop generation is unintended by the subject.

These drop errors occur between “grasping” and “squeezing” at the input of the pressure sensor, and it is therefore important to review the threshold value for bottle error.

4.3 Questionnaire About Device

In a questionnaire about the device, several suggestions were made on the simulation of the sense of the grasping force and the responsiveness of the pressure sensor.

In response to the question “Which figure was the most difficult and why?”, 4 out of 5 answered the star and 1 person said the character. The users who chose the star mentioned that the “figures were too complex such as many sharp corners” and “the figure was relatively small.” The user who chose character stated that “It was easy because there are many straight lines in other figures”. The character is thought to be the most difficult figure because the intermediate grasp operation, “lifting but not squeezing,” is required unlike the other two guidelines. However, from the questionnaire results, it can be inferred that this operation was relatively easy. From these results, we consider responsiveness of the pressure sensor was calibrated relatively accurately.

For the next four items, we conducted a numerical evaluation in 7 stages (1–7). The response, “Whether it felt like the actual feeling of gripping ketchup,” was evaluated at 3.2, with the actual gripping feeling not yet reproduced. The response, “Whether the gripping motion could be done naturally”, was evaluated at 5.6, which corresponds to the idea that it was relatively intuitive to handle the pressure sensor. The responses, “I was able to put ketchup in the position I expected” even though “I was not able to put out the amount of ketchup as expected”, was evaluated at 4.4, it was neither good nor bad.

Some impressions were also seen in impressions on tasks and devices. Regarding the opinion that many ketchup bottles obtained were “deformed when the actual ketchup was depressed, but not punctured in the experiment,” the visually ketchup bottle is deformed, while the force surface does not deform the shape of the device surface Therefore, it seems that there was a difference between force and visual sense modality. Since this leads to a decrease in self-projectability, it can be said that a function of feeding back the deformation of the object is also necessary.

There are multiple opinions that “Learning by experience and learning in the second half was easy” Although this suggests that it can be handled intuitively if accustomed to a certain extent, it can not be handled intuitively from the actual operation at the first touch Respectively.

5 Conclusion and Prospects

5.1 Conclusion

The purpose of this study, in addition to the introduction of the SPIDAR - W wearable force environment, was to increase the immersional degree of grasp operations. In the newly developed end effector, it is now possible to simulate changes in the grasp operation with natural actions, such as fixing the hand using the belt or inputting arbitrary grip conditions via the pressure sensor.

Both bottle and drop errors occurred in the experiments, thus it is necessary to make adjustments to the pressure sensor and equipment used so that more intuitive operations can be performed using threshold settings of “non-grasping”, “grasping.” Despite numerous errors, the newly, developed end effector is capable of replicating intuitive grasping operations, which raised self-projectability related to the sense of grasp. Furthermore, despite various improvements, we found the existence of problems during the precise replication of grasp operations and the representation of grasp forces.

5.2 Prospect

Many improvements were observed in the grasp operation end effector developed in this study.

First, excluded for the sake of simplicity, the addition of a force parameter representing the shape, size, softness and deformation of an object can be used. Although there are limitations on the methods that can be used due to weight and volume relationships, in terms of precisely reproducing the grasp force, it is a task that needs to be evaluated eventually.

In this study, we conducted experiments under the assumption that no error occurs in the actual work. However, even if the same task is done in reality, there is a possibility of the occurrence of a poor degree of agreement with the guideline as well as bottle and drop error. In terms of strictly reproducing real grasp operations, we believe that it is necessary to perform the same task in real life, to be able to compare to simulated scenarios and evaluate the similarities and differences.

The threshold used for input discrimination of “non-grasping”, “grasping” and “squeezing” in the pressure sensor was arbitrarily applied, and thus, could be a source of bottle and drop error, indicated by the results. Accurate grasp operation simulation is accomplished by measuring the grip force when a human actually holds a ketchup bottle as well as the grip force when pushing ketchup out of the bottle and setting the same pressure as the threshold. However, in the current end effector since the pressure sensor does not deform by grasping, the relationship between deformation of the actual bottle and pressure is not expressible. It is therefore necessary to examine threshold values that can be operated in the most intuitive manner possible.

The present device only senses pressure after it is touched by a gripped object, “coincidence between sensory modalities of hand movement” for self-projection properties are not satisfied. In order to satisfy this condition, visual information, for example the degree of bending of a finger, acquired by the use of proximity sensor is needed. The “coincidence between sensory modalities of hand position/posture”, on a larger scale, corresponds to the need of matching the coordinate system of HMD and SPIDAR - W as well as to always draw the hand object at the actual hand position. A conceivable solution, such as incorporating SPIDAR-W into a general-purpose tracking device, can give the SPIDAR-W the ability to estimate its own position, which can be used in the HMD.

In order to simplify the simulation, we did not visually correct the bottle grasping. In the current SPIDAR - W system, you can grab things when the drawn hand overlaps with the object. This is different from the actual movement. Visual information indicating that the hand is buried in a bottle and does not actually penetrate is one factor that lowers the self-projection property. In order to improve this, a contact determination needs to be added to the hand object itself after reflecting the actual position, the posture of the hand and the degree of curvature of the finger in the drawing.