Effects of auditory, haptic and visual feedback on performing gestures by gaze or by hand

Modern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality auditory, haptic or visual and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements up, down, left and right. In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences.