research-article

VisualTouch: Enhancing Affective Touch Communication with Multi-modality Stimulation

Authors:

Zhuoming Zhang,

Eric Lecolinet,

Françoise Detienne,

Stéphane SafinAuthors Info & Claims

ICMI '19: 2019 International Conference on Multimodal Interaction

Pages 114 - 123

https://doi.org/10.1145/3340555.3353733

Published: 14 October 2019 Publication History

Abstract

As one of the most important non-verbal communication channel, touch plays an essential role in interpersonal affective communication. Although some researchers have started exploring the possibility of using wearable devices for conveying emotional information, most of the existing devices still lack the capability to support affective and dynamic touch in interaction. In this paper, we explore the effect of dynamic visual cues on the emotional perception of vibrotactile signals. For this purpose, we developed VisualTouch, a haptic sleeve consisting of a haptic layer and a visual layer. We hypothesized that visual cues would enhance the interpretation of tactile cues when both types of cues are congruent. We first carried out an experiment and selected 4 stimuli producing substantially different responses. Based on that, a second experiment was conducted with 12 participants rating the valence and arousal of 36 stimuli using SAM scales.

References

[1]

Michael Argyle. 1972. Non-verbal communication in human social interaction.(1972).

[2]

Alan Baddeley. 1992. Working memory. Science 255, 5044 (1992), 556–559.

[3]

Bocoup. 2012. Johnny-Five: the JavaScript Robotics and IoT Platform. http://johnny-five.io/, Last accessed on 2019-05-06.

[4]

Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1(1994), 49–59.

[5]

John NI Dieter, Tiffany Field, Maria Hernandez-Reif, Eugene K Emory, and Mercedes Redzepi. 2003. Stable preterm infants gain more weight and sleep less after five days of massage therapy. Journal of pediatric psychology 28, 6 (2003), 403–411.

[6]

Jon Driver and Charles Spence. 2000. Multisensory perception: beyond modularity and convergence. Current biology 10, 20 (2000), R731–R735.

[7]

Cynthia N DuBose, Armand V Cardello, and Owen Maller. 1980. Effects of colorants and flavorants on identification, perceived flavor intensity, and hedonic quality of fruit-flavored beverages and cake. Journal of Food Science 45, 5 (1980), 1393–1399.

[8]

Jacki Eads, G Lorimer Moseley, and Susan Hillier. 2015. Non-informative vision enhances tactile acuity: a systematic review and meta-analysis. Neuropsychologia 75(2015), 179–185.

[9]

Jan Flusser, Barbara Zitova, and Tomas Suk. 2009. Moments and moment invariants in pattern recognition. John Wiley & Sons.

[10]

Alberto Gallace and Charles Spence. 2010. The science of interpersonal touch: an overview. Neuroscience & Biobehavioral Reviews 34, 2 (2010), 246–259.

[11]

Yuan Gao, Nadia Bianchi-Berthouze, and Hongying Meng. 2012. What does touch tell us about emotions in touchscreen-based gameplay?ACM Transactions on Computer-Human Interaction (TOCHI) 19, 4(2012), 31.

[12]

Gilbert Gottlieb. 1971. Ontogenesis of sensory function in birds and mammals. The biopsychology of development(1971), 67–128.

[13]

Jess Hartcher-O’Brien, Alberto Gallace, Benedikt Krings, Camille Koppen, and Charles Spence. 2008. When vision ‘extinguishes’ touch in neurologically-normal people: extending the Colavita visual dominance effect. Experimental brain research 186, 4 (2008), 643–658.

[14]

Y Hasegawa, M Shikida, D Ogura, and K Sato. 2007. Novel type of fabric tactile sensor made from artificial hollow fiber. In 2007 IEEE 20th International Conference on Micro Electro Mechanical Systems (MEMS). IEEE, 603–606.

[15]

Matthew J Hertenstein, Rachel Holmes, Margaret McCullough, and Dacher Keltner. 2009. The communication of emotion via touch.Emotion 9, 4 (2009), 566.

[16]

Matthew J Hertenstein, Dacher Keltner, Betsy App, Brittany A Bulleit, and Ariane R Jaskolka. 2006. Touch communicates distinct emotions.Emotion 6, 3 (2006), 528.

[17]

Gijs Huisman, Aduen Darriba Frederiks, Betsy Van Dijk, Dirk Hevlen, and Ben Kröse. 2013. The TaSSt: Tactile sleeve for social touch. In 2013 World Haptics Conference (WHC). IEEE, 211–216.

[18]

Masayuki Inaba, Yukiko Hoshino, Kenichiro Nagasaka, Tetsuo Ninomiya, Satoshi Kagami, and Hirochika Inoue. 1996. A full-body tactile sensor suit using electrically conductive fabric and strings. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS’96, Vol. 2. IEEE, 450–457.

[19]

Stanley E Jones and A Elaine Yarbrough. 1985. A naturalistic study of the meanings of touch. Communications Monographs 52, 1 (1985), 19–56.

[20]

Veikko Jousmäki and Riitta Hari. 1998. Parchment-skin illusion: sound-biased touch. Current biology 8, 6 (1998), R190–R191.

[21]

Les Kertay and Susan L Reviere. 1998. Touch in context. Touch in psychotherapy: Theory, research, and practice (1998), 16–35.

[22]

Roberta L Klatzky, Susan J Lederman, and Catherine Reed. 1987. There’s more to touch than meets the eye: The salience of object attributes for haptics with and without vision.Journal of experimental psychology: general 116, 4 (1987), 356.

[23]

Mark L Knapp, Judith A Hall, and Terrence G Horgan. 2013. Nonverbal communication in human interaction. Cengage Learning.

[24]

Shinobu Kuroki, Hiroyuki Kajimoto, Hideaki Nii, Naoki Kawakami, and Susumu Tachi. 2007. Proposal for tactile sense presentation that combines electrical and mechanical stimulus. In Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07). IEEE, 121–126.

Digital Library

[25]

Robert W Lindeman, Robert Page, Yasuyuki Yanagida, and John L Sibert. 2004. Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system. In Proceedings of the ACM symposium on Virtual reality software and technology. ACM, 146–149.

Digital Library

[26]

Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating physical impact by combining tactile stimulation with electrical muscle stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 11–19.

Digital Library

[27]

Emiliano Macaluso and Angelo Maravita. 2010. The representation of space near the body through touch and vision. Neuropsychologia 48, 3 (2010), 782–795.

[28]

Flavia Mancini, Armando Bauleo, Jonathan Cole, Fausta Lui, Carlo A Porro, Patrick Haggard, and Gian Domenico Iannetti. 2014. Whole-body mapping of spatial acuity for pain and touch. Annals of neurology 75, 6 (2014), 917–924.

[29]

Haemy Lee Masson and Hans Op de Beeck. 2018. Socio-affective touch expression database. PloS one 13, 1 (2018), e0190921.

[30]

Roger Newport, Benjamin Rabb, and Stephen R Jackson. 2002. Noninformative vision improves haptic spatial perception. Current Biology 12, 19 (2002), 1661–1664.

[31]

Jose Ochoa and Erik Torebjörk. 1983. Sensations evoked by intraneural microstimulation of single mechanoreceptor units innervating the human hand.The Journal of physiology 342, 1 (1983), 633–654.

[32]

Young-Woo Park, Kyoung-Min Baek, and Tek-Jin Nam. 2013. The roles of touch during phone conversations: long-distance couples’ use of POKE in their homes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1679–1688.

Digital Library

[33]

Miles L Patterson, Jack L Powell, and Mary G Lenihan. 1986. Touch, compliance, and interpersonal affect. Journal of Nonverbal behavior 10, 1 (1986), 41–50.

[34]

Betina Piqueras-Fiszman and Charles Spence. 2012. The influence of the color of the cup on consumers’ perception of a hot beverage. Journal of Sensory Studies 27, 5 (2012), 324–331.

[35]

Vilayanur S Ramachandran and Diane Rogers-Ramachandran. 1996. Synaesthesia in phantom limbs induced with mirrors. Proceedings of the Royal Society of London. Series B: Biological Sciences 263, 1369(1996), 377–386.

[36]

James A Russell. 1980. A circumplex model of affect.Journal of personality and social psychology 39, 6(1980), 1161.

[37]

Julia Simner and Vera U Ludwig. 2012. The color of touch: A case of tactile–visual synaesthesia. Neurocase 18, 2 (2012), 167–180.

[38]

Charles Spence, Carmel A Levitan, Maya U Shankar, and Massimiliano Zampini. 2010. Does food color influence taste and flavor perception in humans?Chemosensory Perception 3, 1 (2010), 68–84.

[39]

Paul Strohmeier, Jesse Burstyn, Juan Pablo Carrascal, Vincent Levesque, and Roel Vertegaal. 2016. Reflex: A flexible smartphone with active haptic feedback for bend input. In Proceedings of the TEI’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, 185–192.

Digital Library

[40]

Hyeon-Jeong SUK and Hans IRTEL. 2008. Emotional response to simple color stimuli. Kansei Engineering International 7, 2 (2008), 181–188.

[41]

Juulia T Suvilehto, Enrico Glerean, Robin IM Dunbar, Riitta Hari, and Lauri Nummenmaa. 2015. Topography of social touching depends on emotional bonds between humans. Proceedings of the National Academy of Sciences 112, 45(2015), 13811–13816.

[42]

Julie Wagner, Mathieu Nancel, Sean G Gustafson, Stephane Huot, and Wendy E Mackay. 2013. Body-centric design space for multi-surface interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1299–1308.

Digital Library

[43]

Lois B Wexner. 1954. The degree to which colors (hues) are associated with mood-tones.Journal of applied psychology 38, 6 (1954), 432.

[44]

Lisa Wilms and Daniel Oberfeld. 2018. Color and emotion: effects of hue, saturation, and brightness. Psychological research 82, 5 (2018), 896–914.

[45]

Graham Wilson and Stephen A Brewster. 2017. Multi-Moji: Combining Thermal, Vibrotactile & Visual Stimuli to Expand the Affective Range of Feedback. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1743–1755.

Digital Library

Cited By

An PZhu JZhang ZYin YMa QYan CDu LZhao J(2024)EmoWear: Exploring Emotional Teasers for Voice Message Interaction on SmartwatchesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642101(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642101
Chan AQuek FHowell JLiu T(2024)Closing the Gap Between Long Distance Couples via Asynchronous Remote Touch CommunicationSocial Computing and Social Media10.1007/978-3-031-61312-8_1(3-16)Online publication date: 31-May-2024
https://doi.org/10.1007/978-3-031-61312-8_1
Zhou RSareen HZhang YLeithinger D(2022)EmotiTactor: Exploring How Designers Approach Emotional Robotic TouchProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533487(1330-1344)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533487
Show More Cited By

Recommendations

Towards Pleasant Touch: Vibrotactile Grids for Social Touch Interactions
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

In this paper, we realize a wearable tactile device that delivers smooth pleasant strokes, those resemble of caressing and calming sensations, on the forearm. In Study 1, we develop a psychophysical model of continuous illusory motion on a discrete ...
The Experience of Social Touch in Multi-User Virtual Reality
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

We present user study results on virtual body contact experience in a two-user VR scenario, in which participants performed different touches with a research assistant. The interaction evoked different emotional reactions in perceived relaxation, ...
Survey and perspectives of social touch in HCI
IHM '17: Proceedings of the 29th Conference on l'Interaction Homme-Machine

Touch is one of the primary channel of non-verbal communication. It is used to convey emotions and to establish bounds between people. Its use has already been considered in HCI to interact with devices, but it has rarely been used for direct emotional ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '19: 2019 International Conference on Multimodal Interaction

October 2019

601 pages

ISBN:9781450368605

DOI:10.1145/3340555

Editors:
Wen Gao
Peking University, China
,
Helen Mei Ling Meng
Chinese University of Hong Kong, China
,
Matthew Turk
Toyota Technological Institute at Chicago, USA
,
Susan R. Fussell
Cornell University, USA
,
Björn Schuller
Imperial College London / University of Augsburg, UK
,
Yale Song
Microsoft Research, USA
,
Kai Yu
Shanghai Jiao Tong University, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Agence Nationale de la Recherche

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
476
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)2

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

An PZhu JZhang ZYin YMa QYan CDu LZhao J(2024)EmoWear: Exploring Emotional Teasers for Voice Message Interaction on SmartwatchesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642101(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642101
Chan AQuek FHowell JLiu T(2024)Closing the Gap Between Long Distance Couples via Asynchronous Remote Touch CommunicationSocial Computing and Social Media10.1007/978-3-031-61312-8_1(3-16)Online publication date: 31-May-2024
https://doi.org/10.1007/978-3-031-61312-8_1
Zhou RSareen HZhang YLeithinger D(2022)EmotiTactor: Exploring How Designers Approach Emotional Robotic TouchProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533487(1330-1344)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533487
Zhang ZAlvina JDétienne FLecolinet EBottoni PPanizzi E(2022)Pulling, Pressing, and Sensing with In-Flat: Transparent Touch Overlay for SmartphonesProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3531111(1-9)Online publication date: 6-Jun-2022
https://dl.acm.org/doi/10.1145/3531073.3531111
Günther SRasch JSchön DMüller FSchmitz MRiemann JMatviienko AMühlhäuser M(2022)Smooth as Steel Wool: Effects of Visual Stimuli on the Haptic Perception of Roughness in Virtual RealityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517454(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517454
An PZhou ZLiu QYin YDu LHuang DZhao J(2022)VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social CommunicationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501940(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501940
Price SBianchi-Berthouze NJewitt CYiannoutsou NFotopoulou KDajic SVirdee JZhao YAtkinson DBrudy F(2022)The Making of Meaning through Dyadic Haptic Affective TouchACM Transactions on Computer-Human Interaction10.1145/349049429:3(1-42)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.1145/3490494
Fairhurst MMcGlone FCroy I(2022)Affective touch: a communication channel for social exchangeCurrent Opinion in Behavioral Sciences10.1016/j.cobeha.2021.07.00743(54-61)Online publication date: Feb-2022
https://doi.org/10.1016/j.cobeha.2021.07.007
Zhang ZAlvina JHéron RSafin SDétienne FLecolinet EKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Touch without Touching: Overcoming Social Distancing in Semi-Intimate Relationships with SansTouchProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445612(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445612
Price SJewitt CYiannoutsou N(2021)Conceptualising touch in VRVirtual Reality10.1007/s10055-020-00494-y25:3(863-877)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1007/s10055-020-00494-y

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents