short-paper

Tactile Board: A Multimodal Augmentative and Alternative Communication Device for Individuals with Deafblindness

Authors:

Nils-Krister Persson,

Oliver KornAuthors Info & Claims

MUM '20: Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia

Pages 223 - 228

https://doi.org/10.1145/3428361.3428465

Published: 22 November 2020 Publication History

Abstract

Deafblindness, also known as dual sensory loss, is the combination of sight and hearing impairments of such extent that it becomes difficult for one sense to compensate for the other. Communication issues are a key concern for the Deafblind community. We present the design and technical implementation of the Tactile Board: a mobile Augmentative and Alternative Communication (AAC) device for individuals with deafblindness. The Tactile Board allows text and speech to be translated into vibrotactile signs that are displayed real-time to the user via a haptic wearable. Our aim is to facilitate communication for the deafblind community, creating opportunities for these individuals to initiate and engage in social interactions with other people without the direct need of an intervener.

Supplementary Material

MP4 File (p223-theil-video.mp4)

Download
178.01 MB

References

[1]

Ali Israr and Ivan Poupyrev. 2011. Tactile brush: drawing on skin with a tactile grid display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2019–2028.

Digital Library

[2]

Ali Israr, Seung-Chan Kim, Jan Stec, and Ivan Poupyrev. 2012. Surround haptics: tactile feedback for immersive gaming experiences. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’12). Association for Computing Machinery, New York, NY, USA, 1087–1090.

[3]

Arthur Theil, Lea Buchweitz, Mauricio Fuentes, and Oliver Korn. 2020. Co-Designing Assistive Tools to Support Social Interactions by Individuals Living with Deafblindness. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference (DIS’ 20 Companion). Association for Computing Machinery, New York, NY, USA, 79–83.

Digital Library

[4]

Caleb Southern, James Clawson, Brian Frey, Gregory Abowd, and Mario Romero. 2012. An evaluation of BrailleTouch: mobile touchscreen text entry for the visually impaired. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services (MobileHCI ’12). Association for Computing Machinery, New York, NY, USA, 317–326.

Digital Library

[5]

Chandrika Jayant, Christine Acuario, William Johnson, Janet Hollier, and Richard Ladner. 2010. V-braille: haptic braille perception using a touch-screen and vibration on mobile phones. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility (ASSETS ’10). Association for Computing Machinery, New York, NY, USA, 295–296.

Digital Library

[6]

Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 16–31.

Digital Library

[7]

E. Piateski and L. Jones. 2005. Vibrotactile pattern recognition on the arm and torso. In First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference. IEEE, Pisa, Italy, 90-95.

[8]

European Deafblind Network. 2014. Mapping Opportunities for Deafblind People Across Europe. Indicators of Deafblindness Equality in Europe.

[9]

Eva Lindell, Arthur Theil, Li Guo, Nasrine Olsson, Oliver Korn, Nils-Krister Persson. 2021. Physical Add-Ons for Haptic Human-Surrounding Interaction and Sensorial Augmentation. In Ahram T., Taiar R., Langlois K., Choplin A. (eds) Human Interaction, Emerging Technologies and Future Applications III. IHIET 2020. Advances in Intelligent Systems and Computing. Springer, Cham. 1253, 183-188.

[10]

F. Ramirez-Garibay, C. M. Olivarria, A. F. E. Aguilera and J. C. Huegel. 2014. MyVox—Device for the communication between people: blind, deaf, deaf-blind and unimpaired. In IEEE Global Humanitarian Technology Conference (GHTC 2014). IEEE, San Jose, CA, 506-509.

[11]

Francesca Sorgini, Renato Caliò, Maria Chiara Carrozza, Calogero Maria Oddo. 2018. Haptic-assistive technologies for audition and vision sensory disabilities. Disability and Rehabilitation: Assistive Technology. 13, 4: 394-421.

[12]

Gijs Huismann, Aduén Darriba Frederiks, Betsy Van Dijk, Dirk Heylen. 2013. The TaSST – Tactile Sleeve for Social Touch. In 2013 World Haptics Conference (WHC). IEEE, Piscataway, NJ, 211-216.

[13]

Hendrik Richter, Benedikt Blaha, Alexander Wiethoff, Dominikus Baur, and Andreas Butz. 2011. Tactile feedback without a big fuss: simple actuators for high-resolution phantom sensations. In Proceedings of the 13th international conference on Ubiquitous computing (UbiComp’11). Association for Computing Machinery, New York, NY, USA, 85–88.

Digital Library

[14]

Hiroshi Kato, Yuki Hashimoto, and Hiroyuki Kajimoto. 2010. Basic properties of phantom sensation for practical haptic applications. In Proceedings of the 2010 international conference on Haptics: generating and perceiving tangible sensations, Part I (EuroHaptics’10). Springer-Verlag, Berlin, 271–278.

Digital Library

[15]

Idin Karuei, Karon E. MacLean, Zoltan Foley-Fisher, Russell MacKenzie, Sebastian Koch, and Mohamed El-Zohairy. 2011. Detecting vibrations across the body in mobile contexts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). Association for Computing Machinery, New York, NY, USA, 3267–3276.

Digital Library

[16]

James Gay, Moritz Umfahrer, Arthur Theil, Lea Buchweitz, Eva Lindell, Li Guo, Nils-Krister Persson, and Oliver Korn. 2020. Keep Your Distance: A Playful Haptic Navigation Wearable for Individuals with Deafblindness. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’20), October 26–28, 2020, Virtual Event, Greece. ACM, New York, NY, USA.

Digital Library

[17]

Jani Heikkinen, Thomas Olsson, and Kaisa Väänänen-Vainio-Mattila. 2009. Expectations for user experience in haptic communication with mobile devices. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’09). Association for Computing Machinery, New York, NY, USA, Article 28, 1–10.

Digital Library

[18]

Janice Light and David McNaughton. 2014. Communicative Competence for Individuals who require Augmentative and Alternative Communication: A New Definition for a New Era of Communication? Augmentative and Alternative Communication. 30, 1: 1-18.

[19]

Janice Light, David McNaughton, David Beukelman, Susan Koch Fager, Melanie Fried-Oken, Thomas Jakobs, Erik Jakobs. 2019. Challenges and opportunities in augmentative and alternative communication: Research and technology development to enhance communication and participation for individuals with complex communication needs. Augmentative and Alternative Communication. 35, 1: 1-12.

[20]

Jesper Dammeyer, Anja Nielsen, Emilie Strøm, Ola Hendar and Valgerður Kristín Eiríksdóttir. 2015. A Case Study of Tactile Language and its Possible Structure: A Tentative Outline to Study Tactile Language Systems among Children with Congenital Deafblindness. Communication Disorders, Deaf Studies & Hearing Aids. 3, 2: 133.

[21]

Kimberly Johanna Schelle, Carolina Gomez Naranjo, Martijn ten Bhömer, Oscar Tomico, and Stephan Wensveen. 2015. Tactile Dialogues: Personalization of Vibrotactile Behavior to Trigger Interpersonal Communication. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’15). Association for Computing Machinery, New York, NY, USA, 637–642.

Digital Library

[22]

Lea Buchweitz, Arthur Theil, James Gay, and Oliver Korn. 2020. Exploring Low-Cost Materials to Make Pattern-Based Lock-Screens Accessible for Users with Visual Impairments or Deafblindness. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’20), October 26–28, 2020, Virtual Event, Greece. ACM, New York, NY, USA.

Digital Library

[23]

L. A. Jones, M. Nakamura and B. Lockyer. 2004. Development of a tactile vest. In Proceedings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS '04). IEEE, Chicago, IL, USA, 82-89.

[24]

Manoj Prasad, Murat Russell, and Tracy A. Hammond. 2014. Designing vibrotactile codes to communicate verb phrases. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1, Article 11 (September 2014), 21 pages.

Digital Library

[25]

Mario Enriquez, Karon MacLean, and Christian Chita. 2006. Haptic phonemes: basic building blocks of haptic communication. In Proceedings of the 8th international conference on Multimodal interfaces (ICMI ’06). Association for Computing Machinery, New York, NY, USA, 302–309.

Digital Library

[26]

Marion Hersh. 2013. Deafblind People, Communication, Independence, and Isolation. Journal of Deaf Studies and Deaf Education. 18, 4: 446-463.

[27]

Marion Hersh. 2013. Deafblind people, stigma and the use of communication and mobility assistive devices. Technology and Disablity. 25, 4: 245-261.

[28]

Marlon Twyman, Joe Mullenbach, Craig Shultz, J. Edward Colgate, and Anne Marie Piper. 2015. Designing Wearable Haptic Information Displays for People with Vision Impairments. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’15). Association for Computing Machinery, New York, NY, USA, 341–344.

Digital Library

[29]

Michael T. Clarke, Gloria Soto, Keith Nelson. 2017. Language learning, recasts, and interaction involving AAC: background and potential for intervention. Augmentative and Alternative Communication. 33, 1: 42-50.

[30]

Myrthe A. Plaisier, Daphne S. Vermeer, and Astrid M. L. Kappers. 2020. Learning the Vibrotactile Morse Code Alphabet. ACM Trans. Appl. Percept.17, 3, Article 9 (August 2020).

Digital Library

[31]

Nicholas Caporusso, Luigi Biasi, Giovanni Cinquepalmi, Gianpaolo Francesco Trotta, Antonio Brunetti, Vitoantonio Bevilacqua. 2018. A Wearable Device Supporting Multiple Touch- and Gesture-Based Languages for the Deaf-Blind. In Ahram T., Falcão C. (eds) Advances in Human Factors in Wearable Technologies and Game Design. AHFE 2017. Advances in Intelligent Systems and Computing, 608. Springer, Cham.

[32]

Oliver Korn, James Gay, Rúben Gouveia, Lea Buchweitz, Annika Sabrina Schulz, and Moritz Umfahrer. 2020. Tactile navigation with checkpoints as progress indicators? only when walking longer straight paths. In Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '20). Association for Computing Machinery, New York, NY, USA, Article 32, 1–8.

Digital Library

[33]

Oliver Korn, Raymond Holt, Efstratios Kontopoulos, Astrid M.L. Kappers, Nils-Krister Persson, and Nasrine Olson. 2018. Empowering Persons with Deafblindness: Designing an Intelligent Assistive Wearable in the SUITCEYES Project. In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference (PETRA ’18). ACM, NY, USA, 545–551.

Digital Library

[34]

O. Ozioko, P. Karipoth, M. Hersh and R. Dahiya. 2020. Wearable Assistive Tactile Communication Interface Based on Integrated Touch Sensors and Actuators. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 28, 6: 1344-1352.

[35]

O. Ozioko, W. Taube, M. Hersh and R. Dahiya. 2017. SmartFingerBraille: A tactile sensing and actuation based communication glove for deafblind people. In IEEE 26th International Symposium on Industrial Electronics (ISIE). IEEE, Edinburgh, 2014-2018.

[36]

Peter B. Shull and Dana D. Damian. 2015. Haptic wearables as sensory replacement, sensory augmentation and trainer - a review. Journal of NeuroEngineering and Rehabilitation. 12, 59.

[37]

Pia Hesse and Gerd Nielsen. 2018. Haptic Signals – 139 new and known signals. The Danish DeafBlind Association.

[38]

Riitta Lahtinen and Russ Palmer. 2008. Haptices and Haptemes – Environmental Information through touch. In Proceedings of the 3 rd International Haptic and Auditory Interaction Design Workshop.

[39]

Roger W. Cholewiak and James C. Craig. 1984. Vibrotactile pattern recognition and discrimination at several body sites. Perception & Psychophysics. 35: 503-5014.

[40]

Sándor Darányi, Nasrine Olson, Eva Lindell, Nils-Krister Persson, Marina Riga, Efstratios Kontopoulos, Ioannis Kompatsiaris. 2020. Communicating Semantic Content to Persons with Deafblindness by Haptograms and Smart Textiles: Theoretical Approach and Methodology. International Journal on Advances in Intelligent Systems (IARIA). 13, 1&2: 103-113.

[41]

S. Ohtsuka, N. Sasaki, S. Hasegawa and T. Harakawa, "Helen Keller Phone—a communication system for deaf-blind people using Body-Braille and Skype. 2012. In IEEE Consumer Communications and Networking Conference (CCNC). IEEE, Las Vegas, NV, 30-31.

[42]

S. Ohtsuka, S. Hasegawa, N. Sasaki and T. Harakawa. 2010. Communication System between Deaf-Blind People and Non-Disabled People Using Body-Braille and Infrared Communication. In7th IEEE Consumer Communications and Networking Conference. IEEE, Las Vegas, NV, 1-2.

[43]

Thuong N. Hoang, Hasan S. Ferdous, Frank Vetere, and Martin Reinoso. 2018. Body as a Canvas: An Exploration on the Role of the Body as Display of Digital Information. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS ’18). Association for Computing Machinery, New York, NY, USA, 253–263.

Digital Library

[44]

Viirj Kan, Katsuya Fujii, Judith Amores, Chang Long Zhu Jin, Pattie Maes, and Hiroshi Ishii. 2015. Social Textiles: Social Affordances and Icebreaking Interactions Through Wearable Social Messaging. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’15). Association for Computing Machinery, New York, NY, USA, 619–624.

Digital Library

[45]

V. Saikiran and A. Deepak. 2017. A Braille Based Communication and Translation Glove Assistance for Deaf Blind People. International Journal of Innovative Science and Research Technology. 2, 9: 227-230.

[46]

Y. Matsuda, I. Sakuma, Y. Jimbo, E. Kobayashi, T. Arafune and T. Isomura. 2008. Finger Braille recognition system for people who communicate with deafblind people. In IEEE International Conference on Mechatronics and Automation. IEEE, Takamatsu, 268-273.

Cited By

Curtis HNeate T(2024)Beyond Repairing with Electronic Speech: Towards Embodied Communication and Assistive TechnologyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642274(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642274
Uribe-Quevedo A(2024)Immersive Technologies for Accessible User ExperiencesEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_449(914-921)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_449
Theil AAnderton CCreed COlson NHolt RSarcar S(2023)Accessibility Research and Users with Multiple Disabilities or Complex NeedsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3615651(1-6)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3615651
Show More Cited By

Recommendations

An augmentative and alternative communication tool for children and adolescents with cerebral palsy
Medical teamwork, collaboration and patient-centred care

People with disabilities face many accessibility problems in their daily lives. One such group of disabled people, children who suffer from cerebral palsy, faces specific challenges, which arise due to motor dysfunction and communication disorders. In ...
Developing an augmentative mobile communication system
ICCHP'12: Proceedings of the 13th international conference on Computers Helping People with Special Needs - Volume Part II

The widespread use of smartphones and the inclusion of new tech-nologies as Near Field Communications (NFC) in the mobile devices offer a chance to turn the classic Augmentative and Alternative Communication (AAC) boards into Hi-Tech AAC systems with ...
Chatting with an AAC (augmentative/alternative communication device)
UAHCI'07: Proceedings of the 4th international conference on Universal access in human-computer interaction: applications and services

Communicating with someone using an electronic communication device is different from routine conversation. Despite enormous technological progress, talking with a person with a disability using an electronic communicater is an acquired skill. Very few ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

MUM '20: Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia

November 2020

353 pages

ISBN:9781450388702

DOI:10.1145/3428361

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Horizon 2020

Conference

MUM 2020

MUM 2020: 19th International Conference on Mobile and Ubiquitous Multimedia

November 22 - 25, 2020

Essen, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
392
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)12

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Curtis HNeate T(2024)Beyond Repairing with Electronic Speech: Towards Embodied Communication and Assistive TechnologyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642274(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642274
Uribe-Quevedo A(2024)Immersive Technologies for Accessible User ExperiencesEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_449(914-921)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_449
Theil AAnderton CCreed COlson NHolt RSarcar S(2023)Accessibility Research and Users with Multiple Disabilities or Complex NeedsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3615651(1-6)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3615651
Vihriälä TRaisamo RIhalainen TVirkki J(2023)Towards E-textiles in augmentative and alternative communication – user scenarios developed by speech and language therapistsDisability and Rehabilitation: Assistive Technology10.1080/17483107.2023.222555619:4(1626-1636)Online publication date: 4-Jul-2023
https://doi.org/10.1080/17483107.2023.2225556
Smolin ADidevich JDzhumagulova ABalkanskii ASpiridonova ASopronenko LMagina YVolosiuk A(2023)“Neurotafl” Software and Hardware System for Inclusive EducationProceedings of the 9th International Ergonomics Conference10.1007/978-3-031-33986-8_32(293-304)Online publication date: 20-Jul-2023
https://doi.org/10.1007/978-3-031-33986-8_32
Zdravkova KKrasniqi VDalipi FFerati M(2022)Cutting-edge communication and learning assistive technologies for disabled children: An artificial intelligence perspectiveFrontiers in Artificial Intelligence10.3389/frai.2022.9704305Online publication date: 28-Oct-2022
https://doi.org/10.3389/frai.2022.970430
Curtis HNeate TVazquez Gonzalez C(2022)State of the Art in AAC: A Systematic Review and TaxonomyProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544810(1-22)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544810
Dinulescu STummala NReardon GDandu BGoetz DTopp SVisell Y(2022)A Smart Bracelet Supporting Tactile Communication and Interaction2022 IEEE Haptics Symposium (HAPTICS)10.1109/HAPTICS52432.2022.9765616(1-7)Online publication date: 21-Mar-2022
https://doi.org/10.1109/HAPTICS52432.2022.9765616
Vihriӓlӓ TIhalainen TElo CLintula LVirkki J(2022)Possibilities of intelligent textiles in AAC – perspectives of speech and language therapistsDisability and Rehabilitation: Assistive Technology10.1080/17483107.2022.214190019:3(1019-1031)Online publication date: 13-Nov-2022
https://doi.org/10.1080/17483107.2022.2141900
Uribe-Quevedo A(2022)Immersive Technologies for Accessible User ExperiencesEncyclopedia of Computer Graphics and Games10.1007/978-3-319-08234-9_449-1(1-9)Online publication date: 25-Feb-2022
https://doi.org/10.1007/978-3-319-08234-9_449-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents