Birdbox: Exploring the User Experience of Crossmodal, Multisensory Data Representations
Pages 12 - 21
Abstract
We contribute to an improved understanding of how physical multisensory data representations are experienced and how specific modalities affect the user experience (UX). We investigate how people make sense of Birdbox, a crossmodal data representation that employs combined haptic-audio, audio-visual, or visual-haptic output for data about birds. Findings indicate that participants preferred haptic output for the bodily experience it triggered. Participants further created their own mappings between data and modality; haptic was mapped to aggression, and audio to speed. Especially with (soft) haptic output, Birdbox was experienced as a living entity. This can also be seen in participants’ bodily interactions, holding Birdbox as if it were a small bird. We contribute to a better understanding of the UX of different modalities in multisensory data representations, highlight strengths of the haptic modality, and of metaphorical understandings of modalities.
References
[1]
Elaine N Aron, Arthur Aron, and Jadzia Jagiellowicz. 2012. Sensory processing sensitivity: a review in the light of the evolution of biological responsivity. Pers. Soc. Psychol. Rev. 16, 3 (Aug. 2012), 262–282.
[2]
Stephen Barrass. 2016. Diagnosing blood pressure with Acoustic Sonification singing bowls. International Journal of Human Computer Studies 85 (jan 2016), 68–71. https://doi.org/10.1016/j.ijhcs.2015.08.007
[3]
Alberto Boem, Kai Sasaki, and Shori Kano. 2017. Vital+Morph: A Shape-Changing Interface for Remote Biometric Monitoring. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (Yokohama, Japan) (TEI ’17). Association for Computing Machinery, New York, NY, USA, 503–509. https://doi.org/10.1145/3024969.3025083
[4]
Anna C Bolders, Mattie Tops, Guido P H Band, and Pieter Jan M Stallen. 2017. Perceptual sensitivity and response to strong stimuli are related. Front. Psychol. 8 (Sept. 2017), 1642.
[5]
Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597. https://doi.org/10.1080/2159676X.2019.1628806 arXiv:https://doi.org/10.1080/2159676X.2019.1628806
[6]
Stephen Brewster and Lorna M Brown. 2004. Tactons : Structured Tactile Messages for Non-Visual Information Display. In Proceedings of the Fifth Conference on Australasian User Interface. Australian Computer Society, Inc., AUS, 15–23. www.dcs.gla.ac.uk/$$stephen
[7]
Stephen Brewster, David McGookin, and Christopher Miller. 2006. Olfoto: Designing a Smell-Based Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 653–662. https://doi.org/10.1145/1124772.1124869
[8]
Stuart Card, Jock Mackinlay, and Ben Shneiderman. 1999. Readings in Information Visualization: Using Vision To Think. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[9]
Hao Chen, Arindam Dey, Mark Billinghurst, and Robert W. Lindeman. 2017. Exploring the Design Space for Multi-Sensory Heart Rate Feedback in Immersive Virtual Reality. In Proceedings of the 29th Australian Conference on Computer-Human Interaction (Brisbane, Queensland, Australia) (OZCHI ’17). Association for Computing Machinery, New York, NY, USA, 108–116. https://doi.org/10.1145/3152771.3152783
[10]
Erik Van der Burg, Christian N.L. Olivers, Adelbert W. Bronkhorst, and Jan Theeuwes. 2009. Poke and pop: Tactile–visual synchrony increases visual saliency. Neuroscience Letters 450, 1 (Jan. 2009), 60–64. https://doi.org/10.1016/j.neulet.2008.11.002
[11]
Erik Van der Burg, Christian N. L. Olivers, Adelbert W. Bronkhorst, and Jan Theeuwes. 2008. Pip and pop: Nonspatial auditory signals improve spatial visual search.Journal of Experimental Psychology: Human Perception and Performance 34, 5(2008), 1053–1065. https://doi.org/10.1037/0096-1523.34.5.1053
[12]
Feng Feng and Tony Stockman. 2017. An Investigation of Dynamic Crossmodal Instantiation in TUIs. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (Glasgow, UK) (ICMI ’17). Association for Computing Machinery, New York, NY, USA, 82–90. https://doi.org/10.1145/3136755.3136782
[13]
Daniel J. Finnegan, Eamonn O’Neill, and Michael J. Proulx. 2016. Compensating for Distance Compression in Audiovisual Virtual Environments Using Incongruence. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 200–212. https://doi.org/10.1145/2858036.2858065
[14]
Fritzsch, Bernd (Ed.). 2020. The senses: A comprehensive reference(2 ed.). Academic Press, San Diego, CA.
[15]
Jens C. Hegg, Jonathan Middleton, Ben Luca Robertson, and Brian P. Kennedy. 2018. The sound of migration: exploring data sonification as a means of interpreting multivariate salmon movement datasets. Heliyon 4, 2 (feb 2018), 532. https://doi.org/10.1016/J.HELIYON.2018.E00532
[16]
Trevor Hogan, Uta Hinrichs, and Eva Hornecker. 2017. The Visual and Beyond: Characterizing Experiences with Auditory, Haptic and Visual Data Representations. In Proceedings of the 2017 Conference on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association for Computing Machinery, New York, NY, USA, 797–809. https://doi.org/10.1145/3064663.3064702
[17]
Trevor Hogan and Eva Hornecker. 2012. How Does Representation Modality Affect User-Experience of Data Artifacts?. In Haptic and Audio Interaction Design, Charlotte Magnusson, Delphine Szymczak, and Stephen Brewster (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 141–151.
[18]
Trevor Hogan and Eva Hornecker. 2013. Blending the Repertory Grid Technique with Focus Groups to Reveal Rich Design Relevant Insight. In Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces (Newcastle upon Tyne, United Kingdom) (DPPI ’13). Association for Computing Machinery, New York, NY, USA, 116–125. https://doi.org/10.1145/2513506.2513519
[19]
Trevor Hogan and Eva Hornecker. 2013. In Touch with Space: Embodying Live Data for Tangible Interaction. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction(Barcelona, Spain) (TEI ’13). Association for Computing Machinery, New York, NY, USA, 275–278. https://doi.org/10.1145/2460625.2460671
[20]
Trevor Hogan and Eva Hornecker. 2016. Towards a Design Space for Multisensory Data Representation. Interacting with Computers 29, 2 (05 2016), 147–167. https://doi.org/10.1093/iwc/iww015 arXiv:https://academic.oup.com/iwc/article-pdf/29/2/147/10296429/iww015.pdf
[21]
Eve Hoggan and Stephen Brewster. 2007. Designing Audio and Tactile Crossmodal Icons for Mobile Devices. In Proceedings of the 9th International Conference on Multimodal Interfaces (Nagoya, Aichi, Japan) (ICMI ’07). Association for Computing Machinery, New York, NY, USA, 162–169. https://doi.org/10.1145/1322192.1322222
[22]
Eve Hoggan, Andrew Crossan, Stephen A. Brewster, and Topi Kaaresoja. 2009. Audio or Tactile Feedback: Which Modality When?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 2253–2256. https://doi.org/10.1145/1518701.1519045
[23]
Eve Hoggan, Topi Kaaresoja, Pauli Laitinen, and Stephen Brewster. 2008. Crossmodal Congruence: The Look, Feel and Sound of Touchscreen Widgets. In Proceedings of the 10th International Conference on Multimodal Interfaces (Chania, Crete, Greece) (ICMI ’08). Association for Computing Machinery, New York, NY, USA, 157–164. https://doi.org/10.1145/1452392.1452423
[24]
Kristina Höök. 2018. Designing with the Body - Somaesthetic Interaction Design. The MIT Press, Cambridge, MA. 272 pages. https://mitpress.mit.edu/books/designing-body
[25]
Michael Xuelin Huang, Will W. W. Tang, Kenneth W. K. Lo, C. K. Lau, Grace Ngai, and Stephen Chan. 2012. MelodicBrush: A Novel System for Cross-Modal Digital Art Creation Linking Calligraphy and Music. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New York, NY, USA, 418–427. https://doi.org/10.1145/2317956.2318018
[26]
Caroline Hummels, Kees C. Overbeeke, and Sietske Klooster. 2007. Move to Get Moved: A Search for Methods, Tools and Knowledge to Design for Expressive and Rich Movement-Based Interaction. Personal Ubiquitous Comput. 11, 8 (dec 2007), 677–690. https://doi.org/10.1007/s00779-006-0135-y
[27]
Samuel Huron, Yvonne Jansen, and Sheelagh Carpendale. 2014. Constructing Visual Representations: Investigating the Use of Tangible Tokens. IEEE Transactions on Visualization and Computer Graphics 20, 12(2014), 2102–2111. https://doi.org/10.1109/TVCG.2014.2346292
[28]
Yvonne Jansen, Pierre Dragicevic, and Jean-Daniel Fekete. 2013. Evaluating the Efficiency of Physical Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2593–2602. https://doi.org/10.1145/2470654.2481359
[29]
Yvonne Jansen, Pierre Dragicevic, Petra Isenberg, Jason Alexander, Abhijit Karnik, Johan Kildal, Sriram Subramanian, and Kasper Hornbæk. 2015. Opportunities and Challenges for Data Physicalization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3227–3236. https://doi.org/10.1145/2702123.2702180
[30]
Gregory Kramer. 1994. Auditory Display: Sonification, Audification, and Auditory Interfaces. Computer Music Journal 19, 2 (1994), 110. https://doi.org/10.2307/3680606
[31]
Astrid Twenebowa Larssen, Toni Robertson, and Jenny Edwards. 2006. How It Feels, Not Just How It Looks: When Bodies Interact with Technology. In Proceedings of the 18th Australia Conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (Sydney, Australia) (OZCHI ’06). Association for Computing Machinery, New York, NY, USA, 329–332. https://doi.org/10.1145/1228175.1228236
[32]
Kevin Lefeuvre, Soeren Totzauer, Michael Storz, Albrecht Kurze, Andreas Bischof, and Arne Berger. 2018. Bricks, Blocks, Boxes, Cubes, and Dice: On the Role of Cubic Shapes for the Design of Tangible Interactive Devices. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 485–496. https://doi.org/10.1145/3196709.3196768
[33]
Anan Lin, Meike Scheller, Feng Feng, Michael J Proulx, and Oussama Metatla. 2021. Feeling Colours: Crossmodal Correspondences Between Tangible 3D Objects, Colours and Emotions. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 187, 12 pages. https://doi.org/10.1145/3411764.3445373
[34]
Lian Loke and Toni Robertson. 2013. Moving and Making Strange: An Embodied Approach to Movement-Based Interaction Design. ACM Trans. Comput.-Hum. Interact. 20, 1, Article 7 (apr 2013), 25 pages. https://doi.org/10.1145/2442106.2442113
[35]
Deborah Lupton. 2018. How do data come to matter? Living and becoming with personal data. Big Data & Society 5, 2 (July 2018), 205395171878631. https://doi.org/10.1177/2053951718786314
[36]
Oussama Metatla, Nuno N. Correia, Fiore Martin, Nick Bryan-Kinns, and Tony Stockman. 2016. Tap the ShapeTones: Exploring the Effects of Crossmodal Congruence in an Audio-Visual Interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1055–1066. https://doi.org/10.1145/2858036.2858456
[37]
Keith V. Nesbitt. 2001. Modeling the Multi-Sensory Design Space. (2001), 27–36.
[38]
Keith V. Nesbitt. 2006. Modelling Human Perception to Leverage the Reuse of Concepts across the Multi-Sensory Design Space. In Proceedings of the 3rd Asia-Pacific Conference on Conceptual Modelling - Volume 53 (Hobart, Australia) (APCCM ’06). Australian Computer Society, Inc., AUS, 65–74.
[39]
T. Ogi and M. Hirose. 1996. Multisensory data sensualization based on human perception. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. IEEE, 12. https://doi.org/10.1109/vrais.1996.490512
[40]
Biswaksen Patnaik, Andrea Batch, and Niklas Elmqvist. 2019. Information olfactation: Harnessing scent to convey data. IEEE Transactions on Visualization and Computer Graphics 25, 1 (jan 2019), 726–736. https://doi.org/10.1109/TVCG.2018.2865237
[41]
Nimesha Ranasinghe, Pravar Jain, Nguyen Thi Ngoc Tram, Koon Chuan Raymond Koh, David Tolley, Shienny Karwita, Lin Lien-Ya, Yan Liangkun, Kala Shamaiah, Chow Eason Wai Tung, Ching Chiuan Yen, and Ellen Yi-Luen Do. 2018. Season Traveller: Multisensory Narration for Enhancing the Virtual Reality Experience. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174151
[42]
Jeffrey Rubin and Dana Chisnell. 2008. Handbook of usability testing(2 ed.). John Wiley & Sons, Chichester, England.
[43]
Kim Sauvé, Dominic Potts, Jason Alexander, and Steven Houben. 2020. A Change of Perspective: How User Orientation Influences the Perception of Physicalizations. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376312
[44]
Kim Sauvé, David Verweij, Jason Alexander, and Steven Houben. 2021. Reconfiguration Strategies with Composite Data Physicalizations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 471, 18 pages. https://doi.org/10.1145/3411764.3445746
[45]
Hasti Seifi, Kailun Zhang, and Karon E. MacLean. 2015. VibViz: Organizing, visualizing and navigating vibration libraries. In 2015 IEEE World Haptics Conference (WHC). 254–259. https://doi.org/10.1109/WHC.2015.7177722
[46]
J.G. Sheridan. 2003. Exploring cube affordance: towards a classification of non-verbal dynamics of physical interfaces for wearable computing. In IEE Eurowearable '03. IEE, 113–118. https://doi.org/10.1049/ic:20030156
[47]
Tara Shrimpton-Smith, Bieke Zaman, and David Geerts. 2008. Coupling the Users: The Benefits of Paired User Testing for iDTV. International Journal of Human–Computer Interaction 24, 2(2008), 197–213. https://doi.org/10.1080/10447310701821558 arXiv:https://doi.org/10.1080/10447310701821558
[48]
Simon Stusak, Moritz Hobe, and Andreas Butz. 2016. If Your Mind Can Grasp It, Your Hands Will Help. In Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (Eindhoven, Netherlands) (TEI ’16). Association for Computing Machinery, New York, NY, USA, 92–99. https://doi.org/10.1145/2839462.2839476
[49]
John Sweller, Jeroen J. G. van Merrienboer, and Fred G. W. C. Paas. 1998. Cognitive Architecture and Instructional Design. Educational Psychology Review 10, 3 (Sept. 1998), 251–296. https://doi.org/10.1023/A:1022193728205
[50]
Susanne Tak and Alexander Toet. 2013. Towards Interactive Multisensory Data Representations. In International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications (IVAPP-2013). SciTePress, Portugal, 558–561. https://doi.org/10.5220/0004346405580561
[51]
Rosa van Koningsbruggen and Eva Hornecker. 2021. “It’s Just a Graph” – The Effect of Post-Hoc Rationalisation on InfoVis Evaluation. In Creativity and Cognition (Virtual Event, Italy) (C&C ’21). Association for Computing Machinery, New York, NY, USA, Article 45, 10 pages. https://doi.org/10.1145/3450741.3465257
[52]
Rosa van Koningsbruggen, Hannes Waldschütz, and Eva Hornecker. 2022. What is Data? - Exploring the Meaning of Data in Data Physicalisation Teaching. In Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction(Daejeon, Republic of Korea) (TEI ’22). Association for Computing Machinery, New York, NY, USA, Article 13, 21 pages. https://doi.org/10.1145/3490149.3501319
[53]
Yun Wang, Adrien Segal, Roberta Klatzky, Daniel F. Keefe, Petra Isenberg, Jorn Hurtienne, Eva Hornecker, Tim Dwyer, Stephen Barrass, and Theresa-Marie Rhyne. 2019. An Emotional Response to the Value of Visualization. IEEE Computer Graphics and Applications 39, 5 (Sept. 2019), 8–17. https://doi.org/10.1109/MCG.2019.2923483
[54]
Daniel Wildman. 1995. Getting the Most from Paired-User Testing. Interactions 2, 3 (jul 1995), 21–27. https://doi.org/10.1145/208666.208675
[55]
Wai Yu, Ramesh Ramloll, and Stephen Brewster. 2001. Haptic graphs for blind computer users. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2058 (2001), 41–51. https://doi.org/10.1007/3-540-44589-7_5
[56]
Jack Zhao and Andrew Vande Moere. 2008. Embodiment in Data Sculpture: A Model of the Physical Visualization of Information. In Proceedings of the 3rd International Conference on Digital Interactive Media in Entertainment and Arts (Athens, Greece) (DIMEA ’08). Association for Computing Machinery, New York, NY, USA, 343–350. https://doi.org/10.1145/1413634.1413696
Index Terms
- Birdbox: Exploring the User Experience of Crossmodal, Multisensory Data Representations
Recommendations
The User Experience of Journeys in the Realm of Augmented Reality Television
IMX '22: Proceedings of the 2022 ACM International Conference on Interactive Media ExperiencesAugmented Reality Television (ARTV) can take many forms, from AR content displayed outside the TV frame to video-projected TV screens to social TV watching in VR to immersive holograms in the living room. While the user experience (UX) of individual ...
Exploring how politeness impacts the user experience of chatbots for mental health support
AbstractPoliteness is important in human–human interaction when asking people to engage in sensitive conversations. If politeness manifests similarly in human–chatbot interaction, it may play an important role in the design of sensitive chatbot ...
Highlights- Politeness can both positively and negatively impact the chatbot user experience.
- The Personal politeness chatbot was experienced as caring and encouraging.
- The Passive politeness chatbot was experienced as too apologetic and ...
Measuring retrospective user experience of non-powered hand tools: an exploratory remote study with UX curve
AcademicMindTrek '14: Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & ServicesUser experience (UX) can change over time, when the exciting new product becomes an ordinary part of daily life. With experiences, also our evaluation of a product can change, affecting whether we will recommend the product for others. Studying the long-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2022
315 pages
ISBN:9781450398206
DOI:10.1145/3568444
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 December 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
MUM 2022
MUM 2022: 21th International Conference on Mobile and Ubiquitous Multimedia
November 27 - 30, 2022
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 190 of 465 submissions, 41%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 288Total Downloads
- Downloads (Last 12 months)154
- Downloads (Last 6 weeks)23
Reflects downloads up to 16 Dec 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in