AudioXtend: Assisted Reality Visual Accompaniments for Audiobook Storytelling During Everyday Routine Tasks
Authors: 
Felicia Fang-Yi Tan, 
Peisen Xu, Ashwin Ram, Wei Zhen Suen, Shengdong Zhao, Yun Huang, 
Christophe HurterAuthors Info & Claims
Article No.: 83, Pages 1 - 22
Abstract
The rise of multitasking in contemporary lifestyles has positioned audio-first content as an essential medium for information consumption. We present AudioXtend, an approach to augment audiobook experiences during daily tasks by integrating glanceable, AI-generated visuals through optical see-through head-mounted displays (OHMDs). Our initial study showed that these visual augmentations not only preserved users’ primary task efficiency but also dramatically enhanced immediate auditory content recall by 33.3% and 7-day recall by 32.7%, alongside a marked improvement in narrative engagement. Through participatory design workshops involving digital arts designers, we crafted a set of design principles for visual augmentations that are attuned to the requirements of multitaskers. Finally, a 3-day take-home field study further revealed new insights for everyday use, underscoring the potential of assisted reality (aR) to enhance heads-up listening and incidental learning experiences.
Supplemental Material
- Download
- 7.52 MB
- Transcript
- Download
- 74.17 MB
- Transcript
References
[1]
[n. d.]. OpenAI GPT-4. https://openai.com/gpt-4
[2]
Cynthia Adams, Malcolm C. Smith, Monisha Pasupathi, and Loretta Vitolo. 2002. Social Context Effects on Story Recall in Older and Younger Women: Does the Listener Make a Difference?57, 1 (2002), P28–P40. https://doi.org/10.1093/geronb/57.1.P28
[3]
Simone Ashby, Julian Hanna, Sónia Matos, Callum Nash, and Alexis Faria. 2019. Fourth-Wave HCI Meets the 21st Century Manifesto. In Proceedings of the Halfway to the Future Symposium 2019. Association for Computing Machinery, New York, NY, USA, Article 23, 11 pages. https://doi.org/10.1145/3363384.3363467
[4]
Benjamin Bach, Ronell Sicat, Johanna Beyer, Maxime Cordeil, and Hanspeter Pfister. 2018. The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality?24, 1 (2018), 457–467. https://doi.org/10.1109/TVCG.2017.2745941 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
[5]
Benjamin Baird, Jonathan Smallwood, Michael D. Mrazek, Julia W. Y. Kam, Michael S. Franklin, and Jonathan W. Schooler. 2012. Inspired by distraction: mind wandering facilitates creative incubation. 23, 10 (2012), 1117–1122. https://doi.org/10.1177/0956797612446024
[6]
Mieke Bal and Christine Van Boheemen. 2009. Narratology: Introduction to the theory of narrative. University of Toronto Press.
[7]
Ugo Ballenghein, Johanna K. Kaakinen, Geoffrey Tissier, and Thierry Baccino. 2023. Fluctuation in cognitive engagement during listening and reading of erotica and horror stories. 37, 5 (2023), 874–890. https://doi.org/10.1080/02699931.2023.2215974 Publisher: Routledge _eprint: https://doi.org/10.1080/02699931.2023.2215974.
[8]
Liam J. Bannon. 2006. Forgetting as a feature, not a bug: the dualityof memory and implications for ubiquitous computing. 2, 1 (2006), 3–15. https://doi.org/10.1080/15710880600608230 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/15710880600608230.
[9]
John A Bargh. 2014. The four horsemen of automaticity: Awareness, intention, efficiency, and control in social cognition. In Handbook of social cognition. Psychology Press, 1–40.
[10]
Michael D. Basil. 2012. Multiple Resource Theory. Springer US, Boston, MA, 2384–2385. https://doi.org/10.1007/978-1-4419-1428-6_25
[11]
Tanja Blascheck, Lonni Besançon, Anastasia Bezerianos, Bongshin Lee, and Petra Isenberg. 2019. Glanceable Visualization: Studies of Data Comparison Performance on Smartwatches. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 630–640. https://doi.org/10.1109/TVCG.2018.2865142
[12]
Alexis Boutilier. 2023. Audiobook usage increased by 70% in 2022. https://goodereader.com/blog/audiobooks/audiobook-usage-increased-by-70-in-2022
[13]
Doug A. Bowman and Ryan P. McMahan. 2007. Virtual Reality: How Much Immersion Is Enough?Computer 40, 7 (2007), 36–43. https://doi.org/10.1109/MC.2007.257
[14]
Jerome Bruner. 1986. Actual minds, possible worlds. Harvard University Press. Pages: xi, 201.
[15]
Rick Busselle and Helena Bilandzic. 2009. Measuring narrative engagement. Media psychology 12, 4 (2009), 321–347.
[16]
JJ Cadiz, Gina Venolia, Gavin Jancke, and Anoop Gupta. 2001. Sideshow: Providing Peripheral Awareness of Important Information. Technical Report MSR-TR-2001-83. https://www.microsoft.com/en-us/research/publication/sideshow-providing-peripheral-awareness-of-important-information/
[17]
Runze Cai, Nuwan Nanayakkarawasam Peru Kandage Janaka, Shengdong Zhao, and Minghui Sun. 2023. ParaGlassMenu: Towards Social-Friendly Subtle Interactions in Conversations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 721, 21 pages. https://doi.org/10.1145/3544548.3581065
[18]
Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 981–992. https://doi.org/10.1145/2858036.2858498
[19]
Russell N Carney and Joel R Levin. 2002. Pictorial illustrations still improve students’ learning from text. Educational psychology review 14 (2002), 5–26.
[20]
Wesley P. Chan, Geoffrey Hanks, Maram Sakr, Haomiao Zhang, Tiger Zuo, H. F. Machiel van der Loos, and Elizabeth Croft. 2022. Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks. 11, 3 (2022), 31:1–31:19. https://doi.org/10.1145/3524082
[21]
Soon Hau Chua, Simon T. Perrault, Denys J. C. Matthies, and Shengdong Zhao. 2016. Positioning Glass: Investigating Display Positions of Monocular Optical See-Through Head-Mounted Display. In Proceedings of the Fourth International Symposium on Chinese CHI (San Jose, USA) (ChineseCHI 2016). Association for Computing Machinery, New York, NY, USA, Article 1, 6 pages. https://doi.org/10.1145/2948708.2948713
[22]
Catherine Courage and Kathy Baxter. 2005. Understanding your users: A practical guide to user requirements methods, tools, and techniques. Gulf Professional Publishing.
[23]
Chris Creed, Maadh Al-Kalbani, Arthur Theil, Sayan Sarcar, and Ian Williams. 2023. Inclusive AR/VR: accessibility barriers for immersive technologies. Universal Access in the Information Society (2023), 1–15.
[24]
David B Daniel and William Douglas Woody. 2010. They hear, but do not listen: Retention for podcasted material in a classroom context. Teaching of Psychology 37, 3 (2010), 199–203.
[25]
Thomas H Davenport and Rajeev Ronanki. 2018. HBR: Artificial intelligence for the real world. https://www.bizjournals.com/boston/news/2018/01/09/hbr-artificial-intelligence-for-the-real-world.html
[26]
Fred D. Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 13, 3 (1989), 319–340. https://doi.org/10.2307/249008 Publisher: Management Information Systems Research Center, University of Minnesota.
[27]
Aaron M Dollar. 2014. Classifying human hand use and the activities of daily living. The human hand as an inspiration for robot hand development (2014), 201–216.
[28]
Jason K. Eshraghian. 2020. Human ownership of artificial creativity. 2, 3 (2020), 157–160. https://doi.org/10.1038/s42256-020-0161-x Number: 3 Publisher: Nature Publishing Group.
[29]
Loris P Fagioli. 2008. Effects of Illustrations on Retention and Visual Attention Using Authentic Textbooks.
[30]
Ian C Fiebelkorn and Sabine Kastner. 2019. A rhythmic theory of attention. Trends in cognitive sciences 23, 2 (2019), 87–101.
[31]
Paul M Fitts and Michael I Posner. 1967. Human performance. (1967).
[32]
Neil D Fleming and Colleen Mills. 1992. Not another inventory, rather a catalyst for reflection. To improve the academy 11, 1 (1992), 137–155.
[33]
Christopher Frauenberger. 2020. Entanglement HCI The Next Wave?ACM Transactions on Computer-Human Interaction 27, 1 (Feb. 2020), 1–27. https://doi.org/10.1145/3364998
[34]
Aaron Friedland, Michelle Gilman, Michael Johnson, and Abera Demeke. 2017. Does Reading-While-Listening Enhance Students’ Reading Fluency? Preliminary Results from School Experiments in Rural Uganda.Journal of Education and Practice 8, 7 (2017), 82–95.
[35]
Runshan Fu, Yan Huang, and Param Vir Singh. 2020. AI and Algorithmic Bias: Source, Detection, Mitigation and Implications. https://doi.org/10.2139/ssrn.3681517
[36]
Joseph L. Gabbard, J. Edward Swan, II, and Deborah Hix. 2006. The Effects of Text Drawing Styles, Background Textures, and Natural Lighting on Text Legibility in Outdoor Augmented Reality. 15, 1 (2006), 16–32. https://doi.org/10.1162/pres.2006.15.1.16
[37]
Linda B. Gambrell and Paula Brooks Jawitz. 1993. Mental Imagery, Text Illustrations, and Children’s Story Comprehension and Recall. 28, 3 (1993), 265–276. https://doi.org/10.2307/747998 Publisher: [Wiley, International Reading Association].
[38]
Debjyoti Ghosh, Pin Sym Foong, Shengdong Zhao, Can Liu, Nuwan Janaka, and Vinitha Erusu. 2020. EYEditor: Towards On-the-Go Heads-Up Text Editing Using Voice and Manual Input. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376173
[39]
Katie L. Glonek and Paul E. King. 2014. Listening to Narratives: An Experimental Examination of Storytelling in the Classroom. 28, 1 (2014), 32–46. https://doi.org/10.1080/10904018.2014.861302 Publisher: Routledge _eprint: https://doi.org/10.1080/10904018.2014.861302.
[40]
Xuan Gong, Shuyan Chen, Baochang Zhang, and David Doermann. 2021. Style Consistent Image Generation for Nuclei Instance Segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 3994–4003.
[41]
Arthur C. Graesser, Kathy Hauft-Smith, Andrew D. Cohen, and Leonard D. Pyles. 1980. Advanced Outlines, Familiarity, and Text Genre on Retention of Prose. 48, 4 (1980), 281–290. https://doi.org/10.1080/00220973.1980.11011745 Publisher: Routledge.
[42]
Melanie C Green and Timothy C Brock. 2000. The role of transportation in the persuasiveness of public narratives.Journal of personality and social psychology 79, 5 (2000), 701.
[43]
Yike Guo, Qifeng Liu, Jie Chen, Wei Xue, Henrik Jensen, Fernando Rosas, Jeffrey Shaw, Xing Wu, Jiji Zhang, and Jianliang Xu. 2022. Pathway to Future Symbiotic Creativity. arXiv preprint arXiv:2209.02388 (2022).
[44]
Aleesha Hamid, Rabiah Arshad, and Suleman Shahid. 2022. What Are You Thinking?: Using CBT and Storytelling to Improve Mental Health Among College Students. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 441, 16 pages. https://doi.org/10.1145/3491102.3517603
[45]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
[46]
Iben Have and Birgitte Stougaard Pedersen. 2015. Digital audiobooks: New media, users, and experiences. Routledge.
[47]
Iben Have and Birgitte Stougaard Pedersen. 2021. Reading Audiobooks. Springer International Publishing, Cham, 197–216. https://doi.org/10.1007/978-3-030-49679-1_6
[48]
Jeremy M. Heiner, Scott E. Hudson, and Kenichiro Tanaka. 1999. The Information Percolator: Ambient Information Display in a Decorative Object. In Proceedings of the 12th Annual ACM Symposium on User Interface Software and Technology (Asheville, North Carolina, USA) (UIST ’99). Association for Computing Machinery, New York, NY, USA, 141–148. https://doi.org/10.1145/320719.322595
[49]
Sophie K Herbst and Ayelet N Landau. 2016. Rhythms for cognition: the case of temporal processing. Current Opinion in Behavioral Sciences 8 (2016), 85–93.
[50]
Rebecca J. Hess, Jennifer S. Brach, Sara R. Piva, and Jessie M. VanSwearingen. 2010. Walking Skill Can Be Assessed in Older Adults: Validity of the Figure-of-8 Walk Test. Physical Therapy 90, 1 (01 2010), 89–99. https://doi.org/10.2522/ptj.20080121
[51]
Kevin Huang, Jiannan Li, Mauricio Sousa, and Tovi Grossman. 2022. ImmersivePOV: Filming How-To Videos with a Head-Mounted 360° Action Camera. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 91, 13 pages. https://doi.org/10.1145/3491102.3517468
[52]
Kevin Huang, Thad Starner, Ellen Do, Gil Weinberg, Daniel Kohlsdorf, Claas Ahlrichs, and Ruediger Leibrandt. 2010. Mobile music touch: mobile tactile stimulation for passive learning. In Proceedings of the SIGCHI conference on human factors in computing systems. 791–800.
[53]
Mei lin Jiang and Alex W.C. Tse. 2022. The Effects of Multimedia Glosses, Learning Duration and Working Memory Capacity on High School Students’ Passive English Vocabulary Acquisition.
[54]
Sebastian Kapp, Michael Barz, Sergey Mukhametov, Daniel Sonntag, and Jochen Kuhn. 2021. ARETT: Augmented Reality Eye Tracking Toolkit for Head Mounted Displays. 21, 6 (2021), 2234. https://doi.org/10.3390/s21062234 Number: 6 Publisher: Multidisciplinary Digital Publishing Institute.
[55]
Krzysztof Krejtz, Andrew T. Duchowski, Anna Niedzielska, Cezary Biele, and Izabela Krejtz. 2018. Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze. 13, 9 (2018). https://doi.org/10.1371/journal.pone.0203629 Place: US Publisher: Public Library of Science.
[56]
Masaaki Kurosu and Kaori Kashimura. 1995. Apparent Usability vs. Inherent Usability: Experimental Analysis on the Determinants of the Apparent Usability. In Conference Companion on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’95). Association for Computing Machinery, New York, NY, USA, 292–293. https://doi.org/10.1145/223355.223680
[57]
Marc Lambooij, Marten Fortuin, Ingrid Heynderickx, Wijnand IJsselsteijn, Marten Fortuin, Ingrid Heynderickx, and Wijnand IJsselsteijn. 2009. Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review. 53 (2009), 1–14. https://doi.org/10.2352/J.ImagingSci.Technol.2009.53.3.030201 Publisher: Society for Imaging Science and Technology.
[58]
Lotta C Larson. 2015. E-books and audiobooks: Extending the digital reading experience. The Reading Teacher 69, 2 (2015), 169–177.
[59]
Kyung-Ryong Lee, Beom Kim, Junyoung Kim, Hwajung Hong, and Young-Woo Park. 2021. ADIO: An interactive artifact physically representing the intangible digital audiobook listening experience in everyday living spaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
[60]
Joseph Lemley, Anuradha Kar, Alexandru Drimbarean, and Peter Corcoran. 2023. Efficient CNN Implementation for Eye-Gaze Estimation on Low-Power/Low-Quality Consumer Imaging Systems. arxiv:1806.10890 [cs]http://arxiv.org/abs/1806.10890
[61]
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. 2019. StoryGAN: A Sequential Conditional GAN for Story Visualization. arxiv:1812.02784 [cs.CV]
[62]
Mike E.U. Ligthart, Mark A. Neerincx, and Koen V. Hindriks. 2020. Design Patterns for an Interactive Storytelling Robot to Support Children’s Engagement and Agency. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (Cambridge, United Kingdom) (HRI ’20). Association for Computing Machinery, New York, NY, USA, 409–418. https://doi.org/10.1145/3319502.3374826
[63]
Cindy Lin and Silvia Lindtner. 2021. Techniques of Use: Confronting Value Systems of Productivity, Progress, and Usefulness in Computing and Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–16. https://doi.org/10.1145/3411764.3445237
[64]
D. W. Livingstone. 1999. Exploring the Icebergs of Adult Learning: Findings of the First Canadian Survey of Informal Learning Practices. 13, 2 (1999), 49–72. https://cjsae.library.dal.ca/index.php/cjsae/article/view/2000 Number: 2.
[65]
Gordon D Logan. 2002. An instance theory of attention and memory.Psychological review 109, 2 (2002), 376.
[66]
Feiyu Lu, Shakiba Davari, Lee Lisle, Yuan Li, and Doug Bowman. 2020. Glanceable AR: Evaluating Information Access Methods for Head-Worn Augmented Reality. 930–939. https://doi.org/10.1109/VR46266.2020.00113
[67]
Victoria J. Marsick and Karen E. Watkins. [n. d.]. Informal and Incidental Learning. 2001, 89 ([n. d.]), 25–34. https://doi.org/10.1002/ace.5 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ace.5.
[68]
Tara Matthews. 2006. Designing and Evaluating Glanceable Peripheral Displays. In Proceedings of the 6th Conference on Designing Interactive Systems (University Park, PA, USA) (DIS ’06). Association for Computing Machinery, New York, NY, USA, 343–345. https://doi.org/10.1145/1142405.1142457
[69]
Tara Matthews, Anind K. Dey, Jennifer Mankoff, Scott Carter, and Tye Rattenbury. 2004. A Toolkit for Managing User Attention in Peripheral Displays. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM, USA) (UIST ’04). Association for Computing Machinery, New York, NY, USA, 247–256. https://doi.org/10.1145/1029632.1029676
[70]
Shannon Maughan. 2023. The audiobook market, and its revenue, Keep Growing. https://www.publishersweekly.com/pw/by-topic/industry-news/audio-books/article/92444-the-audiobook-market-and-revenue-keeps-growing.html
[71]
Richard E Mayer. 2002. Multimedia learning. In Psychology of learning and motivation. Vol. 41. Elsevier, 85–139.
[72]
Microsoft Bing. [n. d.]. Microsoft Bing Image Creator (Powered by DALL-E). https://www.bing.com/create
[73]
Natalia Mikidenko and Svetlana Storozheva. 2021. Audiobooks: Reading Practices and Educational Technologies. 97 (2021), 01016. https://doi.org/10.1051/shsconf/20219701016
[74]
Tram Thi Minh Tran, Shane Brown, Oliver Weidlich, Mark Billinghurst, and Callum Parker. 2023. Wearable Augmented Reality: Research Trends and Future Directions from Three Major Venues. 29, 11 (2023), 4782–4793. https://doi.org/10.1109/TVCG.2023.3320231
[75]
Anca Morar, Maria-Anca Băluțoiu, Alin Moldoveanu, Florica Moldoveanu, and Alex Butean. 2021. CultReal—A Rapid Development Platform for AR Cultural Spaces, with Fused Localization. 21, 19 (2021). https://doi.org/10.3390/s21196618
[76]
Roxana Moreno. 2006. Learning in High-Tech and Multimedia Environments. Current Directions in Psychological Science 15, 2 (2006), 63–67. https://doi.org/10.1111/j.0963-7214.2006.00408.x
[77]
Elizabeth L. Murnane, Yekaterina S. Glazko, Jean Costa, Raymond Yao, Grace Zhao, Paula M. L. Moya, and James A. Landay. 2023. Narrative-Based Visual Feedback to Encourage Sustained Physical Activity: A Field Trial of the WhoIsZuki Mobile Health Platform. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 1, Article 23 (mar 2023), 36 pages. https://doi.org/10.1145/3580786
[78]
Janet H Murray. 2017. Hamlet on the Holodeck, updated edition: The Future of Narrative in Cyberspace. MIT press.
[79]
Ali Neshati, Yumiko Sakamoto, Launa Leboe-Mcgowan, Jason Leboe-Mcgowan, Marcos Serrano, and Pourang Irani. 2019. G-Sparks: Glanceable Sparklines on Smartwatches. In 45th Conference on Graphics Interface (GI 2019). Kingston, Ontario, Canada, 1–9. https://doi.org/10.20380/GI2019.23
[80]
Rachel Noorda and Kathi Inman Berens. 2021. Immersive Media and Books 2020: New Insights About Book Pirates, Libraries and Discovery, Millennials, and Cross-Media Engagement: Before and During COVID.
[81]
Donald A Norman. 2004. Emotional design: Why we love (or hate) everyday things. Civitas Books.
[82]
The Editors of Goop. 2015. Konmari method - Marie Kondo folding guide for clothes | Goop. https://goop.com/food/decorating-design/the-illustrated-guide-to-the-kondo-mari-method/
[83]
Eyal Ophir, Clifford Nass, and Anthony D. Wagner. 2009. Cognitive control in media multitaskers. 106, 37 (2009), 15583–15587. https://doi.org/10.1073/pnas.0903620106 _eprint: https://www.pnas.org/doi/pdf/10.1073/pnas.0903620106.
[84]
Josetxu Orrantia, David Múñez, and Julio Tarín. 2014. Connecting goals and actions during reading: The role of illustrations. Reading and Writing 27 (2014), 153–170.
[85]
Anna-Marie Ortloff, Lydia Güntner, Maximiliane Windl, Thomas Schmidt, Martin Kocur, and Christian Wolff. 2019. SentiBooks: Enhancing Audiobooks via Affective Computing and Smart Light Bulbs. In Proceedings of Mensch Und Computer 2019 (Hamburg, Germany) (MuC ’19). Association for Computing Machinery, New York, NY, USA, 863–866. https://doi.org/10.1145/3340764.3345368
[86]
Allan Paivio. 1991. Dual coding theory: Retrospect and current status.Canadian Journal of Psychology/Revue canadienne de psychologie 45, 3 (1991), 255.
[87]
Renate Prins, Lucy Avraamidou, and Martin Goedhart. 2017. Tell me a Story: the use of narrative as a learning tool for natural selection. 54, 1 (2017), 20–33. https://doi.org/10.1080/09523987.2017.1324361 Publisher: Routledge _eprint: https://doi.org/10.1080/09523987.2017.1324361.
[88]
Ashwin Ram and Shengdong Zhao. 2021. LSVP: Towards Effective On-the-Go Video Learning Using Optical Head-Mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 1, Article 30 (mar 2021), 27 pages. https://doi.org/10.1145/3448118
[89]
Ashwin Ram and Shengdong Zhao. 2022. Does Dynamically Drawn Text Improve Learning? Investigating the Effect of Text Presentation Styles in Video Learning. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 89, 12 pages. https://doi.org/10.1145/3491102.3517499
[90]
Luz Rello and Ricardo Baeza-Yates. 2013. Good fonts for dyslexia. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (Bellevue Washington, 2013-10-21). ACM, 1–8. https://doi.org/10.1145/2513383.2513447
[91]
James A. Rose. [n. d.]. To Teach Science, Tell Stories. https://www.semanticscholar.org/paper/To-Teach-Science%2C-Tell-Stories-Rose/8dcd13bbd719b1be288c8b3c60542f74c41411b6
[92]
Louise M. Rosenblatt. [n. d.]. The Transactional Theory: Against Dualisms. 55, 4 ([n. d.]), 377. https://doi.org/10.2307/378648
[93]
David Rothman. 2020. Most ebook and Audiobook Users Multitask: New study explores the details. https://teleread.org/2020/11/06/most-ebook-and-audiobook-users-multitask-new-study-explores-the-details/
[94]
D.C. Rubin. 1995. Memory in Oral Traditions: The Cognitive Psychology of Epic, Ballads, and Counting-out Rhymes. Oxford University Press. https://books.google.com/books?id=yq5A7djClegC
[95]
Rufat Rzayev, Paweł W. Woźniak, Tilman Dingler, and Niels Henze. 2018. Reading on Smart Glasses: The Effect of Text Position, Presentation Type and Walking. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3173574.3173619
[96]
Dario D. Salvucci. 2013. Multitasking. In The Oxford handbook of cognitive engineering. Oxford University Press, 57–67. https://doi.org/10.1093/oxfordhb/9780199757183.001.0001
[97]
Pamela Samuelson. 2023. Generative AI meets copyright. 381, 6654 (2023), 158–161. https://doi.org/10.1126/science.adi0656 Publisher: American Association for the Advancement of Science.
[98]
Shardul Sapkota, Ashwin Ram, and Shengdong Zhao. 2021. Ubiquitous Interactions for Heads-Up Computing: Understanding Users’ Preferences for Subtle Interaction Techniques in Everyday Settings. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (Toulouse & Virtual, France) (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 36, 15 pages. https://doi.org/10.1145/3447526.3472035
[99]
Diane Lemonnier Schallert. 1980. The role of illustrations in reading comprehension. Lawrence Erlbaum, 503–525.
[100]
Diane Lemonnier Schallert. 2017. The role of illustrations in reading comprehension. Theoretical issues in reading comprehension (2017), 503–524.
[101]
Daniel Schugurensky. 2000. The Forms of Informal Learning: Towards a Conceptualization of the Field. (2000).
[102]
Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press.
[103]
Daniel Schwajda, Judith Friedl, Fabian Pointecker, Hans-Christian Jetter, and Christoph Anthes. 2023. Transforming graph data visualisations from 2D displays into augmented reality 3D space: A quantitative study. 4 (2023). https://www.frontiersin.org/articles/10.3389/frvir.2023.1155628
[104]
Jae-eun Shin, Boram Yoon, Dooyoung Kim, and Woontack Woo. 2021. A User-Oriented Approach to Space-Adaptive Augmentation: The Effects of Spatial Affordance on Narrative Experience in an Augmented Reality Detective Game. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 722, 13 pages. https://doi.org/10.1145/3411764.3445675
[105]
Anisha Singh and Patricia A Alexander. 2022. Audiobooks, print, and comprehension: What we know and what we need to know. Educational Psychology Review 34, 2 (2022), 677–715.
[106]
Jacob Somervell, D. Scott McCrickard, Chris North, and Maulik Shukla. 2002. An Evaluation of Information Visualization in Attention-Limited Environments. In Eurographics / IEEE VGTC Symposium on Visualization, D. Ebert, P. Brunet, and I. Navazo (Eds.). The Eurographics Association. https://doi.org/10.2312/VisSym/VisSym02/211-216
[107]
Ramya Srinivasan and Devi Parikh. 2021. Building Bridges: Generative Artworks to Explore AI Ethics. arXiv preprint arXiv:2106.13901 (2021).
[108]
Ramya Srinivasan and Kanji Uchino. 2021. Biases in Generative Art: A Causal Look from the Lens of Art History. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (New York, NY, USA, 2021-03-01) (FAccT ’21). Association for Computing Machinery, 41–51. https://doi.org/10.1145/3442188.3445869
[109]
Felicia Fang-Yi Tan, Ashwin Ram, Chloe Haigh, and Shengdong Zhao. 2023. Mindful Moments: Exploring On-the-go Mindfulness Practice On Smart-glasses. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (, Pittsburgh, PA, USA, ) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 476–492. https://doi.org/10.1145/3563657.3596030
[110]
Xuli Tang, Xin Li, Ying Ding, Min Song, and Yi Bu. 2020. The pace of artificial intelligence innovations: Speed, talent, and trial-and-error. Journal of Informetrics 14, 4 (2020), 101094. https://doi.org/10.1016/j.joi.2020.101094
[111]
Laura Tarantino, Daniela Angelucci, Alessandra Bonomo, Annalisa Cardinali, and Stefania Di Paolo. 2021. Design and applications of GLANCE: GLanceable alarm notification for a user centered experience. Applied Sciences 11, 2 (2021), 669.
[112]
Elisa Tattersall Wallin. 2022. Audiobook apps: exploring reading practices and technical affordances in the player features. Information research 27, 4 (2022).
[113]
Edward R Tufte. 1991. Envisioning information. Optometry and Vision Science 68, 4 (1991), 322–324.
[114]
Jacob Turner. 2019. Responsibility for AI. In Robot Rules : Regulating Artificial Intelligence, Jacob Turner (Ed.). Springer International Publishing, 81–132. https://doi.org/10.1007/978-3-319-96235-1_3
[115]
Unity Real-Time Development Platform. [n. d.]. Unity. https://unity.com/
[116]
Karl Werder, Balasubramaniam Ramesh, and Rongen (Sophia) Zhang. 2022. Establishing Data Provenance for Responsible Artificial Intelligence Systems. 13, 2 (2022), 22:1–22:23. https://doi.org/10.1145/3503488
[117]
Christopher D Wickens. 2002. Multiple resources and performance prediction. Theoretical issues in ergonomics science 3, 2 (2002), 159–177.
[118]
Christopher D. Wickens, Michael Vidulich, and Diane Sandry-Garza. 1984. Principles of S-C-R Compatibility with Spatial and Verbal Tasks: The Role of Display-Control Location and Voice-Interactive Display-Control Interfacing. Human Factors 26, 5 (1984), 533–543. https://doi.org/10.1177/001872088402600505
[119]
Michael B. W. Wolfe and Joseph A. Mienko. 2007. Learning and memory of factual content from narrative and expository text. 77, 3 (2007), 541–564. https://doi.org/10.1348/000709906X143902 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1348/000709906X143902.
[120]
Bang Wong. [n. d.]. Points of view: Color blindness. 8, 6 ([n. d.]), 441–441. https://doi.org/10.1038/nmeth.1618
[121]
Marjorie Woollacott and Anne Shumway-Cook. 2002. Attention and the control of posture and gait: a review of an emerging area of research. Gait & Posture 16, 1 (2002), 1–14. https://doi.org/10.1016/S0966-6362(01)00156-4
[122]
[122] Xreal. [n. d.]. https://www.xreal.com/light/
[123]
Dominyk Zdanovic, Tanja Julie Lembcke, and Toine Bogers. [n. d.]. The Influence of Data Storytelling on the Ability to Recall Information. In Proceedings of the 2022 Conference on Human Information Interaction and Retrieval (New York, NY, USA, 2022-03-14) (CHIIR ’22). Association for Computing Machinery, 67–77. https://doi.org/10.1145/3498366.3505755
[124]
Fang Zhao, Robert Gaschler, Anneli Kneschke, Simon Radler, Melanie Gausmann, Christina Duttine, and Hilde Haider. 2020. Origami folding: Taxing resources necessary for the acquisition of sequential skills. Plos one 15, 10 (2020), e0240226.
[125]
Shengdong Zhao, Felicia Tan, and Katherine Fennedy. 2023. Heads-Up Computing Moving Beyond the Device-Centered Paradigm. Commun. ACM 66, 9 (Aug 2023), 56–63. https://doi.org/10.1145/3571722
Index Terms
- AudioXtend: Assisted Reality Visual Accompaniments for Audiobook Storytelling During Everyday Routine Tasks
Index terms have been assigned to the content through auto-classification.
Recommendations
What is XR? Towards a Framework for Augmented and Virtual Reality
AbstractAugmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and ...
Highlights- XR should not be used as an abbreviation for “Extended Reality”; X should be the placeholder for “all” new reality formats.
Grammar of VR Storytelling: Visual Cues
VRIC '17: Proceedings of the Virtual Reality International Conference - Laval Virtual 2017Visual storytelling is an integral part of the experience in media such as comic books, movies, video games and virtual reality. The grammar of storytelling that has been explored abundantly in virtual reality has close relationship with that of video ...
Compelling experiences in mixed reality interactive storytelling
ACE '04: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technologyEntertainment systems promise to be a significant application for Mixed Reality. Recently, a growing number of Mixed Reality applications have included interaction with virtual actors and storytelling. However, AI-based Interactive Storytelling ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- DesCartes programme, Campus for Research Excellence and Technological Enterprise (CREATE) under the National Research Foundation
- Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 programme
- CityU Start-up Grant
- National Research Foundation, Singapore under its AI Singapore Programme
Conference
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Upcoming Conference
CHI '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 1,048Total Downloads
- Downloads (Last 12 months)1,048
- Downloads (Last 6 weeks)143
Reflects downloads up to 16 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format