research-article

ARTiST: Automated Text Simplification for Task Guidance in Augmented Reality

Authors:

Sonia Castelo Quispe,

Claudio SilvaAuthors Info & Claims

CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Article No.: 939, Pages 1 - 24

https://doi.org/10.1145/3613904.3642772

Published: 11 May 2024 Publication History

Abstract

Text presented in augmented reality provides in-situ, real-time information for users. However, this content can be challenging to apprehend quickly when engaging in cognitively demanding AR tasks, especially when it is presented on a head-mounted display. We propose ARTiST, an automatic text simplification system that uses a few-shot prompt and GPT-3 models to specifically optimize the text length and semantic content for augmented reality. Developed out of a formative study that included seven users and three experts, our system combines a customized error calibration model with a few-shot prompt to integrate the syntactic, lexical, elaborative, and content simplification techniques, and generate simplified AR text for head-worn displays. Results from a 16-user empirical study showed that ARTiST lightens the cognitive load and improves performance significantly over both unmodified text and text modified via traditional methods. Our work constitutes a step towards automating the optimization of batch text data for readability and performance in augmented reality.

Supplemental Material

MP4 File - Video Preview

Video Preview

Transcript for: Video Preview

MP4 File - Video Presentation

Video Presentation

Transcript for: Video Presentation

MP4 File - Video Figure

This supplementary file is a short video describing the paper's content and our user study.

Download
67.52 MB

References

[1]

Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2022. Reason first, then respond: Modular Generation for Knowledge-infused Dialogue. In Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 7112–7132. https://aclanthology.org/2022.findings-emnlp.527

[2]

Murat Akçayır and Gökçe Akçayır. 2017. Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational research review 20 (2017), 1–11.

[3]

Suha S Al-Thanyyan and Aqil M Azmi. 2021. Automated text simplification: a survey. ACM Computing Surveys (CSUR) 54, 2 (2021), 1–36.

Digital Library

[4]

Oliver Alonzo. 2022. The use of automatic text simplification to provide reading assistance to deaf and hard-of-hearing individuals in computing fields. ACM SIGACCESS Accessibility and Computing132, Article 3 (mar 2022), 1 pages. https://doi.org/10.1145/3523265.3523268

Digital Library

[5]

Oliver Alonzo, Sooyeon Lee, Mounica Maddela, Wei Xu, and Matt Huenerfauth. 2022. A Dataset of Word-Complexity Judgements from Deaf and Hard-of-Hearing Adults for Text Simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Virtual), 119–124.

[6]

Oliver Alonzo, Matthew Seita, Abraham Glasser, and Matt Huenerfauth. 2020. Automatic text simplification tools for deaf and hard of hearing adults: Benefits of lexical simplification and providing users with autonomy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13.

Digital Library

[7]

Oliver Alonzo, Jessica Trussell, Matthew Watkins, Sooyeon Lee, and Matt Huenerfauth. 2022. Methods for evaluating the fluency of automatically simplified texts with deaf and hard-of-hearing adults at various literacy levels. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–10.

Digital Library

[8]

KM Augestad, H Han, J Paige, T Ponsky, CM Schlachta, B Dunkin, and J Mellinger. 2017. Educational implications for surgical telementoring: a current review with recommendations for future practice, policy, and research. Surgical endoscopy 31 (2017), 3836–3846.

[9]

Ronald T Azuma. 1997. A survey of augmented reality. Presence: teleoperators & virtual environments 6, 4 (1997), 355–385.

[10]

Beata Beigman Klebanov, Kevin Knight, and Daniel Marcu. 2004. Text simplification for information-seeking applications. In On the Move to Meaningful Internet Systems 2004: CoopIS, DOA, and ODBASE. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 735–747.

[11]

Christoph Bichlmeier, Sandro Michael Heining, Mohammad Rustaee, and Nassir Navab. 2007. Laparoscopic virtual mirror for understanding vessel structure evaluation study by twelve surgeons. In 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. IEEE, IEEE, 125–128.

Digital Library

[12]

Mark Billinghurst, Jerry Bowskill, Nick Dyer, and Jason Morphett. 1998. An evaluation of wearable information spaces. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No. 98CB36180). IEEE, IEEE, 20–27.

[13]

Mark Billinghurst and Andreas Duenser. 2012. Augmented reality in the classroom. Computer 45, 7 (2012), 56–63.

Digital Library

[14]

Matt Bower, Cathie Howe, Nerida McCredie, Austin Robinson, and David Grover. 2014. Augmented Reality in education–cases, places and potentials. Educational Media International 51, 1 (2014), 1–15.

[15]

David E. Breen, Ross T. Whitaker, Eric Rose, and Mihran Tuceryan. 1996. Interactive Occlusion and Automatic Object Placement for Augmented Reality. Computer Graphics Forum 15, 3 (1996), 11–22. https://doi.org/10.1111/1467-8659.1530011 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/1467-8659.1530011

[16]

Glenn W Brier 1950. Verification of forecasts expressed in terms of probability. Monthly weather review 78, 1 (1950), 1–3.

[17]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.

[18]

Josef Buchner, Katja Buntins, and Michael Kerres. 2022. The impact of augmented reality on cognitive load and performance: A systematic review. J. Comput. Assist. Learn. 38, 1 (2022), 285–303. https://doi.org/10.1111/JCAL.12617

[19]

Wolfgang Büschel, Annett Mitschick, Thomas Meyer, and Raimund Dachselt. 2019. Investigating smartphone-based pan and zoom in 3D data spaces in augmented reality. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services. Association for Computing Machinery, New York, NY, USA, Article 2, 13 pages.

Digital Library

[20]

Yvonne Canning, John Tait, Jackie Archibald, and Ros Crawley. 2000. Cohesive generation of syntactically simplified newspaper text. In Text, Speech and Dialogue. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 145–150.

[21]

Julie Carmigniani and Borko Furht. 2011. Augmented reality: an overview. Springer, New York, NY, 3–46.

[22]

John Carroll, Guido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait. 1998. Practical simplification of English newspaper text to assist aphasic readers. In Proceedings of the AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology. Association for the Advancement of Artificial Intelligence, 7–10.

[23]

John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying Text for Language-Impaired Readers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, Henry S. Thompson and Alex Lascarides (Eds.). Association for Computational Linguistics, Bergen, Norway, 269–270. https://aclanthology.org/E99-1042

[24]

Sonia Castelo, Joao Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, Jing Qian, Kyunghyun Cho, He He, Qi Sun, Huy Vo, Juan Bello, Michael Krone, and Claudio Silva. 2024. ARGUS: Visualization of AI-Assisted Task Guidance in AR. IEEE Transactions on Visualization and Computer Graphics 30, 1 (2024), 1313–1323. https://doi.org/10.1109/TVCG.2023.3327396

Digital Library

[25]

Raman Chandrasekar, Christine Doran, and Srinivas Bangalore. 1996. Motivations and methods for text simplification. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.

Digital Library

[26]

Antonina Dattolo and Flaminia L Luccio. 2017. Accessible and usable websites and mobile applications for people with autism spectrum disorders: a comparative study. EAI Endorsed Transactions on Ambient Systems 4, 13 (2017).

[27]

Siobhan Devlin. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic databases (1998).

[28]

Matt Dunleavy and Chris Dede. 2014. Augmented reality teaching and learning. Handbook of research on educational communications and technology (2014), 735–745.

[29]

Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019. Revisiting collaboration through mixed reality: The evolution of groupware. International Journal of Human-Computer Studies 131 (2019), 81–98.

Digital Library

[30]

Richard Evans, Constantin Orăsan, and Iustin Dornescu. 2014. An evaluation of syntactic simplification rules for people with autism. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), Sandra Williams, Advaith Siddharthan, and Ani Nenkova (Eds.). Association for Computational Linguistics, Gothenburg, Sweden, 131–140. https://doi.org/10.3115/v1/W14-1215

[31]

Catarina G Fidalgo, Yukang Yan, Hyunsung Cho, Maurício Sousa, David Lindlbauer, and Joaquim Jorge. 2023. A Survey on Remote Assistance and Training in Mixed Reality Environments. IEEE Transactions on Visualization and Computer Graphics 29, 5 (2023), 2291–2303.

Digital Library

[32]

Michele Fiorentino, Antonio E. Uva, Michele Gattullo, Saverio Debernardis, and Giuseppe Monno. 2014. Augmented reality on large screen for interactive maintenance instructions. Comput. Ind. 65, 2 (2014), 270–278. https://doi.org/10.1016/J.COMPIND.2013.11.004

Digital Library

[33]

Joseph L Gabbard, J Edward Swan, and Deborah Hix. 2006. The effects of text drawing styles, background textures, and natural lighting on text legibility in outdoor augmented reality. Presence 15, 1 (2006), 16–32.

Digital Library

[34]

Joseph L Gabbard, J Edward Swan, Deborah Hix, Si-Jung Kim, and Greg Fitch. 2007. Active text drawing styles for outdoor augmented reality: A user-based study and design implications. In 2007 IEEE Virtual Reality Conference. IEEE, IEEE, 35–42.

[35]

Núria Gala and Johannes Ziegler. 2016. Reducing lexical complexity as a tool to increase text accessibility for children with dyslexia. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC). The COLING 2016 Organizing Committee, Osaka, Japan, 59–66.

[36]

Dmitriy Genzel and Eugene Charniak. 2002. Entropy Rate Constancy in Text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 199–206. https://doi.org/10.3115/1073083.1073117

Digital Library

[37]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017(Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1321–1330. http://proceedings.mlr.press/v70/guo17a.html

[38]

Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.

[39]

Jeremy Hartmann, Yen-Ting Yeh, and Daniel Vogel. 2020. AAR: Augmenting a Wearable Augmented Reality Display with an Actuated Head-Mounted Projector. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 445–458. https://doi.org/10.1145/3379337.3415849

Digital Library

[40]

Steven J Henderson and Steven Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, IEEE, 135–144.

Digital Library

[41]

Firas Hmida, Mokhtar B. Billami, Thomas François, and Núria Gala. 2018. Assisted Lexical Simplification for French Native Children with Reading Difficulties. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), Arne Jönsson, Evelina Rennes, Horacio Saggion, Sanja Stajner, and Victoria Yaneva (Eds.). Association for Computational Linguistics, Tilburg, the Netherlands, 21–28. https://doi.org/10.18653/v1/W18-7004

[42]

Richard L Holloway. 1997. Registration error analysis for augmented reality. Presence: Teleoperators & Virtual Environments 6, 4 (1997), 413–432.

Digital Library

[43]

Aleksander Holynski and Johannes Kopf. 2018. Fast depth densification for occlusion-aware augmented reality. ACM Transactions on Graphics (ToG) 37, 6 (2018), 1–11.

Digital Library

[44]

Michael Hornacek, Hans Küffner-McCauley, Majesa Trimmel, Patrick Rupprecht, and Sebastian Schlund. 2022. A spatial AR system for wide-area axis-aligned metric augmentation of planar scenes. CIRP Journal of Manufacturing Science and Technology 37 (2022), 219–226.

[45]

María-Blanca Ibáñez and Carlos Delgado-Kloos. 2018. Augmented reality for STEM learning: A systematic review. Computers & Education 123 (2018), 109–123.

Digital Library

[46]

Tatyana Ivanova Ivanova. 2017. Ontology-Based Text Simplification for Dyslexics. Science and Technology 3, 10 (2017), 34–47.

[47]

Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF Model for Sentence Alignment in Text Simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, Online, 7943–7960. https://doi.org/10.18653/v1/2020.acl-main.709

[48]

Matthew Kay, Tara Kola, Jessica R. Hullman, and Sean A. Munson. 2016. When (ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, May 7-12, 2016, Jofish Kaye, Allison Druin, Cliff Lampe, Dan Morris, and Juan Pablo Hourcade (Eds.). ACM, 5092–5103. https://doi.org/10.1145/2858036.2858558

Digital Library

[49]

Julian Keil, Annika Korte, Anna Ratmer, Dennis Edler, and Frank Dickmann. 2020. Augmented reality (AR) and spatial cognition: effects of holographic grids on distance estimation and location memory in a 3D indoor scenario. PFG–Journal of Photogrammetry, Remote Sensing and Geoinformation Science 88, 2 (2020), 165–172.

[50]

Frank Keller. 2004. The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Barcelona, Spain, 317–324. https://aclanthology.org/W04-3241

[51]

Kangsoo Kim, Luke Boelling, Steffen Haesler, Jeremy Bailenson, Gerd Bruder, and Greg F Welch. 2018. Does a digital assistant need a body? The influence of visual embodiment and social behavior on the perception of intelligent virtual agents in AR. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, IEEE, 105–114.

[52]

Seungone Kim, Se June Joo, Yul Jang, Hyungjoo Chae, and Jinyoung Yeo. 2023. CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. EACL 2023 - System Demonstrations, Dubrovnik, Croatia, May 2-4, 2023, Danilo Croce and Luca Soldaini (Eds.). Association for Computational Linguistics, 195–208. https://aclanthology.org/2023.eacl-demo.23

[53]

Radha Kumaran, You-Jin Kim, Anne E. Milner, Tom Bullock, Barry Giesbrecht, and Tobias Höllerer. 2023. The Impact of Navigation Aids on Search Performance and Object Recall in Wide-Area Augmented Reality. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (Eds.). ACM, 710:1–710:17. https://doi.org/10.1145/3544548.3581413

Digital Library

[54]

Jean-François Lapointe, Heather Molyneaux, and Mohand Saïd Allili. 2020. A literature review of AR-based remote guidance tasks with user studies. In Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications. Springer International Publishing, Cham, 111–120.

[55]

Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (, New Orleans, LA, USA, ) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 388, 19 pages. https://doi.org/10.1145/3491102.3502030

Digital Library

[56]

Vincent Lepetit and M-O Berger. 2000. A semi-automatic method for resolving occlusion in augmented reality. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), Vol. 2. IEEE, IEEE, 225–230.

[57]

Georgianna Lin, Tanmoy Panigrahi, Jon Womack, Devansh Jatin Ponda, Pramod Kotipalli, and Thad Starner. 2021. Comparing order picking guidance with Microsoft hololens, magic leap, google glass xe and paper. In Proceedings of the 22nd international workshop on mobile computing systems and applications. 133–139.

Digital Library

[58]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692http://arxiv.org/abs/1907.11692

[59]

Heather Lotherington-Woloszyn. 1993. Do Simplified Texts Simplify Language Comprehension for ESL Learners?.English for Specific Purposes 68 (1993), 31–46.

[60]

Michael R Lyu, Irwin King, TT Wong, Edward Yau, and PW Chan. 2005. Arcade: Augmented reality computing arena for digital entertainment. In 2005 IEEE Aerospace Conference. IEEE, 1–9.

[61]

Blair MacIntyre, Enylton Machado Coelho, and Simon J Julier. 2002. Estimating and adapting to registration errors in augmented reality systems. In Proceedings IEEE Virtual Reality 2002. IEEE, IEEE, 73–80.

[62]

Isaias Majil, Mau-Tsuen Yang, and Sophia Yang. 2022. Augmented Reality Based Interactive Cooking Guide. Sensors 22, 21 (2022), 8290. https://doi.org/10.3390/S22218290

[63]

Yuki Matsuura, Tsutomu Terada, Tomohiro Aoki, Susumu Sonoda, Naoya Isoyama, and Masahiko Tsukamoto. 2019. Readability and legibility of fonts considering shakiness of head mounted displays. In Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 150–159.

Digital Library

[64]

Alessandro Mulloni, Andreas Dünser, and Dieter Schmalstieg. 2010. Zooming interfaces for augmented reality browsers. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services. Association for Computing Machinery, New York, NY, USA, 161–170.

Digital Library

[65]

Anton Nijholt. 2022. Towards Social Companions in Augmented Reality: Vision and Challenges. In Distributed, Ambient and Pervasive Interactions. Smart Living, Learning, Well-Being and Health, Art and Creativity: 10th International Conference, DAPI 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II. Springer, Springer-Verlag, Berlin, Heidelberg, 304–319.

Digital Library

[66]

Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: Short papers). Association for Computational Linguistics, Vancouver, Canada, 85–91.

[67]

Mmachi God’sglory Obiorah, Anne Marie Marie Piper, and Michael Horn. 2021. Designing AACs for People with Aphasia Dining in Restaurants. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 496, 14 pages. https://doi.org/10.1145/3411764.3445280

Digital Library

[68]

Jennifer Ockerman and Amy Pritchett. 2000. A review and reappraisal of task guidance: Aiding workers in procedure following. International Journal of Cognitive Ergonomics 4, 3 (2000), 191–212.

[69]

Jason Orlosky, Kiyoshi Kiyokawa, and Haruo Takemura. 2014. Managing mobile text in head mounted displays: studies on visual preference and text placement. ACM SIGMOBILE Mobile Computing and Communications Review 18, 2 (2014), 20–31.

Digital Library

[70]

Gustavo H Paetzold and Lucia Specia. 2017. A survey on lexical simplification. Journal of Artificial Intelligence Research 60 (2017), 549–593.

[71]

Riccardo Palmarini, John Ahmet Erkoyuncu, Rajkumar Roy, and Hosein Torabmostaedi. 2018. A systematic review of augmented reality applications in maintenance. Robotics and Computer-Integrated Manufacturing 49 (2018), 215–228.

[72]

Pranav Parekh, Shireen Patel, Nivedita Patel, and Manan Shah. 2020. Systematic review and meta-analysis of augmented reality in medicine, retail, and games. Visual computing for industry, biomedicine, and art 3 (2020), 1–20.

[73]

Wayne Piekarski and Bruce Thomas. 2002. ARQuake: the outdoor augmented reality gaming system. Commun. ACM 45, 1 (2002), 36–38.

Digital Library

[74]

Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin, John F. Hughes, and Jeff Huang. 2019. Portal-ble: Intuitive Free-hand Manipulation in Unbounded Smartphone-based Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 133–145. https://doi.org/10.1145/3332165.3347904

Digital Library

[75]

Jing Qian, David A. Shamma, Daniel Avrahami, and Jacob Biehl. 2020. Modality and Depth in Touchless Smartphone Augmented Reality Interactions. In Proceedings of the 2020 ACM International Conference on Interactive Media Experiences (Cornella, Barcelona, Spain) (IMX ’20). Association for Computing Machinery, New York, NY, USA, 74–81. https://doi.org/10.1145/3391614.3393648

Digital Library

[76]

Jing Qian, Qi Sun, Curtis Wigington, Han L. Han, Tong Sun, Jennifer Healey, James Tompkin, and Jeff Huang. 2022. Dually Noted: Layout-Aware Annotations with Smartphone Augmented Reality. In CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani (Eds.). ACM, 552:1–552:15. https://doi.org/10.1145/3491102.3502026

Digital Library

[77]

Jing Qian, Tongyu Zhou, Meredith Young-Ng, Jiaju Ma, Angel Cheung, Xiangyu Li, Ian Gonsher, and Jeff Huang. 2021. Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (Virtual Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY, USA, 205–219. https://doi.org/10.1145/3461778.3462098

Digital Library

[78]

Juan Carlos Quiroz, Elena Geangu, and Min Hooi Yong. 2018. Emotion recognition using smart watch sensor data: Mixed-design study. JMIR mental health 5, 3 (2018), e10153.

[79]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.

[80]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67. http://jmlr.org/papers/v21/20-074.html

[81]

Pei-Luen Patrick Rau, Jian Zheng, Zhi Guo, and Jiaqi Li. 2018. Speed reading on virtual reality and augmented reality. Computers & Education 125 (2018), 240–245.

Digital Library

[82]

Luz Rello and Ricardo Baeza-Yates. 2017. How to present more readable text for people with dyslexia. Universal Access in the Information Society 16 (2017), 29–49.

Digital Library

[83]

Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify or help? Text simplification strategies for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (Rio de Janeiro, Brazil). Association for Computing Machinery, New York, NY, USA, Article 15, 10 pages.

[84]

Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013. Frequent words improve readability and short words improve understandability for people with dyslexia. In Human-Computer Interaction – INTERACT 2013. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 203–219.

[85]

Rufat Rzayev, Paweł W Woźniak, Tilman Dingler, and Niels Henze. 2018. Reading on smart glasses: The effect of text position, presentation type and walking. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–9.

Digital Library

[86]

Carolina Scarton, Alessio Palmero Aprosio, Sara Tonelli, Tamara Martín-Wanton, and Lucia Specia. 2017. MUSST: A Multilingual Syntactic Simplification Tool. In Proceedings of the IJCNLP 2017, System Demonstrations, Seong-Bae Park and Thepchai Supnithi (Eds.). Association for Computational Linguistics, Tapei, Taiwan, 25–28. https://aclanthology.org/I17-3007/

[87]

Andreas Schmeil and Wolfgang Broll. 2007. Mara-a mobile augmented reality-based virtual assistant. In 2007 IEEE Virtual Reality Conference. IEEE, IEEE, 267–270.

[88]

Matthew Shardlow. 2014. A survey of automated text simplification. International Journal of Advanced Computer Science and Applications 4, 1 (2014), 58–70.

[89]

Chuhan Shi, Yicheng Hu, Shenan Wang, Shuai Ma, Chengbo Zheng, Xiaojuan Ma, and Qiong Luo. 2023. RetroLens: A Human-AI Collaborative System for Multi-step Retrosynthetic Route Planning. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (, Hamburg, Germany, ) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 770, 20 pages. https://doi.org/10.1145/3544548.3581469

Digital Library

[90]

Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. Association for Computational Linguistics, 373–393. https://aclanthology.org/2022.findings-emnlp.27

[91]

Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023. Getting MoRE out of Mixture of Language Model Reasoning Experts. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 8234–8249. https://doi.org/10.18653/v1/2023.findings-emnlp.552

[92]

Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. 2022. Re-Examining Calibration: The Case of Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2814–2829. https://doi.org/10.18653/v1/2022.findings-emnlp.204

[93]

Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Computation 4 (2006), 77–109.

[94]

Advaith Siddharthan. 2014. A survey of research on text simplification. ITL-International Journal of Applied Linguistics 165, 2 (2014), 259–298.

[95]

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. CoRR abs/2209.11302 (2022). https://doi.org/10.48550/arXiv.2209.11302 arXiv:2209.11302

[96]

Weiwei Sun, Pengjie Ren, and Zhaochun Ren. 2023. Generative Knowledge Selection for Knowledge-Grounded Dialogues. In Findings of the Association for Computational Linguistics: EACL 2023, Andreas Vlachos and Isabelle Augenstein (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 2077–2088. https://doi.org/10.18653/v1/2023.findings-eacl.155

[97]

Arthur Tang, Charles Owen, Frank Biocca, and Weimin Mou. 2003. Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 73–80.

Digital Library

[98]

Lamma Tatwany and Henda Chorfi Ouertani. 2017. A review on using augmented reality in text translation. In 2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA). IEEE, 1–6.

[99]

Bruce H Thomas, Gregory F Welch, Pierre Dragicevic, Niklas Elmqvist, Pourang Irani, Yvonne Jansen, Dieter Schmalstieg, Aurélien Tabard, Neven AM ElSayed, Ross T Smith, 2018. Situated Analytics.Immersive analytics 11190 (2018), 185–220.

[100]

Yuan Tian, Yuxin Ma, Shuxue Quan, and Yi Xu. 2019. Occlusion and collision aware smartphone AR using time-of-flight camera. In Advances in Visual Computing. Springer, Springer International Publishing, Cham, 141–153.

[101]

Antonio E Uva, Michele Gattullo, Vito M Manghisi, Daniele Spagnulo, Giuseppe L Cascella, and Michele Fiorentino. 2018. Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. The International Journal of Advanced Manufacturing Technology 94 (2018), 509–521.

[102]

Chiara Vettori and Ornella Mich. 2011. Supporting deaf children’s reading skills: the many challenges of text simplification. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (Dundee, Scotland, UK) (ASSETS ’11). Association for Computing Machinery, New York, NY, USA, 283–284.

Digital Library

[103]

VIDA-NYU. 2024. ptgctl: A Python Library and Command Line Tool for the PTG API. https://github.com/VIDA-NYU/ptgctl. Available online: https://github.com/VIDA-NYU/ptgctl.

[104]

Boshi Wang, Xiang Deng, and Huan Sun. 2022. Iteratively Prompt Pre-trained Language Models for Chain of Thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. Association for Computational Linguistics, 2714–2730. https://aclanthology.org/2022.emnlp-main.174

[105]

Dakuo Wang, Elizabeth F. Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, Honolulu, HI, USA, April 25-30, 2020, Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjøn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik (Eds.). ACM, 1–6. https://doi.org/10.1145/3334480.3381069

Digital Library

[106]

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: interactive planning with LLMs enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems.

[107]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35. Curran Associates, Inc., 24824–24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf

[108]

Matthias M Wloka and Brian G Anderson. 1995. Resolving occlusion in augmented reality. In Proceedings of the 1995 Symposium on Interactive 3D Graphics (Monterey, California, USA). Association for Computing Machinery, New York, NY, USA, 5–12.

Digital Library

[109]

Guande Wu, Shunan Guo, Jane Hoffswell, Gromit Yeuk-Yin Chan, Ryan A. Rossi, and Eunyee Koh. 2024. Socrates: Data Story Generation via Adaptive Machine-Guided Elicitation of User Feedback. IEEE Transactions on Visualization and Computer Graphics 30, 1 (2024), 131–141. https://doi.org/10.1109/TVCG.2023.3327363

Digital Library

[110]

Guande Wu, Jianzhe Lin, and Cláudio T. Silva. 2022. IntentVizor: Towards Generic Query Guided Interactive Video Summarization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 10493–10502. https://doi.org/10.1109/CVPR52688.2022.01025

[111]

Hsin-Kai Wu, Silvia Wen-Yu Lee, Hsin-Yi Chang, and Jyh-Chong Liang. 2013. Current status, opportunities and challenges of augmented reality in education. Computers & education 62 (2013), 41–49.

[112]

Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, and Shin-Tson Wu. 2021. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light: Science & Applications 10, 1 (2021), 216. https://doi.org/10.1038/s41377-021-00658-8

[113]

Victoria Yaneva. 2015. Easy-read documents as a gold standard for evaluation of text simplification output. In Proceedings of the Student Research Workshop. INCOMA Ltd. Shoumen, BULGARIA, Hissar, Bulgaria, 30–36.

[114]

Xi Ye and Greg Durrett. 2022. Can Explanations Be Useful for Calibrating Black Box Models?. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 6199–6212. https://doi.org/10.18653/v1/2022.acl-long.429

[115]

Xi Ye and Greg Durrett. 2022. The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning. In Advances in Neural Information Processing Systems, Vol. 35. Curran Associates, Inc., 30378–30392. http://papers.nips.cc/paper_files/paper/2022/hash/c402501846f9fe03e2cac015b3f0e6b1-Abstract-Conference.html

[116]

Keyu Zhai, Yiming Cao, Wenjun Hou, and Xueming Li. 2020. Interactive Mixed Reality Cooking Assistant for Unskilled Operating Scenario. In Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications - 12th International Conference, VAMR 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 2020, Proceedings, Part II(Lecture Notes in Computer Science, Vol. 12191), Jessie Y. C. Chen and Gino Fragomeni (Eds.). Springer, 178–195. https://doi.org/10.1007/978-3-030-49698-2_13

Digital Library

[117]

Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. Planning with Large Language Models for Code Generation. In The Eleventh International Conference on Learning Representations. OpenReview.net. https://openreview.net/pdf?id=Lr8cOOtYbfL

[118]

Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing More About Questions Can Help: Improving Calibration in Question Answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021(Findings of ACL, Vol. ACL/IJCNLP 2021), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 1958–1970. https://doi.org/10.18653/v1/2021.findings-acl.172

[119]

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12697–12706. http://proceedings.mlr.press/v139/zhao21c.html

[120]

Chengbo Zheng, Dakuo Wang, April Yi Wang, and Xiaojuan Ma. 2022. Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work. In CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani (Eds.). ACM, 53:1–53:20. https://doi.org/10.1145/3491102.3517615

Digital Library

[121]

Chengbo Zheng, Yuheng Wu, Chuhan Shi, Shuai Ma, Jiehui Luo, and Xiaojuan Ma. 2023. Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (Eds.). ACM, 351:1–351:19. https://doi.org/10.1145/3544548.3581131

Digital Library

[122]

Ting Zheng, Marco Ardolino, Andrea Bacchetti, and Marco Perona. 2021. The applications of Industry 4.0 technologies in manufacturing context: a systematic literature review. International Journal of Production Research 59, 6 (2021), 1922–1954.

[123]

Xianjun Sam Zheng, Cedric Foucault, Patrik Matos da Silva, Siddharth Dasari, Tao Yang, and Stuart Goose. 2015. Eye-wearable technology for machine maintenance: Effects of display position and hands-free operation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2125–2134.

Digital Library

[124]

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. 2022. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision. Springer, Springer Nature Switzerland, Cham, 350–368.

Digital Library

[125]

Jon Zubizarreta, Iker Aguinaga, and Aiert Amundarain. 2019. A framework for augmented reality guidance in industry. The International Journal of Advanced Manufacturing Technology 102 (2019), 4095–4108.

Cited By

Lipkowitz G(2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688875

Index Terms

ARTiST: Automated Text Simplification for Task Guidance in Augmented Reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems

As shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality

Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

May 2024

18961 pages

ISBN:9798400703300

DOI:10.1145/3613904

Editors:
Florian Floyd Mueller
Monash University
,
Penny Kyburz
The Australian National University
,
Julie R. Williamson
University of Glasgow
,
Corina Sas
Lancaster University
,
Max L. Wilson
University of Nottingham
,
Phoebe Toups Dugas
Monash University/New Mexico State University
,
Irina Shklovski
University of Copenhagen

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

DARPA

Conference

CHI '24

Sponsor:

CHI '24: CHI Conference on Human Factors in Computing Systems

May 11 - 16, 2024

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
711
Total Downloads

Downloads (Last 12 months)711
Downloads (Last 6 weeks)254

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lipkowitz G(2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688875

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents