MRTranslate: Bridging Language Barriers in the Physical World Using a Mixed Reality Point-and-Translate System
Article No.: 12, Pages 1 - 9
Abstract
Language barriers pose significant challenges in our increasingly globalized world, hindering effective communication and access to information. Existing translation tools often disrupt the current activity flow and fail to provide seamless user experiences. In this paper, we contribute the design, implementation, and evaluation of MRTranslate, an assistive Mixed Reality (MR) prototype that enables seamless translations of real-world text. We instructed 12 participants to translate items on a food menu using MRTranslate, which we compared to state-of-the-art translation apps, including Google Translate and Google Lens. Findings from our user study reveal that when utilising a fully functional implementation of MRTranslate, participants achieve success in up to 91.67% of their translations whilst also enjoying the visual translation of the unfamiliar text. Although the current translation apps were well perceived, participants particularly appreciated the convenience of not having to grab a smartphone and manually input the text for translation when using MRTranslate. We believe that MRTranslate, along with the empirical insights we have gained, presents a valuable step towards a future where MR transforms language translation and holds the potential to assist individuals in various day-to-day experiences.
Supplemental Material
PDF File
Supplementary material including questions and example study material for the translations.
- Download
- 737.22 KB
References
[1]
Nidhal Baccouri. 2023. DeepTranslator. Retrieved October 24, 2023 from https://pypi.org/project/deep-translator/
[2]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. J. of usability studies.
[3]
Eric A. Bier, Maureen C. Stone, Ken Pier, William Buxton, and Tony D. DeRose. 1993. Toolglass and Magic Lenses: The See-through Interface. In Proc. of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH ’93). ACM, New York, NY, USA.
[4]
Mike Boland. 2023. Google Lens Reaches 10 Billion Monthly Searches. Retrieved October 24, 2023 from https://arinsider.co/2023/02/13/google-lens-reaches-10-billion-monthly-searches
[5]
James V Bradley. 1958. Complete counterbalancing of immediate sequential effects in a Latin square design. J. Amer. Statist. Assoc. 53, 282 (1958).
[6]
John Brooke 1996. SUS-A quick and dirty usability scale. (1996).
[7]
Jas Brooks and Pedro Lopes. 2023. Smell & Paste: Low-Fidelity Prototyping for Olfactory Experiences. In Proc. of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). ACM, New York, NY, USA.
[8]
W. Buxton and B. Myers. 1986. A Study in Two-Handed Input. In CHI (Boston, Massachusetts, USA) (CHI ’86). ACM, New York, NY, USA.
[9]
Shakiba Davari, Feiyu Lu, and Doug A. Bowman. 2022. Validating the Benefits of Glanceable and Context-Aware Augmented Reality for Everyday Information Access Tasks. In IEEE VR.
[10]
Fiona Draxler, Audrey Labrie, Albrecht Schmidt, and Lewis L. Chuang. 2020. Augmented Reality to Enable Users in Learning Case Grammar from Their Real-World Interactions. In Proc. of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). ACM, New York, NY, USA.
[11]
Ruofei Du, Alex Olwal, Mathieu Le Goc, Shengzhi Wu, Danhang Tang, Yinda Zhang, Jun Zhang, David Joseph Tan, Federico Tombari, and David Kim. 2022. Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects into Tangible 6DoF Interfaces Using Ad Hoc UI. In Ext. Abstracts of the CHI Conference (New Orleans, LA, USA) (CHI EA ’22). ACM, New York, NY, USA.
[12]
Pierre Nicolas Durette. 2023. gTTS (Google Text-to-Speech). Retrieved October 24, 2023 from https://pypi.org/project/gTTS/
[13]
Steven K Feiner. 2002. Augmented reality: A new way of seeing. (2002).
[14]
Thomas Franke, Christiane Attig, and Daniel Wessel. 2019. A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale. Int. J. of Human–Computer Interaction (2019).
[15]
Ge Gao, Hao-Chuan Wang, Dan Cosley, and Susan R. Fussell. 2013. Same Translation but Different Experience: The Effects of Highlighting on Machine-Translated Conversations. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). ACM, New York, NY, USA, 10 pages.
[16]
Ge Gao, Naomi Yamashita, Ari MJ Hautasaari, Andy Echenique, and Susan R. Fussell. 2014. Effects of Public vs. Private Automated Transcripts on Multiparty Communication between Native and Non-Native English Speakers. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). ACM, New York, NY, USA.
[17]
Google. 2023. Google Lens - Search What You See. Retrieved October 24, 2023 from https://lens.google/
[18]
Google. 2023. Google Translate. Retrieved October 24, 2023 from https://translate.google.com/about/
[19]
Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. 2017. Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 23, 6 (2017).
[20]
Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior 19, 4 (1987).
[21]
Yves Guiard. 1988. The kinematic chain as a model for human asymmetrical bimanual cooperation. In Advances in Psychology. Vol. 55. Elsevier.
[22]
Kotaro Hara and Shamsi T. Iqbal. 2015. Effect of Machine Translation in Interlingual Conversation: Lessons from a Formative Study. In Proc. of the CHI Conference (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA.
[23]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proc. of the human factors and ergonomics society annual meeting. Sage publications.
[24]
Steven Henderson and Steven Feiner. 2010. Opportunistic Tangible User Interfaces for Augmented Reality. IEEE TVCG 16, 1 (2010).
[25]
Samuel Hoffstaetter. 2023. Python-tesseract. Retrieved October 24, 2023 from https://pypi.org/project/pytesseract/
[26]
Humane. 2023. Humane AI Pin: Beyond touch, beyond screens.Retrieved November 10, 2023 from https://hu.ma.ne/aipin
[27]
Adam Ibrahim, Brandon Huynh, Jonathan Downey, Tobias Höllerer, Dorothy Chun, and John O’donovan. 2018. Arbis pictus: A study of vocabulary learning with augmented reality. IEEE TVCG (2018).
[28]
Itseez 2014. The OpenCV Reference Manual (2.4.9.0 ed.). Itseez.
[29]
Paul Kabbash, William Buxton, and Abigail Sellen. 1994. Two-Handed Input in a Compound Task. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (Boston, Massachusetts, USA). ACM, New York, NY, USA.
[30]
Thomas K Landauer. 1986. Psychology as a mother of invention. ACM (1986).
[31]
Thibault Louis, Jocelyne Troccaz, Amélie Rochet-Capellan, Nady Hoyek, and François Bérard. 2020. When High Fidelity Matters: AR and VR Improve the Learning of a 3D Object. In Proceedings of the International Conference on Advanced Visual Interfaces (Salerno, Italy) (AVI ’20). ACM, New York, NY, USA.
[32]
Feiyu Lu and Doug A. Bowman. 2021. Evaluating the Potential of Glanceable AR Interfaces for Authentic Everyday Uses. In 2021 IEEE VR.
[33]
Feiyu Lu, Leonardo Pavanatto, and Doug A. Bowman. 2023. In-the-Wild Experiences with an Interactive Glanceable AR System for Everyday Use. In Proc. of the 2023 ACM SUI (Sydney, NSW, Australia) (SUI ’23). ACM, New York, NY, USA.
[34]
Jasmine Lu, Ziwei Liu, Jas Brooks, and Pedro Lopes. 2021. Chemical Haptics: Rendering Haptic Sensations via Topical Stimulants. In ACM UIST (Virtual Event, USA) (UIST ’21). ACM, New York, NY, USA.
[35]
Andrés Lucero. 2015. Using Affinity Diagrams to Evaluate Interactive Prototypes. In Human-Computer Interaction – INTERACT 2015, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 231–248.
[36]
Wendy E. Mackay. 1998. Augmented Reality: Linking Real and Virtual Worlds: A New Paradigm for Interacting with Computers. In Proceedings of the Working Conference on Advanced Visual Interfaces (L’Aquila, Italy) (AVI ’98). Association for Computing Machinery, New York, NY, USA.
[37]
Florian Mathis. 2024. Everyday Life Challenges and Augmented Realities: Exploring Use Cases For, and User Perspectives on, an Augmented Everyday Life. In Proceedings of the Augmented Humans International Conference 2024(AHs ’24). Association for Computing Machinery, New York, NY, USA.
[38]
Florian Mathis, Jolie Bonner, Joseph O’Hagan, and Mark McGill. 2023. Breaking Boundaries: Harnessing Mixed Reality to Enhance Social Engagement.
[39]
David Maulsby, Saul Greenberg, and Richard Mander. 1993. Prototyping an intelligent agent through Wizard of Oz. In Proc. of the INTERACT’93 and CHI’93 conference on Human factors in computing systems.
[40]
Meta. 2023. Meta Quest Pro: Built with Privacy in Mind. Retrieved October 24, 2023 from https://www.meta.com/de-de/blog/quest/meta-quest-pro-privacy/
[41]
Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994).
[42]
Shohei Mori, Sei Ikeda, and Hideo Saito. 2017. A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications 9, 1 (2017).
[43]
Joseph O’Hagan, Pejman Saeghe, Jan Gugenheimer, Daniel Medeiros, Karola Marky, Mohamed Khamis, and Mark McGill. 2023. Privacy-Enhancing Technology and Everyday Augmented Reality: Understanding Bystanders’ Varying Needs for Awareness and Consent. IMWUT (2023).
[44]
Cathal O’Madagain, Gregor Kachel, and Brent Strickland. 2019. The origin of pointing: Evidence for the touch hypothesis. Science Advances 5, 7 (2019).
[45]
Ken Pfeuffer, Jason Alexander, and Hans Gellersen. 2016. Partially-Indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction. In Proc. of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA.
[46]
Ingrid Piller. 2022. Language Barriers to Social Participation. Retrieved October 24, 2023 from https://www.languageonthemove.com/language-barriers-to-social-participation/
[47]
Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-Defined Gestures for Augmented Reality. In CHI ’13 Ext. Abstracts on Human Factors in Computing Systems. ACM, New York, NY, USA.
[48]
Michael Rohs, Johannes Schöning, Martin Raubal, Georg Essl, and Antonio Krüger. 2007. Map Navigation with Mobile Devices: Virtual versus Physical Movement with and without Visual Context. In Proc. of the Int. Conference on Multimodal Interfaces (Nagoya, Aichi, Japan). ACM, New York, NY, USA.
[49]
Hanna Kathrin Schraffenberger. 2018. Arguably augmented reality: relationships between the virtual and the real. Ph. D. Dissertation. Leiden University.
[50]
Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Chandima Nanayakkara. 2015. FingerReader: A Wearable Device to Explore Printed Text on the Go. In Proc. of the ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA.
[51]
Maximilian Speicher, Brian D. Hall, and Michael Nebeling. 2019. What is Mixed Reality?. In Proc. of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA.
[52]
Takumi Toyama, Daniel Sonntag, Andreas Dengel, Takahiro Matsuda, Masakazu Iwamura, and Koichi Kise. 2014. A mixed reality head-mounted text translation system using eye gaze input. In Proc. of the International Conference on Intelligent User Interfaces(IUI ’14). ACM, New York, NY, USA.
[53]
Mark Weiser. 1991. The Computer for the 21 st Century. (1991).
[54]
Jacob O. Wobbrock and Julie A. Kientz. 2016. Research Contributions in Human-Computer Interaction. Interactions 23, 3 (apr 2016).
[55]
Shengdong Zhao, Felicia Tan, and Katherine Fennedy. 2023. Heads-Up Computing Moving Beyond the Device-Centered Paradigm. Commun. ACM (2023).
[56]
Annuska Zolyomi, Anushree Shukla, and Jaime Snyder. 2017. Technology-Mediated Sight: A Case Study of Early Adopters of a Low Vision Assistive Technology. In Proc. of the 19th International ACM SIGACCESS Conference on Computers and Accessibility(ASSETS ’17). ACM, New York, NY, USA.
Index Terms
- MRTranslate: Bridging Language Barriers in the Physical World Using a Mixed Reality Point-and-Translate System
Recommendations
Divide and translate: improving long distance reordering in statistical machine translation
WMT '10: Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATRThis paper proposes a novel method for long distance, clause-level reordering in statistical machine translation (SMT). The proposed method separately translates clauses in the source sentence and reconstructs the target sentence using the clause ...
Mixed reality in virtual world teleconferencing
VR '10: Proceedings of the 2010 IEEE Virtual Reality ConferenceIn this paper we present a Mixed Reality (MR) teleconferencing application based on Second Life (SL) and the OpenSim virtual world. Augmented Reality (AR) techniques are used for displaying virtual avatars of remote meeting participants in real physical ...
Lifelogging in Mixed Reality
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing SystemsLifelogging is a growing social phenomenon where personal devices capture records of one’s life for future recalling and reminiscing. At the same time, mixed reality worlds are increasingly available due to advances in head-mounted display technology. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
578 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Data Availability
Supplementary material including questions and example study material for the translations. https://dl.acm.org/doi/10.1145/3656650.3656652#AVI2024_supplementary.pdf
Funding Sources
- International Postdoctoral Fellowships (IPF) at the University of St. Gallen
Conference
AVI 2024
AVI 2024: International Conference on Advanced Visual Interfaces 2024
June 3 - 7, 2024
Arenzano, Genoa, Italy
Acceptance Rates
AVI '24 Paper Acceptance Rate 21 of 82 submissions, 26%;
Overall Acceptance Rate 128 of 490 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 188Total Downloads
- Downloads (Last 12 months)188
- Downloads (Last 6 weeks)51
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in