research-article

Interaction Challenges in AI Equipped Environments Built to Teach Foreign Languages Through Dialogue and Task-Completion

Authors:

Rahul R. Divekar,

Jaimie Drozdal,

Robert Rouhani,

Lilit Balagyozyan,

Hui SuAuthors Info & Claims

DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

Pages 597 - 609

https://doi.org/10.1145/3196709.3196717

Published: 08 June 2018 Publication History

Abstract

As cities around the world become more diverse in culture and language, there is a growing need for learning foreign languages. To further this excitement, we have built a human-scale, immersive room with a virtual AI agent that aids foreign language learning. Our system aids the language learning process through task-completion exercises using multi-modal dialogue. The Cognitive and Immersive Room (CIR) is developed as an immersive Chinese restaurant to teach Mandarin, but the interaction challenges and solutions can be reasonably generalized to other languages taught using similar techniques. As users interact with the immersive environment and the virtual AI agent, they face several user interaction challenges. These challenges arise from new learners' lack of proficiency in the foreign language. By studying user interactions in the CIR, we were able to articulate some of the interaction challenges. We have enhanced the AI agent, virtual environment, and the on-boarding process for new users to mitigate these challenges. The enhancements and the results which show that they were effective are discussed here.

References

[1]

2009. Computer Assisted Language Learning: Critical Concepts in Linguistics. Vol. I. Routledge.

[2]

2013. Modern Chinese Textbook 1A. Better Chinese.

[3]

2017. Effective Language Learning. (Oct 2017). http://www.effectivelanguagelearning.com/ language-guide/language-difficulty

[4]

2018. (2018). http://www.activeworlds.com/

[5]

2018. (2018). http://secondlife.com/

[6]

2018. Amazon Alexa. (2018). https://developer.amazon.com/alexa

[7]

2018. Busuu. (2018). https://www.busuu.com

[8]

2018. CAVE Automatic Virtual Environment. (2018). http://www.visbox.com/products/cave/

[9]

2018. Chinese Pod. (2018). https://chinesepod.com

[10]

2018. eChineseLearning. (2018). https://www.echineselearning.com

[11]

2018. Google Assistant. (2018). https://assistant.google.com

[12]

2018. Hello Talk. (2018). http://home.hellotalk.com

[13]

2018. Mandarin (Chinese) | About World Languages. (2018). http://aboutworldlanguages.com/mandarin

[14]

2018. Smallab Learning. (2018). https://www.smallablearning.com/

[15]

2018. Tutor Mandarin. (2018). https://www.tutormandarin.net/en

[16]

2018. Wai Chinese. (2018). https://waichinese.com/features.html

[17]

2018. Watson Assistant. (2018). https://www.ibm.com/watson/services/conversation/

[18]

2018. Watson Speech To Text. (2018). https://www.ibm.com/watson/services/speech-to-text/

[19]

Muna Mohammed Abbas Alkhateeb. 2017. An Analytical Approach of the Cultural Context in Language Teaching and Learning. Journal of University of Babylon 25, 2 (2017), 832â ? A¸ S840. https: //doaj.org/article/d1e436f141a6483b832b2895c02dd40e?

[20]

Anke Berns, Antonio Gonzalez-Pardo, and David Camacho. 2013. Game-like language learning in 3-D virtual environments. 60 (01 2013), 210--220.

Digital Library

[21]

David Birchfield and Colleen Megowan-Romanowicz. 2009. Earth science learning in SMALLab: A design experiment for mixed reality. International Journal of Computer-Supported Collaborative Learning 4, 4 (01 Dec 2009), 403--421.

[22]

Lynne Bolduc. 1997. Mastering Difficult Concepts Using Software to Support Learning: A Case Study in Teaching Mandarin Chinese. Journal of Science Education and Technology 6, 2 (1997), 91--102.

[23]

Richard Brecht, Dan Davidson, and Ralph Ginsberg. 1995. Predictors of foreign language gain during study abroad. Second language acquisition in a study abroad context 9 (1995), 37.

[24]

Silvia Canto and Kristi Jauregi Ondarra. 2017. Language learning effects through the integration of synchronous online communication: The case of video communication and Second Life. Language Learning in Higher Education 7, 1 (2017), 21--53.

[25]

Dongdong Chen. 2010. Enhancing the learning of Chinese with Second Life. Journal of Technology and Chinese Language Teaching 1, 1 (2010), 14--30.

[26]

Nancy F Chen, Darren Wee, Rong Tong, Bin Ma, and Haizhou Li. 2016. Large-scale characterization of non-native Mandarin Chinese spoken by speakers of European origin: Analysis on iCALL. Speech Communication 84 (2016), 46--56.

Digital Library

[27]

Tina Cok. 2016. ICT-supported language learning tools for Chinese as a foreign Language: a content review/Jezikovnotehnoloska orodja za ucenje kitajscine kot tujega jezika: pregled vsebine. Revija za Elementarno Izobrazevanje 9, 3 (06 2016), 103--120. http://libproxy.rpi.edu/login?url=https://search. proquest.com/docview/1799383943?accountid=28525 Copyright - Copyright Univerza v Mariboru, Faculty of Education/Pedagoska Fakulteta Jun 2016; Document feature - Tables; ; Graphs; Last updated - 2016-06--25.

[28]

Dan E Davidson. 2010. Study abroad: When, how long, and with what results? New data from the Russian front. Foreign Language Annals 43, 1 (2010), 6--26.

[29]

Oxana Dema and Aleidine Kramer Moeller. 2012. Teaching culture in the 21st century language classroom. Faculty Publications: Dept. of Teaching, Learning and Teacher Education (2012), 75â ? A¸ S91.

[30]

Mats Deutschmann, Luisa Panichi, and Judith Molka-Danielsen. 2009. Designing oral participation in Second Life--a comparative study of two language proficiency courses. ReCALL 21, 2 (2009), 206--226.

Digital Library

[31]

Wenhao Diao. 2011. Study abroad, participation and turn taking: A case study. In Selected Proceedings of the 2010 Second Language Research Forum. Cascadilla Proceedings.

[32]

Rahul R. Divekar, Robert Rouhani, Matthew Peveler, Rui Zhao, Jeffrey O. Kephart, David Allen, Kang Wang, Qiang Ji, and Hui Su. CIRA - An Architecture for Building Configurable Immersive Smart-rooms. In To appear in proceedings of Intellisys 2018.

[33]

Hang Du. 2013. The development of Chinese fluency during study abroad in China. The Modern Language Journal 97, 1 (2013), 131--143.

[34]

Michael E Everson. 2011. Best practices in teaching logographic and non-Roman writing systems to L2 learners. Annual Review of Applied Linguistics 31 (2011), 249--274.

[35]

Serhat Güzel and Selami Aydin. 2016. The Effect of Second Life on Speaking Achievement. Online Submission 6, 4 (2016), 236--245.

[36]

Michael Henderson, Hui Huang, Scott Grant, and Lyn Henderson. 2009. Language acquisition in Second Life: Improving self-efficacy beliefs. (2009).

[37]

Murat Hismanoglu. 2012. Integrating second life into an EFL classroom: A new dimension in foreign language learning and teaching. 3 (01 2012), 100--111.

[38]

Yu-Ju Lan, Yu-Hsuan Kan, Indy Y.T. Hsiao, Stephen J.H. Yang, and Kuo-En Chang. 2013. Designing interaction tasks in Second Life for Chinese as a foreign language learners: A preliminary exploration. Australasian Journal of Educational Technology 29, 2 (2013).

[39]

Mark Lee and Barney Dalgarno. 2011. Scaffolding Discovery Learning in 3D Virtual Environments: Challenges and Considerations for Instructional Design. IGI Global, USA, 138--169. Imported on 12 May 2017 - DigiTool details were: publisher = Hershey, PA: IGI-Global, 2011. editor/s (773b) = Shalin Hai-Jew; Issue no. (773s) = 8; Parent title (773t) = Virtual immersive and 3D learning spaces: emerging technologies and trends.

[40]

Hsien-Cheng Liao, Ying-Hua Guan, Jia-Jang Tu, and Jiang-Chun Chen. 2014. A prototype of an adaptive Chinese pronunciation training system. System 45 (2014), 52--66.

[41]

Yuen-kuang Liao and Yu-wen Chen. 2007. The effect of computer simulation instruction on student learning: A meta-analysis of studies in Taiwan. Journal of information Technology and Applications 2, 2 (2007), 69--79.

[42]

Tsun-Ju Lin and Yu-Ju Lan. 2015. Language Learning in Virtual Reality Environments: Past, Present, and Future. Educational Technology & Society 18, 4 (2015), 486--497.

[43]

James Milton, Sunniva Jonsen, Steven Hirst, and Sharn Lindenburn. 2012. Foreign language vocabulary development through activities in an online 3D environment. The Language Learning Journal 40, 1 (2012), 99--112.

[44]

Corey Mitchell. 2016. One million U.S. students could be studying Mandarin by 2020. (Aug 2016). http://www.pbs.org/newshour/rundown/ 2020-u-s-aims-1-million-students-studying-mandarin/

[45]

Max North and Sarah North. 2016. A Comparative Study of Sense of Presence of Traditional Virtual Reality and Immersive Environments. Australasian Journal of Information Systems 20, 0 (2016).

[46]

Mary G. O'Brien, Richard Levy, and Annika Orich. 2009. Virtual Immersion: The Role of CAVE and PC Technology. CALICO Journal 26, 2 (01 2009), 337. http://libproxy.rpi.edu/login?url=https: //search-proquest-com.libproxy.rpi.edu/docview/ 750619686?accountid=28525 Copyright - Copyright CALICO Jan 2009; Last updated - 2014--12-07.

[47]

Mark Peterson. 2011. Towards a Research Agenda for the Use of Three-Dimensional Virtual Worlds in Language Learning. CALICO Journal 29, 1 (2011), 67--80. http://www.jstor.org/stable/calicojournal.29.1.67

[48]

Thomas C Reeves and John G Hedberg. 2003. Chapter 1: A Pragmatic Rationale for Evaluation. Education Technology, 7â ? A¸ S9.

[49]

Wei Ren. 2013. The effect of the study abroad on the pragmatic development in internal modification of refusals. 23 (12 2013), 715--741.

[50]

Roger C. Schank. 1990. Case-Based Teaching: Four Experiences in Educational Software Design. Interactive Learning Environments 1 (1990), 231--253.

[51]

Helen H Shen and Wenjing Xu. 2015. Active learning: Qualitative inquiries into vocabulary instruction in Chinese L2 classrooms. Foreign Language Annals 48, 1 (2015), 82--99.

[52]

Ya-Chun Shih. 2007. The Development of an Online Virtual English Classroom: VEC3D.

[53]

Ya-Chun Shih. 2017. Weaving Web 2.0 and Facial Expression Recognition Into the 3D Virtual English Classroom. Vol. I-III. IGI Global, 1375â ? A¸ S1399.

[54]

Naoko Taguchi, Qiong Li, and Xiaofei Tang. 2017. Learning Chinese Formulaic Expressions in a Scenario-Based Interactive Environment. 50 (10 2017).

[55]

Etsuko Toyoda. 2002. Categorization of Text Chat Communication between Learners and Native Speakers of Japanese.

[56]

Jennifer J. Vogel, David S. Vogel, Jan Cannon-Bowers, Clint A. Bowers, Kathryn Muse, and Michelle Wright. 2006. Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis. Journal of Educational Computing Research 34, 3 (2006), 229--243.

[57]

Airong Wang. 2017. Learning English in a Multi-User Virtual Environment: Exploring Factors Affecting Participation. Ph.D. Dissertation. Mid Sweden University.

[58]

Endymion Porter. Wilkinson. 2015. Chinese History: A New Manual (4 ed.). Harvard University Asia Center for the Harvard-Yenching Institute distributed by Harvard University Press.

[59]

Clare Wright and Z Cong. 2014. Examining the effects of Study Abroad on L2 Chinese development among UK university learners. Newcastle and Northumbria Working Papers in Linguistics 20 (2014), 67--83.

[60]

Zhou Yalun. 2016. Digital Vocabulary Competition as Motivator for Learning in CFL Classrooms. JOURNAL OF TECHNOLOGY AND CHINESE LANGUAGE TEACHING 7, 2 (2016), 1--22.

[61]

Kasumi Yamazaki. 2018. Computer-assisted learning of communication (CALC): A case study of Japanese learning in a 3D virtual world. ReCALL (2018), 1â ? A¸ S18.

[62]

Rui Zhao, Kang Wang, Rahul Divekar, Robert Rouhani, Hui Su, and Qiang Ji. 2018. An Immersive System with Multi-modal Human-computer Interaction. In The IEEE Conference on Automatic Face and Gesture Recognition (FG).

[63]

Wenjing Zuo. 2011. The Effects of Cooperative Learning on Improving College Studentsâ ? A´ Z Reading Comprehension. Theory and Practice in Language Studies 1, 8 (Jan 2011).

Cited By

Zhou YDivekar R(2025)Immersive, Task-Based Language Learning Through XR and AI: From Design Thinking to DeploymentTechTrends10.1007/s11528-025-01048-2Online publication date: 4-Feb-2025
https://doi.org/10.1007/s11528-025-01048-2
Nalyvaiko OMalysh KPrykhodko YChaban S(2024)Neural networks at the service of education: challenges of the new era of educational transformationThe Scientific Notes of the Pedagogical Department10.26565/2074-8167-2024-54-09(98-109)Online publication date: 26-Aug-2024
https://doi.org/10.26565/2074-8167-2024-54-09
Bryan M(2023)Centering Software: Drawing on UX Research to Develop Methods of Collaborative Inquiry into Software Rhetorics within Writing CentersProceedings of the 41st ACM International Conference on Design of Communication10.1145/3615335.3623006(26-31)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3615335.3623006
Show More Cited By

Index Terms

Interaction Challenges in AI Equipped Environments Built to Teach Foreign Languages Through Dialogue and Task-Completion
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Impacts of augmented reality on foreign language teaching: a case study of Persian language
Abstract
The use of information technology in the field of foreign language teaching as an auxiliary tool is very important. In a foreign language classroom, place is just an abstract concept; where the language is separated from the community, culture and ...
Languages and cognition: towards new CALL
ICAIT'2018: Proceedings of the 3rd International Conference on Applications in Information Technology

We address cognitive models and prototype theory in context of foreign language learning and examine some ideas that seem to favour Computer-assisted language learning tools (CALL tools) evolution alongside we present a software prototype of a learning ...
Learning Foreign Languages With Duolingo and Memrise
ICDEL '18: Proceedings of the 2018 International Conference on Distance Education and Learning

Mobile-assisted language learning (MALL) has been prevalent recently due to the advance of technology. In this paper, the writer is using auto ethnography method to compare the efficacy of two language learning apps: Duolingo and Memrise for learning ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

June 2018

1418 pages

ISBN:9781450351980

DOI:10.1145/3196709

General Chairs:
Ilpo Koskinen
University of Twente
,
Youn-kyung Lim
KAIST
,
Program Chairs:
Teresa Cerratto-Pargman
Stockholm University
,
Kenny Chow
The Hong Kong Polytechnic University
,
William Odom
Simon Fraser University

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

IBM

Conference

DIS '18

Sponsor:

SIGCHI

DIS '18: Designing Interactive Systems Conference 2018

June 9 - 13, 2018

Hong Kong, China

Acceptance Rates

DIS '18 Paper Acceptance Rate 107 of 487 submissions, 22%;

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Upcoming Conference

DIS '25

Sponsor:
sigchi

Designing Interactive Systems Conference

July 5 - 9, 2025

Funchal , Portugal

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)5

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou YDivekar R(2025)Immersive, Task-Based Language Learning Through XR and AI: From Design Thinking to DeploymentTechTrends10.1007/s11528-025-01048-2Online publication date: 4-Feb-2025
https://doi.org/10.1007/s11528-025-01048-2
Nalyvaiko OMalysh KPrykhodko YChaban S(2024)Neural networks at the service of education: challenges of the new era of educational transformationThe Scientific Notes of the Pedagogical Department10.26565/2074-8167-2024-54-09(98-109)Online publication date: 26-Aug-2024
https://doi.org/10.26565/2074-8167-2024-54-09
Bryan M(2023)Centering Software: Drawing on UX Research to Develop Methods of Collaborative Inquiry into Software Rhetorics within Writing CentersProceedings of the 41st ACM International Conference on Design of Communication10.1145/3615335.3623006(26-31)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3615335.3623006
Jeon J(2022)Exploring AI chatbot affordances in the EFL classroom: young learners’ experiences and perspectivesComputer Assisted Language Learning10.1080/09588221.2021.202124137:1-2(1-26)Online publication date: 10-Jan-2022
https://doi.org/10.1080/09588221.2021.2021241
Peixoto BMelo MCabral LBessa M(2021)Evaluation of Animation and Lip-Sync of Avatars, and User Interaction in Immersive Virtual Reality Learning Environments2021 International Conference on Graphics and Interaction (ICGI)10.1109/ICGI54032.2021.9655283(1-7)Online publication date: 4-Nov-2021
https://doi.org/10.1109/ICGI54032.2021.9655283
Peixoto BPinto RMelo MCabral LBessa M(2021)Immersive Virtual Reality for Foreign Language Education: A PRISMA Systematic ReviewIEEE Access10.1109/ACCESS.2021.30688589(48952-48962)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3068858
Divekar* RDrozdal* JChabot* SZhou YSu HChen YZhu HHendler JBraasch J(2021)Foreign language acquisition via artificial intelligence and extended reality: design and evaluationComputer Assisted Language Learning10.1080/09588221.2021.187916235:9(2332-2360)Online publication date: 15-Feb-2021
https://doi.org/10.1080/09588221.2021.1879162
Divekar RLepp HChopade PAlbin ABrenner DRamanarayanan V(2021)Conversational Agents in Language Education: Where They Fit and Their Research ChallengesHCI International 2021 - Late Breaking Posters10.1007/978-3-030-90179-0_35(272-279)Online publication date: 6-Nov-2021
https://doi.org/10.1007/978-3-030-90179-0_35
Divekar RSu HKephart JDeBayser MGuerra MMou XPeveler MChen LBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)HUMAINE: Human Multi-Agent Immersive Negotiation CompetitionExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3383001(1-10)Online publication date: 25-Apr-2020
https://dl.acm.org/doi/10.1145/3334480.3383001
Divekar RKephart JMou XChen LSu H(2019)You Talkin’ to Me? A Practical Attention-Aware Embodied AgentHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29387-1_44(760-780)Online publication date: 2-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-29387-1_44
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten