research-article

Improving Multilingual Collaboration by Displaying How Non-native Speakers Use Automated Transcripts and Bilingual Dictionaries

Authors:

Naomi Yamashita,

Ari M.J. Hautasaari,

Susan R. FussellAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 3463 - 3472

https://doi.org/10.1145/2702123.2702498

Published: 18 April 2015 Publication History

Abstract

Conversational grounding, or establishing mutual knowledge that messages have been understood as intended, can be difficult to achieve when some conversational participants are using a non-native language. These difficulties in grounding can be challenging for native speakers to detect. In this paper, we examine the value of signaling potential grounding problems to native speakers (NS) by displaying how non-native speakers (NNS) use automated transcripts and bilingual dictionaries. We conducted a laboratory experiment in which NS and NNS of English collaborated via audio conferencing on a map navigation task. Triads of one NS guider, one NS follower, and one NNS follower performed the task using one of three awareness displays: (a) a no awareness display that showed only the automated transcripts, (b) a general awareness display that showed whether each follower was reading the automated transcripts and/or translating a word; or (c) a detailed awareness display that showed which line of the transcripts a follower was reading and/or which words he/she was translating. NS guiders and NNS followers collaborated most successfully with the detailed awareness display, while NS guiders and NS followers performed equally across conditions. Our findings suggest several ways to improve systems to support multilingual collaboration.

References

[1]

Anderson, A., Bader, M., Bard, E., et al. (1991). The HCRC Map Task Corpus. Language and Speech, 34, 351--366.

[2]

Bardram, J. E., Hansen, T. R., & Soegaard, M. (2006). AwareMedia- A shared interative display supporting social, temporal, and spatial awareness in surgery. Proc. of CSCW 2006, 109--118.

Digital Library

[3]

Beebe, L.M., & Giles, H. (1984). Speechaccommodation theories: A discussion in terms of second-language acquisition. International Journal of the Sociology of Language, 46, 5--32,

[4]

Birnholtz, J., & Ibara, S. (2012). Tracking changes in collaborative writing: Edits, visibility, and group maintenance. Proc. of CSCW 2012, 809--818.

Digital Library

[5]

Carter, J., & Dewan, P. (2010). Are you having difficulty? Proc. of CSCW 2010, 211--214.

Digital Library

[6]

Cheung, V., Chang, Y. L. B., & Scott, S. D. (2012). Communication channels and awareness cues in collocated collaborative time-critical gaming. Proc. of CSCW 2012, 569--578.

Digital Library

[7]

Clark, H.H. (1996). Using Language. Cambridge Press.

[8]

Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley, (Eds.). Perspectives on Socially Shared Cognition (pp. 127--149). APA Press.

[9]

Dabbish, L., & Kraut, R. (2008). Awareness displays and social motivation for coordinating communication. Information Systems Research, 19, 221--238.

Digital Library

[10]

Diamant, E.I., Fussell, S. R., & Lo, F. L. (2008). Where did we turn wrong? Unpacking the effects of culture and technology on attributions of team performance. Proc. CSCW 2008, 383--391.

Digital Library

[11]

Dourish, P., & Bellotti, V. (1992). Awareness and coordination in shared workspaces. Proc. of CSCW 1992, 107--114.

Digital Library

[12]

Feely, A. J., & Harzing, A. W. K. (2003). Language management in multinational companies. CrossCultural Management: An Intl'l Journal, 10, 27--52.

[13]

Fung, L., & Carter, R. (2007). Discourse markers and spoken English: Native and learner use in pedagogic settings. Applied Linguistics, 28, 410--439.

[14]

Gao, G., Yamashita, N., Hautasaari, A., Echenique, A., & Fussell, S. R. (2014). Effects of public vs. private automated transcripts on multiparty communication between native and non-native English speakers. Proc. of CHI 2014, 843--852.

Digital Library

[15]

Gergle, D., Kraut, R. E., & Fussell, S. R. (2013). Using visual information for grounding and awareness in collaborative tasks. Human-Computer Interaction, 28, 1--39.

[16]

Gibson, C. B., & Gibbs, J. L. (2006). Unpacking the concept of virtuality: The effects of geographic dispersion, electronic dependence, dynamic structure, and national diversity on team innovation. Administrative Science Quarterly, 51, 451--495.

[17]

Hart, S. G., & Staveland, L. E. (1998). Development of NASA-TLX: Results of empirical and theoretical research. Advances in Psychology, 52, 139--183.

[18]

Henderson, J.K. (2005). Language diversity in international management teams. International Studies of Management and Organization, 35, 66--82.

[19]

Ishida, T. (2011). The language grid. Springer Press.

[20]

Kraut, R. E., Fussell, S. R., & Siegel, J. (2003). Visual information as a conversational resource in collaborative physical tasks. Human Computer Interaction, 18, 13--49.

Digital Library

[21]

Li, H. Z., Yum, Y., Yates, R., Aguilera, L., Mao, Y., & Zheng, Y. (2005). Interruption and involvement in discourse: Can intercultural interlocutors be trained? J. Intercultural Communication Research, 34, 233--254.

[22]

Li, H. Z. (1999). Grounding and information communication in intercultural and intracultural dyadic discourse. Discourse Processes, 28, 195--215.

[23]

Lim, B. Y., Brdiczka, O., & Bellotti, V. (2010). Show me a good time: Using content to provide activity awareness to collaborators with Activity Spotter. Proc. of GROUP 2010, 263--272.

Digital Library

[24]

Reynolds, L., Birnholtz, J., & Lee, A. (2012). The effect of communication channel and visual awareness display on coordination in online tasks. Proc. iConference 2012, 120--128

Digital Library

[25]

Rogerson-Revell, P. (2008). Participation and performance in international business meetings. English for Specific Purposes, 27, 338--360.

[26]

Setlock, L. D., Fussell, S. R., & Neuwirth, C. (2004). Taking it out of context: Collaborating within and across cultures in face-to-face settings and via instant messaging. Proc. CSCW 2004, 604--613.

Digital Library

[27]

Takano, Y., & Noda, A. (1993). A temporary decline of thinking ability during foreign language processing. Journal of Cross-Cultural Psychology, 24, 445--462.

[28]

Tange, H., & Lauring, J. (2009). Language management and social interaction within the multilingual workplace. J. Communication Management, 13, 218--232.

[29]

Wang, H. C., Fussell, S. R., & Cosley, D. (2013). Machine translation vs. common language: Effects on idea exchange in cross-lingual groups. Proc. of CSCW 2013, 935--944.

Digital Library

[30]

Wang, H-C., Fussell, S. R. & Setlock, L. D. (2009). Cultural difference and adaptation of communication styles in computer-mediated group brainstorming. Proc. CHI 2009, 669--678.

Digital Library

[31]

Wong, J. (2000). The token" yeah" in nonnative speaker English conversation. Research on Language and Social Interaction, 33, 39--67.

[32]

Yamashita, N., Echenique, A., Ishita, T., & Hautasaari, A. (2013). Lost in translation: How transmission lag enhances and deteriorates multilingual collaboration. In Proc. of CSCW 2013, 923--934.

Digital Library

Cited By

Hautasaari AAramaki MChujo RNaemura T(2024)EmoScribe Camera: A Virtual Camera System to Enliven Online Conferencing with Automatically Generated Emotional Text CaptionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650987(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650987
Hwang AWon A(2024)The Sound of Support: Gendered Voice Agent as Support to Minority Teammates in Gender-Imbalanced TeamProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642202(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642202
Huang WAlem LTecchia FDuh HHuang WBillinghurst MAlem LXiao CRasmussen T(2024) Augmenting Hand Gestures in 3D Mixed Reality Computer‐Supported Collaboration10.1002/9781119719830.ch10(243-268)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch10
Show More Cited By

Index Terms

Improving Multilingual Collaboration by Displaying How Non-native Speakers Use Automated Transcripts and Bilingual Dictionaries
1. Human-centered computing

Recommendations

Effects of public vs. private automated transcripts on multiparty communication between native and non-native english speakers
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Real-time transcripts generated by automated speech recognition (ASR) technologies have the potential to facilitate communication between native speakers (NS) and non-native speakers (NNS). Previous studies of ASR have focused on how transcripts aid NNS ...
Multilingual recognition of non-native speech using acoustic model transformation and pronunciation modeling

This article presents an approach for the automatic recognition of non-native speech. Some non-native speakers tend to pronounce phonemes as they would in their native language. Model adaptation can improve the recognition rate for non-native speakers, ...
Investigating the impact of automated transcripts on non-native speakers' listening comprehension
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Real-time transcripts generated by automatic speech recognition (ASR) technologies hold potential to facilitate non-native speakers’ (NNSs) listening comprehension. While introducing another modality (i.e., ASR transcripts) to NNSs provides ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
697
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)6

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hautasaari AAramaki MChujo RNaemura T(2024)EmoScribe Camera: A Virtual Camera System to Enliven Online Conferencing with Automatically Generated Emotional Text CaptionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650987(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650987
Hwang AWon A(2024)The Sound of Support: Gendered Voice Agent as Support to Minority Teammates in Gender-Imbalanced TeamProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642202(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642202
Huang WAlem LTecchia FDuh HHuang WBillinghurst MAlem LXiao CRasmussen T(2024) Augmenting Hand Gestures in 3D Mixed Reality Computer‐Supported Collaboration10.1002/9781119719830.ch10(243-268)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch10
Zhang RDuan WFlathmann CMcNeese NFreeman GWilliams A(2023)Investigating AI Teammate Communication Strategies and Their Impact in Human-AI Teams for Effective TeamworkProceedings of the ACM on Human-Computer Interaction10.1145/36100727:CSCW2(1-31)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610072
Glikson EAsscher O(2023)AI-mediated apology in a multilingual work contextComputers in Human Behavior10.1016/j.chb.2022.107592140:COnline publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1016/j.chb.2022.107592
Li XYamashita NDuan WShirai YFussell S(2022)Improving Non-Native Speakers' Participation with an Automatic Agent in Multilingual GroupsProceedings of the ACM on Human-Computer Interaction10.1145/35675627:GROUP(1-28)Online publication date: 29-Dec-2022
https://dl.acm.org/doi/10.1145/3567562
Duan WYamashita NShirai YFussell S(2021)Bridging Fluency Disparity between Native and Nonnative Speakers in Multilingual Multiparty Collaboration Using a Clarification AgentProceedings of the ACM on Human-Computer Interaction10.1145/34795795:CSCW2(1-31)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479579
Fernando PKuznetsov SBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)OScH in the Wild: Dissemination of Open Science Hardware and Implications for HCIProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376659(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376659
Liao YIeiri YHishiyama RChen KMa Y(2019)A proposal of interaction analysis framework for multilingual discussionProceedings of the 10th International Conference on E-Education, E-Business, E-Management and E-Learning10.1145/3306500.3306561(421-426)Online publication date: 10-Jan-2019
https://dl.acm.org/doi/10.1145/3306500.3306561
CAO XYAMASHITA NISHIDA T(2018)Effects of Automated Transcripts on Non-Native Speakers' Listening ComprehensionIEICE Transactions on Information and Systems10.1587/transinf.2017EDP7255E101.D:3(730-739)Online publication date: 2018
https://doi.org/10.1587/transinf.2017EDP7255
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents