research-article

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Authors:

Ronald E Robertson,

Alexandra Olteanu,

Milad Shokouhi,

Peter BaileyAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 724, Pages 1 - 18

https://doi.org/10.1145/3411764.3445557

Published: 07 May 2021 Publication History

Abstract

In email interfaces, providing users with reply suggestions may simplify or accelerate correspondence. While the “success” of such systems is typically quantified using the number of suggestions selected by users, this ignores the impact of social context, which can change how suggestions are perceived. To address this, we developed a mixed-methods framework involving qualitative interviews and crowdsourced experiments to characterize problematic email reply suggestions. Our interviews revealed issues with over-positive, dissonant, cultural, and gender-assuming replies, as well as contextual politeness. In our experiments, crowdworkers assessed email scenarios that we generated and systematically controlled, showing that contextual factors like social ties and the presence of salutations impacts users’ perceptions of email correspondence. These assessments created a novel dataset of human-authored corrections for problematic email replies. Our study highlights the social complexity of providing suggestions for email correspondence, raising issues that may apply to all social messaging systems.

References

[1]

Herman Aguinis and Kyle J Bradley. 2014. Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods 17, 4 (2014), 351–371.

[2]

Cheryl S Alexander and Henry Jay Becker. 1978. The use of vignettes in survey research. Public Opinion Quarterly 42, 1 (1978), 93–104.

[3]

Sieun An, Li-Jun Ji, Michael Marks, and Zhiyong Zhang. 2017. Two sides of emotion: exploring positivity and negativity in six basic emotions across cultures. Frontiers in Psychology 8 (2017), 610.

[4]

Lynne M. Andersson and Christine M. Pearson. 1999. Tit for Tat? The Spiraling Effect of Incivility in the Workplace. Academy of Management Review 24, 3 (July 1999), 452–471.

[5]

Kenneth Arnold, Krysta Chauncey, and Krzysztof Gajos. 2018. Sentiment Bias in Predictive Text Recommendations Results in Biased Writing. Proceedings of Graphics Interface 2018 Toronto (2018), 8 pages, 730.87 KB.

[6]

Kenneth C Arnold, Krysta Chauncey, and Krzysztof Z Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 128–138.

Digital Library

[7]

Kenneth C. Arnold, Krzysztof Z. Gajos, and Adam T. Kalai. 2016. On Suggesting Phrases vs. Predicting Words for Mobile Text Composition. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST ’16. ACM Press, Tokyo, Japan, 603–608.

Digital Library

[8]

John Langshaw Austin. 1962. How to Do Things with Words: The William James Lectures Delivered at Harvard University in 1955. Technical Report. Oxford University Press, Oxford.

[9]

Ricardo Baeza-Yates. 2018. Bias on the Web. Commun. ACM 61, 6 (2018), 54–61.

Digital Library

[10]

Naomi S. Baron. 1998. Letters by Phone or Speech by Other Means: The Linguistics of Email. Language & Communication 18, 2 (April 1998), 133–170.

[11]

Asaf Beasley and Winter Mason. 2015. Emotional States vs. Emotional Words in Social Media. In Proceedings of the ACM Web Science Conference. ACM, Oxford United Kingdom, 1–10.

Digital Library

[12]

Boomerang. 2020. Respondable: Write Better Email. https://www.boomeranggmail.com/respondable.

[13]

Pia Borlund and Peter Ingwersen. 1997. The Development of a Method for the Evaluation of Interactive Information Retrieval Systems. Journal of Documentation 53, 3 (Aug. 1997), 225–250.

[14]

Jerry Boucher and Charles E. Osgood. 1969. The Pollyanna Hypothesis. Journal of Verbal Learning and Verbal Behavior 8, 1 (Feb. 1969), 1–8.

[15]

Penelope Brown and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Number 4 in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge [Cambridgeshire] ; New York.

[16]

Ulla Bunz and Scott W. Campbell. 2004. Politeness Accommodation in Electronic Mail. Communication Research Reports 21, 1 (Jan. 2004), 11–25.

[17]

Moira Burke, Justin Cheng, and Bethany de Gant. 2020. Social Comparison and Facebook: Feedback, Positivity, and Opportunities for Comparison. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13.

Digital Library

[18]

Robin Campbell and Roger Wales. 1970. The Study of Language Acquisition. In New Horizons in Linguistics, J. Lyons (Ed.). Penguin, Harmondsworth, 242–260.

[19]

Christina Cauterucci. 2015. New Chrome App Helps Women Stop Saying “Just” and “Sorry” in Emails. https://slate.com/human-interest/2015/12/new-chrome-app-helps-women-stop-saying-just-and-sorry-in-emails.html.

[20]

Praveen Chandar, Jean Garcia-Gathright, Christine Hosey, Brian St. Thomas, and Jennifer Thom. 2019. Developing Evaluation Metrics for Instant Search Using Mixed Methods Methods. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR’19. ACM Press, Paris, France, 925–928.

Digital Library

[21]

Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. Gmail Smart Compose: Real-Time Assisted Writing. arXiv:1906.00080 [cs] (May 2019). arxiv:1906.00080 [cs]

[22]

Innocent Chiluwa. 2010. Nigerian English in Informal Email Messages. English World-Wide 31, 1 (2010), 40–61.

[23]

Noam Chomsky. 1965. Aspects of the Theory of Syntax. Number 11 in Massachusetts Institute of Technology (Cambridge, Mass.). Research Laboratory of Electronics. Special Technical Report. MIT Press, Cambridge, MA. OCLC: 245876504.

[24]

Domenic V. Cicchetti, Donald Shoinralter, and Peter J. Tyrer. 1985. The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability : A Monte Carlo Investigation. Applied Psychological Measurement 9, 1 (March 1985), 31–36.

[25]

Lilia M. Cortina, Vicki J. Magley, Jill Hunter Williams, and Regina Day Langhout. 2001. Incivility in the Workplace: Incidence and Impact.Journal of Occupational Health Psychology 6, 1 (2001), 64–80.

[26]

David Crystal 2001. Language and the Internet. Cambridge University Press.

[27]

Jonathon N Cummings, Brian Butler, and Robert Kraut. 2002. The quality of online social relationships. Commun. ACM 45, 7 (2002), 103–108.

Digital Library

[28]

Laura A. Dabbish and Robert E. Kraut. 2006. Email Overload at Work: An Analysis of Factors Associated with Email Strain. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work - CSCW ’06. ACM Press, Banff, Alberta, Canada, 431.

[29]

Laura A. Dabbish, Robert E. Kraut, Susan Fussell, and Sara Kiesler. 2005. Understanding Email Use: Predicting Action on a Message. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’05. ACM Press, Portland, Oregon, USA, 691.

Digital Library

[30]

Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A Computational Approach to Politeness with Application to Social Factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. 10.

[31]

Rita de Cássia Veiga Marriott and Patricia Lupion Torres. 2009. Politeness in Intercultural E-Mail Communication. In Handbook of Research on E-Learning Methodologies for Language Acquisition:. IGI Global.

[32]

Rachele De Felice, Jeannique Darby, Anthony Fisher, and David Peplow. 2013. A Classification Scheme for Annotating Speech Acts in a Business Email Corpus. ICAME Journal 37(2013), 71–105.

[33]

Rachele De Felice and Gregory Garretson. 2018. Politeness at work in the Clinton email Corpus: A first look at the effects of status and gender. Corpus Pragmatics 2, 3 (2018), 221–242.

[34]

Ferdinand de Saussure. 1959. Course in General Linguistics. Philosophical Library, New York.

Digital Library

[35]

Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying Reply Suggestions Using a Matching-Conditional Variational Autoencoder. arXiv:1903.10630 [cs, stat] (March 2019). arxiv:1903.10630 [cs, stat]

[36]

Dotan Di Castro, Zohar Karnin, Liane Lewin-Eytan, and Yoelle Maarek. 2016. You’ve got mail, and here is what you could do with it!: Analyzing and predicting actions on email messages. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. ACM, 307–316.

Digital Library

[37]

Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them Err.Journal of Experimental Psychology: General 144, 1 (2015), 114–126.

[38]

Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, Karine Megerdoomian, Matthew T. McMahon, Brian F. Tivnan, and Christopher M. Danforth. 2015. Human Language Reveals a Universal Positivity Bias. Proceedings of the National Academy of Sciences 112, 8 (Feb. 2015), 2389–2394.

[39]

Peter Sheridan Dodds and Christopher M. Danforth. 2010. Measuring the Happiness of Large-Scale Written Expression: Songs, Blogs, and Presidents. Journal of Happiness Studies 11, 4 (Aug. 2010), 441–456.

[40]

Peter Sheridan Dodds, Kameron Decker Harris, Isabel M. Kloumann, Catherine A. Bliss, and Christopher M. Danforth. 2011. Temporal Patterns of Happiness and Information in a Global Social Network: Hedonometrics and Twitter. PLoS ONE 6, 12 (Dec. 2011), e26752.

[41]

Christa Dürscheid, Carmen Frehner, Susan C Herring, Dieter Stein, and Tuija Virtanen. 2013. Email communication. Handbooks of Pragmatics [HOPS]9 (2013), 35–54.

[42]

William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States. SSRN Electronic Journal(2017).

[43]

Anne Edstrom and Jennifer D. Ewald. 2017. ”Out of the Office”: Conveying Politeness through Auto-Reply Email Messages.Language@Internet 14, 4 (2017).

[44]

Anne Edstrom and Jennifer D. Ewald. 2019. Characteristics of Effective Auto-Reply Emails: Politeness and Perceptions. Technology in Society (Jan. 2019), S0160791X1730218X.

[45]

Serge Egelman, Ed H Chi, and Steven Dow. 2014. Crowdsourcing in hci research. In Ways of Knowing in HCI. Springer, 267–289.

[46]

Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. ”I Always Assumed That I Wasn’t Really That Close to [Her]”: Reasoning about Invisible Algorithms in News Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15. ACM Press, Seoul, Republic of Korea, 153–162.

Digital Library

[47]

Ethan Fast, Binbin Chen, and Michael S. Bernstein. 2016. Empath: Understanding Topic Signals in Large-Scale Text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM Press, Santa Clara, California, USA, 4647–4657.

Digital Library

[48]

Lori Francis, Camilla M. Holmvall, and Laura E. O’Brien. 2015. The Influence of Workload and Civility of Treatment on the Perpetration of Email Incivility. Computers in Human Behavior 46 (May 2015), 191–201.

[49]

Brenden Gallagher. 2018. Gmail’s Smart Reply blurs the line between people and brands. https://www.dailydot.com/debug/thanks-for-letting-me-know/. [Online; accessed Sept-2019].

[50]

Ryan J. Gallagher, Morgan R. Frank, Lewis Mitchell, Aaron J. Schwartz, Andrew J. Reagan, Christopher M. Danforth, and Peter Sheridan Dodds. 2020. Generalized Word Shift Graphs: A Method for Visualizing and Explaining Pairwise Comparisons Between Texts. arXiv:2008.02250 [physics] (Aug. 2020). arxiv:2008.02250 [physics]

[51]

Dana Garbarski, Nora Cate Schaeffer, and Jennifer Dykema. 2015. The Effects of Response Option Order and Question Order on Self-Rated Health. Quality of Life Research 24, 6 (June 2015), 1443–1453.

[52]

Gary W. Giumetti, Andrea L. Hatfield, Jenna L. Scisco, Amber N. Schroeder, Eric R. Muth, and Robin M. Kowalski. 2013. What a Rude E-Mail! Examining the Differential Effects of Incivility versus Support on Mood, Energy, Engagement, and Performance in an Online Context.Journal of Occupational Health Psychology 18, 3 (July 2013), 297–309.

[53]

J. Goldstein and R.E. Sabin. 2006. Using Speech Acts to Categorize Email and Identify Email Genres. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS’06). IEEE, Kauia, HI, USA, 50b–50b.

[54]

Paul H. Grice. 1975. Logic and Conversation. In Syntax and Semantics: Speech Acts, Peter Cole and J. Morgan (Eds.). Vol. 3. Academic Press, New York, 41–58.

[55]

Jacob Groshek and Chelsea Cutino. 2016. Meaner on Mobile: Incivility and Impoliteness in Communicating Contentious Politics on Sociotechnical Networks. Social Media + Society 2, 4 (Nov. 2016), 205630511667713.

Digital Library

[56]

Danielle N. Gunraj, April M. Drumm-Hewitt, Erica M. Dashow, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2016. Texting Insincerely: The Role of the Period in Text Messaging. Computers in Human Behavior 55 (Feb. 2016), 1067–1075.

[57]

Jeffrey T Hancock, Mor Naaman, and Karen Levy. 2020. AI-Mediated Communication: Definition, Research Agenda, and Ethical Considerations. Journal of Computer-Mediated Communication(2020).

[58]

Michael Haugh. 2010. When Is an Email Really Offensive?: Argumentativity and Variability in Evaluations of Impoliteness. Journal of Politeness Research. Language, Behaviour, Culture 6, 1 (Jan. 2010).

[59]

Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient Natural Language Response Suggestion for Smart Reply. arXiv:1705.00652 [cs] (May 2017). arxiv:1705.00652 [cs]

[60]

Susan C Herring. 1994. Politeness in computer culture: Why women thank and men flame. In Cultural Performances: Proceedings of the Third Berkeley Women and Language Conference. Berkeley Women and Language Group Berkeley, CA, 278–294.

[61]

H. W. Hildebrandt and R. D. Snyder. 1981. The Pollyanna Hypothesis in Business Writing: Initial Results, Suggestions for Research. Journal of Business Communication 18, 1 (Jan. 1981), 5–15.

[62]

Erin R. Hoffman, David W. McDonald, and Mark Zachry. 2017. Evaluating a Computational Approach to Labeling Politeness: Challenges for the Application of Machine Classification to Social Computing Data. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (Dec. 2017), 1–14.

Digital Library

[63]

Jess Hohenstein and Malte Jung. 2018. AI-Supported Messaging: An Investigation of Human-Human Text Conversation with AI Support. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, Montreal QC, Canada, 1–6.

Digital Library

[64]

Jess Hohenstein and Malte Jung. 2020. AI as a Moral Crumple Zone: The Effects of AI-Mediated Communication on Attribution and Trust. Computers in Human Behavior 106 (May 2020), 106190.

[65]

Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–16.

Digital Library

[66]

Dirk Holtbrügge, Abigail Weldon, and Helen Rogers. 2013. Cultural Determinants of Email Communication Styles. International Journal of Cross Cultural Management 13, 1 (April 2013), 89–110.

[67]

Kenneth J. Houghton, Sri Siddhi N. Upadhyay, and Celia M. Klin. 2018. Punctuation in Text Messages May Convey Abruptness. Period. Computers in Human Behavior 80 (March 2018), 112–121.

[68]

C J Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media (ICWSM). 10.

[69]

Dell Hymes. 1972. On Communicative Competence. In Sociolinguistics: Selected Readings, J.B. Pride and J. Holmes (Eds.). Penguin, Harmondsworth, 269–293.

[70]

Rumen Iliev, Joe Hoover, Morteza Dehghani, and Robert Axelrod. 2016. Linguistic Positivity in Historical Texts Reflects Dynamic Environmental and Psychological Factors. Proceedings of the National Academy of Sciences 113, 49 (Dec. 2016), E7871–E7879.

[71]

Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. 2019. AI-Mediated Communication: How the Perception That Profile Text Was Written by AI Affects Trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–13.

Digital Library

[72]

Susan Jamieson. 2004. Likert Scales: How to (Ab)Use Them. Medical Education 38, 12 (Dec. 2004), 1217–1218.

[73]

Sherri L. Jessmer and David Anderson. 2001. The Effect of Politeness and Grammar on User Perceptions of Electronic Mail.North American Journal of Psychology 3, 2 (2001), 331–346.

[74]

Anjuli Kannan, Peter Young, Vivek Ramavajjala, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, and Marina Ganea. 2016. Smart Reply: Automated Response Suggestion for Email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16. ACM Press, San Francisco, California, USA, 955–964.

Digital Library

[75]

Jonathon Keats. 2010. Virtual words: Language on the edge of science and technology. Oxford University Press.

[76]

Cory Knobel and Geoffrey C. Bowker. 2011. Values in Design. Commun. ACM 54, 7 (July 2011), 26.

Digital Library

[77]

Birgit Koopmann-Holm and Jeanne L Tsai. 2014. Focusing on the negative: Cultural differences in expressions of sympathy.Journal of Personality and Social Psychology 107, 6(2014), 1092.

[78]

Farshad Kooti, Luca Maria Aiello, Mihajlo Grbovic, Kristina Lerman, and Amin Mantrach. 2015. Evolution of Conversations in the Age of Email Overload. In Proceedings of the 24th International Conference on World Wide Web - WWW ’15. ACM Press, Florence, Italy, 603–613.

Digital Library

[79]

Jon A. Krosnick and Duane F. Alwin. 1987. An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement. Public Opinion Quarterly 51, 2 (1987), 201.

[80]

Jon A. Krosnick and Matthew K. Berent. 1993. Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format. American Journal of Political Science 37, 3 (1993), 941–964.

[81]

Jon A. Krosnick, Allyson L. Holbrook, Matthew K. Berent, Richard T. Carson, W. Michael Hanemann, Raymond J. Kopp, Robert Cameron Mitchell, Stanley Presser, Paul A. Ruud, V. Kerry Smith, Wendy R. Moody, Melanie C. Green, and Michael Conaway. 2002. The Impact of ”No Opinion” Response Options on Data Quality: Non-Attitude Reduction or an Invitation to Satisfice?The Public Opinion Quarterly 66, 3 (2002), 371–403.

[82]

Jon A. Krosnick and Stanley Presser. 2010. Question and Questionnaire Design. In Handbook of Survey Research, Second Edition (2 edition ed.), Peter V. Marsden and James D. Wright (Eds.). Emerald Publishing, Bingley, UK.

[83]

B Levin. 1996. On the Holm, Simes, and Hochberg Multiple Test Procedures.American Journal of Public Health 86, 5 (May 1996), 628–629.

[84]

Sandy Lim, Lilia M. Cortina, and Vicki J. Magley. 2008. Personal and Workgroup Incivility: Impact on Work and Health Outcomes.Journal of Applied Psychology 93, 1 (Jan. 2008), 95–107.

[85]

Vivien K.G. Lim and Thompson S.H. Teo. 2009. Mind Your E-Manners: Impact of Cyber Incivility on Employees’ Work Attitude and Behavior. Information & Management 46, 8 (2009), 419–425.

Digital Library

[86]

Xiao Ma, Jeffery T. Hancock, Kenneth Lim Mingjie, and Mor Naaman. 2017. Self-Disclosure and Perceived Trustworthiness of Airbnb Host Profiles. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, Portland Oregon USA, 2397–2409.

Digital Library

[87]

Shannon L. Marlow, Christina N. Lacerenza, and Chelsea Iwig. 2018. The Influence of Textual Cues on First Impressions of an Email Sender. Business and Professional Communication Quarterly 81, 2 (June 2018), 149–166.

[88]

Heather Newman and David Joyner. 2018. Sentiment Analysis of Student Evaluations of Teaching. In International Conference on Artificial Intelligence in Education, Carolyn Penstein Rosé, Roberto Martínez-Maldonado, H. Ulrich Hoppe, Rose Luckin, Manolis Mavrikis, Kaska Porayska-Pomsta, Bruce McLaren, and Benedict du Boulay (Eds.), Vol. 10948. Springer International Publishing, Cham, 246–250.

[89]

Kazushi Nishimoto and Jianning Wei. 2015. G-IM: An Input Method of Chinese Characters for Character Amnesia Prevention. In ACHI 2015. 118–124.

[90]

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. 2019. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data 2(2019), 13.

[91]

Alexandra Olteanu, Fernando Diaz, and Gabriella Kazai. 2020. When Are Search Completion Suggestions Problematic?Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (Oct. 2020), 1–25.

[92]

Séamas O’Reilly. 2019. How Smart Are Gmail’s ‘Smart Replies’?The Observer (Feb. 2019).

[93]

Soya Park, Amy X. Zhang, Luke S. Murray, and David R. Karger. 2019. Opportunities for Automating Email Processing: A Need-Finding Study. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12.

Digital Library

[94]

Ellie Pavlick and Joel Tetreault. 2016. An Empirical Analysis of Formality in Online Communication. Transactions of the Association for Computational Linguistics 4 (Dec. 2016), 61–74.

[95]

Carolyn C Preston and Andrew M Colman. 2000. Optimal Number of Response Categories in Rating Scales: Reliability, Validity, Discriminating Power, and Respondent Preferences. Acta Psychologica 104, 1 (March 2000), 1–15.

[96]

Kristen Purcell. 2011. Search and Email Still Top the List of Most Popular Online Activities.

[97]

Philip Quinn and Shumin Zhai. 2016. A Cost-Benefit Study of Text Entry Suggestion Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM Press, Santa Clara, California, USA, 83–88.

Digital Library

[98]

Ronald E. Robertson, Shan Jiang, David Lazer, and Christo Wilson. 2019. Auditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation. In Proceedings of the 10th ACM Conference on Web Science - WebSci ’19. ACM Press, Boston, Massachusetts, USA, 235–244.

Digital Library

[99]

Bahareh Sarrafzadeh, Ahmed Hassan Awadallah, Christopher H. Lin, Chia-Jung Lee, Milad Shokouhi, and Susan T. Dumais. 2019. Characterizing and Predicting Email Deferral Behavior. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining - WSDM ’19. ACM Press, Melbourne VIC, Australia, 627–635.

Digital Library

[100]

Gordon B. Schmidt and William M. Jettinghoff. 2016. Using Amazon Mechanical Turk and Other Compensated Crowdsourcing Sites. Business Horizons 59, 4 (July 2016), 391–400.

[101]

Burkard Sievers and Rose Redding Mersky. 2006. The Economy of Vengeance: Some Considerations on the Aetiology and Meaning of the Business of Revenge. Human Relations 59, 2 (Feb. 2006), 241–259.

[102]

Erin L. Spottswood and Jeffrey T. Hancock. 2016. The Positivity Bias and Prosocial Deception on Facebook. Computers in Human Behavior 65 (Dec. 2016), 252–259.

[103]

Keri K. Stephens, Renee L. Cowan, and Marian L. Houser. 2011. Organizational Norm Congruency and Interpersonal Familiarity in E-Mail: Examining Messages From Two Different Status Perspectives. Journal of Computer-Mediated Communication 16, 2 (Jan. 2011), 228–249.

[104]

Wenbin Tang, Honglei Zhuang, and Jie Tang. 2011. Learning to infer social ties in large networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 381–397.

Digital Library

[105]

Yla R. Tausczik and James W. Pennebaker. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology 29, 1 (March 2010), 24–54.

[106]

Benjamin Toff and Rasmus Kleis Nielsen. 2018. “I Just Google It”: Folk Theories of Distributed Discovery. Journal of Communication 68, 3 (June 2018), 636–657.

[107]

Anna K. Turnage. 2007. Email Flaming Behaviors and Organizational Conflict. Journal of Computer-Mediated Communication 13, 1 (Oct. 2007), 43–59.

Digital Library

[108]

Connie K. Varnhagen, G. Peggy McFall, Nicole Pugh, Lisa Routledge, Heather Sumida-MacDonald, and Trudy E. Kwong. 2010. Lol: New Language and Spelling in Instant Messaging. Reading and Writing 23, 6 (July 2010), 719–733.

[109]

Jane A. Vignovic and Lori Foster Thompson. 2010. Computer-Mediated Cross-Cultural Collaboration: Attributing Communication Errors to the Person versus the Situation.Journal of Applied Psychology 95, 2 (2010), 265–276.

[110]

Jocelyne Vincent. 2008. Netiquette Rules OK!...OK?: speculating on rhetorical cleansing and English linguistic and cultural imperialism through email netiquette style guides. In Threads in the complex fabric of language: linguistic and literary studies in honour of Lavinia Merlini Barbaresi, Lavinia Merlini Barbaresi, Marcella Bertuccelli-Papi, Antonio Bertacca, and Silvia Bruti (Eds.). Felici, San Giuliano Terme (Pisa), 409–443.

[111]

James Vincent. 2018. Google removes gendered pronouns from Gmail’s Smart Compose feature. https://www.theverge.com/2018/11/27/18114127/google-gmail-smart-compose-ai-gender-bias-prounouns-removed. [Online; accessed Sept-2019].

[112]

Joan Waldvogel. 2007. Greetings and Closings in Workplace Email. Journal of Computer-Mediated Communication 12, 2 (Jan. 2007), 456–477.

[113]

Joseph B Walther. 1995. Relational aspects of computer-mediated communication: Experimental observations over time. Organization Science 6, 2 (1995), 186–203.

Digital Library

[114]

Dashun Wang, Zhen Wen, Hanghang Tong, Ching-Yung Lin, Chaoming Song, and Albert-László Barabási. 2011. Information spreading in context. In Proceedings of the 20th International Conference on World Wide Web. 735–744.

Digital Library

[115]

Yi-Chia Wang, Alexandros Papangelis, Runze Wang, Zhaleh Feizollahi, Gokhan Tur, and Robert Kraut. 2020. Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents. arxiv:2012.14653 [cs.CL]

[116]

Helen J Watt. 2010. How does the use of modern communication technology influence language and literacy development? A review. Contemporary Issues in Communication Science and Disorders 37 (2010), 141.

[117]

Liu Yang, Susan T. Dumais, Paul N. Bennett, and Ahmed Hassan Awadallah. 2017. Characterizing and Predicting Enterprise Email Reply Behavior. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’17. ACM Press, Shinjuku, Tokyo, Japan, 235–244.

Digital Library

[118]

Xiao Yang, Ahmed Hassan Awadallah, Madian Khabsa, Wei Wang, and Miaosen Wang. 2018. Characterizing and Supporting Question Answering in Human-to-Human Communication. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18. ACM Press, Ann Arbor, MI, USA, 345–354.

[119]

Ning Ye, Ariel Fuxman, Vivek Ramavajjala, Sergey Nazarov, J Patrick McGregor, and Sujith Ravi. 2018. Photoreply: Automatically suggesting conversational responses to photos. In Proceedings of the 2018 World Wide Web Conference. International World Wide Web Conferences Steering Committee, 1893–1899.

Digital Library

[120]

Ming-chung Yu. 2005. Sociolinguistic competence in the complimenting act of native Chinese and American English speakers: A mirror of cultural value. Language and Speech 48, 1 (2005), 91–119.

[121]

Elana Zeide. 2015. This App Wants to Tell Your Next Employer About Your Personality. But It’s Probably Wrong.https://slate.com/technology/2015/05/crystal-app-algorithmic-fortunetelling-for-employers-and-potential-customers.html.

[122]

Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Nithum Thain, Yiqing Hua, and Dario Taraborelli. 2018. Conversations Gone Awry: Detecting Early Signs of Conversational Failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Melbourne, Australia, 1350–1361.

Cited By

Coman ACardon P(2024)Perceptions of Professionalism and Authenticity in AI-Assisted WritingBusiness and Professional Communication Quarterly10.1177/23294906241233224Online publication date: 11-Mar-2024
https://doi.org/10.1177/23294906241233224
Blunden HBrodsky A(2024)A Review of Virtual Impression Management Behaviors and OutcomesJournal of Management10.1177/0149206323122516050:6(2197-2236)Online publication date: 14-Mar-2024
https://doi.org/10.1177/01492063231225160
Truong LLee SSawhney N(2024)Enhancing Conversations in Migrant Counseling Services: Designing for Trustworthy Human-AI CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/36870348:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687034
Show More Cited By

Index Terms

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
1. Human-centered computing
  1. Collaborative and social computing
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Index terms have been assigned to the content through auto-classification.

Recommendations

To Reply or To Reply All: Understanding Replying Behavior in Group Email Communication
CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

“Reply” and “Reply All” buttons in email provide the convenience of a quick reply to those included in the email. Yet this very convenience can be troublesome to both the individual and the group involved if receivers intentionally or unintentionally ...
Characterizing and Predicting Enterprise Email Reply Behavior
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Email is still among the most popular online activities. People spend a significant amount of time sending, reading and responding to email in order to communicate with others, manage tasks and archive personal information. Most previous research on ...
Preventing Spam Email by Delivery Limitation in RMX
IDEAS '15: Proceedings of the 19th International Database Engineering & Applications Symposium

On the rule-based email exchange system called RMX, similar to general mailing lists, anyone can send emails by sending to an address unique to RMX. However, there is a security problem that we cannot prevent spam emails and accidentally sending email ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
761
Total Downloads

Downloads (Last 12 months)211
Downloads (Last 6 weeks)12

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Coman ACardon P(2024)Perceptions of Professionalism and Authenticity in AI-Assisted WritingBusiness and Professional Communication Quarterly10.1177/23294906241233224Online publication date: 11-Mar-2024
https://doi.org/10.1177/23294906241233224
Blunden HBrodsky A(2024)A Review of Virtual Impression Management Behaviors and OutcomesJournal of Management10.1177/0149206323122516050:6(2197-2236)Online publication date: 14-Mar-2024
https://doi.org/10.1177/01492063231225160
Truong LLee SSawhney N(2024)Enhancing Conversations in Migrant Counseling Services: Designing for Trustworthy Human-AI CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/36870348:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687034
Zhou CYan ZRam AGu YXiang YLiu CHuang YOoi WZhao S(2024)GlassMail: Towards Personalised Wearable Assistant for On-the-Go Email Creation on Smart GlassesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660683(372-390)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660683
Buschek D(2024)Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI ToolsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660681(2719-2737)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660681
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Lee MGero KChung JShum SRaheja VShen HVenugopalan SWambsganss TZhou DAlghamdi EAugust TBhat AChoksi MDutta SGuo JHoque MKim YKnight SNeshaei SShibani AShrivastava DShroff LSergeyuk AStark JSterman SWang SBosselut ABuschek DChang JChen SKreminski MPark JPea RRho EShen ZSiangliulue P(2024)A Design Space for Intelligent and Interactive Writing AssistantsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642697(1-35)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642697
Reinhard PLi MPeters CLeimeister J(2024)Augmentierung statt Automatisierung: Nachhaltigkeit in der Entwicklung von KI-Systemen aus der MitarbeitendenperspektiveSustainable Service Management10.1007/978-3-658-45148-6_15(465-492)Online publication date: 26-Jun-2024
https://doi.org/10.1007/978-3-658-45148-6_15
Shah HJaidka KUngar LFagan JGrosser T(2023)Building a Multimodal Classifier of Email Behavior: Towards a Social Network Understanding of Organizational CommunicationInformation10.3390/info1412066114:12(661)Online publication date: 14-Dec-2023
https://doi.org/10.3390/info14120661
Singh NBernal GSavchenko DGlassman E(2023)Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine IntelligenceACM Transactions on Computer-Human Interaction10.1145/351159930:5(1-57)Online publication date: 23-Sep-2023
https://dl.acm.org/doi/10.1145/3511599
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents