Can Voice Assistants Be Microaggressors? Cross-Race Psychological Responses to Failures of Automatic Speech Recognition

Previous work has demonstrated that the accuracy of language technology degrades for certain demographic groups. For example, in one evaluation, Twitter’s language identifier marked tweets using African American English as a foreign language 19.7% more than tweets using white-aligned English [10]. And in online hate speech detection, a false positive bias has been consistently demonstrated toward African American English [86].

Regarding automated speech recognition, VA users with foreign accents are more likely to experience errors [72]. Such errors even extend to natives without foreign accents: In a study of YouTube’s automated captions, Tatman et al. found that captions for Black speakers were significantly less accurate than that of their white counterparts [96]. Most notably, Koenecke et al. found speech recognition systems of Amazon, Apple, Google, IBM, and Microsoft to have an average word error rate of 35% for Black American speakers, in contrast to a 19% word error rate for white American speakers [41]. While such accuracy degradation has been repeatedly established, little work has taken a harm-based approach and been devoted to understanding precisely what effect accuracy degradation in VA systems has on users themselves. Mengesha et al. conducted a diary study evaluating Black users’ subjective experiences with voice assistants, including their responses to VA errors. This study revealed powerful testimonials about Black users’ perceptions of language technologies, including the perception that such technologies are not designed with Black users in mind and require some degree of speech accommodation in order to function well for Black users [55]. The present study builds on this work by using controlled experimental methods to more precisely measure the the psychological effects of experiencing those shortcomings in the technology. To our knowledge, the present study is the first systematic evaluation of the psychological impact of automated speech recognition errors that utilizes an experimental design and quantitative measurement methods.

Racial microaggressions are defined as “brief and commonplace daily verbal, behavioral, or environmental indignities, whether intentional or unintentional, that communicate hostile, derogatory, or negative racial slights and insults toward people of color” [94]. According to psychologist Derald Wing Sue, microaggressions represent a primary form of “modern racism,” subtle and often invisible forms of prejudice or inequity “hiding in the invisible assumptions and beliefs of individuals” and “embedded in the policies and structures of our institutions” [91]. Microaggressions commonly arise in conversational contexts, in which intergroup differences can manifest in the verbal or nonverbal responses of interaction partners from more privileged identity groups. Indeed, people of color identify the common experience of being ignored, being asked to repeat themselves, and/or encountering misunderstandings from white conversation partners due to differences in speech patterns or word choice and, specifically, any deviation from white American English² exhibited by people of color [30, 37, 54, 56, 95].

Although microaggressions tend to be subtle in nature, and often unrecognized by those who commit them, they can have a profound effect on those who experience them. A simple, seemingly innocuous example of being misunderstood or being asked to repeat oneself because of the way one speaks can reinforce the salience of a marginalized identity. This is particularly likely when there are societal stereotypes that associate one’s identity group with lower levels of intelligence and/or poorer communication skills [6]. Specifically, microaggressions can trigger stereotype threat, a “socially premised psychological threat that arises when one is in a situation or doing something for which a negative stereotype about one’s group applies” [90]. Prior work has shown that stereotype threat can have a host of psychological effects, including increased cognitive load [20] and self-focus [15], increased anxiety [71], and decreased self-esteem [19]. Moreover, stereotype threat can hinder targets’ subjective experiences [3], lower their sense of belonging [103], and cause them to dis-identify with or disengage from particular domains associated with the threat [89].

Given that the effects of microaggressions among people in human-human interactions, specifically in occurrences of miscommunication and misunderstanding, are well-documented, the present work aimed to study if these effects may be mirrored in human-computer interactions. Nass et al.’s Computers are Social Actors paradigm affirms that people subconsciously apply social heuristics to technologies, despite their conscious awareness that these technologies are not sentient [68]. This paradigm has been exhibited across multiple contexts: People form first impressions of a voice’s “personality” [53] much like how they form first impressions of people [4], and are attracted to computer voices that demonstrate similar personality characteristics as themselves [44, 66] just as people are attracted to those who are similar to them [59]. People also apply social codes of politeness towards voice assistants [11], much like how we employ politeness among other people [14]. What’s more, stereotyping gender-based attributes is commonplace for voice technology users: Computer tutors with characteristically male voices were rated more competent than female-voiced tutors [67], in line with people’s general perceptions of gender and competence [24, 25, 106]. In more recent work, researchers have found that some voice assistant users even actively personify modern assistants like Amazon Alexa and Google Home [17, 76]. In short, “humans have become voice-activated with brains that are wired to equate voices with people and to act quickly on that identification” regardless of whether the voice is artificial or representative of a real person [69].

Building off of these established phenomena, we predict that the effects of microaggressions and stereotype threat demonstrated in interpersonal interactions will carry over to Black users’ experience interacting with an error-prone VA. Specifically, we designed a controlled experiment to test the following hypothesis:

A total of 108 participants were recruited through a call for study participants on the following Craigslist city pages: Atlanta (n=21), Chicago (n=21), Houston (n=22), New York (n=22), and Washington D.C. (n=22). This sample size was determined using a power analysis based on a predicted effect size of .69, as informed by prior meta-analyses of research documenting the psychological harm of microaggressions. Participants were screened for eligibility before beginning the experiment. Requirements for eligibility included residing in the U.S.A., being aged 18 or older, identifying as either Black or white, having access to a device with a microphone and web camera, and being an active user of voice technology (using a voice technology “multiple times a day” or “multiple times a week”). We required participants to be active users of voice technology to minimize friction in the beginning of the study procedure, which was especially important given that the study was conducted over Zoom. Furthermore, this requirement helped streamline the procedure such that participants had as little direct interaction with the researchers as possible.

The results that follow are based off of the responses of 108 participants, 54 who identified as Black and 54 who identified as white. To determine participants’ race, in the screener form participants were asked to select from a set of race and ethnicity items in response to the question: “What is your race/ethnicity? Please select all that apply.” Only participants who indicated they were “African American/Black” or “White/Caucasian” were invited to participate. Mixed race individuals were not included. The mean age of the participants was 25.7, with an age range 18-52. 48 participants identified as male, 48 identified as female, and 12 identified as another gender or did not specify. All participants were compensated USD 15.

The study utilized a 2x2 between-subjects design, with participants’ self-identified race (Black, white) and their randomly assigned error rate condition (low, high) representing the two independent variables of interest. In the consent form that was completed prior to the study session, participants were told that the purpose of the study was to evaluate and improve the design of a new voice assistant technology that was ready for market. After providing their consent, participants enrolled in an online study session, conducted via the Zoom video conferencing platform by a member of the research team. Half of the participants were randomly assigned to a low error rate condition, and half of the participants were randomly sorted into a high error rate condition. This random assignment occurred before beginning the study procedure. As described below, all participants completed the same basic set of tasks in interacting with the voice assistant; however, based on their assigned experimental condition, the voice assistant’s responses to participants’ queries were pre-determined to exhibit either a higher or lower rate of errors of speech recognition on specific tasks in the sequence created for the study.

After confirming participants’ identity, compensation method, and consent, researchers turned off their web cameras and shared a slide show in full screen. Each slide featured different prompts instructing participants on how they should interact with the voice assistant (Figure 1). Participants were instructed to activate the voice assistant by saying “Hey assistant” before making any requests, and to speak to the assistant using a natural dialogue like they would use with their own voice assistant in their everyday life. Using a “Wizard of Oz” method, the researchers manually delivered all responses from the voice assistant using the text-to-speech AI voice generator from Play.ht [1]. The researchers aimed to replicate the default user experience of popular commercial products, and thus selected a voice representative of a woman speaking white American English [60]. Participants engaged with three “warm-up” prompts to get situated with the VA (Appendix Table 3) before users were presented with eleven evaluative prompts (Table 1). The prompts were selected based on prominent VA user habits, as reported by a 2019 Adobe survey of over 1,000 users [2] and system logs of voice assistant users’ commands [87]. We implemented humorous VA responses in the beginning and end of each participant’s VA interaction to make participants feel more comfortable and enhance their task enjoyment in the study environment [70]. Prior to the study, this procedure was carefully piloted to ensure that participants perceived a high degree of realism and believed they were, in fact, interacting with a functioning voice assistant.

Based on a participants’ randomly assigned error rate condition, the audio response delivered by the voice assistant would either accurately or inaccurately address the participants’ requests. For participants in the high error rate condition, 35.7% of the voice assistant responses were inaccurate. This error rate is based on prior research on the error rates Black individuals experience with voice assistants in everyday environments [41]. For participants in the low error rate condition, 7.1% of the responses were inaccurate. We chose to implement an error rate lower than what white Americans typically experience as we were aiming to simulate an ideal version of the software. That said, we still included one inaccuracy in the low error rate condition, as no commercial voice assistant has perfect accuracy and we wanted our product to be accepted as a realistic product. To this point, in our pilot studies, participants who interacted with a voice assistant displaying perfect accuracy were more skeptical that the voice assistant was real, echoing previous research on agentic errors [57, 77].

After completing their set of interactions with the voice assistant, participants completed a survey about their experience. The survey included the following validated self-report measures of their psychological responses to their experience as well as their perceptions of the technology:

(Figure 2 and Figure 3) To analyze the results from the PANAS-X Scale of affective responses, we first created separate composite subscales for the Positive Affect and Negative Affect items; each subscale achieved a satisfactory level of internal reliability (Cronbach’s alphas > 0.75).

Results from the ANOVA for the Positive Affect scale revealed a significant race x error condition interaction: F (1, 107) = 5.74, p = .007. Planned comparisons revealed that Black participants in the high-error condition reported a significantly lower level of positive affect (M= 2.74, SD = .94) compared to Black participants in the low-error condition (M= 3.68, SD = .56), t(52) = 4.47, p < .01. In comparison, there was no significant difference in the average level of positive affect reported by white participants in the high-error condition (M= 3.18, SD = .92) and low-error condition (M= 3.20, SD = .86), t(52) = .82, p = .47. This pattern supports our hypothesis that Black participants’ positive emotional states would be more negatively affected by encountering a higher rate of errors than would white participants’.

Analysis of the responses to the Negative Affect scale did not reveal a significant race x error condition interaction: F (1, 107) = .17, p = .39. Overall, reported levels of negative affect were relatively low (with means in all conditions falling below the midpoint of the 5-point rating scale). Average levels of negative affect were higher in the high-error conditions (M= 1.59, SD = .56) compared to the low-error conditions (M= 1.26, SD = .30), but this pattern did not differ by participant race. These results did not support our hypothesis: neither Black nor white participants appeared to experience a high level of negative affect overall.

(Figure 4) Participants’ responses to the individual items of the Self-Consciousness Scale were summed and averaged to form a composite score (Cronbach’s alpha = .83). Results from the ANOVA for the composite scale revealed a significant race x error condition interaction: F (1, 107) = 5.61, p < .001. Planned comparisons revealed that Black participants in the high-error condition reported a significantly higher level of self-consciousness (M= 6.09, SD = .72) compared to Black participants in the low-error condition (M= 4.94, SD = .75), t(52) = 7.42, p < .01. In comparison, there was no significant difference in the average level of self-consciousness reported by white participants in the high-error condition (M= 4.50, SD = .75) and low-error condition (M= 4.28, SD = .78), t(52) = .86, p = .29. This pattern supports our hypothesis that Black participants’ state of self-consciousness would be affected more by encountering a higher rate of errors than would white participants’.

(Figure 5 and Figure 6) Responses to both the individual and collective Self-Esteem scales were averaged to form composite scores for each (Cronbach’s alphas > .78). Results from the ANOVA for the composite scale for individual self-esteem revealed a significant race x error condition interaction: F (1, 107) = 2.18, p = .01. Planned comparisons revealed that Black participants in the high-error condition reported a significantly lower level of individual self-esteem (M= 4.17, SD = .74) compared to Black participants in the low-error condition (M= 4.99, SD = .59), t(52) = 4.52, p < .01. The average level of self-esteem reported by white participants in the high-error condition was also lower (M= 4.64, SD = .47) than the average level reported by white participants in the high error rate condition (M= 4.90, SD = .68), t(52) = 2.06, p =.04. However, the difference in means still stands to be greater for Black participants than for white participants, supporting our hypothesis that Black participants’ personal self-esteem would be affected more by encountering a higher rate of errors than would white participants.

For collective self-esteem, results from the ANOVA revealed a significant race x error condition interaction: F (1, 107) = 3.38, p = .003. Planned comparisons revealed that Black participants in the high-error condition reported a significantly lower level of individual self-esteem (M= 4.26, SD = .68) compared to Black participants in the low-error condition (M= 4.84, SD = .56), t(52) = 3.49, p < .01. In comparison, there was no significant difference in the average level of collective self-esteem reported by white participants in the high-error condition (M= 4.57, SD = .59) and low-error condition (M= 4.45, SD = .56), t(52) = .76, p = .45. This pattern supports our hypothesis that Black participants’ group-level self-esteem would be affected more by encountering a higher rate of errors than would white participants’.

(Figure 7) Participants’ responses to the individual items of the Transportation Scale were summed and averaged to form a composite score (Cronbach’s alpha = .88). Results from the ANOVA for the composite scale revealed that the race x error condition interaction was not significant: F (1, 107) = .01, p = .82. Transportation levels reported by participants in the high-error conditions (M= 3.99, SD = .54) were lower than the mean levels reported in the low-error conditions (M= 4.32, SD = .48) to a non-significant degree, and this pattern held for both Black and white participants (see Table 2). Contrary to our hypotheses, Black participants did not show a differential rate of reduced engagement with the task, compared to white participants, when confronted with a more error-prone assistant.

(Figure 8) Participants’ responses to the individual items of the technology evaluation measure were summed and averaged to form a composite score (Cronbach’s alpha = .72). Results from the ANOVA for the composite scale revealed a significant race x error condition interaction: F (1, 107) = 8.52, p < .001. Planned comparisons revealed that Black participants in the high-error condition reported a significantly less positive evaluation of the voice assistant (M= 4.46, SD = 1.14) compared to Black participants in the low-error condition (M= 5.30, SD = .48), t(52) = 3.51, p < .01. In comparison, there was no significant difference in the average level of self-consciousness reported by white participants in the high-error condition (M= 5.06, SD = .68) and low-error condition (M= 4.77, SD = .94), t(52) = 1.30, p = .20. This pattern supports our hypothesis that Black participants’ subjective perceptions of the technology would be more negatively impacted by interacting with a more error-prone VA than would white participants’ perceptions. The pattern of means actually revealed that white participants rated the technology slightly (but not significantly) more positively in the high-error condition.

Taken as a whole, the findings provide strong support for our general hypothesis that Black participants would be more negatively impacted by interacting with a more error-prone voice assistant than would white participants – and, moreover, be impacted in ways consistent with findings from prior research on racial microaggressions. As the results of the study revealed, Black participants randomly assigned to the high-error condition, compared to Black participants in the low-error condition, exhibited higher levels of self-consciousness; lower levels of positive affect as well as individual and collective self-esteem; and less favorable evaluations of the technology. In contrast, white participants were largely unaffected by the error rate displayed by the assistant; across most measures, white participants displayed little difference in their psychological and evaluative responses. Moreover, the differences that were observed between Black and white participants, particularly in the high-error conditions, cannot be attributed to differences in engagement with the task (as we did not observe a significant race x error condition interaction for the measure of psychological transportation).

In other words, despite the fact that white and Black participants in the high error condition experienced an objectively identical set of errors, their subjective experience of the interaction was strikingly different. This pattern is entirely consistent with the findings of prior work on racial microaggressions, which has revealed that the same life experiences (including being misunderstood or misinterpreted by others in social interactions) impact members of racial minority groups more negatively because those occurrences remind members of those groups of stereotypes or biases associated with their identity and trigger a host of threat-related emotional and cognitive responses. Linguistic and communicative misunderstandings are more systemic for Black individuals, but not for white individuals. Moreover, for many people of color, interpersonal microaggressions are constant, continual, and cumulative [92]. The results from the present work indicate that people of color are likely to be affected similarly by acts of bias exhibited by technology and experience those interactions as microaggressions. Due to their innate racial privilege, white participants’ race is not implicated in the same way in experiences of misunderstandings (by other people or by technology). Thus, instead of interpreting speech recognition errors as discriminating against their race or personhood, they are more likely to attribute the errors to other external factors [99]. Indeed, the pattern of Black participants’ internalizing the experience of VA errors (e.g., with heightened self-consciousness and reduced self-esteem) can be contrasted with the finding that white participants exhibited minimal patterns of self-directed focus or blame when confronted with the same display of misunderstanding from the VA. On the one dimension that white participants did appear to be negatively affected by VA errors, individual self-esteem, the impact was nonetheless significantly greater for Black participants.

The present study was designed to be an initial investigation of the disparate impact of voice assistant errors on marginalized and non-marginalized participants. The focus of the study was modeled on the prototype offered by controlled experimental research of racial microaggressions in its prioritization of a high level of experimental control and internal validity (e.g., in pre-designating interaction tasks and keeping the task sequences uniform between conditions), its focus on general differences between two demographic identity categories (Black versus white racial identity), and its use of validated outcome measures utilized by prior work in this space. At the same time, we acknowledge the limitations that these methodological choices pose and the value of follow-up work to extend the results the present study revealed.

First, in using a carefully controlled experimental set-up, we prioritized internal over external validity. While we were careful to design the VA interaction in ways that preserved a sense of believability and realism, this study did not deploy a manipulation check for realism and did not observe users’ interactions with VAs in naturalistic settings. To this end, we have initiated a follow-up study utilizing in-the-wild data collection (including diary entries and usage logs) with participants in their own personal contexts to ascertain if the patterns of findings observed in the present research replicate in more natural, realistic interactions with VAs.

Furthermore, this follow-up study aims to address a second limitation of the present work: its focus on the immediate, short-term psychological impact of VA errors on Black users. In the field study we are currently conducting, we are utilizing repeated measurement of many of the same outcome measures employed in the present study. In addition, we will incorporate a number of measures used in prior work on microaggressions to determine if repeated, cumulative experiences with biases in voice technologies affect users’ susceptibility to health outcomes such as depression [65, 100], anxiety [100], and an overall negative view of the world [65]. Moreover, as researchers have demonstrated, repeated experiences with microaggressions and stereotype threat can have a host of physical health costs [64], including high blood pressure [9, 13] and hypertension [80]. Future studies that utilize longitudinal studies should incorporate these longer-term measures of harm to determine the extent to which technology-driven microaggressions have a similar negative effect on people of color and other marginalized populations. In addition, future investigations, particularly longitudinal studies, could focus on the strategies use to respond to errors in technology – for example, studying what factors predict particular behavioral responses to speech recognition errors, such as code-switching (i.e., assimilation to adjust speech to align with white American English: [35, 40] or dis-engagement from interacting with error-prone technologies [43] and how such patterns of response might either exacerbate or mitigate any harm caused by a technology’s performance.

Another inherent limitation of the present work is its focus on a single facet of identity – racial identity – and, moreover, its comparison of participants who identified their racial identity as primarily Black or white. Future work in this space must not only extend this finding to other facets of identity that may be susceptible to harm caused by patterns of bias in technology – including other racial minority groups, other language groups (e.g., English as a second language speakers, speakers with particular accents or dialects), speakers from lower socio-economic strata, LGBTQ+ users, etc. Ideally, future work will also apply an intersectional approach to identity, understanding that the subjective experiences of individuals are impacted by the interplay between various facets of their identity [78]. For example, the mental and physical health implications of errors and biases in interactions with technology may be of particular significance for disabled Black users [23]. Since speech recognition technologies are utilized by individuals with a variety of accessibility needs [7, 75, 88, 102], when these systems fail, not only are disabled Black users prevented from using assistive technologies that may be central to their day-to-day needs and workflow, but simply attempting to use these requisite technologies can increase their risk of suffering mental and physical health harms due to the psychological threat they may evoke.

Finally, the present research utilized a VA whose voice exhibited the typical features commonly used as the default in the most popular options on the market (e.g., Alexa, Siri, or Google Home): namely, a female voice that prior work has shown is assigned a racial identity of white [60]. Building on a growing body of work examining how various characteristics of voice assistants may affect user trust and acceptance, which has focused primarily on perceived gender [31, 79, 98] and personality [12, 74], understanding the role of perceived race of a VA would be a worthwhile focus for future work. For example, one specific follow-up study to the present research could manipulate both the error rate and perceived race of a VA to determine how users respond to an error-prone VA who shared versus does not share their own racial identity. While prior work has shown that Black users exhibited a preference for conversational agents perceived to be Black [45], would perceived race impact the extent to which Black users experience a VA’s speech recognition errors as a microaggression?

Given the findings of the present study, one vital implication for the design of voice assistants is the importance of addressing or reversing any harm caused by errors in speech recognition, particularly for users from marginalized groups. While there is a growing body of work dedicated to understanding VA error recovery [8, 16, 39, 52, 62, 63], little attention has been paid to how error recovery may be designed specifically for members of marginalized populations, such as Black users. Next, we propose potential directions for designing error recovery strategies that acknowledge the validity of marginalized users’ experiences of speech recognition errors as microaggressions and/or aim to reduce the negative impact caused by these errors. These directions are directly informed by research on effective ways of defusing or mitigating the harm caused by experiences of bias or prejudice in everyday life [93].

“During this study, you will be providing you with 11 questions you will be asking the voice assistant. The questions will be provided on screen as you advance through the study. Due to this being over Zoom there might be a slight lag in the response time or feedback from the agent. This is completely normal. We have tested this and the voice agent accurately hears everything you say to it. Do you have any questions?

As with any assistant, you must call on the assistant before asking it a question. For example with Google Home and Amazon Alexa, you would say “Hey Google, or Alexa” and then follow up with questions such as “What’s my schedule today?”. With this new voice technology, you’ll also have to call on the assistant before asking it a question. To call on the assistant you can say “Hey assistant” and follow up with your question. If the voice assistant doesn’t respond accurately or doesn’t understand what you’ve asked, please refrain from re-asking the assistant. Furthermore, please answer all questions as naturally as possible, as if you were at home or in an environment where you regularly use your voice assistant. Lastly, after the study begins, I will remain in the background with my camera and mic off to encourage a seamless interaction between you and the assistant. Please refrain from asking me about any interactions between you and the assistant.

Let’s run through a few questions to familiarize yourself with the assistant: [Refer to Table 3.]

Do you have any questions or concerns?

Great! You are now going to run through the bulk of the questions. Please imagine these conversations in the context of conversing with an agent at home or in an environment that you regularly use your voice assistant. I will now be turning off my camera and mic so you can converse with the assistant. I’ll pop back in after the questions are over.”