Do Humans Trust Robots that Violate Moral Trust?

Trust in human-robot interaction is an active area of research, and it gained more attention in recent decades. Considerable studies have investigated the effects of diverse factors that influence human-robot trust [15, 27, 37]. Hancock et al. performed a meta-analysis on the factors affecting human-robot trust introduced by various studies. They found that, among all factors affecting trust in HRI, robot-related factors, especially robot performance, are most intensely associated with trust [22]. Therefore, we can say trust in human-robot interaction has been mainly influenced by robot performance. As such, we believe trust in a robotic agent may be compromised after faulty behavior.

Desai et al. performed numerous studies on the effects of robot performance on human trust [13, 14]. Their work mainly focuses on the impact of a robot’s performance on a user’s decision to let the robot operate in an autonomous mode or take over the robot’s control. Robinette et al. investigate the impact of robot reliability on an individual’s decision to follow the robot’s directions in an emergency evacuation scenario [48]. They reported a tendency in participants to over-trust a robot regardless of the robot’s previous faulty behaviors. Salem et al. used a human’s decision to comply with a robot’s commands as a sign of trust in the robot. They experimented to determine the effect of robot failures on human compliance with unusual requests of the robot [49]. They found that participants still completed the odd request made by the robot despite errors.

In a competitive scenario studied by Ragni et al., a robot was programmed to either make some mistakes and exhibit limited memory or to perform correctly continuously [45]. They reported that participants perceived the faulty robot as less intelligent and reliable. However, participants reported that the task was more manageable and commented more positively about the task when the robot showed faulty behavior.

Perceptions of robot failure are not consistent in different contexts. For example, Mirnig et al. reported no significant differences in perceived anthropomorphism or intelligence of a social robot performing a task precisely or making faulty behavior occasionally [35]. Nevertheless, experiment participants liked the erroneous robot significantly better than the flawless robot, which can be interpreted as faulty behaviors that can cause social robots to be perceived as more natural.

The summarized studies reveal numerous factors influencing how individuals perceive a faulty robot, such as the severity of the failure [49] or the task and interaction type [45, 48]. Moreover, there appears to be a capacity to which faults are tolerated, even considered as more favorable and likable [35], and can even have a calibration effect on human performance and help to regulate the tradeoff between performance and satisfaction in human-robot interaction [45].

In the human-robot trust literature, performance-based trust has been the main focus of most studies. The main focus of research on trust in HRI is on factors related to robot performance, capabilities, and mission environments, where components are extensively addressed in terms of rational features such as the robot’s reliability and anthropomorphism or the mission types and complexity [22, 27]. However, the impact of robot personality has obtained less attention [19]. There are fewer concerns about systems being deceptive, betraying, or exploiting humans in human-automation trust literature than the system performance concerns [51]. However, the focus of the studies in the field of human-human trust is more on the affection-based and moral trust aspects. One of the primary questions underlying human-human trust is: “whether a human agent will take advantage of another human’s vulnerability either unconsciously because of a lack of ability or consciously because of a lack of moral integrity” [44].

The multidimensional conceptualization of trust in HRI that incorporates both factors related to the performance and personality of the robot better matches the current state of trust in human-robot interaction. Some researchers have studied different dimensions of trust between humans and robots, and some developed trust scales for measuring these dimensions. Schaefer [50] proposed four categories of trust in robots: the propensity of trust, affect-based trust, cognition-based trust, and trustworthiness. They also developed a 40-item comprehensive questionnaire accounting for all these trust dimensions. Bartneck et al. [3] developed and tested measurement scales for five key concepts affecting human-robot trust, including anthropomorphism, likeability, animacy, intelligence, and perceived safety of robots. Cameron et al. [7] highlighted the multidimensional nature of trust in HRI. They recognized that trust is not a singular, uniform concept but rather comprises various dimensions. However, their work did not offer a specific scale for measuring these dimensions, which was later addressed by Bernotat and colleagues. Bernotat, Eyssel, and Sachse expanded on the multidimensional nature of trust in HRI by distinguishing between cognitive and affective trust. Cognitive trust refers to the belief in the robot’s competence, reliability, and functionality. In contrast, affective trust relates to the emotional bond and sense of security a user feels with the robot. They developed a scale to measure these two components, providing a valuable tool for assessing trust in HRI [4, 5]. Gompei et al. [19] studied different factors affecting cognitive and affective trust and the overlapping elements between these two and developed a 12-item trust scale for measuring cognitive and affective trust in HRI. Chen et al. [9] considered three dimensions for trust in HRI cognitive trust, emotional trust, and behavioral trust, and studied the effects of reciprocity and altruistic behaviors of robots on each trust dimension. Park et al. [44] identify three key dimensions—performance, process, and purpose—associated with perceived trust in service robots during order and payment transactions at a restaurant. Notably, the study establishes a higher-order formative model, revealing that performance has the greatest impact on trust, followed by purpose and process as influential trust dimensions.

Although different names in different studies have referred to trust dimensions, there are similarities among the definitions. Performance-based and competence-based trust is mainly used interchangeably and underlines a robot’s capability, reliability, and efficiency in performing a task. Performance and competence are closely related and often considered synonyms within the cognitive trust framework. Robots that satisfy this trust dimension are an appropriate option to conduct a task that demands heightened preciseness and zero interaction with humans. However, affective-based and relation-based trust signifies the importance of robot acceptance as a trusted social agent and focuses on the relationship between humans and robots. However, based on the definitions, relational trust emphasizes trust aspects like predictability and dependability, and affective trust is mainly based on the feeling of security and attachment. Robots that grant relation-based trust are the right option for tasks requiring close interaction with humans. This class of trust was understudied in the past. However, relation-based trust is increasingly gaining attention with the advancements in social robots [47, 57, 59, 61]. Therefore, more research is expected to be concentrated on factors that may influence this type of trust shortly. Moral trust and integrity are intertwined, with integrity being a crucial aspect of moral trust. Cognitive and affective trusts are broader categories that encompass these various dimensions, addressing rational evaluations and emotional aspects, respectively [60].

Numerous scholars, spanning disciplines from sociology to management, have emphasized competence and integrity as the principal expectations in their examinations of trust in human-human interactions [6, 10, 28, 34]. Based on the human-human trust literature, positive and negative behaviors of a human agent are weighted and perceived by other humans differently concerning a human agent’s competency and integrity. The positive behaviors of a human agent are more intensely weighted than the negative behaviors of the agents when assessing the agent’s competency. However, regarding the human agent’s integrity, negative behaviors are weighted higher than positive behaviors [56]. The inverted weight of positive and negative behavior for assessing competency and integrity of the human agents might be because positive behaviors are a better illustration of one’s competence, and negative behavior better illustrates one’s integrity [55]. Research on the effectiveness of trust repair strategies after a trust violation by a human agent shows that different trust repair strategies should be employed for repairing trust after competency-trust violations and integrity-trust violations [16, 17, 29].

While the consequences of breaches of various trust aspects by automated systems and robots are not as thoroughly investigated as those resulting from human violations, a growing body of research is beginning to shed light on this critical area. Clark et al. [12] categorizes trust violations in HAI under two categories, integrity-based trust violations (IBTV) and competence-based trust violations (CBTV). They also stated that IBTVs affect trust more negatively than CBTVs. Chen et al. [9] studied the effects of selfish behavior by robots and found out that people report lower emotional, cognitive, and behavioral trust toward selfish robots. Anzabi et al. [2] observed that robots displaying a lack of benevolent behavior are met with diminished affective trust from humans, whereas those lacking competence are perceived as less trustworthy in terms of cognitive trust. Sebo et al. [52] also examine the effectiveness of various trust repair strategies, such as apology or denial, in recovering from different classes of trust violations by robots, namely, violations of competency and integrity by a robot. Although this study introduces various trust repair strategies for two different trust violation types by robots, it is not evident how each of these two different trust violation types affects human trust.

Drawing from the human-human trust literature, it is evident that the positive and negative behaviors of a human agent are perceived and weighted differently by others in relation to the agent’s competency and integrity. When assessing an agent’s competency, the agent’s positive behaviors are given more weight than their negative behaviors. In contrast, negative behaviors carry more weight than positive ones for evaluations of the agent’s integrity [55, 56]. Moreover, as discussed in previous sections, moral trust and integrity are closely linked, with integrity being a fundamental component of moral trust. Additionally, the concepts of performance-based trust and competence-based trust are often used interchangeably. Building on these insights, we formulated the following hypothesis:

Our research builds upon the comprehensive trust framework as delineated by Ullman et al. [61]. This framework posits trust as a complex, multi-faceted construct, encompassing performance-related elements commonly emphasized in human-automation contexts and moral elements often highlighted in human-human interactions. Central to this approach is the Multi-Dimensional Measure of Trust (MDMT), a tool proposed by Ullman et al. for effectively quantifying trust. The MDMT identifies two primary dimensions of trust—Performance Trust and Moral Trust–each further divided into subcategories: Reliability and Competence under Performance Trust and Transparency, Ethics, and Benevolence under Moral Trust.

The significant contribution of Ullman et al.’s work is the recognition that this multi-dimensional perspective of trust is applicable in human-robot interactions, though not all dimensions may be pertinent in every interaction. This implies that the MDMT can be selectively employed in various human-robot engagements to assess only the relevant dimensions of trust. Furthermore, should a robot breach a specific trust aspect, this infringement can be distinctly evaluated using the MDMT. This nuanced understanding of trust dynamics in human-robot interactions led us to formulate the following hypothesis:

We developed and evaluated various iterations of human-robot trust games to design an experiment that effectively differentiates between performance and moral trust. Initially, our game designs primarily illustrated violations of performance trust. However, in subsequent versions, we incorporated scenarios that represented breaches of moral trust. Analysis of the results from these early versions indicated a notable pattern: A significant number of participants selected the Not Applicable (N/A) option for moral trust-related items in the MDMT questionnaire [61], particularly in scenarios where moral trust violations were introduced. This observation led us to hypothesize that individuals might not consider moral trust violations as a relevant factor in their perception of robots unless the robots explicitly engage in actions that breach moral trust. Put differently: People might deem moral aspects of trust as irrelevant to robots until there is a direct violation of these moral standards by the robots. These insights informed the formulation of our third hypothesis in this study:

For our experiments, we designed an online human-robot collaborative search game. The game is a simulated search task hosted on a website, and the human plays the game on a computer using a keyboard. In the game procedure, teams composed of one human participant and one robot play 7 rounds of a search game, each round 30 seconds long. Both team members are supposed to search the area and find targets hidden in the area. There are targets in the form of gold coins in the environment, and picking each target leads to gaining one point. The human can search the area with the help of four arrow keys on the keyboard. While the human searches the area, the targets hidden in the searched spots get disclosed. Picking targets is optional. The human can pick targets by navigating to the target cell and pressing the space bar. The human search areas are separate from the robot search areas. The search area is a giant map, so agents can keep moving to the unsearched areas in each round and search for more targets. In the human search area, there are 34 coins, but in the limited game time, the human would not be able to find all of those.

Figure 1 is a screen capture of the human search area on the game page. In this figure, the right side of the screen is the search area in which the human should perform the search task. On the left side, there is a legend showing all the gained scores up to that point and the passed time of the current round. The blue-colored areas are the areas searched by the human. The dark gray-colored areas still need to be searched. The targets with a green border are those the human has already picked. Other targets have not been picked yet. A blue bar on the top edge of the screen also shows the round number.

The human cannot see or control the robot’s operation when the round is running. At the end of each round, the human and the robot should make a trust decision. The trust decision is to decide whether to collaborate and integrate their round scores into the team score or not collaborate and keep their round scores as individual scores. The human and robot must make their trust decision before seeing each other’s score and trust decision. Therefore, it is a blind decision.

The team score at the end of each round is calculated as (human round score \(\times\) robot round score \(\times\) 2), but the team score can be gained only if the human and the robot both choose to add to the team score. If both team members choose to add to their individual scores, then their unchanged scores will be added to their individual scores. However, if one adds to the team score and the other adds to the individual score, then no team score will be gained, and the one who added to the team score will gain no score, but the other who added to the individual score gains an individual score. After making the trust decision, both the human and the robot can see the other teammates’ scores and trust decisions. Then, they can define their strategy for the next round based on the results of previous rounds.

There are two bonus values defined in this game. There is a $7 bonus for gaining 35 team points and a $2 bonus for gaining 17 individual points. Since the game is time-constrained, it is impossible to gain both bonus values. So, participants should decide to work either toward gaining the 35 team points or the 17 individual points. The idea behind this game is to encourage humans to collaborate with the robot and observe the result when the robot violates the trust built between the two. Therefore, we employed four strategies to encourage humans to work together with the robot and maximize the team score. These three strategies are as follows:

It should be noted that the robot scores and trust decisions in all rounds are predefined. However, the human is not aware of that.

In this experiment, we want to study the effects of undesirable robot behavior due to performance and moral trust violation on individuals’ trust in the robot. Undesirable behaviors are any behavior the human participants in this game do not appreciate. Therefore, we can say any behavior by the robot that leads to score loss is undesirable for human participants. These undesirable behaviors by the robot can occur due to the bad performance of the robot (i.e., gaining no points), which is a performance trust violation. They may also occur due to the immorality of the robot (i.e., adding to the individual score), which is a moral trust violation.

In this study, we are interested in seeing if two similar undesirable robot actions that lead to similar score loss would affect human trust differently if one is due to a performance trust violation and the other is due to a moral trust violation. We designed two experiment conditions. In both conditions, participants play 7 rounds of the game. The robot’s behavior is predefined (hard-coded) for consistency in this experiment. We implemented a similar pattern of team scores gained by the robot under different experiment conditions. However, the type of trust violated by the robot varies among those.

As mentioned in Section 3.1, the pattern of score and trust decisions by the robot are predefined. In this pattern, three rounds of desirable robot behavior (i.e., gaining a good score and adding to the team score) are followed by four rounds of undesirable robot behavior (i.e., either gaining no score or adding to the individual score). This pattern of scores is designed to build trust through desirable robot behavior at the beginning of the game. Four rounds of undesirable behavior by the robot are added to study whether the level of trust loss varies if the undesirable behavior is due to moral or performance trust violation.

All the robot notes in both experiment conditions are selected to emphasize the robot’s intentions. In the first three rounds, the robot performs well and encourages the human to keep collaborating. In the last four rounds, the dud-bot emphasizes its poor performance, and the mean-bot emphasizes its immorality through its notes to the human.

While participating in the experiment, participants took six steps. These steps are as follows:

Figure 4 shows the different steps of the experiment procedure and the path participants of each experiment condition follow during this study.

To assess the effects of the robot’s undesirable behavior on the perceived trustworthiness of the robot by participants, we added multiple objective and subjective trust measures to this experiment. These trust measures will be described in detail in the following:

We used the Mann-Whitney, Kruskal-Wallis, binomial, and Z-test on the data of these measures to test our hypotheses and evaluate the significance of the results gained in different conditions of this experiment.

Experiment participants, especially from online crowd sourcing platforms, sometimes do not pay enough attention to the experiment. Suppose experimenters cannot detect participants who fail to pay enough attention to the instructions or tasks they are asked to complete while participating in an experiment. In that case, the noise in the data increases. To prevent validity loss in the experiment data and results, experimenters must add some manipulation check steps to their experiment [39]. To increase the validity of our experiment results, we added some manipulation check questions in different sections of the online games designed for this study. Post-tutorial quiz: We have three post-tutorial quiz questions that participants need to answer correctly before heading to the game. These quiz questions ask about the scoring logic of the game, which participants need to understand correctly to be able to participate in the experiment. Failing to answer these questions correctly returns participants to the beginning of the tutorial.

General game knowledge questions: We have two game knowledge manipulation check questions, which are located at the end of the game procedure, right before the post-survey questionnaire. The first game knowledge question asks about the number of game rounds participants played. The other question asks about the robot’s trust decisions in the last two rounds of the game.

Hidden question in the questionnaire: We have an 8-point Likert scale post-survey questionnaire in this experiment. We added one hidden manipulation check question to the questionnaire. It is of the same format as all other questions in the questionnaire and is not recognizable from other questions. This hidden question asks participants to choose “3” among options on the 8-point Likert scale. This question is added to find careless participants who are not reading questions and select options randomly.

Data points from participants who answered either the hidden questions or any of the two general game knowledge questions incorrectly were removed from the dataset before the final analysis of the experiment results.

We recruited a total of 100 participants for this experiment. We posted this study as a Human Intelligence Task (HIT) on Prolific,¹ and we set three qualifications for participation:

Eligible Prolific workers who accepted our HIT were shown a link to the game’s web-page designed for this experiment. After playing the game and filling out the questionnaire, participants were given a completion verification link that returned them to the Prolific website to be compensated for participation in this experiment. Participants were told that they would receive $6 for participation in this experiment and only if they gained 35 team points or 17 individual points would they be eligible for $7 or $2 bonus values. However, in the end, they were all compensated for $6 baseline compensation plus $7 maximum bonus value. The survey took participants 26.43 minutes on average with 2.53 minutes standard deviation (std). This study was approved by the University of Massachusetts Lowell Institutional Review Board (IRB).

One of the most critical measures in this experiment is the post-survey questionnaire. As mentioned in the Methodology section, we used the MDMT-v2 questionnaire as the post-survey questionnaire. It has separate items for assessing moral-related and performance-related trust dimensions in human-robot interaction. We ran multiple statistical tests on the post-survey questionnaire data and compared the scores gained by the mean-bot and dud-bot in different trust sub-scales. The results of these tests are reported in the following sections.

As mentioned in the Experiment Design section, at the end of each round, we ask participants to answer three end-of-the-round questions. The first question asks participants about the robot’s performance (i.e., referred to as performance rating). The second question asks participants to rate the robot’s honesty/morality (i.e., referred to as morality rating), and the third question asks about the reason behind participants’ trust decisions. In the rest of this section, we will review the results of these three questions.

These two questions serve two main purposes:

To analyze the data from the performance rating and morality rating questions, we employed a round-by-round comparison strategy. This involved conducting Mann-Whitney significance tests between the corresponding rating arrays of the two experiment conditions for rounds 4, 5, 6, and 7, where the robot violates performance or moral trust.

We incorporated two objective measures in this experiment: trust decision and time to respond. As outlined in the Methodology section, the trust decision involved participants making a blind decision to integrate their round score into the team or individual score. Additionally, we measured the time taken to make the trust decision, specifically examining whether there were any variations in the duration required for participants to make trust decisions when the robot exhibited undesirable behavior.

As mentioned in the Experiment Design section, we included both moral trust violation and performance trust violation options for the robot in the designed game. It was observed that the moral trust violation option led to a more significant decrease in the overall trust score. However, it is challenging to determine the exact extent of trust loss caused by different types of trust violations by the robot. This difficulty arises from the unequal representation of trust dimensions in the MDMT-v2 questionnaire, where there are two dimensions defined under performance trust (8 questionnaire items) and three dimensions under moral trust (12 questionnaire items). Furthermore, our results indicated that the reliability trust dimension, classified as a performance trust dimension, was influenced more by moral trust than performance trust. This imbalance in the number of items representing each trust aspect in the questionnaire may have introduced some bias into our results, which could not be entirely prevented.

Another limitation of this experiment is the number of game rounds that participants played. In this study, participants engaged in seven game rounds, with the robot exhibiting either poor performance or poor morality from round four onwards, resulting in zero team scores in rounds four to seven. Despite the robot’s consecutive undesirable behaviors, many participants continued to contribute to the team score. When asked about their reasoning in the third end-of-the-round question, some participants mentioned that they opted to add to the team score because it was too late to start earning an individual bonus. This introduced a bias in the trust decision measure used in this experiment, as many individuals felt compelled to add to the team score due to the lack of alternative options at that stage of the experiment. However, if we had conducted 10 to 12 rounds of the game, then participants may have exhibited different behaviors regarding their trust decisions.

Due to the complexities involved in designing a game that incorporated both performance and moral trust violation options, we were faced with the challenge of creating a game logic that was not easily grasped. To address this, we developed extensive tutorials in both video and interactive formats to ensure that participants fully understood the mechanics of the game and how to maximize their bonus. However, to mitigate the potential issues of prolonged participation time and participant boredom, particularly with online participants, we had to limit the number of game rounds. As a result, our experiment was designed to include only seven game rounds.