Game Awareness: A Questionnaire
Faculty of Economics and Tourism “Dr. Mijo Mirkovic”, Juraj Dobrila University of Pula, 52100 Pula, Croatia
Games 2021, 12(4), 90; https://doi.org/10.3390/g12040090
Submission received: 27 October 2021
/
Revised: 27 November 2021
/
Accepted: 30 November 2021
/
Published: 3 December 2021
(This article belongs to the Section Behavioral and Experimental Game Theory)
Download keyboard_arrow_down
Figure 1
<p>k selection error logs for scenarios S1–S9, 2021 LR coded data. Scenarios 1–9 are denoted S1–S9, respectively.</p> "> Figure A1
<p>k selection error logs for the 2021 LR coded data.</p> "> Figure A2
<p>k selection error logs for the 2021 LR CP coded data.</p> "> Figure A3
<p>k selection error logs for the 2021 SR CP coded data.</p> "> Figure A4
<p>k selection error logs for scenarios S1–S9 for the 2021 LR CP coded data.</p> "> Figure A5
<p>k selection error logs for scenarios S1–S9 for the 2021 SR CP coded data.</p> ">
Versions Notes
<p>k selection error logs for scenarios S1–S9, 2021 LR coded data. Scenarios 1–9 are denoted S1–S9, respectively.</p> "> Figure A1
<p>k selection error logs for the 2021 LR coded data.</p> "> Figure A2
<p>k selection error logs for the 2021 LR CP coded data.</p> "> Figure A3
<p>k selection error logs for the 2021 SR CP coded data.</p> "> Figure A4
<p>k selection error logs for scenarios S1–S9 for the 2021 LR CP coded data.</p> "> Figure A5
<p>k selection error logs for scenarios S1–S9 for the 2021 SR CP coded data.</p> ">
Abstract
:This paper deals with one of the possible methodological approaches to an empirical examination of game awareness. Such an outline should enable the gathering of a deeper understanding of individuals’ awareness. The questionnaire is based on a scenario technique, and it is focused on game awareness in situations with possible strategic interactions. The goal is to assess the instrument regarding its reliability and validity. Internal reliability is assessed using Cronbach’s alpha coefficient. A k-nn analysis was used to estimate divergent construct validity and criterion validity, and nonparametric factor analysis was used to estimate convergent construct validity. The questionnaire satisfies the requirements of reliability and validity. Moreover, there is an indication that each of the eight scenarios may be used separately, as they also meet the criteria for reliability and validity. Possible practical and theoretical implications involve questionnaire use as a basis for educational intervention, other instruments’ development, as well as for the collection and publication of behavioral insights about game awareness. Behavioral insights should enrich the theory and incite behavioral game theory models of game awareness and its implications.
1. Introduction
Game theory allows for the analysis of behavior in strategic situations. While its origins lie in mathematics, its applications spread over various disciplines with an emphasis on economics. The parallel development of behavioral economics revealed the additional potential of game-theoretical analyses of strategic interactions while questioning the boundaries of rationality, which led to the development of behavioral game theory.
Regardless of the type and form of the game, there are basic elements—players, strategies, and payoffs—considered as presumptions and preconditions for the game’s existence. Only recently, another question arose as a possible game presumption, regarding the players’ awareness of the game. The issue can be separated into two questions, where the answers to both have strategic implications. The first one regards whether an individual is aware of the game’s existence. The unaware individual will most likely miss the opportunity for strategy consideration and implementation, namely, miss the opportunity to improve their own outcome. The lack of game awareness may result in situations such as truncated models, where one player passively accepts the consequences of the other player’s decisions (for example, like in a dictator ‘game’). The second question is whether the player is aware of the game in the same way as the modeler. While the answers to the two questions are connected, there is a distinction. The second question can be answered only if the situation is assessed as probably game-like, namely, if an amount of game awareness exists. The answer to the second question requires the awareness and assessment of the game elements: players, strategies, payoffs, and game rules. The players may not be aware of all available and feasible strategies. Still, they may also believe that they can use strategies that are not feasible or that have not been foreseen by the modeler. Similarly, players may not be aware of the existence of another player or may recognize unrelated actors as players. The payoff and outcome awareness are inevitably related to the awareness about the players and strategies. Moreover, a person who believes that there is a game where there is none may initiate a new game. The two questions require a distinction between the existence of game awareness and game awareness, where the latter may also be observed as the awareness of the game elements.
Even in examinations of bounded rationality, standard game representations often implicitly assume players’ awareness and understanding of the game in the same way the modeler does [1]. That also means that the game elements are treated as common knowledge. Still, the growing body of research in behavioral economics regarding human fallacies, biases, and individual differences in perception and decision-making processes calls for a reconsideration of such an approach. Moreover, modeling awareness, or rationality for that matter, as a deviation from the optimum, and behavior deductions from that deviation, is justified if the goal was to prescribe the behavior, and not if the goal was to describe, explain, or predict an actual individual’s behavior.
Depending on the context, the practical consequences for an unaware player may vary from imperceptible to serious. From the modeler’s perspective, the presumption of a player’s awareness requires the model adjustment, as long as the goal is to describe, explain, or predict the behavior and not to prescribe optimal behavior. Therefore, the relevance of the awareness assumption is revealed in various aspects: it enables a more realistic analysis and understanding of individuals’ behavior in strategic interactions; it creates the basis for an intervention; and it allows for an enhancement of the theory and modeling. Nevertheless, the role of awareness should not be misinterpreted, as awareness alone is not a guarantee for success in a game or in decision making, nor for rational reasoning during the game. Instead, it should be observed as a prerequisite.
Due to the importance of an awareness of the game and its elements, and its frequent omission in the game’s theoretical modeling and the scarcity of research, the issue of awareness requires a deeper understanding of its occurrence and chartered methodological approaches for empirical and theoretical aspects. This paper deals with one of the possible methodological approaches to an empirical examination of game awareness that should enable a deeper understanding of individuals’ awareness by using a questionnaire based on scenario techniques. Game awareness is examined through its ‘building blocks,’ the awareness in a narrow sense, and the assessment. As the purpose is to obtain insights at the individual level, it is not assumed that the game awareness must be complete, nor that it must be infallible. In addition, during the process of becoming aware, one might realize that there are some aspects that one is unaware of, or that one’s judgment may be fallible, and thereby assign beliefs or confidence to the accuracy of one’s own awareness and assessment. The goal of this research is to assess the instrument regarding its validity and reliability. Such an instrument could then be used for an exploratory investigation of awareness at the individual level, and to gain new insights into this overlooked but important area.
The following section introduces the methodological aspects, starting with the theoretical basis, and followed by a description of the assembling of the questionnaire and scenarios, of the studies and samples, a comment on the coding issues, and the validity and reliability conditions. The results section presents the reliability and validity results. The discussion and conclusion section tackles the limitations, implications for further research, and possible practical implications.
2. Methodology
2.1. Theoretical Basis
In defining game awareness, it is helpful to start from previously set frameworks. A part of awareness definitions offers a broad and strict approach, where a person is unaware of an event if s/he does not know about an event or does not know if the event is true, and does not know that s/he does not know it [2,3]. Those approaches are exclusive, assuming an absolute state of knowledge, without incomplete information or uncertainty that occurs in reality or personal characteristics that shape one’s awareness. Nevertheless, such approaches are common in economic modeling and standard game representation, where unawareness causes deviations from the optimal outcomes. Considerations that account for an individual’s characteristics result in the inclusion of psychological processes in definitions of the (un)awareness. In that sense, unawareness can be observed as “a real-life phenomenon associated with an unconscious mental state directed toward or lacking positive knowledge about a definite event” [4].
To explain awareness at the individual level, a part of the definitions emphasizes the role of perception [5,6,7,8]. Moreover, awareness involves various psychological processes [6] and is limited with a volume of available time and space [7]. In addition, the notions of seeking, using [5,6], meaning comprehension, and projecting that meaning to the near future [7] are included in behavioral approaches to awareness. That may seem counterintuitive, as seeking, using, and sharing information implies that there is already an amount of awareness that something is going on.
Most definitions, more or less, paint the same picture: there is a piece of information, but it is not used, and the reasons for that may be errors in perception (failure to see) or mistakes in interpretation (due to motivation, preferences, or knowledge). However, the consequences are much the same, as unaware individuals “cannot conceive of, nor articulate, the decision relevant contingencies they are unaware of” [9]. However, this quote calls for a deeper insight. For example, suppose one hears some racket outside of the office. In that case, one is aware that ‘something is going on,’ but, at that point, is still not aware of what exactly is going on in any way relevant for making a decision or taking an action.
From the modeling perspective, in the broadest sense, a game exists if there are: a strategic interaction; at least two players for which the choice of actions of at least one of them should imply the presence of a strategy; and there exists at least two or more possible outcomes shaped by the players’ choices [10]. Given that the basic game elements and presumptions must be satisfied for a game to exist, then the awareness of those elements should also be a threshold for determining whether a game exists or not. Therefore, an individual is game-aware if s/he is aware of the game elements.
Neuberg [11] examined how social behavior can be influenced by information that is perceived outside of conscious awareness. He observed players’ sensitivity to subliminal information through a strategies adjustment. The experiment reveals the intuition of individuals’ who grasp signals about the strategic interaction from their surroundings. The notion of perception outside of conscious awareness implies no deliberate reasoning and no definitive knowledge involved. This captures an aspect of game existence awareness, which may be observed as an aspect of, and as a minimal amount of, the awareness required for the volatile examination of game elements and of the reasoning about the possibilities that result in the game awareness.
In a narrow sense, a person becomes aware of an element as s/he detects an element. However, at that point, a person has yet to become aware of the properties and meaning of the element (identification and evaluation of the objects as the building blocks of awareness, [12]). In that process, one might become aware that there may be some properties or relevant information that one is unaware of ([13], or as in shadow propositions, [14]) or that one’s judgment may be fallible, and assign beliefs or confidence to the accuracy of one’s own awareness and assessment. The dynamic approach to unawareness assumes changes in beliefs as an individual becomes more aware [14]. The awareness and beliefs may change as individuals obtain more information (properties of the object), but the change may arise due to their reasoning about those properties, combined with memory retrievals of previous knowledge and experience, consequently assigning beliefs that their assessment is true. In addition, an individual’s reasoning may be bounded by the cost of reasoning, which may relate to the perceived outcome of the game and individual characteristics [15].
In a game, a player values different objects, or states of those objects, the same if they have the same properties [12]. Even so, the player can value objects differently if they can be categorized differently in any way that s/he is aware of. The player can base their preferences only on the properties that s/he is aware of [12]. This approach assumes a partial awareness. Moreover, a generalized approach to (un)awareness suggests that a person may be involved in a game, but is playing it according to their own subjective representation of the game, which may include ‘virtual’ moves that do not exist in the underlying game [16]. This provides another aspect of awareness—while individuals may not use available information, they may also use information unrelated to the current game [10], which results in a subjective game representation, and as a consequence, payoff beliefs that include moves unrelated to the actual payoff or the underlying game [16]. A standard game approach to unawareness might benefit from behavioral insights.
Blasch et al. [17] offer another approach by differentiating awareness and assessment. However, the awareness and the assessment as psychological terms seem to be entangled in the (un)awareness approaches [12,13,14,15,16], with the addition of understanding [1], seeking, using, and sharing [5,6] information. When combining inductive and deductive approaches [10,11,12,13,14,15,16,17], a compromising approach may be deduced: awareness, in a narrow sense, is a binary occurrence and a precondition for assessing the game elements. At the same time, assessment is required for developing the player’s awareness. Such an approach enables insights into behavioral aspects from empirical data, but in return, it may be compared to the logical and standard game approaches. This is the perspective that will be examined in this research, separating awareness and assessment, but examining both and their relationship to the game awareness.
Such an approach assumes that, if an awareness of game elements occurs, it entices a game elements assessment, resulting in an amount of game awareness. It can be theorized that a ‘fully’ game-aware person realizes the situation, its elements, and the properties of these elements, using all available information in a given framework. Such an approach is relatively close to Laplace’s demon and to a classical approach to rationality. However, the research in behavioral economics and related fields [5,6,10,18] shows that people do not use all available information, and that they use information outside of a given framework. Nonetheless, they also may be aware of only some elements or properties of the elements while disregarding others, and thus, gain an amount of game awareness. Game awareness and its components can but do not have to be observed as a binary occurrence.
Moreover, if the goal is to gain insights into individuals’ awareness, game awareness should not be regarded as a binary occurrence. As bounded and partial awareness approaches—each from its own perspective—denote, the gained awareness does not have to be complete, nor does it have to be infallible, but it enables the articulation of relevant decision-making elements. In addition, it may differ between individuals, as suggested by systematized results of experiments involving awareness and ethical choices [18].
In addition to the experiments, scenarios are often employed to elicit various elements in the decision-making process. While experiments dominate the research of decision making where narrow and particular goals are specified, scenarios are more appropriate for addressing complex and uncertain contexts [19]. Exploring new research directions is always a challenging ordeal, but scenarios are a helpful methodology for challenging existing assumptions while maintaining scientific rigor and usable results [19]. In that sense, this can be observed as an attempt to challenge the standard approach by introducing an examination of behavioral aspects as the first step in the possible implementation of empirical findings into awareness modeling.
If an individual were to play a game and deviate from the optimal choices, it would be hard or even impossible to establish, only by observation, whether the behavior occurred due to the unawareness of some game elements or due to bounded rationality applied to in-game reasoning. Questionnaires help assess phenomena that are otherwise hard to observe or assess. Such is the situation with the game awareness. At the individual level, with human subjects in mind, the awareness inevitably involves psychological processes. Psychological abilities and traits are often explored and assessed using questionnaires. Combining the scenario technique with a questionnaire enables an exploration of game awareness, uniquely positioned within the game theory and behavioral economies, respectively.
Given the chosen approach to the game awareness, scenarios should involve a game-like or a pre-game situation, with a possibility (and not necessarily a requirement) of strategic interaction. Unstructured negotiation games that resemble real-life situations created some of the most valuable findings in pre-game theory [20]. This served as an additional inspiration for the scenario approach and the use of unstructured, open-ended scenarios.
2.2. The Questionnaire and Scenarios
Game-like situations are here considered as realistic situations that involve a possibility for strategic interaction that may lead to virtual or real payoffs (utility). Given that the questionnaire was to be distributed among university students of economics and business economics, the real-life situations chosen for scenarios refer to simple situations.
Chosen situations relate to possibilities of strategic interaction in a purchase situation, a hangout with a friend, a job-related issue, a rumor about a raise, a task division, a competition, an opportunity-seizing situation, a confession (adapted prisoner dilemma), and a sharing situation (adapted trust game). Situations differ given their scope (possible areas of application of findings), complexity, the amount of provided information, and the gravity of potential consequences. While each situation presented in the scenarios would require a specific approach to modeling in the standard game representation, all these situations may occur for a person. This set of scenarios also enables insights into possible differences in the awareness formation given the different settings. Different situations, especially the ones related to economic aspects that an individual may encounter, offer a possibility for the findings’ comparison to the previous approaches to awareness, as well as to other research in behavioral economics. The findings might enrich, but also confirm or refute, existing choices in modeling individuals’ awareness.
Individuals are, at least, partially bound by the evidence and its possible interpretations, meaning that they cannot believe anything they would want to [18]. That means that the awareness, at least partially, arises from the available information. To examine sensitivity to provided information, scenarios should differ concerning the provided information about the player(s), their actions, payoffs, and interaction type. In addition, it can be assumed that the knowledge (possibly related to the situations described in scenarios) and preferences, as well as perception and interpretation errors, vary among individuals. This leads to an assumption that the responses will vary between individuals and between scenarios.
To assess if there is a game, the individual must assess game elements. This can be done by answering a set of questions [10]:
- Are someone’s actions and strategies influencing my possible outcomes, and which are those actions?
- Are there, and who are, the players who choose those actions and strategies?
- What are the possible outcomes, and how do other players’ actions influence the set of possible outcomes?
- What are the other players’ preferences, given the employed actions and strategies?
The questions that follow each scenario regard the list above and consider game elements, namely, the awareness about them, and the assessment of game elements, in line with Blasch et al. [17]. Given that most scenarios are unstructured game-like situations with incomplete information, it is assumed that the respondents are uncertain of their responses. That means that a level of belief, certainty, or confidence is assigned to each answer. These certainties can be observed as: the person believes, at least to a degree p, that there is a game element e (as in [21,22]). The entire range [0, 100] as offered to respondents to express their belief (certainty of confidence), instead of a shortened scale (such as a Likert scale), as it allows respondents the freedom of expressing themselves, follows the theoretical range of probabilities, and enables responses’ analysis as scale variables.
Additionally, observed elements (or signals about elements) may be assessed as deliberate or accidental [23], and such may be their consequences. It seems reasonable to assume that if an activity is assessed as accidental, one will likely assume that the game does not exist. Following that thought, a question about perceived deliberation is added to the questionnaire.
The game awareness questionnaire (GAQ) comprises nine scenarios, with 20 questions related to each scenario, and nine resulting items were created (Table A1). The first seven scenarios regard every-day and simple business/economic situations that may (or may not) be games, while the eighth one presents an adapted prisoner dilemma, and the ninth one presents an adapted trust dilemma.
Simple language should ensure the respondents’ cooperation and engagement [24,25], so common terms were used. The terms used for examining awareness (in a narrow sense) were ‘is there some…’ and ‘detect’, and the terms used for examining assessment were ‘determine’, ‘identify’, ‘describe’. Instead of the word ‘action’, the term ‘activity’ is used, due to its close meaning and to avoid confusing respondents who are not familiar with game theory terms. Instead of ‘players’, a descriptive phrase such as ‘a person (or a group/association/company, etc.)’ was used. To avoid using the term ‘payoff,’ in the first study, the term ‘welfare’ was used in the first question, but was later changed to ‘outcome’.
There are only three open-ended questions per scenario, as they tend to be burdensome to respondents [24]. However, open-ended questions are useful when the presented choices could influence responses, when the answers are not entirely predictable, and when there are potentially unanticipated responses that require flexibility [26]. For example, reasoning, computation, and memory retrieval can, but do not have to be done at the conscious level, as Salehnejad [27] finds that learning through continuous belief-updates makes the homo economicus an intuitive Bayesian statistician—where some of the reasoning does not occur at the conscious level, and neither does the update strictly follow conditional probabilities. The individual’s assessment can be achieved by reasoning or intuition [28], where the intuition, along with the individual’s base of experience and knowledge, might lead to unexpected responses. In addition, an individual’s reasoning may be improved, as it involves skills that can be learned, so revealed fallacies in answers to open-ended questions in a situation assessment may be used (for example, in the classroom) as a basis for intervention [29].
Each scenario is a complex issue, and there are 20 questions per scenario. Examination of a complicated issue with a single question could induce context effects [30]. The sequence of the questions is also relevant for the successful measurement by a questionnaire [24,31,32], so questions are ordered to allow a step-by-step assessment of the situation, which is in line with a specific research goal [33].
The ambiguity of the first seven scenarios is achieved, to an extent, by not introducing the possible action for the respondent. The motivation for such a choice lies in the theoretical distinction between game awareness (a person being ‘fully’ aware) and game existence awareness (a person being aware in a narrow sense). One should first become aware that something is going on, assess what is going on, and only then approach an assessment of what one can do to influence one’s own outcome (introduced as the last two questions, Q19 and Q20). In other words, based on the game’s existence awareness, the situation can be suspected to be a game or not. Only if one believes that there is a possibility for a strategic interaction, it makes sense to put any effort into the examination of the set of possible and feasible strategies/actions as a reaction to other players’ actions. The first seven scenarios (S1-S7) mimic every-day situations, which usually do not occur as games with predefined rules but require individuals’ awareness and assessment. On the contrary, scenarios 8 (S8) and 9 (S9) are structured and, as such, can be used as reference points.
The questionnaire was pre-tested during the winter semester of the 2017/2018 academic year with a small sample of 89 first-year students. Each of them responded to questions from one of the first seven scenarios. The insights revealed that the respondents were able to grasp the received information, assess (at least some) elements and/or their properties (with a few unexpected conjectures), and were aware of (at least some) game elements. The questionnaire was completed in the pen-and-paper form. During its completion, a researcher was present to answer any students’ questions and kept notes to remove unnecessary ambiguity in scenarios or to paraphrase questions for the subsequent studies. While the pre-test was helpful, only a few necessary changes were required, which is in line with Hunt et al. [34].
It is also necessary to note that there is an amount of logical dependency between the questions: awareness and assessment are intertwined, even in theory; the action must be taken by a player; and if a person can be detected as a player, then there should also be a related action. In addition, if there is an action, there should exist different outcomes. At least, this is true from a logical and classical game theory perspective. However, humans are prone to biases and frequently escape the bounds of rationality. The pre-test showed that students could differentiate elements and could determine that there was an activity, even if they could not assess who initiated it, or that they could notice another player and not be able to pinpoint the activity. This was expected, as individuals may evaluate each object or, in this case, game elements, separately [12,14]. Moreover, this confirmed that the questionnaire captured patterns and (in)consistencies, due to the information provided in the scenarios.
Table 1 offers an overview of the provided information. Scenarios S2 and S4 offer the least amount of information, followed by scenarios S1, S3, S6, and S7. Scenarios S5, S8, and S9 offer the most information, whereas scenarios S8 and S9 also offer alternatives for their own action choice and respective outcomes. Scenarios S1, S6, and S8 suggest adverse outcomes, scenarios S4, S7, and S9 suggest positive outcomes, and scenarios S2, S3, and S5 imply possibilities for positive and negative outcomes. Scenarios S6 and S7 allow for an additional examination of short- or long-term respondents’ orientation. Scenarios S1, S4, and S6 offer possibilities for parallel grasping of the situations: at the personal and the market levels (where the latter requires more deduction and abstraction). Scenarios S1, S5, S6, and S8 offer a simple, observable type of activity, while scenarios S2, S3, S4, and S6 introduce a possible activity as a rumor or non-verbal signal. It could be argued that scenario S9 offers a simple, observable activity, but that is true only for the respondent’s actions. In contrast, the colleague’s action is a promise, which may be observed as cheap talk. Scenario S7 is distinct regarding the activity, as it suggests only an invitation, where a further development of the situation does not occur unless the respondent chooses to pitch the idea. For any other scenario, passive acceptance leads to an outcome—as in a different state of virtual or real utility, including affective states (such as satisfaction)—compared to the state before the initiated activity. On the contrary, scenario S7 requires an action by respondents to change the outcome. Noted variations in scenarios should enable the capture of how respondents form game awareness (or a lack thereof), given different amounts and qualities of the provided information. In addition, while the questionnaire may be used as a whole, each scenario may be used separately to examine the desired aspect. Other aspects of the questionnaire’s development are explained in the following sections.
2.3. Studies and Samples
The research ethics assessment committee at Juraj Dobrila University of Pula gave their positive opinion on the questionnaire and research design. Informed consent was obtained from each participant. Three studies were conducted via an online questionnaire during the summer semesters of the 2018/2019, 2019/2020, and 2020/2021 academic years in Pula, Croatia. The first study included only the first seven scenarios, without the questions Q19 and Q20, and with the ‘don’t know’ option. The second study was enriched for the eighth and the ninth scenarios, with minor paraphrasing of the questions and the removal of the ‘don’t know/not sure’ option. That option was removed due to coding and item-interpretation difficulties (which will be discussed in the Coding Issues section). The third study involved nine scenarios with 20 related questions.
In the third study, the students were not present in the classroom while filling in the questionnaire, so an additional written instruction was provided at the beginning of the questionnaire: “Please, stay focused while answering the questions, but also try to answer as spontaneously and honestly as possible. An answer to every question is required. It takes about 30 min to complete the questionnaire, and answering questions about the assessment of the situation requires a higher level of attention. There are nine scenarios in the questionnaire. For each of them, you are asked to notice and assess whether there is an element and to attribute your belief to it, i.e., the probability of how confident you are in your answer”.
In each study, completion was voluntary, but the points incentivized the students if they chose the questionnaire instead of their regular practice assignment. The students received the same amount of points as they would for a regular assignment. They were informed that the responses would be evaluated before the points assignment and that the points would be assigned only if they provided thoughtful answers. For the 2019 study, in the assignment involving the questionnaire, students functioned as pollsters and collected the responses from other students at the university after their own completion of the questionnaire. As a result, the 2019 sample is more heterogeneous than the other two. The samples from the 2020 and 2021 studies involve only first- and second-year undergraduate economics and business economics students. The samples’ characteristics are provided in Table 2 (Here is reported a summary of the data coded by LR without complementary probability (more on this topic in the Coding Issues section). The differences in the number of responses arose because of the removal of unusual cases during the data examination. The summaries for the loosely coded data with complementary probability and the strictly coded data with complementary probability are presented in Table A2 and Table A3, respectively).
The applied incentives may be considered partial or weak when examining economic actions and decisions, which take effort and time to carry out. In this case, only the awareness of possible strategic interaction is examined. In other words, concerning situations precede situations that require extensive in-game deliberation and action. In addition, to ensure that the responses (and deviations from the described situations) were an indicator of awareness and not possible mistakes due to lack of motivation, the responses to the open-ended questions were considered as a proxy for the engagement, and uniform or unusual responses were removed from the further analysis. This is also one of the reasons for the lower percentages of response retention.
In addition, a rate of completion must be discussed. In the first study, 198 respondents approached the survey and completed the questionnaire, and 117 (59.09%) responses were retained after data insights. In the second study, 113 respondents approached the survey, 84 completed the questionnaire, and 77 (68.14%) responses were included for further analysis. In the third study, 94 respondents approached the survey, 65 completed the questionnaire, and 47 (50%) responses were kept. However, each of the retained respondents replied to every scenario, which leads to a much higher number of responses per study. The reason for such a small retention lies in duplicate and uniform responses. Some duplicate responses occurred if a person clicked on ‘Finish’ more than once before receiving feedback from the survey platform. Additionally, the difference between surveys and completion approaches occurs, for most cases, when respondents provide informed consent, but after that, leave the survey (the next approach to filling in the survey registered them as different respondents). Incomplete and duplicate responses to scenarios were excluded, as well as unusual cases detected in the data examination. Uniform responses to open-ended questions (as in ‘I don’t know’ or ‘I cannot assess’) between scenarios are used to exclude all answers by identified respondents from the data. Another reason for the lower shares of usable data may be the questionnaire length, so the possibility of using the scenarios separately must be examined.
While the first sample is more heterogeneous, the second and the third samples regard respondents with similar backgrounds. Despite homogeneity by some characteristics, some of them could have experience or knowledge about one or more situations examined in the scenarios, while others will face the scenarios without prior ‘intuitive probabilities’ assigned to the events. If previous experiences or knowledge vary too much between the respondents, it could affect responses and consequently the questionnaire’s internal reliability.
2.4. Coding Issues
The questionnaire involves qualitative responses, which should be recoded for further analysis. A specified amount of certainty is assigned to each answer as a percentage (0–100%), which does not require recoding. All ‘yes’ and ‘no’ answers are coded 1 and 0, respectively. The ‘don’t know’ options were supposed to highlight uncertainty and were initially coded as 0.5, but the problem occurs in the interpretation: a person does not know and is, for example, 50% confident that s/he does not know. If a complementary probability approach is applied, then it follows that there is 50% certainty that the person does know. However, it remains unclear whether the response would be ‘yes’ or ‘no,’ although it is plausible that the response would lean towards the ‘no’ option. While the interpretation does not make much sense from a mathematical standpoint, the reason for such choices may be the lack of information or the lack of trust in one’s own assessment. While the ‘don’t know’ option offers insight for descriptive, qualitative analysis, it represents a difficulty for quantitative analysis. For further analysis, ‘don’t know’ options were not excluded from the 2019 study data to prevent data loss (all responses for a scenario per respondent should be removed if one ‘don’t know’ option appeared) and were coded here as 0. This issue does not occur for the second and third studies, as there were no ‘don’t know’ options.
The next issue regards open-ended questions (Q3, Q7, and Q13, Table A1) and the coding of explanatory answers. The content of the answers was coded by only one human coder—the researcher. There are two possible approaches, coding by the loose rules (LR) and coding by the strict rules (SR). The first step in both approaches was an a priori approach to the coding [35,36], where the replies containing keywords strictly in line with the scenario were coded 1 (with the help of the search option). The second step included another overview to check for responses with different word choices that reflect the meaning in line with the scenario content. The first seven scenarios do not offer all elements, so the next step required evaluating answers about not offered elements by the criteria of reasoning that directly reflects the information provided in the scenarios. In SR coding, the rest of the replies were coded 0, while these steps created a base for the next step by the LR coding. Within the next step in LR coding, the rest of the responses were additionally examined by the criteria of any possible reasoning of responses in line with the scenarios. Moreover, one of the advantages of the scenario technique—multiple interpretations that can be considered in terms of their plausibility [19]—could not be utilized without such an approach. While this step might be observed through the lenses of content analysis, it must be noted that this part of coding included only an evaluation of whether the offered responses could be regarded as a possible element assessment (coded as 0 or 1), without further description, a subcategories derivation, an additional thematic clustering, or a narration of the answer to the research question, which are the usual elements of content analysis [37,38].
Nevertheless, that step may be observed in line with the interpretivist paradigm, which must be addressed. When a researcher performs coding in the content analysis alone, there is a risk of subjectivism, leading to potential replicability problems [36,37]. The usual approach to the issue is using coders unrelated to the research (or unaware of the research hypothesis) and checking inter-rater reliability. However, that approach is not always recommended and can also be criticized [39,40]. In addition, it can be “beneficial that the analyst is able to use their own unique skills, talents and expertise” in choosing the method and coding qualitative responses [36], but with constant internal scrutiny. The rest of the coding was conducted with a continuous reflective audit to acknowledge and minimize subjective framing [36,41]. While performing the third step in LR coding, due attention was paid to coding similar concepts of responses in the same way [42]. The revealed possible interpretations that can be reasoned from the scenarios were compared and their meaning in relation to scenarios rechecked. The following explanations and examples will describe LR coding issues and display the comparison to SR coding.
The idea behind the LR approach is to allow the respondents to subjectively perceive the available information about a situation. The responses are coded as 1 if respondents provided any assessment that can be reasoned from the scenario. This approach is considered more appropriate for examining how individuals actually assess situations and form their awareness. The SR approach is a ‘modeler’s perspective’ approach, where only the responses deduced directly from scenarios are coded as 1. For the first seven scenarios, the difference between LR and SR data is smaller. An example from scenario 8 paints a picture of the differences in the assessment and game understanding: a part of the respondents perceived the situation as a cheap-talk game with the interrogator (examples of responses for the player identification in such cases: “ethical committee,” “a member of the ethical committee,” “the professor who accused me”), which was coded as 1 by LR, and 0 by SR. In the SR approach, only the responses about player identification that indicated a colleague were coded 1, else 0. In scenario 9, a part of the respondents observed the situation as the problem of commons, where someone took an extra sandwich or the organizers did not prepare enough for everyone (thus, a kind of a protest is in order) with activity descriptions such as “they did not have enough sandwiches to share” in combination with the player described as “the person in charge for the brunch” (coded 1 by LR, 0 by SR).
Another example is from scenario 7, where a few respondents introduced their family or friends as players, explaining that they must have contacted investors who offered to pitch the idea. In the same scenario, a few respondents described the activity as an “economic activity,” coded 1 by LR and 0 by SR (as no explicit description related to the scenario was offered). In scenario 1, a part of the respondents introduced the other buyers or product presenters as players and used their reactions to the price increase to justify their positions (mostly decisions to buy) because of persuasion (“influencers impact my thinking about what I like and love”, “…in the shop, there are employees that perform so-called activities that may influence my purchase”, “other buyers must be protesting to such things”—all coded LR: 1, SR: 0). In the same scenario, if an outcome description was “I will be sad because of a loss of a chocolate,” it was coded 1 by both LR and SR. However, if the outcome was described only as “I will be sad” (or any other affective state, without the explanation or the relationship with the scenario), it was coded 0. The same approach was applied to similar situations in different scenarios. While a relatively small percentage deviates as much from the information provided in the scenario, it is relevant to note the differences and address them. All descriptive responses of “I don’t know”, “I’m not sure”, or “I am uncertain” were coded 0. It might be interesting that in each of the studies, a couple of responses about the player and activity description that sporadically occurred were, for example, “no one can influence me”, “no one can influence the outcomes in my life but me”, or “there may not exist such activity that would influence my outcomes” (as in general), and were coded 0 (although, interestingly, the outcomes’ descriptions of the same respondents indicated that they do recognize the differences in the outcomes). While coding, descriptive responses about the activity, player(s), and the outcome have been considered together to get a comprehensive insight about a person’s grasp of the situation, as it can be seen from some of the quoted descriptions that the activity, players, and the outcome may be interlinked. The frequencies for binary-coded questions are presented in Table A4 and enable a comparison of the data per scenario and study, given the LR and SR approach.
The open-ended questions are relevant, as they provide a deeper insight into awareness and assessment that cannot be revealed by yes/no questions. In this case, they were coded in the simplest way to be utilized for questionnaire reliability and validity examination. However, these questions offer much more by enabling a descriptive, qualitative analysis. In addition, the responses may be used for future questionnaire development by forming statements concerning nine scenarios.
The set of questions was later used to derive nine items: activity awareness (I1), activity assessment (I2), player awareness (I3), player assessment (I4), other players’ outcome awareness (I5), own outcome assessment (I6), game existence awareness (I7), deliberation (I8), and own activity awareness (I9). Two approaches were used for that: with and without assuming that the belief is governed by complementary probability (see Items in Table A1). Following the first approach, the binary answers were multiplied by their respective beliefs and divided by 100 to create a respondent’s value for an item. Following the second approach, the calculation was applied: if the binary coded answers took the value of 1, then the item was created the same way as for the first approach; if the binary coded answers took the value of 0, then the complementary probability (CP) was used to derive a value. The framing effect suggests that individuals would respond differently if asked ‘is there no activity,’ so this represents a modeler’s standpoint, ignoring possible biases. Likewise, Piermont [9] examines decision makers’ unawareness and offers two explanatory sentences: (1) “It is raining or it is not raining”; and (2) “The axiom of choice implies that every vector space has a basis”; and concludes that although they are tautologies, the decision-maker will treat these statements differently if s/he is unaware of set theory. Following the same thought, in this case, respondents should understand complementary probabilities and apply the rule when assigning probabilities. Although there are theoretical considerations that argue against CP coding, that approach will also be explored here, and the results will govern the decision about its applicability.
While the first approach (LR) may seem stricter, it reflects provided choices and responses directly. The second approach (SR CP) can be observed as a modeler’s perspective, with an underlying assumption of logic and complementary probability. It can be argued that the second approach is favored if the goal is to fit the empirical data into the game theory framework and measure deviations. In contrast, the first one is favored if the goal is to gain insights from empirical behavioral data and use them to improve modeling [10], as is the case in the behavioral economy using psychological approaches.
Given the different coding possibilities, three datasets were created from the original data, coded by: loose rules without complementary probability (LR), loose rules with complementary probability (LR CP), and strict rules with complementary probability (SR CP). Given the coding issues, validation and reliability must be examined not only from the perspective of the questionnaire, but the coding variations as well.
2.5. Reliability and Validity
The next step in the questionnaire’s development is an examination of its validity and reliability. Reliability can be observed from three aspects: equivalence, stability, and internal consistency [30]. The first one denotes the equivalence of measures by applying two or more instruments that measure the same phenomenon. This aspect cannot be examined, as no other questionnaire of game awareness is known to the author, so none was applied. Stability denotes how stable the measurement is, in the sense that the same or similar measures are obtained with repeated measurements. This involves repeated instrument administration to the same group of respondents over time. In this case, a proxy for the stability is considered—the results of instrument application to distinct groups of respondents with similar characteristics and backgrounds. Internal reliability, or homogeneity, denotes how consistent the instrument is over all its parts [31,43]. For this purpose, Cronbach’s alpha coefficient is usually used [24,31,43,44] The decision-making criteria for using Cronbach’s alpha are: 0.9–1 is excellent reliability, 0.7–0.9 is high reliability, and 0.5–0.7 is moderate reliability [34]. Kline [45] suggests the value of 0.8 as a threshold for ‘very good’ reliability, and that approach will be used here. While it is a useful indicator, reliability must be combined with the validity, and vice versa, as instruments should be both valid and reliable [46,47].
Validity measures how successfully the questionnaire captures what was intended to be researched. In other words, it answers the questions of whether it measured what was intended to be measured. Several aspects of validity can be explored, such as content validity, construct validity, and criterion validity [43]. Moreover, construct validity should always be examined to justify the meaning of the measure and its subsequent use [48]. Convergent construct validity refers to the degree to which an examined measure correlates with measures that assess the same phenomenon.
In contrast, divergent (discriminant) construct validity indicates variables that are not supposed to be correlated to an examined construct and, that in fact, are not correlated [49]. Most often, it is performed using factor analysis [43]. Exploratory factor analysis can be conducted using different approaches, but principal component analysis (PCA) is often applied. However, whether exploratory or confirmatory, factor analysis requires data that satisfy the assumptions of a linear relationship, multivariate normality, a correlation between variables and factors, and no multicollinearity [50]. While in some cases of minor skewness in the distribution of individual variables, normality may be considered a soft requirement, a detailed examination reveals that the data are not normal, and transformation does not levigate the issue. While there are weak-to-moderate correlations between the variables and no multicollinearity, pair-wise observation reveals that underlying relationships are not linear. These data require nonparametric approaches, such as logistic, logit, and probit models, Spearman correlation coefficient, clustering, decision trees, or k-nn (kth nearest neighbor) analysis.
Each of these methods have pros and cons. Most of the proposed nonparametric approaches allow conclusions about discriminant construct validity, but logit and probit models also allow for decisions on the predictive power of the instrument (criterion validity). However, logit and probit models require dependent variables, meaning that one of the variables should be treated as a dependent; as a logical and theoretically supported choice, that would be I7 (game elements should be detected and assessed to form the game awareness). However, logit models belong to a group of models with limited dependent variables, meaning that I7 should be recoded into a binary or ordinal variable, which implies information loss.
The most frequently used approach for nonparametric discriminant analysis is k-nn analysis, which is a classification method. The method classifies observations, given their distance to the nearest neighbors. The choice of the number of neighbors to use for observation classification is governed by error minimization. The analysis can be conducted with or without a target variable and allows for additional validation by partitioning the sample into training and testing subgroups. There are more approaches to measuring distances between the neighbors, but the Euclidian metric is often applied. Usually, the mean of nearest neighbor values is used to predict scale target, but median values can be used too. The k-nn analysis will be conducted by setting I7 as a target variable (allowing the conclusion about criterion and discriminant construct validity), with partitioning the sample to the training (60%) and testing (40%) part, with medians of the nearest neighbor values used for the prediction of a scale target and, Euclidian metric used for the distances between the neighbors.
There is also a nonparametric alternative for exploratory factor analysis that can be used for convergent construct validity assessment. Such an approach is based on a scale-free, nonparametric, Spearman’s correlation coefficient matrix, instead of a Pearson’s correlation coefficient matrix [51]. As Pearson’s and Spearman’s correlation coefficients lead to the same values if scale variable values are replaced with their ranks, a rank transformation is a link that connects parametric and nonparametric statistics [51,52,53]. In general, Spearman’s rho is a much better choice for heavy tails and asymmetric distributions [52]. If used in a PCA, “factor structure is more stable if researchers simply base their multivariate analysis on the rs matrix” [52] (p. 287). The rest of the exploratory factor analysis procedure follows the standard approach, using matrix algebra to derive the component matrix, where the criteria for the number of components are eigenvalues greater than or equal to one and relevant factor loadings take values greater than or equal to 0.4. The analysis will be conducted using SPSS (IBM, New York, NY, USA) and exploratory factor analysis based on Spearman correlation coefficients matrix with principal component analysis as extraction method will be used following [54]’s approach.
However, validity somewhat relies on the researcher’s interpretation, so reliability is required to assure validity [55,56]. In this case, reliability and validity refer to the questionnaire written in the original language (Croatian), and not the translated English version, both of which are presented in Table A1. If one wanted to use the questionnaire in another language, its reliability and validity examination should be repeated [30].
3. Results
3.1. Reliability
Reliability was examined per coding type, study, and scenario, and for different combinations of items. It can be noticed, in Table 3, that the highest reliability measures appear for the 2021 study and LR coding, pointing out a high (though not excellent) reliability. It can be noticed that only LR and LR CP data from 2021 meet the threshold of 0.8 [45].
At this point, it can be concluded that the last version of the questionnaire, when coded by LR, measures game awareness best. The lowest reliability statistics occurred for the 2019 study. This was expected, due to revealed necessities for changes during the questionnaire’s formation and “don’t know” responses recoded into 0. The lower reliability related to SR CP coding indicates that such coding could be a somewhat worse measure of game awareness between scenarios, while still reaching an acceptable level of reliability for the 2021 study [34].
However, these are statistics for the entire questionnaire, while variations among scenarios might differ. The best results are shown in Table 4, while results for the complete analysis can be found in Table A5 and Table A6.
The results point out that there is high reliability, meaning that there is consistency between the items. While still indicating acceptable reliability according to [34], the lowest measures occurred for scenarios 1, 5, and 9. For scenario 9, the reliability statistics improve (0.803) if examined for six items (removing I6), but the same approach shows no improvement for scenarios 1 and 5, indicating their potential exclusion from the instrument.
As for the proxy for stability, Table A4 and Table A7 show comparable results over the three groups of respondents in terms of frequencies and averages. Only minor differences can be noticed for most frequencies presented in Table A4, with the distinction of frequencies in scenarios 1, 5, and 9, where more variability in frequencies between studies exists. While that variability may be the reason for the somewhat lower reliability statistics for these scenarios, the average certainties do not follow the same pattern. Moreover, an insight into Table A7 reveals similar averages, and the differences between the averages are much smaller than their respective standard deviations. Observing all the indicators combined, it can be concluded that the last version of the questionnaire sufficiently satisfies the reliability requirement.
3.2. Validity
The discriminant construct validity is examined with k-nn analysis. Given the reliability insights, only the results for the 2021 study will be presented.
The k-nn analysis is conducted by setting I7 as a target variable, partitioning the sample to the training (60%) and testing (40%) parts, with medians of the nearest neighbor values used for the prediction of the scale target and Euclidian metric for the distances between the neighbors. By setting the I7 as a target, and other items as predictors, this analysis can also serve as a criterion validity estimation.
The sums of squares errors vary between scenarios, but can still be considered low (Figure 1). The k selection error logs for the 2021 LR, LR CP, and SR CP coded data are presented in Figure A1, Figure A2 and Figure A3, respectively. The k selection error logs for scenarios S1–S9 with LR CP and SR CP coded data are shown in Figure A4 and Figure A5, respectively. The sums of squares errors are higher for entire datasets, but are still acceptably low considering the number of observations.
The number of nearest neighbors ranges from 2 (S6) to 10 (S4). It can be interpreted that in scenarios with a lower number of nearest neighbors, more individual differences between the respondents come to the fore. While lower k values may lead to model over-fitting, the selection criteria used in this case is a minimal error, that is, the minimum of the sum of squares error.
The k-nn analysis also allows for the examination of predictor importance. It can be noticed (Table 5) that predictor importance is almost uniformly distributed for each scenario. That means that all items I1–I6 and I8–I9 are relevant predictors for the target I7. The results for the scenarios in the 2021 LR CP and SR CP coded data are presented in Table A8 and Table A9, respectively. While some minor differences may be observed when comparing Table 5, Table A8 and Table A9, the general conclusion about items’ importance is sustained.
While the additional analysis was conducted after removing the least important predictors, it led to higher sum of squares errors. It can be concluded that none of the predictor variables (I1–I9) are discriminated from further analysis. Furthermore, given the small sum of squares errors, it can be deduced that the scenarios S1–S9 satisfy the criterion validity.
Further analysis requires an examination of convergent validity. For the entire 2021 LR coded dataset, Kaiser–Meyer–Olkin’s measure of sampling adequacy equals 0.887, and Bartlett’s test of sphericity is significant (). Similarly, for the 2021 LR CP coded data, the KMO measure is 0.879, and Bartlett’s test of sphericity indicates that the data is suitable for further analysis (). For 2021 SR CP coded data, the KMO measure is 0.866, and Bartlett’s test of sphericity is significant (). There is only one component with an eigenvalue greater than or equal to one for the data from the 2021 study coded each way. The resulting component matrices are presented in Table 6. The singular components for data coded each way have almost the same order of loadings (for LR CP, the ranks of the loadings for I2 and I5 switch places, and for SR CP the ranking is: I1, I3, I5, I7, I2, I4, I6, I8, I9).
The same approach is taken for each of the scenarios in LR, LR CP, and SR CP coded data from the 2021 study, and the results are presented in Table 7, Table A10 and Table A11, respectively.
The evidence presented in Table 7 shows further support for a decision about S1 exclusion. First, the KMO measure of sampling adequacy is relatively lower in comparison to other scenarios, and this is the only scenario that contains multiple components and cross-loadings, even after an item’s omission (as presented in Table A10 and Table A11), meaning that it measures other concepts in addition to game awareness, and it does not measure game awareness as it was intended. Moreover, I7 appears in the second and third components, showing that the scenario does not allow conclusions about game awareness in relation to game elements.
All other scenarios, except for S7, revealed two components when all items were included. Considering that the goal is to determine whether the scenarios measure game awareness, this exploratory analysis also examines which items do not measure game awareness, aiming for the singular component where the loadings construct game awareness. In most cases, I9 appeared to load the second component, and its exclusion led to unidimensionality. Moreover, I9 refers to the awareness about one’s own possible actions, which requires more intense logical deliberation about possibilities and even in-game reasoning. Hence, it makes sense that it appears in a separate component. The removal of I9 resulted in one component for scenarios 2, 3, and 4. Another problematic item, I8 (deliberation awareness), led to the second component, and its removal prompted a one-component output for scenarios S5 and S6. In comparison, when omitted, these two variables improved reliability statistics (Table 4), too. However, they seem to contribute to the understanding of game awareness in scenarios S7 and S8.
Unusual and somewhat unexpected was the role of I6 (own outcome assessment) in scenarios S8 and S9. However, those are the only two scenarios that offered complete information that enables not guessing but the reading-out of possible outcomes. It is possible that the complete information and the lack of uncertainty led to such variable values and inter-variable correlations that made I6 appear in the second component. Given that interpretation, it may be that the questionnaire better measures game awareness under uncertainty and incomplete information or in a pre-game context, that is, in situations similar to the ones that occur in the real world.
4. Discussion and Conclusions
The goal of this paper was to offer a methodological contribution to the empirical examination of individuals’ game awareness. The proposed questionnaire has been examined regarding its internal reliability, criterion validity, and construct validity. The questionnaire comprises different scenarios that vary regarding provided information, activity type, outcomes, and diverse perspectives. Those variations should enable the capture of how respondents form a game awareness (or a lack thereof), given different amounts and qualities of the provided information. Such an approach allows for empirical insights into the individual’s game awareness formation, which may be used in the future to enrich theoretical comprehension. In addition, while the questionnaire may be used as a whole, there are indications that scenarios may also be used separately to examine the desired aspect.
4.1. Reliability and Validity
The examined reliability per coding type, study, scenarios, and for different combinations of items revealed that the 2021 LR coded data offer the highest reliability measured with Cronbach’s alpha. While this method is not designed to assess the coding approach, infringed reliability sheds light on the consequences for data and the possibility of its use. Undoubtedly, modifications in questionnaire versions contribute most to the higher reliability revealed in the 2021 study data. Given the reliability findings, the validity examination was conducted only for the 2021 study. A k-nn analysis was used to estimate divergent construct validity and criterion validity (by setting I7 as a target), and nonparametric principal component analysis based on the Spearman correlation coefficients matrix was used to estimate convergent construct validity. Reliability and validity analyses point out that the last version of the questionnaire is both reliable and valid. However, the entire questionnaire is lengthy and may be boresome [24] for respondents, and this was the motivation to examine reliability and validity for each scenario. The more profound insights into the scenarios revealed that S1 should be excluded from the questionnaire. The other scenarios show indications of internal reliability, as well as criterion and construct validity requirements. Questions Q19 and Q20 can be omitted from scenarios 2, 3, 4, 5 and 9, or interpreted with caution. Questions Q17 and Q18 can be omitted or interpreted at one’s discretion in scenarios 5, 6, and 9. The same approach is viable for Q13 and Q14 in scenarios 8 and 9 (and 5, for CP coding). While there are differences in results given the different approaches to the coding, overall results’ similarities indicate the questionnaire’s relative robustness. Scenarios S2–S9 reach over the reliability and validity threshold, which may serve as an indication for their separate use.
Moreover, the principal component analysis results in one component for scenarios S2–S9. One component indicates that the included questions measure the same concept and that there are no other underlying latent variables. As a distinction has been made between awareness and assessment, and since separate questions examined them, these results indicate no fundamentally different underlying process that distinguishes the two. This also confirms the approaches that include elements of the assessment in the awareness’ definitions and approaches [5,6,12,14,16], that is, the awareness in a narrow sense seems to be a precondition for the assessment, while assessment is required for the player to obtain an amount of game awareness, but they co-occur and act as building blocks for shaping one’s awareness of a situation.
4.2. Limitations
In this paper, the effort was focused on shedding light on various aspects of the questionnaire, from its topic to its applicability. Nevertheless, there are still questions that remain unanswered. The questionnaire is examined from the perspective of internal consistency reliability, leaving equivalence and stability unchecked. The equivalence was not examined, as no other questionnaire of game awareness is known to the author, so none was applied. Perhaps this questionnaire can serve as a motivation for the further development of research instruments for the behavioral perspective on game awareness, which would enable equivalence assessments in the future.
While descriptive statistics from the three studies were used as a proxy for stability, that cannot be understood as evidence of stability. It is only an indication and is derived based on similarities of corresponding data (not all questions and not all scenarios are present in each study). Furthermore, content validity is suggested only by theoretical basis for questionnaire and scenarios development, as well as by subjective assessment of the researcher (face validity), while no scrupulous appraisal by different experts has been done.
Although a decent amount of data has been gathered overall and validity and reliability have been confirmed at the questionnaire level, there were only 498 observations for the 2021 study spanning nine scenarios, so additional reliability and validity confirmations for each scenario are advised. Moreover, one must keep in mind that reliability and validity refer to the questionnaire written in the original language (Croatian) and not the translated English version. In addition to the stated limitations, the limitations regarding the scenarios’ contents and further enrichment by introducing different scenarios remain for future research.
Considering the coding approaches, the LR approach may seem somewhat subjective, which might influence the results’ replicability. However, a comparison with the results obtained by the SR approach demonstrates a high similarity and indicates the questionnaire’s robustness regarding the coding approaches. Furthermore, the comparison of the convergent validity results suggests that both approaches capture the essence of the phenomenon of interest.
In addition, one might favor the standard game approach and might want to approach the data by measuring deviation. The last two scenarios would be the best choice for proceeding in such an endeavor. This would also require a recoding, such that ‘correct’ responses are denoted from the information offered in scenarios as a baseline (for example, coded 0, which would stand for no deviation) while denoting the other responses as deviations. While that was not the intended purpose of the questionnaire, that option remains a possibility for future instrument use and as a line of development.
Lastly, the provided incentive may be considered weak or partial by some standards, which may influence motivation for providing responses and mistakes in responses. However, the answers to the open-ended questions were considered a proxy for the engagement, and uniform or unusual responses were removed from the further analysis. That is one of the reasons for the lower responses’ retention percentages. Nevertheless, the examination can be repeated in the future with a stronger incentive.
4.3. Implication for Further Research
The questionnaire allows for a quantitative and qualitative (open-ended questions) examination of responses. Open-ended questions provide a deeper insight into awareness and assessment that cannot be revealed by yes/no questions [25]. In addition, the responses may be used for future questionnaire development by forming the statements about scenarios, and thus ease the completion for respondents. However, a possible problem with such an approach is that the respondents may become aware of possible interpretations because they are offered in the statements [25], whereas the current questionnaire form does not enable such potential deviation. Nevertheless, that remains as an option that can be further examined.
Another instrument development direction may be converting the probability assessment into a slider bar or a Likert-type scale based on the possible groupings or by centering percentages around specific values, thus making the instrument more attractive to fill in for the respondents who have a hard time dealing with percentages.
A specific area of possible future in-depth instrument development regards the examination of game element awareness separately. For example, a separate questionnaire could examine what, exactly, is relevant for a person to detect and assess an activity, what properties, what amount of information, and of what quality should be present, etc.
The (un)awareness in game theory is relatively new and mainly dealt with using a logical approach or a standard game representation. The most notable contributions to behavioral aspects of awareness (in general, but also in games) stem from the research of business and ethical questions [5,6,18,29]. Although the contributions are worthwhile, potential consequences of unawareness spread over various disciplines where individuals play a role as a decision maker and engage in strategic interactions. This questionnaire is focused on game awareness in different situations of possible strategic interactions, and hopefully, it will encourage other instruments’ development, as well as the collection and publication of behavioral insights into game awareness. One should notice the difference and the connection between the psychological insights and behavioral game theory: the first one serves as an input for broadening knowledge about human thinking and behavior, and the latter utilizes that knowledge in model creation. Models—including behavioral game theory models—inevitably contain a level of abstraction, idealization, and caricaturing, but with the growing body of theoretical and empirical knowledge, accompanied by the increasing variety and power of quantitative approaches, should strive for more realistic modeling, thus becoming more useful for theoretical and practical advancements. In other words, empirical data enable data-driven conclusions, and in this case, an inductive approach to the role of (un)awareness in games, therefore, complements the existing deductive approach. One possible approach is to use the findings from the data as building blocks for modeling game behavior (instead of modeling behavior as a deviation from the implicitly assumed full awareness and rationality), as has been done for other individuals’ traits, for example, by using personality traits and general attitudes in different situations of bargaining [57,58,59]. The underlying idea is that the behavioral insights should endow the theory and prompt behavioral game theory models of game awareness and its implications. This could also lead to the requirement of revising the current approach to the in-game bounded rationality—a part of the deviations currently ascribed to the bounded rationality might actually stem from unawareness.
4.4. Practical Implications
A certain amount of (un)awareness is present in most situations for everyone. In some cases, consequences may refer to an individual’s life; for example, unawareness of an alternative or its attributes may skew the choices of individuals toward less favorable outcomes. In individuals’ strategic interactions, such occurrences may influence personal and business relationships, careers, personal finance management, health strategies, savings strategies, voting strategies, etc. However, the consequences could multiply if unawareness affects an individual in a position of power, whether it is a managerial position in a company or a political position. Becoming aware of unawareness is not the only requirement, but is a critical one for better decision making and better modeling of actual human behavior. This notion tackles the area of reflective awareness [29], where it was suggested that it can be trained and improved. Accordingly, this questionnaire can be used in the classroom or for professional skill training to assess the individuals’ game awareness and use it as a basis for intervention in the case of revealed fallacies.
Funding
No funding was received for this research.
Institutional Review Board Statement
The Research Ethics Assessment Committee at Juraj Dobrila University of Pula gave their positive opinion on the research design and approved the questionnaire. The instruction for obtaining informed consent is stated on the first page of the questionnaire. The confidentiality of information on the identity of participants in the research is guaranteed. The questionnaires will be located on the online platform. Upon completion of data collection, the aggregated data will be stored on the personal computers of the researcher, protected by a password. Only researchers are allowed access to the data. If the findings of this research are made public, the data will be presented exclusively at the sample level, and individual responses will not be published. The risk of a research procedure is minimal, i.e., that in which the degree of possible damage, discomfort, or injury that the procedure will cause is less than or equal to that to which the participants are exposed in performing daily activities.
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The data presented in this study are available on request from the author. The data are not publicly available due to the restriction by the Statement.
Acknowledgments
I want to thank the students who participated in this research for their thoughtful responses. The research is partially related to the project “The role of emotions in strategic decision-making”—an internal project at the Faculty of Economics and Tourism “ Mijo Mirković” Pula.
Conflicts of Interest
The authors declare no conflict of interest.
Appendix A
Table A1.
GAQ: Scenarios, questions, and items.
| Scenarios | Scenarios in Original Language (Croatian) | |
|---|---|---|
| Scenario 1 1,2,3 | You are going to the shop to do your usual grocery shopping. In addition to the items on your usual shopping list, you intend to buy your favorite chocolate. When you arrive to the shelf with the chocolates, you learn that your favorite chocolate’s price increased by HRK 10 (approx. EUR 1.34). | Ulazite u dućan s namjerom da obavite svoju uobičajenu kupovinu. Između ostalog, namjeravate kupiti svoju omiljenu čokoladu. Dolaskom do police s čokoladama, pogledom nalazite svoju omiljenu čokoladu i uočavate da je poskupila za 10 kuna. |
| Scenario 2 1,2,3,* | You are at a usual cafe hanging out with a friend. You notice that your friend is behaving slightly differently and is somewhat restrained. Moreover, you notice an unusual look that can be interpreted as playful or mischievous. You are trying to figure out what that would mean: maybe it is a secret about preparations for your upcoming birthday, maybe the friend knows something you do not know (e.g., some gossip or some footage that has appeared on YouTube), maybe the friend is about to announce something important, maybe he or she is conducting a scheme, maybe… There may be a lot of options and maybe they have nothing to do with you. | Na uobičajenoj ste kavi s prijateljem/prijateljicom. Primjećujete da se ponaša malo drugačije, pomalo suzdržano. Štoviše, tu i tamo primjećujete neobičan pogled koji može asocirati na zaigranost ili podsmjeh. Pokušavate dokučiti što bi to značilo: možda je to tajanstvenost zbog toga što priprema nešto za Vaš skori rođendan, možda zna nešto što Vi ne znate (npr. trač ili neku snimku koja se pojavila na youtube-u), možda se priprema priopćiti nešto važno, možda Vam priprema spačku, možda… Može biti puno mogućnosti, a možda uopće nema veze s Vama. |
| Scenario 3 1,2,3,* | You arrive to work. You expect the usual tense working pace, and in addition to all your existing obligations, you get another one. Lately, you have caught a few glances and greetings from other colleagues, and the reasons for these changes are not quite clear to you. You figure that you may be promoted to a better position. During the break, you meet a colleague, and you mention the amount of work you need to do. With understanding and good intent, your colleague admits to you that one of your colleagues spoke badly about you. Moreover, it was the person who assigns you your tasks. | Dolazite na posao. Očekujete uobičajen naporan tempo, kad pored svih već postojećih obaveza dobivate još jednu novu. Osim toga, u zadnje ste vrijeme uhvatili nekoliko pogleda i pozdrava kolega, koji Vam nisu bili potpuno jasni. Pada Vam na pamet kako Vam se možda sprema unaprjeđenje na bolje radno mjesto. Tijekom pauze, srećete kolegu kojem se požalite na količinu posla. S razumijevanjem i u dobroj namjeri, kolega Vam prizna da netko od kolega govori ružno o Vama. Štoviše, to je upravo osoba koja Vam dodjeljuje obaveze. |
| Scenario 4 1,2,3,* | A month ago, you started a seasonal job. This season is better than the last one for the company’s turnover. There is a rumor amongst your colleagues that all the employees could get a raise. | Prije mjesec dana dobili ste sezonski posao na kojem i sad radite. Ova je sezona bolja od prethodne po prometu poduzeća za koje radite. Među Vašim kolegama se šuška da bi svi mogli dobiti povišicu na plaću. |
| Scenario 5 1,2,3,* | You have received a team assignment to conduct some research and to report the results in a written form. There are four of you in the team. Your part of the assignment is to create a questionnaire. You are unfamiliar with designing questionnaires, so you contact one of the team members to help you. At that time, you learn that one of the team members already assembled the questionnaire. | Dobili ste zadatak u timu provesti manje istraživanje i o tome sastaviti izvještaj u obliku seminarskog rada. U timu vas je četvero. Vaš je dio posla sastaviti anketu. Ne znate ništa o tome i kontaktirate jednog člana tima da Vam u tome pomogne. U tom kontaktu saznajete da je netko već sastavio upitnik. |
| Scenario 6 1,2,3,* | You are a manager of a department for consulting clients in a leading consulting company. The company is well known for its substantial number of clients and for the low prices of its services. You meet a client and, in an informal conversation at the beginning of the meeting, s/he reveals that s/he has been contacted by a newly opened consultant company that offered their services at a lower price. | Voditelj ste odjela za savjetovanje klijenata u vodećem poduzeću za savjetovanje. Poduzeće je poznato po velikom broju klijenata, ali i najjeftinijoj cijeni usluge. Na sastanak s Vama dolazi klijent. U neformalnom razgovoru na početku savjetovanja, od klijenta saznajete da su ga kontaktirali iz novootvorenog poduzeća koje se odlučilo baviti istom djelatnošću, a savjetovanja nude po nižoj cijeni. |
| Scenario 7 1,2,3,* | You have come up with an idea to start your own business. Even though all the people who hear about the idea agree that it is an excellent idea, you do not have the capital to start the company. You consider giving up the idea, given that you cannot finance it. At that time, you receive an invitation to pitch your idea in front of five investors. If you impress the investors, they could decide to invest. | Osmislili ste ideju za pokretanje svog poduzeća. Iako se svi koji čuju slažu s Vama da se radi o izvrsnoj ideji, Vi nemate dovoljno kapitala za pokretanje poduzeća. Namjeravate odustati od ideje, jer je ne možete financirati. Tad primate poziv na sastanak na kojem ćete održati prezentaciju svoje ideje pred pet investitora. Ako ih oduševite svojom idejom, investitori bi mogli odlučiti investirati u Vas. |
| Scenario 8 1,2,3,* | You and your colleague have been accused of cheating in an exam using unallowed electronic devices and are taken in front of the faculty’s ethical committee for interrogation. You and your colleague are placed in separate rooms and cannot know what the other one is saying during interrogation. The ethical committee member who conducts the interrogation makes the following offer to each of you: ”You may choose to confess or remain silent. If you confess and your accomplice remains silent, you will not be punished, and your testimony will be used to ensure that your accomplice gets a year of suspension. Likewise, if your accomplice confesses while you remain silent, s/he will go free while you get a year of the suspension. If you both confess there will be two punishments, but I’ll see to it that you both get only 6 months of suspension. If you both remain silent, I’ll have to settle for a token punishment for having unallowed devices during the exam and you will get 1 month of suspension”. | Vaš kolega/ica i Vi optuženi ste za varanje na ispitu korištenjem nedopuštenih elektronskih uređaja i odvedeni ste pred Etički odbor fakulteta na ispitivanje. Vi i Vaš/a kolega/ica smješteni ste u odvojene prostorije i ne možete znati što druga osoba govori ispitivaču. Član Etičkog odbora koji provodi ispitivanja daje jednaku ponudu Vama i kolegi/ci u drugoj prostoriji: “Možete odabrati priznati ili šutjeti. Ako priznate, a Vaš/a kolega/ica ostane šutjeti, Vi nećete biti kažnjeni, a Vaša će izjava poslužiti kao dokaz i Vaš/a kolega/ica dobiva godinu dana suspenzije na studiju. Također, ako kolega/ica prizna, a Vi šutite, on/a neće biti kažnjen/a i Vi dobivate godinu dana suspenzije na studiju. Ako oboje/obje/obojica priznate, svaki dobiva po 6 mjeseci suspenzije. Ako oboje/obje/obojica odaberete šutjeti, bit ćete kažnjeni samo za posjedovanje nedopuštenih elektroničkih uređaja tijekom ispita i to s jednim mjesecom suspenzije”. |
| Scenario 9 1,2,3,* | You and your colleague participated in a presentation for your faculty. At lunch, you were given the last sandwich, so your colleague is left with none. Your colleague promises to take you for a coffee later, hoping that you will give him a piece of your sandwich now. You can decide to trust your colleague and give him a piece of the sandwich or keep it all for yourself. | Sudjelovali ste u prezentaciji na fakultetu s kolegom. Na marendi ste dobili posljednji sendvič, tako da je kolega ostao bez. Kolega obećava da će Vas kasnije odvesti na kavu, u nadi da ćete mu sad dati dio sendviča. Možete odabrati vjerovati kolegi i dati mu dio sendviča ili cijelog zadržati za sebe. |
| Questions per scenario | ||
| Q1 1,2,3,* | Is it possible to notice any activity that influences your outcomes (economic or otherwise)? 1,2,3,* Yes 1,2,3,* Don’t know 1 No 1,2,3,* | Je li moguće uočiti ikakvu aktivnost drugih osoba koje utječu na Vaše ishode (ekonomske, ili kakve druge)? |
| Q2 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q3 1,2,3,* | Can you identify (describe) that activity? | Možete li identificirati tu aktivnost? Opišite je: |
| Q4 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q5 1,2,3,* | Is there a person (or a group/association/company, etc.) who initiated the activity? Yes 1,2,3,* Don’t know 1 No 1,2,3,* | Postoji li osoba (ili grupa/udruženje/poduzeće) koja je poduzela tu aktivnost? |
| Q6 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q7 1,2,3,* | Can you identify a person (or group/association/company, etc.) that initiated the noticed activity? Describe. | Možete li identificirati tu osobu (ili grupu/udruženje/poduzeće)—opišite: |
| Q8 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q9 1,2,3,* | Can you determine desirable outcomes for that person (or group/association/company, etc.)? Yes 1,2,3,* Don’t know 1 No 1,2,3,* | Možete li utvrditi koji su poželjni ishodi za tu osobu (ili grupu/udruženje/poduzeće) s obzirom na poduzetu aktivnost? |
| Q10 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q11 1,2,3,* | The outcomes for yourself regarding the initiated activity are Positive Don’t know 1 Negative | Koji su ishodi za Vas s obzirom na poduzetu aktivnost te osobe (ili grupe/udruženja/poduzeća)? |
| Q12 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q13 1,2,3,* | Describe your own outcomes given the initiated activity by a person (or group/association/company, etc.). | Opišite koji su ishodi za Vas s obzirom na poduzetu aktivnost te osobe (ili grupe/udruženja/poduzeća): |
| Q14 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q15 1,2,3,* | Do you believe that something is going on that affects you and that there exists a person (or group/association/company, etc.) whose activities influence your outcomes? Yes 1,2,3,* Don’t know 1 No 1,2,3,* | Vjerujete li da se događa nešto što utječe na Vas i da pritom postoji neka osoba (ili grupa/udruženje/poduzeće) koja svojim aktivnostima utječe na Vas, te da s obzirom na to za Vas nastaju drukčiji ishodi? |
| Q16 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q17 1,2,3* | Do you believe that activity was deliberate or accidental? Deliberate 1,2,3 Don’t know 1 Accidental 1,2,3 | Vjerujete li da je to djelovanje namjerno ili slučajno? |
| Q18 1,2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Q19 2,3,* | Is there an action that you can take to influence outcomes? Yes 2,3 No 2,3 | Možete li Vi donijeti odluku ili poduzeti aktivnost na način da utječete na ishode? |
| Q20 2,3,* | How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.) | Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%): |
| Items | ||
| Without complementary probability | With complementary probability (CP) | |
| I1 1,2,3,* | Activity awareness = Q1 × Q2/100 | if Q1 = 1, then Q1 × Q2, else (100−Q2)/100 |
| I2 1,2,3,* | Activity assessment = Q3 × Q4/100 | if Q3 = 1, then Q3 × Q4, else (100−Q4)/100 |
| I3 1,2,3,* | Player awareness = Q5 × Q6/100 | if Q5 = 1, then Q5 × Q6, else (100−Q6)/100 |
| I4 1,2,3,* | Player assessment = Q7 × Q8/100 | if Q7 = 1, then Q7 × Q8, else (100−Q8)/100 |
| I5 1,2,3,* | Other player’s outcome awareness = Q9 × Q10/100 | if Q9 = 1, then Q9 × Q10, else (100−Q10)/100 |
| I6 1,2,3,* | Own outcome assessment = Q13 × Q14/100 | if Q13 = 1, then Q13 × Q14, else (100−Q14)/100 |
| I7 1,2,3,* | Game existence awareness = Q15 × Q16/100 | if Q15 = 1, then Q15 × Q16, else (100−Q16)/100 |
| I8 1,2,3 | Deliberation = Q17 × Q18/100 | if Q17 = 1, then Q17 × Q18, else (100−Q18)/100 |
| I9 2,3 | Own activity awareness = Q19 × Q20/100 | if Q19 = 1, then Q19 × Q20, else (100−Q20)/100 |
Notes: The term certainty is used in psychological but not epistemic interpretation. A level of certainty in the provided answer is the respondent’s belief that his or her assessment is true/correct. Questions and items denoted with 1, 2, and 3 were used in the first (2019), second (2020), and third (2021) study, respectively. The findings from the second study were published previously, based on a part of the data (respondents without prior game theory knowledge) [10]. In the first two studies, instead of “How certain are you in the previous answer,” a different formulation (“Please provide your determined certainty in the answer”) was implored. In the third study, along with each scenario, respondents were instructed to “Imagine you are a part of the following scenario” and “Take a couple of minutes to engage with the scenario. You may want to close your eyes and imagine the situation. Have you done that? Given the scenario and your part in it, please, answer the following set of questions”. Questions and items denoted with * have been retained after the reliability and validity analyses.
Table A2.
Sample characteristics for loosely coded data with complementary probability.
| Study | Age (Mean, Standard Deviation, Min, Max) | Gender (Frequency Male, Female) | Year of Study (Mean, Standard Deviation, Min, Max) | Previously Learned Game Theory (Frequency) | Used Knowledge of Game Theory (Frequency) | Number of Responses |
|---|---|---|---|---|---|---|
| 2019 | 22.03 4.788 18 46 | 48.54% 51.45% | 1.934 1.1556 1 5 | 54% | 21.8% | 822 |
| 2020 | 19.931 3.687 18 52 | 26.9% 73.1% | 1.049 0.2154 1 2 | 43.8% | 18.8% | 739 |
| 2021 | 19.83 2.31 18 34 * | 33.5% 66.5% * | 1.06 0.2382 1 2 * | 45.74% * | 13.37% | 448 |
| Total | 20.768 4.0822 18 52 | 37.23% 62.77% | 1.414 0.8738 1 5 | 48.41% | 18.82% | 2009 |
* A total of 47 responses for socio-demographic data are missing in the 2021 study; the total number of responses in the 2021 study is 495; the total number of responses for all studies is 2056.
Table A3.
Sample characteristics for strictly coded data with complementary probability.
| Study | Age (Mean, Standard Deviation, Min, Max) | Gender (Frequency Male, Female) | Year of Study (Mean, Standard Deviation, Min, Max) | Previously Learned Game Theory (Frequency) | Used Knowledge of Game Theory (Frequency) | Number of Responses |
|---|---|---|---|---|---|---|
| 2019 | 22.03 4.788 18 46 | 48.54% 51.45% | 1.934 1.1556 1 5 | 54% | 21.8% | 822 |
| 2020 | 19.87 3.469 18 52 | 27.1% 72.9% | 1.048 0.2139 1 2 | 43.6% | 18.7% | 750 |
| 2021 | 19.855 2.315 18 34 * | 33.92% 66.08% * | 1.06 0.2367 1 2 * | 46.04% * | 13.88% | 454 |
| Total | 20.711 3.9348 18 52 | 37.33% 62.67% | 1.38 0.8763 1 5 | 48.37% | 18.88% | 2026 |
* A total of 47 responses for socio-demographic data are missing in the 2021 study; the total number of responses in 2021 study is 501; the total number of responses for all studies is 2073.
Table A4.
Frequencies for binary-coded answers.
| Study | 2019 | 2020 | 2021 | ||||
|---|---|---|---|---|---|---|---|
| Scenario | Questions | % (q = 1) LR | % (q = 1) SR | % (q = 1) LR | % (q = 1) SR | % (q = 1) LR | % (q = 1) SR |
| S1 | Q1 | 83.62 | / | 87.18 | / | 77.08 | / |
| Q3 | 73.28 | 75.00 | 83.33 | 56.96 | 56.25 | 32.65 | |
| Q5 | 76.72 | / | 87.18 | / | 77.08 | / | |
| Q7 | 57.76 | 59.48 | 79.49 | 58.23 | 62.50 | 55.10 | |
| Q9 | 73.28 | / | 79.49 | / | 70.83 | / | |
| Q13 | 76.72 | 72.41 | 92.31 | 87.34 | 85.42 | 61.22 | |
| Q15 | 96.55 | / | 85.90 | / | 89.58 | / | |
| Q17 | 74.14 | / | 67.95 | / | 60.42 | / | |
| Q19 | / | / | 79.49 | / | 77.08 | / | |
| n | 116 | 114 | 78 | 79 | 48 | 49 | |
| S2 | Q1 | 75.86 | / | 91.76 | / | 68.97 | / |
| Q3 | 49.14 | 47.41 | 81.18 | 79.01 | 55.17 | 52.63 | |
| Q5 | 70.69 | / | 78.82 | / | 67.24 | / | |
| Q7 | 47.41 | 51.72 | 65.88 | 70.37 | 56.90 | 54.39 | |
| Q9 | 53.45 | / | 43.53 | / | 53.45 | / | |
| Q13 | 49.14 | 45.69 | 84.71 | 76.54 | 65.52 | 40.35 | |
| Q15 | 95.69 | / | 87.06 | / | 86.21 | / | |
| Q17 | 47.41 | / | 61.18 | / | 55.17 | / | |
| Q19 | / | / | 85.88 | / | 91.38 | / | |
| n | 116 | 113 | 85 | 81 | 58 | 57 | |
| S3 | Q1 | 86.44 | / | 90.59 | / | 80.70 | / |
| Q3 | 69.49 | 65.25 | 82.35 | 76.47 | 64.91 | 61.40 | |
| Q5 | 75.42 | / | 92.94 | / | 78.95 | / | |
| Q7 | 59.32 | 62.71 | 85.88 | 77.65 | 68.42 | 66.67 | |
| Q9 | 61.86 | / | 64.71 | / | 75.44 | / | |
| Q13 | 70.34 | 67.80 | 90.59 | 76.47 | 78.95 | 59.65 | |
| Q15 | 96.61 | / | 90.59 | / | 82.46 | / | |
| Q17 | 74.58 | / | 91.76 | / | 78.95 | / | |
| Q19 | / | / | 83.53 | / | 87.72 | / | |
| n | 118 | 118 | 85 | 85 | 57 | 57 | |
| S4 | Q1 | 84.75 | / | 81.18 | / | 85.00 | / |
| Q3 | 79.66 | 73.73 | 69.41 | 71.43 | 73.33 | 73.77 | |
| Q5 | 73.73 | / | 82.35 | / | 85.00 | / | |
| Q7 | 65.25 | 64.41 | 75.29 | 73.81 | 71.67 | 72.13 | |
| Q9 | 68.64 | / | 78.82 | / | 88.33 | / | |
| Q13 | 76.27 | 0.00 | 96.47 | 95.24 | 83.33 | 72.13 | |
| Q15 | 96.61 | / | 81.18 | / | 91.67 | / | |
| Q17 | 59.32 | / | 62.35 | / | 70.00 | / | |
| Q19 | / | / | 58.82 | / | 71.67 | / | |
| n | 118 | 118 | 85 | 84 | 60 | 61 | |
| S5 | Q1 | 74.79 | / | 92.86 | / | 77.78 | / |
| Q3 | 73.11 | 65.55 | 79.76 | 82.35 | 74.07 | 74.55 | |
| Q5 | 73.95 | / | 89.29 | / | 87.04 | / | |
| Q7 | 60.50 | 60.50 | 77.38 | 77.65 | 79.63 | 80.00 | |
| Q9 | 63.03 | / | 60.71 | / | 75.93 | / | |
| Q13 | 68.07 | 65.55 | 96.43 | 85.88 | 85.19 | 72.73 | |
| Q15 | 96.64 | / | 84.52 | / | 90.74 | / | |
| Q17 | 57.14 | / | 63.10 | / | 66.67 | / | |
| Q19 | / | / | 90.48 | / | 87.04 | / | |
| n | 119 | 119 | 84 | 85 | 54 | 55 | |
| S6 | Q1 | 79.13 | / | 88.24 | / | 90.91 | / |
| Q3 | 72.17 | 63.48 | 85.88 | 79.76 | 83.64 | 80.36 | |
| Q5 | 75.65 | / | 88.24 | / | 87.27 | / | |
| Q7 | 62.61 | 66.09 | 76.47 | 73.81 | 74.55 | 76.79 | |
| Q9 | 68.70 | / | 82.35 | / | 89.09 | / | |
| Q13 | 75.65 | 66.09 | 96.47 | 90.48 | 90.91 | 82.14 | |
| Q15 | 97.39 | / | 84.71 | / | 92.73 | / | |
| Q17 | 64.35 | / | 80.00 | / | 63.64 | / | |
| Q19 | / | / | 90.59 | / | 96.36 | / | |
| n | 115 | 115 | 85 | 84 | 55 | 56 | |
| S7 | Q1 | 83.33 | / | 96.47 | / | 91.07 | / |
| Q3 | 73.33 | 30.83 | 82.35 | 78.82 | 75.00 | 75.00 | |
| Q5 | 72.50 | / | 90.59 | / | 80.36 | / | |
| Q7 | 51.67 | 50.00 | 65.88 | 61.76 | 78.57 | 75.00 | |
| Q9 | 65.83 | / | 92.94 | / | 85.71 | / | |
| Q13 | 71.67 | 59.17 | 95.29 | 95.29 | 87.50 | 83.93 | |
| Q15 | 97.50 | / | 84.71 | / | 92.86 | / | |
| Q17 | 68.33 | / | 77.65 | / | 73.21 | / | |
| Q19 | / | / | 81.18 | / | 92.86 | / | |
| n | 120 | 120 | 85 | 85 | 56 | 56 | |
| S8 | Q1 | / | / | 82.14 | / | 83.64 | / |
| Q3 | / | / | 84.52 | 63.86 | 74.55 | 74.55 | |
| Q5 | / | / | 82.14 | / | 85.45 | / | |
| Q7 | / | / | 78.57 | 36.14 | 63.64 | 25.45 | |
| Q9 | / | / | 73.81 | / | 87.27 | / | |
| Q13 | / | / | 96.43 | 93.98 | 90.91 | 78.18 | |
| Q15 | / | / | 89.29 | / | 87.27 | / | |
| Q17 | / | / | 67.86 | / | 65.45 | / | |
| Q19 | / | / | 92.86 | / | 92.73 | / | |
| n | 0 | 0 | 84 | 83 | 55 | 55 | |
| S9 | Q1 | / | / | 89.41 | / | 83.64 | / |
| Q3 | / | / | 82.35 | 80.95 | 78.18 | 76.36 | |
| Q5 | / | / | 85.88 | / | 78.18 | / | |
| Q7 | / | / | 78.82 | 77.38 | 74.55 | 70.91 | |
| Q9 | / | / | 90.59 | / | 83.64 | / | |
| Q13 | / | / | 90.59 | 88.10 | 83.64 | 78.18 | |
| Q15 | / | / | 78.82 | / | 81.82 | / | |
| Q17 | / | / | 70.59 | / | 49.09 | / | |
| Q19 | / | / | 97.65 | / | 90.91 | / | |
| 0 | 0 | 85 | 84 | 55 | 55 | ||
Table A5.
Reliability statistics for items I1–I9 per scenario, study, and type of coding.
| Coding | LR | LR CP | SR CP | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Reliability Statistics | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items |
| 2020 study S1 | 0.777 | 0.788 | 9 | 0.765 | 0.780 | 9 | 0.668 | 0.683 | 9 |
| 2020 study S2 | 0.766 | 0.775 | 9 | 0.742 | 0.753 | 9 | 0.677 | 0.695 | 9 |
| 2020 study S3 | 0.696 | 0.708 | 9 | 0.691 | 0.695 | 9 | 0.599 | 0.618 | 9 |
| 2020 study S4 | 0.713 | 0.730 | 9 | 0.663 | 0.683 | 9 | 0.621 | 0.644 | 9 |
| 2020 study S5 | 0.595 | 0.640 | 9 | 0.551 | 0.589 | 9 | 0.573 | 0.604 | 9 |
| 2020 study S6 | 0.793 | 0.802 | 9 | 0.788 | 0.795 | 9 | 0.785 | 0.792 | 9 |
| 2020 study S7 | 0.707 | 0.726 | 9 | 0.704 | 0.720 | 9 | 0.714 | 0.738 | 9 |
| 2020 study S8 | 0.758 | 0.754 | 9 | 0.747 | 0.743 | 9 | 0.544 | 0.551 | 9 |
| 2020 study S9 | 0.814 | 0.820 | 9 | 0.773 | 0.785 | 9 | 0.785 | 0.796 | 9 |
| 2021 study S1 | 0.750 | 0.743 | 9 | 0.712 | 0.706 | 9 | 0.680 | 0.667 | 9 |
| 2021 study S2 | 0.861 | 0.853 | 9 | 0.852 | 0.841 | 9 | 0.834 | 0.822 | 9 |
| 2021 study S3 | 0.852 | 0.847 | 9 | 0.848 | 0.845 | 9 | 0.803 | 0.803 | 9 |
| 2021 study S4 | 0.783 | 0.797 | 9 | 0.777 | 0.784 | 9 | 0.773 | 0.781 | 9 |
| 2021 study S5 | 0.727 | 0.735 | 9 | 0.718 | 0.726 | 9 | 0.706 | 0.720 | 9 |
| 2021 study S6 | 0.846 | 0.862 | 9 | 0.799 | 0.815 | 9 | 0.792 | 0.808 | 9 |
| 2021 study S7 | 0.842 | 0.845 | 9 | 0.817 | 0.825 | 9 | 0.816 | 0.825 | 9 |
| 2021 study S8 | 0.824 | 0.822 | 9 | 0.770 | 0.763 | 9 | 0.711 | 0.715 | 9 |
| 2021 study S9 | 0.778 | 0.779 | 9 | 0.778 | 0.779 | 9 | 0.779 | 0.781 | 9 |
Table A6.
Reliability statistics for items I1–I7 per scenario, study, and type of coding.
| LR | LR CP | SR CP | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | |
| 2019 study S1 | 0.793 | 0.793 | 7 | 0.778 | 0.779 | 7 | 0.786 | 0.788 | 7 |
| 2019 study S2 | 0.660 | 0.679 | 7 | 0.664 | 0.675 | 7 | 0.660 | 0.659 | 7 |
| 2019 study S3 | 0.756 | 0.781 | 7 | 0.779 | 0.778 | 7 | 0.753 | 0.757 | 7 |
| 2019 study S4 | 0.350 | 0.760 | 7 | 0.801 | 0.801 | 7 | 0.807 | 0.811 | 7 |
| 2019 study S5 | 0.784 | 0.789 | 7 | 0.747 | 0.749 | 7 | 0.714 | 0.727 | 7 |
| 2019 study S6 | 0.825 | 0.836 | 7 | 0.795 | 0.795 | 7 | 0.804 | 0.808 | 7 |
| 2019 study S7 | 0.823 | 0.826 | 7 | 0.803 | 0.799 | 7 | 0.709 | 0.725 | 7 |
| 2020 study S1 | 0.827 | 0.827 | 7 | 0.827 | 0.827 | 7 | 0.699 | 0.714 | 7 |
| 2020 study S2 | 0.793 | 0.800 | 7 | 0.793 | 0.800 | 7 | 0.709 | 0.727 | 7 |
| 2020 study S3 | 0.674 | 0.680 | 7 | 0.674 | 0.680 | 7 | 0.617 | 0.632 | 7 |
| 2020 study S4 | 0.750 | 0.754 | 7 | 0.750 | 0.754 | 7 | 0.686 | 0.690 | 7 |
| 2020 study S5 | 0.612 | 0.660 | 7 | 0.612 | 0.660 | 7 | 0.580 | 0.611 | 7 |
| 2020 study S6 | 0.792 | 0.804 | 7 | 0.792 | 0.804 | 7 | 0.774 | 0.783 | 7 |
| 2020 study S7 | 0.755 | 0.761 | 7 | 0.755 | 0.761 | 7 | 0.754 | 0.769 | 7 |
| 2020 study S8 | 0.784 | 0.782 | 7 | 0.784 | 0.782 | 7 | 0.578 | 0.581 | 7 |
| 2020 study S9 | 0.798 | 0.802 | 7 | 0.798 | 0.802 | 7 | 0.795 | 0.802 | 7 |
| 2021 study S1 | 0.780 | 0.767 | 7 | 0.747 | 0.744 | 7 | 0.708 | 0.704 | 7 |
| 2021 study S2 | 0.870 | 0.870 | 7 | 0.868 | 0.866 | 7 | 0.849 | 0.849 | 7 |
| 2021 study S3 | 0.875 | 0.875 | 7 | 0.864 | 0.865 | 7 | 0.809 | 0.816 | 7 |
| 2021 study S4 | 0.811 | 0.815 | 7 | 0.790 | 0.791 | 7 | 0.786 | 0.789 | 7 |
| 2021 study S5 | 0.782 | 0.779 | 7 | 0.793 | 0.796 | 7 | 0.766 | 0.781 | 7 |
| 2021 study S6 | 0.884 | 0.890 | 7 | 0.842 | 0.848 | 7 | 0.826 | 0.836 | 7 |
| 2021 study S7 | 0.832 | 0.833 | 7 | 0.782 | 0.785 | 7 | 0.776 | 0.782 | 7 |
| 2021 study S8 | 0.833 | 0.833 | 7 | 0.772 | 0.775 | 7 | 0.688 | 0.707 | 7 |
| 2021 study S9 | 0.794 | 0.794 | 7 | 0.805 | 0.805 | 7 | 0.798 | 0.799 | 7 |
Table A7.
Descriptive statistics for certainties per scenario and study.
| Descriptive Statistics 2019 Study | Descriptive Statistics 2020 Study | Descriptive Statistics 2021 Study | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| S1 n = 116 | Min | Max | s | S1 n = 78 | Min | Max | s | S1 n = 48 | Min | Max | s | ||||||
| Q2 | 20.00 | 100.00 | 85.457 | 1.714 | 18.458 | Q2 | 50.00 | 100.000 | 81.218 | 1.879 | 16.597 | Q2 | 0.000 | 100.000 | 67.542 | 4.091 | 28.344 |
| Q4 | 0.00 | 100.00 | 89.647 | 1.873 | 20.171 | Q4 | 0.00 | 100.000 | 78.949 | 2.553 | 22.545 | Q4 | 0.000 | 100.000 | 80.792 | 3.783 | 26.212 |
| Q6 | 0.00 | 100.00 | 89.224 | 1.886 | 20.309 | Q6 | 0.00 | 100.000 | 84.103 | 2.551 | 22.528 | Q6 | 50.000 | 100.000 | 88.729 | 2.437 | 16.887 |
| Q8 | 0.00 | 100.00 | 86.716 | 1.902 | 20.488 | Q8 | 0.00 | 100.000 | 81.090 | 2.208 | 19.497 | Q8 | 45.000 | 100.000 | 89.229 | 2.474 | 17.141 |
| Q10 | 0.00 | 100.00 | 83.483 | 2.161 | 23.280 | Q10 | 30.00 | 100.000 | 81.231 | 2.180 | 19.249 | Q10 | 20.000 | 100.000 | 82.292 | 3.169 | 21.954 |
| Q14 | 0.00 | 100.00 | 88.284 | 2.127 | 22.913 | Q14 | 30.00 | 100.000 | 86.500 | 1.970 | 17.395 | Q14 | 50.000 | 100.000 | 90.229 | 2.150 | 14.896 |
| Q16 | 2.00 | 100.00 | 85.664 | 1.967 | 21.191 | Q16 | 45.00 | 100.000 | 87.718 | 1.685 | 14.879 | Q16 | 30.000 | 100.000 | 88.375 | 2.650 | 18.359 |
| Q18 | 8.00 | 100.00 | 85.534 | 2.121 | 22.841 | Q18 | 40.00 | 100.000 | 82.462 | 2.013 | 17.780 | Q18 | 50.000 | 100.000 | 85.125 | 2.844 | 19.704 |
| Q20 | Q20 | 50.00 | 100.000 | 86.256 | 1.855 | 16.385 | Q20 | 20.000 | 100.000 | 87.646 | 2.825 | 19.569 | |||||
| S2 n = 116 | Min | Max | s | S2 n = 85 | Min | Max | s | S2 n = 58 | Min | Max | s | ||||||
| Q2 | 3.00 | 100.00 | 82.862 | 2.045 | 22.030 | Q2 | 20.00 | 100.000 | 85.612 | 1.831 | 16.885 | Q2 | 0.000 | 100.000 | 82.845 | 3.100 | 23.606 |
| Q4 | 0.00 | 100.00 | 83.621 | 2.200 | 23.695 | Q4 | 0.00 | 100.000 | 81.529 | 2.445 | 22.546 | Q4 | 20.000 | 100.000 | 88.845 | 2.553 | 19.442 |
| Q6 | 0.00 | 100.00 | 84.741 | 2.251 | 24.243 | Q6 | 20.00 | 100.000 | 85.212 | 2.162 | 19.933 | Q6 | 40.000 | 100.000 | 88.569 | 2.585 | 19.689 |
| Q8 | 0.00 | 100.00 | 85.724 | 2.196 | 23.652 | Q8 | 0.00 | 100.000 | 83.271 | 2.480 | 22.865 | Q8 | 0.000 | 100.000 | 89.431 | 2.653 | 20.202 |
| Q10 | 0.00 | 100.00 | 82.526 | 2.133 | 22.972 | Q10 | 20.00 | 100.000 | 78.976 | 2.152 | 19.836 | Q10 | 20.000 | 100.000 | 81.638 | 3.000 | 22.846 |
| Q14 | 0.00 | 100.00 | 85.129 | 2.069 | 22.279 | Q14 | 0.00 | 100.000 | 83.400 | 2.331 | 21.492 | Q14 | 30.000 | 100.000 | 85.776 | 2.776 | 21.140 |
| Q16 | 0.00 | 100.00 | 86.560 | 1.899 | 20.449 | Q16 | 0.00 | 100.000 | 81.894 | 2.191 | 20.198 | Q16 | 25.000 | 100.000 | 86.190 | 2.623 | 19.975 |
| Q18 | 0.00 | 100.00 | 78.060 | 2.465 | 26.545 | Q18 | 20.00 | 100.000 | 79.871 | 2.237 | 20.620 | Q18 | 20.000 | 100.000 | 80.621 | 3.125 | 23.801 |
| Q20 | Q20 | 20.00 | 100.000 | 87.388 | 1.864 | 17.182 | Q20 | 50.000 | 100.000 | 86.000 | 2.205 | 16.792 | |||||
| S3 n = 118 | Min | Max | s | S3 n = 85 | Min | Max | s | S3 n = 57 | Min | Max | s | ||||||
| Q2 | 20.00 | 100.00 | 90.042 | 1.625 | 17.652 | Q2 | 50.00 | 100.000 | 89.906 | 1.640 | 15.121 | Q2 | 20.000 | 100.000 | 87.895 | 2.517 | 19.006 |
| Q4 | 30.00 | 100.00 | 88.441 | 1.578 | 17.140 | Q4 | 0.00 | 100.000 | 86.082 | 2.186 | 20.153 | Q4 | 0.000 | 100.000 | 84.649 | 3.114 | 23.511 |
| Q6 | 0.00 | 100.00 | 88.958 | 1.821 | 19.778 | Q6 | 10.00 | 100.000 | 89.365 | 1.930 | 17.794 | Q6 | 20.000 | 100.000 | 87.789 | 2.616 | 19.748 |
| Q8 | 0.00 | 100.00 | 87.051 | 2.051 | 22.284 | Q8 | 10.00 | 100.000 | 88.106 | 1.994 | 18.387 | Q8 | 20.000 | 100.000 | 88.175 | 2.798 | 21.128 |
| Q10 | 0.00 | 100.00 | 81.653 | 2.040 | 22.165 | Q10 | 10.00 | 100.000 | 80.600 | 2.200 | 20.281 | Q10 | 20.000 | 100.000 | 85.175 | 2.684 | 20.263 |
| Q14 | 0.00 | 100.00 | 84.373 | 2.089 | 22.697 | Q14 | 0.00 | 100.000 | 85.059 | 2.308 | 21.282 | Q14 | 0.000 | 100.000 | 83.649 | 3.383 | 25.540 |
| Q16 | 0.00 | 100.00 | 84.068 | 2.126 | 23.089 | Q16 | 50.00 | 100.000 | 88.506 | 1.562 | 14.403 | Q16 | 20.000 | 100.000 | 88.018 | 2.453 | 18.523 |
| Q18 | 0.00 | 100.00 | 85.186 | 2.196 | 23.856 | Q18 | 50.00 | 100.000 | 89.200 | 1.603 | 14.783 | Q18 | 50.000 | 100.000 | 88.509 | 2.279 | 17.207 |
| Q20 | Q20 | 46.00 | 100.000 | 86.035 | 2.011 | 18.537 | Q20 | 30.000 | 100.000 | 88.474 | 2.472 | 18.660 | |||||
| S4 n = 118 | Min | Max | s | S4 n = 85 | Min | Max | s | S4 n = 60 | Min | Max | s | ||||||
| Q2 | 0.00 | 100.00 | 87.975 | 1.863 | 20.239 | Q2 | 30.00 | 100.000 | 87.353 | 1.788 | 16.486 | Q2 | 39.000 | 100.000 | 88.217 | 2.455 | 19.016 |
| Q4 | 7.00 | 100.00 | 83.644 | 1.923 | 20.894 | Q4 | 20.00 | 100.000 | 84.247 | 2.021 | 18.635 | Q4 | 20.000 | 100.000 | 83.900 | 2.717 | 21.044 |
| Q6 | 0.00 | 100.00 | 85.398 | 2.032 | 22.076 | Q6 | 20.00 | 100.000 | 85.294 | 2.103 | 19.392 | Q6 | 20.000 | 100.000 | 88.150 | 2.609 | 20.211 |
| Q8 | 0.00 | 100.00 | 85.186 | 2.080 | 22.594 | Q8 | 0.00 | 100.000 | 84.494 | 2.172 | 20.022 | Q8 | 10.000 | 100.000 | 87.700 | 2.745 | 21.266 |
| Q10 | 0.00 | 100.00 | 82.280 | 2.287 | 24.838 | Q10 | 20.00 | 100.000 | 84.012 | 2.029 | 18.706 | Q10 | 40.000 | 100.000 | 86.917 | 2.460 | 19.055 |
| Q14 | 0.00 | 100.00 | 86.568 | 1.984 | 21.555 | Q14 | 10.00 | 100.000 | 86.659 | 1.968 | 18.143 | Q14 | 0.000 | 100.000 | 90.267 | 2.535 | 19.636 |
| Q16 | 20.00 | 100.00 | 87.161 | 1.856 | 20.165 | Q16 | 0.00 | 100.000 | 84.765 | 2.125 | 19.591 | Q16 | 35.000 | 100.000 | 88.583 | 2.323 | 17.996 |
| Q18 | 0.00 | 100.00 | 82.017 | 2.313 | 25.121 | Q18 | 20.00 | 100.000 | 82.082 | 2.053 | 18.930 | Q18 | 34.000 | 100.000 | 86.150 | 2.451 | 18.985 |
| Q20 | Q20 | 50.00 | 100.000 | 84.129 | 1.898 | 17.503 | Q20 | 28.000 | 100.000 | 84.717 | 2.629 | 20.365 | |||||
| S5 n = 119 | Min | Max | s | S5 n = 84 | Min | Max | s | S5 n = 54 | Min | Max | s | ||||||
| Q2 | 0.00 | 100.00 | 85.193 | 2.051 | 22.377 | Q2 | 50.00 | 100.000 | 89.643 | 1.649 | 15.111 | Q2 | 50.000 | 100.000 | 86.852 | 2.539 | 18.661 |
| Q4 | 0.00 | 100.00 | 85.538 | 2.015 | 21.985 | Q4 | 40.00 | 100.000 | 84.369 | 1.874 | 17.172 | Q4 | 30.000 | 100.000 | 88.056 | 2.599 | 19.097 |
| Q6 | 0.00 | 100.00 | 88.345 | 1.917 | 20.908 | Q6 | 40.00 | 100.000 | 88.679 | 1.973 | 18.086 | Q6 | 25.000 | 100.000 | 90.778 | 2.512 | 18.458 |
| Q8 | 0.00 | 100.00 | 86.731 | 2.045 | 22.303 | Q8 | 0.00 | 100.000 | 89.405 | 2.128 | 19.500 | Q8 | 1.000 | 100.000 | 89.444 | 2.666 | 19.591 |
| Q10 | 0.00 | 100.00 | 85.076 | 2.150 | 23.452 | Q10 | 50.00 | 100.000 | 82.917 | 1.962 | 17.983 | Q10 | 20.000 | 100.000 | 84.130 | 2.764 | 20.310 |
| Q14 | 0.00 | 100.00 | 84.361 | 2.112 | 23.037 | Q14 | 10.00 | 100.000 | 85.643 | 2.119 | 19.417 | Q14 | 40.000 | 100.000 | 86.907 | 2.485 | 18.261 |
| Q16 | 0.00 | 100.00 | 85.571 | 2.198 | 23.979 | Q16 | 30.00 | 100.000 | 85.940 | 1.842 | 16.886 | Q16 | 40.000 | 100.000 | 90.667 | 2.081 | 15.293 |
| Q18 | 0.00 | 100.00 | 82.336 | 2.285 | 24.925 | Q18 | 0.00 | 100.000 | 80.036 | 2.254 | 20.660 | Q18 | 20.000 | 100.000 | 80.963 | 3.002 | 22.056 |
| Q20 | Q20 | 50.00 | 100.000 | 87.381 | 1.705 | 15.627 | Q20 | 40.000 | 100.000 | 87.944 | 2.320 | 17.045 | |||||
| S6 n = 115 | Min | Max | s | S6 n = 85 | Min | Max | s | S6 n = 55 | Min | Max | s | ||||||
| Q2 | 0.00 | 100.00 | 89.061 | 2.116 | 22.691 | Q2 | 30.00 | 100.000 | 90.212 | 1.858 | 17.133 | Q2 | 25.000 | 100.000 | 90.345 | 2.409 | 17.868 |
| Q4 | 0.00 | 100.00 | 84.730 | 2.344 | 25.132 | Q4 | 0.00 | 100.000 | 85.929 | 2.431 | 22.413 | Q4 | 10.000 | 100.000 | 86.473 | 3.021 | 22.408 |
| Q6 | 0.00 | 100.00 | 86.061 | 2.104 | 22.559 | Q6 | 30.00 | 100.000 | 89.318 | 2.004 | 18.474 | Q6 | 1.000 | 100.000 | 89.545 | 2.847 | 21.112 |
| Q8 | 0.00 | 100.00 | 86.643 | 2.123 | 22.772 | Q8 | 50.00 | 100.000 | 88.988 | 1.754 | 16.172 | Q8 | 10.000 | 100.000 | 88.473 | 2.950 | 21.881 |
| Q10 | 0.00 | 100.00 | 84.009 | 2.192 | 23.505 | Q10 | 50.00 | 100.000 | 89.788 | 1.712 | 15.787 | Q10 | 20.000 | 100.000 | 84.909 | 3.027 | 22.447 |
| Q14 | 0.00 | 100.00 | 87.861 | 1.908 | 20.460 | Q14 | 0.00 | 100.000 | 85.906 | 2.247 | 20.715 | Q14 | 1.000 | 100.000 | 86.127 | 2.931 | 21.740 |
| Q16 | 0.00 | 100.00 | 86.843 | 1.989 | 21.335 | Q16 | 20.00 | 100.000 | 85.447 | 1.940 | 17.887 | Q16 | 50.000 | 100.000 | 89.545 | 2.269 | 16.830 |
| Q18 | 0.00 | 100.00 | 83.078 | 2.091 | 22.428 | Q18 | 0.00 | 100.000 | 83.988 | 2.254 | 20.780 | Q18 | 34.000 | 100.000 | 82.709 | 2.828 | 20.972 |
| Q20 | Q20 | 10.00 | 100.000 | 86.341 | 2.084 | 19.210 | Q20 | 50.000 | 100.000 | 90.327 | 2.141 | 15.882 | |||||
| S7 n = 120 | Min | Max | s | S7 n = 85 | Min | Max | s | S7 n = 56 | Min | Max | s | ||||||
| Q2 | 0.00 | 100.00 | 88.842 | 1.901 | 20.830 | Q2 | 50.00 | 100.000 | 92.388 | 1.497 | 13.804 | Q2 | 40.000 | 100.000 | 90.357 | 2.385 | 17.850 |
| Q4 | 0.00 | 100.00 | 85.458 | 2.187 | 23.962 | Q4 | 50.00 | 100.000 | 89.506 | 1.732 | 15.966 | Q4 | 40.000 | 100.000 | 85.357 | 2.713 | 20.304 |
| Q6 | 0.00 | 100.00 | 85.367 | 2.047 | 22.426 | Q6 | 46.00 | 100.000 | 87.800 | 1.867 | 17.210 | Q6 | 38.000 | 100.000 | 88.464 | 2.578 | 19.293 |
| Q8 | 0.00 | 100.00 | 85.208 | 2.181 | 23.891 | Q8 | 0.00 | 100.000 | 87.671 | 2.107 | 19.422 | Q8 | 30.000 | 100.000 | 88.625 | 2.649 | 19.820 |
| Q10 | 0.00 | 100.00 | 81.583 | 2.344 | 25.677 | Q10 | 40.00 | 100.000 | 87.400 | 1.882 | 17.354 | Q10 | 20.000 | 100.000 | 85.625 | 2.916 | 21.818 |
| Q14 | 0.00 | 100.00 | 85.275 | 2.161 | 23.675 | Q14 | 50.00 | 100.000 | 90.576 | 1.526 | 14.069 | Q14 | 20.000 | 100.000 | 89.304 | 2.364 | 17.687 |
| Q16 | 0.00 | 100.00 | 86.233 | 2.037 | 22.312 | Q16 | 6.00 | 100.000 | 87.447 | 1.930 | 17.795 | Q16 | 40.000 | 100.000 | 85.661 | 2.663 | 19.930 |
| Q18 | 0.00 | 100.00 | 84.258 | 2.174 | 23.814 | Q18 | 43.00 | 100.000 | 86.576 | 1.881 | 17.343 | Q18 | 44.000 | 100.000 | 84.089 | 2.594 | 19.415 |
| Q20 | Q20 | 20.00 | 100.000 | 88.200 | 1.978 | 18.236 | Q20 | 8.000 | 100.000 | 88.589 | 2.628 | 19.664 | |||||
| S8 n = 84 | Min | Max | s | S8 n = 55 | Min | Max | s | ||||||||||
| Q2 | 30.00 | 100.000 | 89.762 | 1.883 | 17.257 | Q2 | 10.000 | 100.000 | 88.145 | 2.920 | 21.652 | ||||||
| Q4 | 30.00 | 100.000 | 84.464 | 2.272 | 20.823 | Q4 | 0.000 | 100.000 | 86.727 | 3.177 | 23.564 | ||||||
| Q6 | 10.00 | 100.000 | 87.536 | 2.319 | 21.255 | Q6 | 10.000 | 100.000 | 88.345 | 2.914 | 21.613 | ||||||
| Q8 | 1.00 | 100.000 | 85.583 | 2.449 | 22.449 | Q8 | 10.000 | 100.000 | 87.691 | 3.139 | 23.276 | ||||||
| Q10 | 35.00 | 100.000 | 87.548 | 1.954 | 17.907 | Q10 | 10.000 | 100.000 | 85.964 | 3.000 | 22.250 | ||||||
| Q14 | 30.00 | 100.000 | 84.702 | 2.095 | 19.203 | Q14 | 10.000 | 100.000 | 88.473 | 2.613 | 19.381 | ||||||
| Q16 | 50.00 | 100.000 | 88.857 | 1.734 | 15.889 | Q16 | 45.000 | 100.000 | 89.127 | 2.341 | 17.361 | ||||||
| Q18 | 0.00 | 100.000 | 85.524 | 2.141 | 19.623 | Q18 | 20.000 | 100.000 | 86.800 | 2.704 | 20.052 | ||||||
| Q20 | 46.00 | 100.000 | 90.976 | 1.777 | 16.283 | Q20 | 50.000 | 100.000 | 91.855 | 1.918 | 14.222 | ||||||
| S9 n = 85 | Min | Max | s | S9 n = 55 | Min | Max | s | ||||||||||
| Q2 | 43.00 | 100.000 | 89.600 | 1.832 | 16.891 | Q2 | 50.000 | 100.000 | 89.800 | 2.238 | 16.597 | ||||||
| Q4 | 0.00 | 100.000 | 86.306 | 2.222 | 20.487 | Q4 | 20.000 | 100.000 | 90.327 | 2.576 | 19.107 | ||||||
| Q6 | 6.00 | 100.000 | 85.894 | 2.365 | 21.805 | Q6 | 45.000 | 100.000 | 90.055 | 2.360 | 17.501 | ||||||
| Q8 | 17.00 | 100.000 | 88.000 | 2.185 | 20.148 | Q8 | 50.000 | 100.000 | 92.182 | 1.991 | 14.765 | ||||||
| Q10 | 20.00 | 100.000 | 91.082 | 1.806 | 16.647 | Q10 | 0.000 | 100.000 | 86.055 | 3.265 | 24.217 | ||||||
| Q14 | 10.00 | 100.000 | 86.624 | 2.357 | 21.731 | Q14 | 10.000 | 100.000 | 89.782 | 2.418 | 17.935 | ||||||
| Q16 | 0.00 | 100.000 | 87.718 | 2.184 | 20.134 | Q16 | 40.000 | 100.000 | 89.055 | 2.200 | 16.312 | ||||||
| Q18 | 0.00 | 100.000 | 83.494 | 2.567 | 23.667 | Q18 | 10.000 | 100.000 | 87.600 | 2.730 | 20.244 | ||||||
| Q20 | 45.00 | 100.000 | 93.647 | 1.444 | 13.315 | Q20 | 45.000 | 100.000 | 95.364 | 1.565 | 11.607 | ||||||
Figure A1.
k selection error logs for the 2021 LR coded data.
Figure A2.
k selection error logs for the 2021 LR CP coded data.
Figure A3.
k selection error logs for the 2021 SR CP coded data.
Figure A4.
k selection error logs for scenarios S1–S9 for the 2021 LR CP coded data.
Figure A5.
k selection error logs for scenarios S1–S9 for the 2021 SR CP coded data.
Table A8.
Predictor importance for target I7, for scenarios S1–S9 and the 2021 LR CP coded data.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | 2021 LR CP | |
|---|---|---|---|---|---|---|---|---|---|---|
| I1 | 0.12 | 0.13 | 0.12 | 0.13 | 0.12 | 0.14 | 0.12 | 0.15 | 0.13 | 0.12 |
| I2 | 0.12 | 0.12 | 0.12 | 0.12 | 0.13 | 0.1 | 0.13 | 0.14 | 0.12 | 0.13 |
| I3 | 0.13 | 0.14 | 0.13 | 0.12 | 0.11 | 0.14 | 0.12 | 0.14 | 0.13 | 0.12 |
| I4 | 0.13 | 0.12 | 0.11 | 0.13 | 0.11 | 0.15 | 0.13 | 0.09 | 0.13 | 0.13 |
| I5 | 0.13 | 0.12 | 0.13 | 0.13 | 0.13 | 0.16 | 0.12 | 0.14 | 0.12 | 0.12 |
| I6 | 0.13 | 0.12 | 0.13 | 0.13 | 0.13 | 0.08 | 0.12 | 0.11 | 0.12 | 0.13 |
| I8 | 0.13 | 0.12 | 0.12 | 0.13 | 0.13 | 0.09 | 0.13 | 0.12 | 0.13 | 0.13 |
| I9 | 0.12 | 0.13 | 0.13 | 0.12 | 0.13 | 0.14 | 0.12 | 0.13 | 0.13 | 0.13 |
Table A9.
Predictor importance for target I7, for scenarios S1–S9 and the 2021 SR CP coded data.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | 2021 SR CP | |
|---|---|---|---|---|---|---|---|---|---|---|
| I1 | 0.13 | 0.14 | 0.15 | 0.12 | 0.12 | 0.13 | 0.12 | 0.12 | 0.13 | 0.12 |
| I2 | 0.13 | 0.12 | 0.14 | 0.12 | 0.12 | 0.12 | 0.12 | 0.14 | 0.13 | 0.13 |
| I3 | 0.12 | 0.13 | 0.11 | 0.13 | 0.12 | 0.12 | 0.11 | 0.13 | 0.13 | 0.12 |
| I4 | 0.12 | 0.12 | 0.12 | 0.12 | 0.13 | 0.14 | 0.12 | 0.13 | 0.14 | 0.12 |
| I5 | 0.12 | 0.13 | 0.12 | 0.13 | 0.14 | 0.11 | 0.12 | 0.13 | 0.13 | 0.12 |
| I6 | 0.13 | 0.11 | 0.13 | 0.12 | 0.14 | 0.12 | 0.13 | 0.12 | 0.1 | 0.13 |
| I8 | 0.13 | 0.13 | 0.12 | 0.13 | 0.13 | 0.12 | 0.14 | 0.11 | 0.11 | 0.13 |
| I9 | 0.13 | 0.13 | 0.12 | 0.13 | 0.13 | 0.15 | 0.14 | 0.12 | 0.12 | 0.12 |
Table A10.
Component matrices for scenarios S1–S9 from the 2021 LR CP coded dataset.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
| I1 | 0.758 | 0.816 | 0.843 | 0.776 | 0.713 | 0.797 | 0.825 | 0.878 | 0.808 | |
| I2 | 0.738 | 0.772 | 0.664 | 0.664 | 0.808 | 0.843 | 0.626 | 0.716 | 0.578 | |
| I3 | 0.785 | 0.831 | 0.866 | 0.789 | 0.866 | 0.787 | 0.755 | 0.763 | 0.860 | |
| I4 | 0.7 | 0.835 | 0.801 | 0.786 | 0.792 | 0.705 | 0.768 | 0.600 | 0.768 | |
| I5 | 0.693 | 0.706 | 0.670 | 0.732 | 0.679 | 0.811 | 0.667 | 0.747 | 0.604 | |
| I6 | omitted | 0.461 | 0.709 | 0.651 | omitted | 0.611 | 0.645 | omitted | omitted | |
| I7 | 0.752 | 0.657 | 0.659 | 0.718 | 0.682 | 0.691 | 0.699 | 0.728 | 0.840 | |
| I8 | omitted | 0.597 | 0.626 | omitted | omitted | omitted | 0.574 | 0.746 | omitted | |
| I9 | 0.562 | omitted | omitted | omitted | omitted | 0.703 | 0.701 | 0.519 | omitted | |
| variance explained | 57.68% | 51.87% | 54% | 53.73% | 57.76% | 55.81% | 48.93% | 51.74% | 56.43% | |
| KMO | 0.629 | 0.815 | 0.85 | 0.783 | 0.844 | 0.804 | 0.819 | 0.836 | 0.792 | |
| Bartlett’s test | 93.38 (p < 0.001) | 200.37 (p < 0.001) | 206.2 (p < 0.001) | 176.4 (p < 0.001) | 129.66 (p < 0.001) | 219.86 (p < 0.001) | 201.69 (p < 0.001) | 187.97 (p < 0.001) | 146.26 (p < 0.001) | |
Table A11.
Component matrices for scenarios S1–S9 from the 2021 SR CP coded dataset.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
| I1 | 0.691 | 0.823 | 0.818 | 0.787 | 0.713 | 0.794 | 0.823 | 0.85 | 0.819 | ||
| I2 | 0.813 | 0.777 | 0.57 | 0.665 | 0.808 | 0.793 | 0.615 | 0.667 | 0.584 | ||
| I3 | 0.771 | 0.853 | 0.881 | 0.783 | 0.866 | 0.791 | 0.772 | 0.7 | 0.827 | ||
| I4 | 0.666 | 0.802 | 0.806 | 0.796 | 0.792 | 0.724 | 0.722 | omitted * | 0.692 | ||
| I5 | 0.728 | −0.43 | 0.731 | 0.656 | 0.731 | 0.679 | 0.8 | 0.664 | 0.77 | omitted * | |
| I6 | 0.775 | omitted * | 0.537 | 0.612 | omitted * | 0.531 | 0.655 | 0.46 * | 0.592 * | ||
| I7 | 0.87 | 0.65 | 0.66 | 0.705 | 0.682 | 0.685 | 0.692 | 0.749 | 0.864 | ||
| I8 | omitted | 0.628 | 0.606 | omitted * | omitted | omitted | 0.591 | 0.769 | omitted | ||
| I9 | omitted | omitted | omitted | omitted | omitted | 0.688 | 0.697 | 0.536 | omitted | ||
| variance explained | 71.77% | 57.157% | 49.25% | 53.06% | 57.76% | 53.45% | 48.38% | 48.76% | 54.49% | ||
| KMO | 0.716 | 0.832 | 0.835 | 0.779 | 0.844 | 0.832 | 0.75 | 0.808 | 0.773 | ||
| Bartlett’s test | 74.823 (p < 0.001) | 183.84 (p < 0.001) | 178.41 (p < 0.001) | 172.42 (p < 0.001) | 129.66 (p < 0.001) | 189.91 (p < 0.001) | 209.03 (p < 0.001) | 162.62 (p < 0.001) | 132.13 (p < 0.001) | ||
Note: * denotes differences in comparison to LR coded dataset.
References
- Halpern, J.Y.; Rêgo, L. Extensive games with possibly unaware players. arXiv 2007, arXiv:0704.2014. Available online: https://arxiv.org/abs/0704.2014 (accessed on 10 February 2019).
- Li, J. Information structures with unawareness. J. Econ. Theory 2009, 144, 977–993. [Google Scholar] [CrossRef] [Green Version]
- Chen, Y.-C.; Ely, J.C.; Luo, X. Note on unawareness: Negative Introspection versus AU Introspection (and KU Introspection). Int. J. Game Theory 2012, 41, 325–329. [Google Scholar] [CrossRef]
- Chen, G.; Shen, D.; Kwan, C.; Cruz, J.B.; Kruger, M. Game theoretic approach to threat prediction and situation awareness. In Proceedings of the 9th International Conference on Information Fusion, Florence, Italy, 10–13 July 2006; pp. 1–8. [Google Scholar]
- Chugh, D.; Bazerman, M.H. Bounded awareness: What you fail to see can hurt you. Mind Soc. 2007, 6, 1–18. [Google Scholar] [CrossRef]
- Bazerman, M.H.; Chugh, D. Bounded awareness: Focusing failures in negotiation. In Negotiation Theory and Research; Psychology Press: New York, NY, USA, 2006; pp. 7–26. [Google Scholar]
- Endsley, M.R. Toward a theory of situation awareness in dynamic systems. Hum. Factors 1995, 37, 32–64. [Google Scholar] [CrossRef]
- Feinberg, Y. Games with unawareness. BE J. Theor. Econ. 2021, 21, 433–488. Available online: https://www.researchgate.net/profile/Yossi-Feinberg/publication/256039554_Games_with_Unawareness/links/5b4529b6a6fdcc6619170a93/Games-with-Unawareness.pdf (accessed on 10 February 2019). [CrossRef]
- Piermont, E. Unforeseen evidence. J. Econ. Theory 2021, 193, 105235. [Google Scholar] [CrossRef]
- Kostelic, K. Guessing the game: An individual’s awareness and assessment of a game’s existence. Games 2020, 11, 17. [Google Scholar] [CrossRef] [Green Version]
- Neuberg, S.L. Behavioral implications of information presented outside of conscious awareness: The effect of subliminal presentation of trait information on behavior in the Prisoner’s Dilemma Game. Soc. Cogn. 1988, 6, 207–230. [Google Scholar] [CrossRef]
- Halpern, J.Y.; Piermont, E. Partial awareness. arXiv 2018, arXiv:1811.05751. Available online: https://arxiv.org/abs/1811.05751 (accessed on 10 February 2019). [CrossRef]
- Grant, S.; Quiggin, J. Inductive reasoning about unawareness. Econ. Theory 2013, 54, 717–755. [Google Scholar] [CrossRef] [Green Version]
- Halpern, J.Y.; Piermont, E. Dynamic awareness. arXiv 2020, preprint. arXiv:2007.02823. Available online: https://arxiv.org/abs/2007.02823 (accessed on 10 February 2019).
- Zhao, W. Cost of reasoning and strategic sophistication. Games 2020, 11, 40. [Google Scholar] [CrossRef]
- Rêgo, L.C.; Halpern, J.Y. Generalized solution concepts in games with possibly unaware players. Int. J. Game Theory 2012, 41, 131–155. [Google Scholar] [CrossRef]
- Blasch, E.; Shen, D.; Pham, K.D.; Chen, G. Review of game theory applications for situation awareness. In Sensors and Systems for Space Applications VIII; International Society for Optics and Photonics: Bellingham, WA, USA, 2015; Volume 9469. [Google Scholar]
- Gino, F.; Norton, M.I.; Weber, R.A. Motivated Bayesians: Feeling moral while acting egoistically. J. Econ. Perspect. 2016, 30, 189–212. [Google Scholar] [CrossRef] [Green Version]
- Ramirez, R.; Mukherjee, M.; Vezzoli, S.; Kramer, A.M. Scenarios as a scholarly methodology to produce “interesting research”. Futures 2015, 71, 70–87. [Google Scholar] [CrossRef]
- Camerer, C.F. Progress in behavioral game theory. J. Econ. Perspect. 1997, 11, 167–188. [Google Scholar] [CrossRef] [Green Version]
- Aumann, R. Interactive knowledge. Int. J. Game Theory 1999, 28, 263–300. [Google Scholar] [CrossRef]
- Cozic, M. Probabilistic unawareness. Games 2016, 7, 38. [Google Scholar] [CrossRef] [Green Version]
- Burgoon, J.K.; Buller, D.B. Interpersonal deception: III. Effects of deceit on perceived communication and nonverbal behavior dynamics. J. Nonverbal Behav. 1994, 18, 155–184. [Google Scholar] [CrossRef]
- Synodinos, N.E. The “art” of questionnaire construction: Some important considerations for manufacturing studies. Integr. Manuf. Syst. 2003, 14, 221–237. [Google Scholar] [CrossRef]
- Chan, L.L.; Idris, N. Validity and reliability of the instrument using exploratory factor analysis and Cronbach’s alpha. Int. J. Acad. Res. Bus. Soc. Sci. 2017, 7, 400–410. [Google Scholar]
- Peterson, R. Constructing Effective Questionnaires; Sage Publications: Thousand Oaks, CA, USA, 2000. [Google Scholar] [CrossRef]
- Salehnejad, R. Rationality, Bounded Rationality and Microfoundations: Foundations of Theoretical Economics; Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
- Kahneman, D. Maps of bounded rationality: A perspective on intuitive judgment and choice. Nobel Prize Lect. 2002, 8, 351–401. [Google Scholar]
- Božac, M.G.; Kostelić, K.; Paulišić, M.; Smith, C. Business ethics decision-making: Examining partial reflective awareness. Sustainability 2021, 13, 2635. [Google Scholar] [CrossRef]
- Schuman, H.; Presser, S. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context; Sage: Thousand Oaks, CA, USA, 1996. [Google Scholar]
- Patel, H.; Joseph, J. Questionnaire designing process: A review. J. Clin. Trials 2016, 6, 255. [Google Scholar]
- Sreejesh, S.; Mohapatra, S.; Anusree, M.R. Questionnaire design. In Business Research Methods; Springer: Cham, Switzerland, 2014; pp. 143–159. [Google Scholar] [CrossRef]
- Labaw, P. Advanced Questionnaire Design; Abt Books: Cambridge, MA, USA, 1980. [Google Scholar]
- Hunt, S.D.; Sparkman, R.D., Jr.; Wilcox, J.B. The pretest in survey research: Issues and preliminary findings. J. Market. Res. 1982, 19, 269–273. [Google Scholar] [CrossRef]
- Stemler, S. An overview of content analysis. Pract. Assess. Res. Eval. 2001, 7, 137–146. [Google Scholar]
- Blair, E. A reflexive exploration of two qualitative data coding techniques. J. Methods Meas. Soc. Sci. 2015, 6, 14–29. [Google Scholar] [CrossRef] [Green Version]
- Krippendorff, K. Content Analysis: An Introduction to Its Methodology, 2nd ed.; Sage Publications: Thousand Oaks, CA, USA, 2004. [Google Scholar]
- Adu, P. A Step-by-Step Guide to Qualitative Data Coding; Routledge: Oxford, UK, 2019. [Google Scholar] [CrossRef]
- Armstrong, D.; Gosling, A.; Weinman, J.; Marteau, T. The place of inter-rater reliability in qualitative research: An empirical study. Sociology 1997, 31, 597–606. [Google Scholar] [CrossRef]
- Morse, J.M.; Barrett, M.; Mayan, M.; Olson, K.; Spiers, J. Verification strategies for establishing reliability and validity in qualitative research. Int. J. Qual. Methods 2002, 1, 13–22. [Google Scholar] [CrossRef]
- Peshkin, A. In search of subjectivity—One’s own. Educ. Res. 1988, 17, 17–21. [Google Scholar]
- Strauss, A.; Corbin, J. Basics of Qualitative Research Techniques; Sage Publications: Thousand Oaks, CA, USA, 1998. [Google Scholar]
- Taherdoost, H. Validity and reliability of the research instrument; How to test the validation of a questionnaire/survey in a research. Int. J. Acad. Res. Manag. 2016, 5, 28–36. [Google Scholar] [CrossRef]
- Cronbach, L.J. Coefficient alpha and the internal structure of tests. Psychometrika 1951, 16, 297–334. [Google Scholar] [CrossRef] [Green Version]
- Kline, R.B. Principles and Practice of Structural Equation Modeling, 3rd ed.; The Guilford Press: New York, NY, USA, 2011. [Google Scholar]
- Wilson, J. Essentials of Business Research: A Guide to Doing Your Research Project; Sage: Thousand Oaks, CA, USA, 2014. [Google Scholar]
- Buckendahl, C.; Plake, B.S. Evaluating tests. In Handbook of Test Development; Downing, S.M., Haladyna, T.M., Eds.; Lawrence Erlbaum Associates Publishers: Mahwah, NJ, USA, 2006; pp. 725–738. [Google Scholar]
- Messick, S. Validity. In Educational Measurement, 3rd ed.; Linn, R., Ed.; American Council of Education: Washington, DC, USA; Macmillan: New York, NY, USA, 1989; pp. 13–103. [Google Scholar]
- Campbell, D.T.; Fiske, D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol. Bull. 1959, 56, 81. [Google Scholar] [CrossRef] [Green Version]
- Korkmaz, S.; Goksuluk, D.; Zararsiz, G. MVN: An R package for assessing multivariate normality. R J. 2014, 6, 151. [Google Scholar] [CrossRef] [Green Version]
- Mittag, K.C.; Scale-free nonparametric factor analysis: A user-friendly introduction with concrete heuristic examples. Presented at the Annual Meeting of the Southwest Educational Research Association, Austin, TX, USA, 28–30 January 1993. Available online: https://files.eric.ed.gov/fulltext/ED355281.pdf (accessed on 18 October 2021).
- De Winter, J.C.F.; Gosling, S.D.; Potter, J. Comparing the Pearson and Spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychol. Methods 2016, 21, 273–290. [Google Scholar] [CrossRef]
- Conover, W.J.; Iman, R.L. Rank transformations as a bridge between parametric and nonparametric statistics. Am. Stat. 1981, 35, 124–129. [Google Scholar]
- Garcia-Granero, M. Factor Analysis with Spearman Correlation through a Matrix. 2002. Available online: https://www.spsstools.net/en/syntax/syntax-index/factor-analysis/factor-analysis-with-spearman-correlation-through-a-matrix/ (accessed on 18 October 2021).
- Weathington, B.L.; Cunningham, C.J.; Pittenger, D.J. Research Methods for the Behavioral and Social Sciences; Wiley: Hoboken, NJ, USA, 2010. [Google Scholar]
- Kane, M.T. Validating the Interpretations and Uses of Test Scores. J. Educ. Meas. 2013, 50, 1–73. [Google Scholar] [CrossRef]
- Škare, M.; Kostelić, K. Interpersonal communication in the internal marketing: Bounded rationality game theory approach. Econ. Comput. Econ. Cybern. Stud. Res. 2015, 49, 127–149. [Google Scholar]
- Nassiri-Mofakham, F.; Nematbakhsh, M.A.; Ghasem-Aghaee, N.; Baraani-Dastjerdi, A. A heuristic personality-based bilateral multi-issue bargaining model in electronic commerce. Int. J. Hum. Comput. Stud. 2009, 67, 1–35. [Google Scholar] [CrossRef]
- Nassiri-Mofakham, F.; Ghasem-Aghaee, N.; Nematbakhsh, M.A.; Baraani-Dastjerdi, A. A personality-based simulation of bargaining in e-commerce. Simul. Gaming 2008, 39, 83–100. [Google Scholar] [CrossRef] [Green Version]
Figure 1.
k selection error logs for scenarios S1–S9, 2021 LR coded data. Scenarios 1–9 are denoted S1–S9, respectively.
Figure 1.
k selection error logs for scenarios S1–S9, 2021 LR coded data. Scenarios 1–9 are denoted S1–S9, respectively.
Table 1.
Information provided in scenarios.
| Information | Scenario |
|---|---|
| Scenario 1 | |
| action | The price increases |
| player(s) | Not stated explicitly (it can be recognized as shop/shop manager, manufacturer, distributor, etc.) |
| outcome | Loss of 10 HRK/the lack of satisfaction from favorite chocolate |
| Scenario 2 | |
| action | Non-verbal signals |
| player(s) | Not stated explicitly (a friend with whom an individual is drinking coffee, or a third party) |
| outcome | Not stated explicitly (the payoff could be both positive and negative) |
| Scenario 3 | |
| action | Two separate actions are stated: workload and a rumor |
| player(s) | Two players are stated: colleague and task assignor |
| outcome | Not stated explicitly, but both positive and negative payoffs are indicated |
| Scenario 4 | |
| action | A rumor/company’s turnover |
| player(s) | Not stated explicitly (the rumor’s source is the colleagues, but the player is the company) |
| outcome | Not stated explicitly (a raise is expressed as a possible payoff, but the information about the higher workload can be deduced from more tourist arrivals, which could also be related to higher salaries) |
| Scenario 5 | |
| action | Activities within a team assignment |
| player(s) | Team members |
| outcome | A grade/workload |
| Scenario 6 | |
| action | New competitor offers service at a lower price and/or a client’s attempt to negotiate a lower price |
| player(s) | Newly opened consultant company and/or a client |
| outcome | Not stated explicitly, but indicates possible loss (implicitly, in the long run, it can be the loss of a market share to a competitor and/or, in the short-run, a diminished price charged to the client or the loss of the client) |
| Scenario 7 | |
| action | Invitation to pitch the idea |
| player(s) | Five investors |
| outcome | Not stated explicitly (immediate—presentation of the idea (or not), and consequently a possibility of investment—positive payoff—of an unknown amount is indicated) |
| Scenario 8 | |
| action | Confess or remain silent, for both players |
| player(s) | A colleague |
| outcome | 0, 1, 6, or 12 months of suspension |
| Scenario 9 | |
| action | A promise of the colleague and one’s own action about the sandwich allocation |
| player(s) | A colleague |
| outcome | A possibility of a coffee and a piece of/whole sandwich |
Table 2.
Sample characteristics.
| Study | Age (Mean, Standard Deviation, Min, Max) | Gender (Frequency Male, Female) | Year of Study (Mean, Standard Deviation, Min, Max) | Previously Learned Game Theory (Frequency) | Used Knowledge of Game Theory (Frequency) | Number of Responses |
|---|---|---|---|---|---|---|
| 2019 | 22.03 | 48.54% | 1.934 | 54% | 21.8% | 822 |
| 4.788 | 1.1556 | |||||
| 18 | 51.45% | 1 | ||||
| 46 | 5 | |||||
| 2020 | 19.91 | 27% | 1.048 | 43.5% | 18.9% | 756 |
| 3.648 | 0.2131 | |||||
| 18 | 73% | 1 | ||||
| 52 | 2 | |||||
| 2021 | 19.86 | 33.5% | 1.06 | 41.8% * | 12% * | 452 * |
| 2.318 | 0.2373 | |||||
| 18 | 66.5% * | 1 | ||||
| 34 * | 2 * | |||||
| Total | 20.758 | 37.2% | 1.409 | 47.37% | 18.53% | 2030 |
| 4.065 | 0.8703 | |||||
| 18 | 62.8% | 1 | ||||
| 52 | 5 |
* A total of 46 responses for the socio-demographic data are missing in the 2021 study; the total number of responses in the 2021 study is 498; the total number of responses for all studies is 2076.
Table 3.
Reliability statistics per study with 7, 8, and 9 items included, respective to the coding.
Table 3.
Reliability statistics per study with 7, 8, and 9 items included, respective to the coding.
| Coding | LR | LR CP | SR CP | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Study/No. of Items | 9 | 8 | 7 | 9 | 8 | 7 | 9 | 8 | 7 |
| 2019 | / | 0.67 (0.751) | 0.659 (0.755) | / | 0.761 (0.763) | 0.771 (0.771) | / | 0.74 (0.744) | 0.757 (0.761) |
| 2020 | 0.744 (0.751) | 0.758 (0.767) | 0.759 (0.764) | 0.724 (0.73) | 0.742 (0.747) | 0.75 (0.753) | 0.677 (0.684) | 0.689 (0.699) | 0.695 (0.703) |
| 2021 | 0.819 (0.818) | 0.829 (0.823) | 0.841 (0.84) | 0.805 (0.802) | 0.817 (0.819) | 0.826 (0.826) | 0.7812 (0.782) | 0.795 (0.801) | 0.796 (0.802) |
Notes: For 9 examined items: I1–I9; 8 examined items: I1–I8; 7 examined items: I1–I7. Cronbach’s alpha is reported in the brackets, based on standardized items. LR stands for loosely coded data, SR stands for strictly coded data and CP stands for complementary probability applied to certainties.
Table 4.
Reliability statistics for the 2021 study data with I1–I9 and I1–I7 items, coded LR.
| Reliability Statistics | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items | Cronbach’s Alpha | Cronbach’s Alpha Based on Standardized Items | No. of Items |
|---|---|---|---|---|---|---|
| Scenario 1 | 0.750 | 0.743 | 9 | 0.780 | 0.767 | 7 |
| Scenario 2 | 0.861 | 0.853 | 9 | 0.870 | 0.870 | 7 |
| Scenario 3 | 0.852 | 0.847 | 9 | 0.875 | 0.875 | 7 |
| Scenario 4 | 0.783 | 0.797 | 9 | 0.811 | 0.815 | 7 |
| Scenario 5 | 0.727 | 0.735 | 9 | 0.782 | 0.779 | 7 |
| Scenario 6 | 0.846 | 0.862 | 9 | 0.884 | 0.890 | 7 |
| Scenario 7 | 0.842 | 0.845 | 9 | 0.832 | 0.833 | 7 |
| Scenario 8 | 0.824 | 0.822 | 9 | 0.833 | 0.833 | 7 |
| Scenario 9 | 0.778 | 0.779 | 9 | 0.794 | 0.794 | 7 |
Table 5.
Predictor importance for target I7, for scenarios S1–S9, and for the 2021 LR coded data.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | 2021 LR | |
|---|---|---|---|---|---|---|---|---|---|---|
| I1 | 0.15 | 0.12 | 0.13 | 0.13 | 0.12 | 0.1 | 0.13 | 0.15 | 0.13 | 0.12 |
| I2 | 0.12 | 0.13 | 0.12 | 0.12 | 0.11 | 0.11 | 0.14 | 0.15 | 0.12 | 0.12 |
| I3 | 0.11 | 0.12 | 0.13 | 0.12 | 0.13 | 0.1 | 0.11 | 0.15 | 0.13 | 0.13 |
| I4 | 0.12 | 0.11 | 0.13 | 0.13 | 0.12 | 0.12 | 0.14 | 0.1 | 0.12 | 0.12 |
| I5 | 0.13 | 0.14 | 0.13 | 0.13 | 0.12 | 0.15 | 0.11 | 0.1 | 0.13 | 0.13 |
| I6 | 0.12 | 0.14 | 0.13 | 0.12 | 0.14 | 0.15 | 0.12 | 0.1 | 0.13 | 0.12 |
| I8 | 0.12 | 0.13 | 0.12 | 0.13 | 0.12 | 0.16 | 0.13 | 0.12 | 0.11 | 0.13 |
| I9 | 0.12 | 0.12 | 0.12 | 0.12 | 0.14 | 0.1 | 0.12 | 0.14 | 0.12 | 0.13 |
Table 6.
Component matrices for the 2021 data.
| Component (LR) | Component (LR CP) | Component (SR CP) | |
|---|---|---|---|
| I1 | 0.801 | 0.795 | 0.793 |
| I3 | 0.773 | 0.78 | 0.777 |
| I4 | 0.758 | 0.736 | 0.654 |
| I2 | 0.736 | 0.705 | 0.66 |
| I5 | 0.725 | 0.719 | 0.72 |
| I7 | 0.714 | 0.688 | 0.684 |
| I6 | 0.634 | 0.612 | 0.544 |
| I9 | 0.496 | 0.466 | 0.478 |
| I8 | 0.480 | 0.459 | 0.443 |
Table 7.
Component matrices for scenarios S1–S9 from the 2021 LR coded dataset.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |
| I1 | 0.684 | 0.865 | 0.850 | 0.761 | 0.744 | 0.811 | 0.826 | 0.887 | 0.809 | ||
| I2 | 0.712 | 0.763 | 0.759 | 0.681 | 0.799 | 0.867 | 0.686 | 0.747 | 0.590 | ||
| I3 | 0.761 | 0.829 | 0.858 | 0.766 | 0.843 | 0.804 | 0.756 | 0.743 | 0.835 | ||
| I4 | 0.706 | 0.827 | 0.837 | 0.776 | 0.774 | 0.724 | 0.811 | 0.678 | 0.766 | ||
| I5 | 0.736 | 0.662 | 0.693 | 0.751 | 0.640 | 0.849 | 0.673 | 0.757 | 0.614 | ||
| I6 | 0.612 | 0.57 | 0.539 | 0.709 | 0.667 | 0.450 | 0.722 | 0.651 | omitted | omitted | |
| I7 | 0.45 | 0.61 | 0.704 | 0.668 | 0.748 | 0.671 | 0.733 | 0.706 | 0.786 | 0.838 | |
| I8 | 0.505 | 0.65 | 0.624 | 0.588 | 0.490 | omitted | omitted | 0.587 | 0.721 | omitted | |
| I9 | 0.72 | omitted | omitted | omitted | omitted | 0.722 | 0.686 | 0.575 | omitted | ||
| variance explained | 66.48% | 53.94% | 56.41% | 50.52% | 50.93% | 61.00% | 50.82% | 54.98% | 56.07% | ||
| KMO | 0.696 | 0.835 | 0.848 | 0.801 | 0.807 | 0.845 | 0.846 | 0.831 | 0.802 | ||
| Bartlett’s test | 132.32 (p < 0.001) | 212.3 (p < 0.001) | 226.12 (p < 0.001) | 193.36 (p < 0.001) | 135.32 (p < 0.001) | 244.3 (p < 0.001) | 212.75 (p < 0.001) | 215.81 (p < 0.001) | 135.64 (p < 0.001) | ||
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
MDPI and ACS Style
Kostelić, K. Game Awareness: A Questionnaire. Games 2021, 12, 90. https://doi.org/10.3390/g12040090
AMA Style
Kostelić K. Game Awareness: A Questionnaire. Games. 2021; 12(4):90. https://doi.org/10.3390/g12040090
Chicago/Turabian StyleKostelić, Katarina. 2021. "Game Awareness: A Questionnaire" Games 12, no. 4: 90. https://doi.org/10.3390/g12040090
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.