Keywords

1 Introduction

Multitasking interaction with multiple smart devices has become a prevalent phenomenon in the era of ubiquitous computing. The development of various smart devices makes it possible to sequentially or simultaneously interact with multiple smart devices. A survey by Hollywood Reporter showed that 79 % of the respondents always or sometimes visited Facebook while watching TV and 41 % tweeted about the show they were watching and three quarters said that they posted about TV while watching live shows [1]. American teenagers from 8 to 18 years old spent 29 % of the media use time using two or more media concurrently [2]. Besides for entertainment purposes, multitasking is prevalent in working contexts. It is shown that people’s attention duration on the computer screen is only 1.25 min on average [3], meaning that people are easy to switch their attention away from their primary task and attend to interruptions.

Although multitasking interaction with multiple devices serves various needs of multitaskers [4, 5], evidence suggests that multitasking has negative impacts. Multitasking happens accompanied by attention switching between primary works and interruptions. As people resume working from irrelevant interruptions from a different device, they need to exert effort to reorient to the original context. This extra cognitive workload generated in this process hinders individuals’ motivation, ability and opportunity to process the primary media content [6]. As people’s limited attention resources are strained in this process, their performance is degraded [7]. In addition, multitasking causes high levels of stress [8], described as “a complex and often stressful media experience” [9].

Due to the potential harm to multitaskers, an understanding of how people’s attention gets distracted and how the negative experience is produced is necessary for helping multitaskers better interact with multiple devices and manage multiple tasks. In previous studies concerning multitasking, physiological data collected by bio-sensors have been used to reflect multitaskers’ stress levels [10] and experience sampling has been adopted to measure accompanied mood during the multitasking process [11, 12]. These shadowing methods are suitable to monitor the trend of stress or mood for a period of time in field studies. However, we are not aware of any psychological measurement, i.e. an instrument, to directly measure the negative experience in multitasking contexts. Therefore, this study aims at developing an instrument consisting of multiple key constructs to measure negative experience of multitasking interaction. The instrument provides a method to gain insight on how people behave and feel in multitasking contexts and it is beneficial for designing appropriate information infrastructure to support people’s multitasking behaviors.

2 Literature Review

One study summarizes the negative experience of media multitasking from four dimensions: inefficiency, enslavement, disengagement and chaos [9]. Inefficiency, caused by shifting of attention from one stimulus to another, reflects on the deterioration of task performance, which can be defined and measured objectively in different contexts. Enslavement refers to addictive habits, consequences, feelings and corresponding attitudes of a multitasker from the long-term perspective. As we aimed at developing an instrument to measure the extemporaneous negative experience on the spot in a multitasking setting, we did not take the two dimensions into consideration in this study. The current study merely focused on disengagement and chaos. In the following section, we first review previous studies of disengagement and chaos.

2.1 Disengagement

In multitasking contexts, multiple stimuli from different devices compete for individuals’ limited attention resources. Multitaskers become “disengaged and disconnected from attending to any particular media task” [9]. When people use computers and televisions concurrently, they attend to computers primarily compared to televisions, and they switch between such two media with a frequency they are not even aware of [13].

Disengagement is defined as “when participants made an internal decision to stop the activity, or when factors in the participants’ external environment caused them to cease” [14]. Its opposite facet, engagement, is characterized by a series of attributes including challenge, positive affect, endurability, aesthetic and sensory appeal, attention, feedback, variety/novelty, interactivity, and perceived user control [14]. A similar concept with engagement, flow experience, is an “optimal experience” when an individual completely concentrates in an activity [15]. The flow experience is the combination of feeling fully challenged and skillful [16]. An operational decomposition of the flow experience in multitasking contexts includes sense of control, focused attention, curiosity, intrinsic interest and interactivity [17]. Another concept, which has delicate difference with flow experience, is cognitive absorption. It refers to total immersion into an activity, with deep enjoyment, a feeling of control, curiosity and not realizing the passing of time [18]. Engagement, flow experience and cognitive absorption all emphasize individuals’ fully concentrated state, the opposite of which describes disengagement when processing content from multiple media.

2.2 Chaos

Chaos is defined as “an experience of disorder and upheaval” [9] accompanied by multitasking activities. Chaos is a complicated concept consisting of various connotations. The first one is negative emotions caused by interruptions. As found in previous studies, frequent interruptions cause increasing working speed and more stress, no matter in laboratory settings [8] or in real life [10]. Besides, frustration [8] and sense of guilty [9] are often generated in the process of multitasking.

The other connotation refers to the inability because of mental overload. Mental workload in multitasking contexts is higher than in single-task contexts because people not only need to process content on each device, but also exert extra efforts to manage and coordinate different tasks [19]. It can increase the inability to process content effectively in multitasking contexts and make the interactivity between the individual and multiple devices complicated [17]. For example, when people interact with multiple devices, they need to navigate tasks on each single device and switch between multiple devices. Hence, people may feel the sense of disorientation and process content less effectively. Besides, stimuli from multiple devices can cause interruptions to multitaskers, creating an “attentional residue” [20], which means the residual cognitive resources for a prior task when the individual has begun to work on the subsequent task. Thus, multitaskers have to expend extra cognitive effort to reorient back to the original task and the overall cognitive workload in the multitasking process is increased [21].

No matter from emotional perspective or from the cognitive perspective, all the studies consistently indicate that multitasking interaction with multiple devices creates is associated with higher levels of stress and lower levels of positive mood, which is concluded here as “chaos”.

3 Method

Based on previous studies, we summarized variables regarding disengagement and chaos in multitasking contexts and adapted them into questions for composition of a questionnaire. The questionnaire was surveyed online for data collection and then for factor analysis so as to extract a multi-factor model to develop the instrument.

3.1 Questionnaire Design

All the questions were 5-point Likert scales. They were translated into Chinese before being included in the questionnaire. The translations and wordings were checked and modified by one human factors engineering PhD student and one research assistant on psychology. The ambiguity issues were cleared up as well. The questionnaire consisted of two parts. The first part was a general paragraph describing multitasking interaction with multiple smart devices followed by three concrete scenarios. It was emphasized that multitasking interaction in real life had a much broader spectrum beyond the three examples. So we reminded that the quantity and type of devices in multitasking contexts were not limited to the three examples. The second part contained 32 questions with 16 questions regarding disengagement and 16 regarding chaos. They were adapted from the existing literature with syntactical structure and context changes as the main modifications. For instance, an item measuring the flow experience is “I had a strong sense of what I wanted to do” [21]. In the questionnaire, we embodied it in the multitasking scenario as “I have a strong sense of what I want to do while engaging in such multitasking activities”. The questions were adapted from studies by [9, 17, 18, 22, 23]. All the items can be found in Table 1, with “ENGX” or “CHAX” as the code. “X” stands for the sequence number of its appearance in the questionnaire, and “ENG” and “CHA” stand for disengagement and chaos respectively.

Table 1. Rotated component matrix of EFA about negative experience of multitasking interaction with multiple smart devices.
Full size table

Six questions were added according to the results of an interview carried out with three college students in China respectively who have rich experience in multitasking. The interview was to make up for the literature’s lack of focusing on multitasking scenarios. The interview lasted for 30 to 40 min. The participants were encouraged to describe their real multitasking experiences and their accompanying feelings during the process. The moderator encouraged them to elaborate their feelings related to multitasking contexts. The interview was transcribed and the extracted items were noted in Table 1 as well.

3.2 Data Collection

The online questionnaires were distributed via social media in China. RMB 5-worth of mobile phone credit was rewarded to each respondent. Ultimately we collected 437 responses, among which 380 copies were valid.

Among the valid copies, 205 of the respondents were males, while 175 were females. 15 respondents were not older than 20 years old, 322 respondents were between 21 and 30 years old, and 43 respondents were older than 30 years old. 20 respondents have an education level of high school or lower, 160 of them have a university/college degree, and 200 have a graduate or higher degree. 192 of the respondents were students.

All the valid responses were randomly divided into two groups with an equal size. The data in the two groups were for an exploratory factor analysis and a confirmatory factor analysis respectively. Before the factor analysis, the two groups were compared in terms of age, gender and education level. The results suggested no significant difference in age (t = .347, p = .729), gender (\( \chi^{2} \) = .011, p = .918) and education level (t = .124, p = .902) between the two groups.

4 Results

4.1 Exploratory Factor Analysis (EFA)

The EFA was conducted to find the structures of the factors with regard to negative experience in multitasking contexts. The results of the Kaiser–Mayer–Olkin (KMO) test and Bartlett’s test of sphericity were .805 and \( \chi^{2} \) = 1278.57 (p < .001), suggesting being suitable for factor analysis. In the factor extracting and screening phase, the following rules were carried out: extracting components with eigenvalues larger than 1 as principal component; deleting items with loadings smaller than .45 on all common factors; deleting factors containing only one item; making the whole model explicit and simple to explain [24–26]. All the procedures were conducted with SPSS v20.

Finally, 19 items were retained and the component matrix was obliquely rotated to acquire a meaningful explanation of the model. The 19 items composed of six factors and explained 66.63 % of the total variance. The six items and their corresponding items were listed in Table 1. They were named: Confusion (CF), Flow experience (FE), Complexity (CP), Time distortion (TD), Situation awareness (SA), Disorientation (DO).

4.2 Confirmatory Factor Analysis (CFA)

The CFA was conducted to test the model’s goodness-of-fit. The CFA was conducted with SPSS Amos v22. The models were improved according to the modification indices provided by Amos and the professional meanings of each factor. Besides that, the correlation coefficients between the following factors were set zero as their correlation relations did not reach the significant level in the output of the full correlated model: (CF, FE), (SA, DO), (CP, SA), (FE, DO), (FE, CP), (DO, TD).

Goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), comparative fit index (CFI), Tacker-Lewis index (TLI) and root mean square error of approximation (RMSEA) were calculated. The results were summarized at the original model row in Table 2.

Table 2. Model’s goodness-of-fit
Full size table

Previous research suggests that a model with GFI greater than .90 and AGFI larger than .80 demonstrates an acceptable fit [27]. It has also been indicated that RMSEA value between .05 and .08 is acceptable. Another study has indicated that >.90, >.90 and <.06 are cutoff values for CFI, TLI and RMSEA, but the criteria for TLI and RMSEA tend to overreject true population models when the sample size is small [28]. The model satisfied the criteria of the AGFI, the CFI and the RMSEA, but not the GFI and the TLI.

4.3 Reliability and Validity

The Cronbach’s alpha coefficients of the six factors were .852 (CF), .835 (FE), .589 (CP), .593 (TD), .546 (SA) and .601 (DO). For constructing a theoretically reliable instrument, .7 is the cutoff threshold for the Cronbach’s alpha coefficients. The results showed that there were four factors in the model failing to meet the requirement.

The discriminant validity was tested on the original model. Any two factors in the model were combined into one factor, forming a restricted model. Then the chi-square value differences between each restricted model and the original model were compared. As shown in Table 3, the five-factor model combing CP and DO did not have a significant difference with the original six-factor model in the chi-square value. That indicated a direction of model improvement.

Table 3. Chi-square value changes between different models
Full size table

4.4 Model Improvement

Inspired by the results of the discriminant validity, we merged CP and DO, forming a new dimension-reduced model. The Cronbach’s alpha coefficient of the new factor was .679, which showed improved internal consistency.

The goodness-of-fit is shown at the modified model row in Table 2. It can be seen that the modified model showed improved goodness-of-fit compared to the original model.

5 Discussion

5.1 Explanation of Factor Connotations

Via the EFA and the CFA, we constructed a five-factor model measuring negative experience of multitasking interaction with multiple smart devices. The five factors include: Confusion (CF), Flow experience (FE), Complexity and disorientation (CD), Time distortion (TD), Situation awareness (SA).

‘Confusion’ describes the vacant and upheaval feelings in the engagement of multitasking. Due to the mutual interruptions of multiple tasks and unfocused attention, multitaskers tend to lack a clear goal and corresponding strategies and tactics. Thus, they suffer a divided dedication and feel guilty and frustrated during the involvement. The second factor, ‘Flow experience’, describes the extent to which multitaskers show interest, happiness and curiosity in the multiple tasks they are involved in. The third factor, ‘Complexity and disorientation’, is the sum of aforementioned Complexity and Disorientation. It contains items reflecting the task complexity in multitasking contexts, especially the disorientation brought by navigating between multiple devices. Multitaskers not only need to cope with navigation tasks within one device, but also need to manage the coordination between multiple devices. The fourth factor, ‘Time distortion’, describes the phenomenon of losing track of time in multitasking contexts. Time distortion is usually accompanied by flow experience and it happens on game players very frequently [29]. The last factor, ‘Situation awareness’, refers to the perception, understanding and control of what happens in a multitasking context. The five factors all describe the extemporaneous experience on the spot in a multitasking setting.

5.2 Usage of Instrument

This instrument can provide multitaskers with a useful tool to self-exam their states in multitasking contexts. It can benefit the design of smart device and information technology service provision. It can serve as a series of design guidelines for the design of products or services targeting at collaborative working with multiple devices. Besides, it can be used as an evaluation tool for corresponding products or services, especially for those products aiming at providing integrated functions across multiple platforms. Last, this instrument can be used to evaluate the effect of intervention on multitaskers to manage their distracting behaviors in a multitasking context.

5.3 Limitations and Future Work

The current sample size is not large enough and has a limited representativeness because of collecting data only via social media. We cannot arbitrarily regard social media users as multitaskers in real life. Besides, the indices adopted in this study do not reach the realistic standard for constructing a mature psychometric instrument. The items in each factor should be selected based on a more systematic and comprehensive literature summarization. More thorough calculations of various psychometric indices should be performed. Sampling in a broader range of population should be adopted in the future research.

6 Conclusion

In this study we developed an instrument to measure disengaged and chaotic experience of multitasking interaction with multiple smart devices. Via exploratory and confirmatory factor analysis, we constructed a model with five factors with an acceptable goodness-of-fit. The five factors were: Confusion (CF), Flow experience (FE), Complexity and disorientation (CD), Time distortion (TD), Situation awareness (SA). This instrument provides a method to gain insight on how people behave and feel in multitasking contexts and it is beneficial for designing and evaluating information infrastructure to support people’s multitasking behaviors.