Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

Effects of Simultaneous and Sequential Work Structures

on Distributed Collaborative Interdependent Tasks
Paul André, Robert E. Kraut, Aniket Kittur
Carnegie Mellon University
5000 Forbes Avenue, Pittsburgh, PA
{pandre,robert.kraut,nkittur}@cs.cmu.edu
ABSTRACT or tasks. Rather, the optimal structure or coordination
Distributed online groups have great potential for generat- method varies according to factors such as group size and
ing interdependent and complex products like encyclopedia task uncertainty. Routinized assembly-line work is at one
articles or product design. However, coordinating multiple extreme of a task-uncertainty spectrum while complex, cre-
group members to work together effectively while minimiz- ative tasks are at the other. In many creative tasks, the work
ing process losses remains an open challenge. We conduct- process does not follow a rigid sequence and is unable to be
ed an experiment comparing the effectiveness of two coor- defined in advance, the task is interdependent where work
dination strategies (simultaneous vs. sequential work) on a done by one person has unpredictable effects on what a
complex creative task as the number of group members coworker will need to do next, and the output cannot be
increased. Our results indicate that, contrary to prior work, defined in advance.
a sequential work structure was more effective than a sim-
ultaneous work structure as the size of the group increased. Prior research indicates when people are collaborating on
A mediation analysis suggests that social processes such as uncertain and interdependent tasks, they require ‘organic
territoriality partially accounts for these results. A follow up coordination’ or coordination by ‘mutual adjustment.’ That
experiment giving workers specific roles mitigated the det- is, they need rich, fast, responsive communication to help
rimental effects of the simultaneous work structure. These them understand the current state of a task and others’ ac-
results have implications for small group theory and tivities, so that they can adapt their own work in response
crowdsourcing research. [13]. Thus, groups are most successful and better able to
take advantage of a larger workforce—e.g., the workers’
Author Keywords effort, diverse talents and perspectives, cognitive stimula-
group size; coordination; interdependence; small groups. tion, and increased error-checking [23, 28, 29]—when they
ACM Classification Keywords work simultaneously [40, 13].
H.5.m. Information interfaces and presentation (e.g., HCI). On the other hand, simultaneous work can lead to process
INTRODUCTION losses from production blocking, social influence, and so-
Distributed groups routinely collaborate on complex, crea- cial loafing [23, 35]. Even for relatively independent tasks,
tive tasks like designing products, writing encyclopedia such as brainstorming, the presence of more people can
articles, and developing software. The number of people to reduce each person’s quality and productivity, undermining
devote to these creative tasks and the way to organize their the benefit of adding more people [11, 23]. As complexity,
effort are crucial questions for any distributed work group. uncertainty and interdependence of the work increases,
Aiming for optimal output and the best use of each worker, communication and coordination difficulties can increase
factors such as group size and coordination mechanisms even more, undermining the benefits of extra personnel [5,
have been studied both experimentally [20, 23, 29] and in 39, 40].
organizational settings [14, 42]. Because adding more workers to a project brings both bene-
Structural contingency theory [13] suggests there is no sin- fits and challenges, understanding how to best structure
gle effective communication structure for all organizations interdependent tasks is an important and open question
[26]. Typically, researchers have examined the impact of
group size by constructing groups of people simultaneously
Permission to make digital or hard copies of all or part of this work for working on a task [23, 25, 29]. Yet in many cases those
personal or classroom use is granted without fee provided that copies are workers could have potentially performed their tasks se-
not made or distributed for profit or commercial advantage and that copies
quentially, in an iterative process. Research in design has
bear this notice and the full citation on the first page. Copyrights for com-
ponents of this work owned by others than ACM must be honored. Ab- largely focused on the benefits an individual designer re-
stracting with credit is permitted. To copy otherwise, or republish, to post ceives from iteration, showing that the refinement through
on servers or to redistribute to lists, requires prior specific permission iteration improves the design artifact [6, 10]. However, iter-
and/or a fee. Request permissions from permissions@acm.org. ation on a single design also runs the risk of the designer
CHI 2014, April 26 - May 01 2014, Toronto, ON, Canada
Copyright 2014 ACM 978-1-4503-2473-1/14/04…$15.00. persevering, ignoring alternatives [10], and achieving local
http://dx.doi.org/10.1145/2556288.2557158 rather than global optima [6]. Recent crowdsourcing re-

139
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

search has also shown benefits from iterative work process- ways that people fail to optimally combine their work prod-
es in independent tasks, e.g., handwriting recognition [30], ucts when working in groups, such as production blocking
proof-reading [4], and even chair design [44]. However, the when individuals cannot talk at the same time, and thought
benefits and drawbacks of sequential work processes on derailment, when group discussion interferes with one’s
more interdependent tasks remains unknown. ability to start or continue a train of thought; for reviews see
[20, 23, 29, 35].
This paper investigates the benefits and drawbacks of se-
quential vs. simultaneous work structures on interdependent Our research examines how to maximize the benefits of
tasks as the number of workers increases. We explore these additional workers, while minimizing the process losses.
tradeoffs using a creative, interdependent task in the context Two factors moderate these processes: task type and coor-
of a microtask market, Amazon’s Mechanical Turk. Our dination mechanism.
results aim to contribute to theories of small group research
Task Type
as well as the more practical understanding of how to use Performance of groups and individuals differs depending
crowdsourcing for creative and complex tasks [25, 27]. upon type of task. In tasks with objective answers, where it
Specifically, we make the following contributions: is easier to combine individual contributions, larger groups
• We believe this paper is the first to explore tradeoffs in do better than smaller ones [23], for example, in numeric
group size and coordination method for crowd workers, estimation tasks the central limit theory as implemented via
filling a need for research on the size of virtual teams “wisdom of the crowd” suggest more estimators lead to
[31], as well as extending crowd work to rigorously ex- more accurate estimates [41]. In “Eureka” tasks, where eve-
amine interdependent tasks. ryone will recognize a correct answer once it is offered,
larger groups are more likely to contain an individual who
• Our findings show that groups perform uncertain, in- knows the answer. However, for problem-solving tasks,
terdependent tasks better by working sequentially ra- where the answer is more difficult to derive, Laughlin [28]
ther than simultaneously, contrary to prior proposals by demonstrated that groups of two and three workers per-
structural contingency theories [13]. formed better than individuals, as well as larger groups,
• Through mediation analyses and a follow-up experi- potentially because of the process losses previously men-
ment we show analytically and experimentally the role tioned.
of social processes (i.e., territoriality) and cognitive Recent examples in crowdsourcing draw on multiple task
processes (shared mental model) in mitigating the aspects. For example: iterative chair design – a creative
drawbacks of the simultaneous condition. production task, though one that is also judged on practical
RELATED WORK considerations [44]; trip planning – a task that both requires
A recent Science article noted, “if research in psychology generation of ideas and plans yet adherence to global con-
had a Dr. Jekyll and Mr. Hyde Award, it would go to … the straints [34]; and collaborative poetry translation – a crea-
group as a decision-making instrument” [19], emphasizing tive intellectual task that requires negotiation between
the nuanced and often contradictory findings of group per- workers [25]. While these projects have been successful
formance. with groups, they have not focused on the effect of group
size or coordination method. Additionally, there has been
Group Size and Process Losses and Gains little work examining group size in virtual teams [31].
Groups perform better than independent individuals on a
range of tasks, though often fail to meet “a reasonable po- Coordination Mechanism
tential productivity baseline” [23] such as the group’s most Task performance is further affected by coordination mech-
capable member [20, 35, 29]. anism. A group of workers can be utilized in different ways,
and their interactions defined by their environment. Coordi-
Adding members to work groups can offer a variety of pro- nation theorists have proposed that different coordination
cess gains: additional workers can offer new knowledge, styles—pooled coordination, where the group product is
cognitive stimulation and error detection [11, 20, 23, 28, simply the aggregation of individually performed tasks,
29], although the extent of the benefits depends on the task sequential coordination, where one worker builds on anoth-
and environment. er’s output, and team coordination, which requires interac-
Despite the potential benefits of group work for complex, tion among members—are appropriate respectively for
interdependent and uncertain tasks, motivation and coordi- work with increasing degrees of interdependence [39, 40].
nation problems can cause groups to fail, or at least fail to Thus, groups working on uncertain, interdependent work,
live up to their potential, because of social process losses— such as writing a creative document, should be best able to
motivational losses and coordination problems in combin- coordinate by working synchronously, in the ‘team coordi-
ing contributions. Motivation losses include social loafing, nation’ style described above. Synchronous work supports
sucker effects and evaluation apprehension that cause peo- the mutual adjustment (seeing each others work and adjust-
ple to work less hard in groups than when working by
themselves. Coordination losses represent the variety of

140
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

ing), ad hoc arrangements, and rich, fast communication writing also combines a number of attractive task elements
necessary for this type of task [40, 13]. seen in related work: production—creative writing task
requiring generation of ideas and a final product; discus-
However, task uncertainty changes the relationship between
sion—highly interdependent and may require group negoti-
group size and performance. Synchronous work imposes
ation; and problem-solving—though a creative task, there
high coordination costs, and adding workers to complex
are constraints such as specific rhyme and rhythm. Thus,
and interdependent tasks may increase process losses be-
coordination is likely desirable.
yond any benefit the extra workers bring [28, 39, 40].
Study Design
In this work, we argue that sequential work may mitigate We conducted a 2 X 2 experiment, in which groups worked
those process losses, while maintaining the benefit of addi- using a simultaneous or sequential workflow. We treat in-
tional workers. There is little research on sequential work dividuals (1 worker) as a baseline condition, and compare
and interdependent tasks in the small group literature. We to groups of size 2 or 3 workers. Simultaneous groups col-
posit that sequential interaction can allow groups to coordi- laboratively worked online with 2 or 3 workers at once,
nate the interdependent tasks without having to talk to one while in sequential work we add 1 worker in each iteration,
another; that the work artifact contains all the information i.e., 1 worker, 1+1 workers, and 1+1+1 workers. Groups
needed for mutual adjustment. Unlike prior sequential co- wrote and communicated using a single etherpad,1 an online
ordination for assembly line work, the task and artifact are collaborative editing space (see Figure 1).
not divisible, the output is novel, and changes by one per-
son have unpredictable effects on the next person’s task. At To analyze the results, we test two main effects: effect of
a large-scale, this may be similar to Wikipedia, where condition (simultaneous or sequential), and number of
groups of people are seemingly able to coordinate sequen- workers, along with an interaction effect: differential effects
tial editing through the artifact (although with the addition on limerick quality, as a 2 X 2 ANOVA.
of Talk pages), though such interdependent coordination Hypothesis
remains an open question [26]. We conduct controlled em- As we saw in previous work, the potential benefits and
pirical experiments to test these arguments more rigorously. losses from groups depend on the task and environment.
To investigate optimal ways to deploy workers for uncer- Broadly, limericks are a creative, integrated artifact requir-
tain and integrated production work, we propose extending ing an interdependent process. Prior work suggests simulta-
prior work by studying group size in (a) virtual teams, (b) neous work is likely to be most effective, and most able to
with a creative interdependent task, and (c) comparing sim- take advantage of extra workers and diverse perspectives,
ultaneous and sequential coordination methods. We then cognitive stimulation, and error-checking [23, 29]. Howev-
compare the two methods to identify processes that may aid er, adding workers also increases process loss due to coor-
or hinder work in different coordination methods. dination and communication issues [39]. Sequential work,
where each worker works alone, does not incur those pro-
EXPERIMENT 1: GROUP SIZE IN SIMULTANEOUS & cess losses, but on the other hand may incur other types of
SEQUENTIAL WORK
process loss, e.g., not knowing a predecessor’s intention, or
Drawing from small group research, we test groups of size
being left with a difficult story or rhyme.
1, 2, or 3 workers simultaneously or sequentially working
on a task. We hypothesize that iterative work will remove the poten-
tial process loss from simultaneous work—no overhead of
Task
We are interested in complex, creative, collaborative work communication or attempting to coordinate edits to an in-
(i.e., work that has a high degree of uncertainty). We re- terdependent task, but retain some of the process gains—
quired a task that: full effort and attention of each individual, and multiple
eyes correcting and refining the limerick.
• produces a single artifact, interdependent enough that
producing it requires coordination from a group of
workers;
• produces a work product whose quality can be as-
sessed;
• leaves traces, so that researchers can analyze the de-
tailed coordination processes associated with quality.
We chose group limerick writing as the creative task. The Figure 1. The Etherpad collaborative text editor. A shared
editing space with colored contributions on the left, and a
creative and interdependent nature of writing a limerick
chat area on the right.
means the work is highly uncertain: each limerick is unique,
even though it conforms to a specific structure. Its compo-
nents are highly interdependent, e.g., both the content and
1
rhyme in line 5 depend upon line 1. The task of limerick http://etherpad.org

141
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

Procedure We are able to use data from the 1-worker condition in the
From the point of view of a worker, the flow of work was simultaneous condition as our 1-worker condition in the
thus: click on the task in Mechanical Turk, be taken to a sequential condition, as the interface and instructions are
‘waiting room’, perform the task, then take a survey. Partic- the same. Because later stages of the sequential condition
ipants could leave at any time. Workers were filtered to US depend on having the earlier stages complete, this part of
only, greater than 95% approval rate, and could only take the experiment was run at a later time, and subjects were
part in one task. thus not randomly assigned to condition. We revisit this in
the Limitations section.
In the simultaneous condition, workers could sign up for the
task labeled “Help write a limerick (a short rhyme)”, and be Measures
taken to a waiting room (similar to previous real-time Measuring the quality of a creative artifact like a limerick is
crowdsourcing experiments [3]). Payment was $0.70 per challenging, but Amabile [1] has suggested a product or
limerick, a wage in line with MTurk hourly rates. Workers idea is creative to the extent that expert observers agree it is
could leave this page open in the background, and take oth- creative, when making judgments independently, relative to
er tasks, until two or three people were ready to take the the corpus, and on other dimensions such as technical quali-
task. The workers would then be alerted and taken to a page ty.
with instructions (described below), and shown an etherpad
We calculate an outcome measure defined as the average of
to write the limerick.2 No time limit was imposed. Workers
two scores: overall quality, and technical quality (is the
who waited 10 minutes without another worker joining
limerick the right number of lines, the correct rhyme struc-
made up our pool of one worker contributions. The instruc-
ture and meter). We initially split assessment into four crite-
tions of the task explained the collaborative nature of the
ria to capture both objective and subjective aspects (tech-
task (if more than one person), the assessment criteria, the
nical, story coherence, creativity, and overall quality) but
theme of the limerick, and details about rhyme and meter of
initial testing found all to be correlated and for the remain-
a limerick.
der of the experiment we use the two clearest features.
In the sequential condition, the task title and payment were While codebooks or extensive training can be important
changed to reflect the nature and length of the task: “Im- when criteria are unclear, Hak & Bernts [17] warn of code-
prove/edit a limerick (short rhyme)”, with payment of books merely codifying biases, and Hennessey & Amabile
$0.50. Workers could take the task only once. The task be- suggest independent ‘creative’ raters should not confer or
gan straight away. The same instructions and description of be trained prior to judgment [18]. Other techniques such as
the original task was given as above, with the addition of pairwise comparison are valuable but prohibitive in scale.
editing instructions: “We previously asked turkers to write a
Three raters independently rated all limericks blind to con-
short rhyme—a limerick—based on the instructions and
dition; two were undergraduate or master’s students in crea-
assessment criteria below. We are now asking you to edit
tive writing (with additional experience in editing and judg-
and improve one of those limericks. You should improve
ing creative pieces), along with one of this paper’s authors.
the limerick as much as is needed to fulfill the criteria be-
Quality and technical dimensions were rated on a 1—7 lik-
low. (There is no limit to the editing you can do).”
ert scale. The technical dimension was intended to assess
Simultaneous and sequential groups used the same collabo- objective elements of the limerick: “Consider the number of
rative etherpad interface, saw similar instructions, and the lines, the rhyme scheme, and meter.” The quality dimension
same task artifacts—the limerick itself. While groups had was intended to assess the more subjective elements of the
the same interface, the conditions could have led the groups limerick: “A holistic rating, consider the story, the coher-
to use the interface in different ways, but we saw no evi- ence, the creativity or interestingness, more generally: do
dence of this. For example, we initially manipulated wheth- you like it?” We provide examples to illustrate these
er simultaneous workers could use a chat box or not, but no measures in the Results section. We also capture a number
difference in results was found.2 In the sequential condition, of process measures to understand the coordination process:
workers were not aware that others would edit the limerick total time spent, characters written / deleted, distribution of
after them, and thus were not more aware of the ‘process’ edits, and number of author switches.
of the experiment [16], and did not leave, e.g., comments or
We calculated Cronbach’s alpha for the two dimensions, to
to-dos for later workers.
test whether each dimension was independent or whether
we might combine the items to create a more reliable scale.
The resulting alpha was 0.87, i.e., each dimension measures
2
To further investigate the factors affecting interdependent work, we orig- a similar underlying concept. For this reason, and to simpli-
inally also manipulated whether the group could directly communicate fy analysis, we combine (average) the ratings into a single
using a chat box. While some artefacts are created without explicit real-
rating. Agreement between the three raters (Cronbach’s
time coordination (e.g., Wikipedia, shared codebases), prior work suggests
direct communication is key to interdependent work. However, the results alpha) was 0.84, indicating high reliability.
show no difference between workers allowed to directly chat or not, and
we devote the rest of the paper to group size and subsequent experiments.

142
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

Results & Analysis Number of Simultaneous Sequential

Table 1 shows the number of groups and the combined Workers
Mean SE n Mean SE n
quality score for each experimental condition. Overall 716 1 4.05 .10 79 4.05 .10 79
participants wrote a total of 431 limericks working in a total
2 4.03 .09 107 4.48 .07 78
of 79 1-person groups, 185 2-person groups and 167 3-
person groups. Note, although Table 1 shows 79 1-person 3 4.15 .11 89 4.59 .08 78
groups as controls in both the simultaneous and sequential Combined 4.08 .06 275 4.37 .05 235
conditions, these are actually the same groups; groups of
one can be neither simultaneous nor sequential. Table 1. Overall score of final limerick. Simultaneous work-
ers (group sizes of 1, 2, 3 workers), and Sequential work (it-
The limericks below exemplify excellent, average, and poor erations of 1, 1+1, 1+1+1 workers).
output.
There once was a squirrel in a tree In contrast, sequential groups of 2 and 3 workers (M=4.53,
who wanted to dance like a bee SD=.11) produced higher quality limericks than individual
he bumbled around workers (M=4.05, SD=.93). That is, the improvement in
but fell to the ground
work quality that resulted from working in groups of two or
and said "dear, this task's not for me.”
(Overall: 7; Technical: 6; Combined: 6.5) three rather than working individually was reliably greater
when the work was organized sequentially rather than sim-
There once was a wolf in the forest ultaneously, t(233) = 4.53, p<.0001.
who howled at the moon like a chorus The prior literature, which suggests that simultaneous coor-
lonesome seemed he
dination is necessary for interdependent, creative work [39,
to never taste tea
his kidneys became wasted and porous 40], would have led to a prediction that the greatest benefit
(Overall: 3; Technical: 5; Combined: 4) from additional workers would have occurred when work
was organized simultaneously. However, this expectation
A squirrel lives, lives, lives, gives out side was disconfirmed in this experiment. Rather, additional
And talks to a mouse, moose, mole, downtown. workers in the sequential, iterative condition led to an in-
the mouse lives in a house crease in quality.
Where he likes to eat, sleep, but be neat
he saw a cat who looked like a rat Although we designed the environment to minimize some
But he still wants to play all day! known process losses, we may not have been entirely suc-
(Overall: 2; Technical: 1; Combined; 1.5) cessful. We used etherpad, an editor for simultaneous work,
Effect of Adding Workers in Simultaneous vs Sequential which should eliminate one form of production blocking,
We measure the relationship between coordination method since workers were able to write and edit at the same time.
(i.e., simultaneous vs sequential organization), number of However, the presence of others may have led to a virtual
workers, and an interaction effect: differential effects on production blocking, in which workers felt reluctant to
limerick quality. write and especially edit a partner’s work when the partner
was working. Workers were anonymous (or pseudonymous,
We performed a 2 X 2 ANOVA, removing the individual referenced by a nickname of their choosing), which should
(1-worker) condition as a baseline for comparison. The minimize evaluation apprehension; though their contribu-
ANOVA showed a significant effect of coordination meth- tions were colored, increasing the identifiability of their
od (simultaneous or sequential), F(1,352) = 23.15, p<.0001; edits, which should have minimized the social loafing
no significant difference between groups with two or three common in groups [23]. However, we may not have fully
workers, F(1,352) = 1.53, p=.217; and the interaction be- eliminated these problems. The sequential process, on the
tween coordination and number of workers was not signifi- other hand, seemed to be able to take advantage of the extra
cant, F(1,352) = 0.00, p=.998. This suggests that the mean workers, allowing process gains with the extra eyes and
quality for simultaneous and sequential conditions differ, effort. Each iterative worker worked alone, and so there
and the effect is similar in groups of two or three workers. was no process loss from collaboration or communication.
Larger groups (or groups with more steps) don’t differ from
smaller groups. ANALYSIS: COMPARISON OF SEQUENTIAL AND
SIMULTANEOUS
We next contrast the 1-worker conditions to the conditions Our results showed that additional workers improved pro-
wit more than 1-worker. Note we use the same set of sub- duction quality when workers wrote sequentially rather than
jects in the 1-worker condition when comparing multi- simultaneously. However, these results do not explain why
person groups in both simultaneous and sequential condi- sequential workers were able to increase the limerick quali-
tions. Simultaneous groups of 2 and 3 workers (M=4.08, ty while simultaneous workers were not, given that the
SD=0.99) did not produce higher quality limericks than same number of people worked on the limerick.
individual workers (M=4.05, SD=.93), t(273) = .22, p=.827.

143
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

Above we speculated that workers in the simultaneous con- Explaining the Differential Effects of Group Size
dition suffered more from some specific types of process The results of Experiment 1 suggest that groups of two or
losses than those in the sequential condition. To investigate three outperform individuals when working sequentially,
why additional workers improved quality with a sequential but not simultaneously. An alternative analysis of whether
organization of the work process, but not a simultaneous additional workers improved quality, that will allow us to
one, we looked at factors that may mediate the differential simply perform a mediation analysis, is a regression analy-
benefits from adding workers. Specifically, we conducted a sis with coordination method (sequential vs simultaneous),
mediation analysis [2] to test whether territoriality (a form number of workers (one to three), and an interaction effect:
of production blocking) and/or social loafing could account differential effects on limerick quality. The analysis shows
for why adding workers improved quality in the sequential that the experimental manipulations influenced quality: R2
condition but not the simultaneous one. = .06, F(3, 506) = 9.83; p<.001. Neither coordination meth-
od (b=.08, p=.53) nor number of workers (b=.05, p=.49)
Possible Mediators
alone significantly predicted quality, but the interaction did
1. Territoriality. Previous research shows that production
(b=.22, p=.02). Interpreting the coefficients shows that each
blocking – unwillingness or inability to contribute when
additional worker added .22 points to the score in the se-
working with others – can explain why adding workers
quential condition, but only .05 in the simultaneous condi-
does not improve work performance in real groups com-
tion. Similar to the analysis of variance combined with t-
pared to nominal groups where workers do not interact [12].
tests in Experiment 1 results, this suggests that additional
Although the etherpad editor eliminated physical produc-
workers outperformed a single worker in the sequential
tion blocking by allowing workers to simultaneously edit,
condition, but not the simultaneous one.
the presence of co-workers in the simultaneous condition
may have led to social production blocking. In the simulta- In an attempt to explain these effects, we performed two
neous condition, workers may have avoided editing or de- mediation analyses with the variables described earlier—
leting others’ work due to territoriality, in which people are territoriality and social loafing—as mediation variables.
uncomfortable editing another person’s work. Territoriality Territoriality and social loafing are not related to each oth-
has been noted in other collaborative authoring such as er, r = 0.014, indicating suitability for use in mediation. We
Wikipedia [38]. Territoriality could result in missed oppor- use standardized coefficients from now on, to account for
tunities to improve the limerick, failure to fix errors, and a different units in mediation variables. The standardized
lack of creative tension. In contrast, in the sequential condi- coefficient of the interaction (previously 0.22) is 0.18. The
tion, workers could freely edit previous work without fear goal of the mediation analysis is to determine whether add-
of ‘stepping on someone else’s toes’ because the previous ing one of the proposed mediation variables lowers that
worker had departed and would not know about changes to interaction effect. We walk through one analysis, described
their text. graphically in Figure 2.
To test whether territoriality, a form of social production Territoriality. We previously showed that the interaction
blocking, can account for difference in quality, we measure variable (the differential effect of number of workers de-
the amount of deletions to someone else’s work versus pending on coordination method) predicted limerick quali-
one’s own work. (Note: in the sequential case this measure ty; this is the total effect shown in Figure 2. Path a shows
is mostly constant, e.g., in the case of the second worker in how the interaction variable is correlated with the mediator,
the sequential condition, edits are mostly to someone else’s territoriality (beta=.46). Path b shows how the mediator
work. The mediation analysis will allow us to determine if affects the limerick quality score (beta=.12).
this measure can account for output quality in either condi-
The total effect of the interaction variable on quality can be
tion.)
broken into a direct effect and an indirect effect through
2. Social loafing. Simultaneous workers may put forth less territoriality. The direct effect of the interaction variable
effort because they believe other group members will pick accounting for territoriality is the total effect (beta=.18)
up the slack, or they feel their ideas are dispensable [22, minus the effect through the mediator (beta=.06, calculated
35]. However, in the sequential condition, because partici- as product of coefficients in the paths). The effect of inter-
pants worked alone, they would be less susceptible to these action on score drops from a total effect of 0.18 to a direct
sources of social loafing. effect of 0.12 once territoriality is controlled.
To evaluate whether increasing group size reduced effort A Sobel-Goodman test indicates that the mediation effect of
from workers, we measure the edits per person in each con- territoriality is marginally significant, p=.07, with approxi-
dition. It is likely not all edits are equal (some changes mately 32% of the total effect being mediated. In sum, the
might have more of an impact than others), but prior re- evidence weakly suggests that territoriality partially medi-
search (e.g., in Wikipedia [26]) has shown that number of ated the interaction effect of number of workers and coor-
edits correlates highly with other converging measures of dination method on output quality.
effort (e.g., number of characters changed), and so we be-
lieve is a reasonable proxy for effort.

144
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

Condition (two Simultaneous score

workers) Mean SE n
Control 3.70 .24 37
Roles 4.33 .15 40
Table 2. Overall score from Experiment 2: two simultane-
ous workers in control condition, or given assigned roles of
writer and editor.
Figure 2. Mediation analysis for interaction of coordina-
tion method and number of workers. Mediating variable is else’s work. A writer is tasked to create an initial limerick,
territoriality (the amount of deletion of others’ work). while the editor is asked to focus on improving that limer-
Each path is quantified with standardized regression coef- ick. This is similar to the sequential method, but the editor
ficients. The direct effect of interaction on score is calcu- is there for the creation of the limerick, and the writer stays
lated as total effect minus effect through the mediator, to allow back-and-forth once the limerick is written. The
which is calculated as the product of coefficients in the assigning of specific roles may also reduce coordination
paths.
and communication issues [9]. In this experiment, we hold
group size constant (n=2), to focus on the effect of two non-
Social loafing. We perform a parallel analysis to test conflicting roles. Future work may examine the effect of
whether social loafing mediates the interaction effect of roles and group size, as well as multiple workers given
number of workers and coordination method on output complementary or conflicting roles. Given two simultane-
quality. We first test if social loafing is correlated with an ous workers, we hypothesize that assigned roles will in-
increase in workers (in sequential work: beta=.08, p=.28, crease task performance compared to a control condition
and in simultaneous work: beta=-.23, p<.001). This implies similar to prior experiments.
that adding people increases social loafing in simultaneous
Method
substantially more than in the sequential condition. To test A similar procedure to Experiment 1 was followed. On
similar to path a as previous, the interaction variable is cor- MTurk, workers signed up for the task labeled “Help write
related with the mediator, social loafing, beta=.189. Path b a limerick (a short rhyme)” and were taken to a waiting
shows how social loafing affects the limerick quality, be- room. When two workers were ready, they were alerted and
ta=.186. The direct effect of the interaction variable ac- taken to a page with instructions and an etherpad. Workers
counting for social loafing is 0.15. Sobel-Goodman tests
were randomly assigned to a control condition (the same as
indicate this effect is not significant. This suggests that
in Experiment 1), or the role manipulation which added role
while social loafing does affect simultaneous workers more instructions: “You and your partner have different roles: a
than sequential, it does not predict quality and thus cannot writer and an editor. You are the writer. The writer should
account for our interaction. focus on creating new content. Your partner is the editor.
In summary, we proposed two reasons why adding workers The editor should focus on revising the content. These
to sequential work may increase output quality while add- should be happening at the same time.”
ing in simultaneous work does not. The social loafing vari- Experiment 2: Results & Analysis
able implies that while adding workers decreased their indi- Given the high inter-rater reliability in Experiment 1, one
vidual effort, doing so does not affect quality. The territori- rater (blind to condition) rated all limericks. Descriptive
ality variable was able to partially mediate the differential statistics are presented in Table 2. An independent samples
effects of number of workers, demonstrating that increased t-test was conducted to compare score in control and role
edits to others’ work accounted for some quality difference. conditions. There was a significant difference in the scores
EXPERIMENT 2: IMPROVING SIMULTANEOUS WORK for control (M=3.70, SE=.24), and role (M=4.33, SE=.15)
Results from Experiment 1 indicate that additional sequen- conditions; t(75)=-2.23, p=.029. These results suggest that
tial workers outperformed additional simultaneous workers assigning writer and editor roles in simultaneous work is
at a creative interdependent task—limerick writing. Media- able to mitigate process losses, increasing the limerick qual-
tion analysis suggests that a territoriality phenomenon ity above that of a control condition.
whereby workers are reluctant to edit another’s work in the DISCUSSION AND CONCLUSIONS
presence of that person contributed to this difference in Online groups have great potential for creating interde-
quality. pendent and complex work, but coordinating members and
To test this idea, we conducted a follow-up experiment to minimizing process loss is an ongoing challenge [23, 26,
encourage people in the simultaneous condition to edit each 29]. In this paper we examined the effect of group size on
other’s work, by assigning an editor role (as well as a writer two methods of coordination: simultaneous and sequential
role, often used in collaborative writing [32]). Having an work. The results demonstrate that for an interdependent
editor role gives a person explicit license to edit someone creative task like limerick writing, in the context of an

145
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

online environment, adding workers sequentially has more not randomly assigned: the waiting room procedure meant
benefit than adding workers simultaneously. that our 1-person pool generally waited for 10 minutes
(though during this time were able to browse to other pages,
The evidence suggests that this effect occurs because when
complete other tasks) without other workers joining, where-
people work sequentially, the initial person creates, while
as the 2 and 3-person conditions started when a group was
the second does editing. This leads the editor to feel freer to
all present. As a result, subjects in the 1-person conditions
edit the author’s text, i.e., less territorial. This is not hap-
may have been more conscientious than those in the multi-
pening in the simultaneous condition, where workers create
person conditions. Despite their potentially greater consci-
and edit at the same time. These conclusions are supported
entiousness, subjects in the 1-person groups performed
by two types of evidence:
more poorly than those in multi-person groups. Moreover,
1. Mediation analyses suggest that social loafing, a key while workers were allocated non-randomly to this condi-
concept for process loss in groups, does not account for tion, we use the same individual condition for both simulta-
the difference. Although simultaneous group workers neous and sequential conditions, so any pre-existing differ-
did exert less effort (number of edits) than a lone ences should apply to both. All experiments ran for multiple
worker, the mediation analysis suggests this did not ac- days (or even weeks), and some worker effects may average
count for the difference in quality. However, editing out.
others’ work partially accounts for the difference in
Payment differed between conditions ($0.70 in simultane-
quality.
ous, and $0.50 in sequential). Prior work has shown work-
2. A follow-up experiment demonstrates that workers in a ers expect pay to reflect the length of the task [8]. Since
simultaneous condition can work better than a control workers must wait for a period of time before starting in the
condition if they are given writer/editor roles. The ex- simultaneous task, we opted to advertise the task at a higher
plicit assigning of roles lends authoritative weight to payment. Prior work has found that increasing payment
the ability to edit another’s work. attracts more workers faster, but has no consistent effects
on quality of work [e.g., 33]. Future work should further
Our results suggest that one can get the benefits of collabo-
examine the relationship of task pay, uptake, and output
ration with fewer process losses if people work sequential-
quality.
ly. Given the relative lack of research on sequential coordi-
nation and interdependent tasks, these results suggest itera- Since we did not run both experiments concurrently, it is
tion should be further examined. It may be the case that not a fully randomized experiment design. In the sequential
iteration is beneficial under some conditions but not others, condition, since later stages obviously depend on the prior
for example where the artifact does not contain enough in- stages, such a randomization is not possible. Future work
formation or context for a worker to effectively contribute, may investigate how day or time of day affects participant
and in particular other factors of the task and environment pool (prior work has found some seasonality effects with
should be tested in future work. completion rate depending on day of the week [21], but to
our knowledge quality has not been examined) and attempt
Isolating factors to test specific hypotheses, e.g., which el-
to control for quality effects, as well as broaden beyond
ement of creative writing or poetry is beneficial to collabo-
paid marketplaces.
rate on – the ideation, the production, the fitting to con-
straints? – is one promising area for further work. Future We used limericks as an example of the complex, creative
work may also consider ways to optimize workers’ atten- work that has been undertaken in crowdsourcing recently
tion, to focus on poor artifacts or highlight artifacts that (e.g., [7, 24, 25, 27]), and extend beyond the single task
need different types of work, e.g., a rewrite compared to types commonly seen in small group research [29]. Future
light editing. work may focus on particular aspects such as ideation or
editing, as well as extend to different tasks.
Limitations
Territoriality may explain why sequential groups were able Implications
to perform better than simultaneous groups. However, an- We began with a question of optimal resource deployment:
other explanation, that we were unable to test because of how might we get the most benefits from multiple workers
correlation with territoriality, is that of ‘shared mental mod- working on a complex, interdependent task, where it is not
els.’ Subsequent workers in the sequential condition see an clear what the answer is beforehand.
entire limerick, and may have a clearer idea of the direction
Prior work in organizational behavior has focused on small,
of the work before adding their contributions. This is not
synchronous groups, and suggested that for highly interde-
happening in the simultaneous condition, where workers
pendent tasks such as limerick writing, coordination using
create and edit at the same time.
team interdependence is the most valuable [39, 40]. Such
Simultaneous and sequential experiments were not con- coordination allows the fast communication and mutual
ducted at the same time, and so comparisons between them adjustment necessary for such interdependent work. In con-
may introduce a confound. Within condition, workers were trast, while recent crowd work has examined iterative work

146
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

[44, 27, 30], there has been little examination of workers 2. Baron, R. M. & Kenny, D. A. (1986). The moderator--
directly interacting and communicating with each other mediator variable distinction in social psychological re-
synchronously, nor comparison between the two methods. search: Conceptual, strategic, and statistical considera-
tions. J Pers Soc Psychol, 51 (6), 1173.
We believe our paper is the first to explore the tradeoffs
between crowd workers working together simultaneously or 3. Bernstein, M. S., Brandt, J., Miller, R. C., & Karger, D.
iteratively. We focus on limerick writing as an instance of a R. (2011). Crowds in two seconds: Enabling realtime
complex, creative task that has uncertain and interdepend- crowd-powered interfaces. Proc. UIST 2011, 33—42.
ent task aspects. Our findings suggest that, contrary to prior 4. Bernstein, M., Little, G., Miller, R.C., Hartmann, B.,
work, sequential coordination is actually a more beneficial Ackerman, M., Karger, D.R., Crowell, D., & Panovich,
use of multiple workers. Simultaneous work was not able to K. Soylent: A word processor with a crowd inside. Proc.
capture the benefits of additional workers, while iteration UIST 2010. ACM.
was able to do so, and removed communication and coordi-
5. Brooks, F. (1975). The Mythical Man Month: Essays on
nation overheads.
Software Engineering. Addison-Wesley, Mass.
It seems that rather than collaborate on an artifact directly, 6. Buxton, B. 2007. Sketching User Experiences: Getting
sequential workers were able to effectively collaborate the Design Right and the Right Design. Morgan Kauf-
through the artifact. This may be similar to a process such mann.
as Wikipedia that effectively manages to combine collabo-
rative yet sequential editing. This paper has been able to 7. Cheng, J., Kang, L. & Cosley, D. (2013). Storeys – De-
demonstrate and compare in a more rigorous way the poten- signing Collaborative Storytelling Interfaces. Proc. CHI
tial of such coordination, a process that has to date largely 2013 EA (Interactivity).
been ignored in the organizational behavior and small group 8. Chilton, L., Horton, J., Miller, R., Azenkot, S. (2010)
research areas. Task Search in a Human Computation Market. KDD-
HCOMP 2010.
Further, we have demonstrated that territoriality partially
accounts for the difference in quality between simultaneous 9. Cohen, E. G. (1994). Designing groupwork: Strategies
and iterative work. By assigning writer/editor roles to two- for the heterogeneous classroom (2nd ed.). New York:
person simultaneous groups, they were able to outperform Teachers College Press.
control groups. With the desire for crowdsourcing services 10. Cross, N. 2004. Expertise in design: An overview. De-
to create increasingly complex or creative work, these find- sign Studies, 25, 427–441.
ings may have immediate practical implications. With in-
terest in CHI and CSCW specifically in collaborative crowd 11. Dennis, A. R. & Valacich, J. S. (1993). Computer brain-
writing, even very recently, e.g., [7, 24], these results sug- storms: More heads are better than one. J Appl Psychol,
78 (4), 531.
gest possible sources of quality increase, as well as savings
in time and effort (along with traditional factors that should 12. Diehl, M., & Stroebe, W. (1987). Productivity loss in
be further explored, such as coordination method, instruc- brainstorming groups: Toward the solution of a riddle.
tions, and planning stages). Journal of Personality and Social Psychology, 53(3),
497-509.
Finally, these findings suggest that interdependent tasks
requiring team interdependence is an assumption that 13. Donaldson, L. (1999) The normal science of structural
should be challenged and explored more thoroughly, in the contingency theory. Studying organizations: theory and
context of particular tasks and environments. This research method. Sage, pp.51—70.
appears at a time when the very notion of a team and group 14. Gladstein, D. L. (1984). Groups in context: A model of
research paradigm is under discussion by organizational task group effectiveness. Admin Sci Quart, 499—517.
psychologists, as workers and collaboration are increasingly
15. Hackman, J. R. (2012). From causes to conditions in
solely virtually mediated [15, 37, 31]. As teamwork and
group research. J Organ Behav, 33 (3), 428—444.
virtual teams change and grow, understanding how to opti-
mally use workers could have significant impact on organi- 16. Hackman, J. R., Brousseau, K. R., & Weiss, J. A.
zations. (1976). The interaction of task design and group per-
formance strategies in determining group effectiveness.
ACKNOWLEDGMENTS Organizational Behavior and Human Performance,
This work was partially supported by Carnegie Mellon’s 16(2), 350—365.
Center for the Future of Work, Bosch, Google, and NSF
OCI-09-43148, IIS-0968484, IIS-1111124, IIS-1149797. 17. Hak, T. & Bernts, T. (1996). Coder training: Theoretical
training or practical socialization? Qual. Sociology,
REFERENCES 19(2).
1. Amabile, T.M. (1982). Social psychology of creativity:
A consensual assessment technique. J Pers Soc Psychol, 18. Hennessey, B.A., & Amabile, T.T. (1999). Consensual
43, 997—1013. assessment. Encyclopedia of Creativity, 1.

147
Session: Coordination and Collaboration CHI 2014, One of a CHInd, Toronto, ON, Canada

19. Hertwig, R. (2012). Tapping into the Wisdom of the 33. Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Lare-
Crowd—with Confidence. Science, 336(6079), 303-304. do, J., & Vukovic, M. (2011). An Assessment of Intrin-
20. Hill, G. W. (1982). Group versus individual perfor- sic and Extrinsic Motivation on Task Performance in
mance: Are N+1 heads better than one? Psychological Crowdsourcing Markets. Proc. ICWSM 2010.
Bulletin, 91(3), 517. 34. Shepperd, J. A. (1993). Productivity loss in performance
21. Ipeirotis, P. G. (2010). Analyzing the Amazon Mechani- groups: A motivation analysis. Psychol Bull, 113(1), 67.
cal Turk marketplace. XRDS, 17(2), 16—21. 35. Steiner, I. D. (1972). Group process and productivity.
22. Kerr, N. L. & Bruun, S. E. (1983). Dispensability of Academic Press.
member effort and group motivation losses: Free-rider 36. Straus, S. G. & McGrath, J. E. (1994). Does the medium
effects. J Pers Soc Psychol, 44(1), 78. matter? The interaction of task type and technology on
23. Kerr, N. L. & Tindale, R. S. (2004). Group performance group performance and member reactions. J Appl Psy-
and decision making. Annu Rev Psychol, 55, 623—655. chol, 79 (1), 87.
24. Kim, J., Cheng, J. & Bernstein, M. (2014). Exploring 37. Straus, S. G., Parker, A. M., Bruce, J. B., & Dembosky,
Complementary Strengths of Leaders and Crowds in J. W. (2009). The Group Matters: A Review of the Ef-
Creative Collaboration. Proc. CSCW 2014. fects of Group Interaction on Processes and Outcomes
in Analytic Teams. RAND Corporation Working Paper.
25. Kittur, A. (2010). Crowdsourcing, collaboration and
creativity. XRDS, 17(2), 22—26. 38. Thom-Santelli, J., Cosley, D., Gay, G. (2009). What's
Mine is Mine: Territoriality in Collaborative Authoring.
26. Kittur, A., & Kraut, R.E. Harnessing the wisdom of Proc. CHI 2009.
crowds in Wikipedia: Quality through coordination.
Proc. CSCW 2008. ACM. 39. Thompson, J. D. (2003). Organizations in action: Social
science bases of administrative theory. Transaction
27. Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. Pub.
(2011). CrowdForge: crowdsourcing complex work.
Proc. UIST 2011. 40. Van de Ven, A. H., Delbecq, A. L., & Koenig Jr, R.
(1976). Determinants of coordination modes within or-
28. Laughlin, P. R., Hatch, E. C., Silver, J. S., & Boh, L. ganizations. American Sociological Review, 322—338.
(2006). Groups perform better than the best individuals
on letters-to-numbers problems: Effects of group size. J 41. Vul, E. & Pashler, H. (2008). Measuring the Crowd
Pers Soc Psychol, 90 (4), 644. Within Probabilistic Representations Within Individuals.
Psychological Science, 19 (7), 645—647.
29. Levine, J. M. & Moreland, R. L. (1990). Progress in
small group research. Annu Rev Psychol, 41 (1), 585— 42. Wageman, R. (1995). Interdependence and group effec-
634. tiveness. Admin Sci Quart, 145—180.
30. Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. 43. Williams, K., Harkins, S. G., & Latané, B. (1981). Iden-
(2010). TurKit: human computation algorithms on me- tifiability as a deterrant to social loafing: Two cheering
chanical turk. Proc. UIST 2010. experiments. J Pers Soc Psychol, 40 (2), 303.
31. Martins, L. L., Gilson, L. L., & Maynard, M. T. (2004). 44. Yu, L. & Nickerson, J. V. (2011). Cooks or cobblers?:
Virtual teams: What do we know and where do we go crowd creativity through combination. Proc. CHI 2011.
from here? J Manage, 30 (6), 805—835. 45. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., &
32. Posner, I.R. & Baecker, R.M (1993). How People Write Horvitz, E. (2012). Human computation tasks with
Together. In R.M. Baecker (ed.): Readings in Group- global constraints. Proc. CHI 2012.
ware and Computer-Supported Cooperative Work: As-
sisting Human-Human Collaboration. San Mateo, CA:
Morgan Kaufmann, pp. 239–250.

148