Empirical Research For Software Architecture Decis - 2019 - Journal of Systems A

The Journal of Systems and Software 149 (2019) 360–381
Contents lists available at ScienceDirect
The Journal of Systems and Software

journal homepage: www.elsevier.com/locate/jss
Empirical research for software architecture decision making: An

analysis
Maryam Razavian a,∗, Barbara Paech b, Antony Tang c
a
Eindhoven University of Technology, Eindhoven, The Netherlands
b
Heidelberg University, Heidelberg, Germany
c
Swinburne University of Technology, Melbourne, Australia
a r t i c l e i n f o a b s t r a c t
Article history: Context: Despite past empirical research in software architecture decision making, we have not yet sys-
Received 16 November 2017 tematically studied how to perform such empirical research. Software architecture decision making in-
Revised 22 October 2018
volves humans, their behavioral issues and practice. As such, research on decision making needs to in-
Accepted 3 December 2018
volve not only engineering but also social science research methods.
Available online 5 December 2018
Objective: This paper studies empirical research on software architecture decision making. We want to
Keywords: understand what research methods have been used to study human decision making in software archi-
Empirical research tecture. Further, we want to provide guidance for future studies.
Software architecture Method: We analyzed research papers on software architecture decision making. We classified the papers
Decision making according to different sub-dimensions of empirical research design like research logic, research purpose,
Human aspects research methodology and process. We introduce the study focus matrix and the research cycle to capture
the focus and the goals of a software architecture decision making study. We identify gaps in current
software architecture decision making research according to the classification and discuss open research
issues inspired by social science research.
Conclusion: We show the variety of research designs and identify gaps with respect to focus and goals.
Few papers study decision making behavior in software architecture design. Also these researchers study
mostly the process and much less the outcome and the factors influencing decision making. Furthermore,
there is a lack of improvements for software architecture decision making and in particular insights into
behavior have not led to new practices. The study focus matrix and the research cycle are two new
instruments for researchers to position their research clearly. This paper provides a retrospective for the
community and an entry point for new researchers to design empirical studies that embrace the human
role in software architecture decision making.
© 2018 Elsevier Inc. All rights reserved.
1. Introduction work with many stakeholders (Kruchten, 2008) and make impor-
tant design decisions (Kruchten, 2008). They make decisions on
Decision making research involves an understanding of how which architecture style to use, on how to design an API, or what
people make decisions. This is a topic that is starting to receive methods should be included in a class etc. However, there are few
attention in software architecture (van Vliet and Tang, 2016). Re- studies on how software architecture design decisions are made
search in decision making is different from research into inventing even though all these software architecting activities involve de-
or evaluating software engineering methods and tools because decision making (Kruchten, 2008). Decision researchers emphasized
cision making requires a detailed understanding on how humans that decision making is complex and it is not obvious how de-
think and behave. cision makers make decisions (Björklund, 2013). Sometimes de-
Decision making is what software architects do all the time. cision makers themselves cannot tell how they make decisions
Software architects are designers who have a high-level view on (Carroll and Johnson, 1990). We can learn from other disciplines
both business and technical aspects and among other things, they regarding decision making. For instance, classical economics the-
ories assumed that consumers make choices from optimal beliefs
and rationale (Kahneman, 2003), but researchers later found that
∗
Corresponding author. other forces such as bounded rationality (Simon, 1982) and cogni-
E-mail addresses: m.razavian@tue.nl (M. Razavian), paech@informatik.uni- tive biases (Kahneman, 2011; Thaler, 2015) can influence decision
heidelberg (B. Paech), atang@swin.edu.au (A. Tang).
https://doi.org/10.1016/j.jss.2018.12.003
0164-1212/© 2018 Elsevier Inc. All rights reserved.
M. Razavian, B. Paech and A. Tang / The Journal of Systems and Software 149 (2019) 360–381 361
making. The long-held assumption that full market knowledge and search methodology. While these works have a general focus on
rational choices optimize economic decisions does not hold any- research method and software engineering, we instead zoom into
more. Consumer rationality and cognitive biases were taken into the details of empirical research in the area of software architec-
consideration in economic theories, thereby forming the basis of ture DM. This way we are able to picture focus and objectives in
behavioral economics (Kahneman, 2003). Equally, in software de- addition to the research designs. Also, contrary to these works, we
sign we cannot assume designers to have full knowledge of all de- do not want to give specific guidelines, but provide an overview
sign issues and design options to reach an optimal design. The way of and identify current gaps in empirical software architecture DM
each individual designer seeks a design solution depends on his research.
or her preferences, beliefs, knowledge and biases. Thus, decision We ask the following research question (RQ) and the corre-
making in software architecture design can also be more complex sponding sub-questions:
than what researchers assume. Such a shift in research thinking
RQ: How has empirical research on human aspects of software
suggests that software architecture decision making maybe more
architecture decision making been done so far and what can
complex than the traditional software engineering research focus
we learn from that?
on tools and methods only.
Decision making (DM) in software architecture has scarcely We detail this question into three aspects: the focus of the
been studied as a human centric activity. In previous work we studies, their objective and the research design. The first two allow
showed that few research on DM behavior has been conducted in us to position the studies according to their insights about soft-
the software architecture field (Tang et al., 2017). In the software ware architecture:
architecture community, most interest is on the exploration of ar-
RQ1.1: What are the study foci and how to characterize them?
eas such as design rationale (Moran and Carroll, 1996; Tyree and
RQ1.2: What are the objectives of the studies and how do they
Akerman, 2005; Burge and Brown, 20 0 0), creating software knowl-
relate to the focus?
edge management methods (Capilla et al., 2016; Babar et al., 2009),
RQ1.3: What is the research design of the studies?
and creating prescriptive methods and tools (Capilla et al., 2016). In
order to achieve scientific rigor, studies employ empirical research. To understand the focus of DM research (RQ1.1), we look at the
However, the empirical designs appropriate for researching human decision making focus and make explicit whether practice or be-
DM in software architecture have not been investigated systemati- havior aspects are studied. Decision making practice research is
cally. We do not know which research designs are used for certain concerned with the process, techniques and tools to aid software
types of research. decision making, whilst decision making behavior research is con-
In the software engineering field, empirical studies are being cerned with human DM behavior (Tang et al., 2017). To understand
advocated as a way to validate research results (Perry et al., 20 0 0; the details of the focus we look at the data collected with the help
Galster et al., 2018) and research methods like experiments and of the research data focus. Here we characterize and make explicit
surveys are well established (Wohlin et al., 2012; Kitchenham and whether the study focuses on DM process, outcome or factors. As a
Pfleeger, 20 01–20 02). It was found that seventeen percent of stud- result, we provide a new analysis instrument, the Decision Making
ies in software architecture used empirical research methods such Study Focus Matrix, to position the research with respect to these
as case studies and experiments and amongst these studies, half of two foci.
them involve human participants (Galster and Weyns, 2016). How- To identify the objective of DM research (RQ1.2), we look at the
ever, research methods suitable for the study of human behavior design science cycle that embraces three generic study goals: prob-
in software engineering have not been explored in detail. This sub- lem investigation (studying DM problems), treatment design (pro-
ject is challenging because studying human behavior requires not viding solutions), and treatment validation (validation of solutions)
only studying of the technology in use, but also that of the so- (Wieringa, 2014). We refine the design science cycle to create an-
cial and cognitive processes that surround the architects’ thinking other new analysis instrument, the Decision Making Research Cycle
(Dinar et al., 2015). Thus, research in empirical software architec- which also reflects on the DM focus.
ture decision making requires the borrowing of concepts and the- Studying research design (RQ1.3) allows us to understand the
ories from social sciences research. Accordingly, the way empiri- variety and the details of how DM research has been con-
cal studies are designed and conducted can differ from the typi- ducted. We analyze the strategic, tactical and operational designs
cal evaluative research in software engineering. For instance, ap- as explained in common textbooks (Wohlin and Aurum, 2015).
proaches to identify thought patterns are think-aloud protocols and We also consulted general empirical research method books
collaborative design conversations (Dinar et al., 2015). These ap- (Creswell, 2013).
proaches, however, are seldom used in the software architecture To answer these research questions we proceeded as follows: in
field. a previous literature study (Tang et al., 2017), we identified empiri-
Therefore, to understand and conduct empirical research in ar- cal research works in the software architecture discipline that deal
chitecture DM it is important (a) to understand the research de- with different aspects of human software architecture DM. Using
signs of individual studies, (b) to be aware of the insights gained an extended set of literature published until 2017, we analyze the
in terms of the focus and objectives of the study, (c) to be aware of ways DM research has been conducted.
research designs and standards in both, software engineering and We found the following gaps with respect to the focus and data
social science where humans are often the subject of a research. collection (RQ1.1): there is a lack of papers on DM behavior. Also,
Several researchers have addressed the challenges of research the papers study mostly the process of DM and much less the
design in empirical software engineering (Perry et al., 20 0 0; outcome of DM and the factors influencing DM. For the objective
Galster et al., 2018). To address those challenges many have (RQ1.2.) we found that the papers study much less treatments to
proposed means for selecting the right research methodolo- DM problems (few treatment designs and very few treatment val-
gies, and/or provided guidelines for conducting those method- idations). No paper studied behavior in the context of a treatment.
ologies. For instance, Wohlin and Aurum (2015) provide a de- Thus, there is a lack of improvements on software architecture DM
cision model to select the best fitting research designs, while and in particular insights into DM behavior have not led to new
Runeson and Höst (2009) present guidelines for conducting case practices.
studies. Easterbrook et al. (2008) discuss guidelines for selecting From our analysis, we created two new instruments Decision
empirical methods for software engineering which focus on the re- Making Study Focus Matrix and the Decision Making Research Cy-
362 M. Razavian, B. Paech and A. Tang / The Journal of Systems and Software 149 (2019) 360–381
cle that can be helpful in positioning and characterizing software practices of looking at research trends and research methods used
architecture DM papers, and identifying gaps. We give examples in the primary studies similar to e.g. Kitchenham et al. (2009) who
on how to use them. We encourage researchers to position their only performed a manual search on selected well-known venues,
research clearly with the help of these two instruments. For the but did not use automatic searches as the researchers did not syn-
research design (RQ 1.3.) we found basic and inductive research as thesize the research data of the selected literature.
predominant choices on the strategic level and great variation on Instead of using an automated keyword-based database search
tactical and operational designs. It is encouraging that a majority strategy, we opted to go directly to specific and targeted confer-
of the papers studied professionals and thus give insights into the ences and journals that typically publish software architecture lit-
software industry. However, often surveys are used. In comparison erature. This would potentially allow us to find research works on
with the field of social studies we encourage researchers to study human DM in software architecture where the terms and phrases
DM behavior in more detail by experiments and rely on guide- used in the research works are new and varied, and not yet aligned
lines developed in this field. This should be followed-up with real in the community. Following a manual search of these targeted
world case studies. Furthermore data collection should be refined sources, we used backward snowballing (Wohlin, 2014) (i.e. using
by think-aloud protocols or auxiliary information such as gestures reference list to identify new papers to include) to find any rele-
or sketches to unravel mental models. Overall, we see our work vant literature to be included in our study. We judge that such a
as the basis for a systematic approach for researchers to character- manual and targeted search approach would allow us to find suffi-
ize and design empirical studies that embrace human behavior in cient relevant material as a basis for our research purpose. We did
software architecture DM. not perform forward snowballing as we judge that the papers that
The paper is structured as follows: in Section 2 we describe our are in the software architecture field would likely be found any-
literature review and analysis procedure and in Section 3 we intro- way as we have searched through relevant sources until the end
duce our coding schema. We summarize the research results with of 2017. The legitimate finds from forward snowballing in Wohlin’s
respect to the focus in Section 4 and the objectives in Section 5. report represents a very small percentage (Wohlin, 2014).
Strategic, tactical, and operational research designs are discussed In a targeted literature search, we collected software DM re-
in Section 6. In Section 7 we compare the results with other fields search papers that are known to us (Step 1 in Fig. 1). From
and we conclude in Section 8. In the rest of the paper we abbrevi- these papers, we identified eight targeted sources that are likely
ate decision making with DM. to contain the targeted research works (Step 2). We considered
them as the primary sources. These are: (a) Journal of Informa-
2. Literature review tion and Software Technology (IST); (b) Journal of Design Studies
(JDS); (c) Workshop on Sharing and Reusing Architectural Knowl-
This research intends to study the human aspects of empirical edge (SHARK); (d) IEEE Software; (e) Journal of Systems and Soft-
software architecture DM research. We take a broad view of software (JSS); (f) Quality of Software Architecture (QoSA); (g) Work-
ware architecture that includes software requirements and design. ing IEEE/IFIP Conference on Software Architecture (WICSA); (h)
We are looking for literature that studies the human aspects of DM European Conference on Software Architecture (ECSA). Seven of
in software architecture design. Human aspects can be about how the eight sources are platforms where software architecture re-
human decision makers think, communicate, and act; it may also searchers often publish their works. JDS is the exception. We
involve the effects on software architecture DM when humans use picked JDS because it has a focus on design and because there was
software decision supporting tools, methods or processes. a special issue with a number of studies of how software designers
Human DM in software architecture is a subject that is rarely think (Petre et al., 2010). We also considered journals focusing on
studied in software engineering. In order to carry out this study, empirical work, namely Evaluation and Assessment in Software En-
we contemplated our research methodology, which includes a sys- gineering (EASE) and Empirical Software Engineering Journal (ESE),
tematic literature study (SLR) to survey major and relevant jour- but did not find any suitable papers between 2005 and 2017.
nals and conferences (Kitchenham et al., 2009). We have decided The researchers collectively and manually searched the past is-
against the use of automated keyword-based SLR because the use sues of these eight targeted software sources to find relevant pa-
of keyword search can either end up with too much irrelevant lit- pers (Step 3). We looked through all issues for the 11 years from
erature or too little relevant literature. In the former case, there 2005 till 2017. The reason for selecting 2005 is because at that
are many studies on DM in software architecture, but not many time design rationale study started to take off in the software
of them focus on the human aspects of decision making. Instead, architecture field with prominent research papers such as (Tyree
many of the research works focus on the engineering aspects of and Akerman, 2005; Jansen and Bosch, 2005). We started this re-
DM such as methods and processes to model, capture, and reuse search in early 2016 and hence we finished our literature review
decisions. In the latter case, if we use “software architecture” as at the end of 2017. We retrieved research papers from these pri-
one of our search phrases, we would miss useful literature such as mary sources by reading the paper titles and abstracts published
[S14], [S3] and [S7] that describes software architecture DM. This by these eight sources (Step 3). In this step, we examined the title
is because some of these studies do not mention software architec- and the abstract and looked for key phrases like “decision”, “design
ture design even though the activities that they deal with are con- decision” or “decision making”. Some of the studies do not have
sidered as software architecture activities. Human aspects of DM is these keywords and so we also look for any evidence that human
a general term that describes what goes on when humans make aspects of DM is involved in each study, e.g. S71 and S35. This step
decisions in software design. DM can be influenced by many hu- finds potential candidate papers, but it does not guarantee that the
man aspects such as cognition, biases, groupthink, communication, paper would be accepted in the study because a paper may not
and so on. Forming a comprehensive list of search phrases to de- conform to the inclusion criteria such as studying human aspects
scribe all these related human aspects in DM is challenging. There- or conducting empirical research in software architecture. The final
fore the basic difficulty is the alignment of search phrases that can inclusion and exclusion of papers are performed after reading the
reasonably result in a relevant set of literature, and a set in which entire paper in Step 6.
we can confidently say that it is a representative set of literature. There are also secondary sources where we found relevant re-
Additionally, we are not interested in the research data of the stud- search papers. With the results from our search in the primary
ies, but instead we focus on the research designs. Thus, search sources and the known papers (Step 4), we used a backward snow-
completeness is not so important for us. This is consistent with the balling technique (Step 5) (Wohlin, 2014; Jalali and Wohlin, 2012)
Fig. 1. Paper selection process.
to find more relevant papers from citations. These new papers whilst another researcher considered this kind of research as ap-
were then added to the collection of potential articles (Step 4). plied research. After some discussion, we defined case studies that
We also included papers that we knew before this review, some involve practitioners as applied research but not case studies in-
of them are well-known papers dated earlier than 2005 (Step 1). volving students.
Although some of these papers are earlier than 2005, they also
satisfy the criteria defined in Step 6. The papers found from back-
ward snowballing are from Empirical Software Engineering, IEEE 2.1. Limitations
Expert, Communications of ACM, ACM Computing Survey, Interna-
tional Journal of Human-Computer Interaction, IEEE Transaction of In the following we discuss internal and external validity as
Software Engineering, Agile Conference, Automated Software Engi- typical for empirical work. According to Kitchenham and Charters
neering and book chapters (Step 5). We ended up with a prelimi- (2007), internal validity tries to minimize bias and to prevent sys-
nary set of eighty-three (83) papers from both sources. During Step tematic error caused by the design and conduct of the study, while
5, we also found twelve (12) research papers that are relevant to external validity refers to the generalizability and applicability out-
software DM, but the subject of these studies is not software de- side of the study. While there are no agreed guidelines on how
velopment. to mitigate the threats to systematic literature reviews themselves,
Based on this set of papers, we then selected the research pa- Kitchenham suggests four quality assessment (QA) criteria for se-
pers to be included in our analysis by applying selection criteria lection of primary studies (Kitchenham et al., 2009): QA1. Are the
(Step 6). First, a selected paper must study one of the two sub- review’s inclusion and exclusion criteria described and appropri-
jects: (a) human factors that affect software DM; (b) human as- ate? QA2. Is the literature search likely to have covered all rele-
pects of DM practice involving tools, methods or process in a soft- vant studies? QA3. Did the reviewers assess the quality/validity of
ware development environment. Second, a selected paper must the included studies? QA4. Were the basic data/studies adequately
present primary research to yield empirical results. This criterion described?
eliminates papers that are anecdotal or offer secondary research Those qualities relate to the extent to which the study mini-
such as high-level literature surveys. Third, if a paper does not re- mizes bias and maximizes internal and external validity. Therefore,
late the research results to software DM in software architecture we use those four quality assessment criteria in the following dis-
related activities, the work is excluded from our review. These cri- cussion:
teria helped us to eliminate any personal biases in the paper se- Internal validity. Considering QA1, we have explicitly defined
lection. inclusion and exclusion criteria, to avoid results that depart sys-
Each of the four researchers (the authors of Tang et al., 2017) tematically from human aspects of DM or empirical studies. Given
read all selected papers. We arranged the reading, selecting and our interest in research designs of the primary studies, we have
coding of the papers such that (a) each paper was assigned ran- only included studies that are published in well-known venues. As
domly and read and coded by two researchers; (b) each researcher noted in Section 2 the research data and thus the quality of the
had to determine if the paper fits the selection criteria; (c) each re- data and analysis of the studies are not relevant for us (QA3). Con-
searcher read at least forty-five (45) papers; (d) a researcher must sidering QA4 we only analyzed data on research designs that are
not code or select the paper s/he wrote. We have finally selected clearly reported. For instance, we originally wanted to analyze the
a total of thirty-eight (38) papers. Table 1 summarizes the search threats to validity of primary studies, but they were not reported
results. The columns indicate at which stage the papers were iden- clearly in many of the primary studies, so we left them out of
tified and if a paper is selected or not. Step x in signifies that a pa- our analysis and results. A threat to the internal validity of liter-
per from a particular paper source passes the selection criteria in ature reviews in general and of our study is the data collection.
Step 6. For instance, cell “Step1 in / IST” shows that the known soft- We might have classified the papers wrongly. A particular issue
ware decision article (Step 1) in IST has been selected after applying is the full codification of our selected papers. In some cases full
the selection criterion in Step 6. Step x out shows papers that do codification was either not possible or subject to our interpreta-
not meet the selection criteria. Si in each cell is the paper identi- tion, because the papers do not provide sufficient information. As
fier. The references of all papers used for the analysis are given in we interpreted through coding how the research was conducted,
Appendix A. there is a chance for internal inconsistency. We mitigated the data
Each paper was coded by two researchers according to the collection threat by holding consensus meetings with all four re-
schema explained in Section 3. If the codes of the two researchers searchers. In the data extraction process each primary study was
did not match, we discussed this and adjusted them. An example read by two reviewers. One reviewer acted as the main data ex-
for such an adjustment is the interpretation of Research Outcome. tractor, whilst the other reviewer acted as a checker. Any disagree-
Our interpretation of basic and applied research differed initially ments were discussed in the data extraction consensus meetings. A
when one of us coded case studies using students as basic research second threat to internal validity is the general applicability of the
coding for characterization and classification. The start-list of cod-
ing is extracted from Wohlin and Aurum (2015) and it has been
constantly refined through our coding procedure.
Step 5 papers rejected after

External validity A threat to external validity of literature re-
Step 6 filtering(18 papers)
S43,S44,S50,S55,S56,S63,
views in general and also of this study is paper selection (see also
S2,S10,S17,S18,S24,S40,
QA2). The search was organized as a manual search process of a
S86,S87,S88,S89,S95
specific set of journals and conference proceedings not an auto-
mated search process. This is consistent with the practices of other
researchers such as Kitchenham et al. (2009) which looked at re-
search trends and research methods used in the primary studies
as opposed to synthesizing the data or conclusions of those pri-
S4
mary studies. We have restricted the scope to the main software

architecture publishing sources and backward snowballing, and we
Step 5 papers selected after
did not use forward snowballing. The threat here is that relevant
Step 5 – 22 papers found
Step 6 filtering (4 papers)
works from other publication venues may be missing. As such, we

are limited in our claim that all relevant software architecture DM
literature is included. However, we have surveyed mainstream soft-
S14,S15,S62, S77
ware architecture research sources that are likely to publish such

works. We judge that we have a fair representation of the empir-
ical publications on software architecture DM, and our method is
rigorous. As part of the review, we also gathered research papers
from other disciplines, notably psychology, cognitive science, and
design studies to enhance our understanding of DM in general. We
Step 3 papers rejected after
did not conduct any comprehensive literature search in these other

Step 6 filtering (15 papers)
S1,S16,S22,S36,S58,S72,S76
disciplines.
3. Coding schema
S11,S13,S64, S71
Coding was performed by four researchers (authors of

S59,S75
Tang et al., 2017) and every selected paper was coded by two re-
S38
S23
searchers separately. When there was any disagreement between

coders, the issue was discussed amongst all four researchers to
seek an interpretation of the code as well as the intent of the
Step 3 papers selected after
paper. We sought to eliminate any biases or misinterpretation

through this process. During the coding of the papers we used

a coding schema (see Table 2). It enables the characterization of
each research work based on the four dimensions: Study Focus,
S31,S73, S93,S94
S3,S52,S90,S91
Research Cycle Focus, Strategic Design, Tactical and Operational

S19,S26, S61
S21, S39,S92
Design. The two dimensions, i.e. Strategic Design and Tactical and
Operational Design, stem from the work of Wohlin and Aurum
(2015). Although they treat strategic- and tactical- and operational
design in one dimension, we decided to divide them into two dif-
Step1 papers rejected after Step
ferent dimensions so that we can identify the sub-dimensions and

subsequent analysis for each. They address different research de-
sign concerns and we wanted to study the interdependencies be-

6 filtering(13 papers)
tween them: one addresses the strategic design choices and the
other, how those choices are operationalized. In addition, we also
considered the dimension of the Study Focus, since we wanted to
S29,S37, S85
S5,S12, S82
relate the research design of a study with the focus of the study.
S84,S68
S74,S83
Furthermore, we considered the dimension of the Research Cycle

S48
S65
S42
Focus which positions the paper within the design science cycle
(Wieringa, 2014), making explicit the objectives of the study. In the
following, we explain each dimension and their sub-dimensions.
Table 2 gives an overview of the detailed codes for the dimensions.
Step1 papers selected after
As a new dimension we use the Study Focus which determines

the DM aspects to be studied. We use two sub-dimensions in the
S30,S32,S47,S60,S67
S9,S27,S28, S49,S53
Study Focus dimension, namely Decision Making Focus and Re-

search Data Focus.
The decision making focus (DMFocus) coding characterizes DM
S46,S80
S33,S35
S51,S70
Results of paper selection.
research with regard to the distinction between human DM be-

S7,S66
S81
S69
havior (DMBehavior) and DM practices in terms of design process,

techniques, tools or methods (DMPractice). We already used this
distinction when analyzing the content of the papers in our previ-
ous literature review (Tang et al., 2017). DMBehavior papers study
IEEE S/W
SHARK
Others
WICSA
psychological and cognitive aspects of DM and deal with different

Table 1
QoSA
ECSA
JDS
IST
JSS
human thinking aspects. Such studies provide an understanding of

human behavior and its potential influence on the ways software
Table 2
Detailed Coding Schema: The codes are structured into 4 dimensions where we add study focus and research cycle focus to the usual distinction of strategic, tactical and
operational design. For each dimension we distinguish several sub-dimensions and for each sub-dimension we used several codes.
Dimension Sub-dimension Codes
Study Focus Decision Making Focus DMBehavior, DMPractice

Research Data Focus DMFactor, DMProcess, DMOutcome
Research Cycle Focus Basic Research for Problem Investigation Study DMBehavior, Observe DMPractice
Basic Research for Treatment Design Treatment Design only DMPractice, Treatment Design relating DMBehavior
and DMPractice
Applied Research for Treatment Validation Treatment Validation only DMPractice, Treatment Validation relating
DMBehavior and DMPractice
Strategic Design Research Outcome Basic research, Applied research, Open Question, Closed Question
Research Logic Inductive, Deductive
Research Purpose Explanatory, Descriptive, Exploratory, Evaluative
Tactic and Operational Design Research Methodology Real world case study, Limited time case study, Experiment, Survey, Action
Research
Research Process Qualitative, Quantitative, Mixed
Participants Students, Academics, Professionals
Data Collection Interviews, Questionnaires, Focus Groups, Observation, Content analysis
Viewpoint Retrospective, Current, Just after the Fact
engineers make decisions. DMPractice papers either observe DM in study could have both codes, if practice is observed including be-
practice in general or they study specific processes, methods or havior aspects. Based on the insights of the problem investigation
tools. DMPractice papers provide insights into the DM of software a treatment is designed. This is a specific process, method or tool
engineers, or they try new methods to improve DM. for DM. In stage 2 basic research is performed to understand the
The research data focus (RDFocus) coding characterizes DM re- aspects of this treatment before it is applied in practice. Here we
search as to which details of the aspect (in terms of data) are distinguish again between two cases: either the treatment is stud-
studied. Inspired by work of Dorst on decision thinking theory ied with relation to behavioral aspects (Treatment Design relating
(Dorst, 2011), we categorize the DM research data into three cat- DMBehavior and DMPractice) or not (Treatment Design only DMPrac-
egories: (a) decision making factors (abbreviated as DMFactor), (b) tice). For the former ideally insights from the problem investigation
decision making process (DMProcess), and (c) decision making out- with respect to human behavior are taken into account. Finally, in
come (DMOutcome). DMFactors are factors that influence the DM the last stage applied research is performed to validate the treatment
process and constitute the context of the DM process. DMProcess in practice. Again we distinguish between the two cases: either the
describes the activities during DM such as analyzing the problem treatment is applied with study of behavioral aspects (Treatment
or providing arguments for a decision. These activities can be in Validation relating DMBehavior and DMPractice) or not (Treatment
the mind of the people or visible to the outside through documen- Validation only DMPractice).
tation. DMOutcome is the output of DM (e.g. decision documenta- For the Strategic Design dimension we adopt the definition of
tion). Wohlin and Aurum (2015) and consider strategy as consisting of
Note that the two study foci are orthogonal. A study can fo- Research Outcome, the general Research Logic, and Research Purpose.
cus on DMBehavior, and collect data on the factors influencing the However, we skip the research approach (positivist, interpretivist,
behavior (DMFactor), or on how the process is influenced by the and critical), as this is typically not explicitly mentioned in the pa-
behavior (DMProcess), or how the outcome is influenced by the pers.
behavior (DMOutcome). Similarly, a study can focus on DMPractice Research Outcome distinguishes whether the objective is to con-
and collect data on how a particular process, method or tool is in- duct basic or applied research. We use the following corresponding
fluenced by factors, or influences the overall process or outcome, codes: Basic research (a.k.a fundamental research) creates knowl-
or how in general factors influence the process. edge about the world, without calling for an improvement of the
We introduce the Decision Making Research Cycle which world (Wieringa, 2014). We characterize basic research as focus-
positions the paper with respect to the design science cycle ing on understanding a problem or providing a theory rather than
(Wieringa, 2014) and the relation of DMPractice and DMBehavior. solving a real world problem. For example, basic research investi-
The design science cycle is a well-established research method gates how DM is carried out in practice. On the other hand, applied
in software engineering and is typically decomposed into three research is about improving the world (Wieringa, 2014). We char-
tasks, namely: problem investigation (what phenomena must be acterize applied research as providing a solution to a real world
improved? why?), treatment design (design one or more solutions problem by applying knowledge (e.g., a DM practice). For example,
that could treat the problem), and treatment validation (would applying a certain decision documentation approach to improve
these solutions treat the problem in a real world context?). This knowledge vaporization in a real world software project. Besides
is called design cycle, because researchers iterate over these tasks basic and applied research, Wieringa (2014) also proposes classi-
many times in a design science research project. Specifically for fying the research outcome by the range of possible answers that
DM research and building upon the design science cycle we pro- is pre-specified. An open question contains no specification of its
pose the Decision Making Research Cycle. Fig. 2 shows the three possible answers. A closed question contains hypotheses about its
stages of the Decision Making Research Cycle: Problem investiga- possible answers. We adopt this distinction.
tion, Treatment Design, and Treatment validation. Research Logic refers to the approach in which research results
Problem investigation aims at identifying problems (maybe with are reasoned about and derived (Wohlin and Aurum, 2015). Our
a theory) in DM. This is basic research. If we relate that to DM- codes correspond to two common ways of reasoning in empiri-
Behavior and DMPractice, there are two possibilities: the basic re- cal software engineering research: deductive and inductive. Deduc-
search studies human behavior that means psychological or cog- tive research works from a general theory to a specific outcome
nitive aspects of DM (Study DMBehavior). Or the basic research and it mainly addresses theory testing. For example, when a re-
observes the DM in practice (Observe DMPractice). Note that one searcher collects data to confirm or reject a theory. As such, de-
Fig. 2. Decision making research cycle.
ductive research works from a general theory (generalities) to a search in Wohlin and Aurum (2015)). Also, we do not use specific
specific outcome (particularities). Inductive research, on the other codes for data analysis (such as grounded theory), but only use
hand, is based on specific data (particularities) to derive a general these codes for the research process which distinguish qualitative,
theory (generalities). For example, interviews or observations (par- quantitative and mixed analysis. In addition, we distinguish the
ticularities) are used to propose a broad understanding or theories viewpoint when the data was collected. For the retrospective view-
of DM that are intended to apply beyond the sample of partici- point, participants describe DM in past projects, e.g. in surveys. For
pants interviewed or observed. the current viewpoint, participants are performing DM, e.g. in ac-
The Research Purpose refers to the reasons why a research is tion research and real world project case studies. For the just after
conducted and can be classified as exploratory, descriptive, explana- the fact viewpoint, participants describe DM just after they have
tory, and evaluative (Wohlin and Aurum, 2015). Exploratory re- performed it, e.g. in limited time case studies or experiments. It is
search is used when there is not much information available in the preferable to study participants during or just after DM, but often
topic area and the researcher aims to gather insights into the prob- this is not possible so that data is collected retrospectively.
lem. Descriptive research is used to describe a phenomenon or the The codes for all papers are given in Appendix B. A short sum-
characteristics of a problem. Explanatory research is used to exam- mary of the papers is given in Appendix C. In the following sec-
ine the nature of certain relationships between the elements of a tions we provide the results of the analysis with respect to these
problem. Finally, evaluative research aims to determine the impact dimensions. For each dimension we give the results and then dis-
of methods, tools, or frameworks. cuss them.
The Tactical and Operational Design operationalizes the re-
search strategy. We adopt the definition of Wohlin and Aurum 4. Study focus
(2015) where the tactical level comprises Research Process, Research
Methodology, and Participants. The codes are based on the follow- Research design depends foremost on the focus of the study. To
ing distinctions: Research Process can be qualitative (focusing on better distinguish between different research foci we introduce the
the observations of qualitative data and the interpretation of the DM Research Focus Matrix (see Table 3) which relates DMFocus
data), or it can be quantitative (using statistical means to analyze (decision making focus) and RDFocus (research data focus). Position-
data) or mixed. For Research Methodology we take a distinction from ing a research using the matrix, makes explicit whether the focus
Runeson and Höst (2009): surveys, case studies (studying specific of human decision behavior or decision making practice are stud-
real cases, no participation of the researcher), experiments and ac- ied, and what kind of research data is collected. Each cell clearly
tion research (real case with participation of the researcher). Fur- describes an aspect with its details. The cells are illustrated by ex-
thermore, we distinguish two kinds of empirical case studies: real ample research foci formulated in form of research questions. We
world and limited time case studies. Limited time case studies have also show the papers found for each cell. In the following, we first
a preset duration of only a few hours and work on an artificial answer the RQ1.1 where we provide an overview of the focus of
case. Only the former provide insights into real practices in soft- the current studies, then we discuss what we learned and how the
ware architecture while the latter typically give first insights with DM Research Focus Matrix can be used in future studies.
students as participants. As there is evidence (Dinar et al., 2015)
that for DM it makes a difference whether students, academics, or 4.1. Overview of the decision making focus
professionals are studied we also look at the kind of participants.
The operational level in Wohlin and Aurum (2015) comprises For the DMFocus, we classified the papers in terms of whether
Data Collection and Viewpoint. This includes experiment, survey, they are focusing on human DM behavior (DMBehavior) or DM
and simulation as data collection methods. We view the first two practices such as design process, techniques, tools or methods
as research methodology and skip simulation. Then we use the fol- (DMPractice). We found 13 papers on DMBehavior, and 25 papers
lowing codes: For data collection we use interviews, questionnaires, on DMPractice. We show these two categories in Table 4, where we
focus groups, observation, content analysis (also called archival re- sub-classified each category by the main focus of their studies, the
Table 3
The DM Research Focus Matrix to position research with regard to decision making focus and research data focus. Positioning a research on the Matrix makes explicit
whether behavior or practice aspects are studied and with which data.
RDFocus vs. DMBehavior (13 papers) DMPractice (25 papers)

DMFocus
DMFactor Aspect: Collect data on DMFactors to study DMBehavior Aspect: Collect data on DMFactors to study DMPractice
(9 papers) (3 papers: S52,S91,S93) (6 papers: S14,S15,S26,S51,S73,S77)
Examples: Examples:
Which factors influence a rational/ naturalistic DM or Which factors influence the DMProcess in practice?
group DM?
Which cognitive factors (e.g. bias, limitation, mental How does the prescribed method/ agile development/ DM tool/ design
representation) influence DM in general? reasoning/ knowledge management influence the factors relevant for the
DMProcess in practice and vice versa?
DMProcess Aspect: Collect data on DMProcess to study DMBehavior Aspect: Collect data on DMProcess to study DMPractice
(27 papers) (10 papers: S28,S3,S53,S60,S62,S66,S7,S70,S80,S81) (17 papers: S21,S27,S30,S31,S33,S35,S39,S49, S51,S61,S67,S69, S77,S9, S90, S92,
S94)
Examples: Examples:
How is a rational/ naturalistic DMProcess/ group DM (best) How is the DMProcess in practice performed?
performed? Which cognitive factors influence which
DMProcess step and how?
How is the DMProcess in practice performed in the context of the prescribed
method/ agile development/ DM tool/ design reasoning/ knowledge
management?
DMOutcome Aspect: Collect data on DMOutcome to study DMBehavior Aspect: Collect data on DMOutcome to study DMPractice
(8 papers: S19,S21,S32,S46,S47,S51,S73,S77)
(9 papers) (1 paper: S91)
Examples: Examples:
What is the outcome of a rational/ naturalistic DMProcess/ What is the outcome of the DMProcess in practice?
group DM? Which cognitive factors influence which
outcome how?
How does the outcome of DM in the context of the prescribed method/ agile
development/ DM tool/ design reasoning/ knowledge management in
practice look like?
Table 4
Research papers classified by DM behavior and practice.
Decision making focus classes and number of papers found Paper ID
DM Behavior Papers (13 papers)

Naturalistic and Rational Decision Making – 5 papers S52,S66,S80,S81,S91
Cognitive Biases – 2 papers S53, S93
Group Decision Making – 2 papers S3,S60
Cognitive Limitations – 2 papers S28,S70
Mental Representation – 2 papers S7, S62
Behavioral Science – 0 papers Nil
DM Practice Papers (25 papers)
Decision Making Process - 7 papers S26,S27,S51,S61,S73,S90,S94
Decision Making Methods - 4 papers S32,S33,S39,S49
Agile Development Method – 3 papers S14,S15,S92
Decision Making Tools – 3 papers S9,S21,S46
Design Reasoning - 4 papers S30,S31,S67,S69
Knowledge Management - 4 papers S19,S35,S47,S77
number of papers found in each sub-class (i.e. shown within the sonality with respect to architecture DM. The number of behavior
brackets) and the identified papers in each sub-class. The figures science papers shown in Table 4 is zero. Although no such papers
are updated from our previous study in Tang et al. (2017). were found in the software architecture field, we report this cate-
Decision Making Behaviors. There are 13 DMBehavior papers gory because other disciplines have shown that behaviors are con-
that studied psychological and human aspects of DM. In this class, tributing factors to decision making.
the papers deal with different human thinking aspects. We found Decision Making Practice. We found 25 research papers about
five papers that study Naturalistic and Rational Decision Making. DM processes, methods or tools. All of these papers study some
Two papers dealt with Cognitive Biases. Two papers studied Group aspects of software architecture DM practices. We found six sub-
Decision Making. Two papers studied cognitive limitation and sat- classes. Decision Making Process contains seven papers that investi-
isficing behavior, classified as Cognitive Limitations. Finally, two gate the steps software architects take in DM. Four papers on De-
papers studied mental characteristics and experience, and these cision Making Methods investigated how a particular method im-
papers were classified as Mental Representation papers. There is proves DM. Three papers research Agile Development Method. Agile
one group where no papers were found. We call it Behavioral development methods prescribe steps to facilitate a group of de-
Science papers. Behavioral science is one of the psychology ar- velopers to reach goals, schedules, and consensus. We found three
eas that are widely studied in management and organizations papers that describe Decision Making Tools. Four papers describe
(Carroll and Johnson, 1990). There is an awareness and there are how Design Reasoning can aid DM. All decisions require some kind
studies of behavioral science and DM in the information system of knowledge. Four papers focus on the role of Knowledge Manage-
field (Lovallo and Sibony, 2010). In our review, we have found no ment in DM.
works that investigate organization behaviors, motivations, or per-
4.2. Overview of the research data focus • Is the study a DMBehavior study, focusing on human decision
behavior, or is it a DMPractice study, focusing on a practice,
The research data focus characterizes the types of data that de- method, or tool?
pict the focus of the research, i.e. (a) data about the factors that • Which of the three research data categories are collected: DM-
influence DM (DMFactors), (b) data about the activities during DM Factors influencing DM, DMProcess reflecting the activities of
(DMProcess) (c) data about the outputs of DM (DMOutcome). For DM, and/or DMOutcome, i.e., the end result of DM?
the research data focus, we found that 27 papers specifically focus
on DMProcess, nine papers focus on DMFactors, and nine papers on To exemplify the use of the DM Research Focus Matrix, we
DMOutcome (see Table 3). It should be noted that these categories present two examples of research with two different mappings
are not exclusive and that there are studies that have more than on the matrix. Although these two examples cannot represent the
one DM research data focus and are positioned in more than one whole spectrum of study focus variations, they are useful to show
category. how the different coverage of matrix can characterize different
research focuses. The mapping of each example is illustrated in
Tables 5 and 6. The grey cells represent the aspects of DM Research
4.3. Combining the decision making research focus and the research Focus Matrix that are not covered in the corresponding example.
data focus
Example 1. Researcher A decides to conduct research on the role
When we combine the two foci in the DM Research Focus Ma- of intuition and rationality in DM. First, the Matrix enables re-
trix (Table 3), we see that the research aspects focused are subtly searcher A to clearly make the DMFocus of her research explicit: in-
different in each area. We have exemplified these aspects in the tuition and rationality are cognitive processes of the human mind,
Focus Matrix. The Focus Matrix comprises six different generic as- this study falls under the DMBehavior research category. Now re-
pects that can be studied. Table 3 shows that so far DMBehavior searcher A needs to detail the research focus and to decide which
studies have mainly focused on one aspect, i.e., collecting data on types of data to pursue (RDFocus). Using the DMFocus Matrix, re-
the DMProcess to study DMBehavior, while DMPractice studies have searcher A has three options with corresponding research ques-
explored all possible aspects. tions: (a) focus on DMFactors and study, for instance, which situ-
The subcategories of DMBehavior and DMPractice (Table 4) give ational factors induce or affect the use of intuition and rationality
examples for the different research data foci: cognitive bias and in software architects as decision makers (RQA1 in Table 5); (b)
cognitive limitations are possible DMFactors, while group decision focus on the DMProcess and study how intuitive and rational deci-
making is a kind of DMProcess. The mental representation can be sion making are reflected in the architectural DM process (RQA2 in
a part of the DMOutcome or the DMProcess. The subcategories of Table 5); and/or (c) focus on the DMOutcome and study the con-
DMPractice all give examples of important DMProcess aspects. How- sequences of intuition and rationality in DM in terms of for in-
ever, this does not mean that there are only few possible combi- stance decision quality or even project success (RQA3 in Table 5).
nations of the two study foci. When we examine the two foci to- Researcher A could address all three research questions and there-
gether, we can identify the gaps in the current landscape of DM fore cover all details of rational or intuitive DM in terms of fac-
research: tors, process, and outcome, or for instance focus on factors only.
Likewise, she could cover the three research questions in one sin-
• Few papers on DM behavior. As already described in gle study or in separate studies. Table 5 shows the coverage of the
Tang et al. (2017) there is little research on behavioral aspects DMFocus Matrix for the research in this example (grey cells repre-
of DM, and the number is much smaller than in decision prac- sent the aspects that are not covered).
tice research. That means that software researchers in general
do not pay much attention to the role of human behavior in Example 2. Researcher B decides to study design fixation (i.e., cog-
software DM, and we do not really understand how this im- nitive bias) in agile software development. The focus of this re-
pacts the quality of software design. search touches upon both DMBehavior (design fixation) and DM-
• Lack of focus on the DM outcome. In particular, we found only Practice (agile software development) aspects. Now researcher B
one behavioral research paper that addressed the final outcome needs to further detail the research focus and decide on the types
of DM. This means that we do not know how behavioral as- of data that are pursued (RDFocus). As shown in Table 6 Researcher
pects of DM relate to the actual decisions made or the qual- B could focus on DMFactors and study design fixation as a situa-
ity of those. For example, the consequences on decision when tional factor affecting DM. Researcher B could first focus on DMBe-
group DM is used are unknown. Similarly, the effects of cogni- havior and specifically study design fixation as a cognitive factor in
tive biases on the outcome are unknown. architectural decision making (see RQB1 in Table 6). Based on the
• Lack of focus on factors. Only three behavioral research papers results, Researcher B can formulate a hypothesis such as “design
have looked in detail on the factors that influence DM. As such, fixation leads to sub-optimal decisions” and test this hypothesis in
we know very little about which factors influence DM and how. the decision making process (DMProcess) of agile software devel-
opment projects (see RQB2 in Table 6). As shown in Table 6, the
These gaps highlight the lack of important DM knowledge in mapping of Example 2 traverses the DMBehavior and DMPractice
software architecture and they need to be filled. cells. More on this crossover path can be found in Section 5.
4.4. Using the DM research focus matrix by researchers 5. Research cycle focus
We propose that future research studies position themselves The Decision Making Research Cycle shown in Fig. 2 allows po-
clearly in the cells of the DM Research Focus Matrix. This can help sitioning DM research with respect to understanding behavior and
researchers to make the focus of the research explicit and would practice, and for improving software architecture design practice.
make it easier for the community to map out related results to DMBehavior and DMPractice together provide a picture to improve
understand the current state of research in this area. To position software architecture DM. In the following, we first answer RQ1.2
a research project on the matrix a researcher needs to answer the and provide an overview of research objectives as we relate those
following two main questions: objectives to the DM Research Cycle. Then we discuss what we
Table 5
DMFocus Matrix for Example 1: Focus on covering all details of rational or intuitive DM.
RDFocus vs. DMFocus DMBehavior DMPractice
DMFactor RQA1. Which situational factors induce or affect use of intuition and rationality in software architecture DM?
DMProcess RQA2. How intuitive and rational decision making are reflected in architectural DM?
DMOutcome RQA3. What are the consequences of intuition and rationality in DM in terms of for instance decision quality?
Table 6
DMFocus Matrix for Example 2: Focus on covering both DMBehavior and DMPractice.
RDFocus vs. DMBehavior DMPractice

DMFocus
DMFactor RQB1. Is design fixation a cognitive bias in DM ?

DMProcess RQB2. If design fixation lead to sub-optimal decisions in agile software
development projects, if so, how?
DMOutcome
Table 7
Mapping objectives of papers to DM Research Cycle.
Research Cycle Studying or relating to DMBehavior (13 papers) Observing or only studying DMPractice (25 papers)
Basic Research for Problem S28,S3,S52,S53,S60,S62,S66,S7,S70,S80,S81,S91,S93 S14,S15,S26,S27,S30,S31,S35,S51,S67,S73,S77,S92,S94 (13

investigation (26 papers) (13 papers) papers)
Basic Research for Treatment Design none S19,S21,S32,S33,S39,S46,S47,S49,S61,S69 (10 papers)
(10 papers)
Applied research for Treatment none S9, S90 (2 paper)
Validation (2 papers)
learned in the studies using the research cycle and how the DM treatment design, and validation. Such a positioning can help to
research cycle can be used to help researchers in future studies. make explicit the focus of the research in the broader landscape of
solving real world DM problems, and transferring the treatments of
5.1. Overview of the research objectives based on the DM research those problems into real world practice. It would also make it eas-
cycle ier for the community to understand typical DM problems and al-
ready existing treatments of these problems. To exemplify the use
As can be seen in Table 7, the objectives of DM research have of the DM Research Cycle, we extend Example 2 in Section 4.4 and
focused on problem investigation (26 papers) and only half of them show an example covering the full research cycle. Fig. 3 illustrates
study human behavior. Fewer papers (10) have provided treatment the mapping of the example to the DM Research Cycle.
design and only two have applied those treatments in practice. Be-
havioral aspects have not been studied in the context of treat- Example 3. Researcher B would not only like to study design fix-
ments. ation in agile software development, but also provide a treatment
From examining the research objectives in Table 7, we identify for it. In Problem investigation researcher B can conduct basic re-
two main gaps in DM research: search that studies fundamental knowledge of the psychological or
• Lack of studies on treatments (that means treatment design cognitive aspects of DM. Thus, a researcher might study “Is fixation
and treatment validation). Whilst there are many works that in- a cognitive bias in DM?” (see box 1 in Fig. 3). Based on the re-
vestigate the problems, only a third of the papers have looked sults, researcher B can formulate a hypothesis such as “design fix-
at the treatments, and even fewer in treatment validation. Thus, ation leads to sub-optimal decisions”. To intensify problem inves-
we know little of how we really can influence practice. For tigation in practice, researchers can for example, test this hypoth-
software engineering as an applied science, research results in esis in agile software development projects (see box 2 in Fig. 3).
principle should be applicable in practice. The extent to which Next, he could devise a treatment to counteract design fixation or
how this could work, however, has been the subject of debate take a specific method he thinks is relevant in that context. Then,
for decades (Storey, 2011), and remains an unsolved problem. he performs basic research relating the treatment with the design
Dinar et al. (2015) call this the science-practice dichotomy. fixation. Relating means (a) if/how the fundamental DMBehavior
• Lack of studies on behavior in the context of treatments. aspects (design fixation) change in the DMPractice context (treat-
The studies, which looked at treatment of the issues identified ment) and/or (b) if/how the DMBehavior aspects influence the DM-
in the problem investigation, did not study behavioral aspects. Practice processes, methods, or tools. In the example, researcher
Thus, the insights on behavioral aspects of DM in general have B can design a de-biasing technique to be used in retrospective
not been used to understand the effects of treatments more meetings in agile projects and perform again basic research with
deeply. In particular this means, that insights into DMBehavior a first experiment (see box 4 in Fig. 3). Finally, during Treatment
have not led to new practices. Validation, he applies this technique and learns from its use in
real world context. In this example, researcher B may validate the
These gaps highlight the lack of important DM knowledge in de-biasing solution in a real world agile project. He can, for in-
software architecture and they need to be filled. stance, study how design fixation is different from the one in prob-
lem investigation, i.e., is there any improvement in design fixation
5.2. Using the decision making research cycle (DMBehavior) when de-biasing is used in the retrospective meet-
ings of the agile software development (DMPractice) (see box 6 in
By using the DM Research Cycle (see Fig. 2), researchers in this Fig. 3)? These new findings could initiate a new cycle of the DM
field can position their work in terms of problem investigation, research. In the new cycle the researcher may go deeper and fur-
Fig. 3. DM Research Cycle in an example, studying design fixation in Agile Software Development (ASD) and proposing de-biasing techniques as a treatment.
Fig. 5. Distribution of study classification by inductive and deductive reasoning.

Fig. 4. Distribution of study classification by basic/applied research with
open/closed research questions.
ther study aspects of design fixation which still need improvement

or he might focus on a new DMBehavior aspect such as a mental
model. There are also opportunities for synergies where insights in
the behavior of one specific research group can be taken into ac-
count by other researchers applying specific processes, methods or
tools.
6. Strategic, tactical, and operational research design

Fig. 6. Distribution of study classification by research purpose.
In this section we answer RQ1.3 and give an overview of the
strategic, tactical, and operational research designs (see Section 3)
we found and then we discuss the insights based on it
6.1. Overview of strategic research designs
In this subsection we show the results with respect to research

outcome, purpose and logic.
Research outcome is a refinement of the research cycle code.
Of the 38 research works, we found that 30 studies are basic re-
search creating knowledge and answering open knowledge questions;
one applied research paper which answers an open question; six
studies answer closed questions (i.e., hypothesis testing) as a kind
of basic research and one is applied research. In summary, the re-
Fig. 7. Research characterization according to methodology.
search in the DM field is predominantly basic research answering
open questions (see Fig. 4).
Research logic: We found that inductive reasoning leads the re-
search work in DM as 24 studies use induction and only 14 use de- Research purpose: As shown in Fig. 6, we found that half of
duction (see Fig. 5). Note that induction is the prevalent research the research work (19) in DM is descriptive research. Exploratory
logic in both DMBehavior (10 of 13) and DMPractice (14 of 25) pa- and evaluative research are used in seven and eight studies, re-
pers. spectively. Finally, only four studies carry out explanatory work.
In the following we describe the interdependencies between

works with a certain research purpose and their common research
outcomes and logic.
Exploratory research: The seven exploratory studies focus on
exploring and understanding the DMProcess and or the DMFac-
tors involved. To create such an understanding, most of these ap-
proaches aim to understand the perspectives of practitioners. The
main idea is that by observing practitioners or by gaining access
to the perspectives of practitioners experiencing different aspects
of DM, researchers can also gain access to the problems or char-
Fig. 8. Research characterization according to research process.
acteristics of DM. In sum, exploratory research in DM has created
meaning out of what is observed or out of the opinion of practi-
tioners in the field. We found that all exploratory research was ba-
sic research. Furthermore, we found that exploratory research always Behavioral aspects are rarely researched in surveys. Only one
used inductive reasoning. They have inductively created generaliza- paper does explanatory research using a survey. Two of the DM-
tions about the group, process, activity, or situation of DM aspect Practice papers use industrial projects for experiments. This is par-
under study. ticularly interesting as it is difficult to determine the impact of a
Descriptive research: While exploratory research aims to ex- method in a rich context. Real-world case studies are particularly
plore a phenomenon to provide insights for more precise inves- interesting as it is difficult to describe a phenomenon considering
tigation, descriptive research takes the next step and describes a the variations in a real world setting.
phenomenon or characteristic of a problem. All of the 19 descrip- As shown in Fig. 8, the research process was mainly qualitative
tive studies are basic research and the majority creates knowledge (7 DMBehavior and 11 DMPractice) and mixed (6 DMBehavior and
of the DMProcess. In terms of research logic, 14 studies used in- 13 DMPractice) and only one (DMPractice) study was purely quan-
duction to describe DM, whereas only five used deduction to do so. titative. Qualitative data is the staple of the social sciences fields.
The ones using induction took a broader scope, generalized and Not surprisingly, in the software architecture DM research where
then described different aspects of DM. The studies using deduc- humans play an important role, qualitative data is attractive too.
tion however, took a narrower scope and looked at DM through As shown in Fig. 9, eight studies (1 DMBehavior and 7 DMPrac-
the lenses of an existing framework or theory on DM. Those stud- tice) involved student participants only, two studies (1 DMBehav-
ies go from the general — the theory — to the specific — the results ior and 1 DMPractice) involved a mix of students and profession-
that prove/disprove the theory. als. Also, two studies (1 DMBehavior and 1 DMPractice) mixed aca-
Explanatory research: Explanatory research’s investigation is demics and professionals. The rest of the studies (9 DMBehavior
deeper than descriptive research in the sense that explanatory re- and 19 DMPractice) involved professionals only. Thus, 30 studies
search describes phenomena and attempts to explain why a be- give insights into DM in the software industry.
havior is the way it is. This type of research aims at, for instance,
explaining certain ways of DM, or the effects of various factors on
DM. All four explanatory studies carried out basic research and fo-
cused on the DMProcess, of which three used deductive reasoning 6.3. Overview of operational research designs
and one used inductive reasoning.
Evaluative research: Evaluative research aims at determining In this subsection, we show the results with respect to data
the impact of DM approaches, tools, or practices. The research collection and viewpoint. Two thirds of the studies (23) use only
mainly studied the utility of approaches and tools for software one data collection method: In Fig. 10, we show the frequency of
teams, or individuals, and how this utility depends on stakeholder the combination of methods used. Most often the techniques are
goals. Some research works also evaluated the efficiency and effec- used on their own, but also there are many different combinations.
tiveness of the proposed approaches. Two of the eight evaluative Focus group is only used by DMPractice papers, the rest is quite
studies were applied research. The majority (6) of the evaluative evenly spread between DMBehavior and DMPractice
research uses deductive reasoning and few use induction. In the As shown in Fig. 11, predominantly one viewpoint was used (16
studies that use deductive reasoning the evaluation is more spe- only current, 14 only retrospective and 2 only just after the fact).
cific in nature, and typically they are concerned with testing or The other combine the current viewpoint with just after the fact
confirming hypotheses about the effects of use of a certain ap- or retrospective view. Much more DMPractice (12 vs. 4) papers use
proach or tool. The studies that use induction, however, take a the retrospective viewpoint.
broader scope in their evaluations and focus on observing and gen- In the following we compare the use of research methodology
eralizing the utility of the approach in practice. and data collection over time. This is in particular interesting for
Comparing DMBehavior and DMPractice, both are mainly de- the comparison with other disciplines (see Section 7). For our anal-
scriptive (8 vs. 11), few exploratory (3 vs. 4) and few explanatory ysis we grouped the papers into three periods of roughly ten years
(2 vs. 2). As to be expected, all evaluative research focuses on DM- (Fig. 12a and b): the first period starting in 1987 until 1997, the
Practice. second period starting in 1998 and ending in 2007 and the last
period starting in 2008. The papers of the first period use as re-
6.2. Overview of tactical research designs search methodology only case studies, in the middle period also
surveys were added. There are much more papers in the last pe-
In this subsection we show the results with regard to research riod. They use all kinds of methodologies with a focus on surveys
methodology and process and the participants. For the research (see Fig. 12a). For data collection, first only observation and con-
methodology we found that the studies mainly used surveys (3 tent analysis were used, then interviews and questionnaires were
DMBehavior and 9 DMPractice) and limited time case studies (6 applied and in the last period focus groups were added and in-
DMBehavior and 5 DMPractice). In addition, we found eight experi- terviews used most often (see Fig. 12b). Note that some papers
ments (1 DMBehavior and 7 DMPractice) and seven real world case use several research methodologies and data collection methods so
studies (3 DMBehavior and 4 DMPractice) (see Fig. 5a). that the total number exceeds the total number of papers.
Fig. 9. Research characterization by participant type.
Fig. 10. Research data collection methods.
Fig. 11. Research characterization by viewpoint.
Fig. 12. (a) Research methodology used by period; (b) Data collection method used by period.
6.4. Discussion of strategic, tactical and operational research design it is possible to involve practitioners in experimental settings. Data
collection methods used are content analysis (decision outcome),
The results show a big variety of research designs. While there questionnaires, and sometimes also observation by the researchers
is a predominant choice for strategic designs, there is a great va- (facilitated by a tool). The current and the just-after the fact view-
riety on the tactical and operational level. In the following, we points are combined. In some experiments, participants are asked
describe the main designs for the different research methodolo- to evaluate the outcome or report on their approach. Data analy-
gies. Collectively, these variations of research design types can be sis is always quantitative and with one exception also qualitative
reference points for researchers to select an appropriate research as the outcome needs to be coded and the questionnaires include
methodology for their own research. open questions.
Variations of limited time case studies: The research process
is qualitative. Analysis is based on predefined codes, e.g. from deci- 7. Learning from other fields
sion theory or a specific DM approach or general DM activities, or
based on open coding. Some studies use quantitative assessment In Section 4 Study Focus we found that the number of studies
in addition. Tasks can be artificial or real project meetings (only focusing on DMBehavior, i.e. those that focus on the human aspects
once). The task can be to solve a particular problem or to critique of DM, is scant. In this section we highlight a number of points
a given solution. The problem is mostly to create a design or an for research, with the ultimate goal of inspiring and guiding fu-
architecture, but can also be a specific activity like effort estima- ture research on human and behavioral aspects of software archi-
tion. Furthermore, the task can involve a specific technique or tool tecture DM. To do so, we compare the status of empirical architec-
(developed by the researchers) which is being evaluated. Partici- tural DM research with similar research in other domains. As men-
pants come from industry or are students. They work individually tioned in the introduction we refer to Dinar et al. (2015) that gives
or in pairs or larger groups. Data is collected through video, au- an overview of empirical studies of design thinking. They highlight
dio, observation, think aloud, notes, and diagrams. The viewpoint is the difficulties of data collection and analysis, and emphasize the
mainly current. A feedback session just after the task can be used. importance of cognitive studies and empirical research. Our study
The viewpoint can also be retrospective, if the task is to reflect on architectural DM shares similarities to the design thinking study
previous work. as both involve human thinking. We think that the insights pro-
Variations of real world case studies: The research process is vided by Dinar et al. (2015) and other design studies such as Dorst
qualitative. In contrast to limited time case studies the analysis (2011) can provide useful hints to software architecture DMBehavior
is mainly based on open coding. The studied projects last from 6 research. In the following we concentrate on research methodology
weeks to 2 years. Researchers are involved in the projects in the and data collection.
long-term studies. One study involved comparison groups. This is
typically only possible with student participants. Similarly to lim- 7.1. Research methodology for studying DMBehavior
ited time case studies, participants come from industry or are stu-
dents. They work in groups. Data collection in real-life case stud- Studying DMBehavior using experiments. DMBehavior papers
ies is typically less detailed than for limited time case studies. The study how design DM takes place in people’s minds and what fac-
studies used observation, content analysis and interviews or focus tors affect the designers thinking. As shown in Section 6.2, the
groups. The viewpoint is always current with additional interviews majority of these papers use case study designs (both limited
current or just after the fact. time and real world) and only one paper uses experiments. Con-
Variations of surveys: Again, the research process is mainly versely, history of research in design studies and cognitive psychol-
qualitative or mixed. Data analysis uses mostly encoding, and half ogy (Dinar et al., 2015) has shown the necessity of experiments
of these studies apply quantitative analyses in addition. Very few for establishing causality (e.g., between human factors and design
studies use quantitative analyses only. Half of the studies ask the process). Case studies are a great research methodology for iden-
participants to contribute general experiences, the other half ask tifying a certain observation in its real world context. They are,
the participants to describe specific projects or example decisions. however, not the best for explaining the thinking behind design.
The surveys mainly explore how the participants perform DM, but Experiments on the other hand, because they manipulate the in-
one survey prescribes a process to be discussed in the interviews. dependent variables to observe effects on dependent variables, are
Participants are always from practice and one study involves re- strong in explaining what causes certain design DM.
searchers in addition. Three quarters of the studies involve less Dinar et al. (2015) discuss how the empirical studies of de-
than 30 participants, two between 30 and 100, and only one more sign thinking moved from early studies interviewing and observing
than 100. The method of sampling is not clearly reported in most practice to more controlled studies such as experiments. We found
cases. One study reports about a wide search through Google and that experiments were used after the year 2008 (see Fig. 12(a)),
LinkedIn. Three studies mention snowball sampling based on initial but even more surveys have been used. This indicates in our view
personal contacts. Roughly half of the studies use interviews with that also architectural DM studies should advance to more con-
a duration between 30 and 150 min for data collection, the other trolled settings, as in design thinking studies.
half use online questionnaires. The questions are mainly open, Careful design of experiments. Dinar et al. argue that the re-
sometimes closed, and sometimes mixed. Some studies first con- searchers need to be more careful in the following ways:
duct a literature search to derive specific categories to be used in First, it is important to know about the factors that play a role
the questions or codes. The viewpoint is retrospective, except in in the design activities and their interaction. In DMBehavior re-
one study where the participants are asked to consider given deci- search many of those factors are human-related, making the re-
sions. search design of such experiments especially difficult. For instance,
Variations of experiments: The research process is mostly there are differences between individuals; no two human subjects
mixed. Similar to limited time case studies, mostly experimen- have the same background or exact same experience. To control
tal tasks are to perform a part of design exercise, while one ex- this factor in a subgroup of subjects (e.g., expert vs. novices) a
periment studies risk analysis. All but one experiment prescribes large enough sample size is needed so that the backgrounds get
a method or tool. As expected in experiments, most participants randomized and averaged.
are students. One study involves practitioners only and another Second, in treatment vs. control groups the results of the ex-
study involves both practitioners and researchers. This shows that periment should be compared to a base-line such as the con-
trol group that did not have the variable (or the manipulating or it sought to reveal and argued that aspects of designers think-
changing factor). For instance, the control group should follow a ing such as perception and insight were lost in the verbalization
“no method” design task. But for some factors finding the control (Lloyd et al., 2007). One way to address this is to annotate the
group is difficult. For example, if the factor is the level of expe- transcribed data with auxiliary information such as gestures and
rience what would be the control group? We need to define “no sketches. Another way is to collect rough sketches created by de-
experience” in this case. signers to supplement the verbal think aloud data. Many design
Third, the design prompt of the experiments (i.e., the specifi- studies in industrial design or architecture field have accompanied
cation of the problem to be addressed by the participants) needs think aloud protocols with design sketches or design drawings (e.g.
to be carefully designed and reported. It is, for instance, important Suwa and Tversky, 1997) to externalize mental models and ideas
to carefully plan for the complexity of the design prompt as sug- about design. Likewise, in the software architecture field, simple
gested by Pfaff et al. (2013). The reason is that the efficacy of a cer- models of architecture can unravel ideas and mental models in ar-
tain solution is different in complex or simple design problems. To chitect’s minds.
manipulate the complexity of design prompts Pfaff et al., suggest
introducing ambiguity or conflicting requirements in the problem
representation. 8. Conclusion
Forth, not only is the experiment design important but also
the experiment execution. Many of such experiments involve Decision making is an important activity in design, including
some sort of intervention (e.g., asking reflecting questions in software architecture design. DM is a unique human activity in-
Razavian et al., 2016). The interventions and the way they are ap- volving many aspects such as cognition, behaviors and group in-
plied can play a very important role in the quality of the experi- teractions. In software architecture DM research, researchers have
ments. For example, the interventions should be applied in a way investigated both behavior and practice aspects of this activity. In
that feels natural to the designers involved. The design task is also this paper we searched eight different research publication sources,
a critical aspect; it should not be too difficult or too simple for a between 2005 and 2017, for empirical papers on architectural DM.
certain subject or for the dedicated time-span We also used backward snowballing to find other relevant pa-
Complementing experiments with real world case studies. In pers. Using the 38 papers that we found, we investigated four dif-
experiments, the confounding variables such as, educational back- ferent dimensions of research designs covering 13 different sub-
ground, experience, or gender need to be controlled. However, con- dimensions. The results show that there are predominant choices
trolling confounding variables also constrains the applicability of for the study focus, the research cycle focus and strategic design,
DM methods in real world situations. For instance, design teams whilst the choice for the tactical and operational dimensions is
are usually composed of designers with different levels of experi- more diverse. To understand the diversity, we looked at dependen-
ence and expertise and that cannot be controlled practically. Fur- cies and variations in detail.
thermore, observing the effects of individual manipulated variables We introduced two new instruments to position research on
in naturalistic situations can be very difficult. Therefore, we argue software architecture DM: the DM Research Focus Matrix and the
similar to Dinar et al. (2015) that DMBehavior papers, studying the DM Research Cycle. We propose researchers to use these two in-
role of a human factor on decision tasks, should also consider the struments to articulate the foci and objectives of their work and
complexities in real world settings. Thus, on the one hand more clarify their study position with a big picture view of the software
experiments in DMBehavior studies should be performed, but on architecture DM research scene.
the other hand those studies need to be followed-up by case stud- With these instruments we identified several gaps in empirical
ies to ensure the applicability of the results. software architecture DM research: There are few studies on be-
havioral aspects, and in particular we found very few studies that
7.2. Data collection methods for studying DMBehavior consider the factors and the outcome of DM behavior. This means
we know very little about the behavioral aspects of software ar-
Studying DMBehavior using think-aloud protocols. As seen in chitecture DM. Furthermore, there are many problem investigation
Fig. 12b, the majority of papers used content analysis, observation, studies, but very few contemplated new treatments or validating
and questionnaires to collect data on DM. We argue, however, that those treatments, and none looking at the behavioral aspects in
content analysis and questionnaires are not sufficient to reveal the context of new treatments. This implies a lack of insights on pos-
details and complexities of the DMBehavior. The DM of the archi- sible improvements of software architecture DM. These research
tects, to a great extent, takes place in their minds. The use of con- gaps need to be filled. We sketch insights from other disciplines
tent and questionnaires are inadequate to capture their thinking on studying behavioral aspects.
patterns. For instance, questionnaires can only elicit what archi- In conclusion, our work makes the following contributions (a)
tects should do or some levels of agreement and disagreement. The researchers can better understand their research design choices
respondents (e.g., the architects) most probably provide the most and position their works accordingly; (b) given the instruments
“acceptable or appropriate answer”. This might only reflect what that we proposed, researchers can map their DM research works,
they found appropriate in DM. If researchers are interested to un- and build their studies on each other’s results for a more coherent
derstand the process involved in DM or the way how the architects progress in the research on architectural DM. This study has pro-
work, they need to conduct a more in-depth data collection. vided the basis of a systematic approach for researchers to design
To discover the designers thinking patterns, social sciences empirical studies that embrace human behavior in software archi-
studies mainly used think-aloud protocols and protocol analysis tecture decision making.
(Dinar et al., 2015; Mehalik and Schunn, 2007). Think-aloud pro-
tocols ask designers to verbalize their thinking process, uncov-
ering their thought process. Combined with direct observation Appendix A. Selection of decision making literature
they help to study how designers actually perform the design
task. Usually such observations are transcribed and codified. While [S3] K. Børte, S. R. Ludvigsen, and A. I. Mørch, "The role of so-
the protocol analysis can be quite powerful, it has limitations as cial interaction in software effort estimation: Unpacking the “magic
well. Researchers in design studies questioned the effectiveness of step” between reasoning and decision-making," Information and
think aloud protocols to elicit information about the very design Software Technology, vol. 54, pp. 985–996, 2012.
[S7] H. Christiaans and R. A. Almendra, "Accessing decision- [S51] C. Miesbauer and R. Weinreich, "Classification of Design
making in software design," Design Studies, vol. 31, pp. 641–662, Decisions–An Expert Survey in Practice," in Software Architecture,
2010. ed: Springer, 2013, pp. 130–145.
[S9] E. Conklin and K. C. Burgess-Yakemovic, "A Process- [S52] N. B. Moe, A. Aurum, and T. Dybå, "Challenges of shared
Oriented Approach to Design Rationale, Human Computer Interac- decision-making: A multiple case study of agile software develop-
tion, Vol 6, 1991, pp. 357–391. ment," Information and Software Technology, vol. 54, pp. 853–865,
[S14] M. Drury, K. Conboy, and K. Power, "Decision making in 2012.
agile development: A focus group study of decisions and obsta- [S53] R. Mohanani, P. Ralph, and B. Shreeve, "Requirements fixa-
cles," in Agile Conference (AGILE), 2011, 2011, pp. 39–47. tion," in Proceedings of the 36th International Conference on Soft-
[S15] M. Drury, K. Conboy, and K. Power, "Obstacles to decision ware Engineering, 2014, pp. 895–906.
making in Agile software development teams," Journal of Systems [S60] V. Smrithi Rekha and H. Muccini, "A study on group
and Software, vol. 85, pp. 1239–1254, 2012. decision-making in software architecture," in Software Architecture
[S19] D. Falessi, G. Cantone, and P. Kruchten, "Value-based de- (WICSA), 2014 IEEE/IFIP Conference on, 2014, pp. 185–194.
sign decision rationale documentation: Principles and empirical [S61] M. Soliman, M. Riebisch, and U. Zdun, "Enriching Archi-
feasibility study," in Software Architecture, 20 08. WICSA 20 08. Sev- tecture Knowledge with Technology Design Decisions," in Software
enth Working IEEE/IFIP Conference on, 2008, pp. 189–198. Architecture (WICSA), 2015 12th Working IEEE/IFIP Conference on,
[S21] P. Gaubatz, I. Lytra, and U. Zdun, "Automatic enforcement 2015, pp. 135–144.
of constraints in real-time collaborative architectural decision mak- [S62] S. Sonnentag, "Expertise in Professional Software Design:
ing," Journal of Systems and Software, vol. 103, pp. 128–149, 2015. A Process Study," Journal of Applied Psychology, vol. 83, pp. 703–
[S26] I. Groher and R. Weinreich, "A Study on Architectural 715, 1998.
Decision-Making in Context," in Software Architecture (WICSA), [S66] A. Tang, A. Aleti, J. Burge, and H. van Vliet, "What makes
2015 12th Working IEEE/IFIP Conference on, 2015, pp. 11–20. software design effective?," Design Studies, vol. 31, pp. 614–640,
[S27] R. Guindon and B. Curtis, "Control of cognitive processes 2010.
during software design: what tools are needed?," presented at the [S67] A. Tang, M. A. Babar, I. Gorton, and J. Han, "A survey of the
Proceedings of the SIGCHI conference on Human factors in com- use and documentation of architecture design rationale," in Soft-
puting systems, Washington, D.C., United States, 1988. ware Architecture, 20 05. WICSA 20 05. 5th Working IEEE/IFIP Con-
[S28] R. Guindon, H. Krasner, and B. Curtis, "Breakdowns and ference on, 2005, pp. 89–98.
processes during the early activities of software design by profes- [S69] A. Tang, M. H. Tran, J. Han, and H. van Vliet, "Design Rea-
sionals," in Empirical studies of programmers: second workshop, soning Improves Software Design Quality," in Proceedings of the
M. O. Gary, S. Sylvia, and S. Elliot, Eds., ed: Ablex Publishing Corp., Quality of Software-Architectures (QoSA 2008), 2008, pp. 28–42.
1987, pp. 65–82. [S70] A. Tang and H. van Vliet, "Software Designers Satisfice,"
[S30] U. van Heesch and P. Avgeriou, "Mature architecting-a sur- in Software Architecture. vol. 9278, D. Weyns, R. Mirandola, and I.
vey about the reasoning process of professional architects," in Soft- Crnkovic, Eds., ed: Springer International Publishing, 2015, pp. 105–
ware Architecture (WICSA), 2011 9th Working IEEE/IFIP Conference 120.
on, 2011, pp. 260–269. [S73] D. Tofan, M. Galster, and P. Avgeriou, "Difficulty of archi-
[S31] U. van Heesch and P. Avgeriou, "Naive Architecting- tectural decisions–a survey with professional architects," in Euro-
Understanding the Reasoning Process of Students," in Software Ar- pean Conference on Software Architecture, 2013, pp. 192–199.
chitecture, ed: Springer, 2010, pp. 24–37. [S77] R. Weinreich, I. Groher, and C. Miesbauer, "An expert sur-
[S32] U. Van Heesch, P. Avgeriou, and R. Hilliard, "Forces vey on kinds, influence factors and documentation of design de-
on architecture decisions-a viewpoint," in Software Architec- cisions in practice," Future Generation Computer Systems, vol. 47,
ture (WICSA) and European Conference on Software Architecture pp. 145–160, 2015.
(ECSA), 2012 Joint Working IEEE/IFIP Conference on, 2012, pp. 101– [S80] C. Zannier, M. Chiasson, and F. Maurer, "A model of de-
110. sign decision making based on empirical results of interviews with
[S33] U. van Heesch, P. Avgeriou, and A. Tang, "Does decision software designers," Information and Software Technology, vol. 49,
documentation help junior designers rationalize their decisions? A pp. 637–653, 2007.
comparative multiple-case study," Journal of Systems and Software, [S81] C. Zannier and F. Maurer, "Social Factors Relevant to Cap-
vol. 86, pp. 1545–1565, 2013. turing Design Decisions," in Proceedings of the Second Workshop
[S35] J. F. Hoorn, R. Farenhorst, P. Lago, and H. Van Vliet, "The on SHAring and Reusing architectural Knowledge Architecture, Ra-
lonesome architect," Journal of Systems and Software, vol. 84, pp. tionale, and Design Intent, 2007, p. 1.
1424–1435, 2011. [S90] D. Tofan, M. Galster, I. Lytra, P. Avgeriou, U. Zdun, M.-A.
[S39] M. Keil, L. Li, L. Mathiassen, and G. Zheng, "The influ- Fouche, et al., "Empirical evaluation of a process to increase con-
ence of checklists and roles on software practitioner risk percep- sensus in group architectural decision making," Information and
tion and decision-making," Journal of Systems and Software, vol. Software Technology, vol. 72, pp. 31–47, 2016.
81, pp. 908–919, 2008. [S91] T.-M. Hesse, V. Lerche, M. Seiler, K. Knoess, and B. Paech,
[S46] I. Lytra, P. Gaubatz, and U. Zdun, "Two controlled experi- "Documented decision-making strategies and decision knowledge
ments on model-based architectural decision making," Information in open source projects: An empirical study on Firefox issue re-
and Software Technology, vol. 63, pp. 58–75, 2015. ports," Information and Software Technology, vol. 79, pp. 36–51,
[S47] I. Lytra, S. Sobernig, and U. Zdun, "Architectural decision 2016.
making for service-based platform integration: A qualitative multi- [S92] M. L. Drury-Grogan, K. Conboy, and T. Acton, "Examin-
method study," in Software Architecture (WICSA) and European ing decision characteristics & challenges for agile software devel-
Conference on Software Architecture (ECSA), 2012 Joint Working opment," Journal of Systems and Software, vol. 131, pp. 248–265,
IEEE/IFIP Conference on, 2012, pp. 111–120. 2017.
[S49] A. MacLean, R. M. Young, V. M. Bellotti, and T. P. Moran, [S93] A. Zalewski, K. Borowa, and A. Ratkowski, "On cognitive
"Questions, options, and criteria: Elements of design space analy- biases in architecture decision making," in European Conference on
sis," Human–computer interaction, vol. 6, pp. 201–250, 1991. Software Architecture, 2017, pp. 123–137.
[S94] C. Schriek, J. M. E. M. van der Werf, A. Tang, and F. Bex, Appendix C. Details of decision making research studies
"Software Architecture Design Reasoning: A Card Game to Help
Novice Designers," in European Conference on Software Architec- Details of strategy level
ture, 2016, pp. 22–38.
In the following we describe the strategic research designs of
the papers in more detail. We cluster the papers first according to
their research purpose and second according to their research data
Appendix B. Papers and their codes
focus.
Table B1 and Tabel B2
Table B1
Abbreviations for the Resesrach Cycle, Strategic, Tactical and Operational Design Coding.
Dimension Sub-dimension Codes
Research Cycle Focus Basic Research for Problem Investigation Study DMBehavior (ST1a), Observe DMPractice (ST1b)
Basic Research for Treatment Design Treatment Design relating DMBehavior and DMPractice (ST2a), Treatment
Design only DMPractice (ST2b),
Applied Research for Treatment Validation Treatment Validation relating DMBehavior and DMPractice (ST3a), Treatment
Validation only DMPractice (ST3b)
Strategic Design Research Outcome Basic research (b), Applied research (a), Open Question (o), Closed Question (cl)
Research Logic Inductive (i), Deductive (d)
Research Purpose Explanatory (pla), Descriptive (des), Exploratory (plo), Evaluative (eval)
Tactic and Operational Design Research Methodology Real world case study, (caser) Limited time case study (casel), Experiment
(exp), Survey (sur)
Research Process Qualitative (l), Quantitative (n), Mixed (m)
Participants Students (s), Academics (a), Profession (p)
Data Collection Interview (int), Questionnaires (quest), Focus Groups (group), Observation
(Obs), Content analysis (content)
Viewpoint Retrospective (ret), Current (cur), Just after the Fact (jaf)
Table B2
Coding of papers.
Paper Year DM focus RD focus DM Research Research Research Research Research research Data Data Participants
cycle outcome purpose logic methodology process viewpoint collection
S14 2011 DMPractice DMFactors ST1b b-op plo i casel l ret group p
S15 2012 DMPractice DMFactors ST1b b-op plo i casel l ret int, group p
S19 2008 DMPractice DMOutcome ST2b b-op eva d exp n jaf quest s
S21 2015 DMPractice DMOutcome, DMProcess ST2b b-op eva d exp m cur cont, obs s
S26 2015 DMPractice DMFactors ST1b b-op des i sur l ret int p
S27 1988 DMPractice DMProcess ST1b b-op des d casel l cur obs p
S28 1987 DMBehavior DMProcess ST1a b-op plo i casel l cur obs p
S3 2012 DMBehavior DMProcess ST1a b-op des i casel l cur obs p
S30 2011 DMPractice DMProcess ST1b b-op des i sur m ret quest p
S31 2010 DMPractice DMProcess ST1b b-op des i casel m cur, jaf cont, quest s
S32 2012 DMPractice DMOutcome ST3 b-op eva i caser l cur cont, group, s
obs
S33 2013 DMPractice DMProcess ST2b b-cl des d caser l cur cont, group, s
obs
S35 2011 DMPractice DMProcess ST1b b-cl des d sur m ret quest p
S39 2008 DMPractice DMProcess ST2b b-cl eva d exp m cur cont, quest p
S46 2015 DMPractice DMOutcome ST2b b-cl eva d exp m cur cont, obs s
S47 2012 DMPractice DMOutcome ST2b b-op des i sur l cur, ret int p
S49 1991 DMPractice DMProcess ST2b b-op pla d casel l cur obs p
S51 2013 DMPractice DMFactors, DMOutcome, ST1b b-op des i sur l ret int p
DMProcess
S52 2012 DMBehavior DMFactors ST1a b-op des i caser l cur, jaf cont, int,obs p
S53 2014 DMBehavior DMProcess ST1a b-cl pla d exp m jaf cont, quest s
S60 2014 DMBehavior DMProcess ST1a b-op plo i sur m ret quest a, p
S61 2015 DMPractice DMProcess ST2b b-op plo i sur m ret int p
S62 1998 DMBehavior DMProcess ST1a b-op pla d casel m cur cont, obs p
S66 2010 DMBehavior DMProcess ST1a b-op des i casel m cur obs p
S67 2005 DMPractice DMProcess ST1b b-op plo i sur m ret quest p
S69 2008 DMPractice DMProcess ST2b b-op eva d exp m cur, jaf cont, int,obs a, p
S7 2010 DMBehavior DMProcess ST1a b-op des d casel l cur obs p
S70 2015 DMBehavior DMProcess ST1a b-op plo i casel m cur, jaf cont, obs, s, p
quest
S73 2013 DMPractice DMFactors, DMOutcome ST1b b-op des i sur m ret quest p
S77 2015 DMPractice DMFactors, DMOutcome, ST1b b-op des i sur l ret int p
DMProcess
S80 2007 DMBehavior DMProcess ST1a b-op des i sur l ret int p
S81 2007 DMBehavior DMProcess ST1a b-op des i caser l cur, ret int, obs p
S9 1991 DMPractice DMProcess ST3b a-op eva i caser m cur cont p
S90 2016 DMPractice DMProcess ST3b a-cl eva d exp m cur cont s,p
S91 2016 DMBehavior DMFactors, DMOutcome, ST1a b-cl des i caser m ret cont p
DMProcess
S92 2017 DMPractice DMProcess ST1b b-op des d caser l ret int, cont, p
group, obs
S93 2017 DMBehavior DMFactors ST1a b-op des i sur l cur obs, quest p
S94 2016 DMPractice DMProcess ST1b b-op pla i exp m cur cont s
Exploratory research cision making. From a survey, researchers extract and describe the
We only found exploratory research for the two categories of various biases encountered by the participants.
DMFactors and DMProcess. It should be noted that no exploratory DMProcess.S3 as a basic research creates an understanding of
study focused on the DMOutcome. how software professionals in groups invoke knowledge in their
DMFactors. S14 and S15 are exploratory studies that focus on communication, reasoning and DM for software effort estimation.
the factors influencing DM of 43 practitioners in agile software de- S3 uses observations in 3 effort estimation meetings in industry,
velopment. Using inductive reasoning, the researchers generalize which used planning poker and uses inductive reasoning and gen-
decision-making issues according to the experience of the practi- eralizes ways of invoking knowledge by software professionals. S7
tioners. This research work is a basic research that creates knowl- characterize the team behavior in DM and compares it with the
edge about process and issues in agile software development. behavior of the product designers. This basic research creates un-
DMProcess. S28 is a basic research that creates understanding derstanding about the early design phase of design exploration.
about the early activities of design as well as the related design is- To come up to such understanding, S7 uses a descriptive frame-
sues. The designers are given a lift control case as a design assign- work of decision-making (Almendra and Christiaans, 2009) to de-
ment. This research generalizes major issues when designers tack- ductively analyze and describes the designer’s behaviors. S27 de-
led the lift simulation program based on the video record of their fines and creates understanding about the design process strate-
design in action (inductive reasoning). S60 is a basic research that gies using verbal protocol study of professional software designers
creates knowledge about group decision-making. Using inductive (basic research). A cognitive model of software design is used to
reasoning, the researchers generalize mechanisms and technique deductively analyze the observed strategies. S80 defines the rela-
used and challenges involved in shared DM based on researchers tion of Rational DM (RDM) to Naturalistic Decision-Making (NDM).
experience in the software architecture community. S61 aims at NDM was characterized in terms of decision goal, method, environ-
exploring how technology solutions are used in architecting. This ment and knowledge. This basic research work uses inductive rea-
basic research uses induction and combination of literature study, soning to generalize the use of decision-making strategies. S81 de-
survey and interviews to come up to such understanding. S70 is scribes how designers capture decisions and make decisions from
a basic research that investigated the extent to which students different perspectives. This is a basic research that creates under-
and professionals look for alternatives in DM. The researchers in- standing about social factors involved in the designer’s DM. Those
ductively generalized the DMBehavior of students and profession- factors are elicited using inductive reasoning, by generalizing from
als and found satisficing behavior in most of the subjects. S67 is the qualitative data of 3 case studies. S30 also is a basic research
a basic research that creates knowledge about how practitioners that creates understanding about how they reason in real projects.
think about and reason about design decision and design rationale Inductive reasoning is used to define designer’s reasoning using a
practices. Based on a survey the common types of design rationale survey. Likewise, S31 is a basic research about naive reasoning for
were identified using inductive reasoning. architecture DM of undergraduate students. Inductive reasoning is
again used for generalizing and characterizing the naive reasoning.
Descriptive research S35 carried out basic research and investigated knowledge sharing
We found descriptive research for the three categories of DM- between software architects and architecting activities. The study
Factors, DMProcess and DMOutcome. uses survey with architects. To come up to such understanding, S35
DMFactors. S26 is a basic research that creates an understand- uses a framework of architectural activities to deductively analyze
ing about factors in organization. Based on interviews with 25 sys- and describes the knowledge sharing and other architectural ac-
tem analysts, team leads and senior developers S26 inductively tivities. S66 carried out basic research on how software designers
elicits 8 factors that influence DM: company size; business factors; explore the problem and solution space and the role of reasoning
organizational factors; technical factors; cultural factors; individ- in DM. This was done inductively by observing designers who were
ual factors; project factors; and decision scope. The main differ- given a task of designing traffic simulator. S51 described software
ence between such descriptive research and previously discussed architects during their DMProcess. This is a basic research that in-
exploratory studies is that exploratory studies aim to explore a ductively generalized the way decisions are made. For instance, lo-
decision making phenomenon in general and provide insights for cal scoped decisions such as a component are typically made by
more precise investigation, descriptive research takes the next step individuals but architectural decisions are made by a team. S92
and describes a phenomenon or characteristics of a problem. For explores in a case study the failings of decision making in an ag-
example, S14 is an explorative study that explores effectiveness of ile development environment. The research considers decision pro-
decision-making in agile teams. Participants were asked to discuss cess, information intelligence used in decision making, and deci-
topics in importance of decision-making and perception of obsta- sion quality.
cles. S26, however, is a descriptive research that elicits and de- DMOutcome. S77 creates an understanding about the categories
scribes categories of decision making factor. The purpose of S14 of decisions and their documentation. This basic research, induc-
is to generally explore obstacles of decision making in agile teams, tively, extracts such categories based on interviews with practition-
whereas S26 aims at describing and characterizing the influence ers. Categorizing the decisions is carried out according to granu-
factors of the decision making in the agile teams. larity, scope and impact. Likewise, S51 categorizes decisions into
S52 describes the challenges of shared decision-making in ag- structural decisions and technology decisions. S47 defines patterns
ile software development teams. This basic research uses induc- for service-based integration based on a systematic literature re-
tion to generalize challenges from concrete observations. S73 is a view. The identified patterns are grouped into four decision levels-
basic research characterizing factors of difficulty for architectural architecture, platform, integration and application. Such grouping
design decisions following inductive reasoning. S77 and S51 (S77 is carried out inductively.
is follow-up of S51) carry out basic research that elicits DM in-
fluence factors, i.e., personal experiences, requirements and quality Explanatory research
attributes, followed by constraints and cost/benefit. These factors We only found explanatory research focusing on DMProcess.
are identified inductively based on interviews with practitioners. DMProcess. S49 explains how capturing Question, Option and
S91 examines different types of decision-making strategy elements Criteria (QOC) can affect the design DM. This is a basic research
and decision knowledge elements from examining issue reports. creating understanding QOC could be useful. The research logic is
S93 studies the different types of cognitive biases in software de- based on deduction since the analysis is carried out based on the
core elements of QOC (Question, Option and Criteria). S53 explains decision-making approach (QOC) was used in the analysis to un-
how framing/presentation of requirements affects design process derstand how this approach could be useful. In 1998 the lift con-
and design outcome. This basic research carried out deductive rea- trol problem was used again in S62. This time 40 professionals se-
soning by hypothesis testing. S62 study the cognitive character- lected through peer-nomination were asked to solve the task in
istics of high software design performers and explains how such 120 min. Data collection again involved think-aloud protocols and
characteristics affect the design and design outcome. Here also de- notes from the participants. The data was analyzed w.r.t. decision-
ductive reasoning was used. S94 explains how reminder cards can making activities to understand differences between high- and low
help designers less prone to satisficing behavior and improve de- performers. This involved also quantification. In 2010 3 pairs of de-
sign reasoning and discourse. The explanations lie in the experi- signers were given a task (S66). This time it was a traffic simulator.
mental process that the researchers conducted and in the behav- Data was collected through video, but also through feedback ses-
iors of the subjects. sion with the pairs. Analysis focused on general design activities,
but also used part of a specific approach (SEURAT) to code details.
Evaluative research The same material was used in S7 to compare software design-
We found evaluative research for the two categories of DMOut- ers with product designers. In S31 (again in 2010) the task was to
come and DMProcess. exploratoryFurthermore we capture in detail design an architecture in 60 min. In this study participants were
the qualities that were evaluated for each proposed approach or students. They were asked to make design decision explicit. Their
tool. behavior was captured by a mind-map and a follow-up interview
DMOutcome. S19 introduced valued-based documentation of just after the task. Thus, the viewpoint was cur and jaf.
design rationale. The proposed approach is validated using con- Besides general decision-making the limited time case study
trolled experiments. The students had to rate the usefulness of the setting is also used to study particular decision situations. S3
elements of the documentation after making the decision. The re- (2012) studied 3 effort estimation meetings in industry (2 h each)
searchers evaluated if the proposed approach is feasible and how which used planning poker. As these were real projects data col-
useful the elements of the documentation after making the deci- lection was limited to observation. Analysis looked at selected se-
sion. S21 proposed a meta-model to capture DM constraints with quences to understand detailed behavior.
defined semantics and a collaborative architecture DM approach. S14 (2011) and its follow up S15 (2012) describe a study which
The researchers tested the approach and the tool (CoCoADviSE) used a focus group. 43 participants were acquired during a confer-
with controlled experiment. The evaluation assesses the complete- ence. At first they collected individually examples for design deci-
ness and violations of constraints. They also assess time and effort sion and then these were discussed in the group. Open and axial
related efficiency, as well as the effectiveness of users in collabora- coding based on descriptive decision theory was used for analysis.
tive DM. This study was then followed by mini-case studies in agile teams
DMProcess. S32 suggested the use of descriptive forces view- (S15). These teams had specific obstacles which had been identi-
point for architectural decisions. The researchers evaluated the util- fied in S14 and were selected to be studied in depth. The team
ity of the viewpoint by assessing how it supports the DMProcess. experiences were captured through interviews to understand these
S39 assessed the influence of risk checklists and their role on risk obstacles. Thus, the viewpoint was ret.
perception and decision-making of software practitioners. S46 re- S70 (2015) looked at the amount of design reasoning performed
ported two other experiments on CoCoADviSE using students. This before making a decision. 32 students and 40 professionals partic-
study assessed the utility of CoCoADviSE by answering what the ipated in the study. The task (presented online) was to create rea-
value of reusable architectural design decisions are. S69 explore sons in reaction to 10 design scenarios based on some actual sys-
the effects of design reasoning on the quality of design by com- tem cases (without time limit). The professional were also required
paring two groups of practitioners. S9 presented gIBIS as a hyper- to complete a questionnaire on their approach to the task (with
text tool, together with a notation itIBIS. Using a case study, the open and closed questions). In the first round the students and 29
researchers compared and reviewed how design rationale might professionals completed the task without the questionnaire. In the
make design DM more rigorous and error free. S90 reported an second round 11 professionals did the task and the questionnaire.
evaluation of a method, GADGET, to increase consensus in group The data collected were kind and number of the reasons, the time
decision making. The research was conducted using students and and the answers to the questionnaire, the viewpoint therefore cur
the researchers found that GADGET clarifies different points of and jaf. Analysis was both qualitative and quantitative.
views and increases consensus.
Real world case studies
There are 6 papers solely about real world case studies and one
Details of tactical and operational level paper (S81) combining real world case study and survey. We count
it mainly as a real world case study.
In the following we discuss the papers grouped according to The first study is from 1991 (S9) where IBIS was used in a two-
research methodology and sorted according to publication year. year field trial during requirements analysis and design. Data col-
lection relied on content analysis. Data analysis is not clearly de-
Limited time case studies scribed. The results include facts like identified design errors as
There are 11 limited time case studies. Most of them give a spe- well as overall assessments of the use of IBIS.
cific task (typical duration 60–120 min) to the participants, cap- The other studies are multiple case-studies. S81 (2007) focuses
ture the behavior and analyze it to understand some aspect of on social aspects of DM and their influence on tools. Here 3 stud-
decision-making. The first study is from 1987 (S28,) were 8 pro- ies with slightly different focus are reported somewhat shortly. The
fessional designers were given the lift control problem to design first study interviewed 25 designers about previous work. The sec-
a solution. Their behavior was captured with think aloud proto- ond study comprised 9–10 day observations at 3 different compa-
cols, video and diagrams. This was analyzed on the basis of a spe- nies. In the third study the researchers participated in design dis-
cific cognitive model w.r.t. knowledge resources. A similar study cussions at another company for 6 weeks. The collected data was
was done 1991 were 2 pairs of designers jointly worked on the coded with slightly differing focus.
ATM problem in 45 min (S49). The task was to critique a given In S52 (2012) 4 projects from 2 companies were studied to un-
solution. Again video was used for data collection. Here a specific derstand challenges of shared decision-making in agile develop-
ment. The projects lasted 1–3 years and used 2–4 weeks sprints. ysis used several statistical methods. The results were sent to the
Data was collected during 11–12 months through observation and participants asking for general feedback.
content analysis of documentation. Furthermore, all projects par- The interviews (100–150 min) of S47 (2012) involved 9 archi-
ticipants were interviewed at the beginning and were also involved tects (convenience sample?) with the idea to evaluate a set of ar-
in discussions after observations. All the data was coded and ana- chitectural patterns for platform integration. These patterns were
lyzed to identify the challenges. later also applied in a research project with industry. The questions
S32 and S33 (2012 and 2013) report on a comparative multi- asked for an explicit architectural assessment of an application and
ple case study where 4 teams of senior students (21 altogether) relations to prior DMProcesses. Thus the viewpoint was cur and ret.
worked on 4 different projects (duration 20 weeks). Two of the The qualitative data analysis is not described in detail.
teams (projects) had the additional task to use the decision forces S51 (2013) is based on (30–60 min) interviews with 9 experts
viewpoint for documentation so that the effects of this tech- from 6 companies. It aims to understand the kinds and influences
nique could be assessed. Data collection was quite comprehen- on design decisions as well as the process and documentation.
sive through questionnaire (participants data), work diaries, con- Questions were open and then coded based on predefined cate-
tent analysis of documents and weekly focus groups. The data was gories corresponding to the research questions. The participants re-
analyzed through grounded theory. The codes were then related ported general experiences (not a specific project), but were asked
to predefined response variables characterizing design reasoning. to give examples.
S91 (2016) analyze issue reports from Firefox development. The S73 (2013) reports on a questionnaire about characteristics and
information was coded 2 trained coders. The coded results were difficulties of architectural decisions with 43 answers from 23
used to identify NDM/RDM decision making. S92 (2017) report countries on 5 continents. The participants were acquired through
on a case study conducted with practitioners through interviews, LinkedIn and Google. As for S51 participants were asked to give
focus-groups, team meeting observations and document analysis. examples for good and bad decisions and then to describe them in
The data was coded through open coding and axial coding tech- detail. This included the assessment (Likert scale) of 22 statements
niques. The analysis focus on decision process, decision intelligence on the decision and the DMProcess. Thus, qualitative and quantita-
and decision quality. tive analysis was applied. The participants were grouped into ju-
nior and senior architects and their answers compared. Similarly,
assessments for good and bad decisions were compared.
Surveys S60 (2014) distributed an online- questionnaire on group-DM
There are 11 papers solely on surveys and one paper (S81 de- (practices, techniques and challenges) to 23 practitioners and 7 re-
scribed before) combining real world case study and survey. S67 searchers involved in industrial projects. The participants answered
from 2005 and S80 from 2007 are the earliest ones in our sample. based on general experience. The questions were partly multiple-
S67 focused on the use and documentation of architecture de- choice and partly open leading to both qualitative and quantitative
sign rationale. It analyzed 81 responses from practitioners (se- analysis. The answers were clustered and summaries derived for
lected by availability and snowball sampling) to an online ques- clusters.
tionnaire. The questionnaire presented different kinds of generic S77 (2015) reports on (30–80 min) interviews with 25 experts
design rationale and asked for a rating w.r.t. importance, usage, from 22 companies in 10 countries. The open questions focused
documentation (including barriers and methods and tools) and for (similar to S51) on design decisions (kinds, documentation and in-
additions. The answers used Likert scale enabling a quantitative fluences). Participants reported based on general experience. Codes
evaluation. The practitioners answered based on their experiences, were derived from the research questions and then refined during
but not with a special project in mind. coding. Analysis used structuring and summarization.
In S80 25 practitioners (selected by availability and snowball S26 (2015) re-analyzed the interviews of S77 w.r.t. the decision-
sampling, involving participants of a conference) were interviewed making process
(roughly 45 min) about their design DM, in particular cues, knowl- S61 (2015) focuses on technology decisions. 7 experts were se-
edge, options, experience, time pressure and external goals. The lected. (60–120 min) Interviews were hold two days where first
answers were coded and analyzed through content analysis and experiences were asked to give examples regarding several con-
explanation building in order to derive a model on DM in soft- cepts derived from literature. On the second day the refined con-
ware design. The interviews were structured according to the crit- cepts were discussed and rated by the participants. The answers of
ical decision method. The practitioners reported about specific ex- the first day were analyzed to refine the concepts and are reported
periences, but it is not clear, whether this focused on a particular as quotes. The ratings of the second day were quantitatively ana-
project. lyzed.
The other surveys were conducted between 2011 and 2015.
S30 (2011) presents a descriptive survey with 53 answers on ar- Experiments
chitectural DM, in particular architectural analysis (problem space), There are 8 experiment studies, 3 in 2008 and 3 in 2014 or
architectural synthesis (solution proposal) and architectural evalu- 2015, and 2 in 2016.
ation (solution selection). The participants were selected from in- S19 (2008) reports on an experiment to evaluate the feasibil-
dustry by snowball sampling. They were asked to think of one spe- ity of a tailored design decision rational documentation. The ex-
cific project. As for S67 an online questionnaire with mostly Likert periment project was a public transportation system. 50 master
scale and quantitative analysis was used. Two questions were open students were assigned different roles and decisions and had to
and the answers were coded. perform different scenarios for use of the documentation. The stu-
S35 (2011) reports on a large-scale survey with 142 answers to dents had to rate the usefulness of the elements of the documen-
an online questionnaire focusing on knowledge sharing between tation after making the decision. This rating was collected and an-
architectures. The participants were recruited from 4 companies. alyzed quantitatively. Thus, the viewpoint is jaf.
As input to the study a model of architectural activities and sup- S39 (2008) reports a role playing experiment with 128 software
port methods for knowledge sharing was used. The questionnaire practitioners from 4 companies to study the influence of check-
was very carefully designed with a pilot study and statistical anal- lists and roles on risk perception and DM. The experiment project
ysis. It presented items and asked for agreement on a Likert scale was an online bank system development. There were 4 treatments
based on the general experience of the participants. Also the anal- groups (with or without role resp checklist). Through a website
participants were confronted with project scenarios and asked to Available at: http://www. iasdr2009. org/ap/Papers/Orally% 20Presente d% 20Pa-
assess risks and decide about continuation. Thus, the data collec- pers/Design% 20Method/Improving% 20Design% 20Processes% 20through% 20better%
20DecisionMakin g.
tion viewpoint was cur. The risks determined without checklist Babar, M.A., Dingsøyr, T., Lago, P., v. Vliet, H., 2009. Software Architecture Knowledge
were coded. Thus, data analysis was quantitative and qualitative. Management: Theory and Practice. Springer-Verlag, Berlin Heidelberg p.^pp.
S69 (2008) reports an experiment on the effects of design rea- Pages.
Björklund, T.A., 2013. Initial mental representations of design problems: differences
soning on design quality. 20 designers from industry and academia between experts and novices. Des. Stud. 34, 135–160.
were asked to design a user interface, one group with and the Burge, J., Brown, D.C., 20 0 0. Reasoning with design rationale. In: Artificial Intelli-
other group without a predefined design reasoning process. The gence in Design’00. Springer, pp. 611–629.
Capilla, R., Jansen, A., Tang, A., Avgeriou, P., Babar, M.A., 2016. 10 Years of software
experiment project was a monitoring system for cars developed in
architecture knowledge management: practice and future. J. Syst. Softw. 116,
industry. Data was collected based on think-aloud protocols during 191–205.
design and retrospective think aloud protocols on the derived de- Carroll, J., Johnson, E., 1990. Decision Research: A Field Guide. Sage.
Creswell, J.W., 2013. Research Design: Qualitative, Quantitative, and Mixed Methods
signs by the participants, as well as observation and rating of the
Approaches. Sage Publications.
design through the researchers. Also time was measured and the Dinar, M., Shah, J.J., Cagan, J., Leifer, L., Linsey, J., Smith, S.M., et al., 2015. Empir-
interviews after the task included closed question on design qual- ical studies of designer thinking: past, present, and future. J. Mech. Des. 137
ity. Thus, data collection viewpoint was both cur and jaf and both 021101–021101-13.
Dorst, K., 2011. The core of ‘design thinking’ and its application. Des. Stud. 32,
qualitative and quantitative analysis was applied. 521–532.
S53 (2014) involved 42 post-graduate students in a random- Easterbrook, S., Singer, J., Storey, M.-A., Damian, D., 2008. Selecting empirical meth-
ized control experiment to study the influence of the framing re- ods for software engineering research. In: Guide to Advanced Empirical Soft-
ware Engineering. Springer, pp. 285–311.
quirements vs. ideas on design creativity. The experiment project Galster, M., Weyns, D., Tang, A., Kazman, R., Mirakhorli, M., 2018. From craft to sci-
was an existing health-related App. The participants were asked to ence: the road ahead for empirical software engineering research. International
write down designs. The time given was 60 min. Also a post-task Conference on Software Engineering.
Galster, M., Weyns, D., 2016. Empirical, Research in Software Architecture-How far
questionnaire was applied. Thus data collection was based on con- have we come? In: Proceedings of the 13th Working IEEE/IFIP Conference on
tent (cur by the researchers) and interview (jaf by the participants) Software Architecture, pp. 1–8.
and data analysis qualitative and quantitative. Jalali, S., Wohlin, C., 2012. Systematic literature studies: database searches vs. back-
ward snowballing. In: Proceedings of the ACM-IEEE International Symposium on
S21 (2015) investigated a tool enforcing constraints on collabo-
Empirical Software Engineering and Measurement, pp. 29–38.
rative architectural DM. 48 students participated in a practical ex- Jansen, A., Bosch, J., 2005. Software architecture as a set of architectural design de-
ercise (two sessions of 90 min) to design a service-based system. cisions. In: Proceedings 5th IEEE/IFIP Working Conference on Software Architec-
ture, pp. 109–120.
All students worked with the tool which also provided architec-
Kahneman, D., 2003. Maps of bounded rationality: Psychology for behavioral eco-
tural patterns, but in the control group the functionality for enforc- nomics. Am. Econ. Rev. 93, 1449–1475.
ing collaboration constraints was hidden. Only a textual description Kahneman, D., 2011. Thinking, Fast and Slow. Penguin.
of the constraints was supplied. The data collected consisted of the Kitchenham, B., Pfleeger, S.L., 20 01–20 02. Principles of Survey Research, Parts 1 to
6.
actions executed (logged by the tool), the time, the completeness Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., Linkman, S., 2009.
and the violations. Thus, analysis is qualitative (to assess complete- Systematic literature reviews in software engineering–a systematic literature re-
ness and violations) and quantitative. view. Inf. Softw. Technol. 51, 7–15.
Kitchenham, B., Charters, S., 2007. Technical Report, Ver. 2.3 EBSE Technical Report.
S46 (2015) tested the effects of reusable architectural knowl- EBSE sn.
edge on DM in two controlled experiments using the same tool as Kruchten, P., 2008. What do software architects really do? J. Syst. Softw. 81,
S53. In the first experiment 49 bachelor students were involved, 2413–2416.
Lloyd, P., McDonnell, J., Cross, N., 2007. Analysing design behaviour: the design
in the second 122. All students worked in a 90 min session with thinking research symposia series. In: IASDR07. International Association of So-
the tool which provided patterns, but the experiment group also cieties of Desing Research. The Hong Kong Polytechnic University, pp. 12–15.
had access to reusable decision models in the tool. In the 2 experi- Lovallo, D., Sibony, O., 2010. The case for behavioral strategy. McKinsey Q. 2, 30–43.
Mehalik, M., Schunn, C., 2007. What constitutes good design? A review of empirical
ments different projects were used. The data collected consisted of
studies of design processes. Int. J. Eng. Educ. 22, 519–532.
the actions, the time, the number and quality of decisions. Thus, Moran, T., Carroll, J., 1996. Design Rationale: Concepts, Techniques, and Use.
the viewpoint is cur and analysis comprises quantitative and qual- Lawrence Erlbaum Associates, Publishers.
Perry, D.E., Porter, A.A., Votta, L.G., 20 0 0. Empirical studies of software engineering:
itative ratings.
a roadmap. In: in Proceedings of the Conference on The Future of Software Engi-
S90 (2016) tested the effects of GADGET on consensus in archi- neering, pp. 345–355.
tectural decision making. The researchers did an exploratory study Petre, M., van der Hoek, A., Baker, A., 2010. Editorial. Des. Stud. 31, 533–544.
to study the practical applicability of GADGET in the industry, then Pfaff, M.S., Klein, G.L., Drury, J.L., Moon, S.P., Liu, Y., Entezari, S.O., 2013. Supporting
complex decision making through option awareness. J. Cogn. Eng. Decis. Making
conducted an experiment with 113 students to evaluate the effect 7, 155–178.
with and without GADGET on decision consensus. Razavian, M., Tang, A., Capilla, R., Lago, P., 2016. In two minds: how reflections in-
S94 (2016) tested the effects of a reminder card game of de- fluence software design thinking. J. Softw. 28, 394–426.
Runeson, P., Höst, M., 2009. Guidelines for conducting and reporting case study re-
sign reasoning. The experiment involved experimental and control search in software engineering. Empir. Softw. Eng. 14, 131.
groups of students. These students were given the same design ex- Simon, H., 1982. Models of Bounded Rationality. MIT Press, Cambridge, Mass.
ercise, and the experimental group would use the prescribed cards Storey, M.-A., 2011. ICSE 2011 Panel on “What Industry Wants from Research”
Available https://margaretannestorey.wordpress.com/2011/08/05/icse-2011-
during their design session. The design dialogue was captured and panel- on- %E2%80%9Cwhat- industry- wants- from- research%E2%80%9D/.
analyzed to see whether the groups equipped with the reminder Suwa, M., Tversky, B., 1997. What do architects and students perceive in their design
cards reason more during their design. sketches? A protocol analysis. Des. Stud. 18, 385–403.
Tang, A., Razavian, M., Paech, B., Hesse, T.M., 2017. Human aspects in software ar-
chitecture decision making: a literature review. In: 2017 IEEE International Con-
Supplementary materials ference on Software Architecture, pp. 107–116.
Thaler, R.H., 2015. Misbehaving: The Making of Behavioral Economics. WW Norton
& Company.
Supplementary material associated with this article can be
Tyree, J., Akerman, A., 2005. Architecture decisions: demystifying architecture. IEEE
found, in the online version, at doi:10.1016/j.jss.2018.12.003. Softw. 22, 19–27.
van Vliet, H., Tang, A., 2016. Decision making in software architecture. J. Syst. Softw.
References 117, 638–644.
Wieringa, R.J., 2014. Design Science Methodology For Information Systems and Soft-
Almendra, R., Christiaans, H., 2009. Improving design processes through better de- ware Engineering. Springer-Verlag, Berlin Heidelberg.
cision-making: an experiment with a decision-making support tool. In: Pro- Wohlin, C., Runeson, P., Hst, M., Ohlsson, M.C., Regnell, B., Wessln, A., 2012. Experi-
ceedings of International Association of Societies of Design Research, vol. 20 mentation in Software Engineering. Springer Publishing Company, Incorporated.
Wohlin, C., Aurum, A., 2015. Towards a decision-making structure for selecting ity of software with adequate effort. Since many years she is particularly active in
a research design in empirical software engineering. Empir. Softw. Eng. 20, the area of requirements and rational engineering. Based on her experiences as de-
1427–1455. partment head at the Fraunhofer Institute Experimental Software Engineering her
Wohlin, C., 2014. Guidelines for snowballing in systematic literature studies and a research is often empirical and in close cooperation with industry. She holds a PhD
replication in software engineering. In: Proceedings of the 18th International in Computer Science from the Ludwig-Maximilans-Universität München and a Ha-
Conference on Evaluation and Assessment in Software Engineering, p. 38. bilitation in Computer Science from the Technical Universität München.
Maryam Razavian is an assistant professor of industrial engineering in Technical Antony Tang is Associate Professor in Swinburne University of Technology, Aus-
University of Eindhoven. Her research interests include software design reasoning, tralia. He received a PhD degree in Information Technology from Swinburne in
human aspects of software design, software architecture, and service orientation. 2007. Prior to being a researcher, he had spent many years designing and devel-
Razavian received her PhD in Computer Science from VU University Amsterdam. oping software systems. His main research interests are software architecture de-
She’s a member of ACM and IEEE. sign reasoning, software development processes, software architecture knowledge
management.
Barbara Paech holds the chair “Software Engineering” at the University of Heidel-
berg. Her teaching and research focuses on methods and processes to ensure qual-

Empirical Research For Software Architecture Decis - 2019 - Journal of Systems A

Uploaded by

Copyright:

Available Formats

Empirical Research For Software Architecture Decis - 2019 - Journal of Systems A

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Empirical Research For Software Architecture Decis - 2019 - Journal of Systems A

Uploaded by

Copyright:

Available Formats

The Journal of Systems and Software 149 (2019) 360–381

Contents lists available at ScienceDirect

The Journal of Systems and Software

Empirical research for software architecture decision making: An

Fig. 1. Paper selection process.

Step 5 papers rejected after

Step 6 ﬁltering(18 papers)

mary studies. We have restricted the scope to the main software

Step 6 ﬁltering (4 papers)

works from other publication venues may be missing. As such, we

ware architecture research sources that are likely to publish such

did not conduct any comprehensive literature search in these other

Coding was performed by four researchers (authors of

searchers separately. When there was any disagreement between

paper. We sought to eliminate any biases or misinterpretation

through this process. During the coding of the papers we used

Research Cycle Focus, Strategic Design, Tactical and Operational

ferent dimensions so that we can identify the sub-dimensions and

sign concerns and we wanted to study the interdependencies be-

Furthermore, we considered the dimension of the Research Cycle

As a new dimension we use the Study Focus which determines

Study Focus dimension, namely Decision Making Focus and Re-

research with regard to the distinction between human DM be-

havior (DMBehavior) and DM practices in terms of design process,

psychological and cognitive aspects of DM and deal with different

human thinking aspects. Such studies provide an understanding of

Dimension Sub-dimension Codes

Study Focus Decision Making Focus DMBehavior, DMPractice

Fig. 2. Decision making research cycle.

RDFocus vs. DMBehavior (13 papers) DMPractice (25 papers)

Decision making focus classes and number of papers found Paper ID

DM Behavior Papers (13 papers)

RDFocus vs. DMFocus DMBehavior DMPractice

RDFocus vs. DMBehavior DMPractice

DMFactor RQB1. Is design ﬁxation a cognitive bias in DM ?

Basic Research for Problem S28,S3,S52,S53,S60,S62,S66,S7,S70,S80,S81,S91,S93 S14,S15,S26,S27,S30,S31,S35,S51,S67,S73,S77,S92,S94 (13

Fig. 5. Distribution of study classiﬁcation by inductive and deductive reasoning.

ther study aspects of design ﬁxation which still need improvement

6. Strategic, tactical, and operational research design

6.1. Overview of strategic research designs

In this subsection we show the results with respect to research

In the following we describe the interdependencies between

Fig. 9. Research characterization by participant type.

Fig. 10. Research data collection methods.

Fig. 11. Research characterization by viewpoint.

Dimension Sub-dimension Codes

You might also like