Organizing Empirical Studies as Learning Iterations in Design Science Research Projects
Authors: 
Monalessa Barcellos, Gleison Santos, Tayana Conte, Bianca Trinkenreich, Patricia MatsubaraAuthors Info & Claims
Article No.: 17, Pages 1 - 10
Abstract
Software Quality is a relevant topic that interests both Academy and Industry. Hence, research on this topic should be aligned with the Industry needs, which demands the adoption of research approaches that enable closer interaction between researchers and practitioners. In this context, Design Science Research (DSR) stands out as a way to reduce the gap between theory and practice. DSR is a methodological approach to building innovative artifacts to solve real-world problems and, at the same time, making a scientific contribution. As a problem-oriented research method, DSR seeks to understand the problem, build and evaluate artifacts that allow transforming situations, changing their conditions to better or desirable states. In DSR projects, empirical studies have been usually applied to evaluate the proposed artifact. However, they can also be used to support other activities. Over the last eight years, we have successfully used empirical studies with different purposes in DSR projects. We organized the studies as learning iterations that provide useful knowledge to understand the problem, ground the artifact, develop, evaluate and improve it. As a result, we have experienced a more fluid DSR process and the proposed artifacts have been better grounded and suitable for solving the aimed problem. In this paper, we share our experience by discussing how we have used empirical studies as learning iterations in DSR projects, presenting our approach to organizing empirical studies in a DSR project according to the study purpose and the knowledge it intends to capture, and summarizing two DSR projects that address software quality issues and were developed by using such approach.
References
[1]
2022. Short communication: Evolution of secondary studies in software engineering. Inf Softw Technol 145(2022), 106840.
[2]
Icek Ajzen. 2020. The theory of planned behavior: Frequently asked questions. Human Behavior and Emerging Technologies 2, 4 (2020), 314–324. https://doi.org/10.1002/hbe2.195 Preprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/hbe2.195.
[3]
J.E. Aken, van. 2004. Management research based on the paradigm of the design sciences: the quest for field-tested and grounded technological rules. J. Manag. Stud. 41, 2 (2004), 219–246. https://doi.org/10.1111/j.1467-6486.2004.00430.x
[4]
Aline Barbosa, Geraldo Galindo, Maria Lencastre, Roberta Fagundes, and Wylliams Santos. 2020. Fostering Industry-Academia Collaboration in Software Engineering using Action Research: A Case Study. In Anais do XIX Simpósio Brasileiro de Qualidade de Software (São Luiz do Maranhão). SBC, Porto Alegre, RS, Brasil, 411–419. https://sol.sbc.org.br/index.php/sbqs/view/14239
[5]
Victor Basili, Adam Trendowicz, Martin Kowalczyk, Jens Heidrich, Carolyn Seaman, Jürgen Münch, and Dieter Rombach. 2014. GQM+ Strategies in a Nutshell. In Aligning Organizations Through Measurement. Springer, 9–17.
[6]
Richard Baskerville. 2008. What design science is not. European Journal of Information Systems 17, 5 (2008), 441–443. https://doi.org/10.1057/ejis.2008.45
[7]
Richard L Baskerville. 1997. Distinguishing action research from participative case studies. Journal of systems and information technology (1997).
[8]
Joelma Choma, Luciana A. M. Zaina, and Tiago Silva Da Silva. 2015. Towards an Approach Matching CMD and DSR to Improve the Academia-Industry Software Development Partnership: A Case of Agile and UX Integration. In Proceedings of the 29th Brazilian Symposium on Software Engineering. IEEE Computer Society, USA, 51–60. https://doi.org/10.1109/SBES.2015.18
[9]
Aldo Dagnino. 2013. Estimating software-intensive projects in the absence of historical data. In 2013 35th International Conference on Software Engineering (ICSE). 941–950. https://doi.org/10.1109/ICSE.2013.6606643 ISSN: 1558-1225.
[10]
Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. 2008. Selecting Empirical Methods for Software Engineering Research. Springer London, London, 285–311. https://doi.org/10.1007/978-1-84800-044-5_11
[11]
Michael Felderer and Guilherme Horta Travassos. 2020. The Evolution of Empirical Methods in Software Engineering. Springer International Publishing, Cham, 1–24. https://doi.org/10.1007/978-3-030-32489-6_1
[12]
Eduardo Ferreira, Bianca Trinkenreich, Monalessa Perini Barcellos, and Gleison Santos. 2018. Using SINIS and GQM+Strategies to Align Organizational Goals and Service Level Agreement Indicators. In Proceedings of the 17th Brazilian Symposium on Software Quality. ACM, New York, NY, USA, 324–333. https://doi.org/10.1145/3275245.3275274
[13]
Eduardo Ferreira, Bianca Trinkenreich, Monalessa Perini Barcellos, and Gleison Santos. 2021. SINIS-LA Method for IT Alignment Considering Service Level Management. In XVII Brazilian Symposium on Information Systems (Uberlândia, Brazil) (SBSI 2021). ACM, New York, NY, USA. https://doi.org/10.1145/3466933.3466977
[14]
Tony Gorschek and Krzysztof Wnuk. 2020. Third Generation Industrial Co-production in Software Engineering. Springer Int. Publishing, Cham, 503–525.
[15]
A. R. Hevner. 2007. A Three Cycle View of Design Science Research. Scandinavian Journal of Information Systems(2007), 87–92.
[16]
Alan R. Hevner, Salvatore T. March, Jinsoo Park, and Sudha Ram. 2004. Design Science in Information Systems Research. MIS Quarterly 28, 1 (mar 2004), 75–105.
[17]
Pertti Järvinen. 2007. Action Research is Similar to Design Science. Quality & Quantity 41(2007), 37–54. Issue 1. https://doi.org/10.1007/s11135-005-5427-1
[18]
Fernando Kamei, Igor Wiese, Gustavo Pinto, Márcio Ribeiro, and Sérgio Soares. 2020. On the Use of Grey Literature: A Survey with the Brazilian Software Engineering Research Community. In Proc. of the 34th Brazilian Symposium on Software Engineering(SBES ’20). ACM, New York, NY, USA, 183–192.
[19]
Matthew P. H. Kan and Leandre R. Fabrigar. 2017. Theory of Planned Behavior. In Encyclopedia of Personality and Individual Differences, Virgil Zeigler-Hill and Todd K. Shackelford (Eds.). Springer International Publishing, Cham, 1–8. https://doi.org/10.1007/978-3-319-28099-8_1191-1
[20]
Staffs Keele 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01. Keele University & University of Durham.
[21]
Barbara A. Kitchenham. 2004. Procedures for Performing Systematic Reviews. Joint Technical Report TR/SE-0401. Keele University & National ICT Australia Ltd., Keele, UK.
[22]
Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidence-Based Software Engineering and Systematic Reviews (1 ed.). CRC Press.
[23]
Barbara A. Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-Based Software Engineering. In Proceedings of the 26th International Conference on Software Engineering(ICSE ’04). IEEE Computer Society, USA, 273–281.
[24]
Sômulo Nogueira Mafra, Rafael Ferreira Barcelos, and Guilherme Horta Travassos. 2006. Aplicando uma metodologia baseada em evidência na definição de novas tecnologias de software. In Proceedings of the 20th Brazilian Symposium on Software Engineering (SBES 2006), Vol. 1. 239–254.
[25]
Ana Magazinius, Sofia Börjesson, and Robert Feldt. 2012. Investigating Intentional Distortions in Software Cost Estimation - An Exploratory Study. J Syst Softw 85, 8 (aug 2012), 1770–1781. https://doi.org/10.1016/j.jss.2012.03.026
[26]
Patricia Matsubara. 2019. Dealing with Software Estimates Distortions from the Perspective of Negotiation Theories. SIGSOFT Softw. Eng. Notes 44, 3 (nov 2019), 22. https://doi.org/10.1145/3356773.3356794
[27]
Patricia Matsubara, Bruno Gadelha, Igor Steinmacher, and Tayana Conte. 2022. SEXTAMT: A systematic map to navigate the wide seas of factors affecting expert judgment software estimates. J Syst Softw 185(2022), 111148. https://doi.org/10.1016/j.jss.2021.111148
[28]
Patricia Matsubara, Igor Steinmacher, Bruno Gadelha, and Tayana Conte. 2021. Buying time in software development: how estimates become commitments?. In 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, Madrid, Spain, 61–70. https://doi.org/10.1109/CHASE52884.2021.00015
[29]
Patricia Matsubara, Igor Steinmacher, Bruno Gadelha, and Tayana Conte. 2022. The best defense is a good defense: adapting negotiation methods for tackling pressure over software project estimates. In Proceedings of the 44th International Conference in Software Engineering — New Ideas and Emerging Results Track. IEEE, Pittsburgh, Pennsylvania. https://doi.org/10.1145/3510455.3512775
[30]
Steve McConnell. 2006. Politics, Negotiation, and Problem Solving. In Software Estimation: Demystifying the Black Art. Microsoft Press, Redmond, 259–270.
[31]
Philipp Offermann, Olga Levina, Marten Schönherr, and Udo Bub. 2009. Outline of a Design Science Research Process. In Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology(Philadelphia, Pennsylvania) (DESRIST ’09). ACM, New York, NY, USA. https://doi.org/10.1145/1555619.1555629
[32]
Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger, and Samir Chatterjee. 2007. A Design Science Research Methodology for Information Systems Research. J Manag Inf Syst 24, 3 (2007), 45–77. https://doi.org/10.2753/MIS0742-1222240302
[33]
Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for Conducting Systematic Mapping Studies in Software Engineering. Inf. Softw. Technol. 64, C (aug 2015), 1–18. https://doi.org/10.1016/j.infsof.2015.03.007
[34]
Rodrigo Rebouças de Almeida, Christoph Treude, and Uirá Kulesza. 2019. Tracy: A Business-Driven Technical Debt Prioritization Framework. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). 181–185.
[35]
Per Runeson, Emelie Engström, and Margaret-Anne Storey. 2020. The Design Science Paradigm as a Frame for Empirical Software Engineering. Springer International Publishing, Cham, 127–147. https://doi.org/10.1007/978-3-030-32489-6_5
[36]
Per Runeson, Sten Minör, and Johan Svenér. 2014. Get the Cogs in Synch: Time Horizon Aspects of Industry–Academia Collaboration. In Proceedings of the 2014 International Workshop on Long-Term Industrial Collaboration on Software Engineering (Vasteras, Sweden) (WISE ’14). ACM, New York, NY, USA, 25–28. https://doi.org/10.1145/2647648.2647652
[37]
Rafaela Sampaio, Cristina Teles Cerdeiral, and Gleison Santos. 2021. A Method to Support Continuous Planning at the Team Level. ACM, New York, NY, USA. https://doi.org/10.1145/3493244.3493257
[38]
Klaas-Jan Stol and Brian Fitzgerald. 2020. Guidelines for Conducting Software Engineering Research. Springer International Publishing, Cham, 27–62. https://doi.org/10.1007/978-3-030-32489-6_2
[39]
Bianca Trinkenreich and Gleison Santos. 2015. Avaliação da Gerência de Incidentes sob a Luz do MR-MPS-SV e Medição para Apoiar a Melhoria da Qualidade do Serviço de TI. In Proceedings of the 14th Brazilian Symposium on Software Quality. SBC, 220–227.
[40]
Bianca Trinkenreich and Gleison Santos. 2015. Avaliação do Processo de Medição para Serviços de TI em uma Empresa Global à Luz do MR-MPS-SV. iSys-Brazilian Journal of Information Systems 8, 2 (2015), 58–77.
[41]
Bianca Trinkenreich. and Gleison Santos.2015. Metrics to Support It Service Maturity Models - A Case Study. In Proceedings of the 17th International Conference on Enterprise Information Systems - Volume 1: ICEIS,. INSTICC, SciTePress, 395–403. https://doi.org/10.5220/0005398003950403
[42]
Bianca Trinkenreich, Gleison Santos, and Monalessa Perini Barcellos. 2015. Metrics to Support IT Service Maturity Models. In Proc. 17th Int. Conf. Enterp. Inf. Syst.(ICEIS 2015). 1–8.
[43]
Bianca Trinkenreich, Gleison Santos, and Monalessa Perini Barcellos. 2015. SINIS: a method to select indicators for IT services. In International Conference on Product-Focused Software Process Improvement. Springer, 68–86.
[44]
Bianca Trinkenreich, Gleison Santos, and Monalessa Perini Barcellos. 2018. SINIS: A GQM+Strategies-based approach for identifying goals, strategies and indicators for IT services. Inf Softw Technol 100(2018), 147–164.
[45]
Bianca Trinkenreich, Gleison Santos, Monalessa Perini Barcellos, and Tayana Conte. 2017. Eliciting strategies for the GQM+ strategies approach in IT service measurement initiatives. In Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 374–383.
[46]
Bianca Trinkenreich, Gleison Santos, Valdemar TF Confort, and Flávia Maria Santoro. 2015. Toward using Business Process Intelligence to Support Incident Management Metrics Selection and Service Improvement. In SEKE. 522–527.
[47]
Roel Wieringa. 2009. Design Science as Nested Problem Solving. In Proc. of the 4th Int. Conf. on Design Science Research in Information Systems and Technology (Philadelphia, Pennsylvania) (DESRIST ’09). ACM, New York, NY, USA. https://doi.org/10.1145/1555619.1555630
[48]
Roelf J. Wieringa. 2014. Design science methodology for information systems and software engineering. Springer, Netherlands. https://doi.org/10.1007/978-3-662-43839-8 10.1007/978-3-662-43839-8.
[49]
Claes Wohlin and Austen Rainer. 2021. Challenges and recommendations to publishing and using credible evidence in software engineering. Inf Softw Technol 134 (June 2021), 106555. https://doi.org/10.1016/j.infsof.2021.106555
[50]
Claes Wohlin and Per Runeson. 2021. Guiding the selection of research methodology in industry–academia collaboration in software engineering. Inf Softw Technol 140(2021), 106678. https://doi.org/10.1016/j.infsof.2021.106678
Index Terms
- Organizing Empirical Studies as Learning Iterations in Design Science Research Projects
Recommendations
A Postmodern Perspective on Socio-technical Design Science Research in Information Systems
DESRIST 2015: Proceedings of the 10th International Conference on New Horizons in Design Science: Broadening the Research Agenda - Volume 9073This paper presents a critical account of the current state of design science research DSR of socio-technical artifacts in the information systems discipline as viewed through a postmodern lens. The paper offers a novel perspective to reflect on DSR and ...
Support mechanisms to conduct empirical studies in software engineering
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementContext: Empirical studies are gaining recognition in the Software Engineering (SE) research community. In order to foster empirical research, it is essential understand the environments, guidelines, process, and other mechanisms available to support ...
Empirical Grounding of Design Science Research Methodology
DESRIST 2015: Proceedings of the 10th International Conference on New Horizons in Design Science: Broadening the Research Agenda - Volume 9073The purpose of this paper is to empirically ground design science research methodology DSR. We claim that popular DSR methodologies lack solid empirical grounding since they are based on reconstructions of studies conducted for other purposes. Thus, we ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2022
352 pages
ISBN:9781450399999
DOI:10.1145/3571473
- Editors:
- Edna Dias Canedo,
- Davi Viana,
- Vinicius Garcia,
- Carla Bezerra,
- Ismayle de Sousa Santos,
- Bruno Gadelha,
- Ivan Machado,
- Sérgio Soares,
- Uirá Kulesza,
- Breno de França,
- Tayana Conte,
- José Carlos Maldonado,
- Sheila Reinehr,
- Andreia Malucelli,
- Adriano Bessa Albuquerque,
- Gleison Santos,
- Ivan Machado,
- Monalessa Perini Barcellos,
- Rodrigo Santos,
- Sheila Reinehr,
- Tayana Conte,
- Crescencio Lima,
- Danilo Monteiro,
- Adriana Damian,
- Larissa Rocha
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 January 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- FAPEAM
- CAPES
- FAPESP
- FAPERJ
- CNPq
Conference
SBQS '22
Acceptance Rates
Overall Acceptance Rate 35 of 99 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 77Total Downloads
- Downloads (Last 12 months)35
- Downloads (Last 6 weeks)8
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format