research-article

Towards continuous software reliability testing in DevOps

Authors:

Roberto Pietrantuono,

Antonia Bertolino,

Guglielmo De Angelis,

Stefano RussoAuthors Info & Claims

AST '19: Proceedings of the 14th International Workshop on Automation of Software Test

Pages 21 - 27

https://doi.org/10.1109/AST.2019.00009

Published: 27 May 2019 Publication History

Abstract

We introduce the DevOpRET approach for continuous reliability testing in DevOps. It leverages information monitored in operation to guide operational-profile based testing, which is conceived as part of the acceptance testing stage before each next release to production. We overview the envisaged test and monitoring pipeline, describe the approach and present a case-study evaluating how reliability assessment evolves over subsequent releases.

References

[1]

A. Dyck, R. Penners, and H. Lichter, "Towards definitions for release engineering and DevOps," in IEEE/ACM 3rd International Workshop on Release Engineering (RELENG). IEEE, 2015, pp. 3--3.

Digital Library

[2]

J. Smeds, K. Nybom, and I. Porres, "DevOps: A definition and perceived adoption impediments," in Agile Processes in Software Engineering and Extreme Programming, C. Lassenius, T. Dingsøyr, and M. Paasivaara, Eds. Cham: Springer International Publishing, 2015, pp. 166--177.

[3]

M. Walls, Building a DevOps culture. "O'Reilly Media, Inc.", 2013.

[4]

L. J. Bass, I. M. Weber, and L. Zhu, DevOps - A Software Architect's Perspective, ser. SEI series in software engineering. Addison-Wesley, 2015.

Digital Library

[5]

J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley Boston, 2011.

Digital Library

[6]

N. Forsgren and M. Kersten, "DevOps Metrics," Communications of the ACM, vol. 61, no. 4, pp. 44--48, 2018.

Digital Library

[7]

R. Jabbari, N. bin Ali, K. Petersen, and B. Tanveer, "Towards a benefits dependency network for DevOps based on a systematic literature review," Journal of Software: Evolution and Process, vol. 30, no. 11, p. e1957, 2018.

Digital Library

[8]

A. Brunnert, A. van Hoorn, F. Willnecker, A. Danciu, W. Hasselbring, C. Heger, N. R. Herbst, P. Jamshidi, R. Jung, J. von Kistowski, A. Koziolek, J. Kroß, S. Spinner, C. Vögele, J. Walter, and A. Wert, "Performance-oriented devops: A research agenda," CoRR, vol. abs/1508.04752, 2015. {Online}. Available: http://arxiv.org/abs/1508.04752

[9]

M. Mazkatli and A. Koziolek, "Continuous integration of performance model," in Companion of the 2018 ACM/SPEC International Conference on Performance Engineering, ser. ICPE '18. New York, NY, USA: ACM, 2018, pp. 153--158. {Online}. Available

Digital Library

[10]

A. A. U. Rahman and L. Williams, "Software security in devops: Synthesizing practitioners' perceptions and practices," in 2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED), May 2016, pp. 70--76.

Digital Library

[11]

J. S. Lee, "The devsecops and agency theory," in 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Oct 2018, pp. 243--244.

[12]

N. Forsgren, J. Humble, and G. Kim, "Accelerate: State of DevOps, strategies for a new economy," pp. 436--440, 2018, accessed: 2019-01-23. {Online}. Available: https://cloudplatformonline.com/2018-state-of-devops.html

[13]

J. D. Musa, "Software-reliability-engineered testing," IEEE Computer, vol. 29, no. 11, pp. 61--68, 1996.

Digital Library

[14]

N. Laranjeiro, M. Vieira, and H. Madeira, "A robustness testing approach for soap web services," Journal of Internet Services and Applications, vol. 3, no. 2, pp. 215--232, Sep 2012. {Online}. Available

[15]

K.-Y. Cai, "Towards a conceptual framework of software run reliability modeling," Inf. Sci., vol. 126, no. 1--4, pp. 137--163, Jul. 2000. {Online}. Available

Digital Library

[16]

P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini, "Evaluating testing methods by delivered reliability {software}," IEEE Transactions on Software Engineering, vol. 24, no. 8, pp. 586--601, Aug 1998.

Digital Library

[17]

T. A. Thayer, M. Lipow, and E. C. Nelson, Software Reliability. North-Holland Publishing, TRW Series of Software Technology, Amsterdam, 1978.

[18]

J. Angara, S. Prasad, and G. Sridevi, "The Factors Driving Testing in DevOps Setting - A Systematic Literature Survey," Indian Journal of Science and Technology, vol. 9, no. 48, 2017.

[19]

B. Fitzgerald and K.-J. Stol, "Continuous software engineering and beyond: trends and challenges," in Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering. ACM, 2014, pp. 1--9.

Digital Library

[20]

M. Soni, "End to end automation on cloud with build pipeline: the case for devops in insurance industry, continuous integration, continuous testing, and continuous delivery," in IEEE International Conference on Cloud Computing in Emerging Markets (CCEM). IEEE, 2015, pp. 85--89.

[21]

E. Di Nitto, P. Jamshidi, M. Guerriero, I. Spais, and D. A. Tamburri, "A software architecture framework for quality-aware DevOps," in Proceedings of the 2nd International Workshop on Quality-Aware DevOps (QUDOS). ACM, 2016, pp. 12--17.

Digital Library

[22]

B. Fitzgerald and K.-J. Stol, "Continuous software engineering: A roadmap and agenda," Journal of Systems and Software, vol. 123, pp. 176--189, 2017.

[23]

C. Trubiani, A. Bran, A. van Hoorn, A. Avritzer, and H. Knoche, "Exploiting load testing and profiling for performance antipattern detection," Information and Software Technology, vol. 95, pp. 329--345, 2018.

[24]

A. Aleti, C. Trubiani, A. van Hoorn, and P. Jamshidi, "An efficient method for uncertainty propagation in robust software performance estimation," Journal of Systems and Software, vol. 138, pp. 222--235, 2018.

[25]

R. Mijumbi, K. Okumoto, A. Asthana, and J. Meekel, "Recent advances in software reliability assurance," in IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 2018, pp. 77--82.

[26]

R. Pietrantuono and S. Russo, "On adaptive sampling-based testing for software reliability assessment," in IEEE International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2016, pp. 1--11.

Cited By

Haensel JAdriano CDyck JGiese HHoniden SNitto ECalinescu R(2020)Collective risk minimization via a bayesian model for statistical software testingProceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/3387939.3388616(45-56)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3387939.3388616

Recommendations

Adoption Issues in DevOps from the Perspective of Continuous Delivery Pipeline
ICSCA '19: Proceedings of the 2019 8th International Conference on Software and Computer Applications

DevOps and Continuous Delivery (CD) are the terms that are always related to each other in Software Delivery and Operation Process area. DevOps introduces a significant agile perspective to deliver the software product in short cycle time that will ...
Selection of DevOps best test practices: A hybrid approach using ISM and fuzzy TOPSIS analysis
Abstract
Testing is a complex phase in DevOps process due to need of an automated process that provides feedback at different strategies of continuous development and operations pipeline. Software organization face several challenges during the testing ...
- Identified the best test practices for DevOps process.
- Empirically validate the best test practices for DevOps process.
- Presented a holistic view of the best test practices to assist practitioners to revise and develop new strategies of testing for the ...
Assessing the Maturity of DevOps Practices in Software Industry: An Empirical Study of HELENA2 Dataset
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering

Currently, the software development organizations are adopting DevOps practices in order to develop quality product. Due to the lack of definition of DevOps, the principles, practices, and methods adopted in DevOps to determine success have changed ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

AST '19: Proceedings of the 14th International Workshop on Automation of Software Test

May 2019

80 pages

Program Chairs:
Byoungju Choi
Ewha Womans University, Korea
,
María José Escalona
University of Seville, Spain
,
Kim Herzig
Microsoft Corporation

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
223
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Haensel JAdriano CDyck JGiese HHoniden SNitto ECalinescu R(2020)Collective risk minimization via a bayesian model for statistical software testingProceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/3387939.3388616(45-56)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3387939.3388616

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents