research-article

Public Access

Usage, costs, and benefits of continuous integration in open-source projects

Authors:

Danny DigAuthors Info & Claims

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Pages 426 - 437

https://doi.org/10.1145/2970276.2970358

Published: 25 August 2016 Publication History

PDF eReader

Abstract

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? Without answering such questions, developers, tool builders, and researchers make decisions based on folklore instead of data. In this paper, we use three complementary methods to study the usage of CI in open-source projects. To understand which CI systems developers use, we analyzed 34,544 open-source projects from GitHub. To understand how developers use CI, we analyzed 1,529,291 builds from the most commonly used CI system. To understand why projects use or do not use CI, we surveyed 442 developers. With this data, we answered several key questions related to the usage, costs, and benefits of CI. Among our results, we show evidence that supports the claim that CI helps projects release more often, that CI is widely adopted by the most popular projects, as well as finding that the overall percentage of projects using CI continues to grow, making it important and timely to focus more research on CI.

References

[1]

7 reasons why you should be using continuous integration. https://about.gitlab.com/2015/02/03/7reasons-why-you-should-be-using-ci/. Accessed: 2016-04-24.

Google Scholar

[2]

AppVeyor. https://www.appveyor.com/. Accessed: 2016-04-26.

Google Scholar

[3]

The benefits of continuous integration. https:// blog.codeship.com/benefits-of-continuous-integration/. Accessed: 2016-04-24.

Google Scholar

[4]

Build in the cloud. http://google-engtools.blogspot.com/2011/08/buildin-cloud-how-build-system-works.html.

Google Scholar

[5]

CircleCI. https://circleci.com/. Accessed: 2016-04-26.

Google Scholar

[6]

CloudBees. http://cloudbees.com/. Accessed: 2016-04-26.

Google Scholar

[7]

Continuous integration. https: //www.thoughtworks.com/continuous-integration. Accessed: 2016-04-24.

Google Scholar

[8]

Continuous integration is dead. http://www.yegor256.com/2014/10/08/continuousintegration-is-dead.html. Accessed: 2016-04-24.

Google Scholar

[9]

CruiseControl. http://cruisecontrol.sourceforge.net/. Accessed: 2016-04-21.

Google Scholar

[10]

CrunchBase. https://www.crunchbase.com/ organization/travis-ci#/entity. Accessed: 2016-04-24.

Google Scholar

[11]

Google Search Trends. https://www.google.com/trends/. Accessed: 2016-04-24.

Google Scholar

[12]

Jenkins. https://jenkins.io/. Accessed: 2016-04-21.

Google Scholar

[13]

Restkit. https://github.com/RestKit/RestKit. Accessed: 2016-04-29.

Google Scholar

[14]

Stackoverflow. http://stackoverflow.com/questions/214695/what-aresome-arguments-against-using-continuous-integration. Accessed: 2016-04-24.

Google Scholar

[15]

Team Foundation Server. https://www.visualstudio.com/en-us/products/tfsoverview-vs.aspx. Accessed: 2016-04-21.

Google Scholar

[16]

Tools for software engineers. http://research.microsoft.com/en-us/projects/tse/. Accessed: 2016-04-24.

Google Scholar

[17]

Travis CI. https://travis-ci.org/. Accessed: 2016-04-21.

Google Scholar

[18]

Werker. http://wercker.com/. Accessed: 2016-04-26.

Google Scholar

[19]

Why don’t we use continuous integration? https://blog.inf.ed.ac.uk/sapm/2014/02/14/why-dontwe-use-continuous-integration/. Accessed: 2016-04-24.

Google Scholar

[20]

Yaml: Yaml ain’t markup language. http://yaml.org/. Accessed: 2016-04-24.

Google Scholar

[21]

J. M. Al-Kofahi, H. V. Nguyen, A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen. Detecting semantic changes in Makefile build code. In ICSM, 2012.

Digital Library

Google Scholar

[22]

J. Allspaw and P. Hammond. 10+ deploys per day: Dev and ops cooperation at Flickr. https://www.youtube.com/watch?v=LdOe18KhtT4. Accessed: 2016-04-21.

Google Scholar

[23]

K. Beck. Embracing change with Extreme Programming. Computer, 32(10):70–77, 1999.

Digital Library

Google Scholar

[24]

M. Beller, G. Gousios, and A. Zaidman. How (much) do developers test? In ICSE, 2015.

Digital Library

Google Scholar

[25]

M. Beller, G. Gousios, and A. Zaidman. Oops, my tests broke the build: An analysis of travis ci builds with github. Technical report, PeerJ Preprints, 2016.

Crossref

Google Scholar

[26]

G. Booch. Object Oriented Design with Applications. Benjamin-Cummings Publishing Co., Inc., 1991.

Digital Library

Google Scholar

[27]

M. Brandtner, E. Giger, and H. C. Gall. Supporting continuous integration by mashing-up software quality information. In CSMR-WCRE, 2014.

Crossref

Google Scholar

[28]

M. Brandtner, S. C. Müller, P. Leitner, and H. C. Gall. SQA-Profiles: Rule-based activity profiles for continuous integration environments. In SANER, 2015.

Crossref

Google Scholar

[29]

A. Celik, A. Knaust, A. Milicevic, and M. Gligoric. Build system with lazy retrieval for Java projects. In FSE, 2016.

Digital Library

Google Scholar

[30]

J. C. M. de Campos, A. Arcuri, G. Fraser, and R. F. L. M. de Abreu. Continuous test generation: Enhancing continuous integration with automated test generation. In ASE, 2014.

Digital Library

Google Scholar

[31]

S. Dösinger, R. Mordinyi, and S. Biffl. Communicating continuous integration servers for increasing effectiveness of automated testing. In ASE, 2012.

Digital Library

Google Scholar

[32]

J. Downs, B. Plimmer, and J. G. Hosking. Ambient awareness of build status in collocated software teams. In ICSE, 2012.

Digital Library

Google Scholar

[33]

S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regression testing in continuous integration development environments. In FSE, 2014.

Digital Library

Google Scholar

[34]

J. Engblom. Virtual to the (near) end: Using virtual platforms for continuous integration. In DAC, 2015.

Digital Library

Google Scholar

[35]

S. Erdweg, M. Lichter, and M. Weiel. A sound and optimal incremental build system with dynamic dependencies. In OOPSLA, 2015.

Digital Library

Google Scholar

[36]

B. Fitzgerald, K. J. Stol, R. O’Sullivan, and D. O’Brien. Scaling agile methods to regulated environments: An industry case study. In ICSE, 2013.

Digital Library

Google Scholar

[37]

M. Fowler. Continuous Integration. http://martinfowler.com/articles/ originalContinuousIntegration.html. Accessed: 2016-04-21.

Google Scholar

[38]

M. Gligoric, L. Eloussi, and D. Marinov. Practical regression test selection with dynamic file dependencies. In ISSTA, 2016.

Digital Library

Google Scholar

[39]

G. Gousios. The GHTorrent dataset and tool suite. In MSR, 2013.

Digital Library

Google Scholar

[40]

J. Humble. Evidence and case studies. http://continuousdelivery.com/evidence-case-studies/. Accessed: 2016-04-29.

Google Scholar

[41]

E. Knauss, M. Staron, W. Meding, O. Söder, A. Nilsson, and M. Castell. Supporting continuous integration by code-churn based test selection. In RCoSE, 2015.

Digital Library

Google Scholar

[42]

P. S. Kochhar, F. Thung, D. Lo, and J. L. Lawall. An empirical study on the adequacy of testing in open source projects. In APSEC, 2014.

Digital Library

Google Scholar

[43]

V. Kuechler, C. Gilbertson, and C. Jensen. Gender differences in early free and open source software joining process. In IFIP, 2012.

Crossref

Google Scholar

[44]

E. Laukkanen, M. Paasivaara, and T. Arvonen. Stakeholder perceptions of the adoption of continuous integration: A case study. In AGILE, 2015.

Digital Library

Google Scholar

[45]

M. Leppänen, S. Mäkinen, M. Pagels, V. P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistä. The highways and country roads to continuous deployment. IEEE Software, 2015.

Digital Library

Google Scholar

[46]

A. Miller. A hundred days of continuous integration. In AGILE, 2008.

Digital Library

Google Scholar

[47]

K. Mu¸slu, Y. Brun, and A. Meliou. Data debugging with continuous testing. In FSE, 2013.

Google Scholar

[48]

V. One. 10th annual state of Agile development survey. https://versionone.com/pdf/VersionOne-10th-Annual-State-of-Agile-Report.pdf, 2016.

Google Scholar

[49]

Puppet and DevOps Research and Assessments (DORA). 2016 state of DevOps Report. https://puppet.com/system/files/2016-06/2016% 20State%20of%20DevOps%20Report 0.pdf, 2016.

Google Scholar

[50]

D. Saff and M. D. Ernst. Continuous testing in Eclipse. In ICSE, 2005.

Digital Library

Google Scholar

[51]

Testing at the speed and scale of Google, Jun 2011. http://google-engtools.blogspot.com/2011/06/testingat-speed-and-scale-of-google.html.

Google Scholar

[52]

Tools for continuous integration at Google scale, October 2011. http://www.youtube.com/watch?v=b52aXZ2yi08.

Google Scholar

[53]

B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. Quality and productivity outcomes relating to continuous integration in GitHub. In FSE, 2015.

Digital Library

Google Scholar

[54]

Z. Xu, M. B. Cohen, W. Motycka, and G. Rothermel. Continuous test suite augmentation in software product lines. In SPLC, 2013.

Digital Library

Google Scholar

[55]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. STVR, 22(2):67–120, 2012.

Digital Library

Google Scholar

[56]

S. Zhou, J. M. Al-Kofahi, T. N. Nguyen, C. Kästner, and S. Nadi. Extracting configuration knowledge from build files with symbolic analysis. In RELENG, 2015.

Digital Library

Google Scholar

Cited By

View all

Stoudt SJernite YMarshall BMarwick BSharan MWhitaker KDanchev V(2024)Ten simple rules for building and maintaining a responsible data science workflowPLOS Computational Biology10.1371/journal.pcbi.101223220:7(e1012232)Online publication date: 18-Jul-2024
https://doi.org/10.1371/journal.pcbi.1012232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Yin MKashiwa YGallaba KAlfadel MKamei YMcIntosh SFilkov VRay BZhou M(2024)Developer-Applied Accelerations in Continuous Integration: A Detection Approach and Catalog of PatternsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695533(1655-1666)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695533
Show More Cited By

Index Terms

Usage, costs, and benefits of continuous integration in open-source projects
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects
SWAN 2018: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics

Continuous integration (CI) tools integrate code changes by automatically compiling, building, and executing test cases upon submission of code changes. Use of CI tools is getting increasingly popular, yet how proprietary projects reap the benefits of ...
(No) influence of continuous integration on the commit activity in GitHub projects
SWAN 2018: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics

A core goal of Continuous Integration (CI) is to make small incremental changes to software projects, which are integrated frequently into a mainline repository or branch. This paper presents an empirical study that investigates if developers adjust ...
How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub Actions
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

Continuous Integration (CI) is a well-established practice in traditional software development, but its nuances in the domain of Machine Learning (ML) projects remain relatively unexplored. Given the distinctive nature of ML development, understanding ...

Reviews

Reviewer: Klerisson Paixao

A profound revolution has been taking place throughout software engineering, led again by the software industry. Continuous integration (CI) was first discussed in 1991, but until recently only a handful of research papers on the subject were published. With CI adoption increasing, it is worth understanding the advantages of it. This study addresses questions on which systems have been used, by whom, and why CI was adopted. After a brief overview on CI history and key industry products, the authors conducted three main analyses. The first analyzed 34,544 open-source projects and answered research questions related to the usage of CI. The growing adoption of such systems is notable. Then, more than a million build processes from the most popular CI system were analyzed to disclose costs associated with CI adoption. This part reveals that CI is not being used by some software developers because of a lack of knowledge of the practice. The final analysis covered questions related to the benefits of CI. The authors surveyed 442 developers to outline the perceived advantages of CI. They mainly found that projects using CI release more often than those that do not use it. This study contributes to the CI body of knowledge in regards to evidence-based software engineering. It presents a greater effort to organize and disseminate CI research to practitioners, so that we can rely on verified evidence rather than personal observations. There is limited empirical understanding of CI practices, and further study could be warranted. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

August 2016

899 pages

ISBN:9781450338455

DOI:10.1145/2970276

General Chair:
David Lo
Singapore Management University, Singapore
,
Program Chairs:
Sven Apel
University of Passau, Germany
,
Sarfraz Khurshid
University of Texas at Austin, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ASE'16

Sponsor:

SIGAI
SIGSOFT
IEEE-CS

ASE'16: ACM/IEEE International Conference on Automated Software Engineering

September 3 - 7, 2016

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

214
Total Citations
View Citations
4,468
Total Downloads

Downloads (Last 12 months)699
Downloads (Last 6 weeks)84

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Stoudt SJernite YMarshall BMarwick BSharan MWhitaker KDanchev V(2024)Ten simple rules for building and maintaining a responsible data science workflowPLOS Computational Biology10.1371/journal.pcbi.101223220:7(e1012232)Online publication date: 18-Jul-2024
https://doi.org/10.1371/journal.pcbi.1012232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Yin MKashiwa YGallaba KAlfadel MKamei YMcIntosh SFilkov VRay BZhou M(2024)Developer-Applied Accelerations in Continuous Integration: A Detection Approach and Catalog of PatternsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695533(1655-1666)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695533
Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Weeraddana NAlfadel MMcIntosh S(2024)Dependency-Induced Waste in Continuous Integration: An Empirical Study of Unused Dependencies in the npm EcosystemProceedings of the ACM on Software Engineering10.1145/36608231:FSE(2632-2655)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660823
Deshpande GCheekati SPatel SRaj PSingh MPindur MAl Soghyar NZhao BBabolhavaeji PTaher MNathan KSpaeth WRoozbahani MJoyner DKim MWang XXia M(2024)Transforming CS Education with DevOps: Streamlined Assignment Validation and Delivery @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664676(259-264)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664676
Wang GSun ZChen YZhao YLiang QHao DChristakis MPradel M(2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680356
Zaidman ASaadatmand MLonetti FBudnik CLi JGuerriero A(2024)An Inconvenient Truth in Software Engineering? The Environmental Impact of Testing Open Source Java ProjectsProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644461(214-218)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644461
Kola-Olawuyi AWeeraddana NNagappan MSpinellis DConstantinou EBacchelli A(2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644924
Bernardo JDa Costa DMedeiros SKulesza USpinellis DConstantinou EBacchelli A(2024)How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub ActionsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644915(665-676)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644915
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects

(No) influence of continuous integration on the commit activity in GitHub projects

How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub Actions

Reviews

Access critical reviews of Computing literature here