research-article

Public Access

Usage, costs, and benefits of continuous integration in open-source projects

Authors:

Michael Hilton,

Timothy Tunnell,

Danny DigAuthors Info & Claims

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Pages 426 - 437

https://doi.org/10.1145/2970276.2970358

Published: 25 August 2016 Publication History

Abstract

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? Without answering such questions, developers, tool builders, and researchers make decisions based on folklore instead of data. In this paper, we use three complementary methods to study the usage of CI in open-source projects. To understand which CI systems developers use, we analyzed 34,544 open-source projects from GitHub. To understand how developers use CI, we analyzed 1,529,291 builds from the most commonly used CI system. To understand why projects use or do not use CI, we surveyed 442 developers. With this data, we answered several key questions related to the usage, costs, and benefits of CI. Among our results, we show evidence that supports the claim that CI helps projects release more often, that CI is widely adopted by the most popular projects, as well as finding that the overall percentage of projects using CI continues to grow, making it important and timely to focus more research on CI.

References

[1]

7 reasons why you should be using continuous integration. https://about.gitlab.com/2015/02/03/7reasons-why-you-should-be-using-ci/. Accessed: 2016-04-24.

[2]

AppVeyor. https://www.appveyor.com/. Accessed: 2016-04-26.

[3]

The benefits of continuous integration. https:// blog.codeship.com/benefits-of-continuous-integration/. Accessed: 2016-04-24.

[4]

Build in the cloud. http://google-engtools.blogspot.com/2011/08/buildin-cloud-how-build-system-works.html.

[5]

CircleCI. https://circleci.com/. Accessed: 2016-04-26.

[6]

CloudBees. http://cloudbees.com/. Accessed: 2016-04-26.

[7]

Continuous integration. https: //www.thoughtworks.com/continuous-integration. Accessed: 2016-04-24.

[8]

Continuous integration is dead. http://www.yegor256.com/2014/10/08/continuousintegration-is-dead.html. Accessed: 2016-04-24.

[9]

CruiseControl. http://cruisecontrol.sourceforge.net/. Accessed: 2016-04-21.

[10]

CrunchBase. https://www.crunchbase.com/ organization/travis-ci#/entity. Accessed: 2016-04-24.

[11]

Google Search Trends. https://www.google.com/trends/. Accessed: 2016-04-24.

[12]

Jenkins. https://jenkins.io/. Accessed: 2016-04-21.

[13]

Restkit. https://github.com/RestKit/RestKit. Accessed: 2016-04-29.

[14]

Stackoverflow. http://stackoverflow.com/questions/214695/what-aresome-arguments-against-using-continuous-integration. Accessed: 2016-04-24.

[15]

Team Foundation Server. https://www.visualstudio.com/en-us/products/tfsoverview-vs.aspx. Accessed: 2016-04-21.

[16]

Tools for software engineers. http://research.microsoft.com/en-us/projects/tse/. Accessed: 2016-04-24.

[17]

Travis CI. https://travis-ci.org/. Accessed: 2016-04-21.

[18]

Werker. http://wercker.com/. Accessed: 2016-04-26.

[19]

Why don’t we use continuous integration? https://blog.inf.ed.ac.uk/sapm/2014/02/14/why-dontwe-use-continuous-integration/. Accessed: 2016-04-24.

[20]

Yaml: Yaml ain’t markup language. http://yaml.org/. Accessed: 2016-04-24.

[21]

J. M. Al-Kofahi, H. V. Nguyen, A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen. Detecting semantic changes in Makefile build code. In ICSM, 2012.

Digital Library

[22]

J. Allspaw and P. Hammond. 10+ deploys per day: Dev and ops cooperation at Flickr. https://www.youtube.com/watch?v=LdOe18KhtT4. Accessed: 2016-04-21.

[23]

K. Beck. Embracing change with Extreme Programming. Computer, 32(10):70–77, 1999.

Digital Library

[24]

M. Beller, G. Gousios, and A. Zaidman. How (much) do developers test? In ICSE, 2015.

Digital Library

[25]

M. Beller, G. Gousios, and A. Zaidman. Oops, my tests broke the build: An analysis of travis ci builds with github. Technical report, PeerJ Preprints, 2016.

[26]

G. Booch. Object Oriented Design with Applications. Benjamin-Cummings Publishing Co., Inc., 1991.

Digital Library

[27]

M. Brandtner, E. Giger, and H. C. Gall. Supporting continuous integration by mashing-up software quality information. In CSMR-WCRE, 2014.

[28]

M. Brandtner, S. C. Müller, P. Leitner, and H. C. Gall. SQA-Profiles: Rule-based activity profiles for continuous integration environments. In SANER, 2015.

[29]

A. Celik, A. Knaust, A. Milicevic, and M. Gligoric. Build system with lazy retrieval for Java projects. In FSE, 2016.

Digital Library

[30]

J. C. M. de Campos, A. Arcuri, G. Fraser, and R. F. L. M. de Abreu. Continuous test generation: Enhancing continuous integration with automated test generation. In ASE, 2014.

Digital Library

[31]

S. Dösinger, R. Mordinyi, and S. Biffl. Communicating continuous integration servers for increasing effectiveness of automated testing. In ASE, 2012.

Digital Library

[32]

J. Downs, B. Plimmer, and J. G. Hosking. Ambient awareness of build status in collocated software teams. In ICSE, 2012.

Digital Library

[33]

S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regression testing in continuous integration development environments. In FSE, 2014.

Digital Library

[34]

J. Engblom. Virtual to the (near) end: Using virtual platforms for continuous integration. In DAC, 2015.

Digital Library

[35]

S. Erdweg, M. Lichter, and M. Weiel. A sound and optimal incremental build system with dynamic dependencies. In OOPSLA, 2015.

Digital Library

[36]

B. Fitzgerald, K. J. Stol, R. O’Sullivan, and D. O’Brien. Scaling agile methods to regulated environments: An industry case study. In ICSE, 2013.

Digital Library

[37]

M. Fowler. Continuous Integration. http://martinfowler.com/articles/ originalContinuousIntegration.html. Accessed: 2016-04-21.

[38]

M. Gligoric, L. Eloussi, and D. Marinov. Practical regression test selection with dynamic file dependencies. In ISSTA, 2016.

Digital Library

[39]

G. Gousios. The GHTorrent dataset and tool suite. In MSR, 2013.

Digital Library

[40]

J. Humble. Evidence and case studies. http://continuousdelivery.com/evidence-case-studies/. Accessed: 2016-04-29.

[41]

E. Knauss, M. Staron, W. Meding, O. Söder, A. Nilsson, and M. Castell. Supporting continuous integration by code-churn based test selection. In RCoSE, 2015.

Digital Library

[42]

P. S. Kochhar, F. Thung, D. Lo, and J. L. Lawall. An empirical study on the adequacy of testing in open source projects. In APSEC, 2014.

Digital Library

[43]

V. Kuechler, C. Gilbertson, and C. Jensen. Gender differences in early free and open source software joining process. In IFIP, 2012.

[44]

E. Laukkanen, M. Paasivaara, and T. Arvonen. Stakeholder perceptions of the adoption of continuous integration: A case study. In AGILE, 2015.

Digital Library

[45]

M. Leppänen, S. Mäkinen, M. Pagels, V. P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistä. The highways and country roads to continuous deployment. IEEE Software, 2015.

Digital Library

[46]

A. Miller. A hundred days of continuous integration. In AGILE, 2008.

Digital Library

[47]

K. Mu¸slu, Y. Brun, and A. Meliou. Data debugging with continuous testing. In FSE, 2013.

[48]

V. One. 10th annual state of Agile development survey. https://versionone.com/pdf/VersionOne-10th-Annual-State-of-Agile-Report.pdf, 2016.

[49]

Puppet and DevOps Research and Assessments (DORA). 2016 state of DevOps Report. https://puppet.com/system/files/2016-06/2016% 20State%20of%20DevOps%20Report 0.pdf, 2016.

[50]

D. Saff and M. D. Ernst. Continuous testing in Eclipse. In ICSE, 2005.

Digital Library

[51]

Testing at the speed and scale of Google, Jun 2011. http://google-engtools.blogspot.com/2011/06/testingat-speed-and-scale-of-google.html.

[52]

Tools for continuous integration at Google scale, October 2011. http://www.youtube.com/watch?v=b52aXZ2yi08.

[53]

B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. Quality and productivity outcomes relating to continuous integration in GitHub. In FSE, 2015.

Digital Library

[54]

Z. Xu, M. B. Cohen, W. Motycka, and G. Rothermel. Continuous test suite augmentation in software product lines. In SPLC, 2013.

Digital Library

[55]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. STVR, 22(2):67–120, 2012.

Digital Library

[56]

S. Zhou, J. M. Al-Kofahi, T. N. Nguyen, C. Kästner, and S. Nadi. Extracting configuration knowledge from build files with symbolic analysis. In RELENG, 2015.

Digital Library

Cited By

Pei YSohn JHabchi SPapadakis M(2025)Non-Flaky and Nearly Optimal Time-Based Treatment of Asynchronous Wait Web TestsACM Transactions on Software Engineering and Methodology10.1145/369598934:2(1-29)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3695989
Stoudt SJernite YMarshall BMarwick BSharan MWhitaker KDanchev V(2024)Ten simple rules for building and maintaining a responsible data science workflowPLOS Computational Biology10.1371/journal.pcbi.101223220:7(e1012232)Online publication date: 18-Jul-2024
https://doi.org/10.1371/journal.pcbi.1012232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Show More Cited By

Index Terms

Usage, costs, and benefits of continuous integration in open-source projects
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects
SWAN 2018: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics

Continuous integration (CI) tools integrate code changes by automatically compiling, building, and executing test cases upon submission of code changes. Use of CI tools is getting increasingly popular, yet how proprietary projects reap the benefits of ...
(No) influence of continuous integration on the commit activity in GitHub projects
SWAN 2018: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics

A core goal of Continuous Integration (CI) is to make small incremental changes to software projects, which are integrated frequently into a mainline repository or branch. This paper presents an empirical study that investigates if developers adjust ...
How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub Actions
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

Continuous Integration (CI) is a well-established practice in traditional software development, but its nuances in the domain of Machine Learning (ML) projects remain relatively unexplored. Given the distinctive nature of ML development, understanding ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

August 2016

899 pages

ISBN:9781450338455

DOI:10.1145/2970276

General Chair:
David Lo
Singapore Management University, Singapore
,
Program Chairs:
Sven Apel
University of Passau, Germany
,
Sarfraz Khurshid
University of Texas at Austin, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ASE'16

Sponsor:

SIGAI
SIGSOFT
IEEE-CS

ASE'16: ACM/IEEE International Conference on Automated Software Engineering

September 3 - 7, 2016

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

217
Total Citations
View Citations
4,592
Total Downloads

Downloads (Last 12 months)705
Downloads (Last 6 weeks)71

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pei YSohn JHabchi SPapadakis M(2025)Non-Flaky and Nearly Optimal Time-Based Treatment of Asynchronous Wait Web TestsACM Transactions on Software Engineering and Methodology10.1145/369598934:2(1-29)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3695989
Stoudt SJernite YMarshall BMarwick BSharan MWhitaker KDanchev V(2024)Ten simple rules for building and maintaining a responsible data science workflowPLOS Computational Biology10.1371/journal.pcbi.101223220:7(e1012232)Online publication date: 18-Jul-2024
https://doi.org/10.1371/journal.pcbi.1012232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Yin MKashiwa YGallaba KAlfadel MKamei YMcIntosh SFilkov VRay BZhou M(2024)Developer-Applied Accelerations in Continuous Integration: A Detection Approach and Catalog of PatternsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695533(1655-1666)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695533
Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Weeraddana NAlfadel MMcIntosh S(2024)Dependency-Induced Waste in Continuous Integration: An Empirical Study of Unused Dependencies in the npm EcosystemProceedings of the ACM on Software Engineering10.1145/36608231:FSE(2632-2655)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660823
Deshpande GCheekati SPatel SRaj PSingh MPindur MAl Soghyar NZhao BBabolhavaeji PTaher MNathan KSpaeth WRoozbahani MJoyner DKim MWang XXia M(2024)Transforming CS Education with DevOps: Streamlined Assignment Validation and Delivery @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664676(259-264)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664676
Wang GSun ZChen YZhao YLiang QHao DChristakis MPradel M(2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680356
Zaidman ASaadatmand MLonetti FBudnik CLi JGuerriero A(2024)An Inconvenient Truth in Software Engineering? The Environmental Impact of Testing Open Source Java ProjectsProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644461(214-218)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644461
Kola-Olawuyi AWeeraddana NNagappan MSpinellis DConstantinou EBacchelli A(2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644924
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten