Resource Usage and Optimization Opportunities in Workflows of GitHub Actions
Article No.: 25, Pages 1 - 12
Abstract
Continuous integration and continuous delivery (CI/CD) has become a prevalent practice in software development. GitHub Actions is emerging as a popular platform for implementing CI/CD pipelines, called workflows, especially because the platform offers 2,000 minutes of computation for free to public repositories each month. To understand what these resources are used for and whether CI/CD could be more efficient, this paper presents the first comprehensive empirical study of resource usage and optimization opportunities of GitHub Action workflows. Our findings show that CI/CD imposes significant costs, e.g., $504 per year for an average paid-tier repository. The majority of the used resources is consumed by testing and building (91.2%), which is triggered by pull requests (50.7%), pushes (30.9%), and regularly scheduled workflows (15.5%). While existing optimizations, such as caching (adopted by 32.9% of paid-tier repositories), demonstrate a positive impact, they overall remain underutilized. This result underscores the need for enhanced documentation and tools to guide developers toward more resource-efficient workflows. Moreover, we show that relatively simple changes in the platform, such as deactivating scheduled workflows when repositories are inactive, could result in reductions of execution time between 1.1% and 31.6% over the impacted workflows. Overall, we envision our findings to help improve the resource efficiency of CI/CD pipelines.
References
[1]
Jeff Anderson, Saeed Salem, and Hyunsook Do. 2015. Striving for failure: an industrial case study about test failure prediction. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 49--58.
[2]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In International Conference on Mining Software Repositories.
[3]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration. In IEEE/ACM International Conference on Mining Software Repositories.
[4]
Giacomo Benedetti, Luca Verderame, and Alessio Merlo. 2022. Automatic Security Assessment of GitHub Actions Workflows. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses. ACM, Los Angeles CA USA, 37--45.
[5]
Rajkumar Buyya, Anton Beloglazov, and Jemal Abawajy. 2010. Energy-efficient management of data center resources for cloud computing: A vision, architectural elements, and open challenges. arXiv preprint arXiv:1006.0308 (2010).
[6]
Bihuan Chen, LinLin Chen, Chen Zhang, and Xin Peng. 2020. BuildFast: History-Aware Build Outcome Prediction for Fast Feedback and Reduced Cost in Continuous Integration. In ASE.
[7]
Tingting Chen, Yang Zhang, Shu Chen, Tao Wang, and Yiwen Wu. 2021. Let's Supercharge the Workflows: An Empirical Study of GitHub Actions. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, Hainan, China, 01--10.
[8]
Ozren Dabic, Emad Aghajani, and Gabriele Bavota. 2021. Sampling Projects in GitHub for MSR Studies. Technical Report arXiv:2103.04682. arXiv. http://arxiv.org/abs/2103.04682 arXiv:2103.04682 [cs] type: article.
[9]
Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. 2022. On the Use of GitHub Actions in Software Development Repositories. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, Limassol, Cyprus, 235--245.
[10]
Sebastian G. Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for improving regression testing in continuous integration development environments. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 -- 22, 2014, Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (Eds.). ACM, 235--245.
[11]
Wagner Felidré, Leonardo Furtado, Daniel A da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous integration theater. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1--10.
[12]
Keheliya Gallaba, John Ewart, Yves Junqueira, and Shane McIntosh. 2022. Accelerating Continuous Integration by Caching Environments and Inferring Dependencies. IEEE Trans. Software Eng. 48, 6 (2022), 2040--2052.
[13]
Keheliya Gallaba, Maxime Lamothe, and Shane McIntosh. 2022. Lessons from Eight Years of Operational Data from a Continuous Integration Service: An Exploratory Case Study of CircleCI. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25--27, 2022. ACM, 1330--1342.
[14]
Keheliya Gallaba and Shane McIntosh. 2018. Use and misuse of continuous integration features: An empirical study of projects that (mis) use Travis CI. IEEE Transactions on Software Engineering 46, 1 (2018), 33--50.
[15]
Mehdi Golzadeh, Alexandre Decan, and Tom Mens. 2022. On the rise and fall of CI services in GitHub. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Honolulu, HI, USA, 662--672.
[16]
Ahmed E. Hassan and Ken Zhang. 2006. Using Decision Trees to Predict the Certification Result of a Build. In Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering (ASE '06). IEEE Computer Society, USA, 189--198.
[17]
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 197--207.
[18]
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 426--437.
[19]
Hwa-You Hsu and Alessandro Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In 2009 IEEE 31st international conference on software engineering. IEEE, 419--429.
[20]
Xianhao Jin and Francisco Servant. 2020. A Cost-efficient Approach to Building in Continuous Integration. In ICSE.
[21]
Xianhao Jin and Francisco Servant. 2021. What helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).
[22]
Xianhao Jin and Francisco Servant. 2022. Which builds are really safe to skip? Maximizing failure observation for build selection in continuous integration. Journal of Systems and Software 188 (June 2022), 111292.
[23]
Timothy Kinsman, Mairieli Wessel, Marco A. Gerosa, and Christoph Treude. 2021. How Do Software Developers Use GitHub Actions to Automate Their Workflows? (2021).
[24]
Igibek Koishybayev, Aleksandr Nahapetyan, Raima Zachariah, Siddharth Muralee, Bradley Reaves, Alexandros Kapravelos, and Aravind Machiry. 2022. Characterizing the Security of Github CI Workflows. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 2747--2763. https://www.usenix.org/conference/usenixsecurity22/presentation/koishybayev
[25]
Adriaan Labuschagne, Laura Inozemtseva, and Reid Holmes. 2017. Measuring the cost of regression testing in practice: A study of Java projects using continuous integration. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 821--830.
[26]
Jingjing Liang, Sebastian Elbaum, and Gregg Rothermel. 2018. Redefining prioritization: continuous prioritization for continuous integration. In Proceedings of the 40th International Conference on Software Engineering. 688--698.
[27]
Yafeng Lu, Yiling Lou, Shiyang Cheng, Lingming Zhang, Dan Hao, Yangfan Zhou, and Lu Zhang. 2016. How does regression test prioritization perform in real-world software evolution?. In Proceedings of the 38th International Conference on Software Engineering. 535--546.
[28]
Mateusz Machalica, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. Predictive test selection. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25--31, 2019, Helen Sharp and Mike Whalen (Eds.). IEEE / ACM, 91--100.
[29]
Atif M. Memon, Zebao Gao, Bao N. Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-Scale Continuous Testing. In 39th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, Buenos Aires, Argentina, May 20--28, 2017. IEEE Computer Society, 233--242.
[30]
Cong Pan and Michael Pradel. 2021. Continuous test suite failure prediction. In ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11--17, 2021, Cristian Cadar and Xiangyu Zhang (Eds.). ACM, 553--565.
[31]
Saeed Parsa and Alireza Khalilian. 2010. On the optimization approach towards test suite minimization. International Journal of Software Engineering and its applications 4, 1 (2010), 15--28.
[32]
David Paterson, José Campos, Rui Abreu, Gregory M Kapfhammer, Gordon Fraser, and Phil McMinn. 2019. An Empirical Study on the Use of Defect Prediction for Test Case Prioritization. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 346--357.
[33]
Chris Preist, Daniel Schien, and Eli Blevis. 2016. Understanding and mitigating the effects of device and cloud service design decisions on the environmental footprint of digital infrastructure. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1324--1337.
[34]
Akond Rahman, Chris Parnin, and Laurie Williams. 2019. The seven sins: Security smells in infrastructure as code scripts. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 164--175.
[35]
T. Rausch, W. Hummer, P. Leitner, and S. Schulte. 2017. An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 345--355.
[36]
Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. 2019. Learning to Fix Build Errors with Graph2Diff Neural Networks. (2019).
[37]
Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. 2019. Automated reporting of anti-patterns and decay in continuous integration. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25--31, 2019, Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.). IEEE / ACM, 105--115.
[38]
Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, and Massimiliano Di Penta. 2020. Configuration smells in continuous delivery pipelines: a linter and a six-month study on GitLab. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8--13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 327--337.
[39]
Mairieli Wessel, Joseph Vargovich, Marco A. Gerosa, and Christoph Treude. 2022. GitHub Actions: The Impact on the Pull Request Process. (2022).
[40]
Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25 (2020), 1095--1135.
[41]
Celal Ziftci and Jim Reardon. 2017. Who Broke the Build? Automatically Identifying Changes That Induce Test Failures in Continuous Integration at Google Scale. In 39th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, Buenos Aires, Argentina, May 20--28, 2017. IEEE Computer Society, 113--122.
Index Terms
- Resource Usage and Optimization Opportunities in Workflows of GitHub Actions
Recommendations
gawd: A Differencing Tool for GitHub Actions Workflows
MSR '24: Proceedings of the 21st International Conference on Mining Software RepositoriesThe GitHub social coding platform introduced GitHub Actions as a way to automate different aspects of collaborative software development through the use of workflow files. It is the most popular CI/CD and workflow automation tool for GitHub. To maintain ...
ActionsRemaker: Reproducing GitHub Actions
ICSE '23: Proceedings of the 45th International Conference on Software Engineering: Companion ProceedingsMining Continuous Integration and Continuous Delivery (CI/CD) has enabled new research opportunities for the software engineering (SE) research community. However, it remains a challenge to reproduce CI/CD build processes, which is crucial for several ...
On the outdatedness of workflows in the GitHub Actions ecosystem
AbstractGitHub Actions was introduced as a way to automate CI/CD workflows in GitHub, the largest social coding platform. Thanks to its deep integration into GitHub, GitHub Actions can be used to automate a wide range of social and technical activities. ...
Highlights- Most GitHub Actions workflows rely on reusable Actions.
- Workflows and Actions are continuously updated and follow semantic versioning.
- Most workflows are using Actions that are outdated for more than seven months.
- Most ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
2942 pages
ISBN:9798400702174
DOI:10.1145/3597503
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 February 2024
Check for updates
Badges
Qualifiers
- Research-article
Funding Sources
Conference
ICSE '24
Sponsor:
ICSE '24: IEEE/ACM 46th International Conference on Software Engineering
April 14 - 20, 2024
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 276 of 1,856 submissions, 15%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 219Total Downloads
- Downloads (Last 12 months)219
- Downloads (Last 6 weeks)28
Reflects downloads up to 26 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in