research-article

Is using deep learning frameworks free?: characterizing technical debt in deep learning frameworks

Authors:

Shanping LiAuthors Info & Claims

ICSE-SEIS '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society

Pages 1 - 10

https://doi.org/10.1145/3377815.3381377

Published: 18 September 2020 Publication History

Abstract

Developers of deep learning applications (shortened as application developers) commonly use deep learning frameworks in their projects. However, due to time pressure, market competition, and cost reduction, developers of deep learning frameworks (shortened as framework developers) often have to sacrifice software quality to satisfy a shorter completion time. This practice leads to technical debt in deep learning frameworks, which results in the increasing burden to both the application developers and the framework developers in future development.

In this paper, we analyze the comments indicating technical debt (self-admitted technical debt) in 7 of the most popular open-source deep learning frameworks. Although framework developers are aware of such technical debt, typically the application developers are not. We find that: 1) there is a significant number of technical debt in all the studied deep learning frameworks. 2) there is design debt, defect debt, documentation debt, test debt, requirement debt, compatibility debt, and algorithm debt in deep learning frameworks. 3) the majority of the technical debt in deep learning framework is design debt (24.07% - 65.27%), followed by requirement debt (7.09% - 31.48%) and algorithm debt (5.62% - 20.67%). In some projects, compatibility debt accounts for more than 10%. These findings illustrate that technical debt is common in deep learning frameworks, and many types of technical debt also impact the deep learning applications. Based on our findings, we highlight future research directions and provide recommendations for practitioners.

References

[1]

Nicolli SR Alves, Leilane F Ribeiro, Vivyane Caires, Thiago S Mendes, and Rodrigo O Spínola. 2014. Towards an ontology of terms on technical debt. In Managing Technical Debt (MTD), 2014 Sixth International Workshop on. IEEE, 1--7.

Digital Library

[2]

Gabriele Bavota and Barbara Russo. 2016. A large-scale empirical study on self-admitted technical debt. In Mining Software Repositories (MSR), 2016 IEEE/ACM 13th Working Conference on. IEEE, 315--326.

Digital Library

[3]

Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, et al. 2010. Managing technical debt in software-reliant systems. In Proceedings of the FSE/SDP workshop on Future of software engineering research. ACM, 47--52.

Digital Library

[4]

Ward Cunningham. 1993. The WyCash portfolio management system. ACM SIGPLAN OOPS Messenger 4, 2 (1993), 29--30.

Digital Library

[5]

Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement 33, 3 (1973), 613--619.

[6]

Francesca Arcelli Fontana, Vincenzo Ferme, and Stefano Spinelli. 2012. Investigating the impact of code smells debt on quality code evaluation. In Proceedings of the Third International Workshop on Managing Technical Debt. IEEE Press, 15--22.

Digital Library

[7]

Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional.

Digital Library

[8]

Elmar Juergens, Florian Deissenboeck, and Benjamin Hummel. 2010. Code similarities beyond copy & paste. In 2010 14th European Conference on Software Maintenance and Reengineering. IEEE, 78--87.

Digital Library

[9]

Elmar Juergens, Florian Deissenboeck, Benjamin Hummel, and Stefan Wagner. 2009. Do code clones matter?. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 485--495.

Digital Library

[10]

Yasutaka Kamei, Everton da S Maldonado, Emad Shihab, and Naoyasu Ubayashi. 2016. Using Analytics to Quantify Interest of Self-Admitted Technical Debt. In QuASoQ/TDA@ APSEC. 68--71.

[11]

Noela Jemutai Kipyegen and William P K Korir. 2013. Importance of Software Documentation. International Journal of Computer Science Issues 10, 5 (2013), 6.

[12]

Tim Klinger, Peri Tarr, Patrick Wagstrom, and Clay Williams. 2011. An enterprise perspective on technical debt. In Proceedings of the 2nd Workshop on managing technical debt. ACM, 35--38.

Digital Library

[13]

Rainer Koschke. 2007. Survey of research on software clones. In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.

[14]

Philippe Kruchten, Robert L Nord, Ipek Ozkaya, and Davide Falessi. 2013. Technical debt: towards a crisper definition report on the 4th international workshop on managing technical debt. ACM SIGSOFT Software Engineering Notes 38, 5 (2013), 51--54.

Digital Library

[15]

Zengyang Li, Paris Avgeriou, and Peng Liang. 2015. A systematic mapping study on technical debt and its management. Journal of Systems and Software 101 (2015), 193--220.

Digital Library

[16]

Erin Lim, Nitin Taksande, and Carolyn Seaman. 2012. A balancing act: What software practitioners have to say about technical debt. IEEE software 29, 6 (2012), 22--27.

[17]

Everton Maldonado, Rabe Abdalkareem, Emad Shihab, and Alexander Serebrenik. 2017. An Empirical Study On the Removal of Self-Admitted Technical Debt. In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME'17). IEEE.

[18]

Everton Maldonado and Emad Shihab. 2015. Detecting and quantifying different types of self-admitted technical Debt. In Proceedings of the 7th IEEE International Workshop on Managing Technical Debt (MTD'15). 9--15.

[19]

Everton Maldonado, Emad Shihab, and Nikolaos Tsantalis. 2017. Using natural language processing to automatically detect self-admitted technical debt. IEEE Transactions on Software Engineering 43, 11 (2017), 1044--1062.

Digital Library

[20]

Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica 22, 3 (2012), 276--282.

[21]

Roman Novak, Yasaman Bahri, Dan Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. 2018. Sensitivity and Generalization in Neural Networks: an Empirical Study. https://openreview.net/pdf?id=HJC2SzZCW

[22]

Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 1--18.

Digital Library

[23]

Aniket Potdar and Emad Shihab. 2014. An Exploratory Study on Self-Admitted Technical Debt. In Proceedings of the 30th IEEE International Conference on Software Maintenance and Evolution (ICSME'14). 91--100.

Digital Library

[24]

Chanchal Kumar Roy and James R Cordy. 2007. A survey on software clone detection research. Queen's School of Computing TR 541, 115 (2007), 64--68.

[25]

David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Advances in neural information processing systems. 2503--2511.

[26]

Carolyn Seaman and Yuepu Guo. 2011. Measuring and monitoring technical debt. In Advances in Computers. Vol. 82. Elsevier, 25--46.

[27]

Securities and E. Commission. [n. d.]. SEC Charges Knight Capital With Violations of Market Access Rule. https://www.sec.gov/news/press-release/2013-222. 2013.

[28]

Hai Shu and Hongtu Zhu. 2019. Sensitivity Analysis of Deep Neural Networks. (01 2019).

[29]

Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media.

[30]

Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering. ACM, 303--314.

Digital Library

[31]

Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. 2016. Examining the impact of self-admitted technical debt on software quality. In Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on, Vol. 1. IEEE, 179--188.

[32]

Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. 2016. Examining the Impact of Self-admitted Technical Debt on Software Quality. In Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER'16). 11.

[33]

Nico Zazworka, Michele A Shaw, Forrest Shull, and Carolyn Seaman. 2011. Investigating the impact of design debt on software quality. In Proceedings of the 2nd Workshop on Managing Technical Debt. ACM, 17--23.

Digital Library

[34]

Nico Zazworka, Michele A. Shaw, Forrest Shull, and Carolyn Seaman. 2011. Investigating the Impact of Design Debt on Software Quality. In Proceedings of the 2Nd Workshop on Managing Technical Debt (MTD '11). ACM, New York, NY, USA, 17--23.

Digital Library

[35]

Nico Zazworka, Rodrigo O. Spínola, Antonio Vetro', Forrest Shull, and Carolyn Seaman. 2013. A Case Study on Effectively Identifying Technical Debt. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE '13). ACM, New York, NY, USA, 42--47.

Digital Library

[36]

Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An Empirical Study on TensorFlow Program Bugs. International Symposium on Software Testing and Analysis (2018).

Cited By

Freire SPacheco ARios NPérez BCastellanos CCorreal DRamač RMandić VTaušan NLópez GMendonça MFalessi DIzurieta CSeaman CSpínola R(2024)A Comprehensive View on TD Prevention Practices and Reasons for Not Preventing ItACM Transactions on Software Engineering and Methodology10.1145/367472733:7(1-44)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3674727
Chen HBabar M(2024)Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and ChallengesACM Computing Surveys10.1145/363853156:6(1-38)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1145/3638531
Zhou ZLi MYu HFan GYang PHuang Z(2024)Learning to Generate Structured Code Summaries From Hybrid Code ContextIEEE Transactions on Software Engineering10.1109/TSE.2024.343956250:10(2512-2528)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3439562
Show More Cited By

Index Terms

Is using deep learning frameworks free?: characterizing technical debt in deep learning frameworks
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software
      2. Software evolution

Recommendations

An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks
Abstract
To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior ...
Cross-sectional Stock Price Prediction using Deep Learning for Actual Investment Management
ASSE '20: Proceedings of the 2020 Asia Service Sciences and Software Engineering Conference

Stock price prediction has been an important research theme both academically and practically. Various methods to predict stock prices have been studied until now. The feature that explains the stock price by a cross-section analysis is called a "factor"...
Financial Portfolio Construction for Quantitative Trading Using Deep Learning Technique
Artificial Intelligence and Soft Computing
Abstract
Stock portfolio construction is a difficult task which involves the simultaneous consideration of dynamic financial data as well as investment criteria (e.g.: investors required return, risk tolerance, goals, and time frame). The objective of this ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIS '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society

June 2020

106 pages

ISBN:9781450371254

DOI:10.1145/3377815

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
465
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)17

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Freire SPacheco ARios NPérez BCastellanos CCorreal DRamač RMandić VTaušan NLópez GMendonça MFalessi DIzurieta CSeaman CSpínola R(2024)A Comprehensive View on TD Prevention Practices and Reasons for Not Preventing ItACM Transactions on Software Engineering and Methodology10.1145/367472733:7(1-44)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3674727
Chen HBabar M(2024)Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and ChallengesACM Computing Surveys10.1145/363853156:6(1-38)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1145/3638531
Zhou ZLi MYu HFan GYang PHuang Z(2024)Learning to Generate Structured Code Summaries From Hybrid Code ContextIEEE Transactions on Software Engineering10.1109/TSE.2024.343956250:10(2512-2528)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3439562
Recupito GPecorelli FCatolino GLenarduzzi VTaibi DDi Nucci DPalomba F(2024)Technical debt in AI-enabled systems: On the prevalence, severity, impact, and management strategies for code and architectureJournal of Systems and Software10.1016/j.jss.2024.112151216(112151)Online publication date: Oct-2024
https://doi.org/10.1016/j.jss.2024.112151
Liu CCai RZhou YChen XHu HYan M(2024)Understanding the implementation issues when using deep learning frameworksInformation and Software Technology10.1016/j.infsof.2023.107367166:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107367
Song YXie XXu B(2024)When debugging encounters artificial intelligence: state of the art and open challengesScience China Information Sciences10.1007/s11432-022-3803-967:4Online publication date: 21-Feb-2024
https://doi.org/10.1007/s11432-022-3803-9
Sheikhaei MTian YWang SXu B(2024)An empirical study on the effectiveness of large language models for SATD identification and classificationEmpirical Software Engineering10.1007/s10664-024-10548-329:6Online publication date: 1-Oct-2024
https://doi.org/10.1007/s10664-024-10548-3
Sun TGao YKhaladkar SLiu SZhao LKim YHong S(2023)Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local ExplanationsProceedings of the ACM on Human-Computer Interaction10.1145/36101877:CSCW2(1-32)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610187
Zhang XChen LZou WCao YRen HWang ZLi YZhou Y(2023)ICG: A Machine Learning Benchmark Dataset and Baselines for Inline Code Comments Generation TaskInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402350054734:02(331-356)Online publication date: 20-Oct-2023
https://doi.org/10.1142/S0218194023500547
Ebrahimi AOliva GHassan A(2023)Self-Admitted Technical Debt in Ethereum Smart Contracts: A Large-Scale Exploratory StudyIEEE Transactions on Software Engineering10.1109/TSE.2023.328980849:9(4304-4323)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3289808
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents