Demystifying API misuses in deep learning applications
Authors: 
Deheng Yang, Kui Liu, Yan Lei, Li Li, Huan Xie, Chunyan Liu, Zhenyu Wang, Xiaoguang Mao, Tegawendé F. BissyandéAuthors Info & Claims
Abstract
Deep Learning (DL) is achieving staggering performance on an increasing number of applications in various areas. Meanwhile, its associated data-driven programming paradigm comes with a set of challenges for the software engineering community, including the debugging activities for DL applications. Recent empirical studies on bugs in DL applications have shown that the API (i.e., Application Program Interface) misuse has been flagged as an important category of DL programming bugs. By exploring this literature towards API misuse bugs in DL applications, we identified three barriers that are locking an entire research direction. However, three barriers are hindering progress in this research direction: misclassification of API misuse bugs, lack of relevant dataset, and limited depth of analysis. Our work unlocks these barriers by providing an in-depth analysis of a frequent bug type that appears as a mystery. Concretely, we first offer a new perspective to a significant misclassification issue in the literature that hinders understanding of API misuses in DL applications. Subsequently, we curate the first dataset MisuAPI of 143 API misuses sampled from real-world DL applications. Finally, we perform systematic analyses to dissect API misuses and enumerate the symptoms of API misuses in DL applications as well as investigate the possibility of detecting them with state-of-the-art static analyzers. Overall, the insights summarized in this work are important for the community: 1) 18-35% of real API misuses are mislabelled in existing DL bug studies; 2) the widely adopted API misuse taxonomy, namely MUC, does not cover the cases of 1 out of 3 encountered API misuses; 3) DL library API misuses show significant differences from the general third-party library API misuses in terms of the API-usage element issue and symptoms; 4) Most (92.3%) API misuses lead to program crashes; 5) 95.8% API misuses remain undetectable by state-of-the-art static analyzers.
References
[1]
A curated list of static analysis (sast) tools for all programming languages. https://github.com/analysis-tools-dev/static-analysis#python. Accessed June 2021
[2]
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, et al (2016) Tensorflow: a system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), p 265–283
[3]
Al-Rfou R, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F, Bayer J, Belikov A, The Theano Development Team et al (2016)Theano: a python framework for fast computation of mathematical expressions. arXiv:1605.02688
[4]
Amann S, Nguyen HA, Nadi S, Nguyen TN, and Mezini M A systematic evaluation of static api-misuse detectors IEEE Trans Softw Eng 2018 45 12 1170-1188
[5]
Amann S, Nadi S, Nguyen HA, Nguyen TN, Mezini M (2016) Mubench: a benchmark for api-misuse detectors. In Proceedings of the 13th international conference on mining software repositories, pp 464–467
[6]
Amann S, Nguyen HA, Nadi S, Nguyen TN, Mezini M (2019) Investigating next steps in static api-misuse detection. In 2019 IEEE/ACM 16th international conference on mining software repositories (MSR), pp 265–275. IEEE
[7]
Artifact page of our study (2023). https://github.com/DehengYang/MisuAPI
[8]
Bonifacio R, Krüger S, Narasimhan K, Bodden E, Mezini M (2021) Dealing with variability in api misuse specification. arXiv:2105.04950
[9]
Cambronero J, Li H, Kim S, Sen K, Chandra S (2019) When deep learning met code search. In Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 964–974
[10]
Cao J, Li M, Chen X, Wen M, Tian Y, Wu B, Cheung S-C (2022) Deepfd: automated fault diagnosis and localization for deep learning programs. In: Proceedings of the 44th international conference on software engineering, pp 573–585
[11]
Casalnuovo C, Suchak Y, Ray B, Rubio-González C (2017) Gitcproc: a tool for processing and classifying github commits. In: Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis, pp 396–399
[12]
CEO Nvidia (2023) Software is eating the world, but AI is going to eat software. T. Simonite
[13]
Chen Z, Yao H, Lou Y, Cao Y, Liu Y, Wang H, Liu X (2021) An empirical study on deployment faults of deep learning based mobile applications. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), pp 674–685. IEEE
[14]
Dilhara M, Ketkar A, Dig D (2021) Understanding software-2.0: a study of machine learning library usage and evolution. ACM Trans Soft Eng Methodol (TOSEM) 30(4):1–42
[15]
Eghbali A, Pradel M (2020) No strings attached: an empirical study of string-related software bugs. In: 2020 35th IEEE/ACM international conference on automated software engineering (ASE), pp 956–967. IEEE
[16]
Example of a missing api with missing exception handling. https://github.com/tensorpack/tensorpack/commit/132dcccd34a831a01e4fcdbd32f869b36f04537e. Accessed June 2021
[17]
Example of a misused api with incorrect api call sequence. https://github.com/deezer/spleeter/commit/55723cfa6296388ea1f584e2591f1d89e4c0afb6. Accessed June 2021
[18]
Example of a misused api with missing api call. https://github.com/tensorflow/models/commit/001a260214ba34f36e149bbd24f7f5d6a6634500. Accessed June 2021
[19]
Example of a misused api with missing condition. https://github.com/tensorpack/tensorpack/commit/ae84b52ad5402ab1716e0f1e9790ce1da9d706d1. Accessed June 2021
[20]
Example of a misused dl library api depending on the specific device. https://github.com/google/prettytensor/commit/01ee67d6e0cc5e9d6ae5f07045024a638564fe78. Accessed June 2021
[21]
Example of an incorrect parameter value. https://github.com/google/tf-quant-finance/commit/258844720a9bccd326c7b33735f7f81c2d483630. Accessed June 2021
[22]
Falleri J-R, Morandat F, Blanc X, Martinez M, Monperrus M (2014) Fine-grained and accurate source code differencing. In: Proceedings of the 29th ACM/IEEE international conference on automated software engineering, pp 313–324
[23]
Forward A, Lethbridge TC (2008) A taxonomy of software types to facilitate search and evidence-based software engineering. In: Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds, pp 179–191
[24]
Github api. https://docs.github.com/en/rest/reference/search. Accessed June 2021
[25]
Gulli A, Pal S (2017) Deep learning with Keras. Packt Publishing Ltd
[26]
Gu Z, Wu J, Liu J, Zhou M, Gu M (2019) An empirical study on api-misuse bugs in open-source c programs. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 1, pp 11–20. IEEE
[27]
Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, pp 1110–1121
[28]
Institute of Electrical and Electronics Engineers (1987) IEEE Standard Taxonomy for Software Engineering Standards
[29]
Islam MdJ (2020) Towards understanding the challenges faced by machine learning software developers and enabling automated solutions
[30]
Islam MdJ, Nguyen HA, Pan R, Rajan H (2019) What do developers ask about ml libraries? a large-scale study using stack overflow. arXiv:1906.11940
[31]
Islam MdJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 510–520
[32]
Islam MdJ, Pan R, Nguyen G, Rajan H (2020) Repairing deep neural networks: fix patterns and challenges. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), pp 1135–1146. IEEE
[33]
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia, pp 675–678
[34]
Just R, Jalali D, Ernst MD (2014) Defects4j: a database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 international symposium on software testing and analysis, pp 437–440
[35]
Kechagia M, Devroey X, Panichella A, Gousios G, van Deursen A (2019) Effective and efficient api misuse detection via exception propagation and search-based testing. In: Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp 192–203
[36]
Kechagia M, Mechtaev S, Sarro F, Harman M (2021) Evaluating automatic program repair capabilities to repair api misuses. IEEE Trans Softw Eng
[37]
Kuutti S, Bowden R, Jin Y, Barber P, and Fallah S A survey of deep learning applications to autonomous vehicle control IEEE Trans Intell Trans Syst 2020 22 2 712-733
[38]
Kwasnik BH (1999) The role of classification in knowledge representation and discovery
[39]
Lamothe M, Guéhéneuc Y-G, and Shang W A systematic review of api evolution literature ACM Comput Surv (CSUR) 2021 54 8 1-36
[40]
Lamothe M, Li H, Shang W (2021) Assisting example-based api misuse detection via complementary artificial examples. IEEE Trans Softw Eng
[41]
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics, pp 159–174
[42]
Li X, Jiang J, Benton S, Xiong Y, Zhang L (2021) A large-scale study on api misuses in the wild. In: 2021 14th IEEE conference on software testing, verification and validation (ICST), pp 241–252. IEEE
[43]
Liu Y, Liu G, and Zhang Q Deep learning and medical diagnosis Lancet 2019 394 10210 1709-1710
[44]
Liu K, Kim D, Koyuncu A, Li L, Bissyandé TF, Le Traon Y (2018) A closer look at real-world patches. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), p 275–286. IEEE
[45]
Mama R (2021) Example of a misused api with incorrect parameter. https://github.com/Rayhane-mamah/Tacotron-2/commit/0ae2901b428afd4127272154b71705e2799a484d. Accessed June 2021
[46]
Mamah R (2023) The example of inner api misuse in dl application. https://github.com/Rayhane-mamah/Tacotron-2/commit/fb5564b7584ae0dc62ffecaa89d463ff24a3c251. Accessed Aug 2023
[47]
McHugh ML Interrater reliability: the kappa statistic Biochem Med 2012 22 3 276-282
[48]
Meijer E (2018) Behind every great deep learning framework is an even greater programming languages concept (keynote). In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 1–1
[49]
mypy. https://github.com/python/mypy. Accessed June 2021
[50]
Nielebock S, Heumüller R, Schott KM, Ortmeier F (2020) Guided pattern mining for api misuse detection by change-based code analysis. arXiv:2008.00277
[51]
Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch
[52]
pylint. https://github.com/PyCQA/pylint. Accessed June 2021
[53]
pyre-check. https://github.com/facebook/pyre-check. Accessed June 2021
[54]
pyright. https://github.com/microsoft/pyright/. Accessed June 2021
[55]
Python standard library. https://docs.python.org/3/library/. Accessed June 2021
[56]
Ren X, Ye X, Xing Z, Xia X, Xu X, Zhu L, Sun J (2020) Api-misuse detection driven by fine-grained api-constraint knowledge graph. In: 2020 35th IEEE/ACM international conference on automated software engineering (ASE), pp 461–472. IEEE
[57]
Scalabrino S, Bavota G, Linares-Vásquez M, Lanza M, Oliveto R (2019) Data-driven solutions to detect api compatibility issues in android: an empirical study. In: 2019 IEEE/ACM 16th international conference on mining software repositories (MSR), pp 288–298. IEEE
[58]
Shen Q, Ma H, Chen J, Tian Y, Cheung S-C, Chen X (2021) A comprehensive study of deep learning compiler bugs. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 968–980
[59]
Šmite D, Wohlin C, Galviņa Z, and Prikladnicki R An empirically based terminology and taxonomy for global software engineering Empir Softw Eng 2014 19 1 105-153
[60]
Svyatkovskiy A, Deng SK, Fu S, Sundaresan N (2020) Intellicode compose: code generation using transformer. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 1433–1443
[61]
Tensorflow repositories in githubs. https://github.com/search?q=tensorflow &type=. Accessed June 2021
[62]
The manual verification results for api bugs provided by Islam et al. https://zenodo.org/record/8302351. Accessed Aug 2023
[63]
Unterkalmsteiner M, Feldt R, and Gorschek T A taxonomy for requirements engineering and software test alignment ACM Trans Softw Engi Methodol (TOSEM) 2014 23 2 1-38
[64]
Usman M, Britto R, Börstler J, and Mendes E Taxonomies in software engineering: a systematic mapping study and a revised taxonomy development method Inf Softw Technol 2017 85 43-59
[65]
Usman M, Gopinath D, Sun Y, Noller Y, Păsăreanu CS (2021) Nn repair: constraint-based repair of neural network classifiers. In: Computer aided verification: 33rd international conference, CAV 2021, Virtual Event, July 20–23, 2021, Proceedings, Part I 33, pp 3–25. Springer
[66]
Vélez TC, Khatchadourian R, Bagherzadeh M, Raja A (2022) Challenges in migrating imperative deep learning programs to graph execution: an empirical study. In: Proceedings of the 19th international conference on mining software repositories, pp 469–481
[67]
Wan C, Liu S, Hoffmann H, Maire M, Lu S (2021) Are machine learning cloud apis used correctly? In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), pp 125–137. IEEE
[68]
Wardat M, Cruz BD, Le W, Rajan H (2022) Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs. In: Proceedings of the 44th international conference on software engineering, pp 561–572
[69]
Wardat M, Le W, Rajan H (2021) Deeplocalize: fault localization for deep neural networks. In 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), p 251–262. IEEE
[70]
Wen M, Liu Y, Wu R, Xie X, Cheung S-C, Su Z (2019) Exposing library api misuses via mutation analysis. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE), pp 866–877. IEEE
[71]
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media
[72]
Wu D, Shen B, Chen Y (2021) An empirical study on tensor shape faults in deep learning systems. arXiv:2106.02887
[73]
Yan M, Chen J, Zhang X, Tan L, Wang G, Wang Z (2021) Exposing numerical bugs in deep learning via gradient back-propagation. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 627–638
[74]
Yang Y, Xia X, Lo D, Grundy J (2020) A survey on deep learning for software engineering. arXiv:2011.14597
[75]
Yu B, Qi H, Guo Q, Juefei-Xu F, Xie X, Ma L, and Zhao J Deeprepair: style-guided repairing for deep neural networks in the real-world operational environment IEEE Trans Reliab 2021 71 4 1401-1416
[76]
Zar JH (2005) Spearman rank correlation. Encyclopedia of Biostatistics, 7
[77]
Zhang Y, Chen Y, Cheung S-C, Xiong Y, Zhang L (2018) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pp 129–140
[78]
Zhang T, Gao C, Ma L, Lyu M, Kim M (2019) An empirical study of common challenges in developing deep learning applications. In: 2019 IEEE 30th international symposium on software reliability engineering (ISSRE), pp 104–115. IEEE
[79]
Zhang T, Upadhyaya G, Reinhardt A, Rajan H, Kim M (2018) Are code examples on an online q &a forum reliable? a study of api misuse on stack overflow. In: Proceedings of the 40th international conference on software engineering, pp 886–896
[80]
Zhang T, Upadhyaya G, Reinhardt A, Rajan H, Kim M (2018) Are online code examples reliable? an empirical study of api misuse on stack overflow. In: International conference on software engineering (ICSE), vol 10
[81]
Zhang R, Xiao W, Zhang H, Liu Y, Lin H, Yang M (2020) An empirical study on program failures of deep learning jobs. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, pp 1159–1170
[82]
Zhong H, Su Z (2015) An empirical study on real bug fixes. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 1, pp 913–923. IEEE
Recommendations
Demystifying and Detecting Misuses of Deep Learning APIs
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software EngineeringDeep Learning (DL) libraries have significantly impacted various domains in computer science over the last decade. However, developers often face challenges when using the DL APIs, as the development paradigm of DL applications differs greatly from ...
An Empirical Study of API Misuses of Data-Centric Libraries
ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementDevelopers rely on third-party library Application Programming Interfaces (APIs) when developing software. However, libraries typically come with assumptions and API usage constraints, whose violation results in API misuse. API misuses may result in ...
Effective and efficient API misuse detection via exception propagation and search-based testing
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisApplication Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are several tools that can detect API misuses, most of them ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 16 February 2024
Accepted: 18 October 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in