Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3597926.3598056acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

A Comprehensive Study on Quality Assurance Tools for Java

Published: 13 July 2023 Publication History

Abstract

Quality assurance (QA) tools are receiving more and more attention and are widely used by developers. Given the wide range of solutions for QA technology, it is still a question of evaluating QA tools. Most existing research is limited in the following ways: (i) They compare tools without considering scanning rules analysis. (ii) They disagree on the effectiveness of tools due to the study methodology and benchmark dataset. (iii) They do not separately analyze the role of the warnings. (iv) There is no large-scale study on the analysis of time performance. To address these problems, in the paper, we systematically select 6 free or open-source tools for a comprehensive study from a list of 148 existing Java QA tools. To carry out a comprehensive study and evaluate tools in multi-level dimensions, we first mapped the scanning rules to the CWE and analyze the coverage and granularity of the scanning rules. Then we conducted an experiment on 5 benchmarks, including 1,425 bugs, to investigate the effectiveness of these tools. Furthermore, we took substantial effort to investigate the effectiveness of warnings by comparing the real labeled bugs with the warnings and investigating their role in bug detection. Finally, we assessed these tools’ time performance on 1,049 projects. The useful findings based on our comprehensive study can help developers improve their tools and provide users with suggestions for selecting QA tools.

References

[1]
Edward Aftandilian, Raluca Sauciuc, Siddharth Priya, and Sundaresan Krishnan. 2012. Building Useful Program Analysis Tools Using an Extensible Java Compiler. In 2012 IEEE 12th International Working Conference on Source Code Analysis and Manipulation. 14–23. https://doi.org/10.1109/SCAM.2012.28
[2]
Bushra Aloraini and Meiyappan Nagappan. 2017. Evaluating State-of-the-Art Free and Open Source Static Analysis Tools Against Buffer Errors in Android Apps. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, Shanghai. 295–306. isbn:978-1-5386-0992-7 https://doi.org/10.1109/ICSME.2017.77
[3]
Nathaniel Ayewah, William Pugh, J. David Morgenthaler, John Penix, and YuQian Zhou. 2007. Evaluating Static Analysis Defect Warnings on Production Software. In Proceedings of the 7th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE ’07). Association for Computing Machinery, New York, NY, USA. 1–8. isbn:9781595935953 https://doi.org/10.1145/1251535.1251536
[4]
Alexandre Braga, Ricardo Dahab, Nuno Antunes, Nuno Laranjeiro, and Marco Vieira. 2017. Practical Evaluation of Static Analysis Tools for Cryptography: Benchmarking Method and Case Study. In 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE). IEEE, Toulouse. 170–181. isbn:978-1-5386-0941-5 https://doi.org/10.1109/ISSRE.2017.27
[5]
Braga, Alexandre and Dahab, Ricardo and Antunes, Nuno and Laranjeiro, Nuno and Vieira, Marco. 2019. Understanding How to Use Static Analysis Tools for Detecting Cryptography Misuse in Software. IEEE Transactions on Reliability, 68, 4 (2019), 1384–1403. https://doi.org/10.1109/TR.2019.2937214
[6]
Cristiano Calcagno, Dino Distefano, Jeremy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter O’Hearn, Irene Papakonstantinou, Jim Purbrick, and Dulma Rodriguez. 2015. Moving Fast with Software Verification. In NASA Formal Methods, Klaus Havelund, Gerard Holzmann, and Rajeev Joshi (Eds.). Springer International Publishing, Cham. 3–11. isbn:978-3-319-17524-9 https://doi.org/10.1007/978-3-319-17524-9_1
[7]
Foteini Cheirdari and George Karabatis. 2018. Analyzing False Positive Source Code Vulnerabilities Using Static Analysis Tools. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, Seattle, WA, USA. 4782–4788. isbn:978-1-5386-5035-6 https://doi.org/10.1109/BigData.2018.8622456
[8]
Sen Chen, Lingling Fan, Guozhu Meng, Ting Su, Minhui Xue, Yinxing Xue, Yang Liu, and Lihua Xu. 2020. An empirical assessment of security risks of global Android banking apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1310–1322. https://doi.org/10.1145/3377811.3380417
[9]
Sen Chen, Yuxin Zhang, Lingling Fan, Jiaming Li, and Yang Liu. 2022. AUSERA: Automated Security Vulnerability Detection for Android Apps. In 37th IEEE/ACM International Conference on Automated Software Engineering. 1–5. https://doi.org/10.1145/3551349.3559524
[10]
Valentin Dallmeier and Thomas Zimmermann. 2007. Extraction of Bug Localization Benchmarks from History. In Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering (ASE ’07). Association for Computing Machinery, New York, NY, USA. 433–436. isbn:9781595938824 https://doi.org/10.1145/1321631.1321702
[11]
Lisa Nguyen Quang Do, James R. Wright, and Karim Ali. 2022. Why Do Software Developers Use Static Analysis Tools? A User-Centered Study of Developer Needs and Motivations. IEEE Transactions on Software Engineering, 48, 3 (2022), March, 835–847. issn:0098-5589, 1939-3520, 2326-3881 https://doi.org/10.1109/TSE.2020.3004525
[12]
Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2020. Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 530–541. https://doi.org/10.1145/3377811.3380364 arXiv:1910.10601 [cs]
[13]
Thomas Durieux and Martin Monperrus. 2016. IntroClassJava: A Benchmark of 297 Small and Buggy Java Programs. Universite Lille 1. https://hal.archives-ouvertes.fr/hal-01272126/document
[14]
Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2018. Large-scale analysis of framework-specific exceptions in Android apps. In Proceedings of the 40th International Conference on Software Engineering. 408–419. https://doi.org/10.1145/3180155.3180222
[15]
The International Organization for Standardization. 2021. Information technology — Software measurement — Software quality measurement — Automated source code quality measures. https://www.iso.org/standard/80623.html
[16]
Asem Ghaleb and Karthik Pattabiraman. 2020. How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual Event USA. 415–427. isbn:978-1-4503-8008-9 https://doi.org/10.1145/3395363.3397385
[17]
GitHub. 2022. GitHub Static Analysis Tool List. https://github.com/analysis-tools-dev/static-analysis (Accessed on 10/11/2022)
[18]
Andrew Habib and Michael Pradel. 2018. How many of all bugs do we find? a study of static bug detectors. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, Montpellier France. 317–328. isbn:978-1-4503-5937-5 https://doi.org/10.1145/3238147.3238213
[19]
Tim Hepher. 2015. Exclusive: A400M Probe Focuses on Impact of Accidental Data Wipe. AEROSPACE AND DEFENSE.
[20]
David Hovemeyer and William Pugh. 2004. Finding Bugs is Easy. SIGPLAN Not., 39, 12 (2004), dec, 92–106. issn:0362-1340 https://doi.org/10.1145/1052883.1052895
[21]
Brittany Johnson. 2012. A study on improving static analysis tools: Why are we not using them? In 2012 34th International Conference on Software Engineering (ICSE). IEEE, Zurich. 1607–1609. isbn:978-1-4673-1066-6 978-1-4673-1067-3 https://doi.org/10.1109/ICSE.2012.6227228
[22]
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). Association for Computing Machinery, New York, NY, USA. 437–440. isbn:9781450326452 https://doi.org/10.1145/2610384.2628055
[23]
Kompar. 2022. The Kompar catalog of software analyzers. https://catalog.kompar.tools/analyzers (Accessed on 10/11/2022)
[24]
Valentina Lenarduzzi, Francesco Lomio, Heikki Huttunen, and Davide Taibi. 2020. Are SonarQube Rules Inducing Bugs? In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 501–511. https://doi.org/10.1109/SANER48275.2020.9054821
[25]
Valentina Lenarduzzi, Nyyti Saarimäki, and Davide Taibi. 2020. Some SonarQube issues have a significant but small effect on faults and changes. A large-scale empirical study. Journal of Systems and Software, 170 (2020), 110750. issn:0164-1212 https://doi.org/10.1016/j.jss.2020.110750
[26]
Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH Companion 2017). Association for Computing Machinery, New York, NY, USA. 55–56. isbn:9781450355148 https://doi.org/10.1145/3135932.3135941
[27]
Jacques-Louis Lions. 1996. ARIANE 5 Flight 501 Failure: Report by the Enquiry Board.
[28]
Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static C code analyzers for vulnerability detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual South Korea. 544–555. isbn:978-1-4503-9379-9 https://doi.org/10.1145/3533767.3534380
[29]
Bailin Lu, Wei Dong, Liangze Yin, and Li Zhang. 2018. Evaluating and Integrating Diverse Bug Finders for Effective Program Analysis. In Software Analysis, Testing, and Evolution, Lei Bu and Yingfei Xiong (Eds.). 11293, Springer International Publishing, Cham. 51–67. isbn:978-3-030-04271-4 978-3-030-04272-1 https://doi.org/10.1007/978-3-030-04272-1_4 Series Title: Lecture Notes in Computer Science
[30]
Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus. 2019. Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies. In Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’19). https://doi.org/10.1109/SANER.2019.8667991
[31]
Marcus Nachtigall, Michael Schlichtig, and Eric Bodden. 2022. A large-scale study of usability criteria addressed by static analysis tools. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual South Korea. 532–543. isbn:978-1-4503-9379-9 https://doi.org/10.1145/3533767.3534374
[32]
Zhen Ni, Bin Li, Xiaobing Sun, Tianhao Chen, Ben Tang, and Xinchen Shi. 2020. Analyzing bug fix for automatic bug cause classification. Journal of Systems and Software, 163 (2020), 110538. issn:0164-1212 https://doi.org/10.1016/j.jss.2020.110538
[33]
NIST. 2022. Source Code Security Analyzers. https://www.nist.gov/itl/ssd/software-quality-group/source-code-security-analyzers (Accessed on 10/11/2022)
[34]
Peter O’Hearn, John Reynolds, and Hongseok Yang. 2001. Local Reasoning about Programs that Alter Data Structures. In Computer Science Logic, Laurent Fribourg (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg. 1–19. isbn:978-3-540-44802-0 https://doi.org/10.1007/3-540-44802-0_1
[35]
OWASP. 2022. Source Code Analysis Tools. https://owasp.org/www-community/Source_Code_Analysis_Tools (Accessed on 10/11/2022)
[36]
Sebastiano Panichella, Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2015. Would static analysis tools help developers with code reviews? In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, Montreal, QC, Canada. 161–170. isbn:978-1-4799-8469-5 https://doi.org/10.1109/SANER.2015.7081826
[37]
PMD. 2022. PMD Source Code Analyzer. https://pmd.github.io (Accessed on 10/11/2022)
[38]
Kevin Poulsen. 2004. Software Bug Contributed to Blackout. SecurityFocus.
[39]
returntocorp. 2022. Semgrep. https://semgrep.dev
[40]
N. Rutar, C.B. Almazan, and J.S. Foster. 2004. A Comparison of Bug Finding Tools for Java. In 15th International Symposium on Software Reliability Engineering. IEEE, Saint-Malo, Bretagne, France. 245–256. isbn:978-0-7695-2215-9 https://doi.org/10.1109/ISSRE.2004.1
[41]
Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R. Prasad. 2018. Bugs.Jar: A Large-Scale, Diverse Dataset of Real-World Java Bugs. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR ’18). Association for Computing Machinery, New York, NY, USA. 10–13. isbn:9781450357166 https://doi.org/10.1145/3196398.3196473
[42]
Xiuhan Shi, Xiaofei Xie, Yi Li, Yao Zhang, Sen Chen, and Xiaohong Li. 2022. Large-scale analysis of non-termination bugs in real-world OSS projects. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 256–268. https://doi.org/10.1145/3540250.3549129
[43]
SonarSource. 2022. Sonarqube. https://www.sonarqube.org (Accessed on 10/11/2022)
[44]
Spotbugs. 2022. Spotbugs. https://spotbugs.github.io (Accessed on 10/11/2022)
[45]
CWE Team. 2022. Common Weakness Enumeration. https://cwe.mitre.org (Accessed on 10/11/2022)
[46]
CWE Team. 2022. CWE VIEW: Hardware Design. https://cwe.mitre.org/data/definitions/1194.html (Accessed on 10/11/2022)
[47]
CWE Team. 2022. CWE VIEW: Research Concept. https://cwe.mitre.org/data/definitions/1000.html (Accessed on 10/11/2022)
[48]
CWE Team. 2022. CWE VIEW: Software Development. https://cwe.mitre.org/data/definitions/699.html (Accessed on 10/11/2022)
[49]
Ferdian Thung, Lucia, David Lo, Lingxiao Jiang, Foyzur Rahman, and Premkumar T. Devanbu. 2012. To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering - ASE 2012. ACM Press, Essen, Germany. 50. isbn:978-1-4503-1204-2 https://doi.org/10.1145/2351676.2351685
[50]
Ferdian Thung, Lucia, David Lo, Lingxiao Jiang, Foyzur Rahman, and Premkumar T. Devanbu. 2015. To what extent could we detect field defects? An extended empirical study of false negatives in static bug-finding tools. Automated Software Engineering, 22, 4 (2015), Dec., 561–602. issn:0928-8910, 1573-7535 https://doi.org/10.1007/s10515-014-0169-8
[51]
David A. Tomassi. 2018. Bugs in the wild: examining the effectiveness of static analyzers at finding real-world bugs. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Lake Buena Vista FL USA. 980–982. isbn:978-1-4503-5573-5 https://doi.org/10.1145/3236024.3275439
[52]
David A. Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar T. Devanbu, Bogdan Vasilescu, and Cindy Rubio-González. 2019. BugSwarm: mining and continuously growing a dataset of reproducible failures and fixes. In ICSE. IEEE / ACM, 339–349. https://doi.org/10.1109/ICSE.2019.00048
[53]
David A. Tomassi and Cindy Rubio-Gonzalez. 2021. On the Real-World Effectiveness of Static Bug Detectors at Finding Null Pointer Exceptions. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Melbourne, Australia. 292–303. isbn:978-1-66540-337-5 https://doi.org/10.1109/ASE51524.2021.9678535
[54]
Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Harald C. Gall, and Andy Zaidman. 2020. How developers engage with static analysis tools in different contexts. Empirical Software Engineering, 25, 2 (2020), March, 1419–1457. issn:1382-3256, 1573-7616 https://doi.org/10.1007/s10664-019-09750-5
[55]
Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald C. Gall. 2018. Context is king: The developer perspective on the usage of static analysis tools. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Campobasso. 38–49. isbn:978-1-5386-4969-5 https://doi.org/10.1109/SANER.2018.8330195
[56]
Stefan Wagner, Jan Jürjens, Claudia Koller, and Peter Trischberger. 2005. Comparing Bug Finding Tools with Reviews and Tests. In Testing of Communicating Systems, Ferhat Khendek and Rachida Dssouli (Eds.). 3502, Springer Berlin Heidelberg, Berlin, Heidelberg. 40–55. isbn:978-3-540-26054-7 978-3-540-32076-0 https://doi.org/10.1007/11430230_4 Series Title: Lecture Notes in Computer Science
[57]
Dave Wichers. 2022. Free for Open Source Application Security Tools. https://owasp.org/www-community/Free_for_Open_Source_Application_Security_Tools (Accessed on 10/11/2022)
[58]
Wikipedia. 2022. List of tools for static code analysis. https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis (Accessed on 10/11/2022)
[59]
J. Zheng, L. Williams, N. Nagappan, W. Snipes, J.P. Hudepohl, and M.A. Vouk. 2006. On the value of static analysis for fault detection in software. IEEE Transactions on Software Engineering, 32, 4 (2006), April, 240–253. issn:0098-5589 https://doi.org/10.1109/TSE.2006.38
[60]
Michael Zhivich and Robert K. Cunningham. 2009. The Real Cost of Software Errors. IEEE Security & Privacy, 7, 2 (2009), 87–90. https://doi.org/10.1109/MSP.2009.56

Cited By

View all
  • (2024)ERD-CQC : Enhanced Rule and Dependency Code Quality Check for JavaProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674820(377-386)Online publication date: 24-Jul-2024
  • (2024)Understanding and Detecting Annotation-Induced Faults of Static AnalyzersProceedings of the ACM on Software Engineering10.1145/36437591:FSE(722-744)Online publication date: 12-Jul-2024
  • (2024)Comprehensive Security Analysis and Threat Mitigation Strategies for React.js Applications: Leveraging SonarQube for Robust Security Assurance2024 IEEE 1st Karachi Section Humanitarian Technology Conference (KHI-HTC)10.1109/KHI-HTC60760.2024.10482157(1-6)Online publication date: 8-Jan-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis
July 2023
1554 pages
ISBN:9798400702211
DOI:10.1145/3597926
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Bug finding
  2. CWE
  3. Quality assurance tools
  4. Scanning rules

Qualifiers

  • Research-article

Funding Sources

  • the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme
  • National Satellite of Excellence in Trustworthy Software Systems (NSOE-TSS) project under the National Cybersecurity R\&D (NCR) Grant
  • National Research Foundation, Singapore, the Cyber Security Agency under its National Cybersecurity R\&D Programme
  • Natural Science Foundation of China and the Israel Science Foundation (NSFC-ISF) Joint Program

Conference

ISSTA '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)182
  • Downloads (Last 6 weeks)19
Reflects downloads up to 16 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)ERD-CQC : Enhanced Rule and Dependency Code Quality Check for JavaProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674820(377-386)Online publication date: 24-Jul-2024
  • (2024)Understanding and Detecting Annotation-Induced Faults of Static AnalyzersProceedings of the ACM on Software Engineering10.1145/36437591:FSE(722-744)Online publication date: 12-Jul-2024
  • (2024)Comprehensive Security Analysis and Threat Mitigation Strategies for React.js Applications: Leveraging SonarQube for Robust Security Assurance2024 IEEE 1st Karachi Section Humanitarian Technology Conference (KHI-HTC)10.1109/KHI-HTC60760.2024.10482157(1-6)Online publication date: 8-Jan-2024
  • (2023)Learning to Locate and Describe Vulnerabilities2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00045(332-344)Online publication date: 11-Sep-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media