research-article

Are static analysis violations really fixed?: a closer look at realistic usage of SonarQube

Authors:

Diego Marcilio,

Rodrigo Bonifácio,

Eduardo Monteiro,

Gustavo PintoAuthors Info & Claims

ICPC '19: Proceedings of the 27th International Conference on Program Comprehension

Pages 209 - 219

https://doi.org/10.1109/ICPC.2019.00040

Published: 25 May 2019 Publication History

Abstract

The use of automatic static analysis tools (ASATs) has gained increasing attention in the last few years. Even though available research have already explored ASATs issues and how they are fixed, these studies rely on revisions of the software, instead of mining real usage of these tools and real issue reports. In this paper we contribute with a comprehensive, multi-method study about the usage of SonarQube (a popular static analysis tool), mining 421,976 issues from 246 projects in four different instance of SonarQube: two hosted in open-source communities (Eclipse and Apache) and two hosted in Brazilian government institutions (Brazilian Court of Account (TCU) and Brazilian Federal Police (PF)). We first surveyed team leaders of the analyzed projects and found that they mostly consider ASATs warning messages as relevant for overall software improvement. Second, we found that both Eclipse and TCU employ highly customized instance of SonarQube, with more than one thousand distinct checkers-though just a subset of these checkers actually led to issues' reports. Surprisingly, we found a low resolution rate per project in all organizations-on average, 13% of the issues have been solved in the systems. We conjecture that just a subset of the checkers reveal real design and coding flaws, and this might artificially increase the technical debt of the systems. Nevertheless, considering all systems, there is a central tendency (median) of fixing issues after 18.99 days they had been reported, faster than the period for fixing bugs as reported in previous studies.

References

[1]

R. K. Saha, Y. Lyu, W. Lam, H. Yoshida, and M. R. Prasad, "Bugs.jar," in Proceedings of the 15th International Conference on Mining Software Repositories - MSR'18. ACM Press, 2018. {Online}. Available

[2]

B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, "Why don't software developers use static analysis tools to find bugs?" in Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 2013, pp. 672--681.

Digital Library

[3]

J. Wang, S. Wang, and Q. Wang, "Is there a "golden" feature set for static warning identification?" in Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement - ESEM'18. ACM Press, 2018. {Online}. Available

Digital Library

[4]

C. Vassallo, S. Panichella, F. Palomba, S. Proksch, A. Zaidman, and H. C. Gall, "Context is king: The developer perspective on the usage of static analysis tools," in 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, mar 2018. {Online}. Available

[5]

F. Zampetti, S. Scalabrino, R. Oliveto, G. Canfora, and M. D. Penta, "How open source projects use static code analysis tools in continuous integration pipelines," in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, may 2017. {Online}. Available

Digital Library

[6]

Y. Lin, S. Okur, and D. Dig, "Study and refactoring of android asynchronous programming (T)," in 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, 2015, pp. 224--235.

Digital Library

[7]

A. Decan, T. Mens, and E. Constantinou, "On the impact of security vulnerabilities in the npm package dependency network," in Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018, Gothenburg, Sweden, May 28-29, 2018, 2018, pp. 181--191.

Digital Library

[8]

G. Pinto, A. Canino, F. Castor, G. H. Xu, and Y. D. Liu, "Understanding and overcoming parallelism bottlenecks in forkjoin applications," in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017, 2017, pp. 765--775.

Digital Library

[9]

M. F. Aniche, G. Bavota, C. Treude, M. A. Gerosa, and A. van Deursen, "Code smells for model-view-controller architectures," Empirical Software Engineering, vol. 23, no. 4, pp. 2121--2157, 2018.

Digital Library

[10]

C. Vassallo, F. Palomba, A. Bacchelli, and H. C. Gall, "Continuous code quality: are we (really) doing that?" in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering - ASE 2018. ACM Press, 2018. {Online}. Available

Digital Library

[11]

M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, "Analyzing the state of static analysis: A large-scale evaluation in open source software," in 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, mar 2016. {Online}. Available

[12]

G. Digkas, M. Lungu, P. Avgeriou, A. Chatzigeorgiou, and A. Ampatzoglou, "How do developers fix issues and pay back technical debt in the apache ecosystem?" in 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, mar 2018. {Online}. Available

[13]

K. Liu, D. Kim, T. F. Bissyande, S. Yoo, and Y. L. Traon, "Mining fix patterns for FindBugs violations," IEEE Transactions on Software Engineering, pp. 1--1, 2018. {Online}. Available

[14]

J. L. C. Izquierdo and J. Cabot, "The role of foundations in open source projects," in Proceedings of the 40th International Conference on Software Engineering Software Engineering in Society - ICSE-SEIS'18. ACM Press, 2018. {Online}. Available

Digital Library

[15]

F. Nielson, H. R. Nielson, and C. Hankin, Principles of Program Analysis. Berlin, Heidelberg: Springer-Verlag, 1999.

Digital Library

[16]

D. Hovemeyer and W. Pugh, "Finding bugs is easy," ACM SIGPLAN Notices, vol. 39, no. 12, p. 92, dec 2004. {Online}. Available

Digital Library

[17]

SonarSource S.A., "Sonarqube," 2019, {accessed 18-January-2019}. {Online}. Available: https://www.sonarqube.org

[18]

I. E. Allen and C. A. Seaman, "Likert scales and data analyses," Quality progress, vol. 40, no. 7, p. 64, 2007.

[19]

G. Bavota, G. Canfora, M. D. Penta, R. Oliveto, and S. Panichella, "The evolution of project inter-dependencies in a software ecosystem: The case of apache," in 2013 IEEE International Conference on Software Maintenance. IEEE, sep 2013. {Online}. Available

Digital Library

[20]

M. Hollander and D. Wolfe, Nonparametric Statistical Methods, ser. Wiley Series in Probability and Statistics. Wiley, 1999.

[21]

O. J. Dunn, "Multiple comparisons among means," Journal of the American statistical association, vol. 56, no. 293, pp. 52--64, 1961.

[22]

M. M. Mukaka, "A guide to appropriate use of correlation coefficient in medical research," Malawi Medical Journal, vol. 24, no. 3, pp. 69--71, 2012.

[23]

E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, "An in-depth study of the promises and perils of mining GitHub," Empirical Software Engineering, vol. 21, no. 5, pp. 2035--2071, sep 2015. {Online}. Available

Digital Library

[24]

L. D. Panjer, "Predicting eclipse bug lifetimes," in Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007), May 2007, pp. 29--29.

Digital Library

[25]

E. Giger, M. Pinzger, and H. Gall, "Predicting the fix time of bugs," in Proceedings of the 2Nd International Workshop on Recommendation Systems for Software Engineering, ser. RSSE '10. New York, NY, USA: ACM, 2010, pp. 52--56. {Online}. Available

Digital Library

[26]

S. Kim and M. D. Ernst, "Which warnings should i fix first?" in Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering - ESEC-FSE'07. ACM Press, 2007. {Online}. Available

Digital Library

[27]

N. E. Fenton and N. Ohlsson, "Quantitative analysis of faults and failures in a complex software system," IEEE Transactions on Software Engineering, vol. 26, no. 8, pp. 797--814, Aug 2000.

Digital Library

[28]

P. Runeson and C. Andersson, "A replicated quantitative analysis of fault distributions in complex software systems," IEEE Transactions on Software Engineering, vol. 33, pp. 273--286, 2007.

Digital Library

[29]

N. Walkinshaw and L. L. Minku, "Are 20% of files responsible for 80% of defects?" in Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018, Oulu, Finland, October 11-12, 2018. ACM, 2018, pp. 2:1--2:10.

Digital Library

[30]

T. Rausch, W. Hummer, P. Leitner, and S. Schulte, "An empirical analysis of build failures in the continuous integration workflows of java-based open-source software," in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, may 2017. {Online}. Available

Digital Library

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Romano SToriello GCassieri PFrancese RScanniello G(2024)A Folklore Confirmation on the Removal of Dead CodeProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661188(333-338)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661188
Johansson AHolmberg CGomes De Oliveira Neto FLeitner PBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)The Impact of Compiler Warnings on Code Quality in C++ ProjectsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644410(270-279)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644410
Show More Cited By

Recommendations

Static Analysis of JNI Programs via Binary Decompilation
JNI programs are widely used thanks to the combined benefits of C and Java programs. However, because understanding the interaction behaviors between two different programming languages is challenging, JNI program development is difficult to get right and ...
Generating REST API Specifications through Static Analysis
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Web Application Programming Interfaces (APIs) allow services to be accessed over the network. RESTful (or REST) APIs, which use the REpresentation State Transfer (REST) protocol, are a popular type of web API. To use or test REST APIs, developers use ...
Sorald: Automatic Patch Suggestions for SonarQube Static Analysis Violations
Previous work has shown that early resolution of issues detected by static code analyzers can prevent major costs later on. However, developers often ignore such issues for two main reasons. First, many issues should be interpreted to determine if they ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '19: Proceedings of the 27th International Conference on Program Comprehension

May 2019

400 pages

General Chair:
Yann-Gaël Guéhéneuc
Concordia University, Montréal, Canada
,
Program Chairs:
Foutse Khomh
Polytechnique Montréal, Canada
,
Federica Sarro
University College London, London, United Kingdom

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 25 May 2019

Check for updates

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 25, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
144
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Romano SToriello GCassieri PFrancese RScanniello G(2024)A Folklore Confirmation on the Removal of Dead CodeProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661188(333-338)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661188
Johansson AHolmberg CGomes De Oliveira Neto FLeitner PBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)The Impact of Compiler Warnings on Code Quality in C++ ProjectsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644410(270-279)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644410
Kree LHelmke RWinter E(2024)Using Semgrep OSS to Find OWASP Top 10 Weaknesses in PHP Applications: A Case StudyDetection of Intrusions and Malware, and Vulnerability Assessment10.1007/978-3-031-64171-8_4(64-83)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-64171-8_4
Dantas CRocha AMaia M(2023)Assessing the Readability of ChatGPT Code Snippet Recommendations: A Comparative StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613413(283-292)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613413
Nocera SFrancese RScanniello G(2023)Training Bachelor Students to Design Better Quality Web Apps: Preliminary Results from a Prospective Empirical InvestigationProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593957(465-469)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593957
Yu PWu YPeng XPeng JZhang JXie PZhao WGrundy JPollock LPenta M(2023)ViolationTracker: Building Precise Histories for Static Analysis ViolationsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00171(2022-2034)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00171
Romano SZampetti FBaldassarre MDi Penta MScanniello G(2022)Do Static Analysis Tools Affect Software Quality when Using Test-driven Development?Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3544902.3546233(80-91)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1145/3544902.3546233
Kim MKim YJeong HHeo JKim SChung HLee ERoychoudhury ACadar CKim M(2022)An empirical study of deep transfer learning-based program repair for Kotlin projectsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558967(1441-1452)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558967
Kang HAw KLo DDwyer MDamian DZeller A(2022)Detecting false alarms from automatic static analysis toolsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510214(698-709)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510214
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents