A Comparison of Bug Finding Tools for Java

Reviewer: Andrew Brooks

Static analysis tools hold great potential in software quality assurance, but how effective are they__?__ Five Java bug finding tools (Bandera, ESC/Java, FindBugs, Jlint, and PMD) were applied to five programs that ranged in size from around 8,000 to 55,000 lines of code. The tools often generated well over 1,000 warnings on any one program, so the authors decided to manually examine several dozen warnings only. Their analysis revealed that there was a wide variation in the warnings provided by the tools, that some warnings were not about real defects (false positives), and that a single bug can create a cascade of warnings. Given the absence of a single best bug finding tool, the authors propose two metrics: the normalized warning count, and the unique warning total, measured at an individual Java class level across the output from several bug finding tools. Metric results for the poorest performing classes (Figure 10) suggest that classes with an unusually high warning count tend also to have a larger breadth of unique warnings. These results, however, do not fully have proof-of-concept value since no analysis was undertaken to match the real defects present in the code with the warnings reported by the tools. This paper serves as a useful introduction to static analysis tools for Java, and makes several recommendations to improve these tools, whose usefulness is severely comprised by the sheer volume of warnings. As such, this paper is strongly recommended to those using and researching static analysis tools. Online Computing Reviews Service