research-article

Public Access

NonDex: a tool for detecting and debugging wrong assumptions on Java API specifications

Authors:

Owolabi Legunsen,

Darko MarinovAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 993 - 997

https://doi.org/10.1145/2950290.2983932

Published: 01 November 2016 Publication History

Abstract

We present NonDex, a tool for detecting and debugging wrong assumptions on Java APIs. Some APIs have underdetermined specifications to allow implementations to achieve different goals, e.g., to optimize performance. When clients of such APIs assume stronger-than-specified guarantees, the resulting client code can fail. For example, HashSet’s iteration order is underdetermined, and code assuming some implementation-specific iteration order can fail. NonDex helps to proactively detect and debug such wrong assumptions. NonDex performs detection by randomly exploring different behaviors of underdetermined APIs during test execution. When a test fails during exploration, NonDex searches for the invocation instance of the API that caused the failure. NonDex is open source, well-integrated with Maven, and also runs from the command line. During our experiments with the NonDex Maven plugin, we detected 21 new bugs in eight Java projects from GitHub, and, using the debugging feature of NonDex, we identified the underlying wrong assumptions for these 21 new bugs and 54 previously detected bugs. We opened 13 pull requests; developers already accepted 12, and one project changed the continuous-integration configuration to run NonDex on every push. The demo video is at: https://youtu.be/h3a9ONkC59c

References

[1]

ASM: A code manipulation tool to implement adaptable systems. In Adaptable and extensible component systems, Grenoble, France, Nov. 2002.

[2]

Checkstyle pull request integrating NonDex. https://github.com/checkstyle/checkstyle/pull/3393.

[3]

Fix internal data ordering. https://github.com/ geosolutions-it/geoserver-manager/commit/5447c06.

[4]

HashSet Javadoc. https://docs.oracle.com/javase/8/ docs/api/java/util/HashSet.html.

[5]

Improvements to HashMap/LinkedHashMap use of bins/buckets and trees (red/black), Sept. 2013. http:// hg.openjdk.java.net/jdk8/jdk8/jdk/rev/d62c911aebbb.

[6]

JUnit 4.11 - What’s new? Test execution order. http://randomallsorts.blogspot.com/2012/12/ junit-411-whats-new-test-execution-order.html.

[7]

JSON. http://www.json.org/.

[8]

JUnit and Java 7. http://intellijava.blogspot.com/ 2012/05/junit-and-java-7.html.

[9]

B. Liskov and J. Guttag. Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley, 2000.

Digital Library

[10]

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov. An empirical analysis of flaky tests. In FSE, 2014.

Digital Library

[11]

Maintaining the order of JUnit3 tests with JDK 1.7. http://www.coderanch.com/t/600985/Testing/ Maintaining-order-JUnit-tests-JDK.

[12]

all and sundry: JUnit test method ordering. http://www.java-allandsundry.com/2013/01/ junit-test-method-ordering.html.

[13]

Maven. https://maven.apache.org/.

[14]

Maven Central. https://repo1.maven.org/maven2/ edu/illinois/nondex-maven-plugin/.

[15]

NonDex Source Code. https://github.com/TestingResearchIllinois/NonDex.

[16]

A. Shi, A. Gyori, O. Legunsen, and D. Marinov. Detecting assumptions on deterministic implementations of non-deterministic specifications. In ICST, 2016.

[17]

Must Use LinkedHashMap and LinkedList... https://github.com/EsotericSoftware/yamlbeans/ commit/1517822.

[18]

A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. TSE, 28(2), 2002.

Digital Library

Cited By

Rahman SShi ARoychoudhury APaiva AAbreu RStorey M(2024)FlakeSync: Automatically Repairing Async Flaky TestsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639115(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639115
ElGazzar MHossny EOmara F(2024)A survey of Detecting Flakiness in Automated Test Regression Suite2024 21st Learning and Technology Conference (L&T)10.1109/LT60077.2024.10469624(330-336)Online publication date: 15-Jan-2024
https://doi.org/10.1109/LT60077.2024.10469624
Alshammari AAmmann PHilton MBell J(2024)230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00031(257-268)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00031
Show More Cited By

Index Terms

NonDex: a tool for detecting and debugging wrong assumptions on Java API specifications
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Google Faculty Research Award
National Science Foundation

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
640
Total Downloads

Downloads (Last 12 months)123
Downloads (Last 6 weeks)14

Reflects downloads up to 30 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rahman SShi ARoychoudhury APaiva AAbreu RStorey M(2024)FlakeSync: Automatically Repairing Async Flaky TestsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639115(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639115
ElGazzar MHossny EOmara F(2024)A survey of Detecting Flakiness in Automated Test Regression Suite2024 21st Learning and Technology Conference (L&T)10.1109/LT60077.2024.10469624(330-336)Online publication date: 15-Jan-2024
https://doi.org/10.1109/LT60077.2024.10469624
Alshammari AAmmann PHilton MBell J(2024)230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00031(257-268)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00031
Chen YYildiz AMarinov DJabbarvand RJust RFraser G(2023)Transforming Test Suites into CroissantsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598119(1080-1092)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598119
Akli AHaben GHabchi SPapadakis MLe Traon Y(2023)FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00018(140-151)Online publication date: May-2023
https://doi.org/10.1109/AST58925.2023.00018
Rasheed SDietrich JTahir A(2023)On the Effect of Instrumentation on Test Flakiness2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00016(123-127)Online publication date: May-2023
https://doi.org/10.1109/AST58925.2023.00016
Yorihiro AJiang PMarqués VCarleton BLegunsen O(2023)eMOP: A Maven Plugin for Evolution-Aware Runtime VerificationRuntime Verification10.1007/978-3-031-44267-4_20(363-375)Online publication date: 1-Oct-2023
https://doi.org/10.1007/978-3-031-44267-4_20
Wong WGao RLi YAbreu RWotawa FLi D(2023)Software Fault Localization: an Overview of Research, Techniques, and ToolsHandbook of Software Fault Localization10.1002/9781119880929.ch1(1-117)Online publication date: 21-Apr-2023
https://doi.org/10.1002/9781119880929.ch1
Ngo KNguyen VNguyen T(2022)Research on Test Flakiness: from Unit to System TestingProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3563242(1-4)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3563242
Dietrich JRasheed STahir A(2022)Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM55253.2022.00037(264-275)Online publication date: Oct-2022
https://doi.org/10.1109/SCAM55253.2022.00037
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents