research-article

Java unit testing tool competition: seventh round

Authors:

Fitsum Kifetew,

Xavier Devroey,

Urko RuedaAuthors Info & Claims

SBST '19: Proceedings of the 12th International Workshop on Search-Based Software Testing

Pages 15 - 20

https://doi.org/10.1109/SBST.2019.00014

Published: 27 May 2019 Publication History

Get Access

Abstract

We report on the results of the seventh edition of the JUnit tool competition. This year, four tools were executed on a benchmark with (i) new classes, selected from real-world software projects, and (ii) challenging classes from the previous edition. We use Randoop and manual test suites from the projects as baselines. Given the interesting findings of last year, we analyzed the effectiveness of the combined test suites generated by all competing tools and compared; results are confronted with the manual test suites of the projects, as well as those generated by the competing tools. This paper describes our methodology and the results, highlight challenges faced during the contest.

References

[1]

A. Arcuri, J. Campos, and G. Fraser. Unit test generation during software development: Evosuite plugins for Maven, IntelliJ and Jenkins. In IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 401--408. IEEE Computer Society, 2016.

Google Scholar

[2]

P. Braione. Tardis concolic test case generator, 2019. Available at: https://github.com/pietrobraione/tardis.

Google Scholar

[3]

P. Braione, G. Denaro, A. Mattavelli, and M. Pezzè. Combining symbolic execution and search-based testing for programs with complex heap inputs. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2017, pages 90--101, New York, NY, USA, 2017. ACM.

Digital Library

Google Scholar

[4]

P. Braione, G. Denaro, A. Mattavelli, and M. Pezzè. SUSHI: A test generator for programs with complex structured inputs. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE '18, pages 21--24, New York, NY, USA, 2018. ACM.

Digital Library

Google Scholar

[5]

G. Fraser and A. Arcuri. A large scale evaluation of automated unit test generation using evosuite. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(2):8, 2014.

Digital Library

Google Scholar

[6]

F. Kifetew, X. Devroey, and U. R. Molina. Java unit testing tool competition - seventh round. Technical report, 2019. Available at: https://github.com/PROSRESEARCHCENTER/junitcontest/blob/master/publications/SBSTcontest2019_detailed_results.pdf.

Google Scholar

[7]

C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, OOPSLA '07, pages 815--816, New York, NY, USA, 2007. ACM.

Digital Library

Google Scholar

[8]

A. Panichella, F. M. Kifetew, and P. Tonella. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering, 44(2):122--158, Feb 2018.

Crossref

Google Scholar

[9]

I. Prasetya. T3, a combinator-based random testing tool for Java: Benchmarking. Int. Workshop Future Internet Testing, Lecture Notes in Computer Science, 8432, 2014.

Google Scholar

[10]

I. Prasetya. T3i: A tool for generating and querying test suites for java. In 10th Joint Meeting of the European Software Engineering Conference (ESEC) and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE). ACM, 2015.

Digital Library

Google Scholar

[11]

U. Rueda Molina, F. Kifetew, and A. Panichella. Java unit testing tool competition: Sixth round. In Proceedings of the 11th International Workshop on Search-Based Software Testing, SBST '18, pages 22--29, New York, NY, USA, 2018. ACM.

Google Scholar

Cited By

View all

Herculano WMongiovi MAlves ECavalcante EDantas FBatista T(2020)Manually Written or Generated Tests?Proceedings of the XXXIV Brazilian Symposium on Software Engineering10.1145/3422392.3422416(273-282)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3422392.3422416
Olsthoorn Mvan Deursen APanichella AGrundy JLe Goues CLo D(2020)Generating highly-structured input data by combining search-based testing and grammar-based fuzzingProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3418930(1224-1228)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3418930
Roy DZhang ZMa MArnaoudova VPanichella APanichella SGonzalez DMirakhorli MGrundy JLe Goues CLo D(2020)DeepTC-enhancerProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416622(287-298)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416622
Show More Cited By

Recommendations

Java Unit Testing Tool Competition: Eighth Round
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

We report on the results of the eighth edition of the Java unit testing tool competition. This year, two tools, EvoSuite and Randoop, were executed on a benchmark with (i) new classes under test, selected from open-source software projects, and (ii) the ...
Java unit testing tool competition: sixth round
SBST '18: Proceedings of the 11th International Workshop on Search-Based Software Testing

We report on the advances in this sixth edition of the JUnit tool competitions. This year the contest introduces new benchmarks to assess the performance of JUnit testing tools on different types of real-world software projects. Following on the ...
Java unit testing tool competition: fifth round
SBST '17: Proceedings of the 10th International Workshop on Search-Based Software Testing

After four successful JUnit tool competitions, we report on the achievements of a new Java Unit Testing Tool Competition. This 5th contest introduces statistical analyses in the benchmark infrastructure and has been validated with significance against ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SBST '19: Proceedings of the 12th International Workshop on Search-Based Software Testing

May 2019

41 pages

Program Chairs:
Alessandra Gorla
IMDEA Software, Madrid, Spain
,
José Miguel Rojas
University of Leicester, United Kingdom

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Herculano WMongiovi MAlves ECavalcante EDantas FBatista T(2020)Manually Written or Generated Tests?Proceedings of the XXXIV Brazilian Symposium on Software Engineering10.1145/3422392.3422416(273-282)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3422392.3422416
Olsthoorn Mvan Deursen APanichella AGrundy JLe Goues CLo D(2020)Generating highly-structured input data by combining search-based testing and grammar-based fuzzingProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3418930(1224-1228)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3418930
Roy DZhang ZMa MArnaoudova VPanichella APanichella SGonzalez DMirakhorli MGrundy JLe Goues CLo D(2020)DeepTC-enhancerProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416622(287-298)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416622
Prasetya IGorla ARojas J(2019)Random testing with austere budgeting in T3Proceedings of the 12th International Workshop on Search-Based Software Testing10.1109/SBST.2019.00015(21-24)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/SBST.2019.00015

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Java Unit Testing Tool Competition: Eighth Round

Java unit testing tool competition: sixth round

Java unit testing tool competition: fifth round