research-article

History-driven test program synthesis for JVM testing

Authors:

Lingming ZhangAuthors Info & Claims

ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Pages 1133 - 1144

https://doi.org/10.1145/3510003.3510059

Published: 05 July 2022 Publication History

Abstract

Java Virtual Machine (JVM) provides the runtime environment for Java programs, which allows Java to be "write once, run anywhere". JVM plays a decisive role in the correctness of all Java programs running on it. Therefore, ensuring the correctness and robustness of JVM implementations is essential for Java programs. To date, various techniques have been proposed to expose JVM bugs via generating potential bug-revealing test programs. However, the diversity and effectiveness of test programs generated by existing research are far from enough since they mainly focus on minor syntactic/semantic mutations. In this paper, we propose JavaTailor, the first history-driven test program synthesis technique, which synthesizes diverse test programs by weaving the ingredients extracted from JVM historical bug-revealing test programs into seed programs for covering more JVM behaviors/paths. More specifically, JavaTailor first extracts five types of code ingredients from the historical bug-revealing test programs. Then, to synthesize diverse test programs, it iteratively inserts the extracted ingredients into the seed programs and strengthens their interactions via introducing extra data dependencies between them. Finally, JavaTailor employs these synthesized test programs to differentially test JVMs. Our experimental results on popular JVM implementations (i.e., HotSpot and OpenJ9) show that JavaTailor outperforms the state-of-the-art technique in generating more diverse and effective test programs, e.g., test programs generated by JavaTailor can achieve higher JVM code coverage and detect many more unique inconsistencies than the state-of-the-art technique. Furthermore, JavaTailor has detected 10 previously unknown bugs, 6 of which have been confirmed/fixed by developers.

References

[1]

2021. ASMTools. https://wiki.openjdk.java.net/display/CodeTools/asmtools.

[2]

2021. Bug-12552. https://github.com/eclipse-openj9/openj9/issues/12552.

[3]

2021. Bug-12819. https://github.com/eclipse-openj9/openj9/issues/12819.

[4]

2021. Gcov. https://gcc.gnu.org/onlinedocs/gcc/Gcov.html.

[5]

2021. GIJ. https://web.archive.org/web/20070509055923/http://gcc.gnu.org/java.

[6]

2021. Hotspot. http://openjdk.java.net.

[7]

2021. JavaTailor. https://github.com/JavaTailor/CFSynthesis

[8]

2021. Lcov. http://ltp.sourceforge.net/coverage/lcov.php.

[9]

2021. OpenJ9. https://www.eclipse.org/openj9.

[10]

2021. Zulu. https://www.azulsystems.com/products/core.

[11]

Bowen Alpern, Ton Ngo, Jong-Deok Choi, and Manu Sridharan. 2000. DejaVu: deterministic Java replay debugger for Jalapeño Java virtual machine. In Object Oriented Programming Systems Languages and Applications Conference. 165--166.

Digital Library

[12]

Stephen M. Blackburn, Robin Garner, Chris Hoffmann, Asjad M. Khan, Kathryn S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony L. Hosking, Maria Jump, Han Bok Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanovic, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo benchmarks: java benchmarking development and analysis. In Proceedings of the 21th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications. 169--190.

Digital Library

[13]

Marcel Böhme, Cristian Cadar, and Abhik Roychoudhury. 2020. Fuzzing: Challenges and Reflections. IEEE Software (2020).

[14]

Abdulazeez S. Boujarwah and Kassem Saleh. 1997. Compiler test case generation methods: a survey and assessment. Inf. Softw. Technol. 39, 9 (1997), 617--625.

[15]

Andrea Calvagna, Andrea Fornaia, and Emiliano Tramontana. 2014. Combinatorial Interaction Testing of a Java Card Static Verifier. In Seventh IEEE International Conference on Software Testing, Verification and Validation. 84--87.

Digital Library

[16]

Andrea Calvagna and Emiliano Tramontana. 2013. Automated Conformance Testing of Java Virtual Machines. In Seventh International Conference on Complex, Intelligent, and Software Intensive Systems. 547--552.

[17]

Andrea Calvagna and Emiliano Tramontana. 2013. Combinatorial Validation Testing of Java Card Byte Code Verifiers. In 2013 Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises. 347--352.

[18]

Felipe Canales, Geoffrey Hecht, and Alexandre Bergel. 2021. Optimization of Java Virtual Machine Flags using Feature Model and Genetic Algorithm. In ICPE '21: ACM/SPEC International Conference on Performance Engineering. 183--186.

Digital Library

[19]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Bing Xie. 2017. Learning to prioritize test programs for compiler testing. In Proceedings of the 39th International Conference on Software Engineering. 700--711.

Digital Library

[20]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. Test Case Prioritization for Compilers: A Text-Vector Based Approach. In 2016 IEEE International Conference on Software Testing, Verification and Validation. 266--277.

[21]

Junjie Chen, Jiaqi Han, Peiyi Sun, Lingming Zhang, Dan Hao, and Lu Zhang. 2019. Compiler bug isolation via effective witness test program generation. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223--234.

Digital Library

[22]

Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An empirical comparison of compiler testing techniques. In Proceedings of the 38th International Conference on Software Engineering. 180--190.

Digital Library

[23]

Junjie Chen, Haoyang Ma, and Lingming Zhang. 2020. Enhanced Compiler Bug Isolation via Memoized Search. In 35th IEEE/ACM International Conference on Automated Software Engineering. 78--89.

Digital Library

[24]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. ACM Comput. Surv. 53, 1 (2020), 4:1--4:36.

[25]

Junjie Chen and Chenyao Suo. 2022. Boosting Compiler Testing via Compiler Optimization Exploration. In TOSEM. to appear.

[26]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Lu Zhang. 2019. History-Guided Configuration Diversification for Compiler Test-Program Generation. In 34th IEEE/ACM International Conference on Automated Software Engineering. 305--316.

[27]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2021. Coverage Prediction for Accelerating Compiler Testing. IEEE Trans. Software Eng. 47, 2 (2021), 261--278.

Digital Library

[28]

Junjie Chen, Ningxin Xu, PeiqiChen, and Hongyu Zhang. 2021. Efficient Compiler Autotuning via Bayesian Optimization. In 43rd IEEE/ACM International Conference on Software Engineering. 1198--1209.

Digital Library

[29]

Lingchao Chen, Foyzul Hassan, Xiaoyin Wang, and Lingming Zhang. 2020. Taming Behavioral Backward Incompatibilities via Cross-Project Testing and Analysis. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 112--124.

Digital Library

[30]

Yuting Chen, Ting Su, and Zhendong Su. 2019. Deep differential testing of JVM implementations. In Proceedings of the 41st International Conference on Software Engineering. 1257--1268.

Digital Library

[31]

Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-directed differential testing of JVM implementations. In 37th ACM SIGPLAN Conference on Programming Language Design and Implementation. 85--99.

Digital Library

[32]

Jaeseung Choi, Kangsu Kim, Daejin Lee, and Sang Kil Cha. 2021. NTFUZZ: Enabling Type-Aware Kernel Fuzzing on Windows with Static Binary Analysis. (2021).

[33]

Stephen N. Freund and John C. Mitchell. 2003. A Type System for the Java Bytecode Language and Verifier. J. Autom. Reason. 30, 3--4 (2003), 271--321.

Digital Library

[34]

Vincenzo Gervasi and Roozbeh Farahbod. 2009. JASMine: Accessing Java Code from CoreASM. In Rigorous Methods for Software Construction and Analysis, Essays Dedicated to Egon Börger on the Occasion of His 60th Birthday, Vol. 5115. 170--186.

[35]

Matthias Grimmer, Manuel Rigger, Roland Schatz, Lukas Stadler, and Hanspeter Mössenböck. 2014. TruffleC: dynamic execution of C on a Java virtual machine. In 2014 International Conference on Principles and Practices of Programming on the Java Platform Virtual Machines, Languages and Tools. 17--26.

Digital Library

[36]

Sungjae Hwang, Sungho Lee, Jihoon Kim, and Sukyoung Ryu. 2021. JUSTGen: Effective Test Generation for Unspecified JNI Behaviors on JVMs. In 43rd IEEE/ACM International Conference on Software Engineering. 1708--1718.

[37]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In ACM SIGPLAN Conference on Programming Language Design and Implementation. 216--226.

Digital Library

[38]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. 386--399.

Digital Library

[39]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation. 65--76.

[40]

Barton P Miller, Louis Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Commun. ACM 33, 12 (1990), 32--44.

Digital Library

[41]

Mayur Naik, Chang-Seo Park, Koushik Sen, and David Gay. 2009. Effective static deadlock detection. In 31st International Conference on Software Engineering. 386--396.

Digital Library

[42]

Manuel Rigger, Matthias Grimmer, Christian Wimmer, Thomas Würthinger, and Hanspeter Mössenböck. 2016. Bringing low-level languages to the JVM: efficient execution of LLVM IR on Truffle. In Proceedings of the 8th International Workshop on Virtual Machines and Intermediate Languages. 6--15.

Digital Library

[43]

Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021. A comprehensive study of deep learning compiler bugs. In 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). 968--980.

[44]

Emin Gün Sirer and Brian N. Bershad. 1999. Using production grammars in software testing. In Proceedings of the Second Conference on Domain-Specific Languages. 1--13.

[45]

James E. Smith and Ravi Nair. 2005. Virtual machines - versatile platforms for systems and processes. Elsevier.

Digital Library

[46]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. 849--863.

Digital Library

[47]

Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie J. Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java bytecode optimization framework. In Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative Research. 13.

Digital Library

[48]

Haichi Wang, Zan Wang, Jun Sun, Shuang Liu, Ayesha Sadiq, and Yuan-Fang Li. 2020. Towards Generating Thread-Safe Classes Automatically. In 35th IEEE/ACM International Conference on Automated Software Engineering. 943--955.

[49]

Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020. Deep learning library testing via effective model generation. In 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). 788--799.

Digital Library

[50]

Zan Wang, Yingquan Zhao, Shuang Liu, Jun Sun, Xiang Chen, and Huarui Lin. 2019. MAP-Coverage: A Novel Coverage Criterion for Testing Thread-Safe Classes. In 34th IEEE/ACM International Conference on Automated Software Engineering. 722--734.

[51]

Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. 2022. Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE).

Digital Library

[52]

Ming Yan, Junjie Chen, Xiangyu Zhang, Lin Tan, Gan Wang, and Zan Wang. 2021. Exposing numerical bugs in deep learning via gradient back-propagation. In ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 627--638.

Digital Library

[53]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation. 283--294.

Digital Library

[54]

Takahide Yoshikawa, Kouya Shimura, and Toshihiro Ozawa. 2003. Random Program Generator for Java JIT Compiler Test System. In 3rd International Conference on Quality Software. 20.

[55]

Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 132--142.

Digital Library

Cited By

Yang WGao CLiu XLi YXue YFilkov VRay BZhou M(2024)Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695059(631-642)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695059
Schwarcz FBerlakovich FBarany GMössenböck HBöhme MNoller YSzekeres L(2024)LOOL: Low-Overhead, Optimization-Log-Guided Compiler Fuzzing (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685533(42-51)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685533
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Show More Cited By

Index Terms

History-driven test program synthesis for JVM testing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers

Recommendations

Vectorizing Program Ingredients for Better JVM Testing
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

JVM testing is one of the most widely-used methodologies for guaranteeing the quality of JVMs. Among various JVM testing techniques, synthesis-based JVM testing, which constructs a test program by synthesizing various code snippets (also called ...
Coverage-directed differential testing of JVM implementations
PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation

Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to ...
Coverage-directed differential testing of JVM implementations
PLDI '16

Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '22: Proceedings of the 44th International Conference on Software Engineering

May 2022

2508 pages

ISBN:9781450392211

DOI:10.1145/3510003

General Chair:
Matthew B Dwyer
University of Virginia
,
Program Chairs:
Daniela Damian
University of Victoria, Canada
,
Andreas Zeller
CISPA, Germany

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
496
Total Downloads

Downloads (Last 12 months)158
Downloads (Last 6 weeks)21

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang WGao CLiu XLi YXue YFilkov VRay BZhou M(2024)Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695059(631-642)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695059
Schwarcz FBerlakovich FBarany GMössenböck HBöhme MNoller YSzekeres L(2024)LOOL: Low-Overhead, Optimization-Log-Guided Compiler Fuzzing (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685533(42-51)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685533
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Suo CChen JLiu SJiang JZhao YWang JChristakis MPradel M(2024)Fuzzing MLIR Compiler Infrastructure via Operation Dependency AnalysisProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680360(1287-1299)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680360
Zhao WZeng RZhou YChristakis MPradel M(2024)Wapplique: Testing WebAssembly Runtime via Execution Context-Aware Bytecode MutationProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680340(1035-1047)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680340
Hu TYe GTang ZTan SWang HLi MWang ZChristakis MPradel M(2024)UPBEAT: Test Input Checks of Q# Quantum LibrariesProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652120(186-198)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652120
Zang ZYu FThimmaiah AShi AGligoric M(2024)Java JIT Testing with Template ExtractionProceedings of the ACM on Software Engineering10.1145/36437771:FSE(1129-1151)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643777
Xia CPaltenghi MLe Tian JPradel MZhang LRoychoudhury APaiva AAbreu RStorey M(2024)Fuzz4All: Universal Fuzzing with Large Language ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639121(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639121
Tian ZWang FChen YChen L(2024)Differential testing solidity compiler through deep contract manipulation and mutationSoftware Quality Journal10.1007/s11219-024-09673-532:2(765-790)Online publication date: 23-Apr-2024
https://dl.acm.org/doi/10.1007/s11219-024-09673-5
Ye GHu TTang ZFan ZTan SZhang BQian WWang ZChandra SBlincoe KTonella P(2023)A Generative and Mutational Approach for Synthesizing Bug-Exposing Test Cases to Guide Compiler FuzzingProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616332(1127-1139)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616332
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents