Java JIT Testing with Template Extraction
Authors: 
Zhiqiang Zang, Fu-Yao Yu, Aditya Thimmaiah, August Shi, Milos GligoricAuthors Info & Claims
Article No.: 51, Pages 1129 - 1151
Abstract
We present LeJit, a template-based framework for testing Java just-in-time (JIT) compilers. Like recent template-based frameworks, LeJit executes a template---a program with holes to be filled---to generate concrete programs given as inputs to Java JIT compilers. LeJit automatically generates template programs from existing Java code by converting expressions to holes, as well as generating necessary glue code (i.e., code that generates instances of non-primitive types) to make generated templates executable. We have successfully used LeJit to test a range of popular Java JIT compilers, revealing five bugs in HotSpot, nine bugs in OpenJ9, and one bug in GraalVM. All of these bugs have been confirmed by Oracle and IBM developers, and 11 of these bugs were previously unknown, including two CVEs (Common Vulnerabilities and Exposures). Our comparison with several existing approaches shows that LeJit is complementary to them and is a powerful technique for ensuring Java JIT compiler correctness.
References
[1]
Azul Systems, Inc. 2018. AzulSystems/JavaFuzzer: Java* Fuzzer for Android*. https://github.com/AzulSystems/JavaFuzzer
[2]
Jonathan Bell and Gail Kaiser. 2014. Unit Test Virtualization with VMVM. In International Conference on Software Engineering. ACM, 550–561. https://doi.org/10.1145/2568225.2568248
[3]
Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated Testing Based on Java Predicates. In International Symposium on Software Testing and Analysis. ACM, 123–133. https://doi.org/10.1145/566171.566191
[4]
Ahmet Celik, Sreepathi Pai, Sarfraz Khurshid, and Milos Gligoric. 2017. Bounded Exhaustive Test-Input Generation on GPUs. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, 94:1–94:25. https://doi.org/10.1145/3133918
[5]
Stefanos Chaliasos, Thodoris Sotiropoulos, Diomidis Spinellis, Arthur Gervais, Benjamin Livshits, and Dimitris Mitropoulos. 2022. Finding Typing Compiler Bugs. In Programming Language Design and Implementation. ACM, 183–198. https://doi.org/10.1145/3519939.3523427
[6]
Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. Comput. Surveys, 53, 1 (2020), 4:1–4:36. https://doi.org/10.1145/3363562
[7]
Yuting Chen, Ting Su, and Zhendong Su. 2019. Deep Differential Testing of JVM Implementations. In International Conference on Software Engineering. IEEE, 1257–1268. https://doi.org/10.1109/ICSE.2019.00127
[8]
Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-Directed Differential Testing of JVM Implementations. In Programming Language Design and Implementation. ACM, 85–99. https://doi.org/10.1145/2908080.2908095
[9]
Wai-Mee Ching and Alex Katz. 1993. The Testing of an APL Compiler. In International Conference on APL. ACM, 55–62. https://doi.org/10.1145/166197.166205
[10]
Koen Claessen and John Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In International Conference on Functional Programming. ACM, 268–279. https://doi.org/10.1145/351240.351266
[11]
Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. ACM, 185–194. https://doi.org/10.1145/1287624.1287651
[12]
R.A. DeMillo, R.J. Lipton, and F.G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer, 11, 4 (1978), 34–41. https://doi.org/10.1109/C-M.1978.218136
[13]
Eclipse Foundation, Inc. 2024. The order of super interface initialization in J9 is strange - Issue #13242 - eclipse-openj9/openj9. https://github.com/eclipse-openj9/openj9/issues/13242
[14]
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. ACM, 416–419. https://doi.org/10.1145/2025113.2025179
[15]
Tianchang Gao, Junjie Chen, Yingquan Zhao, Yuqun Zhang, and Lingming Zhang. 2023. Vectorizing Program Ingredients for Better JVM Testing. In International Symposium on Software Testing and Analysis. ACM, 526–537. https://doi.org/10.1145/3597926.3598075
[16]
GitHub, Inc. 2023. GitHub. https://github.com
[17]
Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test Generation through Programming in UDITA. In International Conference on Software Engineering. ACM, 225–234. https://doi.org/10.1145/1806799.1806835
[18]
James Gosling and Greg Bollella. 2000. The Real-Time Specification for Java. Addison-Wesley. isbn:0201703238
[19]
JavaParser.org. 2024. JavaParser - Home. https://javaparser.org
[20]
Haoxiang Jia, Ming Wen, Zifan Xie, Xiaochen Guo, Rongxin Wu, Maolin Sun, Kang Chen, and Hai Jin. 2023. Detecting JVM JIT Compiler Bugs via Exploring Two-Dimensional Input Spaces. In International Conference on Software Engineering. IEEE, 43–55. https://doi.org/10.1109/ICSE48619.2023.00016
[21]
Yue Jia and Mark Harman. 2008. Constructing Subtle Faults Using Higher Order Mutation Testing. In IEEE International Working Conference on Source Code Analysis and Manipulation. IEEE, 249–258. https://doi.org/10.1109/SCAM.2008.36
[22]
JUnit. 2022. JUnit - About. https://junit.org/junit4/
[23]
George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating Fuzz Testing. In Conference on Computer and Communications Security. ACM, 2123–2138. https://doi.org/10.1145/3243734.3243804
[24]
Cong Li, Yanyan Jiang, Chang Xu, and Zhendong Su. 2023. Validating JIT Compilers via Compilation Space Exploration. In Symposium on Operating Systems Principles. ACM, 66–79. https://doi.org/10.1145/3600006.3613140
[25]
Vsevolod Livinskii, Dmitry Babokin, and John Regehr. 2020. Random Testing for C and C++ Compilers with YARPGen. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, 196:1–196:25. https://doi.org/10.1145/3428264
[26]
William M McKeeman. 1998. Differential Testing for Software. Digital Technical Journal, 10, 1 (1998), 100–107. https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
[27]
Kazuhiro Nakamura and Nagisa Ishiura. 2016. Random Testing of C Compilers Based on Test Program Generation by Equivalence Transformation. In Asia Pacific Conference on Circuits and Systems. IEEE, 676–679. https://doi.org/10.1109/APCCAS.2016.7804063
[28]
Oracle Corporation and/or its affiliates. 2021. The Java HotSpot Performance Engine Architecture. https://www.oracle.com/java/technologies/whitepaper.html
[29]
Oracle Corporation and/or its affiliates. 2023. [JDK-8280126] C2: detect and remove dead irreducible loops - Java Bug System. https://bugs.openjdk.java.net/browse/JDK-8280126
[30]
Oracle Corporation and/or its affiliates. 2023. [JDK-8280126] C2: detect and remove dead irreducible loops - Java Bug System. https://bugs.openjdk.org/browse/JDK-8280126?focusedCommentId=14476253&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14476253
[31]
Oracle Corporation and/or its affiliates. 2023. openjdk/jdk: JDK main-line development. https://github.com/openjdk/jdk
[32]
OW2. 2024. ASM. https://asm.ow2.io
[33]
Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In International Conference on Software Engineering. IEEE, 75–84. https://doi.org/10.1109/ICSE.2007.37
[34]
Priya Khaira-Hanks. 2023. What is a Java Uber-JAR and Why Is It Useful? https://blog.payara.fish/what-is-a-java-uber-jar
[35]
Emin Gün Sirer and Brian N. Bershad. 2000. Using Production Grammars in Software Testing. In Conference on Domain-Specific Languages. ACM, 1–13. https://doi.org/10.1145/331960.331965
[36]
The Apache Software Foundation. 2023. commons-codec/BinaryCodec.java. https://github.com/apache/commons-codec/blob/4de60e/src/main/java/org/apache/commons/codec/binary/BinaryCodec.java
[37]
The Apache Software Foundation. 2023. commons-codec/StringUtils.java. https://github.com/apache/commons-codec/blob/4de60e/src/main/java/org/apache/commons/codec/binary/StringUtils.java
[38]
The Apache Software Foundation. 2023. commons-math/AdamsNordsieckTransformer.java. https://github.com/apache/commons-math/blob/dff1a0/src/main/java/org/apache/commons/math4/ode/nonstiff/AdamsNordsieckTransformer.java
[39]
The Apache Software Foundation. 2023. commons-math/Quaternion.java. https://github.com/apache/commons-math/blob/dff1a0/src/main/java/org/apache/commons/math4/complex/Quaternion.java
[40]
The Apache Software Foundation. 2023. commons-text/StrBuilder.java. https://github.com/apache/commons-text/blob/e62203/src/main/java/org/apache/commons/text/StrBuilder.java
[41]
The Apache Software Foundation. 2023. Maven - Welcome to Apache Maven. https://maven.apache.org/
[42]
The JUnit Team. 2023. JUnit 5. https://junit.org/junit5/
[43]
Vasudev Vikram, Rohan Padhye, and Koushik Sen. 2021. Growing A Test Corpus with Bonsai Fuzzing. In International Conference on Software Engineering. ACM, 723–735. https://doi.org/10.1109/ICSE43902.2021.00072
[44]
Mingyuan Wu, Minghai Lu, Heming Cui, Junjie Chen, Yuqun Zhang, and Lingming Zhang. 2023. JITfuzz: Coverage-guided Fuzzing for JVM Just-in-Time Compilers. In International Conference on Software Engineering. IEEE, 56–68. https://doi.org/10.1109/ICSE48619.2023.00017
[45]
Mingyuan Wu, Yicheng Ouyang, Minghai Lu, Junjie Chen, Yingquan Zhao, Heming Cui, Guowei Yang, and Yuqun Zhang. 2023. SJFuzz: Seed & Mutator Scheduling for JVM Fuzzing. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. ACM, 1062–1074. https://doi.org/10.1145/3611643.3616277
[46]
Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Programming Language Design and Implementation. ACM, 283–294. https://doi.org/10.1145/1993316.1993532
[47]
Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2021. GCC Bug List Found by Random Testing (Total 79). https://embed.cs.utah.edu/csmith/gcc-bugs.html
[48]
Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2021. LLVM Bug List Found by Random Testing (Total 203). https://embed.cs.utah.edu/csmith/llvm-bugs.html
[49]
Guixin Ye, Tianmin Hu, Zhanyong Tang, Zhenye Fan, Shin Tan, Hwei, Bo Zhang, Wenxiang Qian, and Wang Zheng. 2023. A Generative and Mutational Approach for Synthesizing Bug-exposing Test Cases to Guide Compiler Fuzzing. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. ACM, 1127–1139. https://doi.org/10.1145/3611643.3616332
[50]
Takahide Yoshikawa, Kouya Shimura, and Toshihiro Ozawa. 2003. Random Program Generator for Java JIT Compiler Test System. In International Conference on Quality Software. IEEE, 20–23. https://doi.org/10.1109/QSIC.2003.1319081
[51]
Zhiqiang Zang, Nathaniel Wiatrek, Milos Gligoric, and August Shi. 2022. Compiler Testing using Template Java Programs. In International Conference on Automated Software Engineering. ACM, 23:1–23:13. https://doi.org/10.1145/3551349.3556958
[52]
Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal Program Enumeration for Rigorous Compiler Testing. In Programming Language Design and Implementation. ACM, 347–361. https://doi.org/10.1145/3140587.3062379
[53]
Yingquan Zhao, Zan Wang, Junjie Chen, Mengdi Liu, Mingyuan Wu, Yuqun Zhang, and Lingming Zhang. 2022. History-Driven Test Program Synthesis for JVM Testing. In International Conference on Software Engineering. ACM, 1133–1144. https://doi.org/10.1145/3510003.3510059
Index Terms
- Java JIT Testing with Template Extraction
Recommendations
Validating JIT Compilers via Compilation Space Exploration
SOSP '23: Proceedings of the 29th Symposium on Operating Systems PrinciplesThis paper introduces the novel concept of compilation space, which facilitates the thorough validation of just-in-time (JIT) compilers in modern language virtual machines (LVMs). The compilation space, even for a single program, consists of an extensive ...
Compiler Testing using Template Java Programs
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software EngineeringWe present JAttack, a framework that enables template-based testing for compilers. Using JAttack, a developer writes a template program that describes a set of programs to be generated and given as test inputs to a compiler. Such a framework enables ...
JAttack: Java JIT Testing Using Template Programs
ICSE '23: Proceedings of the 45th International Conference on Software Engineering: Companion ProceedingsWe present JAttack, a framework that enables compiler testing using templates. JAttack allows compiler developers to write a template program that describes a set of concrete programs to be used to test compilers. Such a template-based approach ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
2770 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 July 2024
Published in PACMSE Volume 1, Issue FSE
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSF (National Science Foundation)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 179Total Downloads
- Downloads (Last 12 months)179
- Downloads (Last 6 weeks)58
Reflects downloads up to 24 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in