survey

A Survey of Compiler Testing

Authors:

Michael Pradel,

Lu ZhangAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 53, Issue 1

Article No.: 4, Pages 1 - 36

https://doi.org/10.1145/3363562

Published: 06 February 2020 Publication History

Abstract

Virtually any software running on a computer has been processed by a compiler or a compiler-like tool. Because compilers are such a crucial piece of infrastructure for building software, their correctness is of paramount importance. To validate and increase the correctness of compilers, significant research efforts have been devoted to testing compilers. This survey article provides a comprehensive summary of the current state-of-the-art of research on compiler testing. The survey covers different aspects of the compiler testing problem, including how to construct test programs, what test oracles to use for determining whether a compiler behaves correctly, how to execute compiler tests efficiently, and how to help compiler developers take action on bugs discovered by compiler testing. Moreover, we survey work that empirically studies the strengths and weaknesses of current compiler testing research and practice. Based on the discussion of existing work, we outline several open challenges that remain to be addressed in future work.

References

[1]

Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. 2009. Spectrum-based multiple fault localization. In Proceedings of the International Conference on Automated Software Engineering. 88--99.

[2]

Rui Abreu, Peter Zoeteweij, and Arjan J. C. Van Gemund. 2007. On the accuracy of spectrum-based fault localization. In Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION. 89--98.

[3]

Mohammad Amin Alipour, Alex Groce, Rahul Gopinath, and Arpit Christi. 2016. Generating focused random tests using directed swarm testing. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 70--81.

Digital Library

[4]

M. Amodio, S. Chaudhuri, and T. Reps. 2017. Neural attribute machines for program generation. ArXiv e-prints (May 2017). arxiv:cs.AI/1705.09231

[5]

Cyrille Artho. 2011. Iterative delta debugging. Int. J. Softw. Tools Technol. Transf. 13, 3 (2011), 223--246.

Digital Library

[6]

S. M. Austin, D. R. Wilkins, and B. A. Wichmann. 1991. An Ada program test generator. In Proceedings of the Conference on TRI-Ada ’91: Today’s Accomplishments; Tomorrow’s Expectations (TRI-Ada’91). 320--325.

[7]

Gergö Barany. 2018. Finding missed compiler optimizations by differential testing. In Proceedings of the 27th International Conference on Compiler Construction. 82--92.

Digital Library

[8]

Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing program input grammars. In Proceedings of the 38th Conference on Programming Language Design and Implementation, Vol. 52. 95--110.

Digital Library

[9]

F. Bazzichi and I. Spadafora. 1982. An automatic generator for compiler testing. IEEE Trans. Softw. Eng. 8, 4 (1982), 343--353.

Digital Library

[10]

Clément Béra, Eliot Miranda, Marcus Denker, and Stéphane Ducasse. 2016. Practical validation of bytecode to bytecode JIT compiler dynamic deoptimization. J. Obj. Technol. 15, 2 (2016), 1:1--26.

[11]

Daniel M. Berry. 1983. A new methodology for generating test cases for a programming language compiler. ACM SIGPLAN Not. 18, 2 (1983), 46--56.

Digital Library

[12]

D. L. Bird and C. U. Munoz. 1983. Automatic generation of random self-checking test cases. IBM Syst. J. 22, 3 (1983), 229--245.

Digital Library

[13]

Abdulazeez S. Boujarwah and Kassem Saleh. 1997. Compiler test case generation methods: A survey and assessment. Inf. Softw. Technol. 39, 9 (1997), 617--625.

[14]

Abdulazeez S. Boujarwah, Kassem Saleh, and Jehad Al-Dallal. 1999. Testing syntax and semantic coverage of Java language compilers. Inf. Softw. Technol. 41, 1 (1999), 15--28.

[15]

Colin J. Burgess. 1994. The automated generation of test cases for compilers. Softw. Test. Verif. Rel. 4, 2 (1994), 81--99.

[16]

Colin J. Burgess and M. Saidi. 1996. The automatic generation of test cases for optimizing Fortran compilers. Inf. Softw. Technol. 38, 2 (1996), 111--119.

[17]

D. Callahan, J. Dongarra, and D. Levine. 1988. Vectorizing compilers: A test suite and results. In Proceedings of the ACM/IEEE Conference on Supercomputing. 98--105.

[18]

Jacqueline M. Caron and Peter A. Darnell. 1990. Bugfind: A tool for debugging optimizing compilers. ACM SIGPLAN Not. 25, 1 (1990), 17--22.

Digital Library

[19]

Heung Seok Chae, Gyun Woo, Tae Yeon Kim, Jung Ho Bae, and Won Young Kim. 2011. An automated approach to reducing test suites for testing retargeted C compilers for embedded systems. J. Syst. Softw. 84, 12 (2011), 2053--2064.

Digital Library

[20]

Bor-Yuh Evan Chang, Adam Chlipala, George C. Necula, and Robert R. Schneck. 2005. Type-based verification of assembly language for compiler debugging. In Proceedings of the ACM SIGPLAN International Workshop on Types in Languages Design and Implementation. 91--102.

[21]

Junjie Chen. 2018. Learning to accelerate compiler testing. In Proceedings of the 40th International Conference on Software Engineering. 472--475.

Digital Library

[22]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Bing Xie. 2017. Learning to prioritize test programs for compiler testing. In Proceedings of the 39th International Conference on Software Engineering. 700--711.

Digital Library

[23]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. Test case prioritization for compilers: A text-vector based approach. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation. 266--277.

[24]

Junjie Chen, Yanwei Bai, Dan Hao, Lingming Zhang, Lu Zhang, and Bing Xie. 2017. How do assertions impact coverage-based test-suite reduction? In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation. 418--423.

[25]

Junjie Chen, Jiaqi Han, Peiyi Sun, Lingming Zhang, Dan Hao, and Lu Zhang. 2019. Compiler bug isolation via effective witness test program generation. In Proceedings of the 27th Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223--234.

Digital Library

[26]

Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An empirical comparison of compiler testing techniques. In Proceedings of the 38th International Conference on Software Engineering. 180--190.

Digital Library

[27]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Lu Zhang. 2019. History-guided configuration diversification for compiler test-program generation. In Proceedings of the 34th International Conference on Automated Software Engineering. to appear.

Digital Library

[28]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2018. Coverage prediction for accelerating compiler testing. Trans. Softw. Eng. (2018). to appear.

[29]

Tsong Y. Chen, Shing C. Cheung, and Shiu Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong.

[30]

Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation. 197--208.

Digital Library

[31]

Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-directed differential testing of JVM implementations. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 85--99.

Digital Library

[32]

Wai-Mee Ching and Alex Katz. 1993. The testing of an APL compiler. In Proceedings of the International Conference on APL. 55--62.

Digital Library

[33]

Arpit Christi, Matthew Lyle Olson, Mohammad Amin Alipour, and Alex Groce. 2018. Reduce before you localize: Delta-debugging and spectrum-based fault localization. In Proceedings of the International Symposium on Software Reliability Engineering Workshops. 184--191.

[34]

C. J. Burgess. 1986. Towards the automatic generation of executable programs to test a Pascal compiler. (1986).

[35]

Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated testing of refactoring engines. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. 185--194.

Digital Library

[36]

Maulik A. Dave. 2003. Compiler verification: A bibliography. ACM SIGSOFT Softw. Eng. Notes 28, 6 (2003), 2--2.

Digital Library

[37]

Chromium developers. 2019. Chromium Testsuite. Retrieved from: https://www.chromium.org/chromium-os/testing/test-suites.

[38]

GCC developers. 2019. GCC Testsuites. Retrieved from: https://gcc.gnu.org/onlinedocs/gccint/Testsuites.html#Testsuites.

[39]

LLVM developers. 2019. LLVM Testing Infrastructure Guide. Retrieved from: https://llvm.org/docs/TestingGuide.html.

[40]

OpenJDK developers. 2019. OpenJDK Testsuite. Retrieved from: http://openjdk.java.net/jtreg.

[41]

Kyle Dewey, Jared Roesch, and Ben Hardekopf. 2015. Fuzzing the Rust typechecker using CLP (T). In Proceedings of the 30th International Conference on Automated Software Engineering. 482--493.

Digital Library

[42]

Alastair F. Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated testing of graphics shader compilers. In Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 93:1--93:29.

Digital Library

[43]

Alastair F. Donaldson and Andrei Lascu. 2016. Metamorphic testing for (graphics) compilers. In Proceedings of the 1st International Workshop on Metamorphic Testing. 44--47.

[44]

Jack Dongarra, Mark Furtney, Steve Reinhardt, and Jerry Russell. 1991. Parallel loops—A test suite for parallelizing compilers: Description and example results. Parallel Comput. 17, 10 (1991), 1247--1255.

Digital Library

[45]

A. G. Duncan and J. S. Hutchison. 1981. Using attributed grammars to test designs and implementations. In Proceedings of the 5th International Conference on Software Engineering. 170--178.

[46]

ECMAScript. 2019. Test262: ECMAScript Test Suite. Retrieved from: https://github.com/tc39/test262.

[47]

Eric Eide and John Regehr. 2008. Volatiles are miscompiled, and what to do about it. In Proceedings of the 8th ACM International Conference on Embedded Software (EMSOFT’08). 255--264.

Digital Library

[48]

Pierre-Loïc Garoche, Falk Howar, Temesghen Kahsai, and Xavier Thirioux. 2014. Testing-based compiler validation for synchronous languages. In Proceedings of the NASA Formal Methods Symposium. 246--251.

Digital Library

[49]

Alex Groce, Mohammad Amin Alipour, Chaoqiang Zhang, Yang Chen, and John Regehr. 2016. Cause reduction: Delta debugging, even without bugs. Softw. Test. Verif. Rel. 26, 1 (2016), 40--68.

Digital Library

[50]

Alex Groce, Chaoqiang Zhang, Mohammad Amin Alipour, Eric Eide, Chen Yang, and John Regehr. 2013. Help, help, I’m being suppressed! The significance of suppressors in software testing. In Proceedings of the International Symposium on Software Reliability Engineering. 390--399.

[51]

Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm testing. In Proceedings of the International Symposium on Software Testing and Analysis. 78--88.

Digital Library

[52]

Kenneth V. Hanford. 1970. Automatic generation of test cases. IBM Syst. J. 9, 4 (1970), 242--257.

Digital Library

[53]

Atsushi Hashimoto and Nagisa Ishiura. 2016. Detecting arithmetic optimization opportunities for C compilers by randomly generated equivalent programs. IPSJ Trans. Syst. LSI Des. Methodol. 9 (2016), 21--29.

[54]

Chris Hawblitzel, Shuvendu K. Lahiri, Kshama Pawar, Hammad Hashmi, Sedar Gokbulut, Lakshan Fernando, Dave Detlefs, and Scott Wadsworth. 2013. Will you still compile me tomorrow? Static cross-version compiler validation. In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering. 191--201.

Digital Library

[55]

K. Scott Hemmert, Justin L. Tripp, Brad L. Hutchings, and Preston A. Jackson. 2003. Source level debugger for the sea cucumber synthesizing compiler. In Proceedings of the 11th IEEE Symposium on Field-programmable Custom Computing Machines. 228--237.

[56]

Satia Herfert, Jibesh Patra, and Michael Pradel. 2017. Automatically reducing tree-structured test inputs. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. 861--871.

Digital Library

[57]

Renáta Hodován and Ákos Kiss. 2016. Modernizing hierarchical delta debugging. In Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST’16). 31--37.

[58]

Christian Holler, Kim Herzig, and Andreas Zeller. 2012. Fuzzing with code fragments. In Proceedings of the 21st USENIX Conference on Security Symposium. 445--458.

Digital Library

[59]

Josie Holmes and Alex Groce. 2018. Causal distance-metric-based assistance for debugging after compiler fuzzing. In Proceedings of the 29th International Symposium on Software Reliability Engineering. 166--177.

[60]

Josie Holmes, Alex Groce, and Mohammad Amin Alipour. 2016. Mitigating (and exploiting) test reduction slippage. In Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation. 66--69.

Digital Library

[61]

Bernard Houssais. 1977. Verification of an Algol 68 implementation. In Proceedings of the Strathclyde ALGOL 68 Conference. 117--128.

Digital Library

[62]

Plum Hall Inc. 2019. The Plum Hall Validation Suite. Retrieved from: http://www.plumhall.com/stec1.html.

[63]

Bo Jiang and W. K. Chan. 2015. Input-based adaptive randomized test case prioritization: A local beam search approach. J. Syst. Softw. 105, C (2015), 91--106.

[64]

Alexey Kalinov, Alexander Kossatchev, Alexandre Petrenko, Mikhail Posypkin, and Vladimir Shishkov. 2003. Coverage-driven automated compiler test suite generation. Electron. Notes Theoret. Comput. Sci. 82, 3 (2003), 500--514.

[65]

Alexey Kalinov, Alexandre Kossatchev, Alexander Petrenko, Mikhail Posypkin, and Vladimir Shishkov. 2003. Using ASM specifications for compiler testing. In Proceedings of the Abstract State Machines 10th International Conference on Advances in Theory and Practice. 415--415.

Digital Library

[66]

Kota Kitaura and Nagisa Ishiura. 2018. Random testing of compilers’ performance based on mixed static and dynamic code comparison. In Proceedings of the 9th International Workshop on Automating TEST Case Design, Selection, and Evaluation. 38--44.

Digital Library

[67]

Donald E. Knuth. 1968. Semantics of context-free languages. Math. Syst. Theor. 2, 2 (1968), 127--145.

[68]

Alexander S. Kossatchev and M. A. Posypkin. 2005. Survey of compiler testing methods. Prog. Comput. Softw. 31, 1 (2005), 10--19.

Digital Library

[69]

Cornelis H. A. Koster. 1991. Affix grammars for programming languages. In Proceedings of the International Summer School on Attribute Grammars, Applications and Systems. 358--373.

[70]

Philipp W. Kutter and Alfonso Pierantonio. 1997. Montages specifications of realistic programming languages. J. Univ. Comput. Sci. 3, 5 (1997), 416--442.

[71]

Ralf Lämmel. 2001. Grammar testing. In Proceedings of the 4th International Conference on Fundamental Approaches to Software Engineering. 201--216.

Digital Library

[72]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation. 216--226.

Digital Library

[73]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the International Conference on Object-Oriented Programming, Systems, Languages, and Applications. 386--399.

Digital Library

[74]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Randomized stress-testing of link-time optimizers. In Proceedings of the International Symposium on Software Testing and Analysis. 327--337.

Digital Library

[75]

Daniel Lehmann and Michael Pradel. 2018. Feedback-directed differential testing of interactive debuggers. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 610--620.

Digital Library

[76]

Xavier Leroy. 2009. Formal verification of a realistic compiler. Commun. ACM 52, 7 (2009), 107--115.

Digital Library

[77]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th Conference on Programming Language Design and Implementation. 65--76.

[78]

Christian Lindig. 2005. Find a compiler bug in 5 minutes. In Proceedings of the ACM International Symposium on Automated Analysis-Driven Debugging. 3--12.

[79]

Christian Lindig. 2005. Random testing of C calling conventions. In Proceedings of the 6th International Symposium on Automated Analysis-driven Debugging (AADEBUG’05). 3--12.

Digital Library

[80]

Robert Mandl. 1985. Orthogonal Latin squares: An application of experiment design to compiler testing. Commun. ACM 28, 10 (1985), 1054--1058.

Digital Library

[81]

William M McKeeman. 1998. Differential testing for software. Dig. Tech. J. 10, 1 (1998), 100--107.

[82]

L. G. L. T. Meertens and C. H. A. Koster. 1962. Basic English, a generative grammar for a part of English. In Euratom Seminar\Machine en Talen, Amsterdam.

[83]

Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, and Hanne Riis Nielson. 2017. Effect-driven QuickChecking of compilers. Proc. ACM Prog. Lang. 1 (2017), 15:1--15:23.

Digital Library

[84]

Ghassan Misherghi and Zhendong Su. 2006. HDD: Hierarchical delta debugging. In Proceedings of the 28th International Conference on Software Engineering. 142--151.

Digital Library

[85]

Robin Morisset, Pankaj Pawan, and Francesco Zappa Nardelli. 2013. Compiler testing via a theory of sound optimisations in the C11/C++11 memory model. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation. 187--196.

Digital Library

[86]

Tipp Moseley, Dirk Grunwald, and Ramesh Peri. 2009. OptiScope: Performance accountability for optimizing compilers. In Proceedings of the 7th International Symposium on Code Generation and Optimization. 254--264.

Digital Library

[87]

Eriko Nagai, Hironobu Awazu, Nagisa Ishiura, and Naoya Takeda. 2012. Random testing of C compilers targeting arithmetic optimization. In Proceedings of the Workshop on Synthesis And System Integration of Mixed Information Technologies. 48--53.

[88]

Eriko Nagai, Atsushi Hashimoto, and Nagisa Ishiura. 2014. Reinforcing random testing of arithmetic optimization of C compilers by scaling up size and number of expressions. IPSJ Trans. Syst. LSI Des. Methodol. 7 (2014), 91--100.

[89]

Kazuhiro Nakamura and Nagisa Ishiura. 2016. Random testing of C compilers based on test program generation by equivalence transformation. In Proceedings of the IEEE Asia Pacific Conference on Circuits and Systems (APCCAS’16). 676--679.

[90]

Georg Ofenbeck, Tiark Rompf, and Markus Püschel. 2016. RandIR: Differential testing for embedded compilers. In Proceedings of the 7th ACM SIGPLAN Symposium on Scala. 21--30.

Digital Library

[91]

Kazunori Ogata, Tamiya Onodera, Kiyokuni Kawachiya, Hideaki Komatsu, and Toshio Nakatani. 2006. Replay compilation: Improving debuggability of a just-in-time compiler. ACM SIGPLAN Not. 41, 10 (2006), 241--252.

Digital Library

[92]

Michał H. Pałka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an optimising compiler by generating random lambda terms. In Proceedings of the 6th International Workshop on Automation of Software Test. 91--97.

Digital Library

[93]

Jibesh Patra and Michael Pradel. 2016. Learning to Fuzz: Application-Independent Fuzz Testing with Probabilistic, Generative Models of Input Data. Technical Report. TU Darmstadt, Department of Computer Science.

[94]

Moritz Pflanzer, Alastair F. Donaldson, and Andrei Lascu. 2016. Automatic test case reduction for OpenCL. In Proceedings of the 4th International Workshop on OpenCL. 1:1--1:12.

Digital Library

[95]

James Price and Simon McIntosh-Smith. 2015. Oclgrind: An extensible OpenCL device simulator. In Proceedings of the 3rd International Workshop on OpenCL. 1--7.

Digital Library

[96]

Paul Purdom. 1972. A sentence generator for testing parsers. BIT Numer. Math. 12, 3 (1972), 366--375.

[97]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proceedings of the 33rd Conference on Programming Language Design and Implementation, Vol. 47. 335--346.

Digital Library

[98]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. 1999. Test case prioritization: An empirical study. In Proceedings of the International Conference on Software Maintenance. 179--188.

[99]

Hanan Samet. 1976. Compiler testing via symbolic interpretation. In Proceedings of the 1976 Annual Conference (ACM’76). 492--497.

Digital Library

[100]

H. Samet. 1977. A machine description facility for compiler testing. IEEE Trans. Softw. Eng. 3, 5 (1977), 343--351.

Digital Library

[101]

Hanan Samet. 1977. A normal form for compiler testing. In ACM SIGART Bull. Vol. 12. 155--162.

[102]

Hanan Samet. 1977. Toward automatic debugging of compilers. In Proceedings of the 5th International Joint Conference on Artificial Intelligence—Volume 1. Morgan Kaufmann Publishers Inc., 379--379.

[103]

Sriram Sankar. 1989. A note on the detection of an Ada compiler bug while debugging an Anna program. ACM SIGPLAN Not. 24, 6 (1989), 23--31.

Digital Library

[104]

Masataka Sassa and Daijiro Sudosa. 2006. Experience in testing compiler optimizers using comparison checking. In Software Engineering Research and Practice. CSREA Press, 837--843.

[105]

Bauer Scotty, Pascal Cuoq, and John Regehr. 2015. Deniable backdoors using compiler bugs. Int. J. PoC||GTFO 0x08 (2015), 7--9.

[106]

Julian Seward and Nicholas Nethercote. 2005. Using Valgrind to detect undefined value errors with bit-precision. In Proceedings of the USENIX Annual Technical Conference, General Track. 17--30.

Digital Library

[107]

Flash Sheridan. 2007. Practical testing of a C99 compiler using output comparison. Softw.: Pract. Exper. 37, 14 (2007), 1475--1488.

Digital Library

[108]

Emin Gün Sirer and Brian N. Bershad. 1999. Using production grammars in software testing. In Proceedings of the 2nd Conference on Domain-specific Languages. 1--13.

[109]

Evgeniy Stepanov and Konstantin Serebryany. 2015. MemorySanitizer: Fast detector of uninitialized memory use in C++. In Proceedings of the 13th International Symposium on Code Generation and Optimization. 46--55.

[110]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and analyzing compiler warning defects. In Proceedings of the IEEE/ACM 38th International Conference on Software Engineering. 203--213.

Digital Library

[111]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation. In Proceedings of the International Conference on Object-oriented Programming, Systems, Languages, and Applications. 849--863.

Digital Library

[112]

Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su. 2016. Toward understanding compiler bugs in GCC and LLVM. In Proceedings of the 25th ACM SIGSOFT International Symposium on Software Testing and Analysis. 294--305.

Digital Library

[113]

Chengnian Sun, Yuanbo Li, Qirun Zhang, Tianxiao Gu, and Zhendong Su. 2018. Perses: Syntax-guided program reduction. In Proceedings of the 40th International Conference on Software Engineering. 361--371.

Digital Library

[114]

Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An automatic testing approach for compiler based on metamorphic testing technique. In Proceedings of the Asia Pacific Software Engineering Conference. 270--279.

Digital Library

[115]

Stephen W. Thomas, Hadi Hemmati, Ahmed E. Hassan, and Dorothea Blostein. 2014. Static test case prioritization using topic models. Emp. Softw. Eng. 19, 1 (2014), 182--212.

Digital Library

[116]

Sandro Tolksdorf, Daniel Lehmann, and Michael Pradel. 2019. Interactive metamorphic testing of debuggers. In Proceedings of the 28th International Symposium on Software Testing and Analysis. 273--283.

Digital Library

[117]

Michael Tonndorf. 1998. Ten years of tool-based Ada compiler validations an experience report. In Proceedings of the International Conference on Reliable Software Technologies. 176--187.

[118]

A. van Wijngaarden. 1965. Orthogonal Design and Description of a Formal Language. Stichting Mathematisch Centrum.

[119]

David B. Whalley. 1994. Automatic isolation of compiler errors. ACM Trans. Prog. Lang. Syst. 16, 5 (1994), 1648--1659.

Digital Library

[120]

Klaus-Hendrik Wolf and Mike Klimek. 2016. A conformance test suite for Arden syntax compilers and interpreters. Stud. Health Technol. Inf. 228 (2016), 379--383.

[121]

W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. Trans. Softw. Eng. 42, 8 (2016), 707--740.

Digital Library

[122]

Gyun Woo, Heung Seok Chae, and Hanil Jang. 2007. An intermediate representation approach to reducing test suites for retargeted compilers. In Proceedings of the Ada-Europe International Conference on Reliable Software Technologies. 100--113.

[123]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation. 283--294.

Digital Library

[124]

Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection, and prioritization: A survey. Softw. Test. Verif. Rel. 22, 2 (2012), 67--120.

Digital Library

[125]

Takahide Yoshikawa, Kouya Shimura, and Toshihiro Ozawa. 2003. Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software. 20--23.

[126]

Sergey Zelenov and Sophia Zelenova. 2007. Model-based testing of optimizing compilers. In Proceedings of the International Workshop on Formal Approaches to Software Testing and International Conference on Testing of Communicating Systems. Lecture Notes in Computer Science 4581 (2007), 365--377.

[127]

S. V. Zelenov, S. A. Zelenova, A. S. Kossatchev, and A. K. Petrenko. 2003. Test generation for compilers and other formal text processors. Prog. Comput. Softw. 29, 2 (2003), 104--111.

Digital Library

[128]

Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? In Proceedings of the 7th European Software Engineering Conference Held Jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-7). 253--267.

Digital Library

[129]

Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Trans. Softw. Eng. 28, 2 (2002), 183--200.

Digital Library

[130]

Lingming Zhang, Dan Hao, Lu Zhang, Gregg Rothermel, and Hong Mei. 2013. Bridging the gap between the total and additional test-case prioritization strategies. In Proceedings of the International Conference on Software Engineering. 192--201.

[131]

Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal program enumeration for rigorous compiler testing. In Proceedings of the 38th Conference on Programming Language Design and Implementation. 347--361.

Digital Library

[132]

Chen Zhao, Yunzhi Xue, Qiuming Tao, Liang Guo, and Zhaohui Wang. 2009. Automated test program generation for an industrial optimizing compiler. In Proceedings of the ICSE Workshop on Automation of Software Test. 36--43.

Cited By

R. BMohan US. V(2024)Machine Learning Applications in Behavioural Authentication Data Using Gait AnalysisMachine Learning and Cryptographic Solutions for Data Protection and Network Security10.4018/979-8-3693-4159-9.ch013(206-218)Online publication date: 22-Mar-2024
https://doi.org/10.4018/979-8-3693-4159-9.ch013
Gao FChen HZhou YWang KFilkov VRay BZhou M(2024)Shoot Yourself in the Foot — Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695548(1846-1857)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695548
Chen HFilkov VRay BZhou M(2024)Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695369(2426-2428)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695369
Show More Cited By

Index Terms

A Survey of Compiler Testing
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers

Recommendations

Boosting Compiler Testing via Compiler Optimization Exploration
Compilers are a kind of important software, and similar to the quality assurance of other software, compiler testing is one of the most widely-used ways of guaranteeing their quality. Compiler bugs tend to occur in compiler optimizations. Detecting ...
Detecting Optimizing Compiler Bugs via History-Driven Test Program Mutation
Internetware '24: Proceedings of the 15th Asia-Pacific Symposium on Internetware

Compiler testing is an important task for assuring the quality of compilers. However, most mutation-based compiler testing approaches still suffer from the effectiveness issue due to the ineffective mutation strategies. In this paper, we propose CTHist, ...
Compiler bug isolation via effective witness test program generation
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Compiler bugs are extremely harmful, but are notoriously difficult to debug because compiler bugs usually produce few debugging information. Given a bug-triggering test program for a compiler, hundreds of compiler files are usually involved during ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 53, Issue 1

January 2021

781 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3382040

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 February 2020

Accepted: 01 September 2019

Revised: 01 May 2019

Received: 01 August 2018

Published in CSUR Volume 53, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Survey
Refereed

Funding Sources

German Research Foundation within the ConcSys and Perf4JS projects
BMWF/Hessen within CRISP
National Natural Science Foundation of China
National Key Research and Development Program of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

108
Total Citations
View Citations
3,453
Total Downloads

Downloads (Last 12 months)791
Downloads (Last 6 weeks)141

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

R. BMohan US. V(2024)Machine Learning Applications in Behavioural Authentication Data Using Gait AnalysisMachine Learning and Cryptographic Solutions for Data Protection and Network Security10.4018/979-8-3693-4159-9.ch013(206-218)Online publication date: 22-Mar-2024
https://doi.org/10.4018/979-8-3693-4159-9.ch013
Gao FChen HZhou YWang KFilkov VRay BZhou M(2024)Shoot Yourself in the Foot — Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695548(1846-1857)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695548
Chen HFilkov VRay BZhou M(2024)Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695369(2426-2428)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695369
Liu YZhu MDong JYu JHao DFilkov VRay BZhou M(2024)Compiler Bug Isolation via Enhanced Test Program MutationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695074(819-830)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695074
Yang WGao CLiu XLi YXue YFilkov VRay BZhou M(2024)Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695059(631-642)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695059
Kaindlstorfer DIsychev AWüstholz VChristakis MFilkov VRay BZhou M(2024)Interrogation Testing of Program Analyzers for Soundness and Precision IssuesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695034(319-330)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695034
Wang QJung R(2024)Rustlantis: Randomized Differential Testing of the Rust CompilerProceedings of the ACM on Programming Languages10.1145/36897808:OOPSLA2(1955-1981)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689780
Yang CDeng YLu RYao JLiu JJabbarvand RZhang L(2024)WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language ModelsProceedings of the ACM on Programming Languages10.1145/36897368:OOPSLA2(709-735)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689736
Thangamani ALoechner VGenaud S(2024)A Survey of General-purpose Polyhedral CompilersACM Transactions on Architecture and Code Optimization10.1145/367473521:4(1-26)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3674735
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents