Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros Generation

Rust is a relatively new programming language known for its memory safety and numerous advanced features. It has been widely used in system software in recent years. Thus, ensuring the reliability and robustness of the only implementation of the Rust compiler, rustc, is critical. However, compiler testing, as one of the most effective techniques to detect bugs, faces difficulties in generating valid Rust programs with sufficient diversity due to its stringent memory safety mechanisms. Furthermore, existing research primarily focuses on testing rustc to trigger crash errors, neglecting incorrect compilation results - miscompilation. Detecting miscompilation remains a challenge in the absence of multiple implementations of the Rust compiler to serve as a test oracle.

This paper tackles these challenges by introducing rust-twins, a novel and effective approach to performing automated differential testing for rustc to detect both crashes and miscompilations. We devise four Rust-specific mutators and adapt fourteen general mutators for Rust, each intends to produce a syntax and semantic valid Rust program to trigger rustc crashes. Additionally, we develop a macroize approach to rewrite a regular Rust program into dual macros with equivalent behaviors but in different implementations. Furthermore, we design an assessment component to check the equivalence by comparing the expansion results with a simple macro input. Finally, rust-twins attempts to expand the two macros with numerous complex inputs to detect differences. Due to the macro expansion mechanism, the root causes of differences can arise not only from the macro expansion part but also from any other mis-implemented compiler code.

We have evaluated rust-twins on the latest version of rustc. Our experimental results indicate that rust-twins achieves twice the total line coverage and identifies more crashes and differences than the best baseline technique, rustsmith, after 24 hours of testing. In total, rust-twins triggered 10 rustc crashes, and 229 of the generated macros exposed rustc differences. We analyzed and reported 12 previously unknown bugs, of which 8 have already been confirmed and fixed.