Coverage-directed Differential Testing of X.509 Certificate Validation in SSL/TLS Implementations

Secure Sockets Layer (SSL) and Transport Security (TLS) are two secure protocols for creating secure connections over the Internet. X.509 certificate validation is important for security and needs to be performed before an SSL/TLS connection is established. Some advanced testing techniques, such as frankencert, have revealed, through randomly mutating Internet accessible certificates, that there exist unexpected, sometimes critical, validation differences among different SSL/TLS implementations. Despite these efforts, X.509 certificate validation still needs to be thoroughly tested as this work shows.

This article tackles this challenge by proposing transcert, a coverage-directed technique to much more effectively test real-world certificate validation code. Our core insight is to (1) leverage easily accessible Internet certificates as seed certificates and (2) use code coverage to direct certificate mutation toward generating a set of diverse certificates. The generated certificates are then used to reveal discrepancies, thus potential flaws, among different certificate validation implementations.

We implement transcert and evaluate it against frankencert, NEZHA, and RFCcert (three advanced fuzzing techniques) on five widely used SSL/TLS implementations. The evaluation results clearly show the strengths of transcert: During 10,000 iterations, transcert reveals 71 unique validation differences, 12×, 1.4×, and 7× as many as those revealed by frankencert, NEZHA, and RFCcert, respectively; it also supplements RFCcert in conformance testing of the SSL/TLS implementations against 120 validation rules, 85 of which are exclusively covered by transcert-generated certificates. We identify 17 root causes of validation differences, all of which have been confirmed and 11 have never been reported previously. The transcert-generated X.509 certificates also reveal that the primary goal of certificate chain validation is stated ambiguously in the widely adopted public key infrastructure standard RFC 5280.