research-article

Mutation-based Compliance Testing for RISC-V

Authors:

Vladimir Herdt,

Rolf DrechslerAuthors Info & Claims

ASPDAC '21: Proceedings of the 26th Asia and South Pacific Design Automation Conference

Pages 55 - 60

https://doi.org/10.1145/3394885.3431584

Published: 29 January 2021 Publication History

Abstract

Compliance testing for RISC-V is very important. Essentially, it ensures that compatibility is maintained between RISC-V implementations and the ever growing RISC-V ecosystem. Therefore, an official Compliance Test-suite (CT) is being actively developed. However, it is very difficult to achieve that all relevant functional behavior is comprehensively tested.

In this paper, we propose a mutation-based approach to boost RISC-V compliance testing by providing more comprehensive testing results. Therefore, we define mutation classes tailored for RISC-V to assess the quality of the CT and provide a symbolic execution framework to generate new test-cases that kill the undetected mutants. Our experimental results demonstrate the effectiveness of our approach. We identified several serious gaps in the CT and generated new tests to close these gaps.

References

[1]

2017. RISC-V Torture Test Generator. https://github.com/ucb-bar/riscv-torture.

[2]

2019. The Challenge Of RISC-V Compliance. https://semiengineering.com/toward-risc-v-compliance/.

[3]

2019. Imperas delivers highest quality RISC-V RV32I compliance test suites to implementers and adopters of RISC-V. https://riscv.org/2019/11/imperas-delivers-highest-quality-risc-v-rv32i-compliance-test-suites-to-implementers-and-adopters-of-risc-v/.

[4]

2020. angr. https://angr.io/.

[5]

2020. GRIFT-Galois RISC-V ISA Formal Tools. https://github.com/GaloisInc/grift.

[6]

2020. OneSpin 360 DV RISC-V Verification App. https://www.onespin.com/solutions/risc-v.

[7]

2020. RISC-V Compliance Task Group. https://github.com/riscv/riscv-compliance.

[8]

2020. RISC-V Formal Verification Framework. https://github.com/SymbioticEDA/riscv-formal.

[9]

2020. RISC-V Virtual Prototype. https://github.com/agra-uni-bremen/riscv-vp.

[10]

2020. RISCV-DV. https://github.com/google/riscv-dv.

[11]

2020. RISCV Sail Model. https://github.com/rems-project/sail-riscv.

[12]

2020. Spike RISC-VISA Simulator. https://github.com/riscv/riscv-isa-sim.

[13]

A. Adir, E. Almog, L. Fournier, E. Marcus, M. Rimon, M. Vinov, and A. Ziv. 2004. Genesys-Pro: innovations in test program generation for functional processor verification. D&T (2004), 84--93.

[14]

Brian Campbell and Ian Stark. 2014. Randomised Testing of a Microprocessor Model Using SMT-Solver State Generation. In Formal Methods for Industrial Critical Systems, Frédéric Lang and Francesco Flammini (Eds.). 185--199.

[15]

Mikhail Chupilko, Alexander Kamkin, Artem Kotsynyak, and Andrei Tatarnikov. 2017. MicroTESK: Specification-Based Tool for Constructing Test Program Generators. In HVC.

[16]

S. Fine and A. Ziv. 2003. Coverage directed test generation for functional verification using Bayesian networks. In DAC. 286--291.

[17]

Mark Hampton and Stephane Petithomme. 2007. Leveraging a Commercial Mutation Analysis Tool For Research. In MUTATION. 203--209.

[18]

Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Closing the RISC-V Compliance Gap: Looking from the Negative Testing Side. In DAC.

[19]

Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Enhanced Virtual Prototyping: Featuring RISC-V Case Studies. Springer.

[20]

Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Towards Specification and Testing of RISC-V ISA Compliance. In DATE. 995--998.

[21]

Vladimir Herdt, Daniel Große, Hoang M. Le, and Rolf Drechsler. 2018. Extensible and Configurable RISC-V based Virtual Prototype. In FDL. 5--16.

[22]

Vladimir Herdt, Daniel Große, Hoang M. Le, and Rolf Drechsler. 2019. Verifying Instruction Set Simulators using Coverage-guided Fuzzing. In DATE. 360--365.

[23]

Vladimir Herdt, Daniel Große, Pascal Pieper, and Rolf Drechsler. 2020. RISC-V based Virtual Prototype: An Extensible and Configurable Platform for the System-level. JSA (2020).

[24]

Vladimir Herdt, Hoang M. Le, Daniel Große, and Rolf Drechsler. 2019. Verifying SystemC using Intermediate Verification Language and Stateful Symbolic Simulation. TCAD 38, 7 (2019), 1359--1372.

[25]

Charalambos Ioannides, Geoff Barrett, and Kerstin Eder. 2011. Feedback-Based Coverage Directed Test Generation: An Industrial Evaluation. In Hardware and Software: Verification and Testing, Sharon Barner, Ian Harris, Daniel Kroening, and Orna Raz (Eds.).

[26]

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (Sept. 2011), 649--678.

Digital Library

[27]

Y. Katz, M. Rimon, and A. Ziv.2012. Generating instruction streams using abstract CSP. In DATE. 15--20.

[28]

Lingyi Liu and Shobha Vasudevan. 2011. Efficient validation input generation in RTL by hybridized source code analysis. In DATE. 1596--1601.

[29]

Lorenzo Martignoni, Roberto Paleari, Giampaolo Fresi Roglia, and Danilo Bruschi. 2009. Testing CPU Emulators. In ISSTA. 261--272.

[30]

Youssef Serrestou, Vincent Beroulle, and Chantal Robach. 2007. Functional Verification of RTL Designs Driven by Mutation Testing Metrics. In DSD. 222--227.

[31]

Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Audrey Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, and Giovanni Vigna. 2016. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. (2016).

[32]

Andrew Waterman and Krste Asanović. 2019. The RISC-V Instruction Set Manual; Volume I: Unprivileged ISA. SiFive Inc. and CS Division, EECS Department, University of California, Berkeley.

[33]

Andrew Waterman and Krste Asanović. 2019. The RISC-V Instruction Set Manual; Volume II: Privileged Architecture. SiFive Inc. and CS Division, EECS Department, University of California, Berkeley.

[34]

Tao Xie, Wolfgang Mueller, and Florian Letombe. 2012. Mutation-analysis driven functional verification of a soft microprocessor. In SoC. 283--288.

Cited By

Funck MAhmadi-Pour SHerdt VDrechsler R(2023)Identification of ISA-Level Mutation-Classes for Qualification of RISC-V Formal Verification2023 Forum on Specification & Design Languages (FDL)10.1109/FDL59689.2023.10272202(1-8)Online publication date: 13-Sep-2023
https://doi.org/10.1109/FDL59689.2023.10272202
Herdt VDrechsler R(2021)Advanced virtual prototyping for cyber-physical systems using RISC-V: implementation, verification and challengesScience China Information Sciences10.1007/s11432-020-3308-465:1Online publication date: 23-Dec-2021
https://doi.org/10.1007/s11432-020-3308-4

Mutation-based Compliance Testing for RISC-V
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Symbolic Input-Output Conformance Checking for Model-Based Mutation Testing

This paper presents an approach to use symbolic input output conformance checking for mutation-based test case generation. In this approach, a possibly non-deterministic action system model is used as basis for generating a number of mutants. ...
Formal mutation testing for Circus

Context: The demand from industry for more dependable and scalable test-development mechanisms has fostered the use of formal models to guide the generation of tests. Despite many advancements having been obtained with state-based models, such as Finite ...
Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Toward this direction, various automated test data generation tools have been developed. The majority of existing tools aim at ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASPDAC '21: Proceedings of the 26th Asia and South Pacific Design Automation Conference

January 2021

930 pages

ISBN:9781450379991

DOI:10.1145/3394885

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGDA: ACM Special Interest Group on Design Automation
IEEE CAS
IEEE CEDA

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

ASPDAC '21

Sponsor:

SIGDA

ASPDAC '21: 26th Asia and South Pacific Design Automation Conference

January 18 - 21, 2021

Tokyo, Japan

Acceptance Rates

ASPDAC '21 Paper Acceptance Rate 111 of 368 submissions, 30%;

Overall Acceptance Rate 466 of 1,454 submissions, 32%

Upcoming Conference

ASPDAC '25

Sponsor:
sigda

30th Asia and South Pacific Design Automation Conference

January 20 - 23, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
139
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Funck MAhmadi-Pour SHerdt VDrechsler R(2023)Identification of ISA-Level Mutation-Classes for Qualification of RISC-V Formal Verification2023 Forum on Specification & Design Languages (FDL)10.1109/FDL59689.2023.10272202(1-8)Online publication date: 13-Sep-2023
https://doi.org/10.1109/FDL59689.2023.10272202
Herdt VDrechsler R(2021)Advanced virtual prototyping for cyber-physical systems using RISC-V: implementation, verification and challengesScience China Information Sciences10.1007/s11432-020-3308-465:1Online publication date: 23-Dec-2021
https://doi.org/10.1007/s11432-020-3308-4

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents