research-article

QuCAT: A Combinatorial Testing Tool for Quantum Software

Authors:

Shaukat AliAuthors Info & Claims

ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

Pages 2066 - 2069

https://doi.org/10.1109/ASE56229.2023.00062

Published: 26 September 2024 Publication History

Abstract

With the increased developments in quantum computing, the availability of systematic and automatic testing approaches for quantum programs is becoming increasingly essential. To this end, we present the quantum software testing tool QuCAT for combinatorial testing of quantum programs. QuCAT provides two functionalities of use. With the first functionality, the tool generates a test suite of a given strength (e.g., pair-wise). With the second functionality, it generates test suites with increasing strength until a failure is triggered or a maximum strength is reached. QuCAT uses two test oracles to check the correctness of test outputs. We assess the cost and effectiveness of QuCAT with 3 faulty versions of 5 quantum programs. Results show that combinatorial test suites with a low strength can find faults with limited cost, while a higher strength performs better to trigger some difficult faults with relatively higher cost. Repository: https://github.com/Simula-COMPLEX/qucat-tool Video: https://youtu.be/UsqgOudKLio

References

[1]

J. Zhao, "Quantum software engineering: Landscapes and horizons," CoRR, vol. abs/2007.07047, 2020.

[2]

M. A. Serrano, R. Pérez-Castillo, and M. Piattini, Eds., Quantum Software Engineering. Springer International Publishing, 2022.

[3]

A. Miranskyy and L. Zhang, "On testing quantum programs," in Proceedings of the 41st International Conference on Software Engineering: New Ideas and Emerging Results, ser. ICSE-NIER '19. IEEE Press, 2019, pp. 57--60.

[4]

A. Miranskyy, L. Zhang, and J. Doliskani, "Is your quantum program bug-free?" in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, ser. ICSE-NIER '20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 29--32.

Digital Library

[5]

S. Honarvar, M. R. Mousavi, and R. Nagarajan, "Property-based testing of quantum programs in Q#," in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ser. ICSEW'20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 430--435.

Digital Library

[6]

X. Wang, P. Arcaini, T. Yue, and S. Ali, "Generating failing test suites for quantum programs with search," in Search-Based Software Engineering. Cham: Springer International Publishing, 2021, pp. 9--25.

[7]

D. Fortunato, J. Campos, and R. Abreu, "Mutation testing of quantum programs: A case study with Qiskit," IEEE Transactions on Quantum Engineering, vol. 3, pp. 1--17, 2022.

[8]

X. Wang, T. Yu, P. Arcaini, T. Yue, and S. Ali, "Mutation-based test generation for quantum programs with multi-objective search," in Proceedings of the Genetic and Evolutionary Computation Conference, ser. GECCO '22. New York, NY, USA: Association for Computing Machinery, 2022, pp. 1345--1353.

Digital Library

[9]

J. Wang, F. Ma, and Y. Jiang, "Poster: Fuzz testing of quantum program," in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, pp. 466--469.

[10]

R. Abreu, J. Fernandes, L. Llana, and G. Tavares, "Metamorphic testing of oracle quantum programs," in Proceedings of the 3rd International Workshop on Quantum Software Engineering, ser. Q-SE '22. New York, NY, USA: Association for Computing Machinery, 2023, pp. 16--23.

[11]

S. Ali, P. Arcaini, X. Wang, and T. Yue, "Assessing the effectiveness of input and output coverage criteria for testing quantum programs," in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, pp. 13--23.

[12]

E. Mendiluze, S. Ali, P. Arcaini, and T. Yue, "Muskit: A mutation analysis tool for quantum software testing," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, pp. 1266--1270.

[13]

X. Wang, P. Arcaini, T. Yue, and S. Ali, "Quito: a coverage-guided test generator for quantum programs," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, pp. 1237--1241.

[14]

X. Wang, "QuSBT: Search-based testing of quantum programs," in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ser. ICSE '22. New York, NY, USA: Association for Computing Machinery, 2022, pp. 173--177.

Digital Library

[15]

D. Fortunato, J. Campos, and R. Abreu, "QMutPy: A mutation testing tool for quantum algorithms and applications in Qiskit," in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2022. New York, NY, USA: Association for Computing Machinery, 2022, pp. 797--800.

[16]

C. Nie and H. Leung, "A survey of combinatorial testing," ACM Comput. Surv., vol. 43, no. 2, Feb. 2011.

[17]

J. Tao, Y. Li, F. Wotawa, H. Felbinger, and M. Nica, "On the industrial application of combinatorial testing for autonomous driving functions," in 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2019, pp. 234--240.

[18]

L. Ma, F. Juefei-Xu, M. Xue, B. Li, L. Li, Y. Liu, and J. Zhao, "DeepCT: Tomographic combinatorial testing for deep learning systems," in 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 614--618.

[19]

X. Wang, P. Arcaini, T. Yue, and S. Ali, "Application of combinatorial testing to quantum programs," in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), 2021, pp. 179--188.

[20]

R. Wille, R. Van Meter, and Y. Naveh, "IBM's Qiskit tool chain: Working with and developing for real quantum computers," in 2019 Design, Automation Test in Europe Conference Exhibition (DATE), 2019, pp. 1234--1240.

[21]

A. Agresti, An introduction to categorical data analysis, 3rd ed. Wiley-Blackwell, 2019.

[22]

PICT, "Pairwise independent combinatorial testing," https://github.com/microsoft/pict, 2021.

Cited By

Wang XMuqeet AYue TAli SArcaini P(2024)Test Case Minimization with Quantum AnnealersACM Transactions on Software Engineering and Methodology10.1145/368046734:1(1-24)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680467
De Stefano MPecorelli FDi Nucci DPalomba FDe Lucia A(2024)The quantum frontier of software engineeringInformation and Software Technology10.1016/j.infsof.2024.107525175:COnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107525

Index Terms

QuCAT: A Combinatorial Testing Tool for Quantum Software

Index terms have been assigned to the content through auto-classification.

Recommendations

Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
A Combinatorial Test Suite Generator for Gray-Box Testing
SMC-IT '09: Proceedings of the Third IEEE International Conference on Space Mission Challenges for Information Technology

In black-box testing, the system being tested is typically characterized as a number of inputs, where each input can take one of a number of values. Thus each test is a vector of input settings, and the set of possible tests is an N dimensional space, ...
A combinatorial testing strategy for concurrent programs

One approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on-the-fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

November 2023

2161 pages

ISBN:9798350329964

General Chairs:
Tegawendé F. Bissyandé
University of Luxembourg, Luxembourg
,
Jacques Klein
University of Luxembourg, Luxembourg
,
Program Chairs:
Christian Bird
Microsoft Research, United States
,
Federica Sarro
University College London, United Kingdom

Sponsors

In-Cooperation

University of Luxembourg: University of Luxembourg
IEEE CS

Publisher

IEEE Press

Publication History

Published: 26 September 2024

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE '23

Sponsor:

ASE '23: 38th IEEE/ACM International Conference on Automated Software Engineering

November 11 - 15, 2023

Echternach, Luxembourg

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
9
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang XMuqeet AYue TAli SArcaini P(2024)Test Case Minimization with Quantum AnnealersACM Transactions on Software Engineering and Methodology10.1145/368046734:1(1-24)Online publication date: 27-Jul-2024
https://dl.acm.org/doi/10.1145/3680467
De Stefano MPecorelli FDi Nucci DPalomba FDe Lucia A(2024)The quantum frontier of software engineeringInformation and Software Technology10.1016/j.infsof.2024.107525175:COnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107525

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten