research-article

Better predicate testing

Authors:

Jeff OffuttAuthors Info & Claims

AST '11: Proceedings of the 6th International Workshop on Automation of Software Test

Pages 57 - 63

https://doi.org/10.1145/1982595.1982608

Published: 23 May 2011 Publication History

Abstract

Mutation testing is widely recognized as being extremely powerful, but is considered difficult to automate enough for practical use. This paper theoretically addresses two possible reasons for this: the generation of redundant mutants and the lack of integration of mutation analysis with other test criteria. By addressing these two issues, this paper brings an important mutation operator, relational-operator-replacement (ROR), closer to practical use. First, we develop fault hierarchies for the six relational operators, each of which generates seven mutants per clause. These hierarchies show that, for any given clause, only three mutants are necessary. This theoretical result can be integrated easily into mutation analysis tools, thereby eliminating generation of 57% of the ROR mutants. Second, we show how to bring the power of the ROR operator to the widely used Multiple Condition-Decision Coverage (MCDC) test criterion. This theoretical result includes an algorithm to transform any MCDC-adequate test set into a test set that also satisfies RORG, a new version of ROR appropriate for the MCDC context. The transformation does not use traditional mutation analysis, so can easily be integrated into existing MCDC tools and processes.

References

[1]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press, Cambridge, UK, 2008. ISBN 0-52188-038-1.

Digital Library

[2]

P. Ammann, J. Offutt, and H. Huang. Coverage criteria for logical expressions. In Proceedings of the 14th International Symposium on Software Reliability Engineering, pages 99--107, Denver, CO, November 2003. IEEE Computer Society Press.

Digital Library

[3]

T. Y. Chen, M. F. Lau, and Y. T. Yu. MUMCUT: A fault-based strategy for testing boolean specifications. In APSEC '99: Proceedings of the Sixth Asia Pacific Software Engineering Conference, pages 606--613, Takamatsu, Japan, 1999. IEEE Computer Society Press.

Digital Library

[4]

J. Chilenski and L. A. Richey. Definition for a masking form of modified condition decision coverage (MCDC). Technical report, Boeing, Seattle, WA, 1997. http://www.boeing.com/nosearch/mcdc/.

[5]

J. J. Chilenski and S. P. Miller. Applicability of modified condition/decision coverage to software testing. Software Engineering Journal, 9(5):193--200, September 1994.

[6]

M. E. Delamaro and J. C. Maldonado. Proteum-A tool for the assessment of test adequacy for C programs. In Proceedings of the Conference on Performability in Computing Systems (PCS 96), pages 79--95, New Brunswick, NJ, July 1996.

[7]

R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4):34--41, April 1978.

Digital Library

[8]

R. A. DeMillo and J. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900--910, September 1991.

Digital Library

[9]

A. Gargantini and G. Fraser. Generating minimal fault detecting test suites for boolean expressions. In AMOST 2010 - 6th Workshop on Advances in Model Based Testing, pages 37--45, Paris, France, April 2010.

Digital Library

[10]

J. M. Hanks. Testing COBOL programs by mutation: Volume I-introduction to the CMS.1 system, volume II - CMS.1 system documentation. Technical report GIT-ICS-80/04, Georgia Institute of Technology, February 1980.

[11]

Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation, pages 249--258, Beijing, September 2008.

[12]

Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions of Software Engineering, To appear, 2010.

Digital Library

[13]

G. Kaminski and P. Ammann. Using a fault hierarchy to improve the efficiency of DNF logic mutation testing. In 2nd IEEE International Conference on Software Testing, Verification and Validation (ICST 2009), pages 386--395, Denver, CO, April 2009.

Digital Library

[14]

G. Kaminski and P. Ammann. Using logic criterion feasibility to reduce test set size while guaranteeing fault detection. In 2nd IEEE International Conference on Software Testing, Verification and Validation (ICST 2009), pages 356--365, Denver, CO, April 2009.

Digital Library

[15]

G. Kaminski and P. Ammann. Applications of optimization to logic testing. In CSTVA 2010 - 2nd Workshop on Constraints in Software Testing, Verification and Analysis, pages 331--336, Paris, France, April 2010.

Digital Library

[16]

G. Kaminski and P. Ammann. Reducing logic test set size while preserving fault detection. Journal of Software Testing, Verification and Reliability, Wiley, to appear. Special issue from the 2009 International Conference on Software Testing, Verification and Validation.

Digital Library

[17]

G. Kaminski, G. Williams, and P. Ammann. Reconciling perspectives of logic testing for software. Journal of Software Testing, Verification and Reliability, Wiley, 18(3):149--188, September 2008.

Digital Library

[18]

S. Kim, J. A. Clark, and J. A. McDermid. Investigating the effectiveness of object-oriented strategies with the mutation method. In Proceedings of Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centuries, pages 4--100, San Jose, CA, October 2000. Wiley's Software Testing, Verification, and Reliability, December 2001.

[19]

K. N. King and J. Offutt. A Fortran language system for mutation-based software testing. Software-Practice and Experience, 21(7):685--718, July 1991.

Digital Library

[20]

M. F. Lau and Y. T. Yu. An extended fault class hierarchy for specification-based testing. ACM Transactions on Software Engineering Methodology, 14(3):247--276, July 2005.

Digital Library

[21]

N. Li, U. Praphamontripong, and J. Offutt. An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In Fifth Workshop on Mutation Analysis (Mutation 2009), Denver CO, April 2009.

Digital Library

[22]

Y.-S. Ma, J. Offutt, and Y.-R. Kwon. MuJava : An automated class mutation system. Software Testing, Verification, and Reliability, Wiley, 15(2):97--133, June 2005.

Digital Library

[23]

J. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf. An experimental determination of sufficient mutation operators. ACM Transactions on Software Engineering Methodology, 5(2):99--118, April 1996.

Digital Library

[24]

J. Offutt and S. D. Lee. An empirical evaluation of weak mutation. IEEE Transactions on Software Engineering, 20(5):337--344, May 1994.

Digital Library

[25]

J. Offutt, J. Payne, and J. M. Voas. Mutation operators for Ada. Technical report ISSE-TR-96-09, Department of Information and Software Engineering, George Mason University, Fairfax VA, March 1996. http://www.cs.gmu.edu/tr~admin/.

[26]

RTCA-DO-178B. Software considerations in airborne systems and equipment certification, December 1992.

[27]

C. Sun, Y. Dong, R. Lai, K. Y. Sim, and T. Y. Chen. Analyzing and extending MUMCUT for fault-based testing of general boolean expressions. In The Sixth IEEE International Conference on Computer and Information Technology, pages 184--189, Seoul, Korea, September 2006.

Digital Library

Cited By

Ojdanić MRoychoudhury ACadar CKim M(2022)Change-aware mutation testing for evolving systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558911(1785-1789)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558911
Gong DWang TSu X(2022)Golden Mutator Recommendation Based on Mutation Pattern MiningInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S0218001422500252Online publication date: 28-May-2022
https://doi.org/10.1142/S0218001422500252
Godboley SMohapatra D(2022)Towards Agile Mutation Testing Using Branch Coverage Based Prioritization TechniqueLean and Agile Software Development10.1007/978-3-030-94238-0_9(150-169)Online publication date: 12-Jan-2022
https://doi.org/10.1007/978-3-030-94238-0_9
Show More Cited By

Index Terms

Better predicate testing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Accumulator-based BIST approach for stuck-open and delay fault testing
EDTC '95: Proceedings of the 1995 European conference on Design and Test

In this paper a novel accumulator-based Built-In Self Test (BIST) method for complete two-pattern test generation is presented. Complete two-pattern testing has been proposed for stuck-open and delay testing. The proposed scheme is very attractive for a ...
A self-driven test structure for pseudorandom testing of non-scan sequential circuits
VTS '96: Proceedings of the 14th IEEE VLSI Test Symposium

Introduced is a self-driven test point structure which permits at-speed, on-chip, non-scan, sequential testing using parallel pseudorandom test patterns applied only to the primary inputs of the circuit under test. The test network is unique in that ...
The use of IDDQ testing in low stuck-at coverage situations
VTS '95: Proceedings of the 13th IEEE VLSI Test Symposium

Abstract: An unresolved issue in IC testing is what mix of coverages of different types of test is required in order to achieve a given quality goal. This paper investigates the interaction between IDDQ and stuck-at coverage and examines the use of IDDQ ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

AST '11: Proceedings of the 6th International Workshop on Automation of Software Test

May 2011

124 pages

ISBN:9781450305921

DOI:10.1145/1982595

Program Chairs:
Antonia Bertolino
ISTI-CNR, Italy
,
Howard Foster
City University London, UK
,
J. Jenny Li
Avaya Labs Research, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE11

Sponsor:

SIGSOFT

ICSE11: International Conference on Software Engineering

May 23 - 24, 2011

HI, Waikiki, Honolulu, USA

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
314
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ojdanić MRoychoudhury ACadar CKim M(2022)Change-aware mutation testing for evolving systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558911(1785-1789)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558911
Gong DWang TSu X(2022)Golden Mutator Recommendation Based on Mutation Pattern MiningInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S0218001422500252Online publication date: 28-May-2022
https://doi.org/10.1142/S0218001422500252
Godboley SMohapatra D(2022)Towards Agile Mutation Testing Using Branch Coverage Based Prioritization TechniqueLean and Agile Software Development10.1007/978-3-030-94238-0_9(150-169)Online publication date: 12-Jan-2022
https://doi.org/10.1007/978-3-030-94238-0_9
Zhou WWang YCheng XWang X(2020)Predicate Testing Generation for Safety-critical Systems2020 6th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR51244.2020.00032(143-150)Online publication date: Oct-2020
https://doi.org/10.1109/ISSSR51244.2020.00032
Guimaraes MFernandes LRibeiro Md'Amorim MGheyi R(2020)Optimizing Mutation Testing by Discovering Dynamic Mutant Subsumption Relations2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST46399.2020.00029(198-208)Online publication date: Oct-2020
https://doi.org/10.1109/ICST46399.2020.00029
Lin HWang YGong YJin D(2019)Domain-RIP Analysis: A Technique for Analyzing Mutation StubbornnessIEEE Access10.1109/ACCESS.2018.28837767(4006-4023)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2018.2883776
Lin HWang YGong YChaudron MCrnkovic IChechik MHarman M(2018)Subsuming mutation operatorsProceedings of the 40th International Conference on Software Engineering: Companion Proceeedings10.1145/3183440.3194948(236-237)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3183440.3194948
Kintis MPapadakis MJia YMalevris NLe Traon YHarman M(2018)Detecting Trivial Mutant Equivalences via Compiler OptimisationsIEEE Transactions on Software Engineering10.1109/TSE.2017.268480544:4(308-333)Online publication date: 1-Apr-2018
https://doi.org/10.1109/TSE.2017.2684805
Lindström BMárki A(2018)On strong mutation and the theory of subsuming logic‐based mutantsSoftware Testing, Verification and Reliability10.1002/stvr.166729:1-2Online publication date: 19-Apr-2018
https://doi.org/10.1002/stvr.1667
Ahmed IJensen CGroce AMcKenney P(2017)Applying Mutation Analysis on Kernel Test Suites: An Experience Report2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2017.26(110-115)Online publication date: Mar-2017
https://doi.org/10.1109/ICSTW.2017.26
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents