Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3136040.3136053acmconferencesArticle/Chapter ViewAbstractPublication PagesgpceConference Proceedingsconference-collections
research-article

Avoiding useless mutants

Published: 23 October 2017 Publication History

Abstract

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

References

[1]
2017. PITest Mutation Testing Tool for Java. (2017). http://pitest.org/ Accessed: 2017-05-20.
[2]
Konstantinos Adamopoulos, Mark Harman, and Robert M Hierons. 2004. How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Genetic and evolutionary computation conference. 1338-1349.
[3]
Jr. Allen Troy Acree. 1980. On Mutation. Ph.D. Dissertation. Georgia Institute of Technology.
[4]
Cyrille Artho and Lei Ma. 2016. Classification of randomly generated test cases. In Software Analysis, Evolution, and Reengineering (SANER), IEEE 23rd International Conference on. 29-32.
[5]
Douglas Baldwin and Frederick Sayward. 1979. Heuristics for Determining Equivalence of Program Mutations. Technical Report. DTIC Document.
[6]
Timothy Budd and Dana Angluin. 1982. Two notions of correctness and their relation to testing. Acta Informatica 18, 1 (1982), 31-45.
[7]
Richard DeMillo, Richard Lipton, and Frederick Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34-41.
[8]
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. 416-419.
[9]
Bernhard JM Grün, David Schuler, and Andreas Zeller. 2009. The impact of equivalent mutants. In Software Testing, Verification and Validation Workshops. ICSTW. International Conference on. 192-199.
[10]
Rob Hierons, Mark Harman, and Sebastian Danicic. 1999. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability 9, 4 (1999), 233-262.
[11]
Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis. 2nd edition. The MIT Press.
[12]
Daniel Jackson, Ian Schechter, and Hya Shlyahter. 2000. Alcoa: the Alloy constraint analyzer. In Proceedings of the 31st International Conference on Software Engineering (ICSE). IEEE Computer Society, 730-733.
[13]
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649-678.
[14]
René Just, Darioush Jalali, Laura Inozemtseva, Michael Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654-665.
[15]
René Just, Bob Kurtz, and Paul Ammann. 2017. Inferring Mutant Utility from Program Context. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA).
[16]
Rene Just, Franz Schweiggert, and Gregory Kapfhammer. 2011. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering. 612-615.
[17]
Marinos Kintis and Nicos Malevris. 2015. MEDIC: A static analysis framework for equivalent mutant identification. Information and Software Technology 68 (2015), 1-17.
[18]
Marinos Kintis, Mike Papadakis, Yue Jia, Nicos Malevris, Yves Le Traon, and Mark Harman. 2017. Detecting Trivial Mutant Equivalences via Compiler Optimisations. IEEE Transactions on Software Engineering PP, 99 (2017), 1-1.
[19]
Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, and Nicos Malevris. 2016. Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In Source Code Analysis and Manipulation (SCAM), 16th International Working Conference on. 147-156.
[20]
Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon. 2005. MuJava: An automated class mutation system. Software Testing, Verification and Reliability 15, 2 (2005), 97-133.
[21]
Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23-42.
[22]
Melina Mongiovi. 2017. Scaling Testing of Refactoring Engines. Ph.D. Dissertation. Federal University of Campina Grande (UFCG).
[23]
Melina Mongiovi, Gustavo Mendes, Rohit Gheyi, Gustavo Soares, and Márcio Ribeiro. 2014. Scaling testing of refactoring engines. In Software Maintenance and Evolution (ICSME), IEEE International Conference on. 371-380.
[24]
Jeff Offutt, Yu-Seung Ma, and Yong-Rae Kwon. 2006. The Class-level Mutants of MuJava. In Proceedings of the 2006 International Workshop on Automation of Software Test. 78-84.
[25]
Jeff Offutt and Jie Pan. 1996. Detecting equivalent mutants and the feasible path problem. In Computer Assurance, 1996. COMPASS, Systems Integrity. Software Safety. Process Security. Proceedings of the Eleventh Annual Conference on. 224-236.
[26]
Jeff Offutt and Jie Pan. 1997. Automatically detecting equivalent mutants and infeasible paths. Software Testing, Verification and Reliability 7, 3 (1997), 165-192.
[27]
Carlos Pacheco, Shuvendu Lahiri, Michael Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering. 75-84.
[28]
Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. 936-946.
[29]
Corina S. Pasareanu and Neha Rungta. 2010. Symbolic PathFinder: symbolic execution of Java bytecode. In Proceedings of the IEEE/ACM international conference on Automated software engineering. 179-180.
[30]
David Schuler, Valentin Dallmeier, and Andreas Zeller. 2009. Efficient Mutation Testing by Checking Invariant Violations. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis. 69-80.
[31]
David Schuler and Andreas Zeller. 2010. (Un-)Covering Equivalent Mutants. In Third International Conference on Software Testing, Verification and Validation. 45-54.
[32]
David Schuler and Andreas Zeller. 2013. Covering and Uncovering Equivalent Mutants. Software Testing, Verification and Reliability 23, 5 (2013), 353-374.
[33]
Gustavo Soares, Rohit Gheyi, and Tiago Massoni. 2013. Automated behavioral testing of refactoring engines. IEEE Transactions on Software Engineering 39, 2 (2013), 147-162.
[34]
Gustavo Soares, Rohit Gheyi, Dalton Serey, and Tiago Massoni. 2010. Making program refactoring safer. IEEE software 27, 4 (2010), 52-57.
[35]
Jeffrey Voas and Gary McGraw. 1997. Software fault injection: inoculating programs against errors. John Wiley & Sons, Inc.

Cited By

View all
  • (2023)Do Mutations of Strongly Subsuming Second-Order Mutants Really Mask Each Other?2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00079(114-124)Online publication date: 9-Oct-2023
  • (2023)Validation of Mutation Testing in the Safety Critical Industry through a Pilot Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00064(334-343)Online publication date: Apr-2023
  • (2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
GPCE 2017: Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
October 2017
258 pages
ISBN:9781450355247
DOI:10.1145/3136040
  • cover image ACM SIGPLAN Notices
    ACM SIGPLAN Notices  Volume 52, Issue 12
    GPCE '17
    December 2017
    258 pages
    ISSN:0362-1340
    EISSN:1558-1160
    DOI:10.1145/3170492
    Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Code Generation
  2. Duplicated Mutants
  3. Equivalent Mutants
  4. Mutation Testing
  5. Program Transformation

Qualifiers

  • Research-article

Conference

SPLASH '17
Sponsor:

Acceptance Rates

Overall Acceptance Rate 56 of 180 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)30
  • Downloads (Last 6 weeks)3
Reflects downloads up to 26 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2023)Do Mutations of Strongly Subsuming Second-Order Mutants Really Mask Each Other?2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00079(114-124)Online publication date: 9-Oct-2023
  • (2023)Validation of Mutation Testing in the Safety Critical Industry through a Pilot Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00064(334-343)Online publication date: Apr-2023
  • (2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
  • (2022)A Technique to Test Refactoring Detection ToolsProceedings of the XXXVI Brazilian Symposium on Software Engineering10.1145/3555228.3555246(188-197)Online publication date: 5-Oct-2022
  • (2022)Randoop-TSR: Random-based Test Generator with Test Suite ReductionProceedings of the 13th Asia-Pacific Symposium on Internetware10.1145/3545258.3545280(221-230)Online publication date: 11-Jun-2022
  • (2022)Practical Mutation Testing at Scale: A view from GoogleIEEE Transactions on Software Engineering10.1109/TSE.2021.310763448:10(3900-3912)Online publication date: 1-Oct-2022
  • (2021)The Relation of Test-Related Factors to Software Quality: A Case Study on Apache SystemsEmpirical Software Engineering10.1007/s10664-020-09891-y26:2Online publication date: 20-Feb-2021
  • (2021)An ensemble‐based predictive mutation testing approach that considers impact of unreached mutantsSoftware Testing, Verification and Reliability10.1002/stvr.178431:7Online publication date: 2-Jun-2021
  • (2020)A Preliminary Investigation into Using Machine Learning Algorithms to Identify Minimal and Equivalent Mutants2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00056(304-313)Online publication date: Oct-2020
  • (2020)Mutating Code Annotations: An Empirical Evaluation on Java and C# ProgramsScience of Computer Programming10.1016/j.scico.2020.102418(102418)Online publication date: Feb-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media