research-article

Mutation Operators for Preprocessor-Based Variability

Authors:

Mustafa Al-Hajjaji,

Fabian Benduhn,

Gunter SaakeAuthors Info & Claims

VaMoS '16: Proceedings of the 10th International Workshop on Variability Modelling of Software-Intensive Systems

Pages 81 - 88

https://doi.org/10.1145/2866614.2866626

Published: 27 January 2016 Publication History

Abstract

Mutation testing has emerged as one of the most promising techniques to increase the quality of software-intensive systems. In mutation testing, random faults based on a predefined set of mutation operators are automatically injected into a program to evaluate test suites. The effectiveness of mutation testing strongly depends on the representativeness of the mutation operators. Existing operators are not sufficient to represent typical faults caused by variability. Thus, we propose a set of mutation operators for software with preprocessor-based variability. We derive the operators systematically based on a taxonomy of variability-related faults and evaluate them by investigating their applicability to real-world faults that have previously been identified in research on configurable software systems. Our goal is to leverage mutation testing to highly-variable software for its practical application and to enable empirical evaluation of testing techniques.

References

[1]

I. Abal, C. Brabrand, and A. Wasowski. 42 Variability Bugs in the Linux Kernel: A Qualitative Analysis. In Proc. Int'l Conf. Automated Software Engineering (ASE), pages 421--432. ACM, 2014.

Digital Library

[2]

M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In Proc. Int'l Software Product Line Conf. (SPLC), pages 197--206. ACM, 2014.

Digital Library

[3]

P. Anbalagan and T. Xie. Automated Generation of Pointcut Mutants for Testing Pointcuts in AspectJ Programs. In Proc. Int'l Symposium Software Reliability Engineering (ISSRE), pages 239--248, 2008.

Digital Library

[4]

S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013.

[5]

P. Arcaini, A. Gargantini, and P. Vavassori. Generating Tests for Detecting Faults in Feature Models. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 1--10, 2015.

[6]

S. S. Batth, E. R. Vieira, A. Cavalli, and M. U. Uyar. Specification of Timed EFSM Fault Models in SDL. In Proc. IFIP Int'l Conf. Formal Methods for Open Object-Based Distributed Systems (FMOODS), pages 50--65. Springer, 2007.

Digital Library

[7]

P. Borba, L. Teixeira, and R. Gheyi. A Theory of Software Product Line Refinement. Theoretical Computer Science, 455(0):2--30, 2012.

Digital Library

[8]

P. Chevalley. Applying mutation analysis for object-oriented programs using a reflective approach. In Proc. Asia-Pacific Software Engineering Conference (APSEC), pages 267--270, 2001.

Digital Library

[9]

T. R. Devine, K. Goseva-Popstajanova, S. Krishnan, R. R. Lutz, and J. J. Li. An Empirical Study of Pre-Release Software Faults in an Industrial Product Line. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 181--190, 2012.

Digital Library

[10]

F. Ensan, E. Bagheri, and D. Gasevic. Evolutionary Search-Based Test Generation for Software Product Line Feature Models. In Proc. Int'l Conf. Advanced Information Systems Engineering (CAiSE), volume 7328, pages 613--628. Springer, 2012.

Digital Library

[11]

G. Fraser and F. Wotawa. Mutant Minimization for Model-Checker Based Test-Case Generation. In in Proc. of Workshop on Mutation Analysis, Published with Proc. of Testing: Academic and Industrial Conference Practice and Research Techniques, pages 161--168, 2007.

Digital Library

[12]

B. Garvin and M. Cohen. Feature Interaction Faults Revisited: An Exploratory Study. In Proc. Int'l Symposium Software Reliability Engineering (ISSRE), pages 90--99, 2011.

Digital Library

[13]

C. Henard, M. Papadakis, and Y. Le Traon. Mutation-Based Generation of Software Product Line Test Configurations. In C. Le Goues and S. Yoo, editors, Search-Based Software Engineering, volume 8636 of Lecture Notes in Computer Science, pages 92--106. Springer International Publishing, 2014.

[14]

Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Software Engineering (TSE), 37(5):649--678, 2011.

Digital Library

[15]

K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute, 1990.

[16]

C. Kästner, P. G. Giarrusso, T. Rendel, S. Erdweg, K. Ostermann, and T. Berger. Variability-Aware Parsing in the Presence of Lexical Macros and Conditional Compilation. In Proc. Conf. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), pages 805--824. ACM, 2011.

Digital Library

[17]

S. Krishnan, R. R. Lutz, and K. Goševa-Popstojanova. Empirical Evaluation of Reliability Improvement in an Evolving Software Product Line. In Proc. of Working Conf. on Mining Software Repositories (MSR), pages 103--112. ACM, 2011.

Digital Library

[18]

S. Krishnan, C. Strasburg, R. R. Lutz, and K. Goševa-Popstojanova. Are Change Metrics Good Predictors for an Evolving Software Product Line? In Proc. of Int'l Conf. on Predictive Models in Software Engineering (Promise), pages 1--10. ACM, 2011.

Digital Library

[19]

H. Lackner and M. Schmidt. Towards the Assessment of Software Product Line Tests: A Mutation System for Variable Systems. In Proc. Workshop Software Product Line Analysis Tools (SPLat), pages 62--69. ACM, 2014.

Digital Library

[20]

M. Papadakis, C. Henard, and Y. Le Traon. Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 1--10, 2014.

Digital Library

[21]

D. Reuling, J. Bürdek, S. Rotärmel, M. Lochau, and U. Kelter. Fault-Based Product-Line Testing: Effective Sample Generation Based on Feature-Diagram Mutation. In Proc. Int'l Software Product Line Conf. (SPLC), pages 131--140. ACM, 2015.

Digital Library

[22]

H. A. Richard, R. A. Demillo, B. Hathaway, W. Hsu, W. Hsu, E. Krauser, R. Martin, A. P. Mathur, and E. H. Spafford. Design Of Mutant Operators For The C Programming Language. Technical Report SERC-TR-41-P, Purdue University, West Lafayette, Indiana, 1989.

[23]

A. Sánchez, S. Segura, J. Parejo, and A. Ruiz-Cortés. Variability Testing in the Wild: The Drupal Case Study. Software and System Modeling, pages 1--22, 2015.

[24]

A. B. Sánchez, S. Segura, and A. Ruiz-Cortés. A Comparison of Test Case Prioritization Criteria for Software Product Lines. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 41--50. IEEE, 2014.

Digital Library

[25]

T. Thüm, C. Kästner, S. Erdweg, and N. Siegmund. Abstract Features in Feature Modeling. In Proc. Int'l Software Product Line Conf. (SPLC), pages 191--200. IEEE, 2011.

Digital Library

[26]

T. Thüm, J. Meinicke, F. Benduhn, M. Hentschel, A. von Rhein, and G. Saake. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines. In Proc. Int'l Software Product Line Conf. (SPLC), pages 177--186. ACM, 2014.

Digital Library

Cited By

Güthing LBittner PSchaefer IThüm T(2024)Explaining Edits to Variability Annotations in Evolving Software Product LinesProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634725(93-102)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634725
Bittner PTinnes CSchultheiß AViegener SKehrer TThüm TRoychoudhury ACadar CKim M(2022)Classifying edits to variability in source codeProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549108(196-208)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549108
Temple PPerrouin GAcher MBiggio BJézéquel JRoli F(2021)Empirical assessment of generating adversarial configurations for software product linesEmpirical Software Engineering10.1007/s10664-020-09915-726:1Online publication date: 12-Jan-2021
https://dl.acm.org/doi/10.1007/s10664-020-09915-7
Show More Cited By

Index Terms

Mutation Operators for Preprocessor-Based Variability

Recommendations

Mutation Operators for Actor Systems
ICSTW '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops

Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does ...
Mutation Operators for Large Scale Data Processing Programs in Spark
Advanced Information Systems Engineering
Abstract
This paper proposes a mutation testing approach for big data processing programs that follow a data flow model, such as those implemented on top of Apache Spark. Mutation testing is a fault-based technique that relies on fault simulation by ...
Mutation Testing for Aspect-Oriented Programs
ICST '08: Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation

Mutation testing has been shown to be one of the strongest testing criteria for the evaluation of both programs and test suites. Comprehensive sets of mutants require strong test sets to achieve acceptable testing coverage. Moreover, mutation operators ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

VaMoS '16: Proceedings of the 10th International Workshop on Variability Modelling of Software-Intensive Systems

January 2016

116 pages

ISBN:9781450340199

DOI:10.1145/2866614

Editors:
Ina Schaefer
TU Braunschweig, Germany
,
Vander Alves
University of Brasília, Brazil
,
Eduardo Santana de Almeida
Federal University of Bahia and RiSE Labs, Brazil

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SBC: Sociedade Brasileira de Computação
FAPESB: Fundação de Amparo à Pesquisa do Estado da Bahia
University of Brasília: University of Brasília
Technische Universität Braunschweig

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

VaMoS '16

VaMoS '16: Tenth International Workshop on Variability Modelling of Software-intensive Systems

January 27 - 29, 2016

Salvador, Brazil

Acceptance Rates

Overall Acceptance Rate 66 of 147 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Güthing LBittner PSchaefer IThüm T(2024)Explaining Edits to Variability Annotations in Evolving Software Product LinesProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634725(93-102)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634725
Bittner PTinnes CSchultheiß AViegener SKehrer TThüm TRoychoudhury ACadar CKim M(2022)Classifying edits to variability in source codeProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549108(196-208)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549108
Temple PPerrouin GAcher MBiggio BJézéquel JRoli F(2021)Empirical assessment of generating adversarial configurations for software product linesEmpirical Software Engineering10.1007/s10664-020-09915-726:1Online publication date: 12-Jan-2021
https://dl.acm.org/doi/10.1007/s10664-020-09915-7
Ruland SLuthmann LBürdek JLity SThüm TLochau MRibeiro M(2020)Measuring effectiveness of sample-based product-line testingACM SIGPLAN Notices10.1145/3393934.327813053:9(119-133)Online publication date: 7-Apr-2020
https://dl.acm.org/doi/10.1145/3393934.3278130
Ruland SLuthmann LBürdek JLity SThüm TLochau MRibeiro MVan Wyk ERompf T(2018)Measuring effectiveness of sample-based product-line testingProceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3278122.3278130(119-133)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3278122.3278130
Pinheiro PViana JFernandes LRibeiro MFerrari FFonseca BGheyi R(2018)Mutation Operators for Code AnnotationsProceedings of the III Brazilian Symposium on Systematic and Automated Software Testing10.1145/3266003.3266006(77-86)Online publication date: 17-Sep-2018
https://dl.acm.org/doi/10.1145/3266003.3266006
Krüger JAl-Hajjaji MSchulze SSaake GLeich T(2018)Towards automated test refactoring for software product linesProceedings of the 22nd International Systems and Software Product Line Conference - Volume 110.1145/3233027.3233040(143-148)Online publication date: 10-Sep-2018
https://dl.acm.org/doi/10.1145/3233027.3233040
Arab IBourhnane SVozenilek V(2018)Reducing the cost of mutation operators through a novel taxonomyProceedings of the International Conference on Geoinformatics and Data Analysis10.1145/3220228.3220264(47-56)Online publication date: 20-Apr-2018
https://dl.acm.org/doi/10.1145/3220228.3220264
Krieter SKrüger JLeich T(2018)Don't Worry About itProceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3168365.3170426(19-26)Online publication date: 7-Feb-2018
https://dl.acm.org/doi/10.1145/3168365.3170426
Carvalho LGuimarães MRibeiro MFernandes LAl-Hajjaji MGheyi RThüm T(2018)Equivalent Mutants in Configurable SystemsProceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3168365.3168379(11-18)Online publication date: 7-Feb-2018
https://dl.acm.org/doi/10.1145/3168365.3168379
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents