research-article

Towards a fault-detection benchmark for evaluating software product line testing approaches

Authors:

Stefan Fischer,

Roberto Erick Lopez-Herrejon,

Alexander EgyedAuthors Info & Claims

SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

Pages 2034 - 2041

https://doi.org/10.1145/3167132.3167350

Published: 09 April 2018 Publication History

Abstract

Software Product Lines (SPLs) are families of related software systems distinguished by the set of features each one provides. The commonly large number of variants that can be derived from an SPL poses a unique set of challenges, because it is not feasible to test all the individual variants. Over the last few years many approaches for SPL testing have been devised. They usually select a set of variants to test based on some covering criterion. A problem when evaluating these testing approaches is properly comparing them to one another. Even though some benchmarks have been proposed, they focus on covering criteria and do not consider fault data in their analysis. Considering the dire lack of publicly available fault data, in this paper we present the first results of our ongoing project to introduce simulated faults into SPLs along with using evolutionary techniques for synthesizing unit test cases for SPL examples.

References

[1]

Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 variability bugs in the linux kernel: a qualitative analysis. In ACM/IEEE International Conference on Automated Software Engineering, ASE '14, Vasteras, Sweden - September 15 - 19, 2014, Ivica Crnkovic, Marsha Chechik, and Paul Grünbacher (Eds.). ACM, 421--432.

Digital Library

[2]

Sven Apel, Joanne M. Atlee, Luciano Baresi, and Pamela Zave. 2014. Feature Interactions: The Next Generation (Dagstuhl Seminar 14281). Dagstuhl Reports 4, 7 (2014), 1--24.

[3]

Sven Apel, Sergiy S. Kolesnikov, Norbert Siegmund, Christian Kästner, and Brady Garvin. 2013. Exploring feature interactions in the wild: the new feature-interaction challenge. In 5th International Workshop on Feature-Oriented Software Development, FOSD '13, Indianapolis, IN, USA, October 26, 2013. 1--8.

Digital Library

[4]

Don S. Batory, Jacob Neal Sarvela, and Axel Rauschmayer. 2004. Scaling Step-Wise Refinement. IEEE Trans. Software Eng. 30, 6 (2004), 355--371.

Digital Library

[5]

David Benavides, Sergio Segura, and Antonio Ruiz Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Inf. Syst. 35, 6 (2010), 615--636.

Digital Library

[6]

Thorsten Berger, Daniela Lettner, Julia Rubin, Paul GrÃijnbacher, Adeline Silva, Martin Becker, Marsha Chechik, and Krzysztof Czarnecki. 2015. What is a Feature? A Qualitative Study of Features in Industrial Software Product Lines. In Int'l Software Product Line Conference (SPLC'15).

Digital Library

[7]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Trans. Software Eng. 34, 5 (2008), 633--650.

Digital Library

[8]

Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, John D. McGregor, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. A systematic mapping study of software product lines testing. Information & Software Technology 53, 5 (2011), 407--423.

Digital Library

[9]

Ivan do Carmo Machado, John D. McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo Santana de Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information & Software Technology 56, 10 (2014), 1183--1199.

Digital Library

[10]

Ivan do Carmo Machado, John D. McGregor, and Eduardo Santana de Almeida. 2012. Strategies for testing products in software product lines. ACM SIGSOFT Software Engineering Notes 37, 6 (2012), 1--8.

Digital Library

[11]

Emelie Engström and Per Runeson. 2011. Software product line testing - A systematic mapping study. Information & Software Technology 53, 1 (2011), 2--13.

Digital Library

[12]

Gordon Fraser and Andrea Arcuri. 2016. EvoSuite at the SBST 2016 tool competition. In Proceedings of the 9th International Workshop on Search-Based Software Testing, SBST@ICSE 2016, Austin, Texas, USA, May 14-22, 2016. ACM, 33--36.

Digital Library

[13]

MarkHarman, Yue Jia, Jens Krinke, W. B. Langdon, J. Petke, and Y. Zhang. 2014. Search based software engineering for software product line engineering: a survey and directions for future work. In SPLC.

[14]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Eng. 40, 7 (2014), 650--670.

Digital Library

[15]

Arsalan Javeed and Cemal Yilmaz. 2015. Combinatorial interaction testing of tangled configuration options. In Eighth IEEE International Conference on Software Testing, Verification and Validation, ICST 2015 Workshops, Graz, Austria, April 13-17, 2015. IEEE Computer Society, 1--4.

[16]

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Software Eng. 37, 5 (2011), 649--678.

Digital Library

[17]

Roberto E. Lopez-Herrejon and Don S. Batory. 2001. A Standard Problem for Evaluating Product-Line Methodologies. In GCSE (Lecture Notes in Computer Science), Jan Bosch (Ed.), Vol. 2186. Springer, 10--24.

Digital Library

[18]

Roberto E. Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Alexander Egyed, and Enrique Alba. 2015. Computational Intelligence and Quantitative Software Engineering. Springer, Chapter Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges. Accepted for publication.

[19]

Roberto E. Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Evelyn Nicole Haslinger, Alexander Egyed, and Enrique Alba. 2014. Towards a Benchmark and a Comparison Framework for Combinatorial Interaction Testing of Software Product Lines. CoRR abs/1401.5367 (2014).

[20]

Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In ICST Workshops. 1--10.

[21]

Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A Systematic Mapping Study of Search-Based Software Engineering for Software Product Lines. Journal of Information and Software Technology (2015).

Digital Library

[22]

Roberto E. Lopez-Herrejon, Leticia Montalvillo-Mendizabal, and Alexander Egyed. 2011. From Requirements to Features: An Exploratory Study of Feature-Oriented Refactoring. In Software Product Lines - 15th International Conference, SPLC. 181--190.

Digital Library

[23]

Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon. 2005. MuJava: an automated class mutation system. Softw. Test., Verif. Reliab. 15, 2 (2005), 97--133.

Digital Library

[24]

K. Pohl, G. Bockle, and F. J. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles and Techniques. Springer.

Digital Library

[25]

Ana B. Sánchez, Sergio Segura, José Antonio Parejo, and Antonio Ruiz Cortés. 2017. Variability testing in the wild: the Drupal case study. Software and System Modeling 16, 1 (2017), 173--194.

Digital Library

[26]

David J. Sheskin. 2007. Handbook of Parametric and Nonparametric Statistical Procedures (4 ed.). Chapman & Hall/CRC.

Digital Library

[27]

Mikael Svahnberg, Jilles van Gurp, and Jan Bosch. 2005. A taxonomy of variability realization techniques. Softw., Pract. Exper. 35, 8 (2005), 705--754.

[28]

AndrÃąs Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132. arXiv

Cited By

König CRosiak KCleophas LSchaefer I(2023)True Variability Shining Through Taxonomy MiningProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608989(182-193)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608989
Michelon GLinsbauer LAssunção WFischer SEgyed A(2021)A Hybrid Feature Location Technique for Re-engineeringSingle Systems into Software Product LinesProceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3442391.3442403(1-9)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3442391.3442403
Ali SHafeez YJhanjhi NHumayun MImran MNayyar ASingh SRa I(2020)Towards Pattern-Based Change Verification Framework for Cloud-Enabled Healthcare Component-BasedIEEE Access10.1109/ACCESS.2020.30146718(148007-148020)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3014671
Show More Cited By

Index Terms

Towards a fault-detection benchmark for evaluating software product line testing approaches
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing
ICSTW '13: Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops

Needs for mass customization and economies of scale have pushed engineers to develop Software Product Lines (SPLs). SPLs are families of products sharing commonalities and exhibiting differences, built by reusing software assets abstractly represented ...
Risk-based integration testing of software product lines
VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive Systems

Software product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing ...
A concrete product derivation in software product line engineering: a practical approach

Software Product Lines enable the development of a perfect family of products by reusing shared assets in a systematic manner. Product derivation is a critical activity in software product line engineering and one of the most pressing issues that a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

April 2018

2327 pages

ISBN:9781450351911

DOI:10.1145/3167132

Conference Chairs:
Hisham M. Haddad
Kennesaw State University
,
Roger L. Wainwright
University of Tulsa
,
Richard Chbeir
University of Pau & Pays Adour, France

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Sciences and Engineering Research Council of Canada
Austrian Academy of Sciences

Conference

SAC 2018

Sponsor:

SIGAPP

SAC 2018: Symposium on Applied Computing

April 9 - 13, 2018

Pau, France

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
96
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

König CRosiak KCleophas LSchaefer I(2023)True Variability Shining Through Taxonomy MiningProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608989(182-193)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608989
Michelon GLinsbauer LAssunção WFischer SEgyed A(2021)A Hybrid Feature Location Technique for Re-engineeringSingle Systems into Software Product LinesProceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3442391.3442403(1-9)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3442391.3442403
Ali SHafeez YJhanjhi NHumayun MImran MNayyar ASingh SRa I(2020)Towards Pattern-Based Change Verification Framework for Cloud-Enabled Healthcare Component-BasedIEEE Access10.1109/ACCESS.2020.30146718(148007-148020)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3014671
Fischer SMichelon GRamler RLinsbauer LEgyed A(2020)Automated test reuse for highly configurable softwareEmpirical Software Engineering10.1007/s10664-020-09884-xOnline publication date: 16-Sep-2020
https://doi.org/10.1007/s10664-020-09884-x
Ali SHafeez YHussain SYang S(2019)Enhanced regression testing technique for agile software development and continuous integration strategiesSoftware Quality Journal10.1007/s11219-019-09463-4Online publication date: 13-Sep-2019
https://doi.org/10.1007/s11219-019-09463-4
Campos DLima Cdo Carmo Machado IMalucelli AReinehr S(2018)MERCIProceedings of the XVII Brazilian Symposium on Software Quality10.1145/3275245.3275261(151-159)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3275245.3275261

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten