article

Tool demo: testing configurable systems with FeatureIDE

Authors:

Mustafa Al-Hajjaji,

Sebastian Krieter,

Reimar Schröter,

Gunter SaakeAuthors Info & Claims

ACM SIGPLAN Notices, Volume 52, Issue 3

Pages 173 - 177

https://doi.org/10.1145/3093335.2993254

Published: 20 October 2016 Publication History

Abstract

Most software systems are designed to provide custom functionality using configuration options. Testing such systems is challenging as running tests of a single configuration is often not sufficient, because defects may appear in other configurations. Ideally, all configurations of a software system should be tested, which is usually not applicable in practice due to the combinatorial explosion with respect to the configuration options. Multiple sampling strategies aim to reduce the set of tested configurations to a feasible amount, such as T-wise sampling, random configurations, and user-defined configurations. However, these strategies are often not applied in practice as they require manual effort or a specialized testing framework. Within our tool FeatureIDE, we integrate all aforementioned strategies and reduce the manual effort by automating the process of generating and testing configurations. Furthermore, we provide support for unit testing to avoid redundant test executions and for variability-aware testing. With this extension of FeatureIDE, we aim to make recent testing techniques for configurable systems applicable in practice.

References

[1]

M. Al-Hajjaji, S. Krieter, T. Thüm, M. Lochau, and G. Saake. IncLing: Efficient Product-Line Testing Using Incremental Pairwise Sampling. In GPCE, 2016. To appear.

Digital Library

[2]

M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In SPLC, pp. 197–206. ACM, 2014.

Digital Library

[3]

S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013.

Digital Library

[4]

S. Apel, C. Kästner, and C. Lengauer. Language-Independent and Automated Software Composition: The FeatureHouse Experience. TSE, 39(1):63–79, 2013.

Digital Library

[5]

S. Apel, A. von Rhein, P. Wendler, A. Größlinger, and D. Beyer. Strategies for Product-Line Verification: Case Studies and Experiments. In ICSE, pp. 482–491. IEEE, 2013.

Digital Library

[6]

D. Batory. Feature Models, Grammars, and Propositional Formulas. In SPLC, pp. 7–20. Springer, 2005.

Digital Library

[7]

D. Batory, J. N. Sarvela, and A. Rauschmayer. Scaling Step-Wise Refinement. TSE, 30(6):355–371, 2004.

Digital Library

[8]

B. Beckert, R. Hähnle, and P. Schmitt. Verification of Object-Oriented Software: The KeY Approach. Springer, 2007.

Digital Library

[9]

D. Benavides, S. Segura, and A. Ruiz-Cortés. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Information Systems, 35(6):615–708, 2010.

Digital Library

[10]

J. Bürdek, M. Lochau, S. Bauregger, A. Holzer, A. von Rhein, S. Apel, and D. Beyer. Facilitating Reuse in Multi-goal Test-Suite Generation for Software Product Lines. In FASE, pp. 84–99. Springer, 2015.

[11]

V. Chvatal. A Greedy Heuristic for the Set-Covering Problem. MOR, 4(3):233–235, 1979.

Digital Library

[12]

B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Evaluating Improvements to a Meta-Heuristic Search for Constrained Interaction Testing. EMSE, 16(1):61–102, 2011.

Digital Library

[13]

K. Havelund and T. Pressburger. Model Checking Java Programs Using Java PathFinder. STTT, 2(4):366–381, 2000.

[14]

C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. L. Traon. PLEDGE: A Product Line Editor and Test Generation Tool. In SPLC, pp. 126–129. ACM, 2013.

Digital Library

[15]

K. J. Hoffman, P. Eugster, and S. Jagannathan. Semanticsaware Trace Analysis. In PLDI, pp. 453–464. ACM, 2009.

Digital Library

[16]

M. Jackson and P. Zave. Distributed Feature Composition: A Virtual Architecture for Telecommunications Services. TSE, 24(10):831–847, 1998.

Digital Library

[17]

M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In MODELS, pp. 638–652. Springer, 2011.

Digital Library

[18]

M. F. Johansen, Ø. Haugen, and F. Fleurey. An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models. In SPLC, pp. 46–55. ACM, 2012.

Digital Library

[19]

G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.-M. Loingtier, and J. Irwin. Aspect-Oriented Programming. In ECOOP, pp. 220–242. Springer, 1997.

[20]

C. H. P. Kim, D. Marinov, S. Khurshid, D. Batory, S. Souto, P. Barros, and M. d’Amorim. SPLat: Lightweight Dynamic Analysis for Reducing Combinatorics in Testing Configurable Systems. In ESEC/FSE, pp. 257–267. ACM, 2013.

Digital Library

[21]

D. R. Kuhn, D. R. Wallace, and A. M. Gallo Jr. Software Fault Interactions and Implications for Software Testing. TSE, 30(6):418–421, 2004.

Digital Library

[22]

D. Le Berre and A. Parrain. The sat4j Library, Release 2.2, System Description. JSAT, 7:59–64, 2010.

[23]

J. McGregor. Testing a Software Product Line. In Testing Techniques in Software Engineering, pp. 104–140. Springer, 2010.

[24]

F. Medeiros, C. Kästner, M. Ribeiro, S. Nadi, and R. Gheyi. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In ECOOP, pp. 495–518. Schloss Dagstuhl– Leibniz-Zentrum fuer Informatik, 2015.

[25]

J. Meinicke, T. Thüm, R. Schröter, S. Krieter, F. Benduhn, G. Saake, and T. Leich. FeatureIDE: Taming the Preprocessor Wilderness. In ICSE, pp. 629–632. ACM, 2016.

Digital Library

[26]

J. Meinicke, C.-P. Wong, C. Kästner, T. Thüm, and G. Saake. On Essential Configuration Complexity: Measuring Interactions in Highly-configurable Systems. In ASE, pp. 483–494. ACM, 2016.

Digital Library

[27]

J. A. Pereira, S. Krieter, J. Meinicke, R. Schröter, G. Saake, and T. Leich. FeatureIDE: Scalable Product Configuration of Variable Systems. In ICSR, pp. 397–401. Springer, 2016.

Digital Library

[28]

G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines. In ICST, pp. 459–468. IEEE, 2010.

Digital Library

[29]

M. Steffens, S. Oster, M. Lochau, and T. Fogdal. Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VaMoS, pp. 55–62. ACM, 2012.

Digital Library

[30]

R. Tartler, D. Lohmann, J. Sincero, and W. Schröder-Preikschat. Feature Consistency in Compile-Time-Configurable System Software: Facing the Linux 10,000 Feature Problem. In EuroSys, pp. 47–60. ACM, 2011.

Digital Library

[31]

T. Thüm, S. Apel, C. Kästner, I. Schaefer, and G. Saake. A Classification and Survey of Analysis Strategies for Software Product Lines. CSUR, 47(1):6:1–6:45, 2014.

Digital Library

[32]

T. Thüm, C. Kästner, F. Benduhn, J. Meinicke, G. Saake, and T. Leich. FeatureIDE: An Extensible Framework for Feature-Oriented Software Development. SCP, 79(0):70–85, 2014.

Digital Library

[33]

T. Thüm, J. Meinicke, F. Benduhn, M. Hentschel, A. von Rhein, and G. Saake. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines. In SPLC, pp. 177–186. ACM, 2014.

Digital Library

[34]

A. von Rhein, S. Apel, and F. Raimondi. Introducing Binary Decision Diagrams in the Explicit-State Verification of Java code. In JavaPathfinder Workshop, 2011.

[35]

B. Xin, W. N. Sumner, and X. Zhang. Efficient Program Execution Indexing. In PLDI, pp. 238–248. ACM, 2008.

Digital Library

Cited By

Abolfazli ASpiegelberg JPalmer GAnand A(2023)A Deep Reinforcement Learning Approach to Configuration Sampling Problem2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00009(1-10)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00009
Pinnecke MMousavi MSchobbens P(2021)Product-lining the elinvar wealthtech microservice platformProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B10.1145/3461002.3473947(60-68)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461002.3473947
Schlingloff Bernd-Holger Hoppe Niels (2021)A Framework for Cloud-based Testing of Multi-variant Cyber-physical Systems2021 10th Mediterranean Conference on Embedded Computing (MECO)10.1109/MECO52532.2021.9460159(1-4)Online publication date: 7-Jun-2021
https://doi.org/10.1109/MECO52532.2021.9460159
Show More Cited By

Index Terms

Tool demo: testing configurable systems with FeatureIDE
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Recommendations

Tool demo: testing configurable systems with FeatureIDE
GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences

Most software systems are designed to provide custom functionality using configuration options. Testing such systems is challenging as running tests of a single configuration is often not sufficient, because defects may appear in other configurations. ...
Automating Test Reuse for Highly Configurable Software
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Dealing with highly configurable systems is generally very complex. Hundreds of different analysis techniques have been conceived to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This ...
Automated test reuse for highly configurable software
Abstract
Dealing with highly configurable systems is generally very complex. Researchers and practitioners have conceived hundreds of different analysis techniques to deal with different aspects of configurable systems. One large focal point is the testing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 52, Issue 3

GPCE '16

March 2017

212 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/3093335

Editor:
Matthew Fluet

Issue’s Table of Contents

GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
October 2016
212 pages
ISBN:9781450344463
DOI:10.1145/2993236
General Chair:
Bernd Fischer
Stellenbosch University, South Africa
,
Program Chair:
Ina Schaefer
TU Braunschweig, Germany

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2016

Published in SIGPLAN Volume 52, Issue 3

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abolfazli ASpiegelberg JPalmer GAnand A(2023)A Deep Reinforcement Learning Approach to Configuration Sampling Problem2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00009(1-10)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00009
Pinnecke MMousavi MSchobbens P(2021)Product-lining the elinvar wealthtech microservice platformProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B10.1145/3461002.3473947(60-68)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461002.3473947
Schlingloff Bernd-Holger Hoppe Niels (2021)A Framework for Cloud-based Testing of Multi-variant Cyber-physical Systems2021 10th Mediterranean Conference on Embedded Computing (MECO)10.1109/MECO52532.2021.9460159(1-4)Online publication date: 7-Jun-2021
https://doi.org/10.1109/MECO52532.2021.9460159
Ferreira FViggiato MSouza MFigueiredo ELopez-Herrejon R(2020)Testing configurable software systemsProceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A10.1145/3382025.3414725(1-6)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3382025.3414725
Krieter SThüm TSchulze SSaake GLeich T(2020)YASAProceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3377024.3377042(1-10)Online publication date: 5-Feb-2020
https://dl.acm.org/doi/10.1145/3377024.3377042
Ali AHafeez YHussain SYang S(2020)Role of Requirement Prioritization Technique to Improve the Quality of Highly-Configurable SystemsIEEE Access10.1109/ACCESS.2020.29713828(27549-27573)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2971382
Lity SAl-Hajjaji MThüm TSchaefer ISchaefer I(2017)Optimizing product orders using graph algorithms for improving incremental product-line analysisProceedings of the Eleventh International Workshop on Variability Modelling of Software-intensive Systems10.1145/3023956.3023961(60-67)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1145/3023956.3023961
Mendonça WVergilio SMichelon GEgyed AAssunção WFelfernig AFuentes L(2022)Test2FeatureProceedings of the 26th ACM International Systems and Software Product Line Conference - Volume B10.1145/3503229.3547031(62-65)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3503229.3547031
Pett TKrieter SRunge TThüm TLochau MSchaefer I(2021)Stability of Product-Line Samplingin Continuous IntegrationProceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3442391.3442410(1-9)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3442391.3442410
Ferreira FDiniz JSilva CFigueiredo EWeyns DPerrouin GWeyns D(2019)Testing Tools for Configurable Software SystemsProceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3302333.3302344(1-10)Online publication date: 6-Feb-2019
https://dl.acm.org/doi/10.1145/3302333.3302344
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents