research-article

Strategies for testing products in software product lines

Authors:

Ivan do Carmo Machado,

John D. McGregor,

Eduardo Santana de AlmeidaAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 37, Issue 6

Pages 1 - 8

https://doi.org/10.1145/2382756.2382783

Published: 27 November 2012 Publication History

Abstract

The software product line engineering strategy enables the achievement of significant improvements in quality through reuse of carefully crafted software assets across multiple products. However, high levels of quality in the software product line assets, which are used to create products, must be accompanied by effective and efficient test strategies for the products in the software product line. The goal of this study is to understand which strategies for testing products in software product lines have been reported in the literature, enabling discussions on the significant issues, and also pointing out further research directions. A systematic literature review was carried out that identified two hundred seventy-three papers, published from the years 1998 and early in 2012. From such a set of papers, a systematic selection resulted in forty-one relevant papers. The analysis of the reported strategies comprised two important aspects: the selection of products for testing, and the actual test of products. The findings showed a range of strategies, dealing with both aspects, but few empirical evaluations of their effectiveness have been performed, which limits the inferences that can be drawn.

References

[1]

J. Al-Dallal and P. G. Sorenson. Testing software assets of framework-based product families during application engineering stage. Journal of Software, 3(5):11--25, 2008.

[2]

A. Bertolino, A. Fantechi, S. Gnesi, and G. Lami. Product line use cases: Scenario-based specification and testing of requirements. In Software Product Lines - Research Issues in Engineering and Management, pages 425--445. Springer, 2006.

[3]

M. N. Borazjany, L. Yu, Y. Lei, R. Kacker, and R. Kuhn. Combinatorial testing of acts: A case study. In Proc. of the 5th IEEE International Conference on Software Testing, Verification and Validation, ICST, pages 591--600, Montreal, Canada, 2012.

Digital Library

[4]

I. Cabral, M. B. Cohen, and G. Rothermel. Improving the testing and testability of software product lines. In Proc. of the 14th International Conference on Software Product Lines, SPLC, pages 241--255, Jeju Island, South Korea, 2010. Springer-Verlag.

Digital Library

[5]

H. Cichos, S. Oster, M. Lochau, and A. Schürr. Model-based coverage-driven test suite generation for software product lines. In Proc. of the 14th International Conference on Model Driven Engineering Languages and Systems (MODELS), pages 425--439, Wellington, New Zealand, 2011. Springer-Verlag.

Digital Library

[6]

P. Clements and J. D. McGregor. Better, faster, cheaper: Pick any three. Business Horizons, 55(2):201--208, 2012.

[7]

E. Engström and P. Runeson. Software product line testing - a systematic mapping study. Information & Software Technology, 53(1):2--13, 2011.

Digital Library

[8]

E. Engström, M. Skoglund, and P. Runeson. Empirical evaluations of regression test selection techniques: a systematic review. In 2nd Intl. Symposium on Empirical Software Engineering and Measurement, ESEM, pages 22--31, Kaiserslautern, 2008. ACM.

Digital Library

[9]

A. Ensan, E. Bagheri, M. Asadi, D. Gasevic, and Y. Biletskiy. Goal-oriented test case selection and prioritization for product line feature models. In Eighth International Conference on Information Technology: New Generations, ITNG, pages 291--298, Las Vegas, USA, 2011. IEEE Computer Society.

Digital Library

[10]

B. Geppert, J. J. Li, F. Rößler, and D. M. Weiss. Towards generating acceptance tests for product lines. In Proc. of 8th Intl. Conference on Software Reuse, ICSR, pages 35--48, Madrid, Spain, 2004. Springer.

[11]

T. Gustafsson. An approach for selecting software product line instances for testing. In 11th International Conference on Software Product Lines, SPLC, pages 81--86, Tokyo, Japan, 2007.

[12]

J. Hartmann, M. Vieira, and A. Ruder. A UML-based approach for validating product lines. In Proc. of the Intl. Workshop on Software Product Line Testing, SPLiT, pages 58--65, Boston, MA, USA, 2004.

[13]

A. Hervieu, B. Baudry, and A. Gotlieb. PACOGEN: Automatic generation of pairwise test configurations from feature models. In 22nd IEEE International Symposium on Software Reliability Engineering, ISSRE, pages 120--129, Hiroshima, Japan, 2011.

Digital Library

[14]

M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of realistic feature models make combinatorial testing of product lines feasible. In 14th International Conference on Model Driven Engineering Languages and Systems, MODELS, pages 638--652, Wellington, New Zealand, 2011. Springer.

Digital Library

[15]

T. Kahsai, M. Roggenbach, and B.-H. Schlinglo. Specification-based testing for software product lines. In Sixth IEEE International Conference on Software Engineering and Formal Methods, SEFM, pages 149--158, Cape Town, South Africa, 2008. IEEE.

Digital Library

[16]

E. Kamsties, K. Pohl, S. Reis, and A. Reuys. Testing variabilities in use case models. In Proc. of 5th Intl. Workshop Software Product-Family Engineering, PFE, pages 6--18, Siena, Italy, 2003. Springer.

[17]

C. H. P. Kim, D. S. Batory, and S. Khurshid. Reducing combinatorics in testing product lines. In Proc. of the 10th Intl. Conference on Aspect-oriented Software Development, AOSD, pages 57--68. ACM, 2011.

Digital Library

[18]

B. Kitchenham and S. Charters. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report, 2007.

[19]

R. Kolb and D. Muthig. Challenges in testing software product lines. In Proceedings of the 7th Conference on Quality Engineering in Software Technology, CONQUEST, pages 81--95. Fraunhofer Publica, 2003.

[20]

B. P. Lamancha, M. Usaola, and I. de Guzman. Model-driven testing in software product lines. In IEEE Intl. Conference on Software Maintenance, ICSM, pages 511--514, Edmonton, Canada, 2009.

[21]

B. P. Lamancha and M. P. Usaola. Testing product generation in software product lines using pairwise for features coverage. In Proc. of the 22nd IFIP Intl. Conference on Testing Software and Systems, ICTSS, pages 111--125, Natal, Brazil, 2010. Springer-Verlag.

Digital Library

[22]

B. P. Lamancha, M. P. Usaola, and M. P. Velthius. Software product line testing - a systematic review. In Proc. of the 4th International Conference on Software and Data Technologies, ICSOFT, pages 23--30, Soa, Bulgaria, 2009. INSTICC Press.

[23]

K. Lee, K. C. Kang, and J. Lee. Concepts and guidelines of feature modeling for product line software engineering. In Proc. of the 7th Intl. Conference on Software Reuse, ICSR, pages 62--77, London, UK, 2002. Springer-Verlag.

Digital Library

[24]

M. Lochau, S. Oster, U. Goltz, and A. Schürr. Model-based pairwise testing for feature interaction coverage in software product line engineering. Software Quality Journal, pages 1--38, 2011.

Digital Library

[25]

P. R. Mateo, M. Polo, and B. P. Lamancha. Automatic generation of test cases in software product lines. In Proceedings of the 11th International Conference on Enterprise Information Systems, ICEIS, pages 124--130, Milan, Italy, 2009. Springer.

[26]

J. D. McGregor. Testing a software product line. Technical Report TR-022, CMU Software Engineering Institute, 2001.

[27]

S. Mishra. Specification based software product line testing: A case study. In 15th International Workshop on Concurrency, Specification and Programming, CS&P, pages 243--254, Wandlitz, Germany, 2006.

[28]

M. Mohamed Ali and R. Moawad. An approach for requirements based software product line testing. In 7th International Conference on Informatics and Systems, INFOS, pages 1--10, Cairo, Egypt, 2010.

[29]

C. Nebut, S. Pickin, Y. Le Traon, and J.-M. Jézéquel. Automated requirements-based generation of test cases for product families. In Proc. of the 18th IEEE Intl. Conference on Automated Software Engineering, ASE, pages 263--266, Montreal, Canada, 2003.

Digital Library

[30]

C. Nebut, S. Pickin, Y. L. Traon, and J. M. Jézéquel. Reusable test requirements for uml-modeled product lines. In Proceedings of the Workshop on Requirements Engineering for Product Lines, REPL, pages 51--56, Essen, Germany, 2002. Avaya Labs.

[31]

C. Nebut, Y. Traon, and J.-M. Jézéquel. System testing of product lines: From requirements to test cases. In T. Käköla and J. C. Duenas, editors, Software Product Lines, pages 447--477. Springer, 2006.

[32]

P. A. S. Neto, I. C. Machado, J. D. McGregor, E. S. Almeida, and S. R. L. Meira. A systematic mapping study of software product lines testing. Information & Software Technology, 53(5):407--423, 2011.

Digital Library

[33]

E. M. Olimpiew and H. Gomaa. Model-based testing for applications derived from software product lines. In Proc. of the 1st Intl. Workshop on Advances in model-based testing, A-MOST, pages 1--7. ACM, 2005.

Digital Library

[34]

E. M. Olimpiew and H. Gomaa. Customizable requirements-based test models for software product lines. In Proc. of the 3rd International Workshop on Software Product Line Testing, SPLiT, pages 17--22, Baltimore, USA, 2006.

[35]

E. M. Olimpiew and H. Gomaa. Reusable model-based testing. In 11th Intl. Conference on Software Reuse, ICSR, pages 76--85, Falls Church, USA, 2009. Springer.

Digital Library

[36]

S. Oster, F. Markert, and P. Ritter. Automated incremental pairwise testing of software product lines. In Proc. of the 14th International Conference on Software Product Lines, SPLC, pages 196--210, Jeju Island, South Korea, 2010. Springer-Verlag.

Digital Library

[37]

S. Oster, A. Schürr, and I. Weisemöller. Towards software product line testing using story driven modeling. Fakultät Informatik, pages 48--51, 2008.

[38]

S. Oster, M. Zink, M. Lochau, and M. Grechanik. Pairwise feature-interaction testing for spls: potentials and limitations. In Proc. of the 15th International Software Product Line Conference, SPLC, pages 1--8, Munich, Germany, 2011. ACM.

Digital Library

[39]

B. Perez Lamancha, M. Polo Usaola, and M. Piattini Velthius. A model based testing approach for model-driven development and software product lines. In Proc. of the 5th Intl. Conference on Evaluation of Novel Approaches to Software Engineering, pages 193--208, Athens, Greece, 2010.

[40]

G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Automated and scalable T-wise test case generation strategies for software product lines. In 3rd Intl. Conference on Software Testing, Verification and Validation, ICST, pages 459--468, Paris, France, 2010.

Digital Library

[41]

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. Systematic mapping studies in software engineering. In 12th Intl. Conf. on Evaluation and Assessment in Software Engineering, EASE, Bari, Italy, 2008.

Digital Library

[42]

A. P. K. Reis, S.; Metzger. A reuse technique for performance testing of software product lines. In Proc. of the 3rd Intl. Workshop on Software Product Line Testing, SPLiT, pages 5--10, Baltimore, USA, 2006.

[43]

A. Reuys, S. Reis, E. Kamsties, and K. Pohl. Derivation of domain test scenarios from activity diagrams. In Proc. of the International Workshop on Product Line Engineering, PLEES, pages 35--42, Erfut, Germany, 2003. Fraunhofer IESE.

[44]

A. Reuys, S. Reis, E. Kamsties, and K. Pohl. The scented method for testing software product lines. In T. Käköla and J. C. Duenas, editors, Software Product Lines, pages 479--520. Springer, 2006.

[45]

C. Robinson-Mallett, M. Grochtmann, J. Wegener, J. Kohnlein, and S. Kuhn. Modelling requirements to support testing of product lines. In Proc. of the 3rd International Conference on Software Testing, Verification, and Validation Workshops, ICSTW, pages 11--18, Paris, France, 2010. IEEE.

Digital Library

[46]

K. Scheidemann. Optimizing the selection of representative configurations in verification of evolving product lines of distributed embedded systems. In Proc. of the 10th Intl. Software Product Line Conference, SPLC, pages 75--84. IEEE, 2006.

Digital Library

[47]

S. Segura, J. A. Galindo, D. Benavides, J. A. Parejo, and A. Ruiz-Cortés. Betty: benchmarking and testing on the automated analysis of feature models. In 6th Intl. Workshop on Variability Modeling of Software- Intensive Systems, VaMoS, pages 63--71. ACM, 2012.

Digital Library

[48]

S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortés. Automated metamorphic testing on the analyses of feature models. Information and Software Technology, 53(3):245--258, 2011.

Digital Library

[49]

J. Shi, M. Cohen, and M. Dwyer. Integration testing of software product lines using compositional symbolic execution. In 15th International Conference on Fundamental Approaches to Software Engineering, FASE, pages 270--284, Tallinn, Estonia, 2012. Springer.

Digital Library

[50]

V. Stricker, A. Metzger, and K. Pohl. Avoiding redundant testing in application engineering. In Proc. of the 14th international conference on Software product lines, SPLC, pages 226--240. Springer, 2010.

Digital Library

[51]

A. Tevanlinna, J. Taina, and R. Kauppinen. Product family testing: a survey. ACM SIGSOFT Software Engineering Notes, 29(2):12, 2004.

Digital Library

[52]

E. Uzuncaova, S. Khurshid, and D. Batory. Incremental test generation for software product lines. IEEE Transactions on Software Engineering, 36(3):309--322, 2010.

Digital Library

Cited By

Martou PMens KDuhoux BLegay A(2023)Test scenario generation for feature-based context-oriented software systemsJournal of Systems and Software10.1016/j.jss.2022.111570197(111570)Online publication date: Mar-2023
https://doi.org/10.1016/j.jss.2022.111570
Nguyen TNgo KNguyen SVo H(2021)A Variability Fault Localization Approach for Software Product LinesIEEE Transactions on Software Engineering10.1109/TSE.2021.311385948:10(4100-4118)Online publication date: 20-Sep-2021
https://dl.acm.org/doi/10.1109/TSE.2021.3113859
Martínez-Arias JÁlvarez-Vargas GSierra-Galvis MPabón-Burbano MLinares-Ospina DCastillo-Saavedra APortilla-Córdoba AAlmanza-Trujillo V(2020)SATReLO: A tool to support language therapies for children with hearing disabilities using video gamesRevista Facultad de Ingeniería Universidad de Antioquia10.17533/udea.redin.20200586(99-112)Online publication date: 4-Jun-2020
https://doi.org/10.17533/udea.redin.20200586
Show More Cited By

Index Terms

Strategies for testing products in software product lines
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software development techniques
      1. Reusability
  2. Software notations and tools
    1. Software configuration management and version control systems
    2. Software libraries and repositories

Recommendations

On strategies for testing software product lines: A systematic literature review

Context: Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but techniques that can take advantage of these ...
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line Conference

Software Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...
Risk-based integration testing of software product lines
VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive Systems

Software product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 37, Issue 6

November 2012

191 pages

ISSN:0163-5948

DOI:10.1145/2382756

Issue’s Table of Contents

Copyright © 2012 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 November 2012

Published in SIGSOFT Volume 37, Issue 6

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
652
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)2

Reflects downloads up to 17 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Martou PMens KDuhoux BLegay A(2023)Test scenario generation for feature-based context-oriented software systemsJournal of Systems and Software10.1016/j.jss.2022.111570197(111570)Online publication date: Mar-2023
https://doi.org/10.1016/j.jss.2022.111570
Nguyen TNgo KNguyen SVo H(2021)A Variability Fault Localization Approach for Software Product LinesIEEE Transactions on Software Engineering10.1109/TSE.2021.311385948:10(4100-4118)Online publication date: 20-Sep-2021
https://dl.acm.org/doi/10.1109/TSE.2021.3113859
Martínez-Arias JÁlvarez-Vargas GSierra-Galvis MPabón-Burbano MLinares-Ospina DCastillo-Saavedra APortilla-Córdoba AAlmanza-Trujillo V(2020)SATReLO: A tool to support language therapies for children with hearing disabilities using video gamesRevista Facultad de Ingeniería Universidad de Antioquia10.17533/udea.redin.20200586(99-112)Online publication date: 4-Jun-2020
https://doi.org/10.17533/udea.redin.20200586
Bertolino ALonetti Fde Oliveira Neves V(2020)Standing on the Shoulders of Software Product Line Research for Testing Systems of Systems2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW51248.2020.00074(209-214)Online publication date: Oct-2020
https://doi.org/10.1109/ISSREW51248.2020.00074
Lee JKang SJung P(2020)Test coverage criteria for software product line testing: Systematic literature reviewInformation and Software Technology10.1016/j.infsof.2020.106272122(106272)Online publication date: Jun-2020
https://doi.org/10.1016/j.infsof.2020.106272
Fischer SMichelon GRamler RLinsbauer LEgyed A(2020)Automated test reuse for highly configurable softwareEmpirical Software Engineering10.1007/s10664-020-09884-x25:6(5295-5332)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1007/s10664-020-09884-x
Nešić DKrüger JStănciulescu ȘBerger TDumas MPfahl DApel SRusso A(2019)Principles of feature modelingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338974(62-73)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338974
Fischer SRamler RLinsbauer LEgyed A(2019)Automating Test Reuse for Highly Configurable SoftwareProceedings of the 23rd International Systems and Software Product Line Conference - Volume A10.1145/3336294.3336305(1-11)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3336294.3336305
Strüber DMukelabai MKrüger JFischer SLinsbauer LMartinez JBerger T(2019)Facing the TruthProceedings of the 23rd International Systems and Software Product Line Conference - Volume A10.1145/3336294.3336302(177-188)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3336294.3336302
Lity SNieke MThüm TSchaefer I(2019)Retest test selection for product-line regression testing of variants and versions of variantsJournal of Systems and Software10.1016/j.jss.2018.09.090147(46-63)Online publication date: Jan-2019
https://doi.org/10.1016/j.jss.2018.09.090
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents