research-article

Gathering current knowledge about quality evaluation in software product lines

Authors:

Sonia Montagud,

Silvia AbrahãoAuthors Info & Claims

SPLC '09: Proceedings of the 13th International Software Product Line Conference

Pages 91 - 100

Published: 24 August 2009 Publication History

Abstract

Recently, a number of methods and techniques for assessing the quality of software product lines have been proposed. However, to the best of our knowledge, there is no study which summarizes all the existing evidence about them. This paper presents a systematic review that investigates what methods and techniques have been employed (in the last 10 years) to evaluate the quality of software product lines and how they were employed. A total of 39 research papers have been reviewed from an initial set of 1388 papers. The results show that 25% of the papers reported evaluations at the Design phase of the Domain Engineering phase. The most widely used mechanism for modeling quality attributes was extended feature models and the most evaluated artifact was the base architecture. In addition, the results of the review have identified several research gaps. Specifically, 77% of the papers employed case studies as a "proof of concept" whereas 23% of the papers did not perform any type of validation. Our results are particularly relevant in positioning new research activities and in the selection of quality evaluation methods or techniques that best fit a given purpose.

References

[1]

M. A. Babar. "Evaluating Product Line Architectures: Methods and Techniques". 14th Asia-Pacific Software Engineering Conference, 2007, pp. 13.

Digital Library

[2]

J. Bayer. "Design for Quality". In Software Product-Family Engineering, 2003, pp. 370--380.

[3]

D. Benavides, S. Segura, P. Trinidad and A. Ruiz-Cortés. "FAMA: Tooling a Framework for the automated analysis of feature models". In 1st Internacional Workshop on Variability Modelling of Software Intensive Systems, 2007, pp. 129--134.

[4]

J. Bosch. "Design and use of software architectures: adopting and evolving a product line approach". ACM Press/Addison-Wesley Publishing Co., USA, 2000.

Digital Library

[5]

R. Capilla and J. C. Dueñas. "Modeling variabiltit with features in distributed architectures". In Software Product-Family Engineering, 2002, pp. 46--109.

Digital Library

[6]

A. Davis, Ó. Dieste, A. Hickey, N. Juristo N. and A. M. Moreno. "Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review". In 14th IEEE International Conference Requirements Engineering, 2006, pp. 179--188.

Digital Library

[7]

J. Dehlinger and R. R. Lutz. "PLFaultCAT: A Product-Line Software Fault Tree Analysis Tool". In Automated Software Engineering, 2006, pp. 169--193.

Digital Library

[8]

C. Del Rosso. "The process and the lessons learned from performance tuning of a product family software architecture for mobile phones". In 8th European Conference on Software Maintenance and Reengineering, pp. 270--275.

Digital Library

[9]

T. J. Dolan. Architecture Assessment of Information-System Families: a practical perspective. PhD thesis, Tech. Univ. Eindhoven, Netherlands, 2001.

[10]

L. Ejiogu. "Software Engineering with Formal Metrics", QED Publishing, 1991.

Digital Library

[11]

L. Etxeberria, G. Sagardui and L. Belategi. "Quality aware Software Product Line Engineering" Journal of the Brazilian Computer Society (JBCS), vol. 14, no. 1, Mar 2008.

[12]

L. Etxeberria and G. Sagardui. "Variability Driven Quality Evaluation in Software Product Lines". In Software Product Line Conference, 2008, pp. 243--252.

Digital Library

[13]

L. Etxeberria and G. Sagardiu. "Evaluation of Quality Attribute Variability in Software Product Families". In 15th IEEE International Conference and Workshop on the Engineering of Computer Based Systems, 2008, pp. 255--264.

Digital Library

[14]

L. Etxeberria and G. Sagardui. "Quality assessment in software product lines, High Confidence Software Reuse in Large Systems". In 10th International Conference on Software Reuse, ICSR. Beijing, 2008, pp. 178--181.

Digital Library

[15]

N. Fenton and S. L. Pfleeger. Software Metrics: A Rigorous and Practical Approach. In 2nd Edition. International Thomson Computer Press, 1996.

Digital Library

[16]

J. L. Fleiss. Statistical Methods for Rates and Proportions. Wiley, New York, USA, 1981.

[17]

G. C. Gannod and R. R. Lutz. An Approach to Architectural Analysis of Product Lines. In 22nd International Conference on Software Engineering, 2000, pp. 548--557.

Digital Library

[18]

A. Helferich, G. Herzwurm and S. Schockert. "QFD-PPP: Product Line Portfolio Planning using Quality Function Deployment". In Software Product Lines Conference, 2005, pp. 162--173.

Digital Library

[19]

J. S. Her, J. H. Kim, S. H. Oh, S. Y. Rhew and S. D. Kim. "A framework for evaluating reusability of core asset in product line engineering". Information and Software Technology, 2007, vol. 49(7), pp. 740--760.

Digital Library

[20]

Z. Huang, R. R. Raje, A. M. Olson, M. Auguston, B. R. Bryant, C. Burt and C. Sun. "Unified Approach for System-Level Generative Programming". In 5th International Conference on Algorithms and Architectures for Parrallel Processing (ICA3PP), 2002, pp. 136--142.

Digital Library

[21]

IEEE Standard for a software quality metrics methodology, IEEE Std. 1061--1998, 1998.

[22]

M. Jaring and J. Bosch. "Architecting Product Diversification - Formalizing Variability Dependencies in Software Product Family Engineering". IN 4th International Conference on Quality Software, 2004, pp. 154--161.

Digital Library

[23]

T. Kim, I. Y. Ko, S. W. Kang and D. H. Lee. "Extending ATAM to assess product line architecture". In 8th IEEE International Conference on Computer and Information Technology, pp. 790--797, 2008.

[24]

B. Kitchenham. "Guidelines for Performing Systematic Literature Reviews in Software Engineering". Version 2.3, EBSE Technical Report, Keele University, UK.

[25]

B. Kitchenham. "Procedures for Performing Systematic Reviews". Joint Technical Report Software Engineering Group, Keele University, UK and Empirical Software Engineering, National ICT Australia Ltd, Australia, 2004.

[26]

B. Kitchenham, S. Pfleeger and N. Fenton. "Towards a framework for software measurement validation". IEEE Transactions on Software Engineering 21 (12), 1995.

Digital Library

[27]

M. Khurum, T. Gorschek and K. Petterson. "Systematic Review of Solutions Proposed for Product Line Economics". In 2nd International Workshop on Management and Economics of Software Product Lines, Limerick, Ireland, 2008, pp. 386--393.

[28]

R. Kolb, I. John, J. Knodel, D. Muthing, U. Haury and G. Meier. "Experiences with Product Line Development of Embedded Systems at Testo AG". In 10th International Software Product Line Conference, 2006.

Digital Library

[29]

R. R. Lutz and G. C. Gannod. "Analysis of a software product line architecture: an experience report". Journal of Systems and Software, vol. 66(3), 2003, pp. 253--267.

Digital Library

[30]

M. Matinlassi, E. Niemelä and L. Dobrica. "Quality-driven architecture design and quality analysis method: A revolutionary initiation approach to a product line architecture". Technical Report VTT-PUBS-456, VTT, 2002.

[31]

D. Mellado, E. Fernandez-Medina and M. Piatiini. "Security Requirements Variability for Software Product Lines". In 3rd International Conference on Availability, Reliability and Security (ARES), 2008, pp. 1413--1420.

Digital Library

[32]

E. Mendes. "A systematic review of Web engineering research". International Symposium on Empirical Software Engineering. 2005. Noosa Heads, Australia.

[33]

D. Needham and S. Jones. "A Software Fault Tree Metric". 2006. Journal of Systems and Software, 2007, vol. 80(9), pp. 1530--1540.

Digital Library

[34]

E. Niemelä and A. Immonen. "Capturing quality requirements of product family architecture". Information and Software Technology, 2007, vol. 49(11--12), pp. 1107--1120.

Digital Library

[35]

F. G. Olumofin and V. B. Mišić. "A holistic architecture assessment method for software product lines". Information and Software Technology 49, 2007, pp. 309--323.

Digital Library

[36]

O. Osatretin Edwin. Testing in Software Product Lines. School of Engineering, Master Thesis, Blekinge Institute of Technology, Sweden. March 2007.

[37]

G. Raghavan. "Improving software quality in product families through systematic reengineering". ECSQ 2002

Digital Library

[38]

M. Sinnema, J. S. van der Ven and S. Deelstra. "Using Variability Modeling Principles to Capture Architectural Knowledge". In ACM SIGSOFT Software Engineering Notes, 31(5), 2006.

Digital Library

[39]

E. D. de Souza Filho, R. de Oliveira Cavalcanti, D. F. S. Neiva, T. H. B. Oliveira, L. Barachisio Lisboa, E. S. de Almeida and S. R. de Lemos Meira. "Evaluating Domain Design Approaches Using Systematic Review". In 2nd European Conference on Software Architecture, Cyprus, 2008, pp. 50--65.

Digital Library

[40]

S. Thiel. "On the definition of a framework for an architecting process supporting product family development". In 4th International Workshop on Software Product-Family Engineering, Springer-Verlag, London, UK, 2002, pp. 125--142.

Digital Library

[41]

P. Trinidad, D. Benavides, A. Durán, A. Ruiz-Cortés and M. Toro. "Automated error analysis for the agilization of feature modeling". Jouranl of Systems and Software, 2008, vol. 81(6), pp. 883--896.

Digital Library

[42]

A. Van der Hoek, E. Dincel and N. Medvidovic. "Using service utilization metrics to assess the structure of product software architectures". In 9th International Symposium on Software Metrics, 2003, pp. 298.

Digital Library

[43]

F. Van der Linden, K. Schmid and E. Rommes. Software Product Lines in Action. Springer, 2007.

Digital Library

[44]

H. Zhang, S. Jarzabek and B. Yang: "Quality Prediction and Assessment for Product Lines". In 15th International Conference On Advanced Information Systems Engineering, Klagenfurt/Velden, Austria, 2003, pp. 681--695.

Digital Library

Cited By

Bezerra CAndrade RMonteiro JCedraz D(2018)Aggregating Measures using Fuzzy Logic for Evaluating Feature ModelsProceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3168365.3168375(35-42)Online publication date: 7-Feb-2018
https://dl.acm.org/doi/10.1145/3168365.3168375
Thüm TApel SKästner CSchaefer ISaake G(2014)A Classification and Survey of Analysis Strategies for Software Product LinesACM Computing Surveys10.1145/258095047:1(1-45)Online publication date: 1-Jun-2014
https://dl.acm.org/doi/10.1145/2580950
Ghezzi CMolzam Sharifloo A(2013)Model-based verification of quantitative non-functional properties for software product linesInformation and Software Technology10.1016/j.infsof.2012.07.01755:3(508-524)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1016/j.infsof.2012.07.017
Show More Cited By

Index Terms

Gathering current knowledge about quality evaluation in software product lines

Recommendations

Variability Driven Quality Evaluation in Software Product Lines
SPLC '08: Proceedings of the 2008 12th International Software Product Line Conference

Variability is a key aspect in software product lines. Functional variability has been largely studied as a way to obtain all the desired products for a line. Quality variability, less understood and more complex, has not received so much attention by ...
Evaluation of Quality Attribute Variability in Software Product Families
ECBS '08: Proceedings of the 15th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems

Software product family or line is a software engineering paradigm that systematizes reuse. In Software Product Line Engineering, two phases are distinguished: Domain Engineering which is in charge of developing a common infrastructure and assets and ...
Evaluation Process Model of the Software Product Quality Levels
ICIICII '15: Proceedings of the 2015 International Conference on Industrial Informatics - Computing Technology, Intelligent Technology, Industrial Information Integration (ICIICII)

Nowadays, software industry is still facing many problems in controlling and evaluating software product quality. One of the primary reasons for this is that except for the lack of an objective software product quality assessment model, software ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SPLC '09: Proceedings of the 13th International Software Product Line Conference

August 2009

319 pages

Conference Chair:
Dirk Muthig
Lufthansa Systems Passenger Services GmbH, Germany
,
Program Chair:
John D. McGregor
Clemson University

Publisher

Carnegie Mellon University

United States

Publication History

Published: 24 August 2009

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SPLC '09

SPLC '09: 13th International Software Product Line Conference

August 24 - 28, 2009

California, San Francisco, USA

Acceptance Rates

SPLC '09 Paper Acceptance Rate 30 of 82 submissions, 37%;

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
576
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bezerra CAndrade RMonteiro JCedraz D(2018)Aggregating Measures using Fuzzy Logic for Evaluating Feature ModelsProceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3168365.3168375(35-42)Online publication date: 7-Feb-2018
https://dl.acm.org/doi/10.1145/3168365.3168375
Thüm TApel SKästner CSchaefer ISaake G(2014)A Classification and Survey of Analysis Strategies for Software Product LinesACM Computing Surveys10.1145/258095047:1(1-45)Online publication date: 1-Jun-2014
https://dl.acm.org/doi/10.1145/2580950
Ghezzi CMolzam Sharifloo A(2013)Model-based verification of quantitative non-functional properties for software product linesInformation and Software Technology10.1016/j.infsof.2012.07.01755:3(508-524)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1016/j.infsof.2012.07.017
Myllärniemi VRaatikainen MMännistö TSantana de Almeida ESchwanninger CBenavides D(2012)A systematically conducted literature reviewProceedings of the 16th International Software Product Line Conference - Volume 110.1145/2362536.2362546(41-45)Online publication date: 2-Sep-2012
https://dl.acm.org/doi/10.1145/2362536.2362546

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents