Article

Configurations everywhere: implications for testing and debugging in practice

Authors:

Brian RobinsonAuthors Info & Claims

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

Pages 215 - 224

https://doi.org/10.1145/2591062.2591191

Published: 31 May 2014 Publication History

Abstract

Many industrial systems are highly-configurable, complicating the testing and debugging process. While researchers have developed techniques to statically extract, quantify and manipulate the valid system configurations, we conjecture that many of these techniques will fail in practice. In this paper we analyze a highly-configurable industrial application and two open source applications in order to quantify the true challenges that configurability creates for software testing and debugging. We find that (1) all three applications consist of multiple programming languages, hence static analyses need to cross programming language barriers to work, (2) there are many access points and methods to modify configurations, implying that practitioners need configuration traceability and should gather and merge metadata from more than one source and (3) the configuration state of an application on failure cannot be reliably determined by reading persistent data; a runtime memory dump or other heuristics must be used for accurate debugging. We conclude with a roadmap and lessons learned to help practitioners better handle configurability now, and that may lead to new configuration-aware testing and debugging techniques in the future.

References

[1]

N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann. What makes a good bug report? In International Symposium on Foundations of Software Engineering, FSE, pages 308–318, 2008.

Digital Library

[2]

S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. Information needs in bug reports: improving cooperation between developers and users. In Conference on Computer Supported Cooperative Work, CSCW, pages 301–310, 2010.

Digital Library

[3]

J. Clause and A. Orso. A Technique for Enabling and Supporting Debugging of Field Failures. In International Conference on Software Engineering, ICSE, pages 261–270, Minneapolis, Minnesota, May 2007.

Digital Library

[4]

J. Cleland-Huang, J. H. Hayes, and J. M. Domel. Model-based traceability. In ICSE Workshop on Traceability in Emerging Forms of Software Engineering, TEFSE, pages 6–10, 2009.

Digital Library

[5]

M. B. Cohen, M. B. Dwyer, and J. Shi. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering, 34(5):633–650, 2008.

Digital Library

[6]

M. B. Cohen, J. Snyder, and G. Rothermel. Testing across configurations: implications for combinatorial testing. SIGSOFT Software Engineering Notes, 31(6):1–9, 2006.

Digital Library

[7]

T. Dasgupta, M. Grechanik, E. Moritz, B. Dit, and D. Poshyvanyk. Enhancing software traceability by automatically expanding corpora with relevant documentation. In International Conference on Software Maintenance, ICSM, pages 22–28, Sep 2013.

Digital Library

[8]

J.-M. Davril, E. Delfosse, N. Hariri, M. Acher, J. Cleland-Huang, and P. Heymans. Feature model extraction from large collections of informal product descriptions. In The Joint Meeting on Foundations of Software Engineering, ESEC/FSE, pages 290–300, 2013.

Digital Library

[9]

LibreOﬃce. http://libreoffice.org/, 2013.

[10]

E. Dumlu, C. Yilmaz, M. B. Cohen, and A. Porter. Feedback driven adaptive combinatorial testing. In International Symposium on Software Testing and Analysis, ISSTA, pages 243–253, 2011.

Digital Library

[11]

B. Garvin, M. B. Cohen, and M. B. Dwyer. Failure avoidance in configurable systems through feature locality. In J. Camára, R. Lemos, C. Ghezzi, and A. Lopes, editors, Assurances for Self-Adaptive Systems, volume 7740 of Lecture Notes in Computer Science, pages 266–296. Springer Berlin Heidelberg, 2013.

[12]

O. Gotel, J. Cleland-Huang, J. H. Hayes, A. Zisman, A. Egyed, P. Grunbacher, and G. Antoniol. The quest for ubiquity: A roadmap for software and systems traceability research. International Requirements Engineering Conference, RE, 0:71–80, 2012.

Digital Library

[13]

S. Herbold, J. Grabowski, S. Waack, and U. Bünting. Improved bug reporting and reproduction through non-intrusive GUI usage monitoring and automated replaying. In International Conference on Software Testing, Verification and Validation Workshops, ICSTW, pages 232–241, 2011.

Digital Library

[14]

W. Jin and A. Orso. BugRedux: reproducing field failures for in-house debugging. In International Conference on Software Engineering, ICSE, pages 474–484, 2012.

Digital Library

[15]

L. C. Lamb, W. Jirapanthong, and A. Zisman. Formalizing traceability relations for product lines. In ICSE Workshop on Traceability in Emerging Forms of Software Engineering, TEFSE, pages 42–45, 2011.

Digital Library

[16]

A. D. Lucia, F. Fasano, R. Oliveto, and G. Tortora. Recovering traceability links in software artifact management systems using information retrieval methods. ACM Transactions on Software Engineering and Methodology, 16(4), Sept. 2007.

Digital Library

[17]

J. I. Maletic and M. L. Collard. TQL: A query language to support traceability. In ICSE Workshop on Traceability in Emerging Forms of Software Engineering, TEFSE, pages 16–20, 2009.

Digital Library

[18]

A. Marcus, X. Xie, and D. Poshyvanyk. When and how to visualize traceability links? In ICSE Workshop on Traceability in Emerging Forms of Software Engineering, TEFSE, pages 56–61, 2005.

Digital Library

[19]

Firefox. http://www.mozilla.org/en-US/firefox/, 2013.

[20]

X. Qu, M. Acharya, and B. Robinson. Configuration selection using code change impact analysis for regression testing. International Conference on Software Maintenance, ICSM, 0:129–138, 2012.

Digital Library

[21]

X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: An empirical study of sampling and prioritization. In International Symposium on Software Testing and Analysis, ISSTA, pages 75–85, July 2008.

Digital Library

[22]

X. Qu, M. B. Cohen, and K. M. Woolf. Combinatorial interaction regression testing: A study of test case generation and prioritization. In International Conference on Software Maintenance, ICSM, pages 255–264, Oct 2007.

[23]

A. Rabkin and R. Katz. Static extraction of program configuration options. In International Conference on Software Engineering, ICSE, pages 131–140, 2011.

Digital Library

[24]

A. Rabkin and R. H. Katz. Precomputing possible configuration error diagnoses. In Automated Software Engineering, pages 193–202, 2011.

Digital Library

[25]

S. K. Sahoo, J. Criswell, and V. Adve. An empirical study of reported bugs in server software with implications for automated bug diagnosis. In International Conference on Software Engineering, ICSE, pages 485–494, 2010.

Digital Library

[26]

C. Song, A. Porter, and J. S. Foster. iTree: eﬃciently discovering high-coverage configurations using interaction trees. In The International Conference on Software Engineering, ICSE, pages 903–913, 2012.

Digital Library

[27]

The Document Foundation. http://blog.documentfoundation.org/2011/09/28/, 2011.

[28]

C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for eﬃcient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20–34, Jan 2006.

Digital Library

[29]

Z. Yin, X. Ma, J. Zheng, Y. Zhou, L. N. Bairavasundaram, and S. Pasupathy. An empirical study on configuration errors in commercial and open source systems. In Symposium on Operating Systems Principles, SOSP, pages 159–172, 2011.

Digital Library

[30]

S. Zhang and M. D. Ernst. Automated diagnosis of software configuration errors. In International Conference on Software Engineering, ICSE, pages 312–321, 2013.

Digital Library

[31]

T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy. Characterizing and predicting which bugs get reopened. In International Conference on Software Engineering, ICSE, pages 1074–1083, 2012.

Digital Library

Cited By

Chen ZChen PWang PYu GHe ZMai GChandra SBlincoe KTonella P(2023)DiagConfig: Configuration Diagnosis of Performance Violations in Configurable Software SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616300(566-578)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616300
Bessghaier NSayagh MOuni AMkaouer M(2023)What Constitutes the Deployment and Runtime Configuration System? An Empirical Study on OpenStack ProjectsACM Transactions on Software Engineering and Methodology10.1145/360718633:1(1-37)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1145/3607186
Acher MDuarte JJézéquel J(2023)On Programming Variability with Large Language Model-based AssistantProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608972(8-14)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608972
Show More Cited By

Index Terms

Configurations everywhere: implications for testing and debugging in practice
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Safely evolving configurable systems
SPLASH Companion 2015: Companion Proceedings of the 2015 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity

Developers use configuration options to tailor systems to different platforms. This configurability leads to exponential configuration spaces and traditional tools (e.g., gcc) check only one configuration at a time. As a result, developers introduce ...
ECSTATIC: Automatic Configuration-Aware Testing and Debugging of Static Analysis Tools
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Static analyses are powerful tools that can serve as a complement to dynamic approaches such as testing. In order to ensure generality, many static analysis tools are configurable. However, these configurations can make testing and debugging more ...
Convergence debugging
AADEBUG'05: Proceedings of the sixth international symposium on Automated analysis-driven debugging

This paper proposes a new practical automatic debugging method, called Convergence Debugging, which isolates a set of test cases that converge on the internal root cause of a failure. This method involves evaluating the debugging effectiveness of a set ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

May 2014

741 pages

ISBN:9781450327688

DOI:10.1145/2591062

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
433
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen ZChen PWang PYu GHe ZMai GChandra SBlincoe KTonella P(2023)DiagConfig: Configuration Diagnosis of Performance Violations in Configurable Software SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616300(566-578)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616300
Bessghaier NSayagh MOuni AMkaouer M(2023)What Constitutes the Deployment and Runtime Configuration System? An Empirical Study on OpenStack ProjectsACM Transactions on Software Engineering and Methodology10.1145/360718633:1(1-37)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1145/3607186
Acher MDuarte JJézéquel J(2023)On Programming Variability with Large Language Model-based AssistantProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608972(8-14)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608972
Gazzillo PCohen MScholliers CSinger J(2022)Bringing Together Configuration Research: Towards a Common GroundProceedings of the 2022 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3563835.3568737(259-269)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563835.3568737
Acher MMartin HLesoil LBlouin AJézéquel JKhelladi DBarais OPereira JFelfernig AFuentes L(2022)Feature subset selection for learning huge configuration spacesProceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A10.1145/3546932.3546997(85-96)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3546932.3546997
Sayagh MHassan A(2021)ConfigMiner: Identifying the Appropriate Configuration Options for Config-Related User Questions by Mining Online ForumsIEEE Transactions on Software Engineering10.1109/TSE.2020.297399747:12(2907-2918)Online publication date: 1-Dec-2021
https://doi.org/10.1109/TSE.2020.2973997
Fischer S(2021)A Case Study on the Evolution of Configuration Options of a Highly-Configurable Software System2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER50967.2021.00079(630-635)Online publication date: Mar-2021
https://doi.org/10.1109/SANER50967.2021.00079
Le WWang YYang FWang XWang S(2021)A Survey on Tackling Software Configuration Faults2021 7th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR53171.2021.00036(11-17)Online publication date: Sep-2021
https://doi.org/10.1109/ISSSR53171.2021.00036
Zhang YHe HLegunsen OLi SDong WXu T(2021)An Evolutionary Study of Configuration Design and Implementation in Cloud SystemsProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00029(188-200)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00029
Chen QWang TLegunsen OLi SXu TDevanbu PCohen MZimmermann T(2020)Understanding and discovering software configuration dependencies in cloud and datacenter systemsProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409727(362-374)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3409727
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents