research-article

Proactive detection of inadequate diagnostic messages for software configuration errors

Authors:

Michael D. ErnstAuthors Info & Claims

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

Pages 12 - 23

https://doi.org/10.1145/2771783.2771817

Published: 13 July 2015 Publication History

Abstract

This paper presents a technique to detect inadequate (i.e., missing or ambiguous) diagnostic messages for configuration errors issued by a configurable software system. The technique injects configuration errors into the software under test, monitors the software outcomes under the injected configuration errors, and uses natural language processing to analyze the output diagnostic message caused by each configuration error. The technique reports diagnostic messages that may be unhelpful in diagnosing a configuration error. We implemented the technique for Java in a tool, ConfDiagDetector. In an evaluation on 4 real-world, mature configurable systems, ConfDiagDetector reported 43 distinct inadequate diagnostic messages (25 missing and 18 ambiguous). 30 of the detected messages have been confirmed by their developers, and 12 more have been identified as inadequate by users in a user study. On average, Conf- DiagDetector required 5 minutes of programmer time and 3 minutes of compute time to detect each inadequate diagnostic message.

References

[1]

B. Ashok, J. Joy, H. Liang, S. K. Rajamani, G. Srinivasa, and V. Vangala. DebugAdvisor: A recommender system for debugging. In ESEC/FSE, 2009.

Digital Library

[2]

M. Attariyan, M. Chow, and J. Flinn. X-ray: automating root-cause diagnosis of performance anomalies in production software. In OSDI, 2012.

Digital Library

[3]

M. Attariyan and J. Flinn. Using causality to diagnose configuration bugs. In USENIX ATC, 2008.

Digital Library

[4]

M. Attariyan and J. Flinn. Automating configuration troubleshooting with dynamic information flow analysis. In OSDI, 2010.

Digital Library

[5]

R. Barrett, E. Kandogan, P. P. Maglio, E. M. Haber, L. A. Takayama, and M. Prabaker. Field studies of computer system administrators: Analysis of system management tools and practices. In CSCW, 2004.

Digital Library

[6]

D. Bollegala, Y. Matsuo, and M. Ishizuka. Measuring semantic similarity between words using web search engines. In WWW, 2007.

Digital Library

[7]

S. Bordag. A comparison of co-occurrence and similarity measures as simulations of context. In CICLing, 2008.

Digital Library

[8]

GNOME Bug Buddy. https://directory.fsf.org/wiki/Bug-buddy.

[9]

CLOC. http://cloc.sourceforge.net/.

[10]

M. B. Cohen, M. B. Dwyer, and J. Shi. Interaction testing of highly-configurable systems in the presence of constraints. In ISSTA, 2007.

Digital Library

[11]

Computer Science Glossaries. http://www.math.utah.edu/~wisnia/glossary.html.

[12]

Derby. http://db.apache.org/derby/.

[13]

Z. Dong, M. Ghanavati, and A. Andrzejak. Automated diagnosis of software misconfigurations based on static analysis. In ISSRE, 2013.

[14]

M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 69(1–3):35–45, Dec. 2007.

Digital Library

[15]

R. Johnson. More details on today’s outage. https: //www.facebook.com/notes/facebook-engineering/ more-details-on-todaysoutage/431441338919.

[16]

Volatile and Decentralized. http://matt-welsh.blogspot.com/2013/05/ what-i-wish-systems-researchers-would.html.

[17]

P. Gamallo, C. Gasperin, A. Agustini, and J. G. P. Lopes. Syntactic-based methods for measuring word similarity. In TSD, 2001.

Digital Library

[18]

B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Using feature locality: can we leverage history to avoid failures during reconfiguration? In ASAS, 2011.

Digital Library

[19]

V. Gervasi and D. Zowghi. Reasoning about inconsistencies in natural language requirements. ACM Trans. Softw. Eng. Methodol., 14(3):277–330, July 2005.

Digital Library

[20]

J. Ha, C. J. Rossbach, J. V. Davis, I. Roy, H. E. Ramadan, D. E. Porter, D. L. Chen, and E. Witchel. Improved error reporting for software that uses black-box components. In PLDI, 2007.

Digital Library

[21]

S. Hangal and M. S. Lam. Tracking down software bugs using automatic anomaly detection. In ICSE, 2002.

Digital Library

[22]

A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu. On the naturalness of software. In ICSE, 2012.

Digital Library

[23]

M. J. Howard, S. Gupta, L. Pollock, and K. Vijay-Shanker. Automatically mining software-based, semantically-similar words from comment-code mappings. In MSR, 2013.

Digital Library

[24]

A. Hubaux, P. Heymans, P. Schobbens, and D. Deridder. Towards multi-view feature-based configuration. In REFSQ, 2010.

[25]

A. Hubaux, Y. Xiong, and K. Czarnecki. A user survey of configuration challenges in Linux and eCos. In VaMoS, 2012.

Digital Library

[26]

K. Inkumsah and T. Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In ASE, 2008.

Digital Library

[27]

Jetty. http://www.eclipse.org/jetty/.

[28]

JMeter. http://jmeter.apache.org.

[29]

L. Keller, P. Upadhyaya, and G. Candea. ConfErr: A tool for assessing resilience to human configuration errors. In DSN, 2008.

[30]

N. Kushman and D. Katabi. Enabling configuration-independent automation by non-expert users. In OSDI, 2010.

Digital Library

[31]

B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In PLDI, 2005.

Digital Library

[32]

Z. Lin and X. Zhang. Deriving input syntactic structure from execution. In FSE, 2008.

Digital Library

[33]

R. Mihalcea, C. Corley, and C. Strapparava. Corpus-based and knowledge-based measures of text semantic similarity. In AAAI, 2006.

Digital Library

[34]

L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. Pollock, and K. Vijay-Shanker. Automatic generation of natural language summaries for Java classes. In ICPC, 2013.

[35]

Configuration error brings down the Azure cloud platform. http://www.evolven.com/blog/ configuration-error-brings-down-the-azure-cloud\ -platform.html.

[36]

Microsofot’s Online Crash Analysis. http://support.microsoft.com/kb/923800.

[37]

T. T. Nguyen, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. A statistical semantic language model for source code. In ESEC/FSE, 2013.

Digital Library

[38]

X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: an empirical study of sampling and prioritization. In ISSTA, 2008.

Digital Library

[39]

A. Rabkin and R. Katz. Precomputing possible configuration error diagnoses. In ASE, 2011.

Digital Library

[40]

A. Rabkin and R. Katz. Static extraction of program configuration options. In ICSE, 2011.

Digital Library

[41]

V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical language models. In PLDI, 2013.

Digital Library

[42]

Circleid, misconfiguration brings down entire .se domain in Sweden. http://www.circleid.com/posts/misconfiguration_ brings_down_entire_se_domain_in_sweden.

[43]

A. Sinha, S. M. S. Jr., and A. M. Paradkar. Text2Test: Automated inspection of natural language use cases. In ICST, 2010.

Digital Library

[44]

A. Sinha, A. M. Paradkar, P. Kumanan, and B. Boguraev. A linguistic analysis engine for natural language use case description and its application to dependability analysis in industrial use cases. In DSN, 2009.

[45]

Y. J. Song, F. Junqueira, and B. Reed. Bft for the skeptics. In Proc. ACM SOSP Work in Progress Session, 2009.

[46]

K. Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. In P. Willett, editor, Document Retrieval Systems, pages 132–142. Taylor Graham Publishing, London, UK, UK, 1988.

Digital Library

[47]

G. Sridhara, E. Hill, D. Muppaneni, L. Pollock, and K. Vijay-Shanker. Towards automatically generating summary comments for Java methods. In ASE, 2010.

Digital Library

[48]

Stanford NLP Parser. http: //nlp.stanford.edu/software/lex-parser.shtml.

[49]

Y.-Y. Su, M. Attariyan, and J. Flinn. AutoBash: improving configuration management with operating system causality analysis. In SOSP, 2007.

Digital Library

[50]

S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens. @tcomment: Testing Javadoc comments to detect comment-code inconsistencies. In ICST, 2012.

Digital Library

[51]

H. J. Wang, J. C. Platt, Y. Chen, R. Zhang, and Y.-M. Wang. Automatic misconfiguration troubleshooting with PeerPressure. In OSDI, 2004.

Digital Library

[52]

X. Wang, L. Zhang, T. Xie, Y. Xiong, and H. Mei. Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In FSE, 2012.

Digital Library

[53]

Y.-M. Wang, C. Verbowski, J. Dunagan, Y. Chen, H. J. Wang, C. Yuan, and Z. Zhang. Strider: A black-box, state-based approach to change and configuration management and support. In LISA, 2003.

Digital Library

[54]

Weka. www.cs.waikato.ac.nz/ml/weka/.

[55]

A. Whitaker, R. S. Cox, and S. D. Gribble. Configuration debugging as search: finding the needle in the haystack. In OSDI, 2004.

Digital Library

[56]

R. W. White and D. Morris. Investigating the querying and browsing behavior of advanced search engine users. In SIGIR, 2007.

Digital Library

[57]

WordNet. http://wordnet.princeton.edu/.

[58]

X. Xiao, A. Paradkar, S. Thummalapenta, and T. Xie. Automated extraction of security policies from natural-language software documents. In FSE, 2012.

Digital Library

[59]

Y. Xiong, A. Hubaux, S. She, and K. Czarnecki. Generating range fixes for software configuration. In ICSE, 2012.

Digital Library

[60]

Y. Xiong, H. Zhang, A. Hubaux, S. She, J. Wang, and K. Czarnecki. Range fixes: Interactive error resolution for software configuration. Software Engineering, IEEE Transactions on, PP(99):1–1, 2014.

[61]

T. Xu, J. Zhang, P. Huang, J. Zheng, T. Sheng, D. Yuan, Y. Zhou, and S. Pasupathy. Do not blame users for misconfigurations. In SOSP, 2013.

Digital Library

[62]

Z. Yin, X. Ma, J. Zheng, Y. Zhou, L. N. Bairavasundaram, and S. Pasupathy. An empirical study on configuration errors in commercial and open source systems. In SOSP, 2011.

Digital Library

[63]

D. Yuan, Y. Xie, R. Panigrahy, J. Yang, C. Verbowski, and A. Kumar. Context-based online configuration-error detection. In USENIX ATC, 2011.

Digital Library

[64]

D. Yuan, J. Zheng, S. Park, Y. Zhou, and S. Savage. Improving software diagnosability via log enhancement. ACM Trans. Comput. Syst., 30(1):4:1–4:28, Feb. 2012.

Digital Library

[65]

J. Zhang, L. Renganarayana, X. Z. N. Ge, V. Bala, T. Xu, and Y. Zhou. EnCore: Exploiting system environment and correlation information for misconfiguration detection. In ASPLOS, 2014.

Digital Library

[66]

S. Zhang and M. D. Ernst. Automated diagnosis of software configuration errors. In ICSE, 2013.

Digital Library

[67]

S. Zhang and M. D. Ernst. Which configuration option should I change? In ICSE, 2014.

Digital Library

[68]

W. Zhao, L. Zhang, Y. Liu, J. Sun, and F. Yang. SNIAFL: Towards a static non-interactive approach to feature location. In ICSE, 2004.

Digital Library

[69]

H. Zhong and Z. Su. Detecting API documentation errors. In OOPSLA, 2013.

Digital Library

[70]

H. Zhong, L. Zhang, T. Xie, and H. Mei. Inferring resource specifications from natural language api documentation. In ASE, 2009.

Digital Library

Cited By

Qiu YKon PBeckett RChen AWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Unearthing Semantic Checks for Cloud Infrastructure-as-Code ProgramsProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695974(574-589)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695974
Liao HGuo JHuang BHan YYang DShi KDing JXu GYang GZhang LFilkov VRay BZhou M(2024)DeployFix: Dynamic Repair of Software Deployment Failures via Constraint SolvingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695268(2053-2064)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695268
Chen YHuai YLi SHong CGarcia J(2024)Misconfiguration Software Testing for Failure Emergence in Autonomous Driving SystemsProceedings of the ACM on Software Engineering10.1145/36607921:FSE(1913-1936)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660792
Show More Cited By

Index Terms

Proactive detection of inadequate diagnostic messages for software configuration errors
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Systems Approaches to Tackling Configuration Errors: A Survey

In recent years, configuration errors (i.e., misconfigurations) have become one of the dominant causes of system failures, resulting in many severe service outages and downtime. Unfortunately, it is notoriously difficult for system users (e.g., ...
Automated diagnosis of software configuration errors
ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

The behavior of a software system often depends on how that system is configured. Small configuration errors can lead to hard-to-diagnose undesired behaviors. We present a technique (and its tool implementation, called ConfDiagnoser) to identify the ...
Detecting Inter-Component Configuration Errors in Proactive: A Relation-Aware Method
QSIC '14: Proceedings of the 2014 14th International Conference on Quality Software

Configuration errors, similar to bugs, are often the dominant causes of errors in deployed applications, leading to application failures, exceptions and even crashes. Misconfigurations are hard to detect and diagnose. To address this issue, we propose a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

July 2015

447 pages

ISBN:9781450336208

DOI:10.1145/2771783

General Chair:
Michal Young
University of Oregon, USA
,
Program Chair:
Tao Xie
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '15

Sponsor:

SIGSOFT

ISSTA '15: International Symposium on Software Testing and Analysis

July 13 - 17, 2015

MD, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
324
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)6

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qiu YKon PBeckett RChen AWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Unearthing Semantic Checks for Cloud Infrastructure-as-Code ProgramsProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695974(574-589)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695974
Liao HGuo JHuang BHan YYang DShi KDing JXu GYang GZhang LFilkov VRay BZhou M(2024)DeployFix: Dynamic Repair of Software Deployment Failures via Constraint SolvingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695268(2053-2064)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695268
Chen YHuai YLi SHong CGarcia J(2024)Misconfiguration Software Testing for Failure Emergence in Autonomous Driving SystemsProceedings of the ACM on Software Engineering10.1145/36607921:FSE(1913-1936)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660792
Shan SHuo YSu YLi YLi DZheng ZChristakis MPradel M(2024)Face It Yourselves: An LLM-Based Two-Stage Strategy to Localize Configuration Errors via LogsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652106(13-25)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652106
Li JLi SLi KLuo FYu HLi SLi XRoychoudhury APaiva AAbreu RStorey M(2024)ECFuzz: Effective Configuration Fuzzing for Large-Scale SystemsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623315(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623315
Wang TJia ZLi SZheng SYu YXu EPeng SLiao X(2023)Understanding and Detecting On-The-Fly Configuration Bugs2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00062(628-639)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00062
Wang T(2022)Troubleshooting Configuration Errors via Information Retrieval and Configuration Testing2022 4th International Academic Exchange Conference on Science and Technology Innovation (IAECST)10.1109/IAECST57965.2022.10062229(422-426)Online publication date: 9-Dec-2022
https://doi.org/10.1109/IAECST57965.2022.10062229
Tao YChen ZLiu YXuan JXu ZQin SSpinellis DGousios GChechik MDi Penta M(2021)Demystifying “bad” error messages in data science librariesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468560(818-829)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468560
Li WJia ZLi SZhang YWang TXu EWang JLiao XCadar CZhang X(2021)Challenges and opportunities: an in-depth empirical study on configuration error injection testingProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464799(478-490)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464799
Le WWang YYang FWang XWang S(2021)A Survey on Tackling Software Configuration Faults2021 7th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR53171.2021.00036(11-17)Online publication date: Sep-2021
https://doi.org/10.1109/ISSSR53171.2021.00036
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents