research-article

Automated Support for Reproducing and Debugging Field Failures

Authors:

Alessandro OrsoAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 24, Issue 4

Article No.: 26, Pages 1 - 35

https://doi.org/10.1145/2774218

Published: 02 September 2015 Publication History

Abstract

As confirmed by a recent survey conducted among developers of the Apache, Eclipse, and Mozilla projects, two extremely challenging tasks during maintenance are reproducing and debugging field failures—failures that occur on user machines after release. To help developers with these tasks, in this article we present an overall approach that comprises two different techniques: BugRedux and F³. BugRedux is a general technique for reproducing field failures that collects dynamic data about failing executions in the field and uses this data to synthesize executions that mimic the observed field failures. F³ leverages the executions generated by BugRedux to perform automated debugging using a set of suitably optimized fault-localization techniques. To assess the usefulness of our approach, we performed an empirical evaluation of the approach on a set of real-world programs and field failures. The results of our evaluation are promising in that, for all the failures considered, our approach was able to (1) synthesize failing executions that mimicked the observed field failures, (2) synthesize passing executions similar to the failing ones, and (3) use the synthesized executions to successfully perform fault localization with accurate results.

References

[1]

Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. 2006. An evaluation of similarity coefficients for software fault localization. In Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing. 39--46.

Digital Library

[2]

Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. 2008. An observation-based model for fault localization. In Proceedings of the International Workshop on Dynamic Analysis. 64--70.

Digital Library

[3]

Alfred V. Aho, Monica S. Lam, Ravi Sethi, and Jeffrey D. Ullman. 2006. Compilers: Principles, Techniques, and Tools, 2nd Ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA.

Digital Library

[4]

Shay Artzi, Julian Dolby, Frank Tip, and Marco Pistoia. 2010. Directed test generation for effective fault localization. In Proceedings of the 19th International Symposium on Software Testing and Analysis. 49--60.

Digital Library

[5]

S. Artzi, J. Dolby, F. Tip, and M. Pistoia. 2012. Fault localization for dynamic Web applications. IEEE Trans. Softw. Eng. 38, 2, 314--335.

Digital Library

[6]

Shay Artzi, Sunghun Kim, and Michael D. Ernst. 2008. ReCrash: Making software failures reproducible by preserving object states. In Proceedings of the 22nd European Conference on Object-Oriented Programming. 542--565.

Digital Library

[7]

B. Ashok, Joseph Joy, Hongkang Liang, Sriram K. Rajamani, Gopal Srinivasa, and Vipindeep Vangala. 2009. DebugAdvisor: A recommender system for debugging. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. 373--382.

Digital Library

[8]

Thanassis Avgerinos, Sang Kil Cha, Brent Lim Tze Hao, and David Brumley. 2011. AEG: Automatic Exploit Generation. In Proceedings of the 18th Network and Distributed System Security Symposium.

[9]

George K. Baah, Andy Podgurski, and Mary Jean Harrold. 2011. Mitigating the confounding effects of program dependences for effective fault localization. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. 146--156.

Digital Library

[10]

James Bailey and Peter J. Stuckey. 2005. Discovery of minimal unsatisfiable subsets of constraints using hitting set dualization. In Proceedings of the 7th International Conference on Practical Aspects of Declarative Languages. 174--186.

Digital Library

[11]

Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. 209--224.

Digital Library

[12]

Satish Chandra, Stephen J. Fink, and Manu Sridharan. 2009. Snugglebug: A powerful approach to weakest preconditions. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 363--374.

Digital Library

[13]

Trishul M. Chilimbi, Ben Liblit, Krishna Mehra, Aditya V. Nori, and Kapil Vaswani. 2009. HOLMES: Effective statistical debugging via efficient path profiling. In Proceedings of the 31st International Conference on Software Engineering. 34--44.

Digital Library

[14]

Juergen Christ, Evren Ermis, Martin Schäf, and Thomas Wies. 2013. Flow-sensitive fault localization. In Proceedings of the 14th International Conference on Verification, Model Checking, and Abstract Interpretation.

Digital Library

[15]

James Clause and Alessandro Orso. 2007. A technique for enabling and supporting debugging of field failures. In Proceedings of the 29th International Conference on Software Engineering. 261--270.

Digital Library

[16]

James Clause and Alessandro Orso. 2009. PENUMBRA: Automatically identifying failure-relevant inputs using dynamic tainting. In Proceedings of the International Symposium on Software Testing and Analysis. 249--260.

Digital Library

[17]

Holger Cleve and Andreas Zeller. 2005. Locating causes of program failures. In Proceedings of the 27th International Conference on Software Engineering. 342--351.

Digital Library

[18]

Olivier Crameri, Ricardo Bianchini, and Willy Zwaenepoel. 2011. Striking a new balance between program instrumentation and debugging time. In Proceedings of the 6th European Conference on Computer Systems. 199--214.

Digital Library

[19]

Edsger Wybe Dijkstra. 1997. A Discipline of Programming, 1st Ed. Prentice Hall PTR, Upper Saddle River, NJ.

Digital Library

[20]

Sebastian Elbaum and Madeline Diep. 2005. Profiling deployed software: Assessing strategies and testing opportunities. IEEE Trans. Softw. Eng. 31, 4, 312--327.

Digital Library

[21]

W. Eric Wong, Vidroha Debroy, and Byoungju Choi. 2010. A family of code coverage-based heuristics for effective fault localization. J. Syst. Softw. 83, 2, 188--208.

Digital Library

[22]

Evren Ermis, Martin Schäf, and Thomas Wies. 2012. Error invariants. In Proceedings of the 18th International Symposium on Formal Methods. 187--201.

[23]

Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. 2002. Extended static checking for Java. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 234--245.

Digital Library

[24]

GNU. 2005. Findutils -GNU find utilities. http://www.gnu.org/software/findutils/.

[25]

P. Godefroid, N. Klarlund, and K. Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 213--223.

Digital Library

[26]

Patrice Godefroid, Michael Y. Levin, and David A. Molnar. 2008. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium.

[27]

Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. RERAN: Timing- and touch-sensitive record and replay for Android. In Proceedings of the International Conference on Software Engineering. 72--81.

Digital Library

[28]

Google 2012. Breakpad. http://code.google.com/p/google-breakpad/.

[29]

Alex Groce. 2004. Error explanation with distance metrics. In Proceedings of the Conference on Tools and Algorithms for the Construction and Analysis of Systems. 108--122.

[30]

David M. Hilbert and David F. Redmiles. 2000. Extracting usability information from user interface events. Comput. Surv. 32, 4, 384--421.

Digital Library

[31]

John H. Holland. 1992. Adaptation in Natural and Artificial Systems. MIT Press, Cambridge, MA.

Digital Library

[32]

Dennis Jeffrey, Neelam Gupta, and Rajiv Gupta. 2008. Fault localization using value replacement. In Proceedings of the International Symposium on Software Testing and Analysis. 167--178.

Digital Library

[33]

Lingxiao Jiang and Zhendong Su. 2007. Context-aware statistical debugging: From bug predictors to faulty control flow paths. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering. 184--193.

Digital Library

[34]

Wei Jin and Alessandro Orso. 2012. BugRedux: Reproducing field failures for in-house debugging. In Proceedings of the 34th International Conference on Software Engineering. 474--484.

Digital Library

[35]

Wei Jin and Alessandro Orso. 2013. F3: Fault localization for field failures. In Proceedings of the International Symposium on Software Testing and Analysis. 213--223.

Digital Library

[36]

James A. Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering. 467--477.

Digital Library

[37]

Manu Jose and Rupak Majumdar. 2011. Cause clue clauses: Error localization using maximum satisfiability. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 437--446.

Digital Library

[38]

Fitsum Meshesha Kifetew, Wei Jin, Roberto Tiella, Alessandro Orso, and Paolo Tonella. 2013. SBFR: A search based approach for reproducing failures of programs with grammar based input. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering. 604--609.

Digital Library

[39]

Fitsum Meshesha Kifetew, Wei Jin, Roberto Tiella, Alessandro Orso, and Paolo Tonella. 2014. Reproducing field failures for programs with complex grammar based input. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation. 213--223.

Digital Library

[40]

James C. King. 1976. Symbolic execution and program testing. Commun. ACM 19, 7, 385--394.

Digital Library

[41]

Benjamin Liblit. 2004. Cooperative bug isolation. Ph.D. Dissertation, University of California, Berkeley.

Digital Library

[42]

Ben Liblit, Alexander Aiken, Alice X. Zheng, and Michael I. Jordan. 2003. Bug isolation via remote program sampling. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 141--154.

Digital Library

[43]

Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan. 2005. Scalable statistical bug isolation. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 15--26.

Digital Library

[44]

Chao Liu, Xifeng Yan, Long Fei, Jiawei Han, and Samuel P. Midkiff. 2005. SOBER: Statistical model-based bug localization. In Proceedings of the 10th European Software Engineering Conference and 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering. 286--295.

Digital Library

[45]

Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, and Yuanyuan Zhou. 2005. BugBench: Benchmarks for evaluating bug detection tools. In Proceedings of the Workshop on the Evaluation of Software Defect Detection Tools.

[46]

Mac Developer Library 2012. Technical note TN2123: CrashReporter. http://developer.apple.com/technotes/tn2004/tn2123.html.

[47]

Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. 2011. A Model for spectra-based software diagnosis. ACM Trans. Softw. Engin. Methodol. 20, 3, 1--32.

Digital Library

[48]

Mangala Gowri Nanda and Saurabh Sinha. 2009. Accurate interprocedural null-dereference analysis for Java. In Proceedings of the 31st International Conference on Software Engineering. 133--143.

Digital Library

[49]

A Ochiai. 1957. Zoogeographic studies on the soleoid fishes found in Japan and its neighbouring regions. Bull. Japanese Soc. Sci. Fisheries 22, 9, 526--530.

[50]

Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed random testing for Java. In Proceedings of the 22nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications. 815--816.

Digital Library

[51]

Carlos Pacheco, Shuvendu K. Lahiri, and Thomas Ball. 2008. Finding errors in .NET with feedback-directed random testing. In Proceedings of the International Symposium on Software Testing and Analysis. 87--96.

Digital Library

[52]

Soyeon Park, Yuanyuan Zhou, Weiwei Xiong, Zuoning Yin, Rini Kaushik, Kyu H. Lee, and Shan Lu. 2009. PRES: Probabilistic Replay with Execution Sketching on Multiprocessors. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. 177--192.

Digital Library

[53]

Chris Parnin and Alessandro Orso. 2011. Are automated debugging techniques actually helping programmers? In Proceedings of the International Symposium on Software Testing and Analysis. 199--209.

Digital Library

[54]

Christina Pavlopoulou and Michal Young. 1999. Residual test coverage monitoring. In Proceedings of the 21st International Conference on Software Engineering. 277--284.

Digital Library

[55]

J. L. Rodgers and W. A. Nicewander. 1988. Thirteen ways to look at the correlation coefficient.Amer. Statis. 42, 59--66.

[56]

Jeremias Roessler, Gordon Fraser, Andreas Zeller, and Alessandro Orso. 2012. Isolating failure causes through test case generation. In Proceedings of the International Symposium on Software Testing and Analysis.

Digital Library

[57]

Jeremias Roessler, Andreas Zeller, Gordon Fraser, Cristian Zamfir, and George Candea. 2013. Reconstructing core dumps. In Proceedings of the 6th International Conference on Software Testing. 114--123.

Digital Library

[58]

Raul Santelices, James A. Jones, Yanbing Yu, and Mary Jean Harrold. 2009. Lightweight fault-localization using multiple coverage types. In Proceedings of the 31st International Conference on Software Engineering. 56--66.

Digital Library

[59]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A concolic unit testing engine for C. In Proceedings of the 10th European Software Engineering Conference and 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering. 263--272.

Digital Library

[60]

SIR project 2012. Software-artifact Infrastructure Repository. http://sir.unl.edu/.

[61]

William N. Sumner, Tao Bao, and Xiangyu Zhang. 2011. Selecting Peers for Execution Comparison. In Proceedings of the International Symposium on Software Testing and Analysis. 309--319.

Digital Library

[62]

Willem Visser, Corina S. Păsăreanu, and Sarfraz Khurshid. 2004. Test input generation with Java PathFinder. SIGSOFT Softw. Eng. Notes 29, 4, 97--107.

Digital Library

[63]

VMware 2012. The Amazing VM Record/Replay Feature in VMware Workstation 6. http://communities. vmware.com/community/vmtn/cto/steve/blog/2007/04/18/the-amazing-vm-recordreplay-feature-in-vmw- are-workstation-6.

[64]

Dasarath Weeratunge, Xiangyu Zhang, and Suresh Jagannathan. 2010. Analyzing multicore dumps to facilitate concurrency bug reproduction. In Proceedings of the 15th Edition of ASPLOS on Architectural Support for Programming Languages and Operating Systems. 155--166.

Digital Library

[65]

Windows 2012. Windows error reporting: Getting started. http://www.microsoft.com/whdc/maintain/StartWER.mspx.

[66]

Xiaoyuan Xie, Tsong Yueh Chen, Fei-Ching Kuo, and Baowen Xu. 2013. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Trans. Softw. Engin. Methodol. 22, 4, 1--40.

Digital Library

[67]

Shin Yoo and Mark Harman. 2012. Test data regeneration: Generating new test data from existing test data. J. Softw. Test. Verif. Reliabil. 22, 3, 171--201.

Digital Library

[68]

Shin Yoo, Mark Harman, and David Clark. 2013. Fault localization prioritization: Comparing information-theoretic and coverage-based approaches. ACM Trans. Softw. Engin. Methodol. 22, 3.

Digital Library

[69]

Ding Yuan, Haohui Mai, Weiwei Xiong, Lin Tan, Yuanyuan Zhou, and Shankar Pasupathy. 2010. SherLog: Error diagnosis by connecting clues from run-time logs. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems. 143--154.

Digital Library

[70]

Ding Yuan, Jing Zheng, Soyeon Park, Yuanyuan Zhou, and Stefan Savage. 2011. Improving software diagnosability via log enhancement. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems. 3--14.

Digital Library

[71]

Cristian Zamfir and George Candea. 2010. Execution synthesis: A technique for automated software debugging. In Proceedings of the 5th European Conference on Computer Systems. 321--334.

Digital Library

[72]

Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? In Proceedings of the 7th European Software Engineering Conference Held Jointly With the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 253--267.

Digital Library

[73]

Andreas Zeller. 2002. Isolating cause-effect chains from computer programs. In Proceedings of the 10th ACM SIGSOFT Symposium on Foundations of Software Engineering. 1--10.

Digital Library

[74]

Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Trans. Softw. Eng. 28, 2, 183--200.

Digital Library

[75]

Xiangyu Zhang, Neelam Gupta, and Rajiv Gupta. 2006. Locating faults through automated predicate switching. In Proceedings of the 28th International Conference on Software Engineering. 272--281.

Digital Library

[76]

Zhenyu Zhang, W. K. Chan, T. H. Tse, Bo Jiang, and Xinming Wang. 2009. Capturing propagation of infected program states. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. 43--52.

Digital Library

[77]

Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schröter, and Cathrin Weiss. 2010. What makes a good bug report? IEEE Trans. Softw. Eng. 36, 5, 618--643.

Digital Library

Cited By

Fazzini MMoran KBernal-Cárdenas CWendland TOrso APoshyvanyk D(2023)Enhancing Mobile App Bug Reporting via Real-Time Understanding of Reproduction StepsIEEE Transactions on Software Engineering10.1109/TSE.2022.317402849:3(1246-1272)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3174028
He QXie XWang YYe DChen FJin HYang Y(2017)Localizing Runtime Anomalies in Service-Oriented SystemsIEEE Transactions on Services Computing10.1109/TSC.2016.259346210:1(94-106)Online publication date: 1-Jan-2017
https://doi.org/10.1109/TSC.2016.2593462
Tang CChan WYu YZhang Z(2017)Accuracy Graphs of Spectrum-Based Fault Localization FormulasIEEE Transactions on Reliability10.1109/TR.2017.268848766:2(403-424)Online publication date: Jun-2017
https://doi.org/10.1109/TR.2017.2688487
Show More Cited By

Index Terms

Automated Support for Reproducing and Debugging Field Failures
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

F3: fault localization for field failures
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Reproducing and debugging field failures--failures that occur on user machines after release--are challenging tasks for developers. To help the first task, in previous work we have proposed BugRedux, a technique for reproducing, in-house, failures ...
Reproducing and debugging field failures in house
ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

As confirmed by a recent survey among developers of the Apache, Eclipse, and Mozilla projects, failures of the software that occur in the field, after deployment, are difficult to reproduce and investigate in house. To address this problem, we propose ...
BugRedux: reproducing field failures for in-house debugging
ICSE '12: Proceedings of the 34th International Conference on Software Engineering

A recent survey conducted among developers of the Apache, Eclipse, and Mozilla projects showed that the ability to recreate field failures is considered of fundamental importance when investigating bug reports. Unfortunately, the information typically ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 24, Issue 4

Special Issue on ISSTA 2013

August 2015

177 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2820114

Editor:
David S. Rosenblum
National University of Singapore, Singapore

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 September 2015

Accepted: 01 May 2015

Revised: 01 December 2014

Received: 01 January 2014

Published in TOSEM Volume 24, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSF
Google, IBM Research, and Microsoft Research

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
502
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)10

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fazzini MMoran KBernal-Cárdenas CWendland TOrso APoshyvanyk D(2023)Enhancing Mobile App Bug Reporting via Real-Time Understanding of Reproduction StepsIEEE Transactions on Software Engineering10.1109/TSE.2022.317402849:3(1246-1272)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3174028
He QXie XWang YYe DChen FJin HYang Y(2017)Localizing Runtime Anomalies in Service-Oriented SystemsIEEE Transactions on Services Computing10.1109/TSC.2016.259346210:1(94-106)Online publication date: 1-Jan-2017
https://doi.org/10.1109/TSC.2016.2593462
Tang CChan WYu YZhang Z(2017)Accuracy Graphs of Spectrum-Based Fault Localization FormulasIEEE Transactions on Reliability10.1109/TR.2017.268848766:2(403-424)Online publication date: Jun-2017
https://doi.org/10.1109/TR.2017.2688487
Kochhar PXia XLo DLi SZeller ARoychoudhury A(2016)Practitioners' expectations on automated fault localizationProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931051(165-176)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931051

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents