Article

Free access

Discovering, reporting, and fixing performance bugs

Authors:

Lin TanAuthors Info & Claims

MSR '13: Proceedings of the 10th Working Conference on Mining Software Repositories

Pages 237 - 246

Published: 18 May 2013 Publication History

Abstract

Software performance is critical for how users perceive the quality of software products. Performance bugs---programming errors that cause significant performance degradation---lead to poor user experience and low system throughput. Designing effective techniques to address performance bugs requires a deep understanding of how performance bugs are discovered, reported, and fixed. In this paper, we study how performance bugs are discovered, reported to developers, and fixed by developers, and compare the results with those for non-performance bugs. We study performance and non-performance bugs from three popular code bases: Eclipse JDT, Eclipse SWT, and Mozilla. First, we find little evidence that fixing performance bugs has a higher chance to introduce new functional bugs than fixing non-performance bugs, which implies that developers may not need to be overconcerned about fixing performance bugs. Second, although fixing performance bugs is about as error-prone as fixing nonperformance bugs, fixing performance bugs is more difficult than fixing non-performance bugs, indicating that developers need better tool support for fixing performance bugs and testing performance bug patches. Third, unlike many non-performance bugs, a large percentage of performance bugs are discovered through code reasoning, not through users observing the negative effects of the bugs (e.g., performance degradation) or through profiling. The result suggests that techniques to help developers reason about performance, better test oracles, and better profiling techniques are needed for discovering performance bugs.

References

[1]

Bugzilla@Mozilla, “Bugzilla keyword descriptions,” https://bugzilla.mozilla.org/describekeywords.cgi.

[2]

I. Molyneaux, The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O’Reilly Media, 2009.

Digital Library

[3]

R. E. Bryant and D. R. O’Hallaron, Computer Systems: A Programmer’s Perspective. Addison-Wesley, 2010.

Digital Library

[4]

P. Kallender, “Trend Micro will pay for PC repair costs,” 2005, http://www.pcworld.com/article/120612/article.html.

[5]

G. E. Morris, “Lessons from the Colorado benefits management system disaster,” 2004, www.ad-mktreview.com/public html/air/ai200411.html.

[6]

T. Richardson, “1901 census site still down after six months,” 2002, http://www.theregister.co.uk/2002/07/03/1901 census site still down/.

[7]

D. Mituzas, “Embarrassment,” 2009, http://dom.as/2009/06/26/embarrassment/.

[8]

Microsoft Corp., “Connect,” https://connect.microsoft.com/.

[9]

Apache Software Foundation, “Apache’s JIRA issue tracker,” https://issues.apache.org/jira/secure/Dashboard.jspa.

[10]

D. Fields and B. Karagounis, “Inside Windows 7—reliability, performance and PerfTrack,” 2009, http://channel9.msdn.com/Blogs/Charles/Inside-Windows-7-Reliability-Performance-and-PerfTrack.

[11]

S. Han, Y. Dang, S. Ge, D. Zhang, and T. Xie, “Performance debugging in the large via mining millions of stack traces,” in ICSE, 2012.

Digital Library

[12]

E. Coppa, C. Demetrescu, and I. Finocchi, “Input-sensitive profiling,” in PLDI, 2012.

Digital Library

[13]

D. Zaparanuks and M. Hauswirth, “Algorithmic profiling,” in PLDI’12.

Digital Library

[14]

J. Oh, C. J. Hughes, G. Venkataramani, and M. Prvulovic, “LIME: A framework for debugging load imbalance in multi-threaded execution,” in ICSE, 2011.

Digital Library

[15]

N. Siegmund, S. S. Kolesnikov, C. Kästner, S. Apel, D. S. Batory, M. Rosenm üller, and G. Saake, “Predicting performance via automated feature-interaction detection,” in ICSE, 2012.

Digital Library

[16]

C. E. Killian, K. Nagaraj, S. Pervez, R. Braud, J. W. Anderson, and R. Jhala, “Finding latent performance bugs in systems implementations,” in FSE, 2010.

Digital Library

[17]

G. H. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky, “Go with the flow: Profiling copies to find runtime bloat,” in PLDI, 2009.

Digital Library

[18]

D. Yan, G. Xu, and A. Rountev, “Uncovering performance problems in Java applications with reference propagation profiling,” in ICSE, 2012.

Digital Library

[19]

B. Dufour, B. G. Ryder, and G. Sevitsky, “A scalable technique for characterizing the usage of temporaries in framework-intensive Java applications,” in FSE, 2008.

Digital Library

[20]

S. Bhattacharya, M. G. Nanda, K. Gopinath, and M. Gupta, “Reuse, recycle to de-bloat software,” in ECOOP, 2011.

Digital Library

[21]

P. Zhang, S. G. Elbaum, and M. B. Dwyer, “Automatic generation of load tests,” in ASE, 2011.

Digital Library

[22]

M. Grechanik, C. Fu, and Q. Xie, “Automatically finding performance problems with feedback-directed learning software testing,” in ICSE’12.

Digital Library

[23]

J. Burnim, S. Juvekar, and K. Sen, “WISE: Automated test generation for worst-case complexity,” in ICSE, 2009.

Digital Library

[24]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu, “Understanding and detecting real-world performance bugs,” in PLDI, 2012.

Digital Library

[25]

S. Zaman, B. Adams, and A. E. Hassan, “Security versus performance bugs: A case study on Firefox,” in MSR, 2011.

Digital Library

[26]

——, “A qualitative study on performance bugs,” in MSR, 2012.

[27]

D. E. Knuth, “Structured programming with go to statements,” ACM Comput. Surv., vol. 6, no. 4, pp. 261–301, Dec. 1974.

Digital Library

[28]

G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit, “Automated atomicityviolation fixing,” in PLDI, 2011.

Digital Library

[29]

C. L. Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each,” in ICSE, 2012.

[30]

G. Jin, W. Zhang, D. Deng, B. Liblit, and S. Lu, “Automated concurrency-bug fixing,” in OSDI, 2012.

Digital Library

[31]

C. Liu, J. Yang, L. Tan, and M. Hafiz, “R2Fix: Automatically generating bug fixes from bug reports,” in ICST’13.

[32]

Sun Microsystems, “HPROF JVM profiler,” http://java.sun.com/developer/technicalArticles/Programming/HPROF.html.

[33]

Yourkit LLC, “Yourkit profiler,” http://www.yourkit.com.

[34]

T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney, “Evaluating the accuracy of Java profilers,” in PLDI, 2010.

Digital Library

[35]

J. Park, M. Kim, B. Ray, and D.-H. Bae, “An empirical study of supplementary bug fixes,” in MSR, 2012.

[36]

Z. Yin, D. Yuan, Y. Zhou, S. Pasupathy, and L. N. Bairavasundaram, “How do fixes become bugs?” in FSE, 2011.

Digital Library

[37]

P. Fonseca, C. Li, V. Singhal, and R. Rodrigues, “A study of the internal and external effects of concurrency bugs,” in DSN, 2010.

[38]

S. Lu, S. Park, E. Seo, and Y. Zhou, “Learning from mistakes: A comprehensive study on real world concurrency bug characteristics,” in ASPLOS, 2008.

Digital Library

[39]

S. K. Sahoo, J. Criswell, and V. S. Adve, “An empirical study of reported bugs in server software with implications for automated bug diagnosis,” in ICSE, 2010.

Digital Library

[40]

C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. T. Devanbu, “Fair and balanced?: Bias in bug-fix datasets,” in FSE’09.

Digital Library

[41]

J. Aranda and G. Venolia, “The secret life of bugs: Going past the errors and omissions in software repositories,” in ICSE, 2009.

Digital Library

[42]

S. Chandra and P. M. Chen, “Whither generic recovery from application faults? A fault study using open-source software,” in DSN, 2000.

Digital Library

[43]

A. Chou, J. Yang, B. Chelf, S. Hallem, and D. R. Engler, “An empirical study of operating system errors,” in SOSP, 2001.

Digital Library

[44]

E. Shihab, A. Mockus, Y. Kamei, B. Adams, and A. E. Hassan, “Highimpact defects: A study of breakage and surprise defects,” in FSE, 2011.

Digital Library

[45]

M. Asaduzzaman, M. C. Bullock, C. K. Roy, and K. A. Schneider, “Bug introducing changes: A case study with Android,” in MSR, 2012.

[46]

D. Kim, X. Wang, S. Kim, A. Zeller, S.-C. Cheung, and S. Park, “Which crashes should I fix first?: Predicting top crashes at an early stage to prioritize debugging efforts,” TSE, 2011.

Digital Library

[47]

D. C. D’Elia, C. Demetrescu, and I. Finocchi, “Mining hot calling contexts in small space,” in PLDI, 2011.

Digital Library

Cited By

Al-Bataineh OFilkov VRay BZhou M(2024)A Formal Treatment of Performance BugsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695288(2220-2224)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695288
Zhao YXiao LWong S(2024)A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports With Heuristic Linguistic PatternsIEEE Transactions on Software Engineering10.1109/TSE.2024.339062350:7(1704-1725)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3390623
Li XChen JLiu YWu KGallagher J(2023)Combatting Energy Issues for Mobile ApplicationsACM Transactions on Software Engineering and Methodology10.1145/352785132:1(1-44)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3527851
Show More Cited By

Index Terms

Discovering, reporting, and fixing performance bugs
1. Information systems
  1. Information systems applications
    1. Data mining
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Software version control
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software configuration management and version control systems

Recommendations

Classifying field crash reports for fixing bugs: A case study of Mozilla Firefox
ICSM '11: Proceedings of the 2011 27th IEEE International Conference on Software Maintenance

Many software systems support automatic collection of field crash-reports which record the stack traces and other runtime information when crashes occur. Analysis of field crash-reports can help developers to locate and fix bugs. However, the amount of ...
Are Developers Fixing Their Own Bugs?: Tracing Bug-Fixing and Bug-Seeding Committers

The process of fixing software bugs plays a key role in the maintenance activities of a software project. Ideally, code ownership and responsibility should be enforced among developers working on the same artifacts, so that those introducing buggy code ...
A qualitative study on performance bugs
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories

Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experienced developers and require ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

MSR '13: Proceedings of the 10th Working Conference on Mining Software Repositories

May 2013

438 pages

ISBN:9781467329361

General Chair:
Thomas Zimmermann,
Program Chairs:
Massimiliano Di Penta,
Sunghun Kim

Publisher

IEEE Press

Publication History

Published: 18 May 2013

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
589
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)11

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Al-Bataineh OFilkov VRay BZhou M(2024)A Formal Treatment of Performance BugsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695288(2220-2224)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695288
Zhao YXiao LWong S(2024)A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports With Heuristic Linguistic PatternsIEEE Transactions on Software Engineering10.1109/TSE.2024.339062350:7(1704-1725)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3390623
Li XChen JLiu YWu KGallagher J(2023)Combatting Energy Issues for Mobile ApplicationsACM Transactions on Software Engineering and Methodology10.1145/352785132:1(1-44)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3527851
Cao JChen BSun CHu LWu SPeng XRoychoudhury ACadar CKim M(2022)Understanding performance problems in deep learning systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549123(357-369)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549123
Iqbal MKrishna RJavidian MRay BJamshidi PBromberg YKermarrec AKozyrakis C(2022)UnicornProceedings of the Seventeenth European Conference on Computer Systems10.1145/3492321.3519575(199-217)Online publication date: 28-Mar-2022
https://dl.acm.org/doi/10.1145/3492321.3519575
Garg SMoghaddam RSundaresan NWu CDo LUrban C(2021)PerfLens: a data-driven performance bug detection and fix platformProceedings of the 10th ACM SIGPLAN International Workshop on the State Of the Art in Program Analysis10.1145/3460946.3464318(19-24)Online publication date: 22-Jun-2021
https://dl.acm.org/doi/10.1145/3460946.3464318
Velez MJamshidi PSiegmund NApel SKästner C(2021)White-Box Analysis over Machine LearningProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00100(1072-1084)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00100
Nusrat FHassan FZhong HWang X(2021)How Developers Optimize Virtual Reality ApplicationsProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00052(473-485)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00052
Li XYang YLiu YGallagher JWu KKhurshid SPăsăreanu C(2020)Detecting and diagnosing energy issues for mobile applicationsProceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3395363.3397350(115-127)Online publication date: 18-Jul-2020
https://dl.acm.org/doi/10.1145/3395363.3397350
Ghari SHadian MRasolroveicy MFokaefs M(2019)A multi-dimensional quality analysis of Android applicationsProceedings of the 29th Annual International Conference on Computer Science and Software Engineering10.5555/3370272.3370276(34-43)Online publication date: 4-Nov-2019
https://dl.acm.org/doi/10.5555/3370272.3370276
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten