Article

/icomment: bugs or bad comments?/

Authors:

Yuanyuan ZhouAuthors Info & Claims

SOSP '07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles

Pages 145 - 158

https://doi.org/10.1145/1294261.1294276

Published: 14 October 2007 Publication History

Abstract

Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information regarding a program's semantic behavior. As software evolves, they can easily grow out-of-sync, indicating two problems: (1) bugs -the source code does not follow the assumptions and requirements specified by correct program comments; (2) bad comments - comments that are inconsistent with correct code, which can confuse and mislead programmers to introduce bugs in subsequent versions. Unfortunately, as most comments are written in natural language, no solution has been proposed to automatically analyze commentsand detect inconsistencies between comments and source code. This paper takes the first step in automatically analyzing commentswritten in natural language to extract implicit program rulesand use these rules to automatically detect inconsistencies between comments and source code, indicating either bugs or bad comments. Our solution, iComment, combines Natural Language Processing(NLP), Machine Learning, Statistics and Program Analysis techniques to achieve these goals. We evaluate iComment on four large code bases: Linux, Mozilla, Wine and Apache. Our experimental results show that iComment automatically extracts 1832 rules from comments with 90.8-100% accuracy and detects 60 comment-code inconsistencies, 33 newbugs and 27 bad comments, in the latest versions of the four programs. Nineteen of them (12 bugs and 7 bad comments) have already been confirmed by the corresponding developers while the others are currently being analyzed by the developers.

Supplementary Material

JPG File (1294276.jpg)

Download
14.07 KB

index.html (index.html)

Slides from the presentation

Download
.93 KB

ZIP File (p145-slides.zip)

Supplemental material for /*icomment: bugs or bad comments?*/

Download
21.57 MB

Audio only (1294276.mp3)

Download
11.93 MB

Video (1294276.mp4)

Download
164.70 MB

References

[1]

C# XML comments let you build documentation directly from your Visual Studio .NET source files. http://msdn.microsoft.com/msdnmag/issues/02/06/XMLC/.

[2]

CoNLL-2000 shared task web page -- with data, software and systems' outputs availble. http://www.cnts.ua.ac.be/conll/.

[3]

Doxygen -- source code documentation generator tool. http://www.stack.nl/ dimitri/doxygen/.

[4]

FreeBSD problem report database. http://www.freebsd.org/support/bugreports.html.

[5]

Java annotations. http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html.

[6]

Javadoc tool. http://java.sun.com/j2se/javadoc/.

[7]

Lock_Lint -- Static data race and deadlock detection tool for C. http://developers.sun.com/sunstudio/articles/locklint.html.

[8]

MSDN run-time library reference -- SAL annotations. http://msdn2.microsoft.com/en--us/library/ms235402.aspx.

[9]

NLP tools. http://l2r.cs.uiuc.edu/~cogcomp/tools.php.

[10]

RDoc -- documentation from Ruby source files. http://rdoc.sourceforge.net/.

[11]

Sparse -- A semantic parser for C. http://www.kernel.org/pub/software/devel/sparse/.

[12]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003.

Digital Library

[13]

J.-D. Choi, M. Burke, and P. Carini. Efficient flow-sensitive interprocedural computation of pointer-induced aliases and side effects. In Proceedings of the 20th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1993.

Digital Library

[14]

D. L. Detlefs, K. R. M. Leino, G. Nelson, and J. B. Saxe. Extended static checking, SRC research report 159. ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/researchreports/SRC-159.ps.

[15]

D. R. Engler and K. Ashcraft. RacerX: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003.

Digital Library

[16]

D. R. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, 2001.

Digital Library

[17]

M. D. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In Proceedings of the 22nd International Conference on Software Engineering, 2000.

Digital Library

[18]

D. Evans and D. Larochelle. Improving security using extensible lightweight static analysis. IEEE Software, 2002.

Digital Library

[19]

Y. Even-Zohar and D. Roth. A sequential model for multi class classification. In Proceedings of the Conference on Empirical Methods for Natural Language Processing, 2001.

[20]

S. Hallem, B. Chelf, Y. Xie, and D. R. Engler. A system and language for building system-specific, static analyses. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation.

Digital Library

[21]

W. E. Howden. Comments analysis and programming errors. IEEE Transactions on Software Engineering, 1990.

Digital Library

[22]

Z. M. Jiang and A. E. Hassan. Examining the evolution of code comments in PostgreSQL. In Proceedings of the 2006 International Workshop on Mining Software Repositories.

Digital Library

[23]

S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX Annual Technical Conference, 2005.

Digital Library

[24]

T. Kremenek, P. Twohey, G. Back, A. Y. Ng, and D. R. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th USENIX Symposium on Operating System Design and Implementation, 2006.

Digital Library

[25]

T. Li, C. Ellis, A. Lebeck, and D. Sorin. On-demand and semantic-free dynamic deadlock detection with speculative execution. In USENIX Annual Technical Conference, 2005.

Digital Library

[26]

Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2005.

Digital Library

[27]

C. D. Manning and H. Schütze. Foundations Of Statistical Natural Language Processing. The MIT Press, 2001.

Digital Library

[28]

T. Mitchell. Machine Learning. McGraw Hill, 1997.

Digital Library

[29]

M. Musuvathi, D. Y. W. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: A pragmatic approach to model checking real code. In Proceedingts of the 5th Symposium on Operating Systems Design and Implementation, 2002.

Digital Library

[30]

A. C. Myers and B. Liskov. Protecting privacy using the decentralized label model. ACM Trans. Softw. Eng. Methodol., 2000.

Digital Library

[31]

S. E. Perl and W. E. Weihl. Performance assertion checking. In Proceedings of the 14th ACM Symposium on Operating Systems Principles, 1993.

Digital Library

[32]

V. Punyakanok and D. Roth. The use of classifiers in sequential inference. In Proceedings of the Conference on Advances in Neural Information Processing Systems, 2001.

[33]

V. Punyakanok, D. Roth, and W. Yih. The necessity of syntactic parsing for semantic role labeling. In Proceedings of the International Joint Conference on Artificial Intelligence, 2005.

Digital Library

[34]

R. J. Quilan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.

Digital Library

[35]

A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 1996.

[36]

K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. ESC/Java user's manual, SRC technical note 2000-002. http://gatekeeper.dec.com/pub/DEC/SRC/technicalnotes/abstracts/src-tn-2000-002.html.

[37]

S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 1997.

Digital Library

[38]

B. Steensgaard. Points-to analysis in almost linear time. In Proceedings of the 23rd Annual ACM SIGPLAN--SIGACT Symposium on Principles of Programming Languages, 1996.

Digital Library

[39]

N. Sterling. WARLOCK -- A static data race analysis tool. In USENIX Winter Technical Conference, pages 97--106, 1993.

[40]

S. Teufel and M. Moens. Summarizing scientific articles -- experiments with relevance and rhetorical status. Computational Linguistics, 2002.

Digital Library

[41]

I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques (2nd Ed.). Morgan Kaufmann, 2005.

Digital Library

[42]

S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. The effect of modularization and comments on program comprehension. In Proceedings of the 5th International Conference on Software Engineering, 1981.

Digital Library

[43]

A. Yaar, A. Perrig, and D. X. Song. Pi: A path identification mechanism to defend against DDoS attack. In IEEE Symposium on Security and Privacy, 2003.

Digital Library

[44]

A. T. T. Ying, J. L. Wright, and S. Abrams. Source code that talks: An exploration of eclipse task comments and their implication to repository mining. In Proceedings of the 2005 International Workshop on Mining Software Repositories.

Digital Library

[45]

C. Zhai, A. Velivelli, and B. Yu. A cross-collection mixture model for comparative text mining. In Proceedings of the 2004 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining.

Digital Library

[46]

F. Zhou, J. Condit, Z. Anderson, I. Bagrak, R. Ennals, M. Harren, G. Necula, and E. Brewer. SafeDrive: Safe and recoverable extensions using language-based techniques. In Proceedings of the 7th Symposium on Operating System Design and Implementation, 2006.

Digital Library

Cited By

Zhang YLiu ZFeng YXu BFilkov VRay BZhou M(2024)Leveraging Large Language Model to Assist Detecting Rust Code Comment InconsistencyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695010(356-366)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695010
Wang HGao ZBi TGrundy JWang XWu MYang X(2024)What Makes a Good TODO Comment?ACM Transactions on Software Engineering and Methodology10.1145/366481133:6(1-30)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664811
Mai YGao ZHu XBao LLiu YSun J(2024)Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context LearningProceedings of the ACM on Software Engineering10.1145/36608111:FSE(2355-2377)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660811
Show More Cited By

Index Terms

/*icomment: bugs or bad comments?*/
1. General and reference
  1. Cross-computing tools and techniques
    1. Reliability
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Documentation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

/*icomment: bugs or bad comments?*/
SOSP '07

Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information ...
The Secret Life of Commented-Out Source Code
ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

Source code commenting is a common practice to improve code comprehension in software development. While comments often consist of descriptive natural language, surprisingly, there exists a non-trivial portion of comments that are actually code ...
AutoComment: mining question and answer sites for automatic comment generation
ASE '13: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering

Code comments improve software maintainability. To address the comment scarcity issue, we propose a new automatic comment generation approach, which mines comments from a large programming Question and Answer (Q&A) site. Q&A sites allow programmers to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SOSP '07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles

October 2007

378 pages

ISBN:9781595935915

DOI:10.1145/1294261

General Chair:
Thomas C. Bressoud
Denison University, USA
,
Program Chair:
M. Frans Kaashoek
Massachusetts Institute of Technology, USA

ACM SIGOPS Operating Systems Review Volume 41, Issue 6
SOSP '07
December 2007
363 pages
ISSN:0163-5980
DOI:10.1145/1323293
Issue’s Table of Contents

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SOSP07

Sponsor:

SOSP07: ACM SIGOPS 21st Symposium on Operating Systems Principles 2007

October 14 - 17, 2007

Washington, Stevenson, USA

Acceptance Rates

Overall Acceptance Rate 131 of 716 submissions, 18%

Upcoming Conference

SOSP '25

Sponsor:
sigops

ACM SIGOPS 31st Symposium on Operating Systems Principles

October 13 - 16, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

228
Total Citations
View Citations
1,846
Total Downloads

Downloads (Last 12 months)125
Downloads (Last 6 weeks)17

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YLiu ZFeng YXu BFilkov VRay BZhou M(2024)Leveraging Large Language Model to Assist Detecting Rust Code Comment InconsistencyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695010(356-366)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695010
Wang HGao ZBi TGrundy JWang XWu MYang X(2024)What Makes a Good TODO Comment?ACM Transactions on Software Engineering and Methodology10.1145/366481133:6(1-30)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664811
Mai YGao ZHu XBao LLiu YSun J(2024)Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context LearningProceedings of the ACM on Software Engineering10.1145/36608111:FSE(2355-2377)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660811
Endres MFakhoury SChakraborty SLahiri S(2024)Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?Proceedings of the ACM on Software Engineering10.1145/36607911:FSE(1889-1912)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660791
Gong YNie JYou WShi WHuang JLiang BZhang JBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)SICode: Embedding-Based Subgraph Isomorphism Identification for Bug DetectionProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3646556(304-315)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3646556
Yang HChen HKuai ZTu SKuang LBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)ASKDetector: An AST-Semantic and Key Features Fusion based Code Comment Mismatch DetectorProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644405(392-402)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644405
OBrien DBiswas SImtiaz SAbdalkareem RShihab ERajan HRoychoudhury APaiva AAbreu RStorey M(2024)Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub CopilotProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639176(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639176
Mastropaolo ACiniselli MDi Penta MBavota GRoychoudhury APaiva AAbreu RStorey M(2024)Evaluating Code Summarization Techniques: A New Metric and an Empirical CharacterizationProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639174(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639174
Xu ZGuo SWang YChen RLi HLi XJiang H(2024)Code Comment Inconsistency Detection Based on Confidence LearningIEEE Transactions on Software Engineering10.1109/TSE.2024.335848950:3(598-617)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3358489
Wu JHe HGao KXiao WLi JZhou M(2024)A comprehensive analysis of challenges and strategies for software release notes on GitHubEmpirical Software Engineering10.1007/s10664-024-10486-029:5Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10486-0
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents