Article

Dynamic instrumentation of production systems

Authors:

Bryan M. Cantrill,

Michael W. Shapiro,

Adam H. LeventhalAuthors Info & Claims

ATEC '04: Proceedings of the annual conference on USENIX Annual Technical Conference

Page 2

Published: 27 June 2004 Publication History

Abstract

This paper presents DTrace, a new facility for dynamic instrumentation of production systems. DTrace features the ability to dynamically instrument both user-level and kernel-level software in a unified and absolutely safe fashion. When not explicitly enabled, DTrace has zero. probe effect--the system operates exactly as if DTrace were not present at all. DTrace allows for many tens of thousands of instrumentation points, with even the smallest of systems offering on the order of 30,000 such points in the kernel alone. We have developed a C-like high-level control language to describe the predicates and actions at a given point of instrumentation. The language features user-defined variables, including thread-local variables and associative arrays. To eliminate the need for most postprocessing, the facility features a scalable mechanism for aggregating data and a mechanism for speculative tracing. DTrace has been integrated into the Solaris operating system and has been used to find serious systemic performance problems on production systems-problems that could not be found using pre-existing facilities.

References

[1]

{1} Mikhail Auguston, Clinton Jeffery, and Scott Underwood. A monitoring language for run time and post-mortem behavior analysis and visualization. In 5th International Workshop on Automated and Algorithmic Debugging, Ghent, Belgium, 2003.]]

[2]

{2} Brian Bershad, Stefan Savage, Przemyslaw Pardyak, Emin Gun Sirer, David Becker, Marc Fiuczynski, Craig Chambers, and Susan Eggers. Extensibility, safety and performance in the SPIN operating system. In Proceedings of the 15th ACM Symposium on Operating System Principles, 1995.]]

[3]

{3} R. Hastings and B. Joyce. Purify: Fast detection of memory leaks and access errors. In Proceedings of the Winter USENIX Conference, 1992.]]

[4]

{4} Jeffrey K. Hollingsworth, Barton P. Miller, Marcelo J. R. Gonçalves, Oscar Naim, Zhichen Xu, and Ling Zheng. MDL: A language and compiler for dynamic program instrumentation. In Proceedings of the 1997 International Conference on Parallel Architectures and Compilation Techniques, November 1997.]]

[5]

{5} Eric F. Johnson and Kevin Reichard. Professional Graphics Programming in the X Window System. MIS Press, Portland, OR, 1993.]]

[6]

{6} Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kirsten, Jeffrey Palm, and William G. Griswold. An overview of AspectJ. In Proceedings of the 15th European Conference on Object-Oriented Programming, 2001.]]

[7]

{7} Barton P. Miller, 2003. Personal communication.]]

[8]

{8} Barton P. Miller, Mark D. Callaghan, Jonathan M. Cargille, Jeffrey K. Hollingsworth, R. Bruce Irvin, Karen L. Karavanic, Krishna Kunchithapadam, and Tia Newhall. The Paradyn parallel performance measurement tool. IEEE Computer, 28(11):37-46, 1995.]]

[9]

{9} Richard J. Moore. A universal dynamic trace for Linux and other operating systems. In Proceedings of the FREENIX Track, June 2001.]]

[10]

{10} M. I. Seltzer, Y. Endo, C. Small, and K. A. Smith. Dealing with disaster: Surviving misbehaved kernel extensions. In Proceedings of the Second Symposium on Operating Systems Design and Implementation, 1996.]]

[11]

{11} Amitabh Srivastava and Alan Eustace. ATOM: A system for building customized program analysis tools. In Proceedings of the ACM Symposium on Programming Languages Design and Implementation, 1994.]]

[12]

{12} Sun Microsystems, Santa Clara, California. Solaris Dynamic Tracing Guide, 2004.]]

[13]

{13} Ariel Tamches and Barton P. Miller. Fine-grained dynamic instrumentation of commodity operating system kernels. In Proceedings of the Third Symposium on Operating Systems Design and Implementation, 1999.]]

[14]

{14} Robert W. Wisniewski and Bryan Rosenburg. Efficient, unified, and scalable performance monitoring for multiprocessor operating systems. In SC'2003 Conference CD, 2003.]]

[15]

{15} Zhichen Xu, Barton P. Miller, and Oscar Naim. Dynamic instrumentation of threaded applications. In Proceedings of the 7th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 1999.]]

[16]

{16} Karim Yaghmour and Michel R. Dagenais. Measuring and characterizing system behavior using kernel-level event logging. In Proceedings of the 2000 USENIX Annual Technical Conference, 2000.]]

[17]

{17} Tom Zanussi, Karim Yaghmour, Robert Wisniewski, Richard Moore, and Michel Degenais. relayfs: An efficient unified approach for transmitting data from kernel to user space. In Proceedings of the Ottawa Linux Symposium 2003, July 2003.]]

Cited By

Luo ZSon SBali DAmaro EOusterhout ARatnasamy SShenker STsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)Efficient Microsecond-scale Blind Scheduling with Tiny QuantaProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640381(305-319)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640381
Eisner DWuersching RSchnappinger MPretschner AGarrido AWong WDe Angelis GDo HNguyen B(2022)Probe-based syscall tracing for efficient and practical file-level test tracesProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527239(126-137)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524481.3527239
Elsner DWuersching RSchnappinger MPretschner AGraber MDammer RReimer SHarman MMiller H(2022)Build system aware multi-language regression test selection in continuous integrationProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513078(87-96)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510457.3513078
Show More Cited By

Index Terms

Dynamic instrumentation of production systems
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering

Recommendations

Comprehensive kernel instrumentation via dynamic binary translation
ASPLOS '12

Dynamic binary translation (DBT) is a powerful technique that enables fine-grained monitoring and manipulation of an existing program binary. At the user level, it has been employed extensively to develop various analysis, bug-finding, and security ...
Comprehensive kernel instrumentation via dynamic binary translation
ASPLOS '12

Dynamic binary translation (DBT) is a powerful technique that enables fine-grained monitoring and manipulation of an existing program binary. At the user level, it has been employed extensively to develop various analysis, bug-finding, and security ...
Comprehensive kernel instrumentation via dynamic binary translation
ASPLOS XVII: Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems

Dynamic binary translation (DBT) is a powerful technique that enables fine-grained monitoring and manipulation of an existing program binary. At the user level, it has been employed extensively to develop various analysis, bug-finding, and security ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ATEC '04: Proceedings of the annual conference on USENIX Annual Technical Conference

June 2004

572 pages

Publisher

USENIX Association

United States

Publication History

Published: 27 June 2004

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

164
Total Citations
View Citations
19
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Luo ZSon SBali DAmaro EOusterhout ARatnasamy SShenker STsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)Efficient Microsecond-scale Blind Scheduling with Tiny QuantaProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640381(305-319)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640381
Eisner DWuersching RSchnappinger MPretschner AGarrido AWong WDe Angelis GDo HNguyen B(2022)Probe-based syscall tracing for efficient and practical file-level test tracesProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527239(126-137)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524481.3527239
Elsner DWuersching RSchnappinger MPretschner AGraber MDammer RReimer SHarman MMiller H(2022)Build system aware multi-language regression test selection in continuous integrationProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513078(87-96)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510457.3513078
Araújo RHolmes ROnuţ IZulkernine F(2021)Lightweight self-adaptive configuration using machine learningProceedings of the 31st Annual International Conference on Computer Science and Software Engineering10.5555/3507788.3507807(133-142)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.5555/3507788.3507807
Neves FVilaça RPereira J(2021)Detailed black-box monitoring of distributed systemsACM SIGAPP Applied Computing Review10.1145/3477133.347713521:1(24-36)Online publication date: 20-Jul-2021
https://dl.acm.org/doi/10.1145/3477133.3477135
Nelson LVan Geffen JTorlak EWang XLu SHowell J(2020)Specification and verification in the fieldProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488769(41-61)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488769
Yaseen NSonchack JLiu VBhagwan RPorter G(2020)tpprofProceedings of the 17th Usenix Conference on Networked Systems Design and Implementation10.5555/3388242.3388314(1015-1030)Online publication date: 25-Feb-2020
https://dl.acm.org/doi/10.5555/3388242.3388314
Akgun IKuenning GZadok E(2020)Re-AnimatorProceedings of the 13th ACM International Systems and Storage Conference10.1145/3383669.3398276(61-74)Online publication date: 30-May-2020
https://dl.acm.org/doi/10.1145/3383669.3398276
Mace JRoelke RFonseca R(2020)Pivot tracingCommunications of the ACM10.1145/337893363:3(94-102)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.1145/3378933
Wu YChen APhan LLorch JYu M(2019)ZenoProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323268(395-420)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323268
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents