Article

A comparison of file system workloads

Authors:

Jacob R. Lorch,

Thomas E. AndersonAuthors Info & Claims

ATEC '00: Proceedings of the annual conference on USENIX Annual Technical Conference

Page 4

Published: 18 June 2000 Publication History

Abstract

In this paper, we describe the collection and analysis of file system traces from a variety of different environments, including both UNIX and NT systems, clients and servers, and instructional and production systems. Our goal is to understand how modern workloads affect the ability of file systems to provide high performance to users. Because of the increasing gap between processor speed and disk latency, file system performance is largely determined by its disk behavior. Therefore we primarily focus on the disk I/O aspects of the traces. We find that more processes access files via the memory-map interface than through the read interface. However, because many processes memory-map a small set of files, these files are likely to be cached. We also find that file access has a bimodal distribution pattern: some files are written repeatedly without being read; other files are almost exclusively read. We develop a new metric for measuring file lifetime that accounts for files that are never deleted. Using this metric, we find that the average block lifetime for some workloads is significantly longer than the 30-second write delay used by many file systems. However, all workloads show lifetime locality: the same files tend to be overwritten multiple times.

References

[1]

{Bake91} M. Baker, J. Hartman, M. Kupfer, K. Shirriff, and J. Ousterhout, "Measurements of a Distributed File System," Proceedings of the Thirteenth Symposium on Operating Systems Principles, pp. 198-212, December 1991.]]

[2]

{Bake92} M. Baker and M. Sullivan, "The Recovery Box: Using Fast Recovery to Provide High Availability in the UNIX Environment," Proceedings of the 1992 Summer USENIX Conference, pp. 31-41, June 1992.]]

[3]

{Benn91} J. M. Bennett, M. Bauer, and D. Kinchlea, "Characteristics of Files in NFS Environments," Proceedings of the 1991 Symposium on Small Systems, pp 33-40, June 1991.]]

[4]

{Blaz92} M. Blaze, "NFS Tracing by Passive Network Monitoring," Proceedings of the 1992 Winter USENIX Conference , pp. 333-343, January 1992.]]

[5]

{Bozm91} G. Bozman, H. Ghannad, and E. Weinberger, "A Trace-Driven Study of CMS File References," IBM Journal of Research and Development, 35(5-6), pp. 815-828, September-November 1991.]]

[6]

{Chen96} P. Chen, W. Ng, S. Chandra, C. Aycock, G. Rajamani, and D. Lowell, "The Rio File Cache: Surviving Operating System Crashes," Proceedings of the Seventh ASPLOS Conference, pp. 74-83, October 1996.]]

[7]

{Chia93} C. Chiang and M. Mutka, "Characteristics of User File-Usage Patterns," Systems and Software, 23(3), pp. 257-268, December 1993.]]

[8]

{Dahl94} M. Dahlin, C. Mather, R. Wang, T. Anderson, and D. Patterson, "A Quantitative Analysis of Cache Policies for Scalable Network File Systems," Proceedings of the 1994 Sigmetrics Conference, pp. 150-160, May 1994.]]

[9]

{Douc99} J. Douceur and W. Bolosky, "A Large-Scale Study of File-System Contents," Proceedings of the 1999 Sigmetrics Conference, pp. 59-70, June 1999.]]

[10]

{Floy89} R. Floyd and C. Schlatter Ellis, "Directory Reference Patterns in Hierarchical File Systems," IEEE Transactions on Knowledge and Data Engineering, 1(2), pp. 238-247, June 1989.]]

[11]

{Gang97} G. Ganger and M. F. Kaashoek, "Embedded Inodes and Explicit Groupings: Exploiting Disk Bandwidth for Small Files," Proceedings of the USENIX Annual Technical Conference, pp. 1-17, January 1997.]]

[12]

{Grib98} S. Gribble, G. Manku, D. Roselli, E. Brewer, T. Gibson, and E. Miller, "Self-Similarity in File Systems," Proceedings of the 1998 Sigmetrics Conference, pp. 141-150, June 1998.]]

[13]

{Hart93} J. Hartman and J. Ousterhout, "Corrections to Measurements of a Distributed File System," Operating Systems Review, 27(1), pp. 7-10, January 1993.]]

[14]

{Hitz94} D. Hitz, J. Lau, M. Malcolm, "File System Design for an NFS File Server Appliance," Proceedings of the 1994 Winter USENIX Conference, pp. 235-246, January 1994.]]

[15]

{Lorc00} J. Lorch and A. J. Smith, "Building VTrace, a Tracer for Windows NT," Accepted for publication in MSDN Magazine, September-October 2000.]]

[16]

{McKu84} M. McKusick, W. Joy, S. Leffler, and R. Fabry, "A Fast File System for UNIX," ACM Transactions on Computer Systems, 2(3), pp. 181-197, August 1984.]]

[17]

{McVo91} L. McVoy and S. Kleiman, "Extent-like Performance from a UNIX File System," Proceedings of the 1991 Winter USENIX Conference, pp. 33-44, January 1991.]]

[18]

{Mumm94} L. Mummert and M. Satyanarayanan, "Long-term Distributed File Reference Tracing: Implementation and Experience," Software--Practice and Experience, 26(6), pp. 705-736, November 1994.]]

[19]

{Oust85} J. Ousterhout, H. Da Costa, D. Harrison, J. Kunze, M. Kupfer, and J. Thompson, "A Trace-Driven Analysis of the UNIX 4.2 BSD File System," Proceedings of the Tenth Symposium on Operating Systems Principles, pp. 15-24, December 1985.]]

[20]

{Powe77} M. Powell, "The DEMOS File System," Proceedings of the Sixth Symposium on Operating Systems Principles, pp. 39-40, November 1977.]]

[21]

{Rose98} D. Roselli, "Characteristics of File System Workloads," University of California at Berkeley Technical Report CSD-98-1029, December 1998.]]

[22]

{Rose92} M. Rosenblum and J. Ousterhout, "The Design and Implementation of a Log-Structured File System for UNIX," ACM Transactions on Computer Systems, 10(1), pp. 26-52, February 1992.]]

[23]

{Ruem93} C. Ruemmler and J. Wilkes, "UNIX Disk Access Patterns," Proceedings of 1993 Winter USENIX Conference , CA, January 1993.]]

[24]

{Russ97a} M. Russinovich and B. Cogswell, "Windows NT System-Call Hooking," Dr. Dobb's Journal, 22(1), pp. 42-46, January 1997.]]

[25]

{Russ97b} M. Russinovich and B. Cogswell, "Examining the Windows NT Filesystem," Dr. Dobb's Journal, 22(2), pp. 42-50, February 1997.]]

[26]

{Sand85} R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, and B. Lyon, "Design and Implementation of the Sun Network Filesystem," Proceedings of the 1985 Summer USENIX Conference, pp. 119-130, June 1985.]]

[27]

{Saty81} M. Satyanarayanan, "A Study of File Sizes and Functional Lifetimes," Proceedings of the Eighth Symposium on Operating System Principles, pp. 96-108, December 1981.]]

[28]

{Sien94} T. Sienknecht, R. Friedrich, J. Martinka, and P. Friedenbach, "The Implications of Distributed Data in a Commercial Environment on the Design of Hierarchical Storage Management," Performance Evaluation , 20, pp. 3-25, May 1994.]]

[29]

{Smit81} A. J. Smith, "Analysis of Long Term File Reference Patterns for Application to File Migration Algorithms," IEEE Transactions on Software Engineering, SE-7(4), pp. 403-416, July 1981.]]

[30]

{Voge99} W. Vogels, "File System Usage in Windows NT 4.0", Proceedings of the Seventeenth Symposium on Operating Systems Principles, pp. 93-109, December 1999.]]

[31]

{Zhou99} M. Zhou and A. J. Smith, "Analysis of Personal Computer Workloads," Proceedings of the Seventh International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 208-217, October 1999.]]

Cited By

Rodriguez LGonzalez APoudel PRangaswami RLiu JShilane PWon Y(2021)Unifying the data center caching layerProceedings of the 13th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3465332.3470884(50-57)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3465332.3470884
Wortman PChandy J(2020)A trace-based study of SMB network file system workloads in an academic enterpriseProceedings of the 2020 Summer Simulation Conference10.5555/3427510.3427552(1-8)Online publication date: 20-Jul-2020
https://dl.acm.org/doi/10.5555/3427510.3427552
Cao ZDong SVemuri SDu DNoh SWelch B(2020)Characterizing, modeling, and benchmarking RocksDB key-value workloads at facebookProceedings of the 18th USENIX Conference on File and Storage Technologies10.5555/3386691.3386712(209-224)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.5555/3386691.3386712
Show More Cited By

Index Terms

A comparison of file system workloads
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Performance
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Operational analysis
  2. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Communications management
        Input / output
        File systems management

Recommendations

The zebra striped network file system
The Zebra striped network file system

Zebra is a network file system that increases throughput by striping the file data across multiple servers. Rather than striping each file separately, Zebra forms all the new data from each client into a single stream, which it then stripes using an ...
File system performance and transaction support

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ATEC '00: Proceedings of the annual conference on USENIX Annual Technical Conference

June 2000

704 pages

Publisher

USENIX Association

United States

Publication History

Published: 18 June 2000

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

139
Total Citations
View Citations
59
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rodriguez LGonzalez APoudel PRangaswami RLiu JShilane PWon Y(2021)Unifying the data center caching layerProceedings of the 13th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3465332.3470884(50-57)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3465332.3470884
Wortman PChandy J(2020)A trace-based study of SMB network file system workloads in an academic enterpriseProceedings of the 2020 Summer Simulation Conference10.5555/3427510.3427552(1-8)Online publication date: 20-Jul-2020
https://dl.acm.org/doi/10.5555/3427510.3427552
Cao ZDong SVemuri SDu DNoh SWelch B(2020)Characterizing, modeling, and benchmarking RocksDB key-value workloads at facebookProceedings of the 18th USENIX Conference on File and Storage Technologies10.5555/3386691.3386712(209-224)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.5555/3386691.3386712
Liu HDing WChen YGuo WLiu SLi TZhang MZhao JZhu HZhu ZBoncz PManegold SAilamaki ADeshpande AKraska T(2019)CFSProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3314046(1729-1742)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3314046
May MLaron EZoabi KGerhardt H(2019)On the Lifecycle of the FileACM Transactions on Storage10.1145/329546315:1(1-45)Online publication date: 18-Feb-2019
https://dl.acm.org/doi/10.1145/3295463
Cao ZWen HGe XMa JDiehl JDu D(2019)TDDFSACM Transactions on Storage10.1145/329546115:1(1-26)Online publication date: 5-Feb-2019
https://dl.acm.org/doi/10.1145/3295461
Singh HBawa S(2018)Scalable Metadata Management Techniques for Ultra-Large Distributed Storage Systems -- A Systematic ReviewACM Computing Surveys10.1145/321268651:4(1-37)Online publication date: 31-Jul-2018
https://dl.acm.org/doi/10.1145/3212686
Chen YShu JOu JLu Y(2018)HiNFSACM Transactions on Storage10.1145/320445414:1(1-30)Online publication date: 4-Apr-2018
https://dl.acm.org/doi/10.1145/3204454
Kashyap A(2018)Workload Characterization for Enterprise Disk DrivesACM Transactions on Storage10.1145/315184714:2(1-15)Online publication date: 12-Apr-2018
https://dl.acm.org/doi/10.1145/3151847
Lee SHyun SKim HKim Y(2018)Fair bandwidth allocating and strip-aware prefetching for concurrent read streams and striped RAIDs in distributed file systemsThe Journal of Supercomputing10.1007/s11227-018-2396-474:8(3904-3932)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.1007/s11227-018-2396-4
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents