Article

A low-bandwidth network file system

Authors:

Athicha Muthitacharoen,

David MazièresAuthors Info & Claims

SOSP '01: Proceedings of the eighteenth ACM symposium on Operating systems principles

Pages 174 - 187

https://doi.org/10.1145/502034.502052

Published: 21 October 2001 Publication History

Abstract

Users rarely consider running network file systems over slow or wide-area networks, as the performance would be unacceptable and the bandwidth consumption too high. Nonetheless, efficient remote file access would often be desirable over such networks---particularly when high latency makes remote login sessions unresponsive. Rather than run interactive programs such as editors remotely, users could run the programs locally and manipulate remote files through the file system. To do so, however, would require a network file system that consumes less bandwidth than most current file systems.This paper presents LBFS, a network file system designed for low-bandwidth networks. LBFS exploits similarities between files or versions of the same file to save bandwidth. It avoids sending data over the network when the same data can already be found in the server's file system or the client's cache. Using this technique in conjunction with conventional compression and caching, LBFS consumes over an order of magnitude less bandwidth than traditional network file systems on common workloads.

References

[1]

Brian Berliner. CVS II: Parellizing software development. In Proceedings of the Winter 1990 USENIX Technical Conference, Colorado Springs, CO, 1990.]]

[2]

David Bindel, Yen Chen, Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Seen Rhea, Hakim Weatherspoon, Westley Weimer, Westley Weimer, Christopher Wells, Ben Zhao, and John Kubiatowicz. Oceanstore: An exteremely wide-area storage system. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems, pages 190-201, Boston, MA, November 2000.]]

Digital Library

[3]

Sergey Brin, James Davis, and Hector Garcia-Molina. Copy detection mechanisms for digital documents. In Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, pages 398--409, San Jose, CA, May 1995.]]

Digital Library

[4]

Andrei Broder. On the resemblance and containment of documents. Compression and Complexity of Sequences, pages 21-29, 1997.]]

Digital Library

[5]

B. Callaghan, B. Pawlowski, and P. Staubach. NFS version 3 protocol specification. RFC 1813, Network Working Group, June 1995.]]

Digital Library

[6]

FIPS 180-1. Secure Hash Standard. U.S. Department of Commerce/N.I.S.T., National Technical Information Service, Springfield, VA, April 1995.]]

[7]

Cary G. Gray and David R. Cheriton. Leases: An efficient fault-tolerant mechanism for distributed file cache consistency. In Proceedings of the 12th ACM Symposium on Operating Systems Principles, pages 202-210, Litchfield Park, AZ, December 1989.]]

Digital Library

[8]

Bj0m Gr0nvall, Assar Westerlund, and Stephen Pink. The design of a multicast-based distributed file system. In Proceedings of the Third Symposium on Operating System Design and Implementation, pages 251-264, New Orleans, LA, February 1999.]]

Digital Library

[9]

John H. Howard, Michael L. Kazar, Sherri G. Menees, David A. Nichols, M. Satyanarayanan, Robert N. Sidebotham, and Michael J. West. Scale and performance in a distributed file system. ACM Transactions on Computer Systems, 6(1):51-81, February 1988.]]

Digital Library

[10]

James J. Kistler and M. Satyanarayanan. Disconnected operation in the coda file system. ACM Transactions on Computer Systems, 10(1):3-25, February 1992.]]

Digital Library

[11]

Eddie Kohler, Robert Morris, Benjie Chen, John Jannotti, and M. Frans Kaashoek. The Click modular router. ACM Transactions on Computer Systems, 18(4):263-297, November 2000.]]

Digital Library

[12]

Yui-Wah Lee, Kwong-Sak Leung, and M. Satyanarayanan. Operation-based update propagation in a mobile file system. In Proceedings of the 1999 USEN1X Technical Conference, Monterey, CA, June 1999.]]

Digital Library

[13]

Udi Manber. Finding similar files in a large file system. In Proceedings of the Winter 1994 USENIX Technical Conference, San Francisco, CA, January 1994.]]

Digital Library

[14]

Timothy Mann, Andrew D. Birrell, Andy Hisgen, Chuck Jerian, and Garret Swart. A coherent distributed file cache with directory write-behind. ACM Transactions on Computer Systems, 12(2):123-164, May 1994.]]

Digital Library

[15]

David Mazieres. A toolkit for user-level file systems. In Proceedings of the 2001 USENIX Technical Conference, Boston, MA, June 2001.]]

Digital Library

[16]

David Mazieres, Michael Kaminsky, M. Frans Kaashoek, and Emmett Witchel. Separating key management from file system security. In Proceedings of the 17. th ACM Symposium on Operating Systems Principles, pages 124-139, Kiawa Island, SC, 1999.]]

Digital Library

[17]

Jeffrey C. Mogul, Fred Douglis, Anja Feldmann, and Balachander Krishnamurthy. Potential benefits of delta encoding and data compression for http. In Proceedings of the 1997 ACM SIGCOMM Conference, pages 181-194, Cannes, France, September 1997.]]

Digital Library

[18]

Karin Petersen, Mike J. Spreitzer, and Douglas B. Terry. Flexible update propagation for weakly consistent replication. In Proceedings of the 16th ACM Symposium on Operating Systems Principles, pages 288-301, Saint-Male, France, 1997.]]

Digital Library

[19]

Michael O. Rabin. Fingerprinting by random polynomials. Technical Report TR-15-81, Center for Research in Computing Technology, Harvard University, 1981.]]

[20]

S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, and D. Noveck. NFS version 4 protocol. RFC 3010, Network Working Group, December 2000.]]

Digital Library

[21]

Neil T. Spring and David Wetherall. A protocol independent technique for eliminating redundant network traffic. In Proceedings of the 2000 ACM SIGCOMM Conference, pages 87- 95, Stockholm, Sweden, August 2000.]]

Digital Library

[22]

Walter E Tichy. The string-to-string correction problem with block moves. ACM Transactions on Computer Systems, 2(4):309-321, November 1984.]]

Digital Library

[23]

Andrew Tridgell. Efficient Algorithms for Sorting and Synchronization. PhD thesis, Australian National University, April 2000.]]

[24]

Assar Westerlund and Johan Danielsson. Arla--a free AFS client. In Proceedings of the 1998 USENIX, Freenix track, New Orleans, LA, June 1998. USENIX.]]

Digital Library

[25]

Haifeng Yu and Amin Vahdat. Design and evaluation of a continuous consistency model for replicated services. In Proceedings of the 4rd Symposium on Operating Systems Design and Implementation, San Diego, CA, 2000.]]

Digital Library

Cited By

Raaf PBrinkmann ABorba EAsadi HNarasimhamurthy SBent JEl-Batal MSalkhordeh R(2024)From SSDs Back to HDDs: Optimizing VDO to Support Inline Deduplication and Compression for HDDs as Primary Storage MediaACM Transactions on Storage10.1145/367825020:4(1-28)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3678250
Tan HXia WZou XDeng CLiao QGu Z(2024)The Design of Fast Delta Encoding for Delta Compression Based Storage SystemsACM Transactions on Storage10.1145/366481720:4(1-30)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3664817
Zhao NLin MAlbahar HPaul AHuang ZAbraham SChen KTarasov VSkourtis DAnwar AButt A(2024)An End-to-End High-Performance Deduplication Scheme for Docker Registries and Docker Container Storage SystemsACM Transactions on Storage10.1145/3643819Online publication date: 30-Jan-2024
https://doi.org/10.1145/3643819
Show More Cited By

Index Terms

A low-bandwidth network file system
1. Networks
  1. Network protocols
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Memory management
        Distributed memory
    2. Software system structures
      1. Distributed systems organizing principles

Recommendations

A low-bandwidth network file system

Users rarely consider running network file systems over slow or wide-area networks, as the performance would be unacceptable and the bandwidth consumption too high. Nonetheless, efficient remote file access would often be desirable over such networks---...
The zebra striped network file system
The Zebra striped network file system

Zebra is a network file system that increases throughput by striping the file data across multiple servers. Rather than striping each file separately, Zebra forms all the new data from each client into a single stream, which it then stripes using an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SOSP '01: Proceedings of the eighteenth ACM symposium on Operating systems principles

October 2001

254 pages

ISBN:1581133898

DOI:10.1145/502034

General Chair:
Keith Marzullo
University of California, San Diego
,
Program Chair:
M. Satyanarayanan
Carnegie Mellon University and Intel Research, Pittsburgh

ACM SIGOPS Operating Systems Review Volume 35, Issue 5
Dec. 2001
243 pages
ISSN:0163-5980
DOI:10.1145/502059
Issue’s Table of Contents

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SOSP01

Sponsor:

SIGOPS

SOSP01: 18th Symposium on Operating System Principles

October 21 - 24, 2001

Alberta, Banff, Canada

Acceptance Rates

SOSP '01 Paper Acceptance Rate 17 of 85 submissions, 20%;

Overall Acceptance Rate 131 of 716 submissions, 18%

Upcoming Conference

SOSP '25

Sponsor:
sigops

ACM SIGOPS 31st Symposium on Operating Systems Principles

October 13 - 16, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

679
Total Citations
View Citations
3,946
Total Downloads

Downloads (Last 12 months)191
Downloads (Last 6 weeks)17

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Raaf PBrinkmann ABorba EAsadi HNarasimhamurthy SBent JEl-Batal MSalkhordeh R(2024)From SSDs Back to HDDs: Optimizing VDO to Support Inline Deduplication and Compression for HDDs as Primary Storage MediaACM Transactions on Storage10.1145/367825020:4(1-28)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3678250
Tan HXia WZou XDeng CLiao QGu Z(2024)The Design of Fast Delta Encoding for Delta Compression Based Storage SystemsACM Transactions on Storage10.1145/366481720:4(1-30)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3664817
Zhao NLin MAlbahar HPaul AHuang ZAbraham SChen KTarasov VSkourtis DAnwar AButt A(2024)An End-to-End High-Performance Deduplication Scheme for Docker Registries and Docker Container Storage SystemsACM Transactions on Storage10.1145/3643819Online publication date: 30-Jan-2024
https://doi.org/10.1145/3643819
Huang HWang PSu QXu HXue CBrinkmann ATsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)Palantir: Hierarchical Similarity Detection for Post-Deduplication Delta CompressionProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640353(830-845)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640353
Liu HJin XYe CLiao XJin HZhang Y(2024)I/O Causality Based In-Line Data Deduplication for Non-Volatile Memory Enabled Storage SystemsIEEE Transactions on Computers10.1109/TC.2024.336596173:5(1327-1340)Online publication date: May-2024
https://doi.org/10.1109/TC.2024.3365961
Levina ASemenov A(2024)Jump-Based Backup: An Efficient Data Backup2024 13th Mediterranean Conference on Embedded Computing (MECO)10.1109/MECO62516.2024.10577875(1-4)Online publication date: 11-Jun-2024
https://doi.org/10.1109/MECO62516.2024.10577875
Jarah MUdayashankar SBaba AAl-Kiswany S(2024)The Impact of Low-Entropy on Chunking Techniques for Data Deduplication2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00025(134-140)Online publication date: 7-Jul-2024
https://doi.org/10.1109/CLOUD62652.2024.00025
Wu STu ZZhou YWang ZShen ZChen WWang WWang WMao B(2023)FASTSync: A FAST Delta Sync Scheme for Encrypted Cloud Storage in High-bandwidth Network EnvironmentsACM Transactions on Storage10.1145/360753619:4(1-22)Online publication date: 3-Oct-2023
https://dl.acm.org/doi/10.1145/3607536
Tenescu AAvramescu CBercean BMarcu M(2023)Lung Nodule Segmentation Using Federated Active LearningProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594850(17-21)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594850
Panda ASarangi S(2023)SnapStoreProceedings of the 24th International Middleware Conference10.1145/3590140.3629120(261-274)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3590140.3629120
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents