Abstract
Shift is a lightweight framework for high performance local and remote file transfers that provides resiliency across a wide variety of failure scenarios. Shift supports multiple file transport protocols with automatic selection of the most appropriate mechanism between each pair of participating hosts allowing it to adapt to heterogeneous clients with differing software and network access restrictions. File system information is gathered from clients and servers to detect file system equivalence and enable path rewriting so that multiple clients can be automatically spawned in parallel to carry out both single and multi-file transfers to multiple servers selected according to load and availability. This improves both reliability and performance by eliminating single points of failure and overcoming single system bottlenecks. End-to-end integrity is provided using cryptographic hashes at the source and destination with support for partial file retransmission of only corrupted portions. This paper presents the design and implementation of Shift and details the mechanisms utilized to enhance the reliability and performance of file transfers.
Supported by Task ARC-013 (Contract NNA07CA29C) with Computer Sciences Corporation
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M., Dumitrescu, C., Raicu, I., Foster, I.: The Globus Striped GridFTP Framework and Server. In: ACM/IEEE Supercomputing 2005 Conf. (November 2005)
Basney, J., Duda, P.: Clustering the Reliable File Transfer Service. In: 2nd TeraGrid Conf. (June 2007)
BbFTP, http://doc.in2p3.fr/bbftp
Cohen, B.: Incentives Build Robustness in BitTorrent. In: 1st Wkshp. on Economics of Peer-to-Peer Systems (June 2003)
Galbraith, J., Saarenmaa, O.: SSH File Transfer Protocol. IETF Internet Draft (July 2006)
GSI-Enabled OpenSSH, http://grid.ncsa.illinois.edu/ssh
Kolano, P.Z.: Mesh: Secure, Lightweight Grid Middleware Using Existing SSH Infrastructure. In: 12th ACM Symp. on Access Control Models and Technologies (June 2007)
Kolano, P.Z., Ciotti, R.B.: High Performance Multi-Node File Copies and Checksums for Clustered File Systems. In: 24th USENIX Large Installation System Administration Conf. (November 2010)
Kosar, T., Livny, M.: A Framework for Reliable and Efficient Data Placement in Distributed Computing Systems. Jour. of Parallel and Distributed Computing 65(10) (2005)
Kunszt, P., Badino, P., Brito da Rocha, R., Casey, J., Frohner, A., McCance, G.: The gLite File Transfer Service. In: 1st EGEE User Forum (March 2006)
Kunszt, P., Laure, E., Stockinger, H., Stockinger, K.: File-Based Replica Management. Future Generation Computer Systems 21(1) (January 2005)
Lee, Y., Kim, E., Yeom, H.Y.: Replica Aware Reliable File Transfer Service for the Data Grid. In: 4th IEEE Intl. Conf. on eScience (2008)
Lim, S., Fox, G., Pallickara, S., Pierce, M.: Web Service Robust GridFTP. In: 10th Intl. Conf. on Parallel and Distributed Processing Techniques and Applications (June 2004)
Machek, P.: FIles transferred over SHell protocol (V 0.0.2), http://cvs.savannah.gnu.org/viewvc/*checkout*/mc/mc/vfs/README.fish
Madduri, R.K., Hood, C.S., Allcock, W.E.: Reliable File Transfer in Grid Environments. In: 27th IEEE Conf. on Local Computer Networks (November 2002)
Rapier, C., Bennett, B.: High Speed Bulk Data Transfer Using the SSH Protocol. In: 15th ACM Mardi Gras Conf. (February 2008)
Rosenblum, M., Ousterhout, J.K.: The Design and Implementation of a Log-Structured File System. ACM Trans. on Computer Systems 10(1) (February 1992)
Rsync, http://samba.org/rsync
Sultana, A., Bashir, M.F., Qadir, M.A.: CFiTT - Corrupt Free File Transfer Technique Over FTP. In: 1st IEEE Intl. Conf. on Information and Emerging Technologies (July 2007)
Zissimos, A., Doka, K., Chazapis, A., Koziris, N.: GridTorrent: Optimizing Data Transfers in the Grid with Collaborative Sharing. In: 11th Panhellenic Conf. on Informatics (May 2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kolano, P.Z. (2013). High Performance Reliable File Transfers Using Automatic Many-to-Many Parallelization. In: Caragiannis, I., et al. Euro-Par 2012: Parallel Processing Workshops. Euro-Par 2012. Lecture Notes in Computer Science, vol 7640. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36949-0_52
Download citation
DOI: https://doi.org/10.1007/978-3-642-36949-0_52
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36948-3
Online ISBN: 978-3-642-36949-0
eBook Packages: Computer ScienceComputer Science (R0)