Abstract
Given the vast volume of data that needs to be stored reliably, many data-centers and large-scale file systems have started using erasure codes to achieve reliable storage while keeping the storage overhead low. This has invigorated the research on erasure codes tailor made to achieve different desirable storage system properties such as efficient redundancy replenishment mechanisms, resilience against data corruption, degraded reads, to name a few prominent ones. A problem that has mainly been overlooked until recently is that of how the storage system can be efficiently populated with erasure coded data to start with. In this paper, we will look at two distinct but related scenarios: (i) migration to archival - leveraging on existing replicated data to create an erasure encoded archive, and (ii) data insertion - new data being inserted in the system directly in erasure coded format. We will elaborate on coding techniques to achieve better throughput for data insertion and migration, and in doing so, explore the connection of these techniques with recently proposed locally repairable codes such as self-repairing codes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Acedaski, S., Deb, S., Mdard, M., Koetter, R.: How good is random linear coding based distributed networked storage. In: Workshop on Network Coding, Theory, and Applications, NetCod (2005)
Apache.org. HDFS-RAI, http://wiki.apache.org/hadoop/HDFS-RAID
Cleversafe, http://www.cleversafe.com
Datta, A., Oggier, F.: An overview of codes tailor-made for networked distributed data storage. CoRR, abs/1109.2317 (2011)
Oggier, F., Datta, A.: Self-repairing homomorphic codes for distributed storage systems. In: The 30th IEEE Intl. Conference on Computer Communications, INFOCOM (2011)
Oggier, F., Datta, A.: Coding techniques for repairability in networked distributed storage systems (2012), http://sands.sce.ntu.edu.sg/CodingForNetworkedStorage/pdf/longsurvey.pdf
Pamies-Juarez, L., Datta, A., Oggier, F.: In-network redundancy generation for opportunistic speedup of backup. CoRR, abs/1111.4533 (2011)
Pamies-Juarez, L., Datta, A., Oggier, F.: RapidRAID: Pipelined Erasure Codes for Fast Data Archival in Distributed Storage Systems. In: The 32nd IEEE International Conference on Computer Communications (Infocom 2013) (2013)
Pamies-Juarez, L., Oggier, F., Datta, A.: Decentralized Erasure Coding for Efficient Data Archival in Distributed Storage Systems. In: Frey, D., Raynal, M., Sarkar, S., Shyamasundar, R.K., Sinha, P. (eds.) ICDCN 2013. LNCS, vol. 7730, pp. 42–56. Springer, Heidelberg (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pamies-Juarez, L., Oggier, F., Datta, A. (2013). Data Insertion and Archiving in Erasure-Coding Based Large-Scale Storage Systems. In: Hota, C., Srimani, P.K. (eds) Distributed Computing and Internet Technology. ICDCIT 2013. Lecture Notes in Computer Science, vol 7753. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36071-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-36071-8_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36070-1
Online ISBN: 978-3-642-36071-8
eBook Packages: Computer ScienceComputer Science (R0)