Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A Novel Distributed File System Using Blockchain Metadata

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Cluster computing has become an inevitable part of data processing as the huge volume of data being produced from different sources like online social media, IoT, mobiledata, sensor data, black box data and so on increases in an exponentially fast manner. Distributed File System defines different methods to distribute, read and eliminate the files among different cluster computing nodes. It is found that popular distributed file systems such as Google File System and Hadoop Distributed File System store metadata centrally. This creates a chance for a Single Point of Failure that arises the need for backup and alternative solutions to recover the metadata on the failure of the metadata server. Also, the name node server is built using expensive and reliable hardware. For small and medium clusters, it is not cost effective to maintain expensive name node server. Even though cheap commodity hardware may substitute the name node functionality, they are prone to hardware failure. This paper proposes a novel distributed file system to distribute files over a cluster of machines connected in a Peer-to-Peer network. The most significant feature of the file system is its capability to distribute the metadata using distributed consensus, using hash values. Although the distributed metadata is visible to the public, the methodology ensures that it is immutable and irrefutable. As part of the in-depth research, the proposed file system has been successfully tested in the Google Cloud Platform. Also, the basic operations like read, write, and delete on Distributed File System with distributed metadata are compared with that of Hadoop Distributed File System based on distribution time on the same cluster setup. The novel distributed file system provides better results compared to the existing methodologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data Availability

Data sharing not applicable—no new data generated.

Code Availability

We used our own data and coding.

References

  1. Li, X. S., et al. (2011). Analysis and simplification of three-dimensional space vector PWM for three-phase four-leg inverters. IEEE Transactions on Industrial Electronics, 58, 450–464.

    Article  Google Scholar 

  2. https://www.slideshare.net/wahabtl/chapter-8-distributed-file-systems.

  3. Shvachko, K., et al. (2010). The hadoop distributed file system. Yahoo! Sunnyvale, California USA.

  4. White, T. (2009). Hadoop: The definitive guide. O'Reilly Media, Yahoo! Press, 2009

  5. Shvachko, K. V. (2010). HDFS scalability: the limits to growth. LOGIN

  6. Shavchko, K. V., et al. (2017) File systems and storage- scaling namespace operations with giraffa file system. Summer 2017login 42(2):27–30, 2017. www.usenix.org.

  7. Ghemawat, S., et al. (2003). The Google file system. In: Proceedings of the ACM symposium on operating systems principles, Lake George, NY, pp. 29–43.

  8. McKusick, M. K., et al. (2009). GFS: Evolution on fastforward. ACM Queue, Vol. 7, no 7. ACM, New York

  9. http://hadoop.apache.org/releases.pdf.

  10. https://ignite.apache.org/features/igfs.html.

  11. https://www.datastax.com/wp-content/uploads/2012/09/WP-DataStax-HDFSvsCFS.pdf.

  12. Shvachko, K. V. (2006). The hadoop distributed file system requirements. Hadoop Wiki. http://wiki.apache.org/hadoop/DFS_requirements.

  13. https://www.slideshare.net/KonstantinVShvachko/hdfs-design-principles.

  14. Weil, S., et al. (2006). Ceph: A scalable, high-performance distributed file system. In: Proceedings of OSDI ’06: 7th conference on operating systems design and implementation (USENIX Association, 2006)

  15. Lustre: http://www.lustre.org.

  16. Bhaskar et al. (2016). 3–Bitcoin mining technology. Handbook of digital currency: Bitcoin, innovation, financial instruments, and big data. Academic Press. pp. 47–51. Retrieved 2 Dec 2016—via ScienceDirect

  17. Kumar, D. S., et al. (2017). Performance evaluation of apache spark Vs MPI: A practical case study on twitter sentiment analysis. Journal of Computer Sciences, 13(12), 781–794. https://doi.org/10.3844/jcssp.2017.781.794

    Article  Google Scholar 

  18. Steichen, M., et al. (2018). Blockchain-based, decentralized access control for IPFS, 2018. In: IEEE confs on internet of things, green computing and communications, cyber, physical and social computing, smart data, blockchain, computer and information technology, congress on cybermatics.

Download references

Funding

No funding.

Author information

Authors and Affiliations

  1. College of Engineering Munnar, Idukki, India

    Deepa S. Kumar

  2. Centre for Development of Advanced Computing, Thiruvananthapuram, India

    S. Dija

  3. LBS Institute of Technology for Women, Thiruvananthapuram, India

    M. D. Sumithra

  4. LBS Centre for Science and Technology, Thiruvananthapuram, India

    M. Abdul Rahman

  5. Amal Jyothi College of Engineering, Kanjirappally, Kottayam, India

    Praseeda B. Nair

Authors
  1. Deepa S. Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Dija
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. D. Sumithra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Abdul Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Praseeda B. Nair
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors are contributed in this work.

Corresponding author

Correspondence to Deepa S. Kumar.

Ethics declarations

Conflict of interest

No conflicts of interest to disclose.

Human and Animal Rights

Humans and animals are not involved in this research work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, D.S., Dija, S., Sumithra, M.D. et al. A Novel Distributed File System Using Blockchain Metadata. Wireless Pers Commun 129, 501–520 (2023). https://doi.org/10.1007/s11277-022-10108-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-022-10108-2

Keywords

Search

Navigation