Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A Distributed Architecture for Multimedia File Storage, Analysis and Processing

  • Chapter
Intelligent Tools for Building a Scientific Information Platform

Part of the book series: Studies in Computational Intelligence ((SCI,volume 467))

Abstract

The paper presents a prototype of a distributed storage, analysis and processing system for multimedia (video, audio, image) content based on the Apache Hadoop platform. The core of the system is a distributed file system with advanced topology, replication and balancing capabilities. Metadata for stored media objects is being managed in a separate database in conformance with well-known metadata models and standards. In addition, the system allows implementation of efficient multimedia data analysis (e.g. face recognition) and processing (e.g. transcoding, digital watermarking) algorithms. Interoperability with external systems, such as the currently-developed Infona is provided by means of an OAI-PMH-compliant interface for metadata harvesting. More in-depth integration is possible by making storage, analysis and processing services available to external entities using the SYNAT integration platform. The article evaluates several important aspects of the proposed distributed repository, such as security, performance, extensibility and interoperability.

Work financed by the National Centre for Research and Development (NCBiR) within project no. SP/I/1/77065/10.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Open CV Open Source Computer Vision (2012), http://opencv.willowgarage.com

  2. SYNAT Project (2012), http://www.synat.pl

  3. Apache Software foundation: Hadoop 1.0.2 API Documentation (2012), http://hadoop.apache.org/common/docs/r1.0.2/api/

  4. Borthakur, D.: HDFS Architecture Guide, Version 1.0.2 (2012), http://hadoop.apache.org/common/docs/r1.0.2/hdfs_design.html

  5. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: OSDI 2006: Seventh Symposium on Operating System Design and Implementation (2006)

    Google Scholar 

  6. Dean, J., Ghemawat, S.: Map Reduce: Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 1–13 (2008), http://www.usenix.org/events/osdi04/tech/full_papers/dean/dean_html/

    Article  Google Scholar 

  7. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Operating Systems Review 37(5), 29 (2003), http://portal.acm.org/citation.cfm?doid=1165389.945450

    Article  Google Scholar 

  8. Kim, M., Lee, H., Cui, Y.: Performance Evaluation of Image Conversion Module Based on MapReduce for Transcoding and Transmoding in SMCCSE. In: 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing, pp. 396–403 (2011), http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6118756

  9. Kiran, G.V.R.: Online Video Conversion Using Map Reduce. Tech. rep., International Institute of Information Technology, Hyderabad

    Google Scholar 

  10. Lin, J., Dyer, C.: Data-Intensive Text Processing with MapReduce, 1st edn. Morgan & Claypool Publishers (January 2010)

    Google Scholar 

  11. Sierra, S.: A Million Little Files (2008), http://stuartsierra.com/2008/04/24/a-million-little-files

  12. Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., Olmedilla, D., Miklós, Z.: A simple query interface for interoperable learning repositories. In: Proceedings of the 1st Workshop on Interoperability of Web-Based Educational Systems, pp. 11–18 (2005)

    Google Scholar 

  13. Sweeney, C., Liu, L., Arietta, S., Lawrence, J.: HIPI: A Hadoop Image Processing Interface for Image-based MapReduce Tasks. Master’s thesis, University of Virginia (2011)

    Google Scholar 

  14. White, T.: Hadoop: The Definitive Guide. O’Reilly Series, vol. 54. O’Reilly Media (2009)

    Google Scholar 

  15. White, T.: The Small Files Problem (2009), http://www.cloudera.com/blog/2009/02/the-small-files-problem/

Download references

Author information

Authors and Affiliations

  1. Department of Telecommunications, AGH University of Science and Technology, al. Mickiewicza 30, 30-059, Kraków, Poland

    Andrzej Dziech, Andrzej Glowacz & Jacek Wszołek

  2. ACC Cyfronet AGH, AGH University of Science and Technology, ul. Nawojki 11, 30-950, Kraków, Poland

    Sebastian Ernst & Michał Pawłowski

Authors
  1. Andrzej Dziech
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrzej Glowacz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacek Wszołek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sebastian Ernst
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michał Pawłowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrzej Dziech .

Editor information

Editors and Affiliations

  1. , Institute of Computer Science, Warsaw University of Technology, ul. Nowowiejska 15/19, Warszawa, 00-665, Poland

    Robert Bembenik

  2. , Institute of Computer Science, Warsaw University of Technology, ul. Nowowiejska 15/19, Warszawa, 00-665, Poland

    Lukasz Skonieczny

  3. , Institute of Computer Science, Warsaw University of Technology, ul. Nowowiejska 15/19, Warszawa, 00-665, Poland

    Henryk Rybinski

  4. , Institute of Computer Science, Warsaw University of Technology, ul. Nowowiejska 15/19, Warszawa, 00-665, Poland

    Marzena Kryszkiewicz

  5. , Interdisciplinary Centre for, University of Warsaw, Pawińskiego 5a bl. D, Warsaw, 02-106, Poland

    Marek Niezgodka

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Dziech, A., Glowacz, A., Wszołek, J., Ernst, S., Pawłowski, M. (2013). A Distributed Architecture for Multimedia File Storage, Analysis and Processing. In: Bembenik, R., Skonieczny, L., Rybinski, H., Kryszkiewicz, M., Niezgodka, M. (eds) Intelligent Tools for Building a Scientific Information Platform. Studies in Computational Intelligence, vol 467. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35647-6_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-35647-6_26

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-35646-9

  • Online ISBN: 978-3-642-35647-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation