research-article

RAMP: a system for capturing and tracing provenance in MapReduce workflows

Authors:

Hyunjung Park,

Robert Ikeda,

Jennifer WidomAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 4, Issue 12

Pages 1351 - 1354

https://doi.org/10.14778/3402755.3402768

Published: 01 August 2011 Publication History

Get Access

Abstract

RAMP (Reduce And Map Provenance) is an extension to Hadoop that supports provenance capture and tracing for workflows of MapReduce jobs. RAMP uses a wrapper-based approach, requiring little if any user intervention in most cases, while retaining Hadoop's parallel execution and fault tolerance. We demonstrate RAMP on a real-world MapReduce workflow generated from a Pig script that performs sentiment analysis over Twitter data. We show how RAMP's automatic provenance capture and tracing capabilities provide a convenient and efficient means of drilling-down and verifying output elements.

References

[1]

Jaql. http://code.google.com/p/jaql/.

Google Scholar

[2]

Apache. Hadoop. http://hadoop.apache.org/.

Google Scholar

[3]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137--150, 2004.

Digital Library

Google Scholar

[4]

R. Ikeda, H. Park, and J. Widom. Provenance for generalized map and reduce workflows. In CIDR, pages 273--283, 2011.

Google Scholar

[5]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In SIGMOD, pages 1099--1110, 2008.

Digital Library

Google Scholar

[6]

B. Pang and L. Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115--124, 2005.

Digital Library

Google Scholar

[7]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive: A petabyte scale data warehouse using hadoop. In ICDE, pages 996--1005, 2010.

Crossref

Google Scholar

[8]

J. Yang and J. Leskovec. Patterns of temporal variation in online media. In WSDM, pages 177--186, 2011.

Digital Library

Google Scholar

Cited By

View all

Dias LLopes Bde Oliveira D(2024)MAESTRO: a lightweight ontology-based framework for composing and analyzing script-based scientific experimentsKnowledge and Information Systems10.1007/s10115-024-02134-266:10(5959-6000)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s10115-024-02134-2
Rani AGoyal NGadia S(2021)Twitter Data Modelling and Provenance Support for Key-Value Pair DatabasesDatabases Theory and Applications10.1007/978-3-030-69377-0_8(87-98)Online publication date: 29-Jan-2021
https://dl.acm.org/doi/10.1007/978-3-030-69377-0_8
Hallé STremblay HPasquier T(2020)A generic explainability framework for function circuitsProceedings of the 12th USENIX Conference on Theory and Practice of Provenance10.5555/3488890.3488897(7-7)Online publication date: 22-Jun-2020
https://dl.acm.org/doi/10.5555/3488890.3488897
Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 4, Issue 12

August 2011

303 pages

ISSN:2150-8097

Editor:
H. V. Jagadish

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2011

Published in PVLDB Volume 4, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
43
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dias LLopes Bde Oliveira D(2024)MAESTRO: a lightweight ontology-based framework for composing and analyzing script-based scientific experimentsKnowledge and Information Systems10.1007/s10115-024-02134-266:10(5959-6000)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s10115-024-02134-2
Rani AGoyal NGadia S(2021)Twitter Data Modelling and Provenance Support for Key-Value Pair DatabasesDatabases Theory and Applications10.1007/978-3-030-69377-0_8(87-98)Online publication date: 29-Jan-2021
https://dl.acm.org/doi/10.1007/978-3-030-69377-0_8
Hallé STremblay HPasquier T(2020)A generic explainability framework for function circuitsProceedings of the 12th USENIX Conference on Theory and Practice of Provenance10.5555/3488890.3488897(7-7)Online publication date: 22-Jun-2020
https://dl.acm.org/doi/10.5555/3488890.3488897
Rupprecht LDavis JArnold CGur YBhagwat D(2020)Improving reproducibility of data science pipelines through transparent provenance captureProceedings of the VLDB Endowment10.14778/3415478.341555613:12(3354-3368)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3415478.3415556
Wang ZKumar ANi SLi C(2020)Demonstration of interactive runtime debugging of distributed dataflows in TexeraProceedings of the VLDB Endowment10.14778/3415478.341551713:12(2953-2956)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3415478.3415517
Tan ZE HSong M(2020)A Column-Level Data Lineage Processing System Based on HiveProceedings of the 3rd International Conference on Big Data Technologies10.1145/3422713.3422719(47-52)Online publication date: 18-Sep-2020
https://dl.acm.org/doi/10.1145/3422713.3422719
Barika MGarg SZomaya AWang LMoorsel ARanjan R(2019)Orchestrating Big Data Analysis Workflows in the CloudACM Computing Surveys10.1145/333230152:5(1-41)Online publication date: 13-Sep-2019
https://dl.acm.org/doi/10.1145/3332301
Papavasileiou VYocum KDeutsch ABoncz PManegold SAilamaki ADeshpande AKraska T(2019)AriadneProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3300091(521-536)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3300091
Deutch DMoskovitch YRinetzky NBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Hypothetical Reasoning via Provenance AbstractionProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3300084(537-554)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3300084
Whittaker MTeodoropol CAlvaro PHellerstein J(2018)Debugging Distributed Systems with Why-Across-Time ProvenanceProceedings of the ACM Symposium on Cloud Computing10.1145/3267809.3267839(333-346)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3267809.3267839
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Big Data Analytics with R and Hadoop

Big Data Analytics

Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools