research-article

Dfanalyzer: runtime dataflow analysis of scientific applications using provenance

Authors:

Vítor Silva,

Daniel de Oliveira,

Patrick Valduriez,

Marta MattosoAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 11, Issue 12

Pages 2082 - 2085

https://doi.org/10.14778/3229863.3236265

Published: 01 August 2018 Publication History

Get Access

Abstract

We present DfAnalyzer, a tool that enables monitoring, debugging, steering, and analysis of dataflows while being generated by scientific applications. It works by capturing strategic domain data, registering provenance and execution data to enable queries at runtime. DfAnalyzer provides lightweight dataflow monitoring components to be invoked by high performance applications. It can be plugged in scientific code scripts, or Spark applications, in the same way users already plug visualization library components. During this demo, we will show how DfAnalyzer captures the dataflow, provenance, as well as how it provides runtime data analyses of applications. We will also encourage attendees to use DfAnalyzer for their own applications.

References

[1]

Armbrust, M., Zaharia, M., Das, T., Davidson, A., Ghodsi, A., Or, A., Rosen, J., Stoica, I., Wendell, P., et al. Scaling spark in the real world: performance and usability. PVLDB, 8(12): 1840--1843, 2015.

Digital Library

Google Scholar

[2]

Ayachit, U., Bauer, A., Duque, E.P.N., Eisenhauer, G., Ferrier, N., Gu, J., Jansen, K.E., Loring, B., Lukić, Z., et al. Performance Analysis, Design Considerations, and Applications of Extreme-scale in Situ Infrastructures. Supercomputing conference, 79:1--12, 2016.

Digital Library

Google Scholar

[3]

Camata, J.J., Silva, V., Valduriez, P., Mattoso, M., Coutinho, A.L.G.A. In situ visualization and data analysis for turbidity currents simulation. Computers & Geosciences, 110:23--31, 2018.

Digital Library

Google Scholar

[4]

Ikeda, R., Widom, J. Panda: A System for Provenance and Data. IEEE Data Engineering Bulletin, 42--49, 2010.

Google Scholar

[5]

Ogasawara, E., Dias, J., Oliveira, D., Porto, F., Valduriez, P., Mattoso, M. An Algebraic Approach for Data-Centric Scientific Workflows. PVLDB, 4(12):1328--1339, 2011.

Digital Library

Google Scholar

[6]

Olma, M., Karpathiotakis, M., Alagiannis, I., Athanassoulis, M., Ailamaki, A. Slalom: Coasting Through Raw Data via Adaptive Partitioning and Indexing. PVLDB, 10(10): 1106--1117, 2017.

Digital Library

Google Scholar

[7]

Pimentel, J.F., Murta, L., Braganholo, V., Freire, J. no Workflow: a tool for collecting, analyzing, and managing provenance from python scripts. PVLDB, 10(12): 1841--1844, 2017.

Digital Library

Google Scholar

[8]

Silva, V., Camata, J., de Oliveira, D., Coutinho, A.L.G.A., Valduriez, P., Mattoso, M. In Situ Data Steering on Sedimentation Simulation with Provenance Data. Poster session of Supercomputing conference, 2016.

Google Scholar

[9]

Silva, V., Leite, J., Camata, J., Oliveira, D., Coutinho, A.L.G., Valduriez, P., Mattoso, M. Raw Data Queries during Data-intensive Parallel Workflow Execution. Future Generation Computer Systems Journal, 75402--422, 2017.

Crossref

Google Scholar

Cited By

View all

Pina DKunstmann Lde Oliveira DValduriez PMattoso M(2020)Provenance Supporting Hyperparameter Analysis in Deep Neural NetworksProvenance and Annotation of Data and Processes10.1007/978-3-030-80960-7_2(20-38)Online publication date: 22-Jun-2020
https://dl.acm.org/doi/10.1007/978-3-030-80960-7_2
Dias LMattoso MLopes Bde Oliveira D(2020)Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic DataflowsAdvances in Bioinformatics and Computational Biology10.1007/978-3-030-65775-8_10(105-116)Online publication date: 23-Nov-2020
https://dl.acm.org/doi/10.1007/978-3-030-65775-8_10
Barika MGarg SZomaya AWang LMoorsel ARanjan R(2019)Orchestrating Big Data Analysis Workflows in the CloudACM Computing Surveys10.1145/333230152:5(1-41)Online publication date: 13-Sep-2019
https://dl.acm.org/doi/10.1145/3332301

Recommendations

Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic Dataflows
Advances in Bioinformatics and Computational Biology
Abstract
Phylogenomic experiments provide the basis for evolutionary biology inferences. They are data- and CPU-intensive by nature and aim at producing phylogenomic trees based on an input dataset of protein sequences of genomes. These experiments can be ...
Efficient Java exception handling in just-in-time compilation
Research Articles

Java uses exceptions to provide elegant error handling capabilities during program execution. However, the presence of exception handlers complicates the job of the just-in-time (JIT) compiler, while exceptions are rarely used in most programs. This ...
Exception Handling during Asynchronous Method Invocation (Research Note)
Euro-Par '02: Proceedings of the 8th International Euro-Par Conference on Parallel Processing

Exception handling mechanisms provided by sequential programming languages rely upon the call stack for the propagation of exceptions. Unfortunately, this is inadequate for handling exceptions thrown from asynchronously invoked methods. For instance, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 11, Issue 12

August 2018

426 pages

ISSN:2150-8097

Editors:
Sihem Amer-Yahia
University of Grenoble Alpes, CNRS
,
Jian Pei
Simon Fraser University

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2018

Published in PVLDB Volume 11, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
157
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Pina DKunstmann Lde Oliveira DValduriez PMattoso M(2020)Provenance Supporting Hyperparameter Analysis in Deep Neural NetworksProvenance and Annotation of Data and Processes10.1007/978-3-030-80960-7_2(20-38)Online publication date: 22-Jun-2020
https://dl.acm.org/doi/10.1007/978-3-030-80960-7_2
Dias LMattoso MLopes Bde Oliveira D(2020)Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic DataflowsAdvances in Bioinformatics and Computational Biology10.1007/978-3-030-65775-8_10(105-116)Online publication date: 23-Nov-2020
https://dl.acm.org/doi/10.1007/978-3-030-65775-8_10
Barika MGarg SZomaya AWang LMoorsel ARanjan R(2019)Orchestrating Big Data Analysis Workflows in the CloudACM Computing Surveys10.1145/333230152:5(1-41)Online publication date: 13-Sep-2019
https://dl.acm.org/doi/10.1145/3332301

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic Dataflows

Efficient Java exception handling in just-in-time compilation

Exception Handling during Asynchronous Method Invocation (Research Note)