research-article

Dfanalyzer: runtime dataflow analysis of scientific applications using provenance

Authors:

Vítor Silva,

Daniel de Oliveira,

Patrick Valduriez,

Marta MattosoAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 11, Issue 12

Pages 2082 - 2085

https://doi.org/10.14778/3229863.3236265

Published: 01 August 2018 Publication History

Get Access

Abstract

We present DfAnalyzer, a tool that enables monitoring, debugging, steering, and analysis of dataflows while being generated by scientific applications. It works by capturing strategic domain data, registering provenance and execution data to enable queries at runtime. DfAnalyzer provides lightweight dataflow monitoring components to be invoked by high performance applications. It can be plugged in scientific code scripts, or Spark applications, in the same way users already plug visualization library components. During this demo, we will show how DfAnalyzer captures the dataflow, provenance, as well as how it provides runtime data analyses of applications. We will also encourage attendees to use DfAnalyzer for their own applications.

References

[1]

Armbrust, M., Zaharia, M., Das, T., Davidson, A., Ghodsi, A., Or, A., Rosen, J., Stoica, I., Wendell, P., et al. Scaling spark in the real world: performance and usability. PVLDB, 8(12): 1840--1843, 2015.

Digital Library

Google Scholar

[2]

Ayachit, U., Bauer, A., Duque, E.P.N., Eisenhauer, G., Ferrier, N., Gu, J., Jansen, K.E., Loring, B., Lukić, Z., et al. Performance Analysis, Design Considerations, and Applications of Extreme-scale in Situ Infrastructures. Supercomputing conference, 79:1--12, 2016.

Digital Library

Google Scholar

[3]

Camata, J.J., Silva, V., Valduriez, P., Mattoso, M., Coutinho, A.L.G.A. In situ visualization and data analysis for turbidity currents simulation. Computers & Geosciences, 110:23--31, 2018.

Digital Library

Google Scholar

[4]

Ikeda, R., Widom, J. Panda: A System for Provenance and Data. IEEE Data Engineering Bulletin, 42--49, 2010.

Google Scholar

[5]

Ogasawara, E., Dias, J., Oliveira, D., Porto, F., Valduriez, P., Mattoso, M. An Algebraic Approach for Data-Centric Scientific Workflows. PVLDB, 4(12):1328--1339, 2011.

Digital Library

Google Scholar

[6]

Olma, M., Karpathiotakis, M., Alagiannis, I., Athanassoulis, M., Ailamaki, A. Slalom: Coasting Through Raw Data via Adaptive Partitioning and Indexing. PVLDB, 10(10): 1106--1117, 2017.

Digital Library

Google Scholar

[7]

Pimentel, J.F., Murta, L., Braganholo, V., Freire, J. no Workflow: a tool for collecting, analyzing, and managing provenance from python scripts. PVLDB, 10(12): 1841--1844, 2017.

Digital Library

Google Scholar

[8]

Silva, V., Camata, J., de Oliveira, D., Coutinho, A.L.G.A., Valduriez, P., Mattoso, M. In Situ Data Steering on Sedimentation Simulation with Provenance Data. Poster session of Supercomputing conference, 2016.

Google Scholar

[9]

Silva, V., Leite, J., Camata, J., Oliveira, D., Coutinho, A.L.G., Valduriez, P., Mattoso, M. Raw Data Queries during Data-intensive Parallel Workflow Execution. Future Generation Computer Systems Journal, 75402--422, 2017.

Crossref

Google Scholar

Cited By

View all

Ferreira WKunstmann LPaes ABedo Mde Oliveira D(2024)AkôFlow: um Middleware para Execução de Workflows Científicos em Múltiplos Ambientes ConteinerizadosAnais do XXXIX Simpósio Brasileiro de Banco de Dados (SBBD 2024)10.5753/sbbd.2024.241126(27-39)Online publication date: 14-Oct-2024
https://doi.org/10.5753/sbbd.2024.241126
Becker SGoegge RKao O(2024)Measuring Application Interference With System-Level Instrumentation2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825462(3648-3653)Online publication date: 15-Dec-2024
https://doi.org/10.1109/BigData62323.2024.10825462
AlOmeir OLai EMilani MPottinger R(2023)Summarizing Provenance of Aggregate Query Results in Relational DatabasesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.326584035:10(10695-10709)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3265840
Show More Cited By

Recommendations

Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic Dataflows
Advances in Bioinformatics and Computational Biology
Abstract
Phylogenomic experiments provide the basis for evolutionary biology inferences. They are data- and CPU-intensive by nature and aim at producing phylogenomic trees based on an input dataset of protein sequences of genomes. These experiments can be ...
Efficient Java exception handling in just-in-time compilation
Research Articles

Java uses exceptions to provide elegant error handling capabilities during program execution. However, the presence of exception handlers complicates the job of the just-in-time (JIT) compiler, while exceptions are rarely used in most programs. This ...
Exception Handling during Asynchronous Method Invocation (Research Note)
Euro-Par '02: Proceedings of the 8th International Euro-Par Conference on Parallel Processing

Exception handling mechanisms provided by sequential programming languages rely upon the call stack for the propagation of exceptions. Unfortunately, this is inadequate for handling exceptions thrown from asynchronously invoked methods. For instance, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 11, Issue 12

August 2018

426 pages

ISSN:2150-8097

Editors:
Sihem Amer-Yahia
University of Grenoble Alpes, CNRS
,
Jian Pei
Simon Fraser University

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2018

Published in PVLDB Volume 11, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
157
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ferreira WKunstmann LPaes ABedo Mde Oliveira D(2024)AkôFlow: um Middleware para Execução de Workflows Científicos em Múltiplos Ambientes ConteinerizadosAnais do XXXIX Simpósio Brasileiro de Banco de Dados (SBBD 2024)10.5753/sbbd.2024.241126(27-39)Online publication date: 14-Oct-2024
https://doi.org/10.5753/sbbd.2024.241126
Becker SGoegge RKao O(2024)Measuring Application Interference With System-Level Instrumentation2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825462(3648-3653)Online publication date: 15-Dec-2024
https://doi.org/10.1109/BigData62323.2024.10825462
AlOmeir OLai EMilani MPottinger R(2023)Summarizing Provenance of Aggregate Query Results in Relational DatabasesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.326584035:10(10695-10709)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3265840
Akbarinia RBotella CJoly AMasseglia FMattoso MOgasawara Ede Oliveira DPacitti EPorto FPradal CShasha DValduriez P(2023)Life Science Workflow Services (LifeSWS): Motivations and ArchitectureTransactions on Large-Scale Data- and Knowledge-Centered Systems LV10.1007/978-3-662-68100-8_1(1-24)Online publication date: 28-Sep-2023
https://doi.org/10.1007/978-3-662-68100-8_1
Salazar VCavalcante Jde Oliveira DThompson FMattoso M(2021)BioProv - A provenance library for bioinformatics workflowsJournal of Open Source Software10.21105/joss.036226:67(3622)Online publication date: Nov-2021
https://doi.org/10.21105/joss.03622
Ahmad ZJehangiri AAla'anzy MOthman MLatip RZaman SUmar A(2021)Scientific Workflows Management and Scheduling in Cloud Computing: Taxonomy, Prospects, and ChallengesIEEE Access10.1109/ACCESS.2021.30707859(53491-53508)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3070785
Souza RAzevedo LLourenço VSoares EThiago RBrandão RCivitarese DVital Brazil EMoreno MValduriez PMattoso MCerqueira RNetto M(2021)Workflow provenance in the lifecycle of scientific machine learningConcurrency and Computation: Practice and Experience10.1002/cpe.654434:14Online publication date: 22-Aug-2021
https://doi.org/10.1002/cpe.6544
Silva VCampos VGuedes TCamata Jde Oliveira DCoutinho AValduriez PMattoso M(2020)DfAnalyzer: Runtime dataflow analysis tool for Computational Science and Engineering applicationsSoftwareX10.1016/j.softx.2020.10059212(100592)Online publication date: Jul-2020
https://doi.org/10.1016/j.softx.2020.100592
Guedes TMartins LFalci MSilva VOcaña KMattoso MBedo Mde Oliveira D(2020)Capturing and Analyzing Provenance from Spark-based Scientific Workflows with SAMbA-RaPFuture Generation Computer Systems10.1016/j.future.2020.05.031112(658-669)Online publication date: Nov-2020
https://doi.org/10.1016/j.future.2020.05.031
Pina DKunstmann Lde Oliveira DValduriez PMattoso M(2020)Provenance Supporting Hyperparameter Analysis in Deep Neural NetworksProvenance and Annotation of Data and Processes10.1007/978-3-030-80960-7_2(20-38)Online publication date: 22-Jun-2020
https://dl.acm.org/doi/10.1007/978-3-030-80960-7_2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Recommendations

Experiencing DfAnalyzer for Runtime Analysis of Phylogenomic Dataflows

Efficient Java exception handling in just-in-time compilation

Exception Handling during Asynchronous Method Invocation (Research Note)

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations