research-article

Automated packaging of bioinformatics workflows for portability and durability using makeflow

Authors:

Casey Robinson,

Douglas ThainAuthors Info & Claims

WORKS '13: Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

Pages 98 - 105

https://doi.org/10.1145/2534248.2534258

Published: 17 November 2013 Publication History

Get Access

Abstract

Dependency management remains a major challenge for all forms of software. A program implemented in a given environment typically has many implicit dependencies on programs, libraries, and other objects present within that environment. Moving applications between different runtime environments is certain to fail due to the existence of those external dependencies.

Workflows particularly suffer from dependency management problems, precisely because they tie together multiple independent programs into a coherent whole. To address the problem of workflow decay, we propose applying the old idea of a "linker" into the new context of workflow systems. We have implemented a linker for the Makeflow workflow system, and extended the concept to apply recursively to executables and scripted languages within the workflow. We evaluate the system by applying it to a selection of bioinformatics workflows including BLAST, BWA, and SHRiMP, enabling them to be moved across multiple computation environments. We also show that the portability provided by packaging allows for improved performance.

References

[1]

M. Albrecht, P. Donnelly, P. Bui, and D. Thain. Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids. In Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, 2012.

Digital Library

Google Scholar

[2]

P. J. Guo and D. Engler. Cde: Using system call interposition to automatically create portable software packages.

Google Scholar

[3]

W.-J. Li, K. Wang, S. J. Stolfo, and B. Herzog. Fileprints: Identifying file types by n-gram analysis. In Information Assurance Workshop, 2005. IAW'05. Proceedings from the Sixth Annual IEEE SMC, pages 64--71. IEEE, 2005.

Google Scholar

[4]

L. Presser and J. R. White. Linkers and loaders. ACM Computing Surveys (CSUR), 4(3): 149--167, 1972.

Digital Library

Google Scholar

[5]

A. Thrasher, R. Carmichael, P. Bui, L. Yu, D. Thain, and S. Emrich. Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch. In Workshop on Workflows in Support of Large Scale Science, pages 1--6, 2010.

Crossref

Google Scholar

[6]

J. Zhao, J. M. Gomez-Perez, K. Belhajjame, G. Klyne, E. Garcia-Cuesta, A. Garrido, K. Hettne, M. Roos, D. De Roure, and C. Goble. Why workflows break: Understanding and combating decay in taverna workflows. In E-Science (e-Science), 2012 IEEE 8th International Conference on, pages 1--9. IEEE, 2012.

Digital Library

Google Scholar

Cited By

View all

Zhang QKremer-Herman NTovar BThain D(2018)Reduction of Workflow Resource Consumption Using a Density-based Clustering Model2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS.2018.00006(1-9)Online publication date: Nov-2018
https://doi.org/10.1109/WORKS.2018.00006
Zheng CThain DDesprez FLebre AElmroth E(2015)Integrating Containers into WorkflowsProceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing10.1145/2755979.2755984(31-38)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1145/2755979.2755984
Vis JKok J(2014)Meta-analysis of Disjoint Sets of Attributes in Large Cohort StudiesLeveraging Applications of Formal Methods, Verification and Validation. Specialized Techniques and Applications10.1007/978-3-662-45231-8_32(407-419)Online publication date: 2014
https://doi.org/10.1007/978-3-662-45231-8_32

Recommendations

Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids
SWEET '12: Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies

In recent years, there has been a renewed interest in languages and systems for large scale distributed computing. Unfortunately, most systems available to the end user use a custom description language tightly coupled to a specific runtime ...
Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos
CCGrid '17: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Workflows are a widely used abstraction for describing large scientific applications and running them on distributed systems. However, most workflow systems have been silent on the question of what execution environment each task in the workflow is ...
Data parallelism in bioinformatics workflows using Hydra
HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Large scale bioinformatics experiments are usually composed by a set of data flows generated by a chain of activities (programs or services) that may be modeled as scientific workflows. Current Scientific Workflow Management Systems (SWfMS) are used to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WORKS '13: Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

November 2013

133 pages

ISBN:9781450325028

DOI:10.1145/2534248

Conference Chairs:
Johan Montagnat
CNRS
,
Ian Taylor
Cardiff University

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SC13

Sponsor:

SC13: International Conference for High Performance Computing, Networking, Storage and Analysis

November 17, 2013

Colorado, Denver

Acceptance Rates

WORKS '13 Paper Acceptance Rate 13 of 16 submissions, 81%;

Overall Acceptance Rate 30 of 54 submissions, 56%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
64
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhang QKremer-Herman NTovar BThain D(2018)Reduction of Workflow Resource Consumption Using a Density-based Clustering Model2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS.2018.00006(1-9)Online publication date: Nov-2018
https://doi.org/10.1109/WORKS.2018.00006
Zheng CThain DDesprez FLebre AElmroth E(2015)Integrating Containers into WorkflowsProceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing10.1145/2755979.2755984(31-38)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1145/2755979.2755984
Vis JKok J(2014)Meta-analysis of Disjoint Sets of Attributes in Large Cohort StudiesLeveraging Applications of Formal Methods, Verification and Validation. Specialized Techniques and Applications10.1007/978-3-662-45231-8_32(407-419)Online publication date: 2014
https://doi.org/10.1007/978-3-662-45231-8_32

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids

Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos

Data parallelism in bioinformatics workflows using Hydra