We’re excited to announce WfCommons 1.2, a major update to the WfCommons project, an open-source framework designed to accelerate research and development in scientific workflows.

WfCommons provides a comprehensive set of tools to analyze real workflow executions and generate realistic synthetic workflows, enabling the development and benchmarking of new algorithms, runtime systems, and scheduling strategies for increasingly large and complex workflows.

This release supports the entire workflow R&D lifecycle through the following key components:

• WfFormat: A standard schema (v1.5) for representing workflow execution instances
• WfInstances: A curated collection of real-world workflow traces and the WfInstances Browser for exploration
• WfChef: A library of 10 embedded workflow recipes derived from real data
• WfGen: A synthetic workflow generator based on configurable recipes
• WfBench: A benchmarking suite with translators for major workflow systems including Airflow, CWL, Dask, Nextflow, Parsl, Pegasus, PyCOMPSs, Swift/T, and TaskVine

Whether you're building new workflow systems or evaluating performance and scalability, WfCommons 1.2 provides the building blocks you need.

Full documentation and more: https://wfcommons.org/

The WfCommons project is an open source framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution instances, and generating synthetic, yet realistic, workflow instances that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures. This Python package provides methods for analyzing instances, deriving recipes, and generating representative synthetic workflow instances and benchmarks.

This release targets the research and development life cycle of workflow systems and applications, which is composed of the following major components:

• WfFormat: common format for representing workflow execution instances (this release is compatible with schema version 1.5)
• WfInstances: workflow execution instances
• WfChef: workflow recipes
• WfGen: workflow generator
• WfBench: workflow benchmarks
• WfSim: workflow simulators

Documentation and additional information: https://wfcommons.org

The WfCommons project is an open source framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution instances, and generating synthetic, yet realistic, workflow instances that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures. This Python package provides methods for analyzing instances, deriving recipes, and generating representative synthetic workflow instances.

This release targets the research and development life cycle of workflow systems and applications, which is composed of the following major components:

• WfFormat: common format for representing workflow execution instances (this release is compatible with schema version 1.4)
• WfInstances: workflow execution instances
• WfChef: workflow recipes
• WfGen: workflow generator
• WfBench: workflow benchmarks 🆕
• WfSim: workflow simulators

Documentation and additional information: https://wfcommons.org

The WfCommons project is an open source framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution instances, and generating synthetic, yet realistic, workflow instances that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures. This Python package provides methods for analyzing instances, deriving recipes, and generating representative synthetic workflow instances.

This release targets the research and development life cycle of workflow systems and applications, which is composed of the following major components:

• WfFormat: common format for representing workflow execution instances (this release is compatible with schema version 1.3)
• WfInstances: workflow execution instances
• WfChef: workflow recipes
• WfGen: workflow generator
• WfSim: workflow simulators

The WfGen component targets the generation of realistic synthetic workflow instances. WfGen takes as input a workflow recipe produced by WfChef for a particular application and a desired number of tasks. WfGen then automatically generates synthetic, yet realistic, randomized workflow instances with (approximately) the desired number of tasks.

Documentation and additional information: https://wfcommons.org

The WfCommons project is an open source framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution instances, and generating synthetic, yet realistic, workflow instances that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures. This Python package provides methods for analyzing instances, deriving recipes, and generating representative synthetic workflow instances.

This release is aligned with a reorganization of the WfCommons project that targets the research and development life cycle of workflow systems and applications, which is composed of the following major components: (i) workflow execution instances (WfInstances), (ii) workflow recipes (WfChef), (iii) workflow generator (WfGen), and (iv) workflow simulator (WfSim).

The WfGen component targets the generation of realistic synthetic workflow instances. WfGen takes as input a workflow recipe produced by WfChef for a particular application and a desired number of tasks. WfGen then automatically generates synthetic, yet realistic, randomized workflow instances with (approximately) the desired number of tasks.

Documentation and additional information: https://wfcommons.org

The WfCommons project is an open source framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution instances, and generating synthetic, yet realistic, workflow instances that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures. This Python package provides methods for analyzing instances, deriving recipes, and generating representative synthetic workflow instances.

This release is aligned with a reorganization of the WfCommons project that targets the research and development life cycle of workflow systems and applications, which is composed of the following major components: (i) workflow execution instances (WfInstances), (ii) workflow recipes (WfChef), (iii) workflow generator (WfGen), and (iv) workflow simulator (WfSim).

The WfGen component targets the generation of realistic synthetic workflow instances. WfGen takes as input a workflow recipe produced by WfChef for a particular application and a desired number of tasks. WfGen then automatically generates synthetic, yet realistic, randomized workflow instances with (approximately) the desired number of tasks.

Documentation and additional information: https://wfcommons.org

WfCommons is a community framework that provides a collection of tools for analyzing workflow execution instances, producing realistic synthetic workflow traces, and simulating workflow executions.

This Python package provides a collection of tools for: (i) Analyzing instances of actual workflow executions; (ii) Producing recipes structures for creating workflow recipes for workflow generation; and (iii) Generating synthetic realistic workflow instances.

The current list of available workflow recipes include the following workflow applications:

• 1000Genome: A high-throughput data-intensive bioinformatics workflow.
• BLAST: A high-throughput compute-intensive bioinformatics workflow.
• BWA: A high-throughput data-intensive bioinformatics workflow.
• Cycles: A high-throughput compute-intensive scientific workflow for agroecosystems modeling.
• Epigenomics: A high-throughput data-intensive bioinformatics workflow.
• Montage: A high-throughput compute-intensive astronomy workflow.
• Seismology: A high-throughput data-intensive seismology workflow.
• SoyKB: A high-throughput data-intensive bioinformatics workflow.

Documentation and additional information: https://wfcommons.org

The WorkflowHub project is a community framework for a community framework that provides a collection of tools for analyzing workflow execution traces, producing realistic synthetic workflow traces, and simulating workflow executions.

This Python package provides a collection of tools for: (i) Analyzing traces of actual workflow executions; (ii) Producing recipes structures for creating workflow recipes for workflow generation; and (iii) Generating synthetic realistic workflow traces.

The current list of available workflow recipes include the following workflow applications:

• 1000Genome: A high-throughput data-intensive bioinformatics workflow.
• BLAST: A high-throughput compute-intensive bioinformatics workflow.
• BWA: A high-throughput data-intensive bioinformatics workflow.
• Cycles: A high-throughput compute-intensive scientific workflow for agroecosystems modeling.
• Epigenomics: A high-throughput data-intensive bioinformatics workflow.
• Montage: A high-throughput compute-intensive astronomy workflow.
• Seismology: A high-throughput data-intensive seismology workflow.
• SoyKB: A high-throughput data-intensive bioinformatics workflow.

In this version, we have added three new workflow generator recipes (#9, #12), added a parser for Pegasus (#5, #6) and Makeflow (#10) workflow systems, provided the ability to increase/reduce runtime and files size by a factor (#11), fixed an issue with the Montage generator (#7) and file size generation (#8), and performed some enhancements.

Documentation and additional information: https://workflowhub.org

The WorkflowHub project is a community framework for enabling scientific workflow research and development by providing foundational tools for analyzing workflow execution traces, and generating synthetic, yet realistic, workflow traces that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures.

This Python package provides a collection of tools for: (i) Analyzing traces of actual workflow executions; (ii) Producing recipes structures for creating workflow recipes for workflow generation; and (iii) Generating synthetic realistic workflow traces.

The current list of available workflow recipes include the following workflow applications:

• 1000Genome: A high-throughput data-intensive bioinformatics workflow.
• Cycles: A high-throughput compute-intensive scientific workflow for agroecosystems modeling.
• Epigenomics: A high-throughput data-intensive bioinformatics workflow.
• Montage: A high-throughput compute-intensive astronomy workflow.
• Seismology: A high-throughput data-intensive seismology workflow.
• SoyKB: A high-throughput data-intensive bioinformatics workflow.

In this version, we have improved the documentation (#1), fixed an issue with the Montage generator (#2), and performed some enhancements (#3, #4).

Documentation and additional information: https://workflowhub.org

The WorkflowHub project is a community framework for enabling scientific workflow research and education by providing foundational tools for analyzing workflow execution traces, and generating synthetic, yet realistic, workflow traces that can be used to develop new techniques, algorithms and systems that can overcome the challenges of efficient and robust execution of ever larger workflows on increasingly complex distributed infrastructures.

This Python package provides a collection of tools for: (i) Analyzing traces of actual workflow executions; (ii) Producing recipes structures for creating workflow recipes for workflow generation; and (iii) Generating synthetic realistic workflow traces.

The current list of available workflow recipes include the following workflow applications:

• 1000Genome: A high-throughput data-intensive bioinformatics workflow.
• Cycles: A high-throughput compute-intensive scientific workflow for agroecosystems modeling.
• Epigenomics: A high-throughput data-intensive bioinformatics workflow.
• Montage: A high-throughput compute-intensive astronomy workflow.
• Seismology: A high-throughput data-intensive seismology workflow.
• SoyKB: A high-throughput data-intensive bioinformatics workflow.

Documentation and additional information: https://workflowhub.org