Challenges of Running Scientific Workflows in Cloud Environments

This talk will touch upon the challenges of running scientific workflows in distributed environments such as academic and commercial clouds. It will describe the Pegasus Workflow Management System [1] and how it manages the execution of a variety of scientific workflows. Pegasus is a system that maps high-level workflow descriptions onto the available, potentially distributed compute and data resources. It supports workflow reusability by making the same workflow description portable across infrastructures, including different clouds. A single workflow can run on a particular cloud or across heterogeneous clouds. Pegasus also optimizes the workflow from the point of view of performance and reliability. The talk will explore the options for data management, the impact of file system choice on performance, and general issues of resource provisioning. It will examine the interplay between resource provisioning and workflow execution, and how knowledge about the workflow structure and task characteristics can be used to make better resource provisioning decisions at runtime. The talk will describe the concept of a Shadow Queue and how it is used to predict future workflow needs. This information is then made available to a provisioning system such as the one based on ORCA (Open Resource Control Architecture) [2] to provision compute and network resources as needed while the workflow is executing.