research-article

Scalable Execution of Big Data Workflows using Software Containers

Authors:

Yared Dejene Dessalk,

Nikolay Nikolov,

Mihhail Matskin,

Ahmet Soylu,

Dumitru RomanAuthors Info & Claims

MEDES '20: Proceedings of the 12th International Conference on Management of Digital EcoSystems

Pages 76 - 83

https://doi.org/10.1145/3415958.3433082

Published: 27 November 2020 Publication History

Get Access

Abstract

Big Data processing involves handling large and complex data sets, incorporating different tools and frameworks as well as other processes that help organisations make sense of their data collected from various sources. This set of operations, referred to as Big Data workflows, require taking advantage of the elasticity of cloud infrastructures for scalability. In this paper, we present the design and prototype implementation of a Big Data workflow approach based on the use of software container technologies and message-oriented middleware (MOM) to enable highly scalable workflow execution. The approach is demonstrated in a use case together with a set of experiments that demonstrate the practical applicability of the proposed approach for the scalable execution of Big Data workflows. Furthermore, we present a scalability comparison of our proposed approach with that of Argo Workflows - one of the most prominent tools in the area of Big Data workflows.

References

[1]

D Culler Arvind et al. 1984. The tagged token dataflow architecture. Technical Report. Technical report, MIT Laboratory for Computer Science.

Digital Library

Google Scholar

[2]

Mutaz Barika et al. 2019. Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions. Comput. Surveys 52, 5 (2019).

Digital Library

Google Scholar

[3]

Edward Curry. 2005. Message-Oriented Middleware. John Wiley & Sons, Ltd, 1--28.

Google Scholar

[4]

Yared Dejene Dessalk. 2020. Big Data Workflows: DSL-based Specification and Software Containers for Scalable Execution. The Royal Institute of Technology, 1--49.

Google Scholar

[5]

W. Gerlach et al. 2014. Skyport - Container-Based Execution Environment Management for Multi-cloud Scientific Workflows. In Proc. of the DataCloud 2014. 25--32.

Digital Library

Google Scholar

[6]

A. Kashlev et al. 2017. Big Data Workflows: A Reference Architecture and the DATAVIEW System. Services Transactions on Big Data 4, 1 (2017).

Crossref

Google Scholar

[7]

Marjan Mernik et al. 2005. When and How to Develop Domain-Specific Languages. Comput. Surveys 37, 4 (2005), 316--344.

Digital Library

Google Scholar

[8]

Sara Migliorini et al. 2011. Pattern-Based Evaluation of Scientific Workflow Management Systems.

Google Scholar

[9]

N. Naik. 2017. Docker container-based big data processing system in multiple clouds for everyone. In Proc. of the ISSE 2017. 1--7.

Crossref

Google Scholar

[10]

R. Qasha et al. 2016. Dynamic Deployment of Scientific Workflows in the Cloud Using Container Virtualization. In Proc. of the CloudCom 2016. 269--276.

Crossref

Google Scholar

[11]

R. Ranjan et al. 2017. Orchestrating Big Data Analysis Workflows. IEEE Cloud Computing 4, 3 (2017), 20--28.

Crossref

Google Scholar

[12]

Nick Russell et al. 2005. Workflow Data Patterns: Identification, Representation and Tool Support. In Proc. of the ER 2005. 353--368.

Digital Library

Google Scholar

[13]

C. Wulf et al. 2016. Increasing the Throughput of Pipe-and-Filter Architectures by Integrating the Task Farm Parallelization Pattern. In Proc. of the CBSE 2016. 13--22.

Crossref

Google Scholar

[14]

Charles Zheng and Douglas Thain. 2015. Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker. In Proc. of the VTDC 2015. 31--38.

Digital Library

Google Scholar

Cited By

View all

Mikina FZuk PRzadca K(2024)sAirflow: Adopting Serverless in a Legacy Workflow SchedulerEuro-Par 2024: Parallel Processing10.1007/978-3-031-69577-3_18(254-268)Online publication date: 26-Aug-2024
https://doi.org/10.1007/978-3-031-69577-3_18
Cheng JZhao P(2022)Sustainable Big Data Analytics Process Pipeline Using Apache EcosystemEncyclopedia of Data Science and Machine Learning10.4018/978-1-7998-9220-5.ch073(1247-1259)Online publication date: 14-Oct-2022
https://doi.org/10.4018/978-1-7998-9220-5.ch073
Tahmasebi SLayegh ANikolov NPayberah ADinh KMitrovic VRoman DMatskin M(2022)Dataclouddsl: Textual and Visual Presentation of Big Data Pipelines2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00183(1165-1171)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00183
Show More Cited By

Index Terms

Scalable Execution of Big Data Workflows using Software Containers
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing
2. Information systems
  1. Data management systems
  2. Information systems applications
    1. Computing platforms

Recommendations

Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions

Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies, and research outcomes. This has led to advancement in communication, programming, and processing technologies, including cloud ...
Locality-Aware Workflow Orchestration for Big Data
MEDES '21: Proceedings of the 13th International Conference on Management of Digital EcoSystems

The development of the Edge computing paradigm shifts data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructure. Such a paradigm requires data processing solutions that consider data locality in ...
Scalable Big Data Architecture: A practitioners guide to choosing relevant Big Data architecture

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

MEDES '20: Proceedings of the 12th International Conference on Management of Digital EcoSystems

November 2020

170 pages

ISBN:9781450381154

DOI:10.1145/3415958

General Chairs:
Richard Chbeir
University Pau & Pays de l'Adour, France
,
Yannis Manolopoulos
Open University of Cyprus, Cyprus
,
Ernesto Damiani
Khalifa University of Science and Technology, UAE
,
Djamal Benslimane
Lyon 1 University, France
,
Program Chairs:
Ladjel Bellatreche
LIAS/ISAE-ENSMA, France
,
Tadeusz Morzy
Poznan University of Technology, Poland

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Horizon 2020 Framework Programme

Conference

MEDES '20

MEDES '20: 12th International Conference on Management of Digital EcoSystems

November 2 - 4, 2020

Virtual Event, United Arab Emirates

Acceptance Rates

MEDES '20 Paper Acceptance Rate 19 of 27 submissions, 70%;

Overall Acceptance Rate 267 of 682 submissions, 39%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
223
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Mikina FZuk PRzadca K(2024)sAirflow: Adopting Serverless in a Legacy Workflow SchedulerEuro-Par 2024: Parallel Processing10.1007/978-3-031-69577-3_18(254-268)Online publication date: 26-Aug-2024
https://doi.org/10.1007/978-3-031-69577-3_18
Cheng JZhao P(2022)Sustainable Big Data Analytics Process Pipeline Using Apache EcosystemEncyclopedia of Data Science and Machine Learning10.4018/978-1-7998-9220-5.ch073(1247-1259)Online publication date: 14-Oct-2022
https://doi.org/10.4018/978-1-7998-9220-5.ch073
Tahmasebi SLayegh ANikolov NPayberah ADinh KMitrovic VRoman DMatskin M(2022)Dataclouddsl: Textual and Visual Presentation of Big Data Pipelines2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00183(1165-1171)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00183
Mehran NSamani ZKimovski DProdan R(2022)Matching-based Scheduling of Asynchronous Data Processing Workflows on the Computing Continuum2022 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER51413.2022.00021(58-70)Online publication date: Sep-2022
https://doi.org/10.1109/CLUSTER51413.2022.00021
Ciavotta MCutrona VDe Paoli FNikolov NPalmonari MRoman D(2022)Supporting Semantic Data Enrichment at ScaleTechnologies and Applications for Big Data Value10.1007/978-3-030-78307-5_2(19-39)Online publication date: 29-Apr-2022
https://doi.org/10.1007/978-3-030-78307-5_2
Corodescu ANikolov NKhan ASoylu AMatskin MPayberah ARoman D(2021)Big Data Workflows: Locality-Aware Orchestration Using Software ContainersSensors10.3390/s2124821221:24(8212)Online publication date: 8-Dec-2021
https://doi.org/10.3390/s21248212
Corodescu ANikolov NKhan ASoylu AMatskin MPayberah ARoman DChbeir RManolopoulos YBellatreche LBenslimane DIvanovic MMaamar Z(2021)Locality-Aware Workflow Orchestration for Big DataProceedings of the 13th International Conference on Management of Digital EcoSystems10.1145/3444757.3485106(62-70)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1145/3444757.3485106
Nikolov NDessalk YKhan ASoylu AMatskin MPayberah ARoman D(2021)Conceptualization and scalable execution of big data workflows using domain-specific languages and software containersInternet of Things10.1016/j.iot.2021.10044016(100440)Online publication date: Dec-2021
https://doi.org/10.1016/j.iot.2021.100440

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions

Locality-Aware Workflow Orchestration for Big Data

Scalable Big Data Architecture: A practitioners guide to choosing relevant Big Data architecture