AkôFlow: um Middleware para Execução de Workflows Científicos em Múltiplos Ambientes Conteinerizados
Resumo
Diversos workflows produzem um grande volume de dados e requerem técnicas de paralelismo e ambientes distribuídos para reduzir o tempo de execução. Esses workflows são executados por Sistemas de Workflow, que apoiam a execução eficiente, mas focam em ambientes específicos. A tecnologia de contêineres surgiu como solução para que uma aplicação execute em ambientes heterogêneos por meio da virtualização do SO. Embora existam soluções de gerenciamento e orquestração de contêineres, e.g., Kubernetes, elas não focam em workflows científicos. Neste artigo, propomos o AkôFlow, um middleware para execução paralela de workflows científicos em ambientes conteinerizados. O AkôFlow permite ao cientista explorar a execução paralela de atividades, com apoio à captura de proveniência. Avaliamos o AkôFlow com um workflow da astronomia e os resultados foram promissores.
Palavras-chave:
Contêiner, Workflows
Referências
Babuji, Y. N. et al. (2019). Parsl: Pervasive parallel programming in python. In Weissman, J. B., Butt, A. R., and Smirni, E., editors, HPDC’19, pages 25–36. ACM.
Burkat, K., Pawlik, M., Balis, B., Malawski, M., Vahi, K., Rynge, M., da Silva, R. F., and Deelman, E. (2021). Serverless containers – rising viable approach to scientific workflows. In eScience, pages 40–49.
Carrión, C. (2023). Kubernetes scheduling: Taxonomy, ongoing issues and challenges. ACM Comput. Surv., 55(7):138:1–138:37.
de Oliveira, D., Ocaña, K. A. C. S., Baião, F. A., and Mattoso, M. (2012). A provenance-based adaptive scheduling heuristic for parallel scientific workflows in clouds. J. Grid Comput., 10(3):521–552.
de Oliveira, D., Ogasawara, E. S., Baião, F. A., and Mattoso, M. (2010). Scicumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows. In CLOUD’10, pages 378–385.
de Oliveira, D., Silva, V., and Mattoso, M. (2015). How much domain data should be in provenance databases? In 7th USENIX Workshop on the Theory and Practice of Provenance (TaPP 15).
de Oliveira, D. C. M., Liu, J., and Pacitti, E. (2019). Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments. Synthesis Lectures on Data Management. Morgan & Claypool Publishers.
Deelman, E., da Silva, R. F., Vahi, K., Rynge, M., Mayani, R., Tanaka, R., Whitcup, W. R., and Livny, M. (2021). The pegasus workflow management system: Translational computer science in practice. J. Comput. Sci., 52:101200.
Freire, J., Koop, D., Santos, E., and Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in science & engineering, 10(3):11–21.
Guedes, T., Martins, L. B., Falci, M. L. F., Silva, V., Ocaña, K. A., Mattoso, M., Bedo, M., and de Oliveira, D. (2020). Capturing and analyzing provenance from spark-based scientific workflows with samba-rap. Future Generation Computer Systems, 112:658 – 669.
Jiang, Q., Lee, Y. C., and Zomaya, A. Y. (2017). Serverless execution of scientific workflows. In ICSOC 2017, pages 706–721. Springer.
Kunstmann, L., Pina, D., Oliveira, L., Oliveira, D., and Mattoso, M. (2022). Provdeploy: Explorando alternativas de conteinerização com proveniência para aplicações científicas com pad. In Anais do XXIII Simpósio em Sistemas Computacionais de Alto Desempenho, pages 49–60, Porto Alegre, RS, Brasil. SBC.
Kurtzer, G. M., Sochat, V., and Bauer, M. W. (2017). Singularity: Scientific containers for mobility of compute. PloS one, 12(5):e0177459.
Ogasawara, E. S., de Oliveira, D., Valduriez, P., Dias, J., Porto, F., and Mattoso, M. (2011). An algebraic approach for data-centric scientific workflows. Proc. VLDB Endow., 4(12):1328–1339.
Ogasawara, E. S., Dias, J., Silva, V., Chirigati, F. S., de Oliveira, D., Porto, F., Valduriez, P., and Mattoso, M. (2013). Chiron: a parallel engine for algebraic scientific workflows. Concurr. Comput. Pract. Exp., 25(16):2327–2341.
Sakellariou, R. et al. (2009). Mapping workflows on grid resources: Experiments with the montage workflow. In ERCIM W. Group on Grids, pages 119–132.
Shah, S. T., Lahaye, R. J. W. E., Kazmi, S. A. A., Chung, M. Y., and Hasan, S. F. (2014). Htcondor system for running extensive simulations related to D2D communication. In ICTC, pages 283–284. IEEE.
Silva, V., de Oliveira, D., Valduriez, P., and Mattoso, M. (2018). Dfanalyzer: runtime dataflow analysis of scientific applications using provenance. Proceedings of the VLDB Endowment, 11(12):2082–2085.
Struhár, V., Behnam, M., Ashjaei, M., and Papadopoulos, A. V. (2020). Real-time containers: A survey. In Fog-IoT, volume 80 of OASIcs, pages 7:1–7:9.
Teylo, L., de Paula Junior, U., et al. (2017). A hybrid evolutionary algorithm for task scheduling and data assignment of data-intensive scientific workflows on clouds. FGCS, 76:1–17.
Zheng, C., Tovar, B., and Thain, D. (2017). Deploying high throughput scientific workflows on container schedulers with makeflow and mesos. In CCGrid, CCGrid ’17, page 130–139. IEEE Press.
Burkat, K., Pawlik, M., Balis, B., Malawski, M., Vahi, K., Rynge, M., da Silva, R. F., and Deelman, E. (2021). Serverless containers – rising viable approach to scientific workflows. In eScience, pages 40–49.
Carrión, C. (2023). Kubernetes scheduling: Taxonomy, ongoing issues and challenges. ACM Comput. Surv., 55(7):138:1–138:37.
de Oliveira, D., Ocaña, K. A. C. S., Baião, F. A., and Mattoso, M. (2012). A provenance-based adaptive scheduling heuristic for parallel scientific workflows in clouds. J. Grid Comput., 10(3):521–552.
de Oliveira, D., Ogasawara, E. S., Baião, F. A., and Mattoso, M. (2010). Scicumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows. In CLOUD’10, pages 378–385.
de Oliveira, D., Silva, V., and Mattoso, M. (2015). How much domain data should be in provenance databases? In 7th USENIX Workshop on the Theory and Practice of Provenance (TaPP 15).
de Oliveira, D. C. M., Liu, J., and Pacitti, E. (2019). Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments. Synthesis Lectures on Data Management. Morgan & Claypool Publishers.
Deelman, E., da Silva, R. F., Vahi, K., Rynge, M., Mayani, R., Tanaka, R., Whitcup, W. R., and Livny, M. (2021). The pegasus workflow management system: Translational computer science in practice. J. Comput. Sci., 52:101200.
Freire, J., Koop, D., Santos, E., and Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in science & engineering, 10(3):11–21.
Guedes, T., Martins, L. B., Falci, M. L. F., Silva, V., Ocaña, K. A., Mattoso, M., Bedo, M., and de Oliveira, D. (2020). Capturing and analyzing provenance from spark-based scientific workflows with samba-rap. Future Generation Computer Systems, 112:658 – 669.
Jiang, Q., Lee, Y. C., and Zomaya, A. Y. (2017). Serverless execution of scientific workflows. In ICSOC 2017, pages 706–721. Springer.
Kunstmann, L., Pina, D., Oliveira, L., Oliveira, D., and Mattoso, M. (2022). Provdeploy: Explorando alternativas de conteinerização com proveniência para aplicações científicas com pad. In Anais do XXIII Simpósio em Sistemas Computacionais de Alto Desempenho, pages 49–60, Porto Alegre, RS, Brasil. SBC.
Kurtzer, G. M., Sochat, V., and Bauer, M. W. (2017). Singularity: Scientific containers for mobility of compute. PloS one, 12(5):e0177459.
Ogasawara, E. S., de Oliveira, D., Valduriez, P., Dias, J., Porto, F., and Mattoso, M. (2011). An algebraic approach for data-centric scientific workflows. Proc. VLDB Endow., 4(12):1328–1339.
Ogasawara, E. S., Dias, J., Silva, V., Chirigati, F. S., de Oliveira, D., Porto, F., Valduriez, P., and Mattoso, M. (2013). Chiron: a parallel engine for algebraic scientific workflows. Concurr. Comput. Pract. Exp., 25(16):2327–2341.
Sakellariou, R. et al. (2009). Mapping workflows on grid resources: Experiments with the montage workflow. In ERCIM W. Group on Grids, pages 119–132.
Shah, S. T., Lahaye, R. J. W. E., Kazmi, S. A. A., Chung, M. Y., and Hasan, S. F. (2014). Htcondor system for running extensive simulations related to D2D communication. In ICTC, pages 283–284. IEEE.
Silva, V., de Oliveira, D., Valduriez, P., and Mattoso, M. (2018). Dfanalyzer: runtime dataflow analysis of scientific applications using provenance. Proceedings of the VLDB Endowment, 11(12):2082–2085.
Struhár, V., Behnam, M., Ashjaei, M., and Papadopoulos, A. V. (2020). Real-time containers: A survey. In Fog-IoT, volume 80 of OASIcs, pages 7:1–7:9.
Teylo, L., de Paula Junior, U., et al. (2017). A hybrid evolutionary algorithm for task scheduling and data assignment of data-intensive scientific workflows on clouds. FGCS, 76:1–17.
Zheng, C., Tovar, B., and Thain, D. (2017). Deploying high throughput scientific workflows on container schedulers with makeflow and mesos. In CCGrid, CCGrid ’17, page 130–139. IEEE Press.
Publicado
14/10/2024
Como Citar
FERREIRA, Wesley; KUNSTMANN, Liliane; PAES, Aline; BEDO, Marcos; DE OLIVEIRA, Daniel.
AkôFlow: um Middleware para Execução de Workflows Científicos em Múltiplos Ambientes Conteinerizados. In: SIMPÓSIO BRASILEIRO DE BANCO DE DADOS (SBBD), 39. , 2024, Florianópolis/SC.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 27-39.
ISSN 2763-8979.
DOI: https://doi.org/10.5753/sbbd.2024.241126.