research-article

Damaris: Addressing Performance Variability in Data Management for Post-Petascale Simulations

Authors:

Matthieu Dorier,

Gabriel Antoniu,

Franck Cappello,

Robert Sisneros,

Leigh OrfAuthors Info & Claims

ACM Transactions on Parallel Computing (TOPC), Volume 3, Issue 3

Article No.: 15, Pages 1 - 43

https://doi.org/10.1145/2987371

Published: 25 October 2016 Publication History

Abstract

With exascale computing on the horizon, reducing performance variability in data management tasks (storage, visualization, analysis, etc.) is becoming a key challenge in sustaining high performance. This variability significantly impacts the overall application performance at scale and its predictability over time.

In this article, we present Damaris, a system that leverages dedicated cores in multicore nodes to offload data management tasks, including I/O, data compression, scheduling of data movements, in situ analysis, and visualization. We evaluate Damaris with the CM1 atmospheric simulation and the Nek5000 computational fluid dynamic simulation on four platforms, including NICS’s Kraken and NCSA’s Blue Waters. Our results show that (1) Damaris fully hides the I/O variability as well as all I/O-related costs, thus making simulation performance predictable; (2) it increases the sustained write throughput by a factor of up to 15 compared with standard I/O approaches; (3) it allows almost perfect scalability of the simulation up to over 9,000 cores, as opposed to state-of-the-art approaches that fail to scale; and (4) it enables a seamless connection to the VisIt visualization software to perform in situ analysis and visualization in a way that impacts neither the performance of the simulation nor its variability.

In addition, we extended our implementation of Damaris to also support the use of dedicated nodes and conducted a thorough comparison of the two approaches—dedicated cores and dedicated nodes—for I/O tasks with the aforementioned applications.

References

[1]

Hasan Abbasi, Matthew Wolf, Greg Eisenhauer, Scott Klasky, Karsten Schwan, and Fang Zheng. 2009. DataStager: Scalable data staging services for petascale applications. In Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing (HPDC’09). ACM, New York, NY, 39--48.

Digital Library

[2]

Nawab Ali, Philip Carns, Kamil Iskra, Dries Kimpe, Samuel Lang, Robert Latham, Robert Ross, Lee Ward, and Ponnuswamy Sadayappan. 2009. Scalable I/O forwarding framework for high-performance computing systems. In Proceedings of the IEEE International Conference on Cluster Computing and Workshops, 2009 (CLUSTER’09).

[3]

ANL. 2015. MPICH. Retrieved from http://www.mpich.org.

[4]

George H. Bryan and J. Michael Fritsch. 2002. A benchmark simulation for moist nonhydrostatic numerical models. Monthly Weather Review 130, 12 (2002), 2917--2928.

[5]

Philip H. Carns, Walter B. Ligon, III, Robert B. Ross, and Rajeev Thakur. 2000. PVFS: A parallel file system for Linux clusters. In Proceedings of the 4th Annual Linux Showcase 8 Conference - Volume 4. USENIX Association, Berkeley, CA.

Digital Library

[6]

Christian M. Chilan, M. Yang, Albert Cheng, and Leon Arber. 2006. Parallel I/O performance study with HDF5, a scientific data package. TeraGrid 2006: Advancing Scientific Discovery (2006).

[7]

Hank Childs, David Pugmire, Sean Ahern, Brad Whitlock, Mark Howison, Prabhat, Gunther Weber, and E. Wes Bethel. 2010. Extreme scaling of production visualization software on diverse architectures. IEEE Computer Graphics and Applications 30, 3 (2010), 22--31.

Digital Library

[8]

Ciprian Docan, Manish Parashar, and Scott Klasky. 2010. Enabling high-speed asynchronous data extraction and transfer using DART. Concurrency and Computation: Practice and Experience 22, 9 (2010), 1181--1204.

Digital Library

[9]

Stephanie Donovan, Gerrit Huizenga, Andrew J. Hutton, C. Craig Ross, Martin K. Petersen, and Philip Schwan. 2003. Lustre: Building a file system for 1000-node clusters. In Proceedings of the 2003 Linux Symposium. Citeseer.

[10]

Matthieu Dorier, Gabriel Antoniu, Franck Cappello, Marc Snir, and Leigh Orf. 2012a. Damaris: How to efficiently leverage multicore parallelism to achieve scalable, jitter-free I/O. In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER’12). IEEE.

Digital Library

[11]

Matthieu Dorier, Gabriel Antoniu, Franck Cappello, Marc Snir, and Leigh Orf. 2012b. Damaris: Leveraging Multicore Parallelism to Mask I/O Jitter. Research Report RR-7706. INRIA. 36 pages.

[12]

Matthieu Dorier, Gabriel Antoniu, Robert Ross, Dries Kimpe, and Shadi Ibrahim. 2014. CALCioM: Mitigating I/O interference in HPC systems through cross-application coordination. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS’14).

Digital Library

[13]

Matthieu Dorier, R. Sisneros, Roberto, Tom Peterka, Gabriel Antoniu, and B. Semeraro, Dave. 2013. Damaris/Viz: A nonintrusive, adaptable and user-friendly in situ visualization framework. In Proceedings of the IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV’13). http://hal.inria.fr/hal-00859603.

[14]

Matthieu Dreher, Jessica Prevoteau-Jonquet, Mikael Trellet, Marc Piuzzi, Marc Baaden, Bruno Raffin, Nicolas Férey, Sophie Robert, and Sébastien Limet. 2014. Exaviz: A flexible framework to analyse, steer and interact with molecular dynamics simulations. Faraday Discussions (2014).

[15]

Matthieu Dreher and Bruno Raffin. 2014. A flexible framework for asynchronous in situ and in transit analytics for scientific simulations. ACM/IEEE International Symposium on Cluster, Cloud and Grid Computing (CCGrid’14).

Digital Library

[16]

David Ellsworth, Bryan Grenn, Chris Henze, Patrick Moran, and Timothy Sandstrom. 2006. Concurrent visualization in a production supercomputing environment. IEEE Transactions on Visualization and Computer Graphics (TVGC) 12, 5 (Sept.-- Oct. 2006), 997--1004.

Digital Library

[17]

ERDC DSRC. 2015. EzViz. http://daac.hpc.mil/software/ezViz/.

[18]

Aurélien Esnard, Nicolas Richart, and Olivier Coulaud. 2006. A steering environment for online parallel visualization of legacy parallel simulations. In Proceedings of the IEEE/ACM International Symposium on Distributed Simulation and Real-Time Applications (DS-RT’06). IEEE, 7--14.

Digital Library

[19]

Nathan Fabian, Kenneth Moreland, David Thompson, Andrew C. Bauer, Pat Marion, Nerk Geveci, Michel Rasquin, and Kenneth E. Jansen. 2011. The ParaView coprocessing library: A scalable, general purpose in situ visualization library. In Proceedings of the IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV’11).

[20]

P. F. Fischer, James W. Lottes, and Stefan G. Kerkemeier. 2008. Nek5000 Web page. http://nek5000. mcs.anl.gov.

[21]

Mike Folk, Albert Cheng, and Kim Yates. 1999. HDF5: A file format and I/O library for high performance computing applications. In Proceedings of the ACM/IEEE International Conference on High Performance Computing, Networking, Storage and Analysis (SC’99).

[22]

Jing Fu, Robert Latham, Misun Min, and Christopher D. Carothers. 2012. I/O threads to reduce checkpoint blocking for an electromagnetics solver on Blue Gene/P and Cray XK6. In Proceedings of the International Workshop on Runtime and Operating Systems for Supercomputers (ROSS’12).

Digital Library

[23]

Ana Gainaru, Guillaume Aupy, Anne Benoit, Franck Cappello, Yves Robert, and Marc Snir. 2014. Scheduling the I/O of HPC Applications Under Congestion. Rapport de recherche RR-8519. INRIA. Retrieved from http://hal.inria.fr/hal-00983789.

[24]

Grid’5000. 2015. Inria testbed. Retrieved from http://www.grid5000.fr.

[25]

Ajay Gulati, Arif Merchant, and Peter J. Varman. 2007. pClock: An arrival curve based approach for QoS guarantees in shared storage systems. In Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’07). ACM, New York, NY, 13--24.

Digital Library

[26]

HDF5. 2015. Hierarchical Data Format. http://www.hdfgroup.org/HDF5/.

[27]

Mark Hereld, Michael E. Papka, and V. Vishwanath. 2011. Toward simulation-time data analysis and I/O acceleration on leadership-class systems. In Proceedings of the IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV’11).

[28]

Florin Isaila, Javier Garcia Blas, Jesus Carretero, Robert Latham, and Robert Ross. 2010. Design and evaluation of multiple level data staging for Blue Gene systems. IEEE Transactions on Parallel and Distributed Systems (TPDS’10).

Digital Library

[29]

Christopher Johnson, Steven Parker, Charles Hansen, Gordgon Kindlmann, and Yarden Livnat. 1999. Interactive simulation and visualization. Computer 32, 12 (1999), 59--65.

Digital Library

[30]

Donald B. Johnston. 2014. First-of-a-Kind Supercomputer at Lawrence Livermore Available for Collaborative Research. Retrieved from https://www.llnl.gov/news/newsreleases/2014/May/NR-14-05-02.html.

[31]

KitWare. 2015a. eXtensible Data Model and Format (XDMF). Retrieved from http://www.xdmf.org/.

[32]

KitWare. 2015b. ParaView. Retrieved from http://www.paraview.org/.

[33]

Min Li, Sudharshan S. Vazhkudai, Ali R. Butt, Fei Meng, Xiaosong Ma, Youngjae Kim, Christian Engelmann, and Galen Shipman. 2010. Functional partitioning to optimize end-to-end performance on many-core architectures. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC’10). IEEE Computer Society.

Digital Library

[34]

Ning Liu, Jason Cope, Philip Carns, Christopher Carothers, Robert Ross, Gary Grider, Adam Crume, and Carlos Maltzahn. 2012. On the role of burst buffers in leadership-class storage systems. In Proceedings of the 28th IEEE Symposium on Mass Storage Systems and Technologies (MSST’12). IEEE.

[35]

LLNL. 2015. VisIt, Lawrence Livermore National Laboratory. Retrieved from https://wci.llnl.gov/simulation/computer-codes/visit.

[36]

Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten Schwan, and Matthew Wolf. 2010. Managing variability in the IO performance of petascale storage systems. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC’10). IEEE Computer Society, 12.

Digital Library

[37]

Jay F. Lofstead, Scott Klasky, Karsten Schwan, Norbert Podhorszki, and Chen Jin. 2008. Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments (CLADE’08). ACM, New York, NY.

Digital Library

[38]

Kwan-Liu Ma. 2009. In situ visualization at extreme scale: Challenges and opportunities. IEEE Computer Graphics and Applications 29, 6 (Nov.-- Dec. 2009), 14--19.

[39]

Kwan-Liu Ma, Chaoli Wang, Hongfeng Yu, and Anna Tikhonova. 2007. In-situ processing and visualization for ultrascale simulations. Journal of Physics: Conference Series 78, 1 (2007).

[40]

Xiaosong Ma, Jonghyun Lee, and Marianne Winslett. 2006. High-level buffering for hiding periodic output cost in scientific simulations. IEEE Transactions on Parallel and Distributed Systems (TPDS) 17 (2006), 193--204.

Digital Library

[41]

Preeti Malakar, Vijay Natarajan, and Sathish S. Vadhiyar. 2010. An adaptive framework for simulation and online remote visualization of critical climate applications in resource-constrained environments. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC’10). IEEE Computer Society, 11.

Digital Library

[42]

Adam Moody, Greg Bronevetsky, Kathryn Mohror, and Bronis R. de Supinski. 2010. Design, modeling, and evaluation of a scalable multi-level checkpointing system. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC’10). IEEE Computer Society, Los Alamitos, CA.

Digital Library

[43]

Kenneth Moreland, Ron Oldfield, Pat Marion, Sebastien Jourdain, Norbert Podhorszki, Venkatram Vishwanath, Nathan Fabian, Ciprian Docan, Manish Parashar, Mark Hereld, Michael E. Papka, and Scott Klasky. 2011. Examples of in transit visualization. In Proceedings of the 2nd International Workshop on Petascale Data Analytics: Challenges and Opportunities (PDAC’11). ACM.

Digital Library

[44]

NCSA. 2015. Blue Waters supercomputer, National Center for Supercomputing Applications. http://www.ncsa.illinois.edu/BlueWaters/.

[45]

NICS. 2015. Kraken supercomputer, National Institute for Computational Sciences. http://www.nics.tennessee.edu/computing-resources/kraken.

[46]

Arifa Nisar, Wei-keng Liao, and Alok Choudhary. 2008. Scaling parallel I/O performance through I/O delegate and caching system. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC’08).

Digital Library

[47]

Christina M. Patrick, Seung Woo Son, and Mahmut Kandemir. 2008. Comparative evaluation of overlap strategies with study of I/O overlap in MPI-IO. Operating Systems Review (SIGOPS) 42, 6 (Oct. 2008), 43--49.

Digital Library

[48]

Tom Peterka, Robert Ross, Wesley Kendall, Attila Gyulassy, Valerio Pascucci, Han-Wei Shen, Teng-Yok Lee, and Abon Chaudhuri. 2011. Scalable parallel building blocks for custom data analysis. In Proceedings of Large Data Analysis and Visualization Symposium (LDAV’11). Providence, RI.

[49]

Ramya Prabhakar, Sudharshan S. Vazhkudai, Youngjae Kim, Ali R. Butt, Min Li, and Mahmut Kandemir. 2011. Provisioning a multi-tiered data staging area for extreme-scale machines. In Proceedings of the 31st International Conference on Distributed Computing Systems (ICDCS’11).

Digital Library

[50]

Jean-Pierre Prost, Richard Treumann, Richard Hedges, Bin Jia, and Alice Koniges. 2001. MPI-IO/GPFS an optimized implementation of MPI-IO on top of GPFS. In Proceedings of the ACM/IEEE International Conference on High Performance Computing, Networking, Storage and Analysis (SC’01). IEEE Computer Society, Los Alamitos, CA.

Digital Library

[51]

Xing Pu, Ling Liu, Yiduo Mei, S. Sivathanu, Younggyun Koh, and C. Pu. 2010. Understanding performance interference of I/O workload in virtualized cloud environments. In Proceedings of the IEEE International Conference on Cloud Computing (Cloud’10). 51--58.

Digital Library

[52]

Michel Rasquin, Patrick Marion, Venkatram Vishwanath, Benjamin Matthews, Mark Hereld, Kenneth Jansen, Raymond Loy, Andrew Bauer, Min Zhou, and Onkar Sahni. 2011. Electronic poster: Co-visualization of full data and in situ data extracts from unstructured grid CFD at 160k cores. In ACM/IEEE SC Companion. ACM, 103--104.

Digital Library

[53]

Marzia Rivi, Luigi Calori, Giuseppa Muscianisi, and Vladimir Slavnic. 2011. In-situ visualization: State-of-the-art and some use cases. PRACE White Paper (2012). Retrieved from http://www.prace-ri.eu/ Visualisation.

[54]

Rutgers. 2015. DataSpace. http://www.dataspaces.org/.

[55]

Douglas C. Schmidt. 1995. Reactor - An object behavioral pattern for demultiplexing and dispatching handles for synchronous events.

[56]

William J. Schroeder, Lisa Avila, and William Hoffman. 2000. Visualizing with VTK: A tutorial. IEEE Computer Graphics and Applications 20, 5 (Sep.-- Oct. 2000), 20--27.

Digital Library

[57]

Hongzhang Shan and John Shalf. 2007. Using IOR to analyze the I/O performance for HPC platforms. In Proceedings of the Cray User Group Conference (CUG’07). Seattle, WA.

[58]

David Skinner and William Kramer. 2005. Understanding the causes of performance variability in HPC workloads. In Proceedings of the IEEE Workload Characterization Symposium (IISWC’05). IEEE Computer Society, 137--149.

[59]

N. T. B. Stone, D. Balog, B. Gill, B. Johanson, J. Marsteller, P. Nowoczynski, D. Porter, R. Reddy, J. R. Scott, D. Simmel, J. Sommerfield, K. Vargo, and C. Vizino. 2006. PDIO: High-performance remote file I/O for portals enabled compute nodes. In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA’06). http://www.scientificcommons.org/43489982.

[60]

Rajeev Thakur, William Gropp, and Ewing Lusk. 1999a. Data sieving and collective I/O in ROMIO. In Proceedings of the Symposium on the Frontiers of Massively Parallel Processing. 182.

Digital Library

[61]

Rajeev Thakur, William Gropp, and Ewing Lusk. 1999b. On implementing MPI-IO portably and with high performance. In Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems (IOPADS’99). ACM, 23--32.

Digital Library

[62]

Top500. 2015. Top500 List of Supercomputers. http://www.top500.org/.

[63]

Tiankai Tu, Hongfeng Yu, Leonardo Ramirez-Guzman, Jacobo Bielak, Omar Ghattas, Kwan-Liu Ma, and David R. O’Hallaron. 2006. From mesh generation to scientific visualization: An end-to-end approach to parallel supercomputing. In Proceedings of the ACM/IEEE International Conference on High Performance Computing, Networking, Storage and Analysis (SC’06). ACM, New York, NY, Article 91.

Digital Library

[64]

Unidata. 2015. NetCDF. http://www.unidata.ucar.edu/software/netcdf/.

[65]

Andrew Uselton, Mark Howison, Nicholas J. Wright, David Skinner, Noel Keen, John Shalf, Karen L. Karavanic, and Leonid Oliker. 2010. Parallel I/O performance: From events to ensembles. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS’10).

[66]

Matthew Wachs, Michael Abd-El-Malek, Eno Thereska, and Gregory R. Ganger. 2007. Argon: Performance insulation for shared storage servers. In Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST’07). USENIX Association, Berkeley, CA, 1. http://dl.acm.org/citation.cfm?id=1267903.1267908

Digital Library

[67]

Brad Whitlock, Jean M. Favre, and Jeremy S. Meredith. 2011. Parallel in situ coupling of simulation with a fully featured visualization system. In Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization (EGPGV’10). Eurographics Association.

Digital Library

[68]

Hongfeng Yu and Kwan-Liu Ma. 2005. A study of I/O methods for parallel visualization of large-scale data. Journal of Parallel Computing - Parallel Graphics and Visualization 31, 2 (2005), 167--183.

Digital Library

[69]

Hongfeng Yu, Chaoli Wang, R. W. Grout, J. H. Chen, and Kwan-Liu Ma. 2010. In situ visualization for large-scale combustion simulations. IEEE Computer Graphics and Applications 30, 3 (May-- June 2010), 45--57.

Digital Library

[70]

Fan Zhang, Solomon Lasluisa, Tong Jin, Ivan Rodero, Hoang Bui, and Manish Parashar. 2012a. In-situ feature-based objects tracking for large-scale scientific simulations. In ACM/IEEE SC Companion. IEEE.

Digital Library

[71]

Fan Zhang, Manish Parashar, Ciprian Docan, Scott Klasky, Norbert Podhorszki, and Hasan Abbasi. 2012b. Enabling in-situ execution of coupled scientific workflow on multi-core platform. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS’12). IEEE.

Digital Library

[72]

Xuechen Zhang, Kei Davis, and Song Jiang. 2011. QoS support for end users of I/O-intensive applications using shared storage systems. In Proceedings of the ACM/IEEE International Conference on High Performance Computing, Networking, Storage and Analysis (SC’11).

Digital Library

[73]

Fang Zheng, Hasan Abbasi, Jianting Cao, Jai Dayal, Karsten Schwan, Matthew Wolf, Scott Klasky, and Norbert Podhorszki. 2011. In-situ I/O processing: A case for location flexibility. In Proceedings of the 6th Workshop on Parallel Data Storage (PDSW’11). ACM, New York, NY, 37--42.

Digital Library

[74]

Fang Zheng, H. Abbasi, C. Docan, J. Lofstead, Qing Liu, S. Klasky, M. Parashar, N. Podhorszki, K. Schwan, and M. Wolf. 2010. PreDatA -- preparatory data analytics on peta-scale machines. In Proceedings of the IEEE International Symposium on Parallel Distributed Processing (IPDPS’10).

[75]

Fang Zheng, Jianting Cao, Jai Dayal, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Hasan Abbasi, Scott Klasky, and Norbert Podhorszki. 2011. High end scientific codes with computational I/O pipelines: Improving their end-to-end performance. In Proceedings of the 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities (PDAC’11). ACM, New York, NY, 23--28.

Digital Library

Cited By

Yildiz OMorozov DNigmetov ANicolae BPeterka T(2024)Wilkins: HPC in situ workflows made easyFrontiers in High Performance Computing10.3389/fhpcp.2024.14727192Online publication date: 20-Nov-2024
https://doi.org/10.3389/fhpcp.2024.1472719
Belghit HSpivak MDauchez MBaaden MJonquet-Prevoteau J(2024)From complex data to clear insights: visualizing molecular dynamics trajectoriesFrontiers in Bioinformatics10.3389/fbinf.2024.13566594Online publication date: 11-Apr-2024
https://doi.org/10.3389/fbinf.2024.1356659
Carretero Perez JCascajo ASingh DCarretero J(2024)Detecting interference between applications and improving the scheduling using malleable application clonesInternational Journal of High Performance Computing Applications10.1177/1094342023122089838:2(108-133)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1177/10943420231220898
Show More Cited By

Index Terms

Recommendations

Lessons Learned from Building In Situ Coupling Frameworks
ISAV2015: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

Over the past few years, the increasing amounts of data produced by large-scale simulations have motivated a shift from traditional offline data analysis to in situ analysis and visualization. In situ processing began as the coupling of a parallel ...
On the energy footprint of I/O management in Exascale HPC systems

The advent of unprecedentedly scalable yet energy hungry Exascale supercomputers poses a major challenge in sustaining a high performance-per-watt ratio. With I/O management acquiring a crucial role in supporting scientific simulations, various I/O ...
Damaris: How to Efficiently Leverage Multicore Parallelism to Achieve Scalable, Jitter-free I/O
CLUSTER '12: Proceedings of the 2012 IEEE International Conference on Cluster Computing

With exascale computing on the horizon, the performance variability of I/O systems represents a key challenge in sustaining high performance. In many HPC applications, I/O is concurrently performed by all processes, which leads to I/O bursts. This ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Parallel Computing

ACM Transactions on Parallel Computing Volume 3, Issue 3

December 2016

145 pages

ISSN:2329-4949

EISSN:2329-4957

DOI:10.1145/3012407

Editor:
Phillip B. Gibbons
Carnegie Mellon University, Pittsburgh, USA

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2016

Accepted: 01 August 2016

Revised: 01 February 2016

Received: 01 February 2015

Published in TOPC Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
243
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yildiz OMorozov DNigmetov ANicolae BPeterka T(2024)Wilkins: HPC in situ workflows made easyFrontiers in High Performance Computing10.3389/fhpcp.2024.14727192Online publication date: 20-Nov-2024
https://doi.org/10.3389/fhpcp.2024.1472719
Belghit HSpivak MDauchez MBaaden MJonquet-Prevoteau J(2024)From complex data to clear insights: visualizing molecular dynamics trajectoriesFrontiers in Bioinformatics10.3389/fbinf.2024.13566594Online publication date: 11-Apr-2024
https://doi.org/10.3389/fbinf.2024.1356659
Carretero Perez JCascajo ASingh DCarretero J(2024)Detecting interference between applications and improving the scheduling using malleable application clonesInternational Journal of High Performance Computing Applications10.1177/1094342023122089838:2(108-133)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1177/10943420231220898
Yildiz OGueroudji ABigot JRaffin BBadia RPeterka T(2024)Extreme-scale workflows: A perspective from the JLESC international communityFuture Generation Computer Systems10.1016/j.future.2024.07.041161(502-513)Online publication date: Dec-2024
https://doi.org/10.1016/j.future.2024.07.041
Gueroudji ABigot JRaffin BRoss R(2023)Dask-Extended External Tasks for HPC/ML In transit WorkflowsProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624151(831-838)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624151
Bez JByna SIbrahim S(2023)I/O Access Patterns in HPC Applications: A 360-Degree SurveyACM Computing Surveys10.1145/361100756:2(1-41)Online publication date: 15-Sep-2023
https://doi.org/10.1145/3611007
Bruder VLarsen MErtl TChilds HFrey S(2023)A Hybrid in Situ Approach for Cost Efficient Image Database GenerationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.316959029:9(3788-3798)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3169590
Martinelli ATorquati MAldinucci MColonnelli ICantalupo B(2023)CAPIO: a Middleware for Transparent I/O Streaming in Data- Intensive Workflows2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)10.1109/HiPC58850.2023.00031(153-163)Online publication date: 18-Dec-2023
https://doi.org/10.1109/HiPC58850.2023.00031
Dorier MWang ZRamesh SAyachit USnyder SRoss RParashar M(2023)Towards elastic in situ analysis for high-performance computing simulationsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.02.014177(106-116)Online publication date: Jul-2023
https://doi.org/10.1016/j.jpdc.2023.02.014
Honore VDo TPottier LDa Silva RDeelman ESuter F(2022)SIM-SITU: A Framework for the Faithful Simulation of in situ Processing2022 IEEE 18th International Conference on e-Science (e-Science)10.1109/eScience55777.2022.00032(182-191)Online publication date: Oct-2022
https://doi.org/10.1109/eScience55777.2022.00032
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents