Article

Free access

Experiments with in-transit processing for data intensive grid workflows

Authors:

Manish Parashar,

Scott KlaskyAuthors Info & Claims

GRID '07: Proceedings of the 8th IEEE/ACM International Conference on Grid Computing

Pages 193 - 200

https://doi.org/10.1109/GRID.2007.4354133

Published: 19 September 2007 Publication History

Abstract

Efficient and robust data streaming and in-transit data manipulations are critical requirements of emerging scientific and engineering application workflows, which are based on seamless interactions and coupling between geographically distributed application components. The overall goal of this research is to address these requirements and develop a data streaming and in-transit data manipulation service. In this paper, we experimentally investigate reactive management strategies for in-transit data manipulation, as well as cooperative end-to-end management for wide-area data-streaming and in-transit data manipulation for data-intensive scientific and engineering workflows.

References

[1]

V. Bhat, M. Parashar, M. Khandekar, N. Kandasamy, and S. Klasky, "A Self-Managing Wide-Area Data Streaming Service using Model-based Online Control," in 7th IEEE International Conference on Grid Computing (Grid 2006). Barcelona, Spain: IEEE Computer Society, 2006, pp. 176-183.

Digital Library

[2]

S. Klasky, B. Ludäscher, and M. Parashar, "The Center for Plasma Edge Simulation Workflow Requirements," in 22nd International Conference on Data Engineering Workshops (ICDEW'06). Atlanta, GA, USA: IEEE Computer Society, 2006, p. 73.

Digital Library

[3]

H. Liu, "Accord: A Programming System for Autonomic Self-Managing Applications," Ph.D. dissertation, Rutgers University, 2005.

Digital Library

[4]

Lawrence-Berkeley-National-Laboratory, "Energy Sciences Network (ESNET-4)," 2006. {Online}. Available: http://www.es.net/

[5]

J. Plank and M. Beck, "The Logistical Computing Stack - A Design For Wide-Area, Scalable, Uninterruptible Computing," in Dependable Systems and Networks, Workshop on Scalable, Uninterruptible Computing (DNS 2002), Bethesda, Maryland, USA, 2002.

[6]

M. Eriksen, "Trickle: A Userland Bandwidth Shaper for Unix-like Systems," in USENIX Annual Technical Conference (USENIX'05). Anaheim, CA, USA: SAGE, 2005, pp. 61-70.

Digital Library

[7]

A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, and S. Tuecke, "The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets," in Network Storage Symposium (NetStore'99), vol. 23. Seattle, WA, USA: Journal of Network and Computer Applications, 1999, pp. 187-200.

[8]

A. Shoshani, A. Sim, and J. Gu, "Storage Resource Managers: Middleware Components for Grid Storage," in 10th Goddard Conference on Mass Storage Systems and Technologies, 19th IEEE Symposium on Mass Storage Systems, College Park, MD, USA, 2002, p. 14.

[9]

W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, "The Globus Striped Gridftp Framework and Server," in Super Computing (SC'05). Seattle, WA, USA: IEEE Computer Society, 2005, pp. 54-64.

Digital Library

[10]

J. S. Plank, M. Beck, W. R. Elwasif, T. Moore, M. Swany, and R. Wolski, "The Internet Backplane Protocol: Storage in the Network," in NetStore99, Seattle, WA, USA, 1999.

[11]

A. Hanushevsky, "bbcp peer to peer cp program," 2002. {Online}. Available: http://www.slac.stanford.edu/~abh/bbcp/

[12]

Y. Gu and R. Grossman, "SABUL: A Transport Protocol for Grid Computing," Journal of Grid Computing, vol. 1, no. 4, pp. 377-386, 2003.

[13]

M. Beynon, C. Chang, U. Catalyurek, T. Kurc, A. Sussman, R. Andrade, H. and Ferreira, and J. Saltz, "Processing Large-Scale Multi-Dimensional Data in Parallel and Distributed Environments," Parallel dataintensive algorithms and applications, vol. 28, no. 5, pp. 827-859, 2002.

Digital Library

[14]

G. Singh, C. Kesselman, and E. Deelman, "Optimizing Grid-based Workflow Execution," Journal of Grid Computing, vol. 3, no. 3-4, pp. 201-219, September 2005.

[15]

J. In, P. Avery, R. Cavanaugh, L. Chitnis, M. Kulkarni, and S. Ranka, "SPHINX: A Fault-Tolerant System for Scheduling in Dynamic Grid Environments," in 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05). Denver, Colorado, USA: IEEE Computer Society, 2005, p. 12.2.

Digital Library

[16]

S. Venugopal, R. Buyya, and L. Winton, "A Grid Service Broker for Scheduling e-Science Applications on Global Data Grids," Concurrency and Computation: Practice & Experience, vol. 18, no. 6, pp. 685 - 699, May 2006.

Digital Library

[17]

G. v. Laszewski, K. Amin, and et. al, "GridAnt: A Client-Controllable Grid Workflow System," in 37th Hawaii International Conference on System Science (HICSS'04), Waikoloa, Big Island, Hawaii, 2004, p. 10.

Digital Library

[18]

R. D. Stevens, A. J. Robinson, and C. A. Goble, "myGrid: Personalised Bioinformatics on the Information Grid," in 11th International Conference on Intelligent Systems in Molecular Biology, vol. 19. Brisbane, Australia: Bioinformatics, Oxford University Press, 2003, pp. i302-i304.

[19]

X. Ma, J. Lee, and M. Winslett, "High-Level Buffering for Hiding Periodic Output Cost in Scientific Simulations," IEEE Transactions Parallel Distributed Systems, vol. 17, no. 3, pp. 193-204, March 2006.

Digital Library

[20]

Z. Cai, V. Kumar, and K. Schwan, "IQ-Paths: Predictably High Performance Data Streams Across Dynamic Network Overlays," Journal of Grid Computing, vol. 5, no. 2, pp. 129-150, June 2007.

Cited By

Docan CZhang FJin TBui HSun QCummings JPodhorszki NKlasky SParashar M(2015)ActiveSpacesConcurrency and Computation: Practice & Experience10.1002/cpe.340727:14(3724-3745)Online publication date: 25-Sep-2015
https://dl.acm.org/doi/10.1002/cpe.3407
Aktas MHaldeman GParashar MBalman MByna STierney B(2014)Flexible scheduling and control of bandwidth and in-transit services for end-to-end application workflowsProceedings of the Fourth International Workshop on Network-Aware Data Management10.1109/NDM.2014.9(28-31)Online publication date: 16-Nov-2014
https://dl.acm.org/doi/10.1109/NDM.2014.9
Jha SParashar MRana O(2009)Self-adaptive architectures for autonomic computational scienceProceedings of the First international conference on Self-organizing architectures10.5555/1880569.1880581(177-197)Online publication date: 14-Sep-2009
https://dl.acm.org/doi/10.5555/1880569.1880581
Show More Cited By

Experiments with in-transit processing for data intensive grid workflows

Recommendations

Brokering multi-grid workflows in the P-GRADE portal
Euro-Par'06: Proceedings of the CoreGRID 2006, UNICORE Summit 2006, Petascale Computational Biology and Bioinformatics conference on Parallel processing

Grid computing has gone through some generations and as a result only a few widely used middleware architectures remain. The Globus Toolkit is the most widespread middleware in most of the current production grid systems, but the LCG-2 middleware ...
Monitoring of Grid scientific workflows
Large-Scale Programming Tools and Environments

Scientific workflows are a means of conducting in silico experiments in modern computing infrastructures for e-Science, often built on top of Grids. Monitoring of Grid scientific workflows is essential not only for performance analysis but also to ...
A Data-Intensive Workflow Scheduling Algorithm for Grid Computing
CHINAGRID '09: Proceedings of the 2009 Fourth ChinaGrid Annual Conference

The data-intensive workflow in scientific and enterprise grids has gained popularity in recent times. Data-intensive workflow needs to access, process and transfer large datasets that may each be replicated on different data hosts. Because of the large ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

GRID '07: Proceedings of the 8th IEEE/ACM International Conference on Grid Computing

September 2007

339 pages

ISBN:9781424415595

Publisher

IEEE Computer Society

United States

Publication History

Published: 19 September 2007

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
109
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)3

Reflects downloads up to 30 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Docan CZhang FJin TBui HSun QCummings JPodhorszki NKlasky SParashar M(2015)ActiveSpacesConcurrency and Computation: Practice & Experience10.1002/cpe.340727:14(3724-3745)Online publication date: 25-Sep-2015
https://dl.acm.org/doi/10.1002/cpe.3407
Aktas MHaldeman GParashar MBalman MByna STierney B(2014)Flexible scheduling and control of bandwidth and in-transit services for end-to-end application workflowsProceedings of the Fourth International Workshop on Network-Aware Data Management10.1109/NDM.2014.9(28-31)Online publication date: 16-Nov-2014
https://dl.acm.org/doi/10.1109/NDM.2014.9
Jha SParashar MRana O(2009)Self-adaptive architectures for autonomic computational scienceProceedings of the First international conference on Self-organizing architectures10.5555/1880569.1880581(177-197)Online publication date: 14-Sep-2009
https://dl.acm.org/doi/10.5555/1880569.1880581
Jha SParashar MRana OGermain-Renaud C(2009)Investigating autonomic behaviours in grid-basedcomputational science applicationsProceedings of the 6th international conference industry session on Grids meets autonomic computing10.1145/1555301.1555305(29-38)Online publication date: 15-Jun-2009
https://dl.acm.org/doi/10.1145/1555301.1555305

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents