research-article

Event-based systems: opportunities and challenges at exascale

Authors:

Greg Eisenhauer,

Karsten SchwanAuthors Info & Claims

DEBS '09: Proceedings of the Third ACM International Conference on Distributed Event-Based Systems

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/1619258.1619261

Published: 06 July 2009 Publication History

Abstract

Streaming data models have been shown to be useful in many applications requiring high-performance data exchange. Application-level overlay networks are a natural way to realize these applications' data flows and their internal computations, but existing middleware is not designed to scale to the data rates and low overhead computations necessary for the high performance domain. This paper describes EVPath, a middleware infrastructure that supports the construction and management of overlay networks that can be customized both in topology and in the data manipulations being performed. Extending from a previous high-performance publish-subscribe system, EVPath not only provides for the low overhead movement and in-line processing of large data volumes, but also offers the flexibility needed to support the varied data flow and control needs of alternative higher-level streaming models. We explore some of the challenges of high performance event systems, including those experienced when operating an event infrastructure used to transport IO events at the scale of hundred+ thousand nodes. Specifically, when transporting output data from a large-scale simulation running on the ORNL Cray Jaguar petascale machine, a surprising new issue seen in experimentation at scale is the potential for strong perturbation of running applications from inappropriate speeds at which IO is performed. This requires the IO system's event transport to be explicitly scheduled to constrain resource competition, in addition to dynamically setting and changing the topologies of event delivery.

References

[1]

H. Abbasi, M. Wolf, F. Zheng, G. Eisenhauer, S. Klasky, and K. Schwan. Scalable data staging services for petascale applications. In hpdc2009, 2009.

Digital Library

[2]

S. Agarwala, G. Eisenhauer, and K. Schwan. Lightweight morphing support for evolving data exchanges in distributed applications. In Proc. of the 25th International Conference on Distributed Computer Systems (ICDCS-25), June 2005.

Digital Library

[3]

K. Birman, A. Schiper, and P. Stephenson. Lightweight Causal and Atomic Group Multicast. ACM Transactions on Computer Systems, 9(3), Aug. 1991.

Digital Library

[4]

D. Box, L. Cabrera, C. Critchley, F. Curbera, D. Ferguson, A. Geller, S. Graham, D. Hull, G. Kakivaya, A. Lewis, et al. Web Services Eventing (WS-Eventing). W3C Member Submission, 2006.

[5]

L. Brenna, A. Demers, J. Gehrke, M. Hong, J. Ossher, B. Panda, M. Riedewald, M. Thatte, and W. White. Cayuga: a high-performance event processing engine. In SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pages 1100--1102, New York, NY, USA, 2007. ACM.

Digital Library

[6]

Z. Cai, G. Eisenhauer, Q. He, V. Kumar, K. Schwan, and M. Wolf. Iq-services: Network-aware middleware for interactive large-data applications. Concurrency&Computation. Practice and Exprience Journal, 2005.

[7]

A. Carzaniga, D. S. Rosenblum, and A. L. Wolf. Challenges for distributed event services: Scalability vs. expressiveness. In Proc. of Engineering Distributed Objects (EDO '99), ICSE 99 Workshop, May 1999.

[8]

A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White. A general algebra and implementation for monitoring event streams. Technical Report TR2005-1997, Cornell University, 2005.

[9]

A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White. Towards expressive publish/subscribe systems. Lecture Notes in Computer Science, 3896:627, 2006.

Digital Library

[10]

G. Eisenhauer, F. Bustamante, and K. Schwan. Publish-subscribe for high-performance computing. IEEE Internet Computing - Asynchronous Middleware and Services, 10(1):8--25, January 2006.

Digital Library

[11]

G. Eisenhauer and L. K. Daley. Fast heterogenous binary data interchange. In Proc. of the Heterogeneous Computing Workshop (HCW2000), May 3--5 2000.

Digital Library

[12]

P. Eugster. Type-based publish/subscribe: Concepts and experiences. ACM Trans. Program. Lang. Syst., 29(1):6, 2007.

Digital Library

[13]

P. Eugster, P. Felber, R. Guerraoui, and A.-M. Kerrmarec. The many faces of publish/subscribe. Tech. Report DSC-ID:200104, École Polythechnique Fédérale de Lausanne, Lausanne, France, January 2001.

[14]

I. Foster, K. Czajkowski, D. Ferguson, J. Frey, S. Graham, T. Maguire, D. Snelling, and S. Tuecke. Modeling and managing state in distributed systems: The role of OGSI and WSRF. Proceedings of the IEEE, 93(3):604--612, 2005.

[15]

M. K. Gardner, W.-c. Feng, J. S. Archuleta, H. Lin, and X. Ma. Parallel Genomic Sequence-Searching on an Ad-Hoc Grid: Experiences, Lessons Learned, and Implications. In ACM/IEEE SC\06: The International Conference on High-Performance Computing, Networking, Storage, and Analysis, Tampa, FL, November 2006. Best Paper Nominee.

Digital Library

[16]

A. Grimshaw, M. Morgan, D. Merrill, H. Kishimoto, A. Savva, D. Snelling, C. Smith, and D. Berry. An open grid services architecture primer. Computer, 42(2):27--34, 2009.

Digital Library

[17]

Hewlet-Packard. The netperf network performance benchmark, http://www.netperf.org.

[18]

Y. Huang, A. Slominski, C. Herath, and D. Gannon. Ws-messenger: A web services based messaging system for service-oriented grid computing. In 6th IEEE International Symposium on Cluster Computing and the Grid (CCGrid06), 2006.

Digital Library

[19]

N. Jiang, A. Quiroz, C. Schmidt, and M. Parashar. Meteor: a middleware infrastructure for content-based decoupled interactions in pervasive grid environments. Concurr. Comput.: Pract. Exper., 20(12):1455--1484, 2008.

Digital Library

[20]

V. Kumar, B. F. Cooper, Z. Cai, G. Eisenhauer, and K. Schwan. Resource-aware distributed stream management using dynamic overlays. In Proceedings of the 25th IEEE International Conference on Distributed Computing Systems (ICDCS-2005), 2005.

Digital Library

[21]

J. Lofstead, K. Schwan, S. Klasky, N. Podhorszki, and C. Jin. Flexible io and integration for scientific codes through the adaptable io system (adios). In Challenges of Large Applications in Distributed Environments (CLADE), 2008.

Digital Library

[22]

L. Oliker, J. Carter, michael Wehner, A. Canning, S. Ethier, A. Mirin, G. Bala, D. parks, patrick Worley Shigemune Kitawaki, and Y. Tsuda. Leading computational methods on scalar and vector hec platforms. In Proceedings of SuperComputing 2005, 2005.

Digital Library

[23]

O. M. G. (OMG). Notification service specification 1.0. ftp://www.omg.org/pub/doc/formal/00-06-20.pdf, June 2000.

[24]

O. M. G. (OMG). Event service specification 1.1. ftp://www.omg.org/pub/docs/formal/01-03-01.pdf, March 2001.

[25]

B. Segall and D. Arnold. Elvin has left the building: A publish/subscribe notification service with quenching. In Proc. of the AUUG (Australian users group for Unix and Open Systems) 1997 Conference, September 1997.

[26]

D. Skeen. The enterprise-capable publish-subscribe server. http://www.vitria.com.

[27]

R. Strom, G. Banavar, T. Chandra, M. Kaplan, K. Miller, B. Mukherjee, D. Sturman, and M. Ward. Gryphon: An information flow based approach to message brokering. In International Symposium on Software Reliability Engineering '98 Fast abstract, 1998.

[28]

Tibco. TIB/rendezvous. http://www.rv.tibco.com/rvwhitepaper.html.

[29]

M. Wolf, H. Abbasi, B. Collins, D. Spain, and K. Schwan. Service Augmentation for High End Interactive Data Services. In Proceedings of Cluster 2005, 2005.

[30]

M. Wolf, Z. Cai, W. Huang, and K. Schwan. Smartpointers: Personalized scientific data portals in your hand. In Proceedings of the Proceedings of the IEEE/ACM SC2002 Conference, page 20. IEEE Computer Society, 2002.

Digital Library

Cited By

Huang DQin ZLiu QPodhorszki NKlasky S(2022)Identifying Challenges and Opportunities of In-Memory Computing on Large HPC SystemsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2022.02.002Online publication date: Feb-2022
https://doi.org/10.1016/j.jpdc.2022.02.002
Wolf MLogan JMehta KJacobson DCashman MWalker AEisenhauer GWidener PClif A(2021)Reusability First: Toward FAIR Workflows2021 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/Cluster48925.2021.00053(444-455)Online publication date: Sep-2021
https://doi.org/10.1109/Cluster48925.2021.00053
Huang DQin ZLiu QPodhorszki NKlasky S(2020)A Comprehensive Study of In-Memory Computing on Large HPC Systems2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS47774.2020.00045(987-997)Online publication date: Nov-2020
https://doi.org/10.1109/ICDCS47774.2020.00045
Show More Cited By

Index Terms

Event-based systems: opportunities and challenges at exascale
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Communications management

Recommendations

On Event-Based Middleware for Location-Aware Mobile Applications

As mobile applications become more widespread, programming paradigms and middleware architectures designed to support their development are becoming increasingly important. The event-based programming paradigm is a strong candidate for the development ...
Event Services in High Performance Systems

The Internet and the Grid are changing the face of high performance computing. Rather than tightly-coupled SPMD-style components running in a single cluster, on a parallel machine, or even on the Internet programmed in MPI, applications are evolving ...
Efficient message transport interface between agent framework and event service

Multi-agent techniques have been continuously evolving as ubiquitous and pervasive computing emerges as a key post-internet paradigm. In the ubiquitous environment, the applications need to exchange the requests asynchronously using an event-based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DEBS '09: Proceedings of the Third ACM International Conference on Distributed Event-Based Systems

July 2009

292 pages

ISBN:9781605586656

DOI:10.1145/1619258

Conference Chairs:
Aniruddha Gokhale
Vanderbilt University
,
Douglas C. Schmidt
Vanderbilt University

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

DEBS '09

Sponsor:

DEBS '09: The 3rd ACM International Conference on Distributed Event-based Systems

July 6 - 9, 2009

Tennessee, Nashville

Acceptance Rates

Overall Acceptance Rate 145 of 583 submissions, 25%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
487
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang DQin ZLiu QPodhorszki NKlasky S(2022)Identifying Challenges and Opportunities of In-Memory Computing on Large HPC SystemsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2022.02.002Online publication date: Feb-2022
https://doi.org/10.1016/j.jpdc.2022.02.002
Wolf MLogan JMehta KJacobson DCashman MWalker AEisenhauer GWidener PClif A(2021)Reusability First: Toward FAIR Workflows2021 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/Cluster48925.2021.00053(444-455)Online publication date: Sep-2021
https://doi.org/10.1109/Cluster48925.2021.00053
Huang DQin ZLiu QPodhorszki NKlasky S(2020)A Comprehensive Study of In-Memory Computing on Large HPC Systems2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS47774.2020.00045(987-997)Online publication date: Nov-2020
https://doi.org/10.1109/ICDCS47774.2020.00045
Wolf MChoi JEisenhauer GEthier SHuck KKlasky SLogan JMalony AWood CDominski JMerlo G(2019)Scalable Performance Awareness for In Situ Scientific Applications2019 15th International Conference on eScience (eScience)10.1109/eScience.2019.00037(266-276)Online publication date: Sep-2019
https://doi.org/10.1109/eScience.2019.00037
Zhang XZheng FNguyen B(2019)DeStagerDistributed and Parallel Databases10.1007/s10619-018-7235-337:1(209-231)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1007/s10619-018-7235-3
Xue HZhang YZeng WZhou X(2018)Application of Publish/Subscribe Pattern on Data Transmission between Hybrid NetworksProceedings of the 3rd International Conference on Intelligent Information Processing10.1145/3232116.3232127(63-68)Online publication date: 19-May-2018
https://dl.acm.org/doi/10.1145/3232116.3232127
Fu YLi FSong FChen ZZhao MChandra ARamakrishnan L(2018)Performance analysis and optimization of in-situ integration of simulation with data analysisProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208049(192-205)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208049
Choi JChurchill MDavis PDi SEisenhauer GEthier SFoster IGeveci BGuo HHuck KJenko FChang CKim MKress JKu SLiu QLogan JMalony AMehta KMoreland KMunson TParashar MDominski JPeterka TPodhorszki NPugmire DTugluk OWang RWhitney BWolf MWood CKlasky SMerlo GSuchyta EAinsworth MAllen BCappello F(2018)Coupling Exascale Multiphysics Applications: Methods and Lessons Learned2018 IEEE 14th International Conference on e-Science (e-Science)10.1109/eScience.2018.00133(442-452)Online publication date: Oct-2018
https://doi.org/10.1109/eScience.2018.00133
Klasky SWolf MAinsworth MAtkins CChoi JEisenhauer GGeveci BGodoy WKim MKress JKurc TLiu QLogan JMaccabe AMehta KOstrouchov GParashar MPodhorszki NPugmire DSuchyta EWan LWang R(2018)A View from ORNL: Scientific Data Research Opportunities in the Big Data Age2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2018.00136(1357-1368)Online publication date: Jul-2018
https://doi.org/10.1109/ICDCS.2018.00136
Dorier MDreher MPeterka TRoss RMohror KWelch B(2017)CoSSProceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems10.1145/3149393.3149396(13-18)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3149393.3149396
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten