research-article

TCEP: Adapting to Dynamic User Environments by Enabling Transitions between Operator Placement Mechanisms

Authors:

Manisha Luthra,

Boris Koldehofe,

Pascal Weisenburger,

Guido Salvaneschi,

Raheel ArifAuthors Info & Claims

DEBS '18: Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems

Pages 136 - 147

https://doi.org/10.1145/3210284.3210292

Published: 25 June 2018 Publication History

Abstract

Operator placement has a profound impact on the performance of a distributed complex event processing system (DCEP). Since the behavior of a placement mechanism strongly depends on its environment; a single placement mechanism is often not enough to fulfill stringent performance requirements under environmental changes. In this paper, we show how DCEP can benefit from the adaptive use of multiple placement mechanisms. We propose Tcep, a DCEP system to integrate multiple placement mechanisms. By enabling transitions, Tcep can seamlessly exchange distinct operator mechanisms at runtime. We make two main contributions that are highly important for a cost-efficient transition: i) a transition strategy for efficiently scheduling state migrations and ii) a lightweight learning algorithm to adaptively select an appropriate placement mechanism as a consequence of a transition. Our evaluations for important decentralized placement mechanisms in the context of an IoT scenario show that transitions can better fulfill QoS demands in a dynamic environment. Thereby efficient scheduling of state migrations can help to faster complete transitions by up to 94 %.

References

[1]

EsperTech -- Esper. http://www.espertech.com/esper/, 2006. Accessed 03.05.2018.

[2]

Akka. http://akka.io/, 2009. Accessed 03.05.2018.

[3]

Docker -- Community edition. https://www.docker.com/community-edition, 2013. Accessed 03.05.2018.

[4]

Gartner Says 8.4 Billion Connected "Things" Will Be in Use in 2017. https://www.gartner.com/newsroom/id/3598917, 2017. Accessed 03.05.2018.

[5]

Y. Ahmad and U. Çetintemel. Network-aware query processing for stream-based applications. In Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30, VLDB '04, pages 456--467. VLDB Endowment, 2004.

Digital Library

[6]

L. Aniello, R. Baldoni, and L. Querzoni. Adaptive online scheduling in storm. In Proceedings of the 7th ACM International Conference on Distributed Event-based Systems, DEBS '13, pages 207--218, 2013.

Digital Library

[7]

M. Baguena, G. Samaras, A. Pamboris, M. L. Sichitiu, P. Pietzuch, and P. Manzoni. Towards enabling hyper-responsive mobile apps through network edge assistance. In 13th IEEE Annual Consumer Communications Networking Conference (CCNC), pages 399--404, 2016.

[8]

J. V. d. Bercken and B. Seeger. Query processing techniques for multiversion access methods. In Proceedings of the 22th International Conference on Very Large Data Bases, VLDB '96, pages 168--179. Morgan Kaufmann Publishers Inc., 1996.

Digital Library

[9]

T. Blickle and L. Thiele. A comparison of selection schemes used in evolutionary algorithms. Evolutionary Computation, 4(4):361--394, 1996.

Digital Library

[10]

V. Cardellini, V. Grassi, F. Lo Presti, and M. Nardelli. Optimal operator placement for distributed stream processing applications. In Proceedings of the 10th ACM International Conference on Distributed and Event-Based Systems, DEBS 2016, pages 69--80, 2016.

Digital Library

[11]

G. Cookson and B. Pishue. INRIX Global Traffic Scorecard. Technical report, 2018.

[12]

G. Cugola and A. Margara. Deployment strategies for distributed complex event processing. Computing, 95(2):129--156, 2013.

Digital Library

[13]

R. Dwarakanath, B. Koldehofe, and R. Steinmetz. Operator migration for distributed complex event processing in device-to-device based networks. In Proceedings of the 3rd ACM Workshop on Middleware for Context-Aware Applications in the IoT, M4IoT 2016, pages 13--18, 2016.

Digital Library

[14]

A. Frömmgen, S. Haas, M. Stein, R. Rehner, A. Buchmann, and M. Mühlhäuser. Always the best: Executing transitions between search overlays. In Proceedings of the 2015 European Conference on Software Architecture Workshops, ECSAW '15, pages 8:1--8:4, 2015.

Digital Library

[15]

A. Frömmgen, B. Richerzhagen, R. Julius, D. Hausheer, R. Steinmetz, and A. Buchmann. Towards the Description and Execution of Transitions in Networked Systems. In Intelligent Mechanisms for Network Configuration and Security, pages 17--29. Springer International Publishing, 2015.

[16]

T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer. Latency-aware elastic scaling for distributed data stream processing systems. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, DEBS '14, pages 13--22, 2014.

Digital Library

[17]

T. Heinze, M. Zia, R. Krahn, Z. Jerzak, and C. Fetzer. An adaptive replication scheme for elastic data stream processing systems. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, pages 150--161,2015.

Digital Library

[18]

J. O. Kephart and D. M. Chess. The vision of autonomic computing. Computer, 36(1):41--50, 2003.

Digital Library

[19]

G. G. Koch, B. Koldehofe, and K. Rothermel. Cordies: expressive event correlation in distributed systems. In Proceedings of the 4th ACM International Conference on Distributed Event-Based Systems (DEBS 2010), pages 26--37, 2010.

Digital Library

[20]

B. Koldehofe, R. Mayer, U. Ramachandran, K. Rothermel, and M. Völz. Rollback-recovery without checkpoints in distributed event processing systems. In Proceedings of the 7th ACM international conference on Distributed event-based systems (DEBS '13, page 27, 2013.

Digital Library

[21]

G. T. Lakshmanan, Y. Li, and R. Strom. Placement strategies for internet-scale data stream systems. IEEE Internet Computing, 12(6):50--60, 2008.

Digital Library

[22]

M. Luthra, B. Koldehofe, and R. Steinmetz. Adaptive Complex Event Processing over Fog-Cloud Infrastructure Supporting Transitions. In GI/ITG KuVS-Fachgespraech Fog-Computing, 2018.

[23]

T. D. Matteis and G. Mencagli. Proactive elasticity and energy awareness in data stream processing. Journal of Systems and Software, 127:302 -- 319, 2017.

Digital Library

[24]

D. L. Mills. Internet time synchronization: the network time protocol. IEEE Transactions on Communications, 39(10):1482--1493, 1991.

[25]

D. O'Keeffe, T. Salonidis, and P. Pietzuch. Network-aware stream query processing in mobile ad-hoc networks. In MILCOM 2015 -- 2015 IEEE Military Communications Conference, pages 1335--1340, 2015.

[26]

B. Ottenwälder, B. Koldehofe, K. Rothermel, K. Hong, D. Lillethun, and U. Ramachandran. MCEP: A Mobility-Aware Complex Event Processing System. ACM Transactions on Internet Technology (TOIT), 14(1):1--24, 2014.

Digital Library

[27]

P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and M. Seltzer. Network-aware operator placement for stream-processing systems. In 22nd International Conference on Data Engineering (ICDE'06), pages 49--49, 2006.

Digital Library

[28]

M. A. Rahmani, L.-S. P. Preden, and A. Jantsch. Fog Computing in the Internet of Things. Springer International Publishing, 2018.

[29]

B. Richerzhagen, B. Koldehode, and R. Steinmetz. Immense Dynamism. German Research, 37:24--27, 2015.

[30]

B. Richerzhagen, N. Richerzhagen, J. Zobel, S. Schönherr, B. Koldehofe, and R. Steinmetz. Seamless transitions between filter schemes for location-based mobile applications. In IEEE 41st Conference on Local Computer Networks (LCN), pages 348--356, 2016.

[31]

B. Richerzhagen, M. Schiller, M. Lehn, D. Lapiner, and R. Steinmetz. Transition-enabled event dissemination for pervasive mobile multiplayer games. In IEEE 16th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM), pages 1--3, 2015.

[32]

B. Richerzhagen, S. Wilk, J. Rückert, D. Stohr, and W. Effelsberg. Transitions in live video streaming services. In Proceedings of ACM VideoNEXT 2014, 2014.

Digital Library

[33]

N. Richerzhagen, P. Lieser, B. Richerzhagen, B. Koldehofe, I. Stavrakakis, and R. Steinmetz. Change as chance: Transition-enabled monitoring for dynamic networks and environments. In 14th Annual Conference on Wireless On-demand Network Systems and Services (WONS), 2018.

[34]

S. Rizou, F. Durr, and K. Rothermel. Solving the multi-operator placement problem in large-scale operator networks. In Proceedings of 19th International Conference on Computer Communications and Networks, pages 1--6, 2010.

[35]

B. Schilling, B. Koldehofe, and K. Rothermel. Efficient and distributed rule placement in heavy constraint-driven event systems. In Proceedings of the 13th IEEE International Conference on High Performance Computing and Communications (HPCC-2011), pages 355--364, 2011.

Digital Library

[36]

F. Starks, V. Goebel, S. Kristiansen, and T. Plagemann. Mobile Distributed Complex Event Processing- Ubi Sumus? Quo Vadimus? Mobile Big Data- A Roadmap from Models to Technologies, Springer 2017, (1):1--34, 2017.

[37]

F. Starks and T. P. Plagemann. Operator placement for efficient distributed complex event processing in manets. In 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pages 83--90, 2015.

[38]

T. M. Sutherland, B. Pielech, Y. Zhu, L. Ding, and E. A. Rundensteiner. An adaptive multi-objective scheduling selection framework for continuous query processing. In 9th International Database Engineering Application Symposium (IDEAS'05), pages 445--454, 2005.

Digital Library

[39]

P. Weisenburger, M. Luthra, B. Koldehofe, and G. Salvaneschi. Quality-aware runtime adaptation in complex event processing. In Proceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS '17, pages 140--151, 2017.

Digital Library

[40]

R. Wermund. Privacy-Aware and Reliable Complex Event Processing in the Internet of Things. PhD thesis, Technical University of Darmstadt, 2018.

[41]

D. Whitley. The genitor algorithm and selection pressure: Why rank-based allocation of reproductive trials is best. In Proceedings of the Third International Conference on Genetic Algorithms, pages 116--121. Morgan Kaufmann Publishers Inc., 1989.

Digital Library

[42]

Y. Zhu, E. A. Rundensteiner, and G. T. Heineman. Dynamic plan migration for continuous queries over data streams. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, SIGMOD '04, pages 431--442, 2004.

Digital Library

Cited By

Heinrich RBinnig CKornmayer HLuthra M(2024)Costream: Learned Cost Models for Operator Placement in Edge-Cloud Environments2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00015(96-109)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00015
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Lotfian Delouee MKoldehofe BDegeler V(2022)Towards adaptive quality-aware Complex Event Processing in the Internet of Things2022 18th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN57253.2022.00095(571-575)Online publication date: Dec-2022
https://doi.org/10.1109/MSN57253.2022.00095
Show More Cited By

Index Terms

TCEP: Adapting to Dynamic User Environments by Enabling Transitions between Operator Placement Mechanisms
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
2. Networks
  1. Network properties
    1. Network dynamics

Recommendations

Adapting to Dynamic User Environments in Complex Event Processing System using Transitions
DEBS '18: Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems

Complex Event Processing (CEP) system enables extraction of higher-level information from real-time data streams produced by distributed sources. However, these systems are subject to changes in the user environment e.g., density of sources, rate at ...
Understanding the Behavior of Operator Placement Mechanisms on Large-Scale Networks
Middleware '18: Proceedings of the 19th International Middleware Conference (Posters)

Operator Placement (OP) mechanism dictates the mapping of compute units, aka. operators on the network infrastructure based on the performance objectives requested by the end-user. The behavior of a placement is strongly influenced by the underlying ...
Optimizing Query Routing Trees in Wireless Sensor Networks
ICTAI '10: Proceedings of the 2010 22nd IEEE International Conference on Tools with Artificial Intelligence - Volume 02

The focus of this work is to adaptively optimize continuous queries in a wireless sensor network. Therefore, we analyze and categorize the different types of queries used by wireless sensor applications. We present a short survey on the existing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DEBS '18: Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems

June 2018

289 pages

ISBN:9781450357821

DOI:10.1145/3210284

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft

Conference

DEBS '18

Sponsor:

DEBS '18: The 12th ACM International Conference on Distributed and Event-based Systems

June 25 - 29, 2018

Hamilton, New Zealand

Acceptance Rates

DEBS '18 Paper Acceptance Rate 12 of 31 submissions, 39%;

Overall Acceptance Rate 145 of 583 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
311
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Heinrich RBinnig CKornmayer HLuthra M(2024)Costream: Learned Cost Models for Operator Placement in Edge-Cloud Environments2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00015(96-109)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00015
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Lotfian Delouee MKoldehofe BDegeler V(2022)Towards adaptive quality-aware Complex Event Processing in the Internet of Things2022 18th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN57253.2022.00095(571-575)Online publication date: Dec-2022
https://doi.org/10.1109/MSN57253.2022.00095
Rafiuzzaman MGopalakrishnan SPattabiraman K(2022)$\pi$-Configurator: Enabling Efficient Configuration of Pipelined Applications on the Edge2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)10.1109/IoTDI54339.2022.00009(156-169)Online publication date: May-2022
https://doi.org/10.1109/IoTDI54339.2022.00009
Agnihotri PZhang KGherbi AVenkatasubramanian NVeiga L(2021)Autonomous resource management in distributed stream processing systemsProceedings of the 22nd International Middleware Conference: Doctoral Symposium10.1145/3491087.3493680(19-22)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.1145/3491087.3493680
Luthra MKoldehofe BDanger NWeisenberger PSalvaneschi GStavrakakis I(2021)TCEP: Transitions in operator placement to adapt to dynamic network environmentsJournal of Computer and System Sciences10.1016/j.jcss.2021.05.003Online publication date: Jul-2021
https://doi.org/10.1016/j.jcss.2021.05.003
Liu XBuyya R(2020)Resource Management and Scheduling in Distributed Stream Processing SystemsACM Computing Surveys10.1145/335539953:3(1-41)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3355399
Luthra MHennig SRazavi KWang LKoldehofe B(2020)Operator as a Service: Stateful Serverless Complex Event Processing2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378142(1964-1973)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378142
Eskandari LMair JHuang ZEyers D(2020)I-Scheduler: Iterative scheduling for distributed stream processing systemsFuture Generation Computer Systems10.1016/j.future.2020.11.011Online publication date: Nov-2020
https://doi.org/10.1016/j.future.2020.11.011
Esmaeilyfard RNaderi M(2020)Distributed composition of complex event services in IoT networkThe Journal of Supercomputing10.1007/s11227-020-03498-2Online publication date: 19-Nov-2020
https://doi.org/10.1007/s11227-020-03498-2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents