research-article

Real time data analysis of taxi rides using StreamMine3G

Authors:

André Martin,

Andrey Brito,

Christof FetzerAuthors Info & Claims

DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

Pages 269 - 276

https://doi.org/10.1145/2675743.2772584

Published: 24 June 2015 Publication History

Get Access

Abstract

In this paper, we present our approach for solving the DEBS Grand Challenge 2015 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We first provide an overview about the system architecture of StreamMine3G followed by a thorough description of our implementation for the two queries that provide continuously up-to-date information about (i) the top-k most frequently driven routes and (ii) most profitable areas.

Novel aspects of our implementation include two self-balancing double linked list implementations to efficiently update and determine a top-k as well as a median from a set of samples. Furthermore, we present a solution that supports data partitioning which allows the application to scale without bounds while still guaranteeing semantic transparency through the deterministic processing approach offered by the StreamMine3G runtime. In our evaluation, we provide measurements that show that our system can scale horizontally as well as vertically and can process 13 kEvents/s on a single node which translates to a processing of 3.8 hours of real time data within a second and a latency under 1 ms.

References

[1]

Apache s4 - distributed stream computing platform. https://incubator.apache.org/s4/, 2015.

Google Scholar

[2]

Apache samza - a distributed stream processing framework. http://samza.incubator.apache.org/, 2015.

Google Scholar

[3]

Apache storm - distributed and fault-tolerant realtime computation. https://storm.incubator.apache.org/, 2015.

Google Scholar

[4]

Hadoop mapreduce open source implementation. http://hadoop.apache.org/, 2015.

Google Scholar

[5]

Seep - a parallel data processing system. https://github.com/lsds/Seep/, 2015.

Google Scholar

[6]

J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(1):107--113, Jan. 2008.

Digital Library

Google Scholar

[7]

V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez. Streamcloud: An elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst., 23(12):2351--2365, Dec. 2012.

Digital Library

Google Scholar

[8]

A. Martin, R. Marinho, A. Brito, and C. Fetzer. Predicting energy consumption with streammine3g. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, DEBS '14, pages 270--275, New York, NY, USA, 2014. ACM.

Digital Library

Google Scholar

[9]

L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. In Proceedings of the 2010 IEEE International Conference on Data Mining Workshops, ICDMW '10, pages 170--177, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

Google Scholar

Cited By

View all

Dayarathna MPerera S(2018)Recent Advancements in Event ProcessingACM Computing Surveys10.1145/317043251:2(1-36)Online publication date: 13-Feb-2018
https://dl.acm.org/doi/10.1145/3170432
Martin ABrito AFetzer CGal AWeidlich MKalogeraki VVenkasubramanian N(2016)Real-time social network graph analysis using StreamMine3GProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933514(322-329)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933514

Index Terms

Real time data analysis of taxi rides using StreamMine3G
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Organizing principles for web applications

Recommendations

Real-time social network graph analysis using StreamMine3G
DEBS '16: Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems

In this paper, we present our approach for solving the DEBS Grand Challenge 2016 using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant event stream processing (ESP) system. We first provide an overview about StreamMine3G with ...
Predicting energy consumption with StreamMine3G
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

In this paper, we present our approach on solving the DEBS Grand Challenge using StreamMine3G, a distributed, highly scalable, elastic and fault tolerant ESP system. We will provide an overview about the system architecture of Stream-Mine3G and ...
Scalable and elastic realtime click stream analysis using StreamMine3G
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

Click stream analysis is a common approach for analyzing customer behavior during the navigation through e-commerce or social network sites. Performing such an analysis in real-time opens up new business opportunities as well as increases revenues as ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

June 2015

385 pages

ISBN:9781450332866

DOI:10.1145/2675743

General Chairs:
Frank Eliassen
University of Oslo, Norway
,
Roman Vitenberg
University of Oslo, Norway

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Commission

Conference

DEBS '15

Sponsor:

DEBS '15: The 9th ACM International Conference on Distributed Event-Based Systems

June 29 - July 3, 2015

Oslo, Norway

Acceptance Rates

Overall Acceptance Rate 145 of 583 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
193
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dayarathna MPerera S(2018)Recent Advancements in Event ProcessingACM Computing Surveys10.1145/317043251:2(1-36)Online publication date: 13-Feb-2018
https://dl.acm.org/doi/10.1145/3170432
Martin ABrito AFetzer CGal AWeidlich MKalogeraki VVenkasubramanian N(2016)Real-time social network graph analysis using StreamMine3GProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933514(322-329)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933514

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Real-time social network graph analysis using StreamMine3G

Predicting energy consumption with StreamMine3G

Scalable and elastic realtime click stream analysis using StreamMine3G