short-paper

Toward elastic memory management for cloud data analytics

Authors:

Magdalena BalazinskaAuthors Info & Claims

BeyondMR '16: Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond

Article No.: 7, Pages 1 - 4

https://doi.org/10.1145/2926534.2926541

Published: 26 June 2016 Publication History

Abstract

We present several key elements towards elastic memory management in modern big data systems. The goal of our approach is to avoid out-of-memory failures without over-provisioning but also to avoid garbage-collection overheads when possible.

References

[1]

Tungsten: Memory management and binary processing on spark. https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html.

[2]

Memory management in the Java HotSpot™virtual machine. http://www.oracle.com/technetwork/java/javase/memorymanagement-whitepaper-150215.pdf, 2006.

[3]

N. Anciaux et al. Memory requirements for query execution in highly constrained devices. In VLDB, 2003.

Digital Library

[4]

K. P. Brown et al. Managing memory to meet multiclass workload response time goals. In VLDB, 1993.

Digital Library

[5]

C. Chen et al. Adaptive database buffer allocation using query feedback. In VLDB, 1993.

Digital Library

[6]

J. E. Cook et al. Semi-automatic, self-adaptive control of garbage collection rates in object databases. In SIGMOD, 1996.

Digital Library

[7]

D. L. Davison et al. Dynamic resource brokering for multi-user query execution. In SIGMOD, 1995.

Digital Library

[8]

C. Faloutsos et al. Predictive load control for flexible buffer allocation. In VLDB, 1991.

Digital Library

[9]

M. N. Garofalakis et al. Parallel query scheduling and optimization with time- and space-shared resources. In VLDB, 1997.

Digital Library

[10]

M. Hall et al. The weka data mining software: An update. 2009.

[11]

D. Halperin et al. Demo of the Myria big data management service. In SIGMOD, 2014.

Digital Library

[12]

H. Herodotou et al. No one (cluster) size fits all: automatic cluster sizing for data-intensive analytics. In SoCC, 2011.

Digital Library

[13]

H. Herodotou et al. Starfish: A self-tuning system for big data analytics. In CIDR, 2011.

[14]

B. Hindman et al. Mesos: A platform for fine-grained resource sharing in the data center. In NSDI, 2011.

Digital Library

[15]

M. Kornacker et al. Impala: A modern, open-source SQL engine for hadoop. In CIDR, 2015.

[16]

W. Lang et al. Towards multi-tenant performance slos. IEEE Trans. Knowl. Data Eng., 2014.

Digital Library

[17]

J. Li et al. Resource bricolage for parallel database systems. Proc. of the VLDB Endow., 2014.

Digital Library

[18]

Y. Low et al. Distributed GraphLab: a framework for machine learning and data mining in the cloud. In VLDB, 2012.

Digital Library

[19]

D. G. Murray et al. Naiad: A timely dataflow system. In SOSP, 2013.

Digital Library

[20]

V. R. Narasayya et al. Sharing buffer pool memory in multi-tenant relational database-as-a-service. Proc. of the VLDB Endow., 2015.

Digital Library

[21]

R. T. Ng et al. Flexible buffer allocation based on marginal gains. In SIGMOD, 1991.

Digital Library

[22]

H. Pang et al. Managing memory for real-time queries. In SIGMOD, 1994.

Digital Library

[23]

T. A. Project. Apache Giraph, http://giraph.apache.org/.

[24]

J. Schaffner et al. Predicting in-memory database performance for automating cluster management tasks. In ICDE, 2011.

Digital Library

[25]

A. J. Storm et al. Adaptive self-tuning memory in DB2. In VLDB, 2006.

Digital Library

[26]

P. Tembey et al. Merlin: Application- and platform-aware resource allocation in consolidated server systems. In SoCC, 2014.

Digital Library

[27]

V. K. Vavilapalli et al. Apache hadoop YARN: yet another resource negotiator. In SoCC, 2013.

Digital Library

[28]

M. Weimer et al. REEF: retainable evaluator execution framework. In SIGMOD, 2015.

Digital Library

[29]

T. White. Hadoop: The Definitive Guide. 2009.

Digital Library

[30]

M. Zaharia et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012.

Digital Library

Cited By

Panwar GLaghari MChoukse EJian X(2024)DyLeCT: Achieving Huge-page-like Translation Performance for Hardware-compressed Memory2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00085(1129-1143)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00085
Wang QLi HHu ZZhang RYang CCai PZhou XZhou A(2024)Mirage: Generating Enormous Databases for Complex Workloads2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00306(3989-4001)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00306
Wang YYu JYu Z(2023)Resource scheduling techniques in cloud from a view of coordination: a holistic survey从协同视角论云资源调度技术：综述Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.210029824:1(1-40)Online publication date: 23-Jan-2023
https://doi.org/10.1631/FITEE.2100298
Show More Cited By

Toward elastic memory management for cloud data analytics
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Memory management

Recommendations

Big Data Analytics
Big data analytics in Cloud computing: an overview
Abstract
Big Data and Cloud Computing as two mainstream technologies, are at the center of concern in the IT field. Every day a huge amount of data is produced from different sources. This data is so big in size that traditional processing tools are unable ...
Cloud computing for big data analytics How cloud computing can handle procesing large amounts of data and improve real-time data analytics
Abstract
With the increasing volume, velocity, and variety of data generated by various sources, big data has become a critical challenge for many organizations. Cloud computing provides an efficient and cost-effective solution to address the challenges of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

BeyondMR '16: Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond

June 2016

70 pages

ISBN:9781450343114

DOI:10.1145/2926534

Program Chairs:
Foto Afrati
National Technical University of Athens
,
Jacek Sroka
University of Warsaw, Poland
,
Jan Hidders
Vrije Universiteit Brussel, Belgium

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data
Google Inc.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Funding Sources

Conference

SIGMOD/PODS'16

Sponsor:

SIGMOD

SIGMOD/PODS'16: International Conference on Management of Data

June 26 - July 1, 2016

California, San Francisco

Acceptance Rates

BeyondMR '16 Paper Acceptance Rate 10 of 19 submissions, 53%;

Overall Acceptance Rate 19 of 36 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
215
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Panwar GLaghari MChoukse EJian X(2024)DyLeCT: Achieving Huge-page-like Translation Performance for Hardware-compressed Memory2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00085(1129-1143)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00085
Wang QLi HHu ZZhang RYang CCai PZhou XZhou A(2024)Mirage: Generating Enormous Databases for Complex Workloads2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00306(3989-4001)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00306
Wang YYu JYu Z(2023)Resource scheduling techniques in cloud from a view of coordination: a holistic survey从协同视角论云资源调度技术：综述Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.210029824:1(1-40)Online publication date: 23-Jan-2023
https://doi.org/10.1631/FITEE.2100298
Chen WPi AWang SZhou X(2019)PufferfishProceedings of the ACM Symposium on Cloud Computing10.1145/3357223.3362730(259-271)Online publication date: 20-Nov-2019
https://dl.acm.org/doi/10.1145/3357223.3362730
Farhana SLajis ALong Z(2019)Binary Algorithm for Big Data Management and Analytics of MyRA Data2019 IEEE International Conference on Smart Instrumentation, Measurement and Application (ICSIMA)10.1109/ICSIMA47653.2019.9057309(1-4)Online publication date: Aug-2019
https://doi.org/10.1109/ICSIMA47653.2019.9057309
Xu GXu C(2019)MEER: Online Estimation of Optimal Memory Reservations for Long Lived Containers in In-Memory Cluster Computing2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2019.00012(23-34)Online publication date: Jul-2019
https://doi.org/10.1109/ICDCS.2019.00012
Yu ZBei ZQian X(2018)Datasize-Aware High Dimensional Configurations Auto-Tuning of In-Memory Cluster ComputingACM SIGPLAN Notices10.1145/3296957.317318753:2(564-577)Online publication date: 19-Mar-2018
https://dl.acm.org/doi/10.1145/3296957.3173187
Liu LXu H(2018)ElasecutorProceedings of the ACM Symposium on Cloud Computing10.1145/3267809.3267818(107-120)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3267809.3267818
Perez TZhou XCheng D(2018)Reference-distance Eviction and Prefetching for Cache Management in SparkProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225087(1-10)Online publication date: 13-Aug-2018
https://dl.acm.org/doi/10.1145/3225058.3225087
Yu ZBei ZQian XShen XTuck JBianchini RSarkar V(2018)Datasize-Aware High Dimensional Configurations Auto-Tuning of In-Memory Cluster ComputingProceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3173162.3173187(564-577)Online publication date: 19-Mar-2018
https://dl.acm.org/doi/10.1145/3173162.3173187

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents