research-article

Multi-Tier Resource Allocation for Data-Intensive Computing

Authors:

Young Choon LeeAuthors Info & Claims

Big Data Research, Volume 2, Issue 3

Pages 110 - 116

https://doi.org/10.1016/j.bdr.2015.03.001

Published: 01 September 2015 Publication History

Abstract

As distributed computing systems are used more widely, driven by trends such as 'big data' and cloud computing, they are being used for an increasingly wide range of applications. With this massive increase in application heterogeneity, the ability to have a general purpose resource management technique that performs well in heterogeneous environments is becoming increasingly important.In this paper, we present Multi-Tier Resource Allocation (MTRA) as a novel fine-grained resource management technique for distributed systems. The core idea is based on allocating resources to individual tasks in a tiered or layered approach. To account for heterogeneity, we propose a dynamic resource allocation method that adjusts resource allocations to individual tasks on a cluster node based on resource utilisation levels. We demonstrate the efficacy of this technique in a data-intensive computing environment, MapReduce data processing framework in Hadoop YARN. Our results demonstrate that MTRA is an effective general purpose resource management technique particularly for data-intensive computing environments. On a range of MapReduce benchmarks in a Hadoop YARN environment, our MTRA technique improves performance by up to 18%. In a Facebook workload model it improves job execution times by 10% on average, and up to 56% for individual jobs.

References

[1]

J. Dean, S. Ghemawat, MapReduce: simplified data processing on large clusters, in: Proceedings of the 6th Conference on Symposium on Operating Systems Design and Implementation, USENIX Association, 2004, pp. 137-150.

[2]

G. Malewicz, M.H. Austern, A.J. Bik, J.C. Dehnert, I. Horn, N. Leiser, G. Czajkowski, Pregel: a system for large-scale graph processing, in: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, ACM, New York, NY, USA, 2010, pp. 135-146.

Digital Library

[3]

J.C. Jacob, D.S. Katz, Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking, Int. J. Comput. Sci. Eng., 4 (2009) 73-87.

[4]

Montage: an astronomical image mosaic engine. http://montage.ipac.caltech.edu/

[5]

V.K. Vavilapalli, A.C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, E. Baldeschwieler, Apache Hadoop YARN: yet another resource negotiator, in: Proceedings of the 4th Annual Symposium on Cloud Computing, ACM, New York, NY, USA, 2013, pp. 5:1-5:16.

Digital Library

[6]

P. Lu, Y.C. Lee, V. Gramoli, L.M. Leslie, A.Y. Zomaya, Local resource shaper for MapReduce, in: Proceedings of the IEEE International Conference on Cloud Computing Technology and Science (CloudCom), 2014, pp. 483-490.

Digital Library

[7]

The Fourth Paradigm: Data-Intensive Scientific Discovery, in: The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft, 2009.

[8]

Amazon, Amazon elastic compute cloud. http://aws.amazon.com/ec2/

[9]

Google, Google compute engine. https://cloud.google.com/compute/

[10]

M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, M. Zaharia, A view of cloud computing, Commun. ACM, 53 (2010) 50-58.

Digital Library

[11]

R. Nathuji, A. Kansal, A. Ghaffarkhah, Q-clouds: managing performance interference effects for QoS-aware clouds, in: Proceedings of the 5th European Conference on Computer Systems, ACM, 2010, pp. 237-250.

Digital Library

[12]

G. Jung, M.A. Hiltunen, K.R. Joshi, R.D. Schlichting, C. Pu, Mistral: dynamically managing power, performance, and adaptation cost in cloud infrastructures, in: Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems, IEEE Computer Society, 2010, pp. 62-73.

Digital Library

[13]

C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, A. Warfield, Live migration of virtual machines, in: Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation, vol. 2, USENIX Association, 2005, pp. 273-286.

[14]

A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, I. Stoica, Dominant resource fairness: fair allocation of multiple resource types, in: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, USENIX Association, Berkeley, CA, USA, 2011, pp. 323-336.

[15]

Hortonworks, How to plan and configure YARN in Hadoop 2.0. http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

[16]

G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, E. Harris, Reining in the outliers in map-reduce clusters using Mantri, in: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, USENIX Association, Berkeley, CA, USA, 2010, pp. 1-16.

[17]

P. Lu, Y.C. Lee, C. Wang, B.B. Zhou, J. Chen, A. Zomaya, Workload characteristic oriented scheduler for MapReduce, in: Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems (ICPADS), 2012, pp. 156-163.

[18]

J. Polo, C. Castillo, D. Carrera, Y. Becerra, I. Whalley, M. Steinder, J. Torres, E. Ayguadé, Resource-aware adaptive scheduling for MapReduce clusters, in: Proceedings of the 12th ACM/IFIP/USENIX International Conference on Middleware, Springer, Berlin, Heidelberg, 2011, pp. 187-207.

Digital Library

[19]

A. Verma, L. Cherkasova, R. Campbell, Resource provisioning framework for MapReduce jobs with performance goals, in: Lect. Notes Comput. Sci., vol. 7049, Springer, Berlin, Heidelberg, 2011, pp. 165-186.

[20]

M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, I. Stoica, Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling, in: Proceedings of the 5th European Conference on Computer Systems, ACM, New York, NY, USA, 2010, pp. 265-278.

Digital Library

[21]

B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A.D. Joseph, R. Katz, S. Shenker, I. Stoica, Mesos: a platform for fine-grained resource sharing in the data center, in: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, 2011, pp. 295-308.

[22]

M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, J. Wilkes, Omega: flexible, scalable schedulers for large compute clusters, in: Proceedings of the 8th ACM European Conference on Computer Systems, ACM, New York, NY, USA, 2013, pp. 351-364.

[23]

J. Park, D. Lee, B. Kim, J. Huh, S. Maeng, Locality-aware dynamic VM reconfiguration on MapReduce clouds, in: Proceedings of the 21st International Symposium on High-Performance Parallel and Distributed Computing, ACM, New York, NY, USA, 2012, pp. 27-36.

[24]

M. Li, L. Zeng, S. Meng, J. Tan, L. Zhang, A.R. Butt, N. Fuller, MRONLINE: MapReduce online performance tuning, in: Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, ACM, New York, NY, USA, 2014, pp. 165-176.

[25]

C. Delimitrou, C. Kozyrakis, QoS-aware scheduling in heterogeneous datacenters with paragon, ACM Trans. Comput. Syst., 31 (2013) 12:1-12:34.

Digital Library

[26]

H. Goudarzi, M. Pedram, Multi-dimensional SLA-based resource allocation for multi-tier cloud computing systems, in: Proceedings of the 2011 IEEE 4th International Conference on Cloud Computing, IEEE Computer Society, 2011, pp. 324-331.

[27]

F. Jrad, J. Tao, I. Brandic, A. Streit, Multi-dimensional resource allocation for data-intensive large-scale cloud applications, in: Proceedings of the 4th International Conference on Cloud Computing and Services Science, 2014.

Cited By

Bhatnagar VJain A(2018)Olympics Big Data PrognosticationsInternational Journal of Rough Sets and Data Analysis10.4018/IJRSDA.20161001033:4(32-45)Online publication date: 11-Dec-2018
https://dl.acm.org/doi/10.4018/IJRSDA.2016100103
Jain ABhatnagar V(2017)Hadoop Map Only Job for Enciphering Patient-Generated Health DataInternational Journal of Information Retrieval Research10.4018/IJIRR.20171001057:4(72-86)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJIRR.2017100105
Bhatnagar VJain A(2017)Concoction of Ambient Intelligence and Big Data for Better Patient Ministration ServicesInternational Journal of Ambient Computing and Intelligence10.4018/IJACI.20171001028:4(19-30)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJACI.2017100102
Show More Cited By

Multi-Tier Resource Allocation for Data-Intensive Computing
1. General and reference
  1. Cross-computing tools and techniques
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures

Recommendations

Multi-policy-aware MapReduce resource allocation and scheduling for smart computing cluster

When a user submit a MapReduce job in the smart computing cluster, we first need to allocate cluster resource for the job. It is widely concerned that how to save time and resource costs to provide users with computing capacity and services. Here, we ...
Improvement of job completion time in data-intensive cloud computing applications
Abstract
Task stragglers in MapReduce jobs dramatically impede job execution of data-intensive computing in cloud data centers. This impedance is due to the uneven distribution of input data, heterogeneous data nodes, resource contention situations, and ...
QoS-aware Resource Provisioning for Big Data Processing in Cloud Computing Environment
CSCI '14: Proceedings of the 2014 International Conference on Computational Science and Computational Intelligence - Volume 02

Nowadays, the quantity of collected data from many different sources is increasing dramatically. As the traditional on-hand computing resources are not sufficient enough to handle Big data, deploying the processing services into clouds is becoming an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Big Data Research

Big Data Research Volume 2, Issue 3

September 2015

41 pages

ISSN:2214-5796

EISSN:2214-5796

Issue’s Table of Contents

Copyright © Elsevier Inc.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 September 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bhatnagar VJain A(2018)Olympics Big Data PrognosticationsInternational Journal of Rough Sets and Data Analysis10.4018/IJRSDA.20161001033:4(32-45)Online publication date: 11-Dec-2018
https://dl.acm.org/doi/10.4018/IJRSDA.2016100103
Jain ABhatnagar V(2017)Hadoop Map Only Job for Enciphering Patient-Generated Health DataInternational Journal of Information Retrieval Research10.4018/IJIRR.20171001057:4(72-86)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJIRR.2017100105
Bhatnagar VJain A(2017)Concoction of Ambient Intelligence and Big Data for Better Patient Ministration ServicesInternational Journal of Ambient Computing and Intelligence10.4018/IJACI.20171001028:4(19-30)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJACI.2017100102
Misra RPanda BTiwary M(2016)Big data and ICT applicationsProceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies10.1145/2905055.2905099(1-6)Online publication date: 4-Mar-2016
https://dl.acm.org/doi/10.1145/2905055.2905099

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents