poster

Coupling scheduler for MapReduce/Hadoop

Authors:

Jian Tan,

Xiaoqiao Meng,

Li ZhangAuthors Info & Claims

HPDC '12: Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing

Pages 129 - 130

https://doi.org/10.1145/2287076.2287097

Published: 18 June 2012 Publication History

Get Access

Abstract

Current schedulers of MapReduce/Hadoop are quite successful in providing good performance. However improving spaces still exist: map and reduce tasks are not jointly optimized for scheduling, albeit there is a strong dependence between them. This can cause job starvation and bad data locality. We design a resource-aware scheduler for Hadoop, which couples the progresses of mappers and reducers, and jointly optimize the placements for both of them. This mitigates the starvation problem and improves the overall data locality. Our experiments demonstrate improvements to job response times by up to an order of magnitude.

References

[1]

Fair Scheduler, http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html.

Google Scholar

[2]

J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Commun. ACM, 51:107--113, January 2008.

Digital Library

Google Scholar

[3]

Hadoop. http://hadoop.apache.org.

Google Scholar

[4]

M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Job scheduling for multi-user mapreduce clusters. Technical Report, University of California, Berkeley, April 2009.

Google Scholar

Cited By

View all

Pandey VSaini P(2019)Survey on Various MapReduce Scheduling AlgorithmsHandbook of Research on Cloud Computing and Big Data Applications in IoT10.4018/978-1-5225-8407-0.ch022(499-515)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-8407-0.ch022
Shen HSarker AYu LDeng F(2016)Probabilistic Network-Aware Task Placement for MapReduce Scheduling2016 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER.2016.48(241-250)Online publication date: Sep-2016
https://doi.org/10.1109/CLUSTER.2016.48
Tiwari NSarkar SBellur UIndrawan M(2015)Classification Framework of MapReduce Scheduling AlgorithmsACM Computing Surveys10.1145/269331547:3(1-38)Online publication date: 16-Apr-2015
https://dl.acm.org/doi/10.1145/2693315
Show More Cited By

Index Terms

Coupling scheduler for MapReduce/Hadoop
1. Information systems
  1. Information systems applications
2. Software and its engineering
  1. Software creation and management
    1. Designing software

Recommendations

Delay tails in MapReduce scheduling
SIGMETRICS '12: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE joint international conference on Measurement and Modeling of Computer Systems

MapReduce/Hadoop production clusters exhibit heavy-tailed characteristics for job processing times. These phenomena are resultant of the workload features and the adopted scheduling algorithms. Analytically understanding the delays under different ...
TaskTracker aware scheduler with resource availability control for Hadoop MapReduce

Schedulers are playing a vital role in task assignment for Hadoop MapReduce. In some scenario, the default schedulers of Hadoop spawn tasks in TaskTracker without checking the external dependency and may fail. As a result, Hadoop should rerun the tasks in ...
Delay tails in MapReduce scheduling
Performance evaluation review

MapReduce/Hadoop production clusters exhibit heavy-tailed characteristics for job processing times. These phenomena are resultant of the workload features and the adopted scheduling algorithms. Analytically understanding the delays under different ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

HPDC '12: Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing

June 2012

308 pages

ISBN:9781450308052

DOI:10.1145/2287076

General Chair:
Dick Epema
Delft University of Technology and Eindhoven University of Technology, The Netherlands
,
Program Chairs:
Thilo Kielmann
Vrije Universiteit, The Netherlands
,
Matei Ripeanu
The University of British Columbia, Canada

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

HPDC'12

Sponsor:

University of Arizona
SIGARCH

HPDC'12: The 21st International Symposium on High-Performance Parallel and Distributed Computing

June 18 - 22, 2012

Delft, The Netherlands

Acceptance Rates

HPDC '12 Paper Acceptance Rate 23 of 143 submissions, 16%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
535
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Pandey VSaini P(2019)Survey on Various MapReduce Scheduling AlgorithmsHandbook of Research on Cloud Computing and Big Data Applications in IoT10.4018/978-1-5225-8407-0.ch022(499-515)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-8407-0.ch022
Shen HSarker AYu LDeng F(2016)Probabilistic Network-Aware Task Placement for MapReduce Scheduling2016 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER.2016.48(241-250)Online publication date: Sep-2016
https://doi.org/10.1109/CLUSTER.2016.48
Tiwari NSarkar SBellur UIndrawan M(2015)Classification Framework of MapReduce Scheduling AlgorithmsACM Computing Surveys10.1145/269331547:3(1-38)Online publication date: 16-Apr-2015
https://dl.acm.org/doi/10.1145/2693315
D'Souza SChandrasekaran K(2015)Analysis of MapReduce scheduling and its improvements in cloud environment2015 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES)10.1109/SPICES.2015.7091470(1-5)Online publication date: Mar-2015
https://doi.org/10.1109/SPICES.2015.7091470
Vieira TFernandes SGarcia VPescapé AFernandes S(2013)Evaluating MapReduce for profiling application trafficProceedings of the first edition workshop on High performance and programmable networking10.1145/2465839.2465846(45-52)Online publication date: 18-Jun-2013
https://dl.acm.org/doi/10.1145/2465839.2465846
Tan JMeng SMeng XZhang L(2013)Improving ReduceTask data locality for sequential MapReduce jobs2013 Proceedings IEEE INFOCOM10.1109/INFCOM.2013.6566959(1627-1635)Online publication date: Apr-2013
https://doi.org/10.1109/INFCOM.2013.6566959
Tan JMeng SMeng XZhang L(2012)Delay asymptotics for heavy-tailed MapReduce jobs2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton)10.1109/Allerton.2012.6483417(1637-1639)Online publication date: Oct-2012
https://doi.org/10.1109/Allerton.2012.6483417

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Delay tails in MapReduce scheduling

TaskTracker aware scheduler with resource availability control for Hadoop MapReduce

Delay tails in MapReduce scheduling