research-article

MR-scope: a real-time tracing tool for MapReduce

Authors:

Hai JinAuthors Info & Claims

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Pages 849 - 855

https://doi.org/10.1145/1851476.1851598

Published: 21 June 2010 Publication History

Abstract

MapReduce programming model is emerging as an efficient tool for data-intensive applications. Hadoop, an open-source implementation of MapReduce, has been widely adopted and experienced by both academia and enterprise. Recently, lots of efforts have been done on improving the performance of MapReduce system and on analyzing the MapReduce process based on the log files generated during the Hadoop execution. Visualizing log files seems to be a very useful tool to understand the behavior of the Hadoop process. In this paper, we present MR-Scope, a real-time MapReduce tracing tool. MR-Scope provides a real-time insight of the MapReduce process, including the ongoing progress of every task hosted in Task Tracker. In addition, it displays the health of the Hadoop cluster data nodes, the distribution of the file system's blocks and their replicas and the content of the different block splits of the file system. We implement MR-Scope in native Hadoop 0.1. Experimental results demonstrat that MR-Scope's overhead is less than 4% when running wordcount benchmark.

References

[1]

}}Dean, J. and Ghemawat, S. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of OSDI. 2004, 137--150.

Digital Library

[2]

}}Hadoop: http://hadoop.apache.org/2010.

[3]

}}Ghemawat, S., Gobioff, H., and Leung, S. The Google file system. In Proceedings of SOSP. 2003, 29--43.

Digital Library

[4]

}}Tan, J., Pan, X., Kavulya, S., Gandhi, R., and Narasimhan, P. Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop. CMU-PDL-09-103, Technical Report, May 2009.

[5]

}}Fonseca, R., Porter, G., Katz, R. H., Shenker, S., and Stoica, I. X-Trace: A Pervasive Network Tracing Framework. In Proceedings of NSDI. 2007.

Digital Library

[6]

}}Konwinski, A., Zaharia, M., Katz, R., and Stoica I. X-Tracing Hadoop. Presentation in Hadoop Summit, Mar 2008.

[7]

}}Massie, M. L., Chun, B. N., and Culler, D. E. The ganglia distributed monitoring system: design, implementation, and experience. In Parallel Computing. Vol. 30, No. 7, 2004, 817--840.

[8]

}}Quiroz, A., Gnanasambandam, N., Parashar, M., and Sharma, N. Robust clustering analysis for the management of self-monitoring distributed systems. In Proceedings of Cluster Computing. 2009, 73--85.

Digital Library

[9]

}}Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R. H., and Stoica, I. Improving MapReduce Performance in Heterogeneous Environments. In Proceedings of OSDI. 2008, 29--42.

Digital Library

[10]

}}Boulon, J., Konwinski, A., Qi, R., Rabkin, A., Yang, E., and Yang, M. Chukwa: A large-scale monitoring system. In Proceedings of Cloud Computing and Its Applications, 2008.

[11]

}}Ibrahim, S., Jin, H., Cheng, B., Cao, H., Wu, S., and Qi, L. CLOUDLET: towards mapreduce implementation on virtual machines. In Proceedings of HPDC. 2009, 65--66.

Digital Library

[12]

}}Ibrahim, S., Jin, H., Lu, L., Qi, L., Wu, S., and Shi, X. Evaluating MapReduce on Virtual Machines: The Hadoop Case. In Proceedings of CloudCom. 2009, 519--528.

Digital Library

[13]

}}Cretu-Ciocarlie, G. F., Budiu, M., and Goldszmidt, M. Hunting for Problems with Artemis. In Proceedings of WASL. 2008.

Digital Library

[14]

}}He, B. S., Fang, W., Luo, Q., Govindaraju, N. K., and Wang, T. Mars: a MapReduce framework on graphics processors. In Proceedings of PACT. 2008, 260--269.

Digital Library

[15]

}}Chen, S, and Schlosser, S. W. Map-Reduce Meets Wider Varieties of Applications. IRP-TR-08-05, Technical Report, Intel Research Pittsburgh, May 2008.

[16]

}}ParaGraph User Guide. ParaGraph: A Performance Visualization Tool for MPI http://www.csar.illinois.edu/software/paragraph/

[17]

}}MPI standard: http://www.mpi-forum.org/docs/mpi-11-html/mpi-report.html 2010.

Cited By

Kundu SMaulik U(2021)Cloud deployment of game theoretic categorical clustering using apache spark: An application to car recommendationMachine Learning with Applications10.1016/j.mlwa.2021.100100(100100)Online publication date: Jul-2021
https://doi.org/10.1016/j.mlwa.2021.100100
Yildiz OIbrahim SAntoniu G(2017)Enabling fast failure recovery in shared Hadoop clusters: Towards failure-aware schedulingFuture Generation Computer Systems10.1016/j.future.2016.02.01574(208-219)Online publication date: Sep-2017
https://doi.org/10.1016/j.future.2016.02.015
Memishi BIbrahim SPerez MAntoniu G(2016)On the Dynamic Shifting of the MapReduce TimeoutManaging and Processing Big Data in Cloud Computing10.4018/978-1-4666-9767-6.ch001(1-22)Online publication date: 2016
https://doi.org/10.4018/978-1-4666-9767-6.ch001
Show More Cited By

Index Terms

MR-scope: a real-time tracing tool for MapReduce
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management
        Monitors
    2. Software system structures
      1. Distributed systems organizing principles

Recommendations

MapReduce: Review and open challenges

The continuous increase in computational capacity over the past years has produced an overwhelming flow of data or big data, which exceeds the capabilities of conventional processing tools. Big data signify a new era in data exploration and utilization. ...
Disease Surveillance System for Big Climate Data Processing and Dengue Transmission

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of ...
Design and Development of a Medical Big Data Processing System Based on Hadoop

Secondary use of medical big data is increasingly popular in healthcare services and clinical research. Understanding the logic behind medical big data demonstrates tendencies in hospital information technology and shows great significance for hospital ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

June 2010

911 pages

ISBN:9781605589428

DOI:10.1145/1851476

General Chairs:
Salim Hariri
University of Arizona
,
Kate Keahey
University of Chicago

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Intel Corporation
Ministry of Education of the People's Republic of China
China Next Generation Internet Project
Ministry of Science and Technology of the People's Republic of China
Key Project in the National Science & Technology Pillar Program of China

Conference

HPDC '10

Sponsor:

University of Arizona
SIGARCH

HPDC '10: The 19th International Symposium on High Performance Distributed Computing

June 21 - 25, 2010

Illinois, Chicago

Acceptance Rates

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
571
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kundu SMaulik U(2021)Cloud deployment of game theoretic categorical clustering using apache spark: An application to car recommendationMachine Learning with Applications10.1016/j.mlwa.2021.100100(100100)Online publication date: Jul-2021
https://doi.org/10.1016/j.mlwa.2021.100100
Yildiz OIbrahim SAntoniu G(2017)Enabling fast failure recovery in shared Hadoop clusters: Towards failure-aware schedulingFuture Generation Computer Systems10.1016/j.future.2016.02.01574(208-219)Online publication date: Sep-2017
https://doi.org/10.1016/j.future.2016.02.015
Memishi BIbrahim SPerez MAntoniu G(2016)On the Dynamic Shifting of the MapReduce TimeoutManaging and Processing Big Data in Cloud Computing10.4018/978-1-4666-9767-6.ch001(1-22)Online publication date: 2016
https://doi.org/10.4018/978-1-4666-9767-6.ch001
Li RHu HLi HWu YYang J(2016)MapReduce Parallel Programming ModelInternational Journal of Parallel Programming10.1007/s10766-015-0395-044:4(832-866)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1007/s10766-015-0395-0
Phan TIbrahim SAntoniu GBouge L(2015)On Understanding the Energy Impact of Speculative Execution in HadoopProceedings of the 2015 IEEE International Conference on Data Science and Data Intensive Systems (DSDIS)10.1109/DSDIS.2015.45(396-403)Online publication date: 11-Dec-2015
https://dl.acm.org/doi/10.1109/DSDIS.2015.45
Tang YMiao LChen F(2014)Peer-Comparison Based Fault Diagnosis for Hadoop SystemsApplied Mechanics and Materials10.4028/www.scientific.net/AMM.621.235621(235-240)Online publication date: Aug-2014
https://doi.org/10.4028/www.scientific.net/AMM.621.235
Tedesco JDudko RSharma AFarivar RCampbell R(2013)TheiusProceedings of the 2013 IEEE International Conference on Cloud Engineering10.1109/IC2E.2013.36(177-182)Online publication date: 25-Mar-2013
https://dl.acm.org/doi/10.1109/IC2E.2013.36
Wirtz TRong Ge (2011)Improving MapReduce energy efficiency for computation intensive workloadsProceedings of the 2011 International Green Computing Conference and Workshops10.1109/IGCC.2011.6008564(1-8)Online publication date: 25-Jul-2011
https://dl.acm.org/doi/10.1109/IGCC.2011.6008564

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents