article

Reliable MapReduce computing on opportunistic resources

Authors:

Wu-Chun FengAuthors Info & Claims

Cluster Computing, Volume 15, Issue 2

Pages 145 - 161

https://doi.org/10.1007/s10586-011-0158-7

Published: 01 June 2012 Publication History

Abstract

MapReduce offers an ease-of-use programming paradigm for processing large data sets, making it an attractive model for opportunistic compute resources. However, unlike dedicated resources, where MapReduce has mostly been deployed, opportunistic resources have significantly higher rates of node volatility. As a consequence, the data and task replication scheme adopted by existing MapReduce implementations is woefully inadequate on such volatile resources.

In this paper, we propose MOON, short for MapReduce On Opportunistic eNvironments, which is designed to offer reliable MapReduce service for opportunistic computing. MOON adopts a hybrid resource architecture by supplementing opportunistic compute resources with a small set of dedicated resources, and it extends Hadoop, an open-source implementation of MapReduce, with adaptive task and data scheduling algorithms to take advantage of the hybrid resource architecture. Our results on an emulated opportunistic computing system running atop a 60-node cluster demonstrate that MOON can deliver significant performance improvements to Hadoop on volatile compute resources and even finish jobs that are not able to complete in Hadoop.

References

[1]

Hadoop. http://hadoop.apache.org/core/

[2]

Spot Instances on Amazon EC2. http://aws.amazon.com/ec2/ spot-instances/

[3]

Adya, A., Bolosky, W., Castro, M., Chaiken, R., Cermak, G., Douceur, J., Howell, J., Lorch, J., Theimer, M., Wattenhofer, R.: FARSITE: federated, available, and reliable storage for an incompletely trusted environment. In: Proceedings of the 5th Symposium on Operating Systems Design and Implementation (2002).

Digital Library

[4]

Anderson, D.: Boinc: a system for public-resource computing and storage. In: IEEE/ACM International Workshop on Grid Computing (2004).

[5]

Apple Inc. Xgrid. http://www.apple.com/server/macosx/ technology/xgrid.html

[6]

Averitt, S., Bugaev, M., Peeler, A., Shaffer, H., Sills, E., Stein, S., Thompson, J., Vouk, M.: Virtual computing laboratory (VCL). In: International of the International Conference on Virtual Computing Initiative (2007).

[7]

Chen, S., Schlosser, S.: Map-reduce meets wider varieties of applications meets wider varieties of applications. Technical report IRP-TR-08-05, Intel research (2008).

[8]

Chien, A., Calder, B., Elbert, S., Bhatia, K.: Entropia: Architecture and performance of an enterprise desktop grid system. J. Parallel Distrib. Comput. 63, 597-610 (2003).

Digital Library

[9]

Chun, B.-G., Dabek, F., Haeberlen, A., Sit, E., Weatherspoon, H., Kaashoek, M.F., Kubiatowicz, J., Morris, R.: Efficient replica maintenance for distributed storage systems. In: NSDI'06: Proceedings of the 3rd conference on Networked Systems Design & Implementation, Berkeley, CA, USA, pp. 4-4. USENIX Association, Berkeley (2006).

[10]

Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107-113 (2008).

Digital Library

[11]

Fedak, G., He, H., Cappello, F.: Bitdew: a programmable environment for large-scale data management and distribution. In: SC '08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing, Piscataway, NJ, USA, pp. 1-12. IEEE Press, New York (2008).

Digital Library

[12]

Gharaibeh, A., Ripeanu, M.: Exploring data reliability tradeoffs in replicated storage systems. In: HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing, New York, NY, USA, pp. 217-226. ACM, New York (2009).

Digital Library

[13]

Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. In: Proceedings of the 19th Symposium on Operating Systems Principles (2003).

[14]

Grant, M., Sehrish, S., Bent, J., Wang, J.: Introducing map-reduce to high end computing. In: 3rd Petascale Data Storage Workshop, Nov (2008).

[15]

GridGain Systems, LLC. Gridgain. http://www.gridgain.com/

[16]

Gupta, A., Lin, B., Dinda, P.A.: Measuring and understanding user comfort with resource borrowing. In: HPDC '04: Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing, Washington, DC, USA, pp. 214- 224. IEEE Computer Society, Los Alamitos (2004).

[17]

Haeberlen, A., Mislove, A., Druschel, P.: Glacier: Highly durable, decentralized storage despite massive correlated failures. In: Proceedings of the 2nd Symposium on Networked Systems Design and Implementation (NSDI'05), May (2005).

Digital Library

[18]

Ko, S., Hoque, I., Cho, B., Gupta, I.: On availability of intermediate data in cloud computations. In: 12th Workshop on Hot Topics in Operating Systems (HotOS XII) (2009).

[19]

Kondo, D., Taufe, M., Brooks, C., Casanova, H., Chien, A.: Characterizing and evaluating desktop grids: an empirical study. In: Proceedings of the 18th International Parallel and Distributed Processing Symposium (2004).

[20]

Matsunaga, A., Tsugawa, M., Fortes, J.: Cloudblast: combining mapreduce and virtualization on distributed resources for bioinformatics. In: Microsoft eScience Workshop (2008).

[21]

Strickland, J., Freeh, V., Ma, X., Vazhkudai, S.: Governor: Autonomic throttling for aggressive idle resource scavenging. In: Proceedings of the 2nd IEEE International Conference on Autonomic Computing (2005).

[22]

Sun Microsystems. Compute server. https://computeserver.dev. java.net/

[23]

Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. In: Concurrency and Computation: Practice and Experience (2004).

[24]

Vazhkudai, S., Ma, X., Freeh, V., Strickland, J., Tammineedi, N., Scott, S.: Freeloader: scavenging desktop storage resources for bulk, transient data. In: Proceedings of Supercomputing (2005).

[25]

Zaharia, M., Konwinski, A., Joseph, A., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: OSDI (2008).

[26]

Zhong, M., Shen, K., Seiferas, J.: Replication degree customization for high availability. SIGOPS Oper. Syst. Rev. 42(4), 55-68 (2008).

Digital Library

Cited By

Esposito CFicco M(2016)Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduceInternational Journal of Data Warehousing and Mining10.4018/IJDWM.201601010412:1(49-68)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.4018/IJDWM.2016010104
Memishi BPérez MAntoniu G(2016)Feedback-Based Resource Allocation in MapReduce-Based SystemsScientific Programming10.1155/2016/72419282016Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1155/2016/7241928

Reliable MapReduce computing on opportunistic resources

Recommendations

MOON: MapReduce On Opportunistic eNvironments
HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

MapReduce offers an ease-of-use programming paradigm for processing large data sets, making it an attractive model for distributed volunteer computing systems. However, unlike on dedicated resources, where MapReduce has mostly been deployed, such ...
MapReduce on opportunistic resources leveraging resource availability
CLOUDCOM '12: Proceedings of the 2012 IEEE 4th International Conference on Cloud Computing Technology and Science (CloudCom)

MapReduce is a popular large-scale parallel data processing framework. In the context of MapReduce processing on volunteer computing environments, it is important to devise scheduling and data placement policies that account for characteristics of ...
Adaptive Combiner for MapReduce on cloud computing

MapReduce is a programming model to process a massive amount of data on cloud computing. MapReduce processes data in two phases and needs to transfer intermediate data among computers between phases. MapReduce allows programmers to aggregate ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Cluster Computing

Cluster Computing Volume 15, Issue 2

June 2012

119 pages

ISSN:1386-7857

Issue’s Table of Contents

Copyright © Copyright © 2012 Springer Science+Business Media, LLC.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Esposito CFicco M(2016)Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduceInternational Journal of Data Warehousing and Mining10.4018/IJDWM.201601010412:1(49-68)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.4018/IJDWM.2016010104
Memishi BPérez MAntoniu G(2016)Feedback-Based Resource Allocation in MapReduce-Based SystemsScientific Programming10.1155/2016/72419282016Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1155/2016/7241928

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents