research-article

Conductor: orchestrating the clouds

Authors:

Alexander Wieder,

Pramod Bhatotia,

Ansley Post,

Rodrigo RodriguesAuthors Info & Claims

LADIS '10: Proceedings of the 4th International Workshop on Large Scale Distributed Systems and Middleware

Pages 44 - 48

https://doi.org/10.1145/1859184.1859197

Published: 28 July 2010 Publication History

Get Access

Abstract

Cloud computing enables customers to access virtually unlimited resources on demand and without any fixed upfront cost. However, the commoditization of computing resources imposes new challenges in how to manage them: customers of cloud services are no longer restricted to the resources they own, but instead choose from a variety of different services offered by different providers, and the impact of these choices on price and overall performance is not always clear. Furthermore, having to take into account new cloud products and services, the cost of recovering from faults, or price fluctuations due to spot markets makes the picture even more unclear.

This position paper highlights a series of challenges that must be overcome in order to allow customers to better lever-age cloud resources. We also make the case for a system called Conductor that automatically manages resources in cloud computing to meet user-specifiable optimization goals, such as minimizing monetary cost or completion time. Finally, we discuss some of the challenges we will face in building such a system.

References

[1]

}}Chohan, N., Castillo, C., Spreitzer, M., Steinder, M., Tantawi, A., and Krintz, C. See Spot Run: Using spot instances for MapReduce workflows. In 2nd USENIX Workshop on Hot Topics in Cloud Computing (June 2010).

Digital Library

Google Scholar

[2]

}}Dusseau, A. C., Arpaci, R. H., and Culler, D. E. Effective distributed scheduling of parallel workloads. In SIGMETRICS '96: Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems (1996), pp. 25--36.

Digital Library

Google Scholar

[3]

}}Gottfrid, D. Self-service, Prorated Super Computing Fun! http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/.

Google Scholar

[4]

}}Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., and Goldberg, A. Quincy: fair scheduling for distributed computing clusters. In SOSP '09: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (2009), pp. 261--276.

Digital Library

Google Scholar

[5]

}}Keeton, K., Kelly, T., Merchant, A., Santos, C., Wiener, J., Zhu, X., and Beyer, D. Don't settle for less than the best: Use optimization to make decisions. In HOTOS'07: Proceedings of the 11th USENIX workshop on Hot topics in operating systems (May 2007), pp. 1--6.

Digital Library

Google Scholar

[6]

}}Ko, S. Y., Hoque, I., Cho, B., and Gupta, I. On availability of intermediate data in cloud computations. In Proceedings of HotOS'09: 12th Workshop on Hot Topics in Operating Systems.

Digital Library

Google Scholar

[7]

}}Krawczyk, S., and Bubendorfer, K. Grid resource allocation: allocation mechanisms and utilisation patterns. In AusGrid '08: Proceedings of the sixth Australasian workshop on Grid computing and e-research (Darlinghurst, Australia, Australia, 2008), Australian Computer Society, Inc., pp. 73--81.

Digital Library

Google Scholar

[8]

}}Muthitacharoen, A., Morris, R., Gil, T. M., and Chen, B. Ivy: a read/write peer-to-peer file system. SIGOPS Operating Systems Review 36, SI (2002), 31--44.

Digital Library

Google Scholar

[9]

}}Olston, C., Reed, B., Srivastava, U., Kumar, R., and Tomkins, A. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data (2008), pp. 1099--1110.

Digital Library

Google Scholar

[10]

}}Qureshi, A., Weber, R., Balakrishnan, H., Guttag, J., and Maggs, B. Cutting the electric bill for internet-scale systems. In SIGCOMM '09: Proceedings of the ACM SIGCOMM 2009 conference on Data communication (2009), pp. 123--134.

Digital Library

Google Scholar

[11]

}}Raman, R., Livny, M., and Solomon, M. Matchmaking: Distributed resource management for high throughput computing. In HPDC '98: Proceedings of the 7th IEEE International Symposium on High Performance Distributed Computing (Washington, DC, USA, 1998), IEEE Computer Society, p. 140.

Digital Library

Google Scholar

[12]

}}Wang, G., Butt, A. R., Pandey, P., and Gupta, K. A simulation approach to evaluating design decisions in mapreduce setups. In MASCOTS '09: Proceedings of the International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems.

Google Scholar

[13]

}}Wieder, A., Bhatotia, P., Post, A., and Rodrigues, R. Brief Announcement: Modelling MapReduce for Optimal Execution in the Cloud. In PODC '10: Proceedings of the 29th ACM symposium on Principles of distributed computing.

Digital Library

Google Scholar

[14]

}}Yin, Q., Schüpbach, A., Cappos, J., Baumann, A., and Roscoe, T. Rhizoma: a runtime for self-deploying, self-managing overlays. In Middleware '09: Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware (2009), pp. 1--20.

Digital Library

Google Scholar

Cited By

View all

Bhatotia PAcar UJunqueira FRodrigues R(2019)Incremental Sliding Window AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_156(1007-1015)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_156
Quoc DChen RBhatotia PFetzer CHilt VStrufe T(2019)Approximate Computing for Stream AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_153(90-97)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_153
Quoc DBeck MBhatotia PChen RFetzer CStrufe T(2019)Privacy-Preserving Data AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_152(1292-1300)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_152
Show More Cited By

Index Terms

Conductor: orchestrating the clouds
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles

Recommendations

Guest Editors' introduction: Special section on optimizing the cloud

Cloud computing is defined as a pool of virtualized computer resources. Based on this virtualization, the cloud computing paradigm allows workloads to be deployed and scaled-out quickly through the rapid provisioning of virtual machines or physical ...
Winds of Change: From Vendor Lock-In to the Meta Cloud

The emergence of yet more cloud offerings from a multitude of service providers calls for a meta cloud to smoothen the edges of the jagged cloud landscape. This meta cloud could solve the vendor lock-in problems that current public and hybrid cloud ...
Challenges in real-time virtualization and predictable cloud computing

Cloud computing and virtualization technology have revolutionized general-purpose computing applications in the past decade. The cloud paradigm offers advantages through reduction of operation costs, server consolidation, flexible system configuration ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

LADIS '10: Proceedings of the 4th International Workshop on Large Scale Distributed Systems and Middleware

July 2010

65 pages

ISBN:9781450304061

DOI:10.1145/1859184

General Chairs:
Gregory Chockler
IBM Research Haifa, Israel
,
Ymir Vigfusson
IBM Research Haifa, Israel
,
Program Chairs:
Marcos K. Aguilera
Microsoft Research, USA
,
Marc Shapiro
INRIA & LIP6, France

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PODC '10

Sponsor:

PODC '10: ACM Symposium on Principles of Distributed Computing

July 28 - 29, 2010

Zürich, Switzerland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
337
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Bhatotia PAcar UJunqueira FRodrigues R(2019)Incremental Sliding Window AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_156(1007-1015)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_156
Quoc DChen RBhatotia PFetzer CHilt VStrufe T(2019)Approximate Computing for Stream AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_153(90-97)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_153
Quoc DBeck MBhatotia PChen RFetzer CStrufe T(2019)Privacy-Preserving Data AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_152(1292-1300)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_152
Quoc DKrishnan DBhatotia PFetzer CRodrigues R(2019)Incremental Approximate ComputingEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_151(1000-1007)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_151
Bhatotia PAcar UJunqueira FRodrigues R(2018)Incremental Sliding Window AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_156-1(1-8)Online publication date: 6-Mar-2018
https://doi.org/10.1007/978-3-319-63962-8_156-1
Le Quoc DChen RBhatotia PFetzer CHilt VStrufe T(2018)Approximate Computing for Stream AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_153-1(1-8)Online publication date: 6-Feb-2018
https://doi.org/10.1007/978-3-319-63962-8_153-1
Quoc DBeck MBhatotia PChen RFetzer CStrufe T(2018)Privacy-Preserving Data AnalyticsEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_152-1(1-8)Online publication date: 1-Feb-2018
https://doi.org/10.1007/978-3-319-63962-8_152-1
Quoc DKrishnan DBhatotia PFetzer CRodrigues R(2018)Incremental Approximate ComputingEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_151-1(1-8)Online publication date: 1-Feb-2018
https://doi.org/10.1007/978-3-319-63962-8_151-1
Quoc DChen RBhatotia PFetzer CHilt VStrufe TJayaram KGandhi AKemme BPietzuch P(2017)StreamApproxProceedings of the 18th ACM/IFIP/USENIX Middleware Conference10.1145/3135974.3135989(185-197)Online publication date: 11-Dec-2017
https://dl.acm.org/doi/10.1145/3135974.3135989
Thalheim JRodrigues AAkkus IBhatotia PChen RViswanath BJiao LFetzer CJayaram KGandhi AKemme BPietzuch P(2017)SieveProceedings of the 18th ACM/IFIP/USENIX Middleware Conference10.1145/3135974.3135977(14-27)Online publication date: 11-Dec-2017
https://dl.acm.org/doi/10.1145/3135974.3135977
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Guest Editors' introduction: Special section on optimizing the cloud

Winds of Change: From Vendor Lock-In to the Meta Cloud

Challenges in real-time virtualization and predictable cloud computing