Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Mirror Mirror on the Wall, How Do I Dimension My Cloud After All?

  • Chapter
  • First Online:
Cloud Computing

Part of the book series: Computer Communications and Networks ((CCN))

Abstract

Clouds are a reality both in commercial and scientific domains. It is a fact that clouds are not only an IT outsourcing, but an opportunity to foster the development of complex scientific applications over distributed resources in several domains from bioinformatics to astronomy. Although clouds provide several advantages such as elasticity and a pay-as-you-go model, such characteristics come at a price. One important drawback of clouds is how do estimate the amount of resources to deploy. Depending on the type of application, it may be not simple to estimate the necessary amount of resources. This complexity may lead to over- or under-dimensioning, which are not desired. This chapter addresses the problem of dimensioning the amount of virtual machines (VMs) in clouds for executing high performance computing (HPC) scientific applications. The aim of this chapter is to present existing approaches that estimate in a static or dynamic way the amount of VMs for several types of applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://aws.amazon.com/

  2. 2.

    cloud.google.com/

  3. 3.

    http://www.ibm.com/cloud-computing/

  4. 4.

    https://www.rackspace.com/

  5. 5.

    https://azure.microsoft.com/

  6. 6.

    http://magellan.alcf.anl.gov

  7. 7.

    http://nebula.nasa.gov

  8. 8.

    https://aws.amazon.com/ec2/faqs/

  9. 9.

    https://github.com/s3fs-fuse/s3fs-fuse

References

  1. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ (1990) Basic local alignment search tool. J Mol Biol 215(3):403–410. citeseer.nj.nec.com/akutsu99identification.html

  2. Alvares de Oliveira F, Sharrock R, Ledoux T (2012) Synchronization of multiple autonomic control loops: application to cloud computing. In: Proceedings of the 14th international conference on coordination models and languages, COORDINATION 2012. Springer, Berlin/Heidelberg, pp 29–43

    Google Scholar 

  3. Blom J, Albaum SP, Doppmeier D, Puhler A, Vorholter FJ, Zakrzewski M, Goesmann A (2009) EDGAR: a software framework for the comparative analysis of prokaryotic genomes. BMC Bioinform 10(1):154. doi: 10.1186/1471-2105-10-154, http://www.biomedcentral.com/1471-2105/10/154

  4. Buyya R, Ranjan R, Calheiros R (2010) InterCloud: utility-oriented federation of cloud computing environments for scaling of application services. In: Hsu CH, Yang L, Park J, Yeo SS (eds) Algorithms and architectures for parallel processing. Lecture notes in computer science, vol 6081. Springer, Berlin/Heidelberg, pp 13–31

    Chapter  Google Scholar 

  5. Chaisiri S, Lee BS, Niyato D (2012) Optimization of resource provisioning cost in cloud computing. IEEE Trans Serv Comput 5(2):164–177

    Article  Google Scholar 

  6. Chard R, Chard K, Bubendorfer K, Lacinski L, Madduri R, Foster I (2015) Cost-aware elastic cloud provisioning for scientific workloads. In: 2015 IEEE 8th international conference on cloud computing (CLOUD), pp 971–974

    Google Scholar 

  7. Collela P (2004) Defining software requirements for scientific computing. In: DARPA reports, pp 315–320

    Google Scholar 

  8. Coutinho R, Drummond L, Frota Y (2014) Optimization of a cloud resource management problem from a consumer perspective. In: Euro-Par 2013: parallel processing workshops. Lecture notes in computer science, vol 8374. Springer, Berlin/Heidelberg, pp 218–227

    Google Scholar 

  9. Coutinho R, Drummond L, Frota Y, de Oliveira D, Ocaña K (2014) Evaluating grasp-based cloud dimensioning for comparative genomics: a practical approach. In: IEEE international conference on cluster computing (CLUSTER), pp 371–379

    Google Scholar 

  10. Coutinho R, Drummond L, Frota Y, de Oliveira D (2015) Optimizing virtual machine allocation for parallel scientific workflows in federated clouds. Future Gener Comput Syst 46(0):51–68

    Google Scholar 

  11. Coutinho R, Frota Y, Ocaña K, de Oliveira D, Drummond LMA (2016) A dynamic cloud dimensioning approach for parallel scientific workflows: a case study in the comparative genomics domain. J Grid Comput 1–19

    Google Scholar 

  12. Crawl D, Wang J, Altintas I (2011) Provenance for MapReduce-based data-intensive workflows. In: Proceedings of the 6th workshop on workflows in support of large-scale science, WORKS ’11. ACM, New York, pp 21–30

    Google Scholar 

  13. Dean J, Ghemawat S (2004) MapReduce: simplified data processing on large clusters. In: Proceedings of the 6th conference on symposium on opearting systems design & implementation, OSDI’04, vol 6. USENIX Association, Berkeley, pp 10–10

    Google Scholar 

  14. Deelman E, Singh G, Su MH, Blythe J, Gil Y, Kesselman C, Mehta G, Vahi K, Berriman GB, Good J, Laity AC, Jacob JC, Katz DS (2005) Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci Program 13(3):219–237

    Google Scholar 

  15. Deng K, Song J, Ren K, Iosup A (2013) Exploring portfolio scheduling forlong-term execution of scientific workloads in IaaS clouds. In: Proceedings of SC13: international conference for high performance computing, networking, storage and analysis, SC ’13. ACM, New York, pp 55:1–55:12

    Google Scholar 

  16. de Oliveira D, Ogasawara E, Baião F, Mattoso M: Scicumulus: a lightweight cloud middleware to explore many task computing paradigm in scientific workflows. In: 3rd international conference on cloud computing (2010), pp 378–385

    Google Scholar 

  17. de Oliveira D, Ocaña KA, Ogasawara E, Dias J, Gonçalves J, Baião F, Mattoso M (2013) Performance evaluation of parallel strategies in public clouds: a study with phylogenomic workflows. Future Gener Comput Syst 29(7):1816–1825

    Google Scholar 

  18. de Oliveira D, Viana V, Ogasawara E, Ocaña K, Mattoso M (2013) Dimensioning the virtual cluster for parallel scientific workflows in clouds. In: Proceedings of the 4th ACM workshop on scientific cloud computing, science cloud ’13. ACM, New York, pp 5–12

    Google Scholar 

  19. Emeakaroha V, Maurer M, Stern P, Łabaj P, Brandic I, Kreil D (2013) Managing and optimizing bioinformatics workflows for data analysis in clouds. J Grid Comput 11(3):407–428

    Article  Google Scholar 

  20. Endo PT, de Almeida Palhares AV, Pereira NN, Goncalves GE, Sadok D, Kelner J, Melander B, Mangs J (2011) Resource allocation for distributed cloud: concepts and research challenges. IEEE Network 25(4):42–46

    Google Scholar 

  21. Engen V, Papay J, Phillips SC, Boniface M (2012) Predicting application performance for multi-vendor clouds using dwarf benchmarks. In: Proceedings of the 13th international conference on web information systems engineering, WISE’12. Springer, Berlin/Heidelberg, pp 659–665. doi: 10.1007/978-3-642-35063-4_50, http://dx.doi.org/10.1007/978-3-642-35063-4_50

  22. Fadika Z, Dede E, Hartog J, Govindaraju M (2012) Marla: mapreduce for heterogeneous clusters. In: Proceedings of the 2012 12th IEEE/ACM international symposium on cluster, cloud and grid computing (Ccgrid 2012), CCGRID ’12. IEEE Computer Society, Washington, DC, pp 49–56. doi: 10.1109/CCGrid.2012.135, http://dx.doi.org/10.1109/CCGrid.2012.135

  23. Feng H, Misra V, Rubenstein D (2007) Pbs: a unified priority-based scheduler. In: Proceedings of the 2007 ACM SIGMETRICS international conference on measurement and modeling of computer systems, SIGMETRICS ’07. ACM, New York, pp 203–214. doi: 10.1145/1254882.1254906, http://doi.acm.org/10.1145/1254882.1254906

  24. Foster I, Kesselman C (2003) The grid 2: blueprint for a new computing infrastructure. The Elsevier series in grid computing, 2nd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  25. Freire J, Koop D, Santos E, Silva CT (2008) Provenance for computational tasks: a survey. Comput Sci Eng 10(3):11–21

    Article  Google Scholar 

  26. Habib I (2006) Getting started with condor. Linux J 2006(149):2–. http://dl.acm.org/citation.cfm?id=1152899.1152901

  27. Heilig L, Lalla-Ruiz E, Voß S (2016) A cloud brokerage approach for solving the resource management problem in multi-cloud environments. Comput Ind Eng 95:16–26

    Article  Google Scholar 

  28. Hey T, Tansley S, Tolle K (eds) (2009): The fourth paradigm: data-intensive scientific discovery. Microsoft Research, Redmond

    Google Scholar 

  29. ILOG SA (2008) Cplex 11 user’s manual

    Google Scholar 

  30. Jackson KR, Ramakrishnan L, Runge KJ, Thomas RC (2010) Seeking supernovae in the clouds: a performance study. In: Proceedings of the 19th ACM international symposium on high performance distributed computing, HPDC ’10. ACM, New York, pp 421–429

    Chapter  Google Scholar 

  31. Jamshidi P, Ahmad A, Pahl C (2013) Cloud migration research: a systematic review. IEEE Trans Cloud Comput 1(2):142–157. doi: 10.1109/TCC.2013.10

    Article  Google Scholar 

  32. Joshi SB (2012) Apache hadoop performance-tuning methodologies and best practices. In: Proceedings of the 3rd ACM/SPEC international conference on performance engineering, ICPE ’12. ACM, New York, pp 241–242. doi: 10.1145/2188286.2188323, http://doi.acm.org/10.1145/2188286.2188323

  33. Juve G, Deelman E (2010) Scientific workflows and clouds. Crossroads 16(3):14–18. doi: 10.1145/1734160.1734166, http://doi.acm.org/10.1145/1734160.1734166

  34. Kitchenham B, Brereton P, Turner M, Niazi M, Linkman S, Pretorius R, Budgen D (2009) The impact of limited search procedures for systematic literature reviews #x2014; a participant-observer case study. In: 2009 3rd international symposium on empirical software engineering and measurement, pp 336–345. doi: 10.1109/ESEM.2009.5314238

  35. Lama P, Zhou X (2012) AROMA: automated resource allocation and configuration of MapReduce environment in the cloud. In: Proceedings of the 9th international conference on autonomic computing, ICAC ’12. ACM, New York, pp 63–72

    Chapter  Google Scholar 

  36. Lord E, Leclercq M, Boc A, Diallo AB, Makarenkov V (2012) Armadillo 1.1: an original workflow platform for designing and conducting phylogenetic analysis and simulations. PLoS ONE 7(1):e29903. doi: 10.1371/journal.pone.0029903, http://dx.plos.org/10.1371/journal.pone.0029903

  37. Ludäscher B, Altintas I, Berkley C, Higgins D, Jaeger E, Jones MB, Lee EA, Tao J, Zhao Y (2006) Scientific workflow management and the Kepler system. Concurr Comput: Pract Exp 18(10):1039–1065. doi: 10.1002/cpe.994, http://dx.doi.org/10.1002/cpe.994

  38. Maheshwari K, Jung ES, Meng J, Morozov V, Vishwanath V, Kettimuthu R (2016) Workflow performance improvement using model-based scheduling over multiple clusters and clouds. Future Gener Comput Syst 54:206–218

    Article  Google Scholar 

  39. Malawski M, Juve G, Deelman E, Nabrzyski J (2015) Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. Future Gener Comput Syst 48:1–18. Special Section: Business and Industry Specific Cloud

    Google Scholar 

  40. Manfroi LF, Ferro M, Yokoyama AM, Mury AR, Schulze B (2013) A walking dwarf on the clouds. In: 2013 IEEE/ACM 6th international conference on utility and cloud computing (UCC), pp 399–404. doi: 10.1109/UCC.2013.80

  41. Matsunaga A, Tsugawa M, Fortes J (2008) Cloudblast: combining mapreduce and virtualization on distributed resources for bioinformatics applications. In: IEEE fourth international conference on eScience, eScience ’08, pp 222–229. doi: 10.1109/eScience.2008.62

  42. Mattoso M, Werner C, Travassos GH, Braganholo V, Ogasawara E, Oliveira DD, Cruz SM, Martinho W, Murta L (2010) Towards supporting the life cycle of large scale scientific experiments. Int J Bus Process Integr Manag 5(1):79+

    Google Scholar 

  43. Moustafa A, Bhattacharya D, Allen AE (2010) iTree: a high-throughput phylogenomic pipeline. IEEE, Cairo, pp 103–107. doi: 10.1109/CIBEC.2010.5716071, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5716071

  44. Nguyen P, Halem M (2011) A MapReduce workflow system for architecting scientific data intensive applications. In: Proceedings of the 2nd international workshop on software engineering for cloud computing, SECLOUD ’11. ACM, New York, pp 57–63

    Chapter  Google Scholar 

  45. Niemenmaa M, Kallio A, Schumacher A, Klemela P, Korpelainen E, Heljanko K (2012) Hadoop-BAM: directly manipulating next generation sequencing data in the cloud. Bioinformatics 28(6):876–877. doi: 10.1093/bioinformatics/bts054, http://bioinformatics.oxfordjournals.org/cgi/doi/10.1093/bioinformatics/bts054

  46. Ocaña K, de Oliveira D, Ogasawara ES, Dávila AMR, Lima AAB, Mattoso M (2011) SciPhy: a cloud-based workflow for phylogenetic analysis of drug targets in protozoan genomes. In: de Souza ON, Telles GP, Palakal MJ (eds) BSB. Lecture notes in computer science, vol 6832. Springer, pp 66–70

    Google Scholar 

  47. Paranjape K, Hebert S, Masson B (2012) Heterogeneous computing in the cloud: crunching big data and democratizing HPC access for the life sciences. Technical report, Intel Corporation

    Google Scholar 

  48. Phillips SC, Engen V, Papay J (2011) Snow white clouds and the seven dwarfs. In: 2011 IEEE third international conference on cloud computing technology and science (CloudCom), pp 738–745 doi: 10.1109/CloudCom.2011.114

  49. Prodan R, Wieczorek M, Fard H (2011) Double auction-based scheduling of scientific applications in distributed grid and cloud environments. J Grid Comput 9(4):531–548

    Article  Google Scholar 

  50. Rodero I, Viswanathan H, Lee EK, Gamell M, Pompili D, Parashar M (2012) Energy-efficient thermal-aware autonomic management of virtualized HPC cloud infrastructure. J Grid Comput 10(3):447–473

    Article  Google Scholar 

  51. Severin J, Beal K, Vilella AJ, Fitzgerald S, Schuster M, Gordon L, Ureta-Vidal A, Flicek P, Herrero J (2010) eHive: an artificial intelligence workflow system for genomic analysis. BMC Bioinform 11(1):240. doi: 10.1186/1471-2105-11-240, http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-240

  52. Shanahan JG, Dai L (2015) Large scale distributed data science using apache spark. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’15. ACM, New York, pp 2323–2324 doi: 10.1145/2783258.2789993, http://doi.acm.org/10.1145/2783258.2789993

  53. Shen Z, Subbiah S, Gu X, Wilkes J (2011) Cloudscale: elastic resource scaling for multi-tenant cloud systems. In: Proceedings of the 2nd ACM symposium on cloud computing, SOCC ’11. ACM, New York, pp 5:1–5:14

    Google Scholar 

  54. Singh A, Chen C, Liu W, Mitchell W, Schmidt B: A hybrid computational grid architecture for comparative genomics. IEEE Trans Inf Technol Biomed 12(2):218–225 (2008). doi: 10.1109/TITB.2007.908462, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4358919

  55. Szabo C, Sheng Q, Kroeger T, Zhang Y, Yu J (2014) Science in the cloud: allocation and execution of data-intensive scientific workflows. J Grid Comput 12(2):245–264

    Article  Google Scholar 

  56. Taylor IJ, Deelman E, Gannon DB (2007) Workflows for e-science: scientific workflows for grids. Springer, London

    Book  Google Scholar 

  57. Tian W (2009) adaptive dimensioning of cloud data centers. In: Proceedings of the 8th international conference on dependable, autonomic and secure computing, DASC ’09. IEEE Computer Society, Washington, pp 5–10

    Google Scholar 

  58. Vaquero LM, Rodero-Merino L, Caceres J, Lindner M (2008) A break in the clouds: towards a cloud definition. SIGCOMM Comput Commun Rev 39(1):50–55

    Article  Google Scholar 

  59. Wall DP, Kudtarkar P, Fusaro VA, Pivovarov R, Patil P, Tonellato PJ (2010) Cloud computing for comparative genomics. BMC Bioinform 11(1):259. doi: 10.1186/1471-2105-11-259, http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-259

  60. Wang J, Crawl D, Altintas I (2009) Kepler + Hadoop: a general architecture facilitating data-intensive applications in scientific workflow systems. In: Proceedings of the 4th workshop on workflows in support of large-scale science, WORKS ’09. ACM, New York, pp 12:1–12:8

    Google Scholar 

  61. Wolstencroft K, Haines R, Fellows D, Williams AR, Withers D, Owen S, Soiland-Reyes S, Dunlop I, Nenadic A, Fisher P, Bhagat J, Belhajjame K, Bacall F, Hardisty A, de la Hidalga AN, Vargas MPB, Sufi S, Goble CA (2013) The Taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucleic Acids Res 41(Webserver-Issue):557–561. doi: 10.1093/nar/gkt328, http://dx.doi.org/10.1093/nar/gkt328

  62. Wozniak JM, Armstrong TG, Maheshwari K, Lusk EL, Katz DS, Wilde M, Foster IT (2013) Turbine: a distributed memory dataflow engine for high performance many-task applications. Fundamenta Informaticae Journal 128(3):337–366

    Google Scholar 

  63. Xiao Z, Song W, Chen Q (2013) dynamic resource allocation using virtual machines for cloud computing environment. IEEE Trans Parallel Distrib Syst 24(6):1107–1117

    Article  Google Scholar 

  64. Xu L, Zeng Z, Ye X (2012) Multi-objective optimization based virtual resource allocation strategy for cloud computing. In: Proceedings of the 11th international conference on computer and information science, ICIS ’12. IEEE Computer Society, Washington, DC, pp 56–61

    Google Scholar 

Download references

Acknowledgements

Authors would like to thank CNPq and FAPERJ for partially sponsoring this research.

Author information

Authors and Affiliations

  1. Federal Center of Technological Education, Nova Iguaçu, Brazil

    Rafaelli Coutinho

  2. Fluminense Federal University, Niterói, Brazil

    Yuri Frota, Daniel de Oliveira & Lúcia M. A. Drummond

  3. National Laboratory of Scientific Computing, Petrópolis, Brazil

    Kary Ocaña

Authors
  1. Rafaelli Coutinho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuri Frota
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kary Ocaña
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lúcia M. A. Drummond
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rafaelli Coutinho .

Editor information

Editors and Affiliations

  1. University of Derby, Derby, Derbyshire, United Kingdom

    Nick Antonopoulos

  2. University of Surrey, Guildford, Surrey, United Kingdom

    Lee Gillam

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Coutinho, R., Frota, Y., Ocaña, K., de Oliveira, D., Drummond, L.M.A. (2017). Mirror Mirror on the Wall, How Do I Dimension My Cloud After All?. In: Antonopoulos, N., Gillam, L. (eds) Cloud Computing. Computer Communications and Networks. Springer, Cham. https://doi.org/10.1007/978-3-319-54645-2_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54645-2_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54644-5

  • Online ISBN: 978-3-319-54645-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation