Article

Workload analysis of a cluster in a grid environment

Author:

Emmanuel MedernachAuthors Info & Claims

JSSPP'05: Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing

Pages 36 - 61

https://doi.org/10.1007/11605300_2

Published: 19 June 2005 Publication History

Abstract

With Grids, we are able to share computing resources and to provide for scientific communities a global transparent access to local facilities. In such an environment the problems of fair resource sharing and best usage arise. In this paper, the analysis of the LPC cluster usage (Laboratoire de Physique Corpusculaire, Clermont-Ferrand, France) in the EGEE Grid environment is done, and from the results a model for job arrival is proposed.

References

[1]

Dror G. Feitelson. Workload modeling for performance evaluation. In Maria Carla Calzarossa and Salvatore Tucci, editors, Performance Evaluation of Complex Systems: Techniques and Tools, pages 114-141. Springer-Verlag, Sep 2002. Lect. Notes Comput. Sci. vol. 2459.

Digital Library

[2]

Darin England and Jon B. Weissman. Costs and benefits of load sharing in the computational grid. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing. Springer-Verlag, 2004.

Digital Library

[3]

M. Garey and D.S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman, San Francisco, CA., 1979.

Digital Library

[4]

Stephan Mertens. The easiest hard problem: Number partitioning. In A.G. Percus, G. Istrate, and C. Moore, editors, Computational Complexity and Statistical Physics, New York, 2004. Oxford University Press.

[5]

Dror G. Feitelson and Larry Rudolph. Parallel job scheduling: Issues and approaches. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 1-18. Springer-Verlag, 1995. Lect. Notes Comput. Sci. vol. 949.

Digital Library

[6]

David Jackson, Quinn Snell, and Mark Clement. Core algorithms of the Maui scheduler. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 87-102. Springer Verlag, 2001. Lect. Notes Comput. Sci. vol. 2221.

Digital Library

[7]

Brett Bode, David M. Halstead, Ricky Kendall, and Zhou Lei. The Portable Batch Scheduler and the Maui Scheduler on Linux Clusters, USENIX Association. 4th Annual Linux Showcase Conference, 2000.

Digital Library

[8]

S. Agostinelli et al. Geant 4 (GEometry ANd Tracking): a Simulation toolkit. Nuclear Instruments and Methods in Physics Research, pages 250-303, 2003.

[9]

Ian Foster and Carl Kesselman. Globus: A metacomputing infrastructure toolkit. The International Journal of Supercomputer Applications and High Performance Computing, 11(2):115-128, Summer 1997.

Digital Library

[10]

EGEE Design Team. EGEE middleware architecture, EGEE-DJRA1.1-476451- v1.0, August 2004. Also available as https://edms.cern.ch/document/476451/ 1.0.

[11]

Dmitry Zotkin and Peter J. Keleher. Job-length estimation and performance in backfilling schedulers. In HPDC, 1999.

Digital Library

[12]

Antonio Delgado Peris, Patricia Méndez Lorenzo, Flavia Donno, Andrea Sciabà, Simone Campana, and Roberto Santinelli. LCG User guide, 2004.

[13]

G. Avellino, S. Beco, B. Cantalupo, A. Maraschini, F. Pacini, M. Sottilaro, A. Terracina, D. Colling, F. Giacomini, E. Ronchieri, A. Gianelle, R. Peluso, M. Sgaravatto, A. Guarise, R. Piro, A. Werbrouck, D. Kouřil, A. Křenek, L. Matyska, M. Mulač, J. Pospíšil, M. Ruda, Z. Salvet, J. Sitera, J. Škrabal, M. Voců, M. Mezzadri, F. Prelz, S. Monforte, and M. Pappalardo. The datagrid workload management system: Challenges and results. Kluwer Academic Publishers, 2004.

[14]

Dror G. Feitelson and Larry Rudolph. Toward convergence in job schedulers for parallel supercomputers. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 1-26. Springer-Verlag, 1996. Lect. Notes Comput. Sci. vol. 1162.

Digital Library

[15]

Su-Hui Chiang, Andrea Arpaci-Dusseau, and Mary K. Vernon. The impact of more accurate requested runtimes on production job scheduling performance. In Dror G. Feitelson, Larry Rudolph, and Uwe Schwiegelshohn, editors, Job Scheduling Strategies for Parallel Processing, pages 103-127. Springer Verlag, 2002. Lect. Notes Comput. Sci. vol. 2537.

Digital Library

[16]

Maria Calzarossa and Giuseppe Serazzi. Workload characterization: A survey. Proc. IEEE, 81(8):1136-1150, 1993.

[17]

Steve J. Chapin, Walfredo Cirne, Dror G. Feitelson, James Patton Jones, Scott T. Leutenegger, Uwe Schwiegelshohn, Warren Smith, and David Talby. Benchmarks and standards for the evaluation of parallel job schedulers. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 67-90. Springer-Verlag, 1999. Lect. Notes Comput. Sci. vol. 1659.

Digital Library

[18]

Walfredo Cirne and Francine Berman. A comprehensive model of the supercomputer workload, 2001.

[19]

Allen B. Downey and Dror G. Feitelson. The elusive goal of workload characterization. Perf. Eval. Rev., 26(4):14-29, 1999.

Digital Library

[20]

Dror G. Feitelson and Bill Nitzberg. Job characteristics of a production parallel scientific workload on the NASA Ames iPSC/860. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 337-360. Springer-Verlag, 1995. Lect. Notes Comput. Sci. vol. 949.

Digital Library

[21]

Vern Paxson and Sally Floyd. Wide area traffic: the failure of Poisson modeling. IEEE/ACM Transactions on Networking, 3(3):226-244, 1995.

Digital Library

[22]

Hui Li, David Groep, and Lex Wolters. Workload characteristics of a multicluster supercomputer. In Dror G. Feitelson, Larry Rudolph, and Uwe Schwiegelshohn, editors, Job Scheduling Strategies for Parallel Processing. Springer Verlag, 2004.

Digital Library

[23]

Richard Kelsey, William Clinger, and Jonathan Rees (Editors). Revised⁵ report on the algorithmic language Scheme. ACM SIGPLAN Notices, 33(9):26-76, 1998.

Digital Library

[24]

Dror G. Feitelson. Metrics for parallel job scheduling and their convergence. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 188-205. Springer Verlag, 2001. Lect. Notes Comput. Sci. vol. 2221.

Digital Library

[25]

Joefon Jann, Pratap Pattnaik, Hubertus Franke, Fang Wang, Joseph Skovira, and Joseph Riodan. Modeling of workload in MPPs. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 95-116. Springer Verlag, 1997. Lect. Notes Comput. Sci. vol. 1291.

Digital Library

[26]

David Talby, Dror G. Feitelson, and Adi Raveh. Comparing logs and models of parallel workloads using the co-plot method. In Dror G. Feitelson and Larry Rudolph, editors, Job Scheduling Strategies for Parallel Processing, pages 43-66. Springer Verlag, 1999. Lect. Notes Comput. Sci. vol. 1659.

Digital Library

[27]

Yossi Azar, Bala Kalyansasundaram, Serge A. Plotkin, Kirk Pruhs, and Orli Waarts. On-line load balancing of temporary tasks. J. Algorithms, 22(1):93-110, 1997.

Digital Library

[28]

Yossi Azar, Andrei Z. Broder, and Anna R. Karlin. On-line load balancing. Theoretical Computer Science, 130(1):73-84, 1994.

Digital Library

[29]

A. Bar-Noy, A. Freund, and J. Naor. New algorithms for related machines with temporary jobs. In E.K. Burke, editor, Journal of Scheduling, pages 259-272. Springer-Verlag, 2000.

[30]

Tak-Wah Lam, Hing-Fung Ting, Kar-Keung To, and Wai-Ha Wong. On-line load balancing of temporary tasks revisited. Theoretical Computer Science, 270(1-2): 325-340, 2002.

Digital Library

[31]

Nazareno Andrade, Walfredo Cirne, Francisco Brasileiro, and Paulo Roisenberg. OurGrid: An approach to easily assemble grids with equitable resource sharing. In Proceedings of the 9th Workshop on Job Scheduling Strategies for Parallel Processing, June 2003.

[32]

EGEE Design Team. Design of the EGEE middleware grid services. EGEE JRA1, 2004. Also available as https://edms.cern.ch/document/487871/1.0.

Cited By

Quang BKim JRho SKim SKim SHwang SMedernach EBreton VBalaji PXu C(2015)A comparative analysis of scheduling mechanisms for virtual screening workflow in a shared resource environmentProceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2015.123(853-862)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1109/CCGrid.2015.123
Bertin RHunold SLegrand ATouati C(2014)Fair scheduling of bag-of-tasks applications using distributed Lagrangian optimizationJournal of Parallel and Distributed Computing10.1016/j.jpdc.2013.08.01174:1(1914-1929)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1016/j.jpdc.2013.08.011
Wen JMa YLiu PSun S(2014)Distributed multipliers in MWM for analyzing job arrival processes in massive HPC workload datasetsFuture Generation Computer Systems10.1016/j.future.2013.12.00937:C(335-344)Online publication date: 1-Jul-2014
https://dl.acm.org/doi/10.1016/j.future.2013.12.009
Show More Cited By

Recommendations

A grid workflow environment for brain imaging analysis on distributed systems
Special Issue: 3rd International Workshop on Workflow Management and Applications in Grid Environments (WaGe2008)

Scientific applications like neuroscience data analysis are usually compute and data-intensive. With the use of the additional capacity offered by distributed resources and suitable middlewares, we can achieve much shorter execution time, distribute ...
ATLAS grid workload on NDGF resources

Evaluating new ideas for job scheduling or data transfer algorithms in large-scale grid systems is known to be notoriously challenging. Existing grid simulators expect to receive a realistic workload as an input. Such input is difficult to obtain in the ...
Analyzing the EGEE Production Grid Workload: Application to Jobs Submission Optimization
Job Scheduling Strategies for Parallel Processing

Grids reliability remains an order of magnitude below clusters on production infrastructures. This work is aims at improving grid application performances by improving the job submission system. A stochastic model, capturing the behavior of a complex ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

JSSPP'05: Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing

June 2005

283 pages

ISBN:354031024X

Editors:
Dror Feitelson
Department of Computer Science, The Hebrew University of Jerusalem
,
Eitan Frachtenberg
Department of Computer Science, Powerset, Inc.,
,
Larry Rudolph
Department of Computer Science, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge
,
Uwe Schwiegelshohn
Department of Computer Science, No Affiliations,, 77 Massachusetts Avenue, Cambridge

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 19 June 2005

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Quang BKim JRho SKim SKim SHwang SMedernach EBreton VBalaji PXu C(2015)A comparative analysis of scheduling mechanisms for virtual screening workflow in a shared resource environmentProceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2015.123(853-862)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1109/CCGrid.2015.123
Bertin RHunold SLegrand ATouati C(2014)Fair scheduling of bag-of-tasks applications using distributed Lagrangian optimizationJournal of Parallel and Distributed Computing10.1016/j.jpdc.2013.08.01174:1(1914-1929)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1016/j.jpdc.2013.08.011
Wen JMa YLiu PSun S(2014)Distributed multipliers in MWM for analyzing job arrival processes in massive HPC workload datasetsFuture Generation Computer Systems10.1016/j.future.2013.12.00937:C(335-344)Online publication date: 1-Jul-2014
https://dl.acm.org/doi/10.1016/j.future.2013.12.009
Quang BQuang NMedernach EBreton VLong PQuyet TThanh BVan TBui MHong S(2013)Stretch optimization for virtual screening on multi-user pilot-agent platforms on grid/cloudProceedings of the 4th Symposium on Information and Communication Technology10.1145/2542050.2542063(301-310)Online publication date: 5-Dec-2013
https://dl.acm.org/doi/10.1145/2542050.2542063
Agmon Ben-Yehuda OBen-Yehuda MSchuster ATsafrir D(2013)Deconstructing Amazon EC2 Spot Instance PricingACM Transactions on Economics and Computation10.1145/2509413.25094161:3(1-20)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1145/2509413.2509416
da Silva RGlatard T(2012)A science-gateway workload archive to study pilot jobs, user activity, bag of tasks, task sub-steps, and workflow executionsProceedings of the 18th international conference on Parallel processing workshops10.1007/978-3-642-36949-0_10(79-88)Online publication date: 27-Aug-2012
https://dl.acm.org/doi/10.1007/978-3-642-36949-0_10
Tran NWolters LMaccabe AThain D(2011)Towards a profound analysis of bags-of-tasks in parallel systems and their performance impactProceedings of the 20th international symposium on High performance distributed computing10.1145/1996130.1996148(111-122)Online publication date: 8-Jun-2011
https://dl.acm.org/doi/10.1145/1996130.1996148
Mocicki JLamanna MBubak MSloot P(2011)Processing moldable tasks on the gridFuture Generation Computer Systems10.1016/j.future.2011.02.00227:6(725-736)Online publication date: 1-Jun-2011
https://dl.acm.org/doi/10.1016/j.future.2011.02.002
Berten VJeannot E(2009)Modeling resubmission in unreliable gridsProceedings of the 2009 international conference on Parallel processing10.5555/1884795.1884815(150-161)Online publication date: 25-Aug-2009
https://dl.acm.org/doi/10.5555/1884795.1884815
Borissov NWirström N(2008)Q-StrategyProceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE 2008. Part I on On the Move to Meaningful Internet Systems:10.1007/978-3-540-88871-0_52(744-761)Online publication date: 9-Nov-2008
https://dl.acm.org/doi/10.1007/978-3-540-88871-0_52
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents