Abstract
Achieving fair resource sharing is rapidly becoming an essential requirement in cluster computing systems. Although many fair scheduling algorithms have been proposed in recent decades, controlling resource sharing among jobs on servers remains a challenging problem that, if not handled well, may result in chaotic resource contention and service-level agreement violation of jobs. To address this problem, we propose a resource container–based job management approach for fair resource sharing. In our approach, we first design and implement a general container-based job management module, providing lightweight and fine-grained resource allocation and isolation for job execution. With this module, we propose a resource-aware management scheme to enable fair resource sharing in job scheduling and dispatching. We conduct experiments by implementing the proposed module and applying the scheme on TCluster, a self-developed cluster computing system of a worldwide top Internet corporation. Results show that our approach performs well in guaranteeing fair resource sharing with negligible overhead.
This work was partially supported by NSFC 61202417, 60903116 and 61003063.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
byte-unixbench, http://code.google.com/p/byte-unixbench/
Cgroup, http://www.kernel.org/doc/documentation/cgroups/cgroups.txt
Geekbench, http://www.primatelabs.ca/geekbench/
Linux container, http://lxc.sourceforge.net/
Linux kernel namespace, http://lxc.sourceforge.net/index.php/about/kernel-namespaces/
Vmware, http://www.vmware.com/
Hadoop fair scheduler (2010), http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html
Banga, G., Druschel, P., Mogul, J.C.: Resource containers: A new facility for resource management in server systems. In: OSDI, pp. 45–58 (1999)
Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: SOSP, pp. 164–177 (2003)
Chang, F., Ren, J., Viswanathan, R.: Optimal resource allocation in clouds. In: ICCC (2010)
Diaz, C.O., Guzek, M., Pecero, J.E., Danoy, G., Bouvry, P., Khan, S.U.: Energyaware fast scheduling heuristics in heterogeneous computing systems. In: HPCS (2011)
Gambosi, G., Postiglione, A., Talamo, M.: Algorithms for the relaxed online binpacking model. SIAM Journal on Computing 30(5), 1532–1551 (2000)
Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: Fair allocation of multiple resource types. In: NSDI (2011)
Grit, L., Irwin, D., Marupadi, V., Shivam, P., Yumerefendi, A., Chase, J., Albrecht, J.: Harnessing virtual machine resource control for job management. In: VTDC (2007)
He, Y., Elnikety, S., Sun, H.: Tians scheduling: Using partial processing in beste ort applications. In: ICDCS (2011)
Huang, Q., Huang, T.: An optimistic job scheduling strategy based on QoS for cloud computing. In: ICISS (2010)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: SOSP, pp. 261–276 (2009)
Lee, C.C., Lee, D.T.: A simple on-line bin-packing algorithm. J. ACM 32, 562–572 (1985)
Liu, H., Abraham, A., Hassanien, A.E.: Scheduling jobs on computational grids using a fuzzy particle swarm optimization algorithm. In: FGCS (2009)
Menage, P.B.: Adding generic process containers to the linux kernel. In: OLS (2007)
Pinel, F., Pecero, J.E., Bouvry, P., Khan, S.U.: A two-phase heuristic for the scheduling of independent tasks on computational grids. In: HPCS (2011)
Sanders, P., Sivadasan, N., Skutella, M.: Online scheduling with bounded migration. Journal of Mathematics of Operations Research 34(2) (2009)
Soltesz, S., Pötzl, H., Fiuczynski, M.E., Bavier, A., Peterson, L.: Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors. In: EuroSys, pp. 275–288 (2007)
Waldspurger, C.A.: Lottery and stride scheduling: Flexible proportional-share resource management (1995)
Wang, M., Meng, X., Zhang, L.: Consolidating virtual machines with dynamic bandwidth demand in data centers. In: INFOCOM (2011)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Job scheduling for multi-user mapreduce clusters. Technical Report UCB/EECS-2009-55 (2009), http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-55.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hong, J. et al. (2013). Container-Based Job Management for Fair Resource Sharing. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds) Supercomputing. ISC 2013. Lecture Notes in Computer Science, vol 7905. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38750-0_22
Download citation
DOI: https://doi.org/10.1007/978-3-642-38750-0_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38749-4
Online ISBN: 978-3-642-38750-0
eBook Packages: Computer ScienceComputer Science (R0)