research-article

Open access

How to design a library OS for practical containers?

Authors:

Ryo NakamuraAuthors Info & Claims

VEE 2021: Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 15 - 28

https://doi.org/10.1145/3453933.3454011

Published: 07 April 2021 Publication History

Abstract

Container engines with operating-system virtualization have been widely used and now offer extensions to replace core functionalities that are derived from the host kernel. Because such extensions with an alternate kernel, which is often implemented in a library operating system (libOS), can be designed to have free choice, developers are tempted to take a clean-slate approach, i.e., implement the kernels from scratch. However, this design decision makes it difficult to cover broad features of the original Linux kernel, and some application programs may not work on such kernels. Precise emulation of the huge codebase and rich feature set of the Linux kernel is not easily possible. In this paper, we have tried to improve the level of compatibility in a libOS by using the source code of the Linux kernel as the container kernel. We present µKontainer, an alternate container kernel based on a libOS by extending the existing open-source software, Linux Kernel Library, while preserving the lightweight property of conventional containers. We have studied the level of compatibility with the conformance tests of network protocol implementation of nine different libOSs, and µKontainer performs identically like the Linux kernel. The network-related benchmark shows mostly comparable results with a conventional container and a native Linux host; in the best case, the goodput of the short-sized packet is up to 84% faster than that of a native Linux host. This paper sheds light on the design space of the libOS when we introduced the extended container kernel.

References

[1]

Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. Firecracker: Lightweight virtualization for serverless applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 419-434, Santa Clara, CA, February 2020. USENIX Association.

[2]

Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O'Keefe, Mark L. Stillwell, David Goltzsche, Dave Eyers, Rüdiger Kapitza, Peter Pietzuch, and Christof Fetzer. SCONE: Secure Linux Containers with Intel SGX. In OSDI, volume 16, pages 689-703, 2016.

[3]

Arnd Bergmann. [PATCH 00 /16] remove eight obsolete architectures. http://lkml.iu.edu/hypermail/linux/kernel/ 1803.1/06845.html. (Accessed January 14th 2020 ).

[4]

F. Baker. Requirements for IP Version 4 Routers. RFC 1812 (Proposed Standard), June 1995. Updated by RFCs 2644, 6633.

Digital Library

[5]

Andrew Baumann, Dongyoon Lee, Pedro Fonseca, Lisa Glendenning, Jacob R. Lorch, Barry Bond, Reuben Olinsky, and Galen C. Hunt. Composing OS Extensions Safely and Eficiently with Bascule. In Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys ' 13, pages 239-252, New York, NY, USA, 2013. ACM.

Digital Library

[6]

Alfred Bratterud, Alf-Andre Walla, Hårek Haugerud, Paal E Engelstad, and Kyrre Begnum. IncludeOS: A minimal, resource eficient unikernel for cloud services. In Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on, pages 250-257. IEEE, 2015.

Digital Library

[7]

Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. BBR: Congestion-based Congestion Control. Commun. ACM, 60 ( 2 ): 58-66, January 2017.

Digital Library

[8]

HK Jerry Chu and Yuan Liu. User Space TCP-Getting LKL Ready for the Prime Time. Linux Netdev 1.2, October 2016.

[9]

Cloudius Systems. Seastar. http://www.seastar-project.org/. (Accessed Jan 26th 2017 ).

[10]

Justin Cormack. frankenlibc. https://github.com/justincormack/ frankenlibc. ( Accessed Jan 26th 2017 ).

[11]

J. Dike. User Mode Linux. In Proceedings of the 5th Anual Linux Showcase and Conference, ALS' 01, pages 3-14. USENIX Association, 2001.

[12]

Docker Inc. A modern, performant 9P library for Go. https://github. com/docker/go-p9p.

[13]

Docker Inc. Docker for Mac. https://www.docker.com/docker-mac. (Accessed Apr 18th 2018 ).

[14]

Adam Dunkels. Design and Implementation of the lwIP TCP/IP Stack. Swedish Institute of Computer Science, 2 : 77, 2001.

[15]

Rich Felker. musl libc. https://www.musl-libc.org/. (Accessed Jan 26th 2017 ).

[16]

Google Inc. gVisor: Container Runtime Sandbox. https://github.com/ google/gvisor. (Accessed May 8th 2018 ).

[17]

Google Inc. IPv4 and IPv6 userland network stack. https://github.com/ google/netstack. ( Accessed Sep 14th 2018 ).

[18]

Sangtae Ha, Injong Rhee, and Lisong Xu. CUBIC: a new TCP-friendly high-speed TCP variant. ACM SIGOPS operating systems review, 42 ( 5 ): 64-74, 2008.

[19]

Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, Bob Lantz, and Nick McKeown. Reproducible Network Experiments Using Container-based Emulation. In Proceedings of ACM CoNEXT 2012, pages 253-264. ACM, 2012.

[20]

Brian N. Handy, Rich Murphey, and Jim Mock. Linux Binary Compatibility. https://www.freebsd.org/doc/handbook/linuxemu-lbc-install. html. (Accessed Apr 18th 2018 ).

[21]

Mike Harsh. Run Bash on Ubuntu on Windows. https://blogs.windows. com/buildingapps/2016/03/30/run-bash-on-ubuntu-on-windows/. (Accessed Apr 18th 2018 ).

[22]

IBM. Nabla Containers. https://github.com/nabla-containers/runnc. ( Accessed July 3rd 2019 ).

[23]

Solarflare Communications Inc. OpenOnload. http://www.openonload. org/. (Accessed 14th January 2015 ).

[24]

Ixia. IxANVL. https://www.keysight.com/us/en/products/networksecurity/ixanvl.html. (Accessed March 8th 2021 ).

[25]

EunYoung Jeong, Shinae Wood, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 489-502, Seattle, WA, April 2014. USENIX Association.

Digital Library

[26]

Jessie Frazelle. Docker Containers on the Desktop. https://blog.jessfraz. com/post/docker-containers-on-the-desktop /, 2015. ( Accessed Aug 15th 2018 ).

[27]

Venkateswararao Jujjuri, Eric Van Hensbergen, Anthony Liguori, and Badari Pulavarty. VirtFS-a virtualization aware file system passthrough. In Ottawa Linux Symposium (OLS), pages 109-120, 2010.

[28]

M. Frans Kaashoek, Dawson R. Engler, Gregory R. Ganger, Hector M. Briceño, Russell Hunt, David Mazières, Thomas Pinckney, Robert Grimm, John Jannotti, and Kenneth Mackenzie. Application performance and flexibility on exokernel systems. In Proceedings of the sixteenth ACM symposium on Operating systems principles, SOSP '97, pages 52-65, New York, NY, USA, 1997. ACM.

Digital Library

[29]

Antti Kantee. The Design and Implementation of the Anykernel and Rump Kernels, 2nd Edition. http://book.rumpkernel.org, 2016.

[30]

Antti Kantee and Justin Cormack. Rump Kernels: No OS? No Problem! USENIX ;login :, 39 ( 5 ): 11-17, 2014.

[31]

Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav Har'El, Don Marti, and Vlad Zolotarov. OSv-Optimizing the Operating System for Virtual Machines. In 2014 USENIX Annual Technical Conference (USENIX ATC 14), pages 61-72, Philadelphia, PA, June 2014. USENIX Association.

[32]

Ricardo Koller and Dan Williams. Will Serverless End the Dominance of Linux in the Cloud ? In Proceedings of the 16th Workshop on Hot Topics in Operating Systems, HotOS '17, pages 169-173, New York, NY, USA, 2017. ACM.

Digital Library

[33]

Hsuan-Chi Kuo, Dan Williams, Ricardo Koller, and Sibin Mohan. A linux in unikernel clothing. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys ' 20, New York, NY, USA, 2020. Association for Computing Machinery.

Digital Library

[34]

Linux Programmer's Manual. namespaces-overview of Linux namespaces. http://man7.org/linux/man-pages /man7/namespaces.7.html. (Accessed Sep 14th 2018 ).

[35]

Craig Loewen. Announcing WSL 2. https://devblogs.microsoft.com/ commandline/announcing-wsl-2 /. ( Accessed July 24th 2019 ).

[36]

Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. Unikernels: Library operating systems for the cloud. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 461-472, New York, NY, USA, 2013. ACM.

[37]

Ilias Marinos, Robert N.M. Watson, and Mark Handley. Network Stack Specialization for Performance. In Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM '14, pages 175-186, New York, NY, USA, 2014. ACM.

Digital Library

[38]

Moby Project. LinuxKit. https://github.com/linuxkit/linuxkit. ( Accessed Sep 14th 2018 ).

[39]

Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. SOCK: Rapid Task Provisioning with Serverless-Optimized Containers. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC'18), 2018.

[40]

Pierre Olivier, Daniel Chiba, Stefan Lankes, Changwoo Min, and Binoy Ravindran. A Binary-compatible Unikernel. In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE 2019, pages 59-73, New York, NY, USA, 2019. ACM.

Digital Library

[41]

Open Container Initiative. OCI Runtime Specification. https://github. com/opencontainers/runtime-spec. (Accessed Sep 14th 2018 ).

[42]

Cristina Opriceana and Hajime Tazaki. Network stack personality in Android phone. In Linux netdev 2.2, THE Technical Conference on Linux Networking netdev 2.2, 2017.

[43]

Donald E. Porter, Silas Boyd-Wickizer, Jon Howell, Reuben Olinsky, and Galen C. Hunt. Rethinking the Library OS from the Top Down. In Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVI, pages 291-304, New York, NY, USA, 2011. ACM.

Digital Library

[44]

Octavian Purdila, Lucian Adrian Grijincu, and Nicolae Tapus. LKL: The Linux kernel library. In Roedunet International Conference (RoEduNet), 2010 9th, pages 328-333, 2010.

[45]

Quagga. Quagga Routing Suite. https://www.quagga.net/. (Accessed Sep 14th 2018 ).

[46]

Ali Raza, Parul Sohal, James Cadden, Jonathan Appavoo, Ulrich Drepper, Richard Jones, Orran Krieger, Renato Mancuso, and Larry Woodman. Unikernels: The Next Stage of Linux's Dominance. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS '19, pages 7-13, New York, NY, USA, 2019. ACM.

Digital Library

[47]

Rusty Russell. Virtio: Towards a De-facto Standard for Virtual I/O Devices. SIGOPS Oper. Syst. Rev., 42 ( 5 ): 95-103, July 2008.

Digital Library

[48]

Takaya Saeki, Yuichi Nishiwaki, Takahiro Shinagawa, and Shinichi Honiden. Bash on Ubuntu on macOS. In Proceedings of the 8th AsiaPacific Workshop on Systems, APSys '17, pages 17 : 1-17 : 8, New York, NY, USA, 2017. ACM.

Digital Library

[49]

Dan Schatzberg, James Cadden, Han Dong, Orran Krieger, and Jonathan Appavoo. EbbRT: A Framework for Building Per-Application Library Operating Systems. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 671-688, GA, November 2016. USENIX Association.

[50]

Zhiming Shen, Zhen Sun, Gur-Eyal Sela, Eugene Bagdasaryan, Christina Delimitrou, Robbert Van Renesse, and Hakim Weatherspoon. X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers. In Proceedings of the TwentyFourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '19, pages 121-135, New York, NY, USA, 2019. ACM.

Digital Library

[51]

Hajime Tazaki, Ryo Nakamura, and Yuji Sekiya. Library operating system with mainline Linux kernel. In Linux netdev 0.1, THE Technical Conference on Linux Networking netdev 0.1, 2015.

[52]

The OpenStack Foundation. Kata Containers. https://katacontainers. io/. (Accessed Aug 15th 2018 ).

[53]

The OpenStack Foundation. Kata Containers Architecture. https://github.com/kata-containers/documentation/blob/master/ design/architecture.md. (Accessed January 15th 2020 ).

[54]

Chia-Che Tsai. Library OS is the New Container. https://sched.co/FxXc. (Accessed November 21st 2019 ).

[55]

Chia-Che Tsai, Kumar Saurabh Arora, Nehal Bandi, Bhushan Jain, William Jannen, Jitin John, Harry A. Kalodner, Vrushali Kulkarni, Daniela Oliveira, and Donald E. Porter. Cooperation and Security Isolation of Library OSes for Multi-process Applications. In Proceedings of the Ninth European Conference on Computer Systems, EuroSys ' 14, pages 9 : 1-9 : 14, New York, NY, USA, 2014. ACM.

Digital Library

[56]

Chia-Che Tsai, Bhushan Jain, Nafees Ahmed Abdul, and Donald E. Porter. A Study of Modern Linux API Usage and Compatibility: What to Support when You'Re Supporting. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys ' 16, pages 16 : 1-16 : 16, New York, NY, USA, 2016. ACM.

Digital Library

[57]

Chia-Che Tsai, Donald E Porter, and Mona Vij. Graphene-SGX: A Practical Library OS for Unmodified Applications on SGX. In 2017 USENIX Annual Technical Conference (USENIX ATC '17), pages 645-658, 2017.

[58]

Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. Peeking Behind the Curtains of Serverless Platforms. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 133-146. USENIX Association, 2018.

[59]

Dan Williams and Ricardo Koller. Unikernel monitors: Extending minimalism outside of the box. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16 ). USENIX Association, 2016.

[60]

Dan Williams, Ricardo Koller, Martin Lucina, and Nikhil Prakash. Unikernels As Processes. In Proceedings of the ACM Symposium on Cloud Computing, SoCC '18, pages 199-211, New York, NY, USA, 2018. ACM.

[61]

Irene Zhang, Jing Liu, Amanda Austin, Michael Lowell Roberts, and Anirudh Badam. I'm not dead yet!: The role of the operating system in a kernel-bypass era. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS '19, pages 73-80, New York, NY, USA, 2019. ACM.

Digital Library

Cited By

JANG SSUH YTAK B(2024)Understanding File System Operations of a Secure Container Runtime Using System Call Tracing TechniqueIEICE Transactions on Information and Systems10.1587/transinf.2023EDL8039E107.D:2(229-233)Online publication date: 1-Feb-2024
https://doi.org/10.1587/transinf.2023EDL8039
Wada TYamada H(2024)Reboot-Based Recovery of Unikernels at the Component Level2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN58291.2024.00017(15-28)Online publication date: 24-Jun-2024
https://doi.org/10.1109/DSN58291.2024.00017
Shaikh SKumar SMishra D(2023)KalpaVriksh: Efficient and Cost-effective GUI Application Hosting using Singleton Snapshots2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid57682.2023.00026(180-190)Online publication date: May-2023
https://doi.org/10.1109/CCGrid57682.2023.00026
Show More Cited By

Index Terms

How to design a library OS for practical containers?
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
      2. Software infrastructure
        Virtual machines

Recommendations

X-Containers: Breaking Down Barriers to Improve Performance and Isolation of Cloud-Native Containers
ASPLOS '19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems

"Cloud-native" container platforms, such as Kubernetes, have become an integral part of production cloud environments. One of the principles in designing cloud-native applications is called Single Concern Principle, which suggests that each container ...
My VM is Lighter (and Safer) than your Container
SOSP '17: Proceedings of the 26th Symposium on Operating Systems Principles

Containers are in great demand because they are lightweight when compared to virtual machines. On the downside, containers offer weaker isolation than VMs, to the point where people run containers in virtual machines to achieve proper isolation. In this ...
A binary-compatible unikernel
VEE 2019: Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Unikernels are minimal single-purpose virtual machines. They are highly popular in the research domain due to the benefits they provide. A barrier to their widespread adoption is the difficulty/impossibility to port existing applications to current ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE 2021: Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

April 2021

200 pages

ISBN:9781450383943

DOI:10.1145/3453933

General Chair:
Ben L. Titzer
Google, Germany
,
Program Chairs:
Harry Xu
University of California at Los Angeles, USA
,
Irene Zhang
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EC Horizon 2020 Work Programme

Conference

VEE '21

Sponsor:

VEE '21: 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

April 16, 2021

Virtual, USA

Acceptance Rates

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
2,040
Total Downloads

Downloads (Last 12 months)331
Downloads (Last 6 weeks)38

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

JANG SSUH YTAK B(2024)Understanding File System Operations of a Secure Container Runtime Using System Call Tracing TechniqueIEICE Transactions on Information and Systems10.1587/transinf.2023EDL8039E107.D:2(229-233)Online publication date: 1-Feb-2024
https://doi.org/10.1587/transinf.2023EDL8039
Wada TYamada H(2024)Reboot-Based Recovery of Unikernels at the Component Level2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN58291.2024.00017(15-28)Online publication date: 24-Jun-2024
https://doi.org/10.1109/DSN58291.2024.00017
Shaikh SKumar SMishra D(2023)KalpaVriksh: Efficient and Cost-effective GUI Application Hosting using Singleton Snapshots2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid57682.2023.00026(180-190)Online publication date: May-2023
https://doi.org/10.1109/CCGrid57682.2023.00026
Tsakanikas VDagiuklas TIqbal MWang XMumtaz S(2023)An intelligent model for supporting edge migration for virtual function chains in next generation internet of thingsScientific Reports10.1038/s41598-023-27674-513:1Online publication date: 19-Jan-2023
https://doi.org/10.1038/s41598-023-27674-5
Wang KWu SLi SHuang ZFan HYu CJin H(2023)Precise control of page cache for containersFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-022-2455-018:2Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1007/s11704-022-2455-0
Thyagaturu AShantharama PNasrallah AReisslein M(2022)Operating Systems and Hypervisors for Network Functions: A Survey of Enabling Technologies and Research StudiesIEEE Access10.1109/ACCESS.2022.319491310(79825-79873)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3194913

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten