research-article

Safe, Fast Sharing of memcached as a Protected Library

Authors:

Chris Kjellqvist,

Mohammad Hedayati,

Michael L. ScottAuthors Info & Claims

ICPP '20: Proceedings of the 49th International Conference on Parallel Processing

Article No.: 6, Pages 1 - 8

https://doi.org/10.1145/3404397.3404443

Published: 17 August 2020 Publication History

Abstract

Memcached is a widely used key-value store. It is structured as a multithreaded user-level server, accessed over socket connections by a potentially distributed collection of clients. Because socket communication is so much more expensive than a single operation on a K-V store, much of the client library is devoted to batching of requests. Batching is not always feasible, however, and the cost of communication seems particularly unfortunate when—as is often the case—clients are co-located on a single machine with the server, and have access to the same physical memory.

Fortunately, recent work on protected libraries has shown that it is possible, on current Intel processors, to amplify access rights quickly when calling into a specially configured user-level library. Library instances in separate processes can then share data safely, even in the face of independent process failures. We have used protected libraries to implement a new version of memcached in which client threads execute the code of the server themselves, without the need to send messages. Compared to the original, our new version is both significantly simpler, containing 24% less code, and dramatically faster, with a 11–56 × reduction in latency and a roughly 2 × increase in throughput.

References

[1]

Adam Belay, Andrea Bittau, Ali Mashtizadeh, David Terei, David Mazières, and Christos Kozyrakis. 2012. Dune: Safe User-level Access to Privileged CPU Features. In 10th USENIX Symp. on Operating Systems Design and Implementation (OSDI). Hollywood, CA, 335–348.

[2]

Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christos Kozyrakis, and Edouard Bugnion. 2014. IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In 11th USENIX Symp. on Operating Systems Design and Implementation (OSDI). Broomfield, CO, 49–65.

[3]

Wentao Cai, Haosen Wen, H. Alan Beadle, Chris Kjellqvist, Mohammad Hedayati, and Michael L. Scott. 2020. Understanding and Optimizing Persistent Memory Allocation. In 19th Intl. Symp. on Memory Management. Earlier version published as arXiv:2003.06718 [cs.DC] and TR 1008, Computer Science Dept, Univ. of Rochester, March 2020. Extended abstract presented at PPoPP 2020.

[4]

Dhruva R. Chakrabarti, Hans-J. Boehm, and Kumud Bhandari. 2014. Atlas: Leveraging Locks for Non-volatile Memory Consistency. In ACM Intl. Conf. on Object Oriented Programming Systems Languages & Applications (OOPSLA). Portland, OR, 433–452. https://doi.org/10.1145/2660193.2660224

Digital Library

[5]

Guoyang Chen, Lei Zhang, Richa Budhiraja, Xipeng Shen, and Youfeng Wu. 2017. Efficient Support of Position Independence on Non-volatile Memory. In 50th IEEE/ACM Intl. Symp. on Microarchitecture (MICRO). Cambridge, MA, 191–203. https://doi.org/10.1145/3123939.3124543

Digital Library

[6]

Joel Coburn, Adrian M. Caulfield, Ameen Akel, Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala, and Steven Swanson. 2011. NV-Heaps: Making Persistent Objects Fast and Safe with Next-generation, Non-volatile Memories. In 16th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Newport Beach, CA, 105–118. https://doi.org/10.1145/1950365.1950380

Digital Library

[7]

Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In 1st ACM Symp. on Cloud Computing (SoCC). Indianapolis, IN, 143–154. https://doi.org/10.1145/1807128.1807152

Digital Library

[8]

Mingkai Dong, Heng Bu, Jifei Yi, Benchao Dong, and Haibo Chen. 2019. Performance and Protection in the ZoFS User-Space NVM File System. In 27th ACM Symp. on Operating Systems Principles (SOSP). Huntsville, Ontario, Canada, 478–493. https://doi.org/10.1145/3341301.3359637

Digital Library

[9]

Kevin Elphinstone, Amirreza Zarrabi, Kent Mcleod, and Gernot Heiser. 2017. A Performance Evaluation of Rump Kernels as a Multi-Server OS Building Block on SeL4. In 8th Asia-Pacific Workshop on Systems(APSys ’17). Mumbai, India, Article 11, 8 pages. https://doi.org/10.1145/3124680.3124727

Digital Library

[10]

D. R. Engler, M. F. Kaashoek, and J. O’Toole. 1995. Exokernel: An Operating System Architecture for Application-Level Resource Management. SIGOPS Oper. Syst. Rev. 29, 5 (Dec. 1995), 251–266. https://doi.org/10.1145/224057.224076

Digital Library

[11]

Jim Gray. 1981. The Transaction Concept: Virtues and Limitations (Invited Paper). In 7th Intl. Conf. Very Large Data Bases (VLDB). Cannes, France, 144–154.

[12]

Mohammad Hedayati, Spyridoula Gravani, Ethan Johnson, John Criswell, Michael L. Scott, Kai Shen, and Mike Marty. 2019. Hodor: Intra-Process Isolation for High-Throughput Data Plane Libraries. In 2019 USENIX Annual Technical Conf. (ATC). Renton, WA, 489–504.

[13]

Intel Corp.2018. Intel DPDK: Data Plane Development Kit. http://www.dpdk.org.

[14]

Intel Corp.2018. Intel SPDK: Storage Performance Development Kit. http://www.spdk.io.

[15]

Intel Corp.2018. PCI-SIG SR-IOV Primer: An Introduction to SR-IOV Technology. http://www.intel.sg/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf.

[16]

Joseph Izraelevitz, Terence Kelly, and Aasheesh Kolli. 2016. Failure-Atomic Persistent Memory Updates via JUSTDO Logging. In 21st Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Atlanta, GA, 427–442. https://doi.org/10.1145/2872362.2872410

Digital Library

[17]

Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R. Dulloor, Jishen Zhao, and Steven Swanson. 2019. Basic Performance Measurements of the Intel Optane DC Persistent Memory Module. CoRR abs/1903.05714(2019). http://arxiv.org/abs/1903.05714

[18]

Eun Young Jeong, Shinae Woo, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems. In 11th USENIX Conf. on Networked Systems Design and Implementation (NSDI). Seattle, WA, 489–502.

[19]

Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, and et al.2009. SeL4: Formal Verification of an OS Kernel. In 22nd ACM Symp. on Operating Systems Principles (SOSP). Big Sky, MT, 207–220. https://doi.org/10.1145/1629575.1629596

Digital Library

[20]

James Larus and Galen Hunt. 2010. The Singularity System. Commun. ACM 53, 8 (Aug. 2010), 72–79. https://doi.org/10.1145/1787234.1787253

Digital Library

[21]

libMemcached.org. 2011. libMemcached. libmemcached.org/libMemcached.html.

[22]

Qingrui Liu, Joseph Izraelevitz, Se Kwon Lee, Michael L. Scott, Sam H. Noh, and Changhee Jung. 2018. iDO: Compiler-Directed Failure Atomicity for Nonvolatile Memory. In 51st IEEE/ACM Intl. Symp. on Microarchitecture (MICRO). Fukuoka, Japan, 258–270. https://doi.org/10.1109/MICRO.2018.00029

Digital Library

[23]

I. Marinos, R. N. M. Watson, and M. Handley. 2014. Network Stack Specialization for Performance. In ACM SIGCOMM Conf.Chicago, IL, 175–186. https://doi.org/10.1145/2619239.2626311

Digital Library

[24]

Michael Marty, Marc de Kruijf, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Michael Dalton, Nandita Dukkipati, William C. Evans, Steve Gribble, and et al.2019. Snap: A Microkernel Approach to Host Networking. In 27th ACM Symp. on Operating Systems Principles (SOSP). Huntsville, ON, Canada, 399–413. https://doi.org/10.1145/3341301.3359657

Digital Library

[25]

Zeyu Mi, Dingji Li, Zihan Yang, Xinran Wang, and Haibo Chen. 2019. SkyBridge: Fast and Secure Inter-process Communication for Microkernels. In 14th ACM SIGOPS European Conf. on Computer Systems (EuroSys). Dresden, Germany, Article 9, 15 pages. https://doi.org/10.1145/3302424.3303946

Digital Library

[26]

Faisal Nawab, Joseph Izraelevitz, Terrence Kelly, Charles B. Morrey III, Dhruva R. Chakrabarti, and Michael L. Scott. 2017. Dalí: A Periodically Persistent Hash Map. In 31st Intl. Symp. on Distributed Computing (DISC). Vienna, Austria, Article 37, 16 pages.

[27]

Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe. 2014. Arrakis: The Operating System is the Control Plane. In 11th USENIX Symp. on Operating Systems Design and Implementation (OSDI). Broomfield, CO, 1–16.

[28]

Donald E. Porter, Silas Boyd-Wickizer, Jon Howell, Reuben Olinsky, and Galen C. Hunt. 2011. Rethinking the Library OS from the Top Down. SIGPLAN Not. 46, 3 (March 2011), 291–304. https://doi.org/10.1145/1961296.1950399

Digital Library

[29]

George Prekas, Marios Kogias, and Edouard Bugnion. 2017. ZygOS: Achieving Low Tail Latency for Microsecond-scale Networked Tasks. In 26th Symp. on Operating Systems Principles (SOSP). Shanghai, China, 325–341. https://doi.org/10.1145/3132747.3132780

Digital Library

[30]

Dan Schatzberg, James Cadden, Han Dong, Orran Krieger, and Jonathan Appavoo. 2016. EbbRT: A Framework for Building per-Application Library Operating Systems. In 12th USENIX Conf. on Operating Systems Design and Implementation (OSDI). Savannah, GA, 671–688.

[31]

Andrew S. Tanenbaum and Linus Torvalds. 1999. The Tanenbaum-Torvalds Debate. In Open Sources: Voices from the Open Source Revolution, Chris DiBona, Sam Ockman, and Mark Stone (Eds.). O’Reilly & Associates. Appendix A. https://www.oreilly.com/openbook/opensources/book/appa.html.

[32]

Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, and Samuel Madden. 2013. Speedy Transactions in Multicore In-memory Databases. In 24th ACM Symp. on Operating Systems Principles (SOSP). Farmington, PA, 18–32. https://doi.org/10.1145/2517349.2522713

Digital Library

[33]

Úlfar Erlingsson, Martín Abadi, Michael Vrable, Mihai Budiu, and George C. Necula. 2006. XFI: Software Guards for System Address Spaces. In 7th USENIX Symp. on Operating Systems Design and Implementation (OSDI). Seattle, WA, 75–88.

[34]

Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, and Deepak Garg. 2019. ERIM: Secure, Efficient in-Process Isolation with Protection Keys (MPK). In 28th USENIX Security Symp. (SEC). Santa Clara, CA, 1221–1238.

[35]

Haris Volos, Andres Jaan Tack, and Michael M. Swift. 2011. Mnemosyne: Lightweight Persistent Memory. In 16th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Newport Beach, CA, 91–104. https://doi.org/10.1145/1950365.1950379

Digital Library

[36]

Robert Wahbe, Steven Lucco, Thomas E. Anderson, and Susan L. Graham. 1993. Efficient Software-based Fault Isolation. In 14th ACM Symp. on Operating Systems Principles (SOSP). Asheville, NC, 203–216. https://doi.org/10.1145/168619.168635

Digital Library

[37]

Jean Yang and Chris Hawblitzel. 2010. Safe to the Last Instruction: Automated Verification of a Type-Safe Operating System. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). Toronto, ON, Canada, 99–110. https://doi.org/10.1145/1806596.1806610

Digital Library

Cited By

Lefeuvre HBădoiu VJung ATeodorescu SRauch SHuici FRaiciu COlivier PFalsafi BFerdman MLu SWenisch T(2022)FlexOS: towards flexible OS isolationProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3503222.3507759(467-482)Online publication date: 28-Feb-2022
https://dl.acm.org/doi/10.1145/3503222.3507759
Gao BTee HSanaee AJun SJevdjic D(2022)OS-level Implications of Using DRAM Caches in Memory Disaggregation2022 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)10.1109/ISPASS55109.2022.00020(153-155)Online publication date: May-2022
https://doi.org/10.1109/ISPASS55109.2022.00020
Moti NSalkhordeh RBrinkmann A(2022)Protected Functions: User Space Privileged Function CallsArchitecture of Computing Systems10.1007/978-3-031-21867-5_8(117-131)Online publication date: 13-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-21867-5_8
Show More Cited By

Recommendations

Search lookaside buffer: efficient caching for index data structures
SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing

With the ever increasing DRAM capacity in commodity computers, applications tend to store large amount of data in main memory for fast access. Accordingly, efficient traversal of index structures to locate requested data becomes crucial to their ...
An Empirical Analysis on Memcached's Replacement Policies
MEMSYS '23: Proceedings of the International Symposium on Memory Systems

The performance of large-scale web services heavily relies on the hit ratio of the key-value caches. One core component of a high-performance key-value cache is the replacement policy. A right replacement policy can help the caching system achieve a ...
Performance and protection in the ZoFS user-space NVM file system
SOSP '19: Proceedings of the 27th ACM Symposium on Operating Systems Principles

Non-volatile memory (NVM) can be directly accessed in user space without going through the kernel. This encourages several recent studies on building user-space NVM file systems. However, for the sake of file system protection, none of the existing file ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP '20: Proceedings of the 49th International Conference on Parallel Processing

August 2020

844 pages

ISBN:9781450388160

DOI:10.1145/3404397

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPP '20

ICPP '20: 49th International Conference on Parallel Processing

August 17 - 20, 2020

AB, Edmonton, Canada

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
205
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lefeuvre HBădoiu VJung ATeodorescu SRauch SHuici FRaiciu COlivier PFalsafi BFerdman MLu SWenisch T(2022)FlexOS: towards flexible OS isolationProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3503222.3507759(467-482)Online publication date: 28-Feb-2022
https://dl.acm.org/doi/10.1145/3503222.3507759
Gao BTee HSanaee AJun SJevdjic D(2022)OS-level Implications of Using DRAM Caches in Memory Disaggregation2022 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)10.1109/ISPASS55109.2022.00020(153-155)Online publication date: May-2022
https://doi.org/10.1109/ISPASS55109.2022.00020
Moti NSalkhordeh RBrinkmann A(2022)Protected Functions: User Space Privileged Function CallsArchitecture of Computing Systems10.1007/978-3-031-21867-5_8(117-131)Online publication date: 13-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-21867-5_8
Wen HCai WDu MJenkins LValpey BScott M(2021)A Fast, General System for Buffered Persistent Data StructuresProceedings of the 50th International Conference on Parallel Processing10.1145/3472456.3472458(1-11)Online publication date: 9-Aug-2021
https://dl.acm.org/doi/10.1145/3472456.3472458
Gravani SHedayati MCriswell JScott MBilge LDumitras T(2021)Fast Intra-kernel Isolation and Security with IskiOSProceedings of the 24th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3471621.3471849(119-134)Online publication date: 6-Oct-2021
https://dl.acm.org/doi/10.1145/3471621.3471849

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents