research-article

HHVM Performance Optimization for Large Scale Web Services

Authors:

Abhishek Gupta,

Maximilian Balandat,

Arun Kejariwal,

Benjamin LeeAuthors Info & Claims

ICPE '23: Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering

Pages 137 - 148

https://doi.org/10.1145/3578244.3583720

Published: 15 April 2023 Publication History

Abstract

HHVM is commonly developed for large online web services, yet there remains much room for optimizing HHVM performance. This paper discusses challenges and techniques in optimizing HHVM performance for Meta's web service. We begin by evaluating the effectiveness of semantic request routing, a request routing method aimed at enhancing code cache performance in HHVM, and examine its implications for optimizing HHVM performance. Second, we characterize HHVM performance for a large-scale datacenter and identify the challenges brought by uncontrollable confounding factors. Finally, we present the performance management framework for autotuning HHVM performance at scale.

References

[1]

Approximations for Mean and Variance of a Ratio. https://www.stat.cmu.edu/~hseltman/files/ratio.pdf, 2021. Online; accessed 12 Jan 2022.

[2]

Ax:Adaptive Experimentation Platform. https://ax.dev/, 2021. Online; accessed 29 Nov 2021.

[3]

Engineering at Meta: Redesigning the HHVM JIT compiler for better performance. engineering.fb.com/redesigning-the-hhvm-jit-compiler-\for-better- performance/, 2021. Online; accessed 29 Nov 2021.

[4]

github:baidu-hhvm-301. https://github.com/baidu-lamp/baidu-hhvm-301, 2021. Online; accessed 2 Dec 2021.

[5]

Go Faster! HHVM. https://hhvm.com/blog/4061/go-faster, 2021. Online; accessed 01 Jan 2022.

[6]

Hacklang at Slack: A Better PHP. https://slack.engineering/hacklang-at-slack-a-better-php/, 2021. Online; accessed 2 Dec 2021.

[7]

HHVM Configuration: INI Settings. https://docs.hhvm.com/hhvm/configuration/INI-settings, 2021. Online; accessed 2 Dec 2021.

[8]

Hill climbing-Wikipedia. https://en.wikipedia.org/wiki/Hill_climbing, 2021. Online; accessed 12 Jan 2022.

[9]

Introducing mcrouter: A memcached protocol router for scaling memcached deployments. https://engineering.fb.com/introducing-mcrouter-a-memcached-protocol-\router-for-scaling-memcached-deployments/, 2021. Online; accessed 29 Nov 2021.

[10]

Introducing Proxygen, Facebook's C HTTP framework. https://engineering.fb.com/2014/11/05/production-engineering/introducing-proxygen-facebook-s-c-http-framework//, 2021. Online; accessed 1 Dec 2021.

[11]

Partition around limits. https://docs.microsoft.com/en-us/azure/architecture/guide/design-principles/partition, 2021. Online; accessed 29 Nov 2021.

[12]

Partition (database)--Wikipedia. https://en.wikipedia.org/wiki/Partition_ (database), 2021. Online; accessed 27 Dec 2021.

[13]

Partition service fabric reliable services. https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-concepts-partitioning, 2021. Online; accessed 27 Dec 2021.

[14]

Randomized controlled trial. https://en.wikipedia.org/wiki/Randomized_controlled_trial, 2021. Online; accessed 29 Nov 2021.

[15]

Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17), pages 469--482, 2017.

[16]

Grant Ayers, Jung Ho Ahn, Christos Kozyrakis, and Parthasarathy Ranganathan. Memory hierarchy for web search. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 643--656. IEEE, 2018.

[17]

Sikha Bagui and Loi Tang Nguyen. Database sharding: to provide fault tolerance and scalability of big data on the cloud. International Journal of Cloud Applications and Computing (IJCAC), 5(2):36--52, 2015.

Digital Library

[18]

Nathan Bronson, Zach Amsden, George Cabrera, Prasad Chakka, Peter Dimov, Hui Ding, Jack Ferris, Anthony Giardullo, Sachin Kulkarni, Harry Li, et al. {TAO}: Facebook's distributed data store for the social graph. In 2013 {USENIX} Annual Technical Conference ({USENIX} {ATC} 13), pages 49--60, 2013.

[19]

Kevin K Chang, Abhijith Kashyap, Hasan Hassan, Saugata Ghose, Kevin Hsieh, Donghyuk Lee, Tianshi Li, Gennady Pekhimenko, Samira Khan, and Onur Mutlu. Understanding latency variation in modern dram chips: Experimental characterization, analysis, and optimization. In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science, pages 323--336, 2016.

Digital Library

[20]

Christina Delimitrou and Christos Kozyrakis. Paragon: Qos-aware scheduling for heterogeneous datacenters. ACM SIGPLAN Notices, 48(4):77--88, 2013.

Digital Library

[21]

Dibakar Gope, David J Schlais, and Mikko H Lipasti. Architectural support for server-side php processing. In 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pages 507--520. IEEE, 2017.

Digital Library

[22]

Chang-Hong Hsu, Yunqi Zhang, Michael A Laurenzano, David Meisner, Thomas Wenisch, Jason Mars, Lingjia Tang, and Ronald G Dreslinski. Adrenaline: Pinpointing and reining in tail queries with quick voltage boosting. In 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), pages 271--282. IEEE, 2015.

[23]

Ana Klimovic, Heiner Litz, and Christos Kozyrakis. Selecta: Heterogeneous cloud storage configuration for data analytics. In 2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18), pages 759--773, 2018.

[24]

Carl Knospe. Pid control. IEEE Control Systems Magazine, 26(1):30--31, 2006.

[25]

Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1168--1176, 2013.

Digital Library

[26]

Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. Con- trolled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18(1):140--181, 2009.

[27]

Ron Kohavi, Diane Tang, and Ya Xu. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press, 2020.

[28]

Ron Kohavi, Diane Tang, Ya Xu, Lars G Hemkens, and John PA Ioannidis. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials, 21(1):1--9, 2020.

[29]

Vasileios Kontorinis, Liuyi Eric Zhang, Baris Aksanli, Jack Sampson, Houman Homayoun, Eddie Pettis, Dean M Tullsen, and Tajana Simunic Rosing. Managing distributed ups energy for effective power capping in data centers. In 2012 39th Annual International Symposium on Computer Architecture (ISCA), pages 488--499. IEEE, 2012.

[30]

Rakesh Kumar, Boris Grot, and Vijay Nagarajan. Blasting through the front-end bottleneck with shotgun. ACM SIGPLAN Notices, 53(2):30--42, 2018.

Digital Library

[31]

Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. Constrained bayesian optimization with noisy experiments. Bayesian Analysis, 14(2):495--519, 2019.

[32]

Zhao Lucis Li, Chieh-Jan Mike Liang, Wenjia He, Lianjie Zhu, Wenjun Dai, Jin Jiang, and Guangzhong Sun. Metis: Robustly tuning tail latencies of cloud systems. In 2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18), pages 981--992, 2018.

[33]

David Lo, Liqun Cheng, Rama Govindaraju, Parthasarathy Ranganathan, and Christos Kozyrakis. Heracles: Improving resource efficiency at scale. In Proceedings of the 42nd Annual International Symposium on Computer Architecture, pages 450--462, 2015.

Digital Library

[34]

Ashraf Mahgoub, Alexander Michaelson Medoff, Rakesh Kumar, Subrata Mitra, Ana Klimovic, Somali Chaterji, and Saurabh Bagchi. {OPTIMUSCLOUD}: Heterogeneous configuration optimization for distributed databases in the cloud. In 2020 {USENIX} Annual Technical Conference ({USENIX} {ATC} 20), pages 189--203, 2020.

[35]

Ashraf Mahgoub, Paul Wood, Alexander Medoff, Subrata Mitra, Folker Meyer, Somali Chaterji, and Saurabh Bagchi. {SOPHIA}: Online reconfiguration of clustered nosql databases for time-varying workloads. In 2019 {USENIX} Annual Technical Conference ({USENIX} {ATC} 19), pages 223--240, 2019.

[36]

Hosein Mohammadi Makrani and Houman Homayoun. Mena: A memory navigator for modern hardware in a scale-out environment. In 2017 IEEE International Symposium on Workload Characterization (IISWC), pages 2--11. IEEE, 2017.

[37]

Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. Taming performance variability. In 13th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 18), pages 409--425, 2018.

[38]

Michael Mitzenmacher. The power of two choices in randomized load balancing. IEEE Transactions on Parallel and Distributed Systems, 12(10):1094--1104, 2001.

Digital Library

[39]

Usama Naseer, Luca Niccolini, Udip Pant, Alan Frindell, Ranjeeth Dasineni, and Theophilus A Benson. Zero downtime release: Disruption-free load balancing of a multi-billion user website. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pages 529--541, 2020.

Digital Library

[40]

Shamkant Navathe, Stefano Ceri, Gio Wiederhold, and Jinglie Dou. Vertical partitioning algorithms for database design. ACM Transactions on Database Systems (TODS), 9(4):680--710, 1984.

[41]

Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, et al. Scaling memcache at facebook. In 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13), pages 385--398, 2013.

[42]

Rajiv Nishtala, Vinicius Petrucci, Paul Carpenter, and Magnus Sjalander. Twig: Multi-agent task management for colocated latency-critical cloud services. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 167--179. IEEE, 2020.

[43]

Guilherme Ottoni. Hhvm jit: A profile-guided, region-based compiler for php and hack. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 151--165, 2018.

Digital Library

[44]

Guilherme Ottoni and Bin Liu. Hhvm jump-start: Boosting both warmup and steady-state performance at scale. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pages 340--350. IEEE, 2021.

Digital Library

[45]

Varun Sakalkar, Vasileios Kontorinis, David Landhuis, Shaohong Li, Darren De Ronde, Thomas Blooming, Anand Ramesh, James Kennedy, Christopher Malone, Jimmy Clidaras, et al. Data center power oversubscription with a medium voltage power plane and priority-aware capping. In Proceedings of the Twenty- Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 497--511, 2020.

Digital Library

[46]

Akshitha Sriraman and Abhishek Dhanotia. Accelerometer: Understanding acceleration opportunities for data center overheads at hyperscale. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 733--750, 2020.

Digital Library

[47]

Akshitha Sriraman, Abhishek Dhanotia, and Thomas F Wenisch. Softsku: Optimizing server architectures for microservice diversity@ scale. In Proceedings of the 46th International Symposium on Computer Architecture, pages 513--526, 2019.

Digital Library

[48]

Shu Wang, Chi Li, Henry Hoffmann, Shan Lu, William Sentosa, and Achmad Imam Kistijantoro. Understanding and auto-adjusting performance-sensitive configurations. ACM SIGPLAN Notices, 53(2):154--168, 2018.

Digital Library

[49]

Qiang Wu, Qingyuan Deng, Lakshmi Ganesh, Chang-Hong Hsu, Yun Jin, Sanjeev Kumar, Bin Li, Justin Meza, and Yee Jiun Song. Dynamo: Facebook's data center-wide power management system. ACM SIGARCH Computer Architecture News, 44(3):469--480, 2016.

Digital Library

[50]

Neeraja J Yadwadkar, Bharath Hariharan, Joseph E Gonzalez, Burton Smith, and Randy H Katz. Selecting the best vm across multiple public clouds: A data-driven performance modeling approach. In Proceedings of the 2017 Symposium on Cloud Computing, pages 452--465, 2017.

Digital Library

[51]

Owen Yamauchi. Hack and HHVM: programming productivity without breaking things. " O'Reilly Media, Inc.", 2015.

[52]

Laiping Zhao, Yanan Yang, Kaixuan Zhang, Xiaobo Zhou, Tie Qiu, Keqiu Li, and Yungang Bao. Rhythm: component-distinguishable workload deployment in datacenters. In Proceedings of the Fifteenth European Conference on Computer Systems, pages 1--17, 2020.

[53]

Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, and Yingchun Yang. Bestconfig: tapping the performance potential of systems via automatic configuration tuning. In Proceedings of the 2017 Symposium on Cloud Computing, pages 338--350, 2017.

Digital Library

Cited By

Frenot LPereira FRodríguez GSadayappan PSukumaran-Rajam A(2024)Reducing the Overhead of Exact Profiling by Reusing Affine VariablesProceedings of the 33rd ACM SIGPLAN International Conference on Compiler Construction10.1145/3640537.3641569(150-161)Online publication date: 17-Feb-2024
https://dl.acm.org/doi/10.1145/3640537.3641569

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '23: Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering

April 2023

244 pages

ISBN:9798400700682

DOI:10.1145/3578244

General Chairs:
Marco Vieira
University of Coimbra, Portugal
,
Valeria Cardellini
University of Rome Tor Vergata, Italy
,
Program Chairs:
Antinisca Di Marco
University of L'Aquila, Italy
,
Petr Tuma
Charles University, Czechia

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICPE '23

Sponsor:

ICPE '23: ACM/SPEC International Conference on Performance Engineering

April 15 - 19, 2023

Coimbra, Portugal

Acceptance Rates

ICPE '23 Paper Acceptance Rate 15 of 46 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
115
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Frenot LPereira FRodríguez GSadayappan PSukumaran-Rajam A(2024)Reducing the Overhead of Exact Profiling by Reusing Affine VariablesProceedings of the 33rd ACM SIGPLAN International Conference on Compiler Construction10.1145/3640537.3641569(150-161)Online publication date: 17-Feb-2024
https://dl.acm.org/doi/10.1145/3640537.3641569

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents