The Next 700 ML-Enabled Compiler Optimizations
Authors: 
S. VenkataKeerthy, Siddharth Jain, Umesh Kalvakuntla, Pranav Sai Gorantla, Rajiv Shailesh Chitale, Eugene Brevdo, 
Albert Cohen, Mircea Trofin, Ramakrishna UpadrastaAuthors Info & Claims
Pages 238 - 249
Abstract
There is a growing interest in enhancing compiler optimizations with ML models, yet interactions between compilers and ML frameworks remain challenging. Some optimizations require tightly coupled models and compiler internals, raising issues with modularity, performance and framework independence. Practical deployment and transparency for the end-user are also important concerns. We propose ML-Compiler-Bridge to enable ML model development within a traditional Python framework while making end-to-end integration with an optimizing compiler possible and efficient. We evaluate it on both research and production use cases, for training and inference, over several optimization problems, multiple compilers and its versions, and gym infrastructures.
References
[1]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/ Software available from tensorflow.org
[2]
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51, 4 (2018), 81.
[3]
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. code2vec: learning distributed representations of code. 3, POPL (2019), Article 40, jan, 29 pages. https://doi.org/10.1145/3290353
[4]
Saman P. Amarasinghe. 2020. Compiler 2.0: Using Machine Learning to Modernize Compiler Technology. In Proceedings of the 21st ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems, LCTES 2020, London, UK, June 16, 2020, Jingling Xue and Changhee Jung (Eds.). ACM, 1–2. https://doi.org/10.1145/3372799.3397167
[5]
Amir H. Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer Kulkarni, and John Cavazos. 2017. MiCOMP: Mitigating the Compiler Phase-Ordering Problem Using Optimization Sub-Sequences and Machine Learning. ACM Trans. Archit. Code Optim., 14, 3 (2017), Article 29, sep, 28 pages. issn:1544-3566 https://doi.org/10.1145/3124452
[6]
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: a learnable representation of code semantics. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Curran Associates Inc., Red Hook, NY, USA. 3589–3601.
[7]
Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer. In Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’08). Association for Computing Machinery, New York, NY, USA. 101–113. isbn:9781595938602 https://doi.org/10.1145/1375581.1375595
[8]
Michael Boulton. [n. d.]. TSVC_2. https://github.com/UoB-HPC/TSVC_2.git Accessed 2015-09-16
[9]
Alexander Brauckmann, Andrés Goens, and Jeronimo Castrillon. 2021. PolyGym: Polyhedral Optimizations as an Environment for Reinforcement Learning. In 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT). 17–29. https://doi.org/10.1109/PACT52795.2021.00009
[10]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arxiv:arXiv:1606.01540.
[11]
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation (OSDI’18). USENIX Association, USA. 579–594. isbn:9781931971478
[12]
Keith D. Cooper, Devika Subramanian, and Linda Torczon. 2002. Adaptive Optimizing Compilers for the 21st Century. J. Supercomput., 23, 1 (2002), 7–22. https://doi.org/10.1023/A:1015729001611
[13]
Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun, Torsten Hoefler, Michael F. P. O’Boyle, and Hugh Leather. 2021. ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Marina Meila and Tong Zhang (Eds.) (Proceedings of Machine Learning Research, Vol. 139). PMLR, 2244–2253. http://proceedings.mlr.press/v139/cummins21a.html
[14]
Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017. Synthesizing benchmarks for predictive modeling. In 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 86–99. https://doi.org/10.1109/CGO.2017.7863731
[15]
Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and Hugh Leather. 2022. CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 92–105. https://doi.org/10.1109/CGO53902.2022.9741258
[16]
Anderson Faustino da Silva, Bruno Conde Kind, José Wesley de Souza Magalhães, Jerônimo Nunes Rocha, Breno Campos Ferreira Guimarães, and Fernando Magno Quintão Pereira. 2021. AnghaBench: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction. In Proceedings of the 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO ’21). IEEE Press, 378–390. isbn:9781728186139 https://doi.org/10.1109/CGO51591.2021.9370322
[17]
Dibyendu Das, Shahid Asghar Ahmad, and Venkataramanan Kumar. 2020. Deep Learning-based Approximate Graph-Coloring Algorithm for Register Allocation. In 2020 IEEE/ACM 6th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC) and Workshop on Hierarchical Parallelism for Exascale Computing (HiPar). 23–32. https://doi.org/10.1109/LLVMHPCHiPar51896.2020.00008
[18]
[n. d.]. FlatBuffers. https://flatbuffers.dev/index.html [Online; accessed 29-Aug-2022]
[19]
Grigori Fursin, Yuriy Kashnikov, Abdul Wahid Memon, Zbigniew Chamski, Olivier Temam, Mircea Namolaru, Elad Yom-Tov, Bilha Mendelson, Ayal Zaks, Eric Courtois, François Bodin, Phil Barnard, Elton Ashton, Edwin V. Bonilla, John Thomson, Christopher K. I. Williams, and Michael F. P. O’Boyle. 2011. Milepost GCC: Machine Learning Enabled Self-tuning Compiler. Int. J. Parallel Program., 39, 3 (2011), 296–327. https://doi.org/10.1007/s10766-010-0161-2
[20]
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design patterns: elements of reusable object-oriented software. Pearson Deutschland GmbH.
[21]
Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica. 2020. NeuroVectorizer: end-to-end vectorization with deep reinforcement learning. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization (CGO 2020). Association for Computing Machinery, New York, NY, USA. 242–255. isbn:9781450370479 https://doi.org/10.1145/3368826.3377928
[22]
Hado van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double Q-Learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI’16). AAAI Press, 2094–2100.
[23]
Qijing Huang, Ameer Haj-Ali, William Moses, John Xiang, Ion Stoica, Krste Asanovic, and John Wawrzynek. 2019. AutoPhase: Compiler Phase-Ordering for HLS with Deep Reinforcement Learning. In 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE Computer Society, Los Alamitos, CA, USA. 308–308. https://doi.org/10.1109/FCCM.2019.00049
[24]
Shalini Jain, Yashas Andaluri, S. VenkataKeerthy, and Ramakrishna Upadrasta. 2022. POSET-RL: Phase ordering for Optimizing Size and Execution Time using Reinforcement Learning. In International IEEE Symposium on Performance Analysis of Systems and Software, ISPASS 2022, Singapore, May 22-24, 2022. IEEE, 121–131. https://doi.org/10.1109/ISPASS55109.2022.00012
[25]
Shalini Jain, S. VenkataKeerthy, Rohit Aggarwal, Tharun Kumar Dangeti, Dibyendu Das, and Ramakrishna Upadrasta. 2022. Reinforcement Learning assisted Loop Distribution for Locality and Vectorization. In 2022 IEEE/ACM Eighth Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC). 1–12. https://doi.org/10.1109/LLVM-HPC56686.2022.00006
[26]
Minsu Kim, Jeong-Keun Park, and Soo-Mook Moon. 2022. Solving PBQP-Based Register Allocation Using Deep Reinforcement Learning. In Proceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization (CGO ’22). IEEE Press, 230–241. isbn:9781665405843 https://doi.org/10.1109/CGO53902.2022.9741272
[27]
Sameer Kulkarni, John Cavazos, Christian Wimmer, and Douglas Simon. 2013. Automatic construction of inlining heuristics using machine learning. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 1–12. https://doi.org/10.1109/CGO.2013.6495004
[28]
P. J. Landin. 1966. The next 700 Programming Languages. Commun. ACM, 9, 3 (1966), mar, 157–166. issn:0001-0782 https://doi.org/10.1145/365230.365257
[29]
Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-Directed and Runtime Optimization (CGO ’04). IEEE Computer Society, USA. 75. isbn:0769521029
[30]
Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. 2021. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 2–14. https://doi.org/10.1109/CGO51591.2021.9370308
[31]
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated Graph Sequence Neural Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). arxiv:1511.05493
[32]
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for Distributed Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning, Jennifer Dy and Andreas Krause (Eds.) (Proceedings of Machine Learning Research, Vol. 80). PMLR, 3053–3062. https://proceedings.mlr.press/v80/liang18b.html
[33]
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Distributed Model Selection and Training. arxiv:1807.05118.
[34]
ONNX (Linux Foundation). 2017. ONNX: Open Neural Network Exchange. https://github.com/onnx/onnx [Online; accessed 11-Mar-2023]
[35]
LLVM-Org. [n. d.]. LLVM Test Suite. https://github.com/llvm/llvm-test-suite Accessed 2021-08-25
[36]
Charith Mendis, Cambridge Yang, Yewen Pu, Saman Amarasinghe, and Michael Carbin. 2019. Compiler auto-vectorization with imitation learning. Curran Associates Inc., Red Hook, NY, USA. https://doi.org/10.5555/3454287.3455597
[37]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, Maria Florina Balcan and Kilian Q. Weinberger (Eds.) (Proceedings of Machine Learning Research, Vol. 48). PMLR, New York, New York, USA. 1928–1937. https://proceedings.mlr.press/v48/mniha16.html
[38]
William S. Moses, Lorenzo Chelini, Ruizhe Zhao, and Oleksandr Zinenko. 2021. Polygeist: Raising C to Polyhedral MLIR. In 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT). 45–59. https://doi.org/10.1109/PACT52795.2021.00011
[39]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035. https://doi.org/10.5555/3454287.3455008
[40]
Louis-Noel Pouchet, C´edric Bastoul, and Uday Bondhugula. 2010. PoCC: the polyhedral compiler collection. https://web.cs.ucla.edu/ pouchet/software/pocc Accessed 2023-10-25
[41]
[n. d.]. Protocol Buffers. https://developers.google.com/protocol-buffers [Online; accessed 29-Aug-2022]
[42]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arxiv:1707.06347.
[43]
M. Stephenson and S. Amarasinghe. 2005. Predicting unroll factors using supervised classification. In International Symposium on Code Generation and Optimization. 123–134. https://doi.org/10.1109/CGO.2005.29
[44]
The IREE Team. 2023. https://github.com/openxla/iree Accessed 2023-11-13
[45]
The Triton Team. 2023. https://github.com/openai/triton Accessed 2023-11-13
[46]
Mircea Trofin, Yundi Qian, Eugene Brevdo, and David Li. 2021. RFC: MLGO Regalloc: learned eviction policy for regalloc. https://lists.llvm.org/pipermail/llvm-dev/2021-November/153639.html [Online; accessed 08-May-2022]
[47]
Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choromanski, and David Li. 2021. MLGO: a Machine Learning Guided Compiler Optimizations Framework. CoRR, abs/2101.04808 (2021), arXiv:2101.04808. arxiv:2101.04808
[48]
S. VenkataKeerthy, R Aggarwal, S Jain, M S Desarkar, R Upadrasta, and Y. N. Srikant. 2020. IR2Vec: LLVM IR Based Scalable Program Embeddings. ACM Trans. Archit. Code Optim., 17, 4 (2020), Article 32, Dec., 27 pages. issn:1544-3566 https://doi.org/10.1145/3418463
[49]
S. VenkataKeerthy and Siddharth Jain. 2024. ML-Compiler-Bridge: The Next 700 ML-Enabled Compiler Optimizations. https://doi.org/10.5281/zenodo.10574579
[50]
S. VenkataKeerthy, Siddharth Jain, Anilava Kundu, Rohit Aggarwal, Albert Cohen, and Ramakrishna Upadrasta. 2023. RL4ReAl: Reinforcement Learning for Register Allocation. In CC 2023. Association for Computing Machinery, New York, NY, USA. 133–144. isbn:9798400700880 https://doi.org/10.1145/3578360.3580273
[51]
Huanting Wang, Zhanyong Tang, Cheng Zhang, Jiaqi Zhao, Chris Cummins, Hugh Leather, and Zheng Wang. 2022. Automating Reinforcement Learning Architecture Design for Code Optimization. In Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction (CC 2022). Association for Computing Machinery, New York, NY, USA. 129–143. isbn:9781450391832 https://doi.org/10.1145/3497776.3517769
[52]
X Wang, H Zhao, and J Zhu. 1993. GRPC: A Communication Cooperation Mechanism in Distributed Systems. SIGOPS Oper. Syst. Rev., 27, 3 (1993), jul, 75–86. issn:0163-5980 https://doi.org/10.1145/155870.155881
[53]
Zheng Wang and Michael O’Boyle. 2018. Machine Learning in Compiler Optimization. Proc. IEEE, 106, 11 (2018), 1879–1901. https://doi.org/10.1109/JPROC.2018.2817118
Index Terms
- The Next 700 ML-Enabled Compiler Optimizations
Recommendations
Source-Level Compiler Optimizations for High-Level Synthesis
SEEDA-CECNSM '16: Proceedings of the SouthEast European Design Automation, Computer Engineering, Computer Networks and Social Media ConferenceWith high-level synthesis becoming the preferred method for hardware design, tools that operate on high-level programming languages and optimize hardware output are crucial for successful synthesis. In high-level synthesis, conventional programming ...
Verifying Compiler Optimisations: (Invited Paper)
Formal Methods and Software EngineeringAbstractCompilers are a vital tool but errors in a compiler can lead to errors in the myriad of programs it compiles. Our research focuses on verifying the optimisation phase because it is a common source of errors within compilers. In programming ...
The next 700 compiler correctness theorems (functional pearl)
Compiler correctness is an old problem, with results stretching back beyond the last half-century. Founding the field, John McCarthy and James Painter set out to build a "completely trustworthy compiler". And yet, until quite recently, even despite truly ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
February 2024
261 pages
ISBN:9798400705076
DOI:10.1145/3640537
- General Chair:
- Gabriel Rodríguez,
- Program Chairs:
- P. Sadayappan,
- Aravind Sukumaran-Rajam
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 February 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Conference
CC '24
Sponsor:
CC '24: 33rd ACM SIGPLAN International Conference on Compiler Construction
March 2 - 3, 2024
Edinburgh, United Kingdom
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 909Total Downloads
- Downloads (Last 12 months)909
- Downloads (Last 6 weeks)130
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in