Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3579990.3580020acmconferencesArticle/Chapter ViewAbstractPublication PagescgoConference Proceedingsconference-collections
research-article
Open access

Looplets: A Language for Structured Coiteration

Published: 22 February 2023 Publication History

Abstract

Real world arrays often contain underlying structure, such as sparsity, runs of repeated values, or symmetry. Specializing for structure yields significant speedups. But automatically generating efficient code for structured data is challenging, especially when arrays with different structure interact. We show how to abstract over array structures so that the compiler can generate code to coiterate over any combination of them. Our technique enables new array formats (such as 1DVBL for irregular clustered sparsity), new iteration strategies (such as galloping intersections), and new operations over structured data (such as concatenation or convolution).

References

[1]
2022. TIFF, Revision 6.0. https://www.loc.gov/preservation/digital/formats/fdd/fdd000022.shtml
[2]
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283. https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
[3]
Willow Ahrens, Fredrik Kjolstad, and Saman Amarasinghe. 2022. Autoscheduling for sparse tensor algebra with an asymptotic cost model. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI 2022). Association for Computing Machinery, New York, NY, USA. 269–285. isbn:978-1-4503-9265-5 https://doi.org/10.1145/3519939.3523442
[4]
Willow Ahrens, Daniel Donenfeld, Fredrik Kjolstad, and Saman Amarasinghe. 2023. Looplets: A Language For Structured Coiteration (The Artifact). https://doi.org/10.5281/zenodo.7499790 Language: eng
[5]
Riyadh Baghdadi, Jessica Ray, Malek Ben Romdhane, Emanuele Del Sozzo, Abdurrahman Akkas, Yunming Zhang, Patricia Suriana, Shoaib Kamil, and Saman Amarasinghe. 2019. Tiramisu: a polyhedral compiler for expressing fast and portable code. In Proceedings of the 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO 2019). IEEE Press, Washington, DC, USA. 193–205. isbn:978-1-72811-436-1
[6]
Jérémy Barbay, Alejandro López-Ortiz, Tyler Lu, and Alejandro Salinger. 2010. An experimental investigation of set intersection algorithms for text searching. ACM Journal of Experimental Algorithmics, 14 (2010), Jan., 7:3.7–7:3.24. issn:1084-6654 https://doi.org/10.1145/1498698.1564507
[7]
Aart J. C. Bik. 1996. Compiler Support for Sparse Matrix Computations. Ph. D. Dissertation. LIACS, Leiden University. https://theses.liacs.nl/1315
[8]
Aart J. C. Bik, Penporn Koanantakool, Tatiana Shpeisman, Nicolas Vasilache, Bixia Zheng, and Fredrik Kjolstad. 2022. Compiler Support for Sparse Tensor Computations in MLIR. arXiv:2202.04305 [cs], Feb., arxiv:2202.04305 arXiv: 2202.04305
[9]
Aart J. C. Bik and Harry A. G. Wijshoff. 1993. Compilation techniques for sparse matrix computations. In Proceedings of the 7th international conference on Supercomputing (ICS ’93). Association for Computing Machinery, New York, NY, USA. 416–424. isbn:978-0-89791-600-4 https://doi.org/10.1145/165939.166023
[10]
Aart J. C. Bik and Harry A. G. Wijshoff. 1994. On automatic data structure selection and code generation for sparse computations. In Languages and Compilers for Parallel Computing, Utpal Banerjee, David Gelernter, Alex Nicolau, and David Padua (Eds.) (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg. 57–75. isbn:978-3-540-48308-3 https://doi.org/10.1007/3-540-57659-2_4
[11]
L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, and others. 2002. An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Software, 28, 2 (2002), 135–151. issn:0098-3500 https://doi.org/10.1145/567806.567807
[12]
Stephen Chou, Fredrik Kjolstad, and Saman Amarasinghe. 2018. Format abstraction for sparse tensor algebra compilers. Proceedings of the ACM on Programming Languages, 2, OOPSLA (2018), Oct., 123:1–123:30. https://doi.org/10.1145/3276493
[13]
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. EMNIST: an extension of MNIST to handwritten letters. https://doi.org/10.48550/arXiv.1702.05373 arXiv:1702.05373 [cs]
[14]
Timothy A. Davis. 2019. Algorithm 1000: SuiteSparse:GraphBLAS: Graph Algorithms in the Language of Sparse Linear Algebra. ACM Trans. Math. Software, 45, 4 (2019), Dec., 44:1–44:25. issn:0098-3500 https://doi.org/10.1145/3322125
[15]
Daniel Donenfeld, Stephen Chou, and Saman Amarasinghe. 2022. Unified Compilation for Lossless Compression and Sparse Computing. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 205–216. https://doi.org/10.1109/CGO53902.2022.9741282
[16]
I. S. Duff, Roger G. Grimes, and John G. Lewis. 1989. Sparse matrix test problems. ACM Trans. Math. Software, 15, 1 (1989), March, 1–14. issn:0098-3500 https://doi.org/10.1145/62038.62043
[17]
Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM Transactions on Graphics, 31, 4 (2012), July, 44:1–44:10. issn:0730-0301 https://doi.org/10.1145/2185520.2185540
[18]
Pratik Fegade, Tianqi Chen, Phillip B. Gibbons, and Todd C. Mowry. 2021. The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding. arXiv:2110.10221 [cs], Oct., arxiv:2110.10221 arXiv: 2110.10221
[19]
Shashi Gowda, Yingbo Ma, Alessandro Cheli, Maja Gwóźzdź, Viral B. Shah, Alan Edelman, and Christopher Rackauckas. 2022. High-performance symbolic-numerics via multiple dispatch. ACM Communications in Computer Algebra, 55, 3 (2022), Jan., 92–96. issn:1932-2240 https://doi.org/10.1145/3511528.3511535
[20]
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. Nature, 585, 7825 (2020), Sept., 357–362. issn:1476-4687 https://doi.org/10.1038/s41586-020-2649-2 Number: 7825 Publisher: Nature Publishing Group
[21]
Rawn Henry, Olivia Hsu, Rohan Yadav, Stephen Chou, Kunle Olukotun, Saman Amarasinghe, and Fredrik Kjolstad. 2021. Compilation of sparse array programming models. Proceedings of the ACM on Programming Languages, 5, OOPSLA (2021), Oct., 128:1–128:29. https://doi.org/10.1145/3485505
[22]
Paul N. Hilfinger and Philip Colella. 1989. 5. FIDIL: A Language for Scientific Programming. In Symbolic Computation, Robert Grossman (Ed.). Society for Industrial and Applied Mathematics, 97–138. isbn:978-0-89871-239-1 978-1-61197-103-3 https://doi.org/10.1137/1.9781611971033.ch5
[23]
Olivia Hsu, Maxwell Strange, Jaeyeon Won, Ritvik Sharma, Kunle Olukotun, Joel Emer, Mark Horowitz, and Fredrik Kjolstad. 2022. The Sparse Abstract Machine. https://doi.org/10.48550/arXiv.2208.14610 arXiv:2208.14610 [cs]
[24]
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. 2019. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics, 38, 6 (2019), Nov., 201:1–201:16. issn:0730-0301 https://doi.org/10.1145/3355089.3356506
[25]
Kenneth E. Iverson. 1962. A programming language. John Wiley & Sons, Inc., USA. isbn:978-0-471-43014-8
[26]
Jeremy Kepner, Peter Aaltonen, David Bader, Aydin Buluç, Franz Franchetti, John Gilbert, Dylan Hutchison, Manoj Kumar, Andrew Lumsdaine, Henning Meyerhenke, Scott McMillan, Carl Yang, John D. Owens, Marcin Zalewski, Timothy Mattson, and Jose Moreira. 2016. Mathematical foundations of the GraphBLAS. In 2016 IEEE High Performance Extreme Computing Conference (HPEC). 1–9. https://doi.org/10.1109/HPEC.2016.7761646
[27]
Oleg Kiselyov, Aggelos Biboudis, Nick Palladinos, and Yannis Smaragdakis. 2017. Stream fusion, to completeness. ACM SIGPLAN Notices, 52, 1 (2017), Jan., 285–299. issn:0362-1340 https://doi.org/10.1145/3093333.3009880
[28]
Fredrik Kjolstad, Willow Ahrens, Shoaib Kamil, and Saman Amarasinghe. 2019. Tensor Algebra Compilation with Workspaces. In 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 180–192. https://doi.org/10.1109/CGO.2019.8661185
[29]
Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman Amarasinghe. 2017. The Tensor Algebra Compiler. Proc. ACM Program. Lang., 1, OOPSLA (2017), Oct., 77:1–77:29. issn:2475-1421 https://doi.org/10.1145/3133901
[30]
Fredrik Berg Kjølstad. 2020. Sparse tensor algebra compilation. Massachusetts Institute of Technology. https://dspace.mit.edu/handle/1721.1/128314 Accepted: 2020-11-03T20:30:04Z ISBN: 9781201259824
[31]
Vladimir Kotlyar. 1999. Relational Algebraic Techniques for the Synthesis of Sparse Matrix Programs. Cornell. 00000
[32]
Vladimir Kotlyar, Keshav Pingali, and Paul Stodghill. 1997. Compiling parallel sparse code for user-defined data structures. Cornell. 00000
[33]
Vladimir Kotlyar, Keshav Pingali, and Paul Stodghill. 1997. A relational approach to the compilation of sparse matrix programs. In Euro-Par’97 Parallel Processing, Christian Lengauer, Martin Griebl, and Sergei Gorlatch (Eds.) (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg. 318–327. isbn:978-3-540-69549-3 https://doi.org/10.1007/BFb0002751
[34]
Scott Kovach and Fredrik Kjolstad. 2022. Correct Compilation of Semiring Contractions. arxiv:2207.13291 arXiv:2207.13291 [cs]
[35]
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350, 6266 (2015), Dec., 1332–1338. https://doi.org/10.1126/science.aab3050 Publisher: American Association for the Advancement of Science
[36]
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, 86, 11 (1998), Nov., 2278–2324. issn:1558-2256 https://doi.org/10.1109/5.726791 Conference Name: Proceedings of the IEEE
[37]
Jure Leskovec and Andrej Krevl. 2014. SNAP Datasets: Stanford Large Network Dataset Collection. http://snap.stanford.edu/data
[38]
Amanda Liu, Gilbert Louis Bernstein, Adam Chlipala, and Jonathan Ragan-Kelley. 2022. Verified tensor-program optimization via high-level scheduling rewrites. Proceedings of the ACM on Programming Languages, 6, POPL (2022), Jan., 55:1–55:28. https://doi.org/10.1145/3498717
[39]
Geoffrey Mainland, Roman Leshchinskiy, and Simon Peyton Jones. 2017. Exploiting vector instructions with generalized stream fusion. Commun. ACM, 60, 5 (2017), April, 83–91. issn:0001-0782 https://doi.org/10.1145/3060597
[40]
Tim Mattson, David Bader, Jon Berry, Aydin Buluc, Jack Dongarra, Christos Faloutsos, John Feo, John Gilbert, Joseph Gonzalez, Bruce Hendrickson, Jeremy Kepner, Charles Leiserson, Andrew Lumsdaine, David Padua, Stephen Poole, Steve Reinhardt, Mike Stonebraker, Steve Wallach, and Andrew Yoo. 2013. Standards for graph algorithm primitives. In 2013 IEEE High Performance Extreme Computing Conference (HPEC). 1–2. https://doi.org/10.1109/HPEC.2013.6670338
[41]
Abhijeet Mohapatra and Michael Genesereth. 2012. Incrementally maintaining run-length encoded attributes in column stores. In Proceedings of the 16th International Database Engineering & Applications Sysmposium (IDEAS ’12). Association for Computing Machinery, New York, NY, USA. 146–154. isbn:978-1-4503-1234-9 https://doi.org/10.1145/2351476.2351493
[42]
Thomas Neumann. 2011. Efficiently compiling efficient query plans for modern hardware. Proceedings of the VLDB Endowment, 4, 9 (2011), June, 539–550. issn:2150-8097 https://doi.org/10.14778/2002938.2002940
[43]
Hung Q. Ngo, Ely Porat, Christopher Ré, and Atri Rudra. 2018. Worst-case Optimal Join Algorithms. J. ACM, 65, 3 (2018), March, 16:1–16:40. issn:0004-5411 https://doi.org/10.1145/3180143
[44]
William Pugh and Tatiana Shpeisman. 1999. SIPR: A New Framework for Generating Efficient Code for Sparse Matrix Computations. In Languages and Compilers for Parallel Computing, Siddhartha Chatterjee, Jan F. Prins, Larry Carter, Jeanne Ferrante, Zhiyuan Li, David Sehr, and Pen-Chung Yew (Eds.) (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg. 213–229. isbn:978-3-540-48319-9 https://doi.org/10.1007/3-540-48319-5_14
[45]
Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, and Frédo Durand. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. ACM Transactions on Graphics, 31, 4 (2012), July, 32:1–32:12. issn:0730-0301 https://doi.org/10.1145/2185520.2185528
[46]
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13). Association for Computing Machinery, New York, NY, USA. 519–530. isbn:978-1-4503-2014-6 https://doi.org/10.1145/2491956.2462176
[47]
Ryan Senanayake, Changwan Hong, Ziheng Wang, Amalee Wilson, Stephen Chou, Shoaib Kamil, Saman Amarasinghe, and Fredrik Kjolstad. 2020. A sparse iteration space transformation framework for sparse tensor algebra. Proceedings of the ACM on Programming Languages, 4, OOPSLA (2020), Nov., 158:1–158:30. https://doi.org/10.1145/3428226
[48]
Jessica Shi, Stephen Chou, Fredrik Kjolstad, and Saman Amarasinghe. 2021. An Attempt to Generate Code for Symmetric Tensor Computations. https://doi.org/10.48550/arXiv.2110.00186 arXiv:2110.00186 [cs]
[49]
Edgar Solomonik and Torsten Hoefler. 2015. Sparse Tensor Algebra as a Parallel Programming Model. arXiv:1512.00066 [cs], Nov., arxiv:1512.00066 arXiv: 1512.00066
[50]
Edgar Solomonik, Devin Matthews, Jeff R. Hammond, John F. Stanton, and James Demmel. 2014. A massively parallel tensor contraction framework for coupled-cluster computations. J. Parallel and Distrib. Comput., 74, 12 (2014), Dec., 3176–3190. issn:07437315 https://doi.org/10.1016/j.jpdc.2014.06.002
[51]
Michelle Mills Strout, Mary Hall, and Catherine Olschanowsky. 2018. The Sparse Polyhedral Framework: Composing Compiler-Generated Inspector-Executor Code. Proc. IEEE, 106, 11 (2018), Nov., 1921–1934. issn:1558-2256 https://doi.org/10.1109/JPROC.2018.2857721 Conference Name: Proceedings of the IEEE
[52]
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. 2020. Efficient Processing of Deep Neural Networks. Synthesis Lectures on Computer Architecture, 15, 2 (2020), June, 1–341. issn:1935-3235 https://doi.org/10.2200/S01004ED1V01Y202004CAC050 Publisher: Morgan & Claypool Publishers
[53]
William Thies, Steven Hall, and Saman Amarasinghe. 2009. Manipulating lossless video in the compressed domain. In Proceedings of the 17th ACM international conference on Multimedia (MM ’09). Association for Computing Machinery, New York, NY, USA. 331–340. isbn:978-1-60558-608-3 https://doi.org/10.1145/1631272.1631319
[54]
Ruiqin Tian, Luanzheng Guo, Jiajia Li, Bin Ren, and Gokcen Kestor. 2021. A High-Performance Sparse Tensor Algebra Compiler in Multi-Level IR. arXiv:2102.05187 [cs], Feb., arxiv:2102.05187 arXiv: 2102.05187
[55]
Todd Veldhuizen. 2014. Triejoin: A Simple, Worst-Case Optimal Join Algorithm. https://doi.org/10.5441/002/ICDT.2014.13 Type: dataset
[56]
Richard Vuduc, James W. Demmel, and Katherine A. Yelick. 2005. OSKI: A library of automatically tuned sparse matrix kernels. Journal of Physics: Conference Series, 16 (2005), Jan., 521–530. issn:1742-6596 https://doi.org/10.1088/1742-6596/16/1/071 Publisher: IOP Publishing
[57]
Rohan Yadav, Alex Aiken, and Fredrik Kjolstad. 2022. DISTAL: the distributed tensor algebra compiler. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI 2022). Association for Computing Machinery, New York, NY, USA. 286–300. isbn:978-1-4503-9265-5 https://doi.org/10.1145/3519939.3523437
[58]
Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. 2022. SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning. https://doi.org/10.48550/arXiv.2207.04606 arXiv:2207.04606 [cs]
[59]
Tuowen Zhao, Tobi Popoola, Mary Hall, Catherine Olschanowsky, and Michelle Mills Strout. 2022. Polyhedral Specification and Code Generation of Sparse Tensor Contraction with Co-Iteration. https://doi.org/10.48550/arXiv.2208.11858 arXiv:2208.11858 [cs]
[60]
Ningxin Zheng, Bin Lin, Quanlu Zhang, Lingxiao Ma, Yuqing Yang, Fan Yang, Yang Wang, Mao Yang, and Lidong Zhou. 2022. SparTA: Deep Learning Model Sparsity via Tensor with Sparsity Attribute. 21.

Cited By

View all
  • (2024)Compilation of Shape Operators on Sparse ArraysProceedings of the ACM on Programming Languages10.1145/36897528:OOPSLA2(1162-1188)Online publication date: 8-Oct-2024
  • (2024)Compiler Support for Sparse Tensor ConvolutionsProceedings of the ACM on Programming Languages10.1145/36897218:OOPSLA2(275-303)Online publication date: 8-Oct-2024
  • (2024)Mechanised Hypersafety Proofs about Structured DataProceedings of the ACM on Programming Languages10.1145/36564038:PLDI(647-670)Online publication date: 20-Jun-2024
  • Show More Cited By

Index Terms

  1. Looplets: A Language for Structured Coiteration
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CGO '23: Proceedings of the 21st ACM/IEEE International Symposium on Code Generation and Optimization
    February 2023
    262 pages
    ISBN:9798400701016
    DOI:10.1145/3579990
    This work is licensed under a Creative Commons Attribution 4.0 International License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 February 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    Author Tags

    1. Array
    2. Coiteration
    3. Compressed
    4. Sparse
    5. Tensor

    Qualifiers

    • Research-article

    Conference

    CGO '23

    Acceptance Rates

    Overall Acceptance Rate 312 of 1,061 submissions, 29%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)655
    • Downloads (Last 6 weeks)62
    Reflects downloads up to 12 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Compilation of Shape Operators on Sparse ArraysProceedings of the ACM on Programming Languages10.1145/36897528:OOPSLA2(1162-1188)Online publication date: 8-Oct-2024
    • (2024)Compiler Support for Sparse Tensor ConvolutionsProceedings of the ACM on Programming Languages10.1145/36897218:OOPSLA2(275-303)Online publication date: 8-Oct-2024
    • (2024)Mechanised Hypersafety Proofs about Structured DataProceedings of the ACM on Programming Languages10.1145/36564038:PLDI(647-670)Online publication date: 20-Jun-2024
    • (2024)GraphBLAS.jl v0.1: An Update on GraphBLAS in Julia2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW63119.2024.00104(516-519)Online publication date: 27-May-2024
    • (2023)Runtime Composition of Iterations for Fusing Loop-carried Sparse DependenceProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3607097(1-15)Online publication date: 12-Nov-2023

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media