research-article

Open access

U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning

Authors:

Rodrigo Girão SerrãoAuthors Info & Claims

ARRAY 2023: Proceedings of the 9th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming

Pages 22 - 35

https://doi.org/10.1145/3589246.3595371

Published: 06 June 2023 Publication History

Abstract

The APL notation would appear to be a clear match for convolutional neural networks, but traditional implementations of APL have lagged behind the performance of highly tuned, specialized frameworks designed to execute CNNs on the GPU. Moreover, most demonstrations of APL for neural networking have involved relatively small examples. We explore a more complex example in the U-net architecture and utilize a modern APL compiler with GPU support, Co-dfns, to compare the state of the art of APL against the current crop of specialized neural network frameworks in the form of PyTorch. We compare performance as well as the language design of APL for neural network programming and the clarity and transparency of the resulting code.

We found that the complete “from scratch” APL source was on par with the complexity of the PyTorch reference implementation, albeit more foreign, while being more concise and complete. We also found that when compiled with Co-dfns, despite the naïve implementation both of Co-dfns and our own code, performance on the GPU and the CPU were within a factor of 2.2 - 2.4 times that of the PyTorch implementation. We believe this suggests significant avenues of future exploration for machine learning language design, pedagogy, and implementation, both inside and outside of the APL community.

References

[1]

Manuel Alfonseca. 1990. Neural Networks in APL. SIGAPL APL Quote Quad, 20, 4 (1990), may, 2–6. issn:0163-6006 https://doi.org/10.1145/97811.97816

Digital Library

[2]

Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. 2017. Julia: A fresh approach to numerical computing. SIAM review, 59, 1 (2017), 65–98.

[3]

Albert Cardona, Stephan Saalfeld, Stephan Preibisch, Benjamin Schmid, Anchi Cheng, Jim Pulokas, Pavel Tomancak, and Volker Hartenstein. 2010. An Integrated Micro- and Macroarchitectural Analysis of the Drosophila Brain by Computer-Assisted Serial Section Electron Microscopy. PLOS Biology, 8, 10 (2010), 10, 1–17. https://doi.org/10.1371/journal.pbio.1000502

[4]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. 248–255.

[5]

Pedro Domingos. 2012. A few useful things to know about machine learning. Commun. ACM, 55, 10 (2012), 78–87.

Digital Library

[6]

Vincent Dumoulin and Francesco Visin. 2016. A guide to convolution arithmetic for deep learning. arXiv preprint arXiv:1603.07285.

[7]

Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. Nature, 585 (2020), 357–362. https://doi.org/10.1038/s41586-020-2649-2

[8]

Troels Henriksen, Niels GW Serup, Martin Elsman, Fritz Henglein, and Cosmin E Oancea. 2017. Futhark: purely functional GPU-programming with nested parallelism and in-place array updates. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation. 556–571.

Digital Library

[9]

Aaron Wen-yao Hsu. 2019. A data parallel compiler hosted on the gpu. Ph. D. Dissertation. Indiana University.

[10]

Roger Hui. 2017. Stencil Lives. https://www.dyalog.com/blog/2017/07/stencil-lives/

[11]

Roger Hui. 2020. Towards Improvements to Stencil. https://www.dyalog.com/blog/2020/06/towards-improvements-to-stencil/

[12]

Roger KW Hui and Morten J Kromberg. 2020. APL since 1978. Proceedings of the ACM on Programming Languages, 4, HOPL (2020), 1–108.

Digital Library

[13]

Kenneth E Iverson. 1962. A programming language. In Proceedings of the May 1-3, 1962, spring joint computer conference. 345–351.

[14]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093.

[15]

JSoftware. 2014. Vocabulary/semidot. https://code.jsoftware.com/wiki/Vocabulary/semidot

[16]

D Knuth. 1993. Computer literacy bookshops interview. Also available as http://yurichev. com/mirrors/C/knuth-interview1993. txt.

[17]

Donald E Knuth. 2007. Computer programming as an art. In ACM Turing award lectures. 1974.

[18]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25 (2012).

[19]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, 86, 11 (1998), 2278–2324.

[20]

Bernard Legrand. 2009. Mastering Dyalog APL (1 ed.). Dyalog Ltd.

[21]

Inc Math Works. 1992. MATLAB reference guide. Math Works, Incorporated.

[22]

Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. 2018. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378.

[23]

Keiron O’Shea and Ryan Nash. 2015. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458.

[24]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

Digital Library

[25]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. 234–241.

[26]

Dominik Scherer, Andreas Müller, and Sven Behnke. 2010. Evaluation of pooling operations in convolutional architectures for object recognition. In International conference on artificial neural networks. 92–101.

[27]

Rodrigo Girão Serrão. 2022. Transposed convolution. https://mathspp.com/blog/til/033##transposed-convolution

[28]

Artjoms Šinkarovs, Robert Bernecky, and Sven-Bodo Scholz. 2019. Convolutional neural networks in APL. In Proceedings of the 6th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming. 69–79.

Digital Library

[29]

Duc Minh Tran, Troels Henriksen, and Martin Elsman. 2019. Compositional Deep Learning in Futhark. In Proceedings of the 8th ACM SIGPLAN International Workshop on Functional High-Performance and Numerical Computing (FHPNC 2019). Association for Computing Machinery, New York, NY, USA. 47–59. isbn:9781450368148 https://doi.org/10.1145/3331553.3342617

Digital Library

[30]

Guido Van Rossum and Fred L. Drake. 2009. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA. isbn:1441412697

[31]

Artjoms Šinkarovs, Hans-Nikolai Vieß mann, and Sven-Bodo Scholz. 2021. Array Languages Make Neural Networks Fast. In Proceedings of the 7th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming (ARRAY 2021). Association for Computing Machinery, New York, NY, USA. 39–50. isbn:9781450384667 https://doi.org/10.1145/3460944.3464312

Digital Library

Cited By

Oladapo BOlawumi MOmigbodun F(2024)Machine Learning for Optimising Renewable Energy and Grid EfficiencyAtmosphere10.3390/atmos1510125015:10(1250)Online publication date: 19-Oct-2024
https://doi.org/10.3390/atmos15101250
Núñcz-Corrales SFrenkel MAbreu B(2023)quAPL: Modeling Quantum Computation in an Array Programming Language2023 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE57702.2023.00114(1001-1012)Online publication date: 17-Sep-2023
https://doi.org/10.1109/QCE57702.2023.00114

Index Terms

U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning
1. Software and its engineering
  1. Software notations and tools
    1. Compilers
      1. Runtime environments
    2. General programming languages
      1. Language types
        Parallel programming languages

Recommendations

Convolutional neural networks in APL
ARRAY 2019: Proceedings of the 6th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming

This paper shows how a Convolutional Neural Network (CNN) can be implemented in APL. Its first-class array support ideally fits that domain, and the operations of APL facilitate rapid and concise creation of generically reusable building blocks. For our ...
Compiling a Subset of APL Into a Typed Intermediate Language
ARRAY'14: Proceedings of ACM SIGPLAN International Workshop on Libraries, Languages, and Compilers for Array Programming

We present a compiler and a typed intermediate language for a subset of APL. The intermediate language treats all numeric data as multi-dimensional arrays and the type system makes explicit the ranks of arrays. Primitive operators are polymorphic in ...
APL since 1978

The Evolution of APL, the HOPL I paper by Falkoff and Iverson on APL, recounted the fundamental design principles which shaped the implementation of the APL language in 1966, and the early uses and other influences which shaped its first decade of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ARRAY 2023: Proceedings of the 9th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming

June 2023

74 pages

ISBN:9798400701696

DOI:10.1145/3589246

General Chairs:
Troels Henriksen
University of Copenhagen, Denmark
,
Artjoms Sinkarovs
Heriot-Watt University, UK

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ARRAY '23

Sponsor:

SIGPLAN

ARRAY '23: 9th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming

June 18, 2023

FL, Orlando, USA

Acceptance Rates

Overall Acceptance Rate 17 of 25 submissions, 68%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
2,082
Total Downloads

Downloads (Last 12 months)543
Downloads (Last 6 weeks)64

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oladapo BOlawumi MOmigbodun F(2024)Machine Learning for Optimising Renewable Energy and Grid EfficiencyAtmosphere10.3390/atmos1510125015:10(1250)Online publication date: 19-Oct-2024
https://doi.org/10.3390/atmos15101250
Núñcz-Corrales SFrenkel MAbreu B(2023)quAPL: Modeling Quantum Computation in an Array Programming Language2023 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE57702.2023.00114(1001-1012)Online publication date: 17-Sep-2023
https://doi.org/10.1109/QCE57702.2023.00114

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents