Article

Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays

Authors:

Stephen J. Carey,

Walterio W. Mayol-CuevasAuthors Info & Claims

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX

Pages 488 - 503

https://doi.org/10.1007/978-3-030-58526-6_29

Published: 23 August 2020 Publication History

Abstract

We present a novel method of CNN inference for pixel processor array (PPA) vision sensors, designed to take advantage of their massive parallelism and analog compute capabilities. PPA sensors consist of an array of processing elements (PEs), with each PE capable of light capture, data storage and computation, allowing various computer vision processes to be executed directly upon the sensor device. The key idea behind our approach is storing network weights “in-pixel” within the PEs of the PPA sensor itself to allow various computations, such as multiple different image convolutions, to be carried out in parallel. Our approach can perform convolutional layers, max pooling, ReLu, and a final fully connected layer entirely upon the PPA sensor, while leaving no untapped computational resources. This is in contrast to previous works that only use a sensor-level processing to sequentially compute image convolutions, and must transfer data to an external digital processor to complete the computation. We demonstrate our approach on the SCAMP-5 vision system, performing inference in a MNIST digit classification network at over 3000 frames per second and over 93% classification accuracy. This is the first work demonstrating CNN inference conducted entirely upon a PPA vision sensor, requiring no external processing.

References

[1]

Aimar A et al. NullHop: a flexible convolutional neural network accelerator based on sparse representations of feature maps IEEE Trans. Neural Netw. Learn. Syst. 2018 99 1-13

[2]

Bose, L., Chen, J., Carey, S.J., Dudek, P., Mayol-Cuevas, W.: Visual odometry for pixel processor arrays. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4604–4612 (2017)

[3]

Bose, L., Chen, J., Carey, S.J., Dudek, P., Mayol-Cuevas, W.: A camera that CNNs: towards embedded neural networks on pixel processor arrays. arXiv preprint arXiv:1909.05647 (2019). (ICCV 2019 Accepted Submission)

[4]

Carey, S.J., Lopich, A., Barr, D.R., Wang, B., Dudek, P.: A 100,000 fps vision sensor with embedded 535GOPS/W 256 256 SIMD processor array. In: 2013 Symposium on VLSI Circuits, pp. C182–C183. IEEE (2013)

[5]

Carey, S.J., Zarándy, Á., Dudek, P.: Characterization of processing errors on analog fully-programmable cellular sensor-processor arrays. In: 2014 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1580–1583. IEEE (2014)

[6]

Chen, J., Carey, S.J., Dudek, P.: Scamp5d vision system and development framework. In: Proceedings of the 12th International Conference on Distributed Smart Cameras, p. 23. ACM (2018)

[7]

Chen, Y.H., Emer, J., Sze, V.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. In: ACM SIGARCH Computer Architecture News, vol. 44, pp. 367–379. IEEE Press (2016)

[8]

Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, pp. 3123–3131 (2015)

[9]

Du, Z., et al.: ShiDianNao: shifting vision processing closer to the sensor. In: ACM SIGARCH Computer Architecture News, vol. 43, pp. 92–104. ACM (2015)

[10]

Guillard, B.: Optimising convolutional neural networks for super fast inference on focal-plane sensor-processor arrays. Master’s thesis, Imperial College London (2019)

[11]

Hubara I, Courbariaux M, Soudry D, El-Yaniv R, and Bengio Y Quantized neural networks: training neural networks with low precision weights and activations J. Mach. Learn. Res. 2017 18 1 6869-6898

Digital Library

[12]

Komuro T, Kagami S, and Ishikawa M A dynamically reconfigurable SIMD processor for a vision chip IEEE J. Solid-State Circuits 2004 39 1 265-268

[13]

Liang S, Yin S, Liu S, Luk W, and Wei S FP-BNN: binarized neural network on FPGA Neurocomputing 2018 275 1072-1086

Digital Library

[14]

Rodriguez-Vazquez A, Fernández-Berni J, Leñero-Bardallo JA, Vornicu I, and Carmona-Galán R CMOS vision sensors: embedding computer vision at imaging front-ends IEEE Circuits Syst. Mag. 2018 18 2 90-107

[15]

Sim, J., Park, J.S., Kim, M., Bae, D., Choi, Y., Kim, L.S.: A 1.42 TOPS/W deep convolutional neural network recognition processor for intelligent IoE systems. In: 2016 IEEE International Solid-State Circuits Conference (ISSCC), pp. 264–265. IEEE (2016)

[16]

Wong, M.: Analog vision - neural network inference acceleration using analog SIMD computation in the focal plane. M.Sc. dissertation, Imperial College London (2018)

[17]

Zhao, R., et al.: Accelerating binarized convolutional neural networks with software-programmable FPGAs, pp. 15–24 (02 2017)

[18]

Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: towards lossless CNNs with low-precision weights. arXiv preprint arXiv:1702.03044 (2017)

[19]

Zhu, C., Han, S., Mao, H., Dally, W.J.: Trained ternary quantization. arXiv preprint arXiv:1612.01064 (2016)

Cited By

Kaiser MDatta GSarkar SKundu SYin ZGarg MJacob ABeerel PJaiswal AThapliyal HDeMara RPartin-Vaisband IKatkoori S(2023)Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)Proceedings of the Great Lakes Symposium on VLSI 202310.1145/3583781.3590235(613-618)Online publication date: 5-Jun-2023
https://dl.acm.org/doi/10.1145/3583781.3590235
Datta GYin ZJacob AJaiswal ABeerel P(2022)Towards Energy-Efficient Hyperspectral Image Processing Inside Camera PixelsComputer Vision – ECCV 2022 Workshops10.1007/978-3-031-25075-0_22(303-316)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-25075-0_22

Index Terms

Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays
1. Computer systems organization

Index terms have been assigned to the content through auto-classification.

Recommendations

Polymorphic Processor Arrays

Polymorphic processor arrays (PPAs), two-dimensional mesh-connected arrays ofprocessors in which each processor is equipped with a switch able to interconnect itsfour NEWS ports, are discussed. The main features of PPA are that it models a ...
Convolutional Neural Networks based classifications of soil images
Abstract
The utilization of Artificial Intelligence (AI) and Machine Learning(ML) in the image processing domain is useful to detect and recognize the types of soil. The main aim of the present work is to process the soil images and classify them ...
A High-Performance Reconfigurable Accelerator for Convolutional Neural Networks
ICMSSP '18: Proceedings of the 3rd International Conference on Multimedia Systems and Signal Processing

In this paper, we propose a new high-performance accelerator that supports a variety of convolutional neural networks (CNNs) such as GoogLeNet, ResNet and AlexNet. The proposed accelerator mainly includes 24 parallel PEs (processing engines) for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX

Aug 2020

843 pages

ISBN:978-3-030-58525-9

DOI:10.1007/978-3-030-58526-6

Editors:
Andrea Vedaldi
University of Oxford, Oxford, UK
,
Horst Bischof
Graz University of Technology, Graz, Austria
,
Thomas Brox
University of Freiburg, Freiburg im Breisgau, Germany
,
Jan-Michael Frahm
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

© Springer Nature Switzerland AG 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 August 2020

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kaiser MDatta GSarkar SKundu SYin ZGarg MJacob ABeerel PJaiswal AThapliyal HDeMara RPartin-Vaisband IKatkoori S(2023)Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)Proceedings of the Great Lakes Symposium on VLSI 202310.1145/3583781.3590235(613-618)Online publication date: 5-Jun-2023
https://dl.acm.org/doi/10.1145/3583781.3590235
Datta GYin ZJacob AJaiswal ABeerel P(2022)Towards Energy-Efficient Hyperspectral Image Processing Inside Camera PixelsComputer Vision – ECCV 2022 Workshops10.1007/978-3-031-25075-0_22(303-316)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-25075-0_22

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents