HSB-GDM: a Hybrid Stochastic-Binary Circuit for Gradient Descent with Momentum in the Training of Neural Networks
Authors: 
Han Li, Heng Shi, Honglan Jiang, Siting LiuAuthors Info & Claims
Article No.: 22, Pages 1 - 6
Abstract
To enable an energy-efficient training of neural networks, this paper proposes a hybrid stochastic-binary (HSB) computing circuit for implementing the gradient descent with momentum (GDM) algorithm. By accumulating the weight-update values step by step, the proposed design executes the weight optimization of a neural network. At each step, the weight-update value is obtained by a linear combination of its previous value and the current gradient. In this design, it is computed in a hybrid stochastic-binary manner and encoded as a dynamic stochastic sequence consisting of 0, +1 and -1. Then, the weights are updated by accumulating the bits in the dynamic stochastic sequence. With the hybrid stochastic-binary design, this circuit can be readily integrated into a neural network accelerator to support online training with a small footprint. Experimental results show that, with little accuracy loss, the area efficiency of the proposed HSB-GDM is improved by 2.68× and energy efficiency by 4.41× compared to a floating-point design using bfloat16 data format.
References
[1]
Armin Alaghi, Weikang Qian, and John P. Hayes. 2018. The Promise and Challenge of Stochastic Computing. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37, 8 (2018), 1515--1531.
[2]
Sina Asadi, M. Hassan Najafi, and Mohsen Imani. 2021. A Low-Cost FSM-based Bit-Stream Generator for Low-Discrepancy Stochastic Computing. In DATE. 908--913.
[3]
Yu-Hsin Chen, Tushar Krishna, Joel S. Emer, and Vivienne Sze. 2017. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-State Circuits 52, 1 (2017), 127--138.
[4]
Brian R Gaines. 1969. Stochastic computing systems. Advances in information systems science (1969), 37--172.
[5]
Warren J Gross and Vincent C Gaudet. 2019. Stochastic Computing: Techniques and Applications. Springer.
[6]
John L Gustafson and Isaac T Yonemoto. 2017. Beating floating point at its own game: Posit arithmetic. Supercomputing frontiers and innovations 4, 2 (2017), 71--86.
[7]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR. 770--778.
[8]
Honglan Jiang, Francisco Javier Hernandez Santiago, Hai Mo, Leibo Liu, and Jie Han. 2020. Approximate Arithmetic Circuits: A Survey, Characterization, and Recent Applications. Proc. IEEE 108, 12 (2020), 2108--2135.
[9]
Zhendong Lin, Guangjun Xie, Shaowei Wang, Jie Han, and Yongqiang Zhang. 2021. A Review of Deterministic Approaches to Stochastic Computing. In NANOARCH. 1--6.
[10]
Siting Liu and Jie Han. 2020. Dynamic Stochastic Computing for Digital Signal Processing Applications. In DATE. 604--609.
[11]
Siting Liu, Honglan Jiang, Leibo Liu, and Jie Han. 2018. Gradient Descent Using Stochastic Circuits for Efficient Training of Learning Machines. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37, 11 (2018), 2530--2541.
[12]
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. 2018. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. In Computer Vision - ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 122--138.
[13]
M. Hassan Najafi, Shiva Jamali-Zavareh, David J. Lilja, Marc D. Riedel, Kia Bazargan, and Ramesh Harjani. 2017. Time-Encoded Values for Highly Efficient Stochastic Circuits. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 25, 5 (2017), 1644--1657.
[14]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR. 4510--4520.
[15]
Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In ICML (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 6105--6114.
[16]
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. 2020. Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. In ICLR.
[17]
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. In CVPR. 6848--6856. Received 30 September 2022; revised 20 November 2022; accepted 7 December 2022
Index Terms
- HSB-GDM: a Hybrid Stochastic-Binary Circuit for Gradient Descent with Momentum in the Training of Neural Networks
Recommendations
Neural Network Classifiers Using a Hardware-Based Approximate Activation Function with a Hybrid Stochastic Multiplier
Special Issue on Emerging Networks-on-Chip and Regular PapersNeural networks are becoming prevalent in many areas, such as pattern recognition and medical diagnosis. Stochastic computing is one potential solution for neural networks implemented in low-power back-end devices such as solar-powered devices and ...
Low-Cost Stochastic Hybrid Multiplier for Quantized Neural Networks
Special Issue on HALO for Energy-Constrained On-Chip Machine LearningWith increased interests of neural networks, hardware implementations of neural networks have been investigated. Researchers pursue low hardware cost by using different technologies such as stochastic computing (SC) and quantization. More specifically, ...
Robust Adaptive Gradient-Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain
For a recurrent neural network (RNN), its transient response is a critical issue, especially for real-time signal processing applications. The conventional RNN training algorithms, such as backpropagation through time (BPTT) and real-time recurrent ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2022
140 pages
Copyright © 2022 Owner/Author(s).
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 31 May 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Shanghai Sailing Program
- National Natural Science Foundation of China
Conference
NANOARCH '22
Sponsor:
NANOARCH '22: 17th ACM International Symposium on Nanoscale Architectures
December 7 - 9, 2022
OR, Virtual, USA
Acceptance Rates
NANOARCH '22 Paper Acceptance Rate 25 of 31 submissions, 81%;
Overall Acceptance Rate 55 of 87 submissions, 63%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 223Total Downloads
- Downloads (Last 12 months)180
- Downloads (Last 6 weeks)42
Reflects downloads up to 13 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in