Cited By
View all- Pietrołaj MBlok M(2024)Resource constrained neural network trainingScientific Reports10.1038/s41598-024-52356-114:1Online publication date: 29-Jan-2024
Dual-Precision Deep Neural Network
Pages 30 - 34
Abstract
On-line Precision scalability of the deep neural networks(DNNs) is a critical feature to support accuracy and complexity trade-off during the DNN inference. In this paper, we propose dual-precision DNN that includes two different precision modes in a single model, thereby supporting an on-line precision switch without re-training. The proposed two-phase training process optimizes both low- and high-precision modes.
References
[1]
Banner, R., Nahshan, Y., Hoffer, E., Soudry, D.: Aciq: Analytical clipping for integer quantization of neural networks (2018).
[2]
Cai, J., Takemoto, M., Nakajo, H.: A deep look into logarithmic quantization of model parameters in neural networks. In: Proceedings of the 10th International Conference on Advances in Information Technology. pp. 1--8 (2018).
[3]
Chung, J., Shin, T.: Simplifying deep neural networks for neuromorphic architec- tures. In: 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC). pp. 1--6. IEEE (2016).
[4]
Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect:Trainin gdeepneuralnet- works with binary weights during propagations. In: Advances in neural information processing systems. pp. 3123--3131 (2015).
[5]
Gupta, S., Agrawal, A., Gopalakrishnan, K., Narayanan, P.: Deep learning with limited numerical precision. In: International Conference on Machine Learning. pp. 1737--1746 (2015).
[6]
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).
[7]
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarizedneural networks. In: Advances in neural information processing systems. pp. 4107--4115 (2016).
[8]
Hwang, K., Sung, W.: Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In: 2014 IEEE Workshop on Signal Processing Systems (SiPS). pp. 1--6. IEEE (2014).
[9]
Jain, S.R., Gural, A., Wu, M., Dick, C.H.: Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. arXiv preprint arXiv:1903.08066 2(3), 7 (2019).
[10]
Krizhevsky, A., Sutskever, I., Hinton, G.: '2012 alexnet. Advances In Neural Infor- mation Processing Systems pp. 1--9 (2012).
[11]
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009).
[12]
Lee, E.H., Miyashita, D., Chai, E., Murmann, B., Wong, S.S.: Lognet: Energy- efficient neural networks using logarithmic computation. In: 2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5900- 5904. IEEE (2017).
[13]
Li, F., Zhang, B., Liu, B.: Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016).
[14]
Liu, X., Ye, M., Zhou, D., Liu, Q.: Post-training quantization with multiple points: Mixed precision without mixed precision. arXiv preprint arXiv:2002.09049 (2020).
[15]
Migacz, S.: 8-bit inference with tensorrt. In: GPU technology conference. vol. 2, p. 5 (2017).
[16]
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnornet: Imagenet classification using binary convolutional neural networks. In: European conference on computer vision. pp. 525--542. Springer (2016).
[17]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[18]
Sung, W., Shin, S., Hwang, K.: Resiliency of deep neural networks under quantization. arXiv preprint arXiv:1511.06488 (2015).
[19]
Wu, S., Li, G., Chen, F., Shi, L.: Training and inference with integers in deep neural networks. arXiv preprint arXiv:1802.04680 (2018).
[20]
Zhao, R., Hu, Y., Dotzel, J., De Sa, C., Zhang, Z.: Improving neural network quantization using outlier channel splitting. arXiv preprint arXiv:1901.09504 (2019)
[21]
Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044 (2017).
Index Terms
- Dual-Precision Deep Neural Network
Recommendations
Deep Neural Network Regularization (DNNR) on Denoised Image
Image dehazing in supervised learning models suffers from overfitting and underfitting problems. To avoid overfitting, the authors use regularization techniques like dropout and L2 norm. Dropout helps in reducing overfitting and batch normalization ...
EEG-based prediction of driver's cognitive performance by deep convolutional neural network
We considered the prediction of driver's cognitive states related to driving performance using EEG signals. We proposed a novel channel-wise convolutional neural network (CCNN) whose architecture considers the unique characteristics of EEG data. We also ...
On the challenges in programming mixed-precision deep neural networks
MAPL 2020: Proceedings of the 4th ACM SIGPLAN International Workshop on Machine Learning and Programming LanguagesDeep Neural Networks (DNNs) are resilient to reduced data precision, which motivates exploiting low-precision data formats for more efficient computation, especially on custom hardware accelerators. Multiple low-precision types can be mixed to fit the ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2020
250 pages
ISBN:9781450375511
DOI:10.1145/3430199
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 December 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
AIPR 2020
AIPR 2020: 2020 3rd International Conference on Artificial Intelligence and Pattern Recognition
June 26 - 28, 2020
Xiamen, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 89Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025
Other Metrics
Citations
Cited By
View all- Pietrołaj MBlok M(2024)Resource constrained neural network trainingScientific Reports10.1038/s41598-024-52356-114:1Online publication date: 29-Jan-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in