research-article

NeuroSim: A Circuit-Level Macro Model for Benchmarking Neuro-Inspired Architectures in Online Learning

Authors:

Shimeng YuAuthors Info & Claims

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Volume 37, Issue 12

Pages 3067 - 3080

https://doi.org/10.1109/TCAD.2018.2789723

Published: 01 December 2018 Publication History

Abstract

Neuro-inspired architectures based on synaptic memory arrays have been proposed for on-chip acceleration of weighted sum and weight update in machine/deep learning algorithms. In this paper, we developed NeuroSim, a circuit-level macro model that estimates the area, latency, dynamic energy, and leakage power to facilitate the design space exploration of neuro-inspired architectures with mainstream and emerging device technologies. NeuroSim provides flexible interface and a wide variety of design options at the circuit and device level. Therefore, NeuroSim can be used by neural networks (NNs) as a supporting tool to provide circuit-level performance evaluation. With NeuroSim, an integrated framework can be built with hierarchical organization from the device level (synaptic device properties) to the circuit level (array architectures) and then to the algorithm level (NN topology), enabling instruction-accurate evaluation on the learning accuracy as well as the circuit-level performance metrics at the run-time of online learning. Using multilayer perceptron as a case-study algorithm, we investigated the impact of the “analog” emerging nonvolatile memory (eNVM)’s “nonideal” device properties and benchmarked the tradeoffs between SRAM, digital, and analog eNVM-based architectures for online learning and offline classification.

References

[1]

J. Hasler and B. Marr, “Finding a roadmap to achieve large neuromorphic hardware systems,” Front. Neurosci., vol. 7, no. 118, pp. 1–29, 2013.

[2]

J. Von Neumann, “The principles of large-scale computing machines,” Ann. History Comput., vol. 3, no. 3, pp. 263–273, Jul./Sep. 1981.

Digital Library

[3]

R. Ananthanarayanan, S. K. Esser, H. D. Simon, and D. S. Modha, “The cat is out of the bag: Cortical simulations with 10⁹ neurons, 10¹³ synapses,” in Proc. Conf. High Perform. Comput. Netw. Stor. Anal., Portland, OR, USA, 2009, pp. 1–12.

[4]

S. Schmittet al., “Neuromorphic hardware in the loop: Training a deep spiking network on the brainscales wafer-scale system,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), 2017, pp. 2227–2234.

[5]

B. V. Benjaminet al., “Neurogrid: A mixed-analog–digital multichip system for large-scale neural simulations,” Proc. IEEE, vol. 102, no. 5, pp. 699–716, May 2014.

[6]

P. A. Merollaet al., “A million spiking-neuron integrated circuit with a scalable communication network and interface,” Science, vol. 345, no. 6197, pp. 668–673, 2014.

[7]

Y.-H. Chen, T. Krishna, J. Emer, and V. Sze, “14.5 eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” in Proc. IEEE Int. Solid-State Circuits Conf. (ISSCC), San Francisco, CA, USA, 2016, pp. 262–263.

[8]

D. Shin, J. Lee, J. Lee, and H.-J. Yoo, “14.2 DNPU: An 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks,” in Proc. Int. Solid-State Circuits Conf. (ISSCC), San Francisco, CA, USA, 2017, pp. 240–241.

[9]

S. Yu, Ed., Neuro-Inspired Computing Using Resistive Synaptic Devices. Cham, Switzerland: Springer, 2017.

[10]

P.-Y. Chenet al., “Mitigating effects of non-ideal synaptic device characteristics for on-chip learning,” in Proc. ACM/IEEE Int. Conf. Comput.-Aided Design (ICCAD), Austin, TX, USA, 2015, pp. 194–199.

[11]

G. W. Burret al., “Experimental demonstration and tolerancing of a large-scale neural network (165 000 synapses) using phase-change memory as the synaptic weight element,” IEEE Trans. Electron Devices, vol. 62, no. 11, pp. 3498–3507, Nov. 2015.

[12]

S. Agarwalet al., “Resistive memory device requirements for a neural algorithm accelerator,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Vancouver, BC, Canada, 2016, pp. 929–938.

[13]

P. Chiet al., “PRIME: A novel processing-in-memory architecture for neural network computation in ReRAM-based main memory,” in Proc. ACM/IEEE Int. Symp. Comput. Archit. (ISCA), Seoul, South Korea, 2016, pp. 27–39.

[14]

X. Liuet al., “Harmonica: A framework of heterogeneous computing systems with memristor-based neuromorphic computing accelerators,” IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 63, no. 5, pp. 617–628, May 2016.

[15]

A. Shafieeet al., “ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,” in Proc. Int. Symp. Comput. Archit. (ISCA), Seoul, South Korea, 2016, pp. 14–26.

[16]

L. Xiaet al., “MNSIM: Simulation platform for memristor-based neuromorphic computing system,” in Proc. ACM/IEEE Design Autom. Test Europe Conf. Exhibit. (DATE), Dresden, Germany, 2016, pp. 469–474.

[17]

MLP Simlator (+NeuroSim) Version 1.0. [Online]. Available: https://github.com/neurosim/MLP_NeuroSim

[18]

D. Kuzum, S. Yu, and H.-S. P. Wong, “Synaptic electronics: Materials, devices and applications,” Nanotechnology, vol. 24, no. 38, 2013, Art. no.

[19]

X. Guoet al., “Temperature-insensitive analog vector-by-matrix multiplier based on 55 nm NOR flash memory cells,” in Proc. IEEE Custom Integr. Circuits Conf. (CICC), Austin, TX, USA, 2017, pp. 1–4.

[20]

S. H. Joet al., “Nanoscale memristor device as synapse in neuromorphic systems,” Nano Lett., vol. 10, no. 4, pp. 1297–1301, 2010.

[21]

L. Gaoet al., “Fully parallel write/read in resistive synaptic array for accelerating on-chip learning,” Nanotechnology, vol. 26, no. 45, 2015, Art. no.

[22]

S. Parket al., “Neuromorphic speech systems using advanced ReRAM-based synapse,” in Proc. IEEE Int. Electron Device Meeting (IEDM), Washington, DC, USA, 2013, pp. 625–628.

[23]

J. Wooet al., “Improved synaptic behavior under identical pulses using AlO_x/HfO₂ bilayer RRAM array for neuromorphic systems,” IEEE Electron Device Lett., vol. 37, no. 8, pp. 994–997, Aug. 2016.

[24]

S. Yuet al., “A low energy oxide-based electronic synaptic device for neuromorphic visual systems with tolerance to device variation,” Adv. Mater., vol. 25, no. 12, pp. 1774–1779, 2013.

[25]

M. Preziosoet al., “Training and operation of an integrated neuromorphic network based on metal-oxide memristors,” Nature, vol. 521, no. 7550, pp. 61–64, 2015.

[26]

D. Kuzum, R. G. D. Jeyasingh, B. Lee, and H.-S. P. Wong, “Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing,” Nano Lett., vol. 12, no. 5, pp. 2179–2186, 2011.

[27]

M. Suriet al., “Phase change memory as synapse for ultra-dense neuromorphic systems: Application to complex visual pattern extraction,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Washington, DC, USA, 2011, pp. 79–82.

[28]

M. Hu, H. Li, Q. Wu, and G. S. Rose, “Hardware realization of BSB recall function using memristor crossbar arrays,” in Proc. Design Autom. Conf. (DAC), San Francisco, CA, USA, 2012, pp. 498–503.

[29]

P.-Y. Chenet al., “Technology-design co-optimization of resistive cross-point array for accelerating learning algorithms on chip,” in Proc. Design Autom. Test Europe Conf. Exhibit. (DATE), Grenoble, France, 2015, pp. 854–859.

[30]

J. Liang and H.-S. P. Wong, “Cross-point memory array without cell selectors—Device characteristics and data storage pattern dependencies,” IEEE Trans. Electron Devices, vol. 57, no. 10, pp. 2531–2538, Oct. 2010.

[31]

P.-Y. Chen, L. Gao, and S. Yu, “Design of resistive synaptic array for implementing on-chip sparse learning,” IEEE Trans. Multi-Scale Comput. Syst., vol. 2, no. 4, pp. 257–264, Oct./Dec. 2016.

[32]

S. Yu and P.-Y. Chen, “Emerging memory technologies: Recent trends and prospects,” IEEE Solid State Circuits Mag., vol. 8, no. 2, pp. 43–56, Jun. 2016.

[33]

S. Yuet al., “Scaling-up resistive synaptic arrays for neuro-inspired architecture: Challenges and prospect,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Washington, DC, USA, 2015, pp. 451–454.

[34]

S. J. E. Wilton and N. P. Jouppi, “CACTI: An enhanced cache access and cycle time model,” IEEE J. Solid-State Circuits, vol. 31, no. 5, pp. 677–688, May 1996.

[35]

X. Dong, C. Xu, Y. Xie, and N. P. Jouppi, “NVSim: A circuit-level performance, energy, and area model for emerging nonvolatile memory,” IEEE Trans. Comput.-Aided Design Integr. Circuits Syst., vol. 31, no. 7, pp. 994–1007, Jul. 2012.

Digital Library

[36]

D. Kadetotadet al., “Parallel architecture with resistive crosspoint array for dictionary learning acceleration,” IEEE J. Emerg. Sel. Topic Circuits Syst., vol. 5, no. 2, pp. 194–204, Jun. 2015.

[37]

Predictive Technology Model (PTM). [Online]. Available: http://ptm.asu.edu/

[38]

W. Qianet al., “Energy-efficient adaptive computing with multifunctional memory,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 64, no. 2, pp. 191–195, Feb. 2017.

[39]

P.-Y. Chen and S. Yu, “Partition SRAM and RRAM based synaptic arrays for neuro-inspired computing,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), Montreal, QC, Canada, 2016, pp. 2310–2313.

[40]

P. Rosenfeld, E. Cooper-Balis, and B. Jacob, “DRAMSim2: A cycle accurate memory system simulator,” IEEE Comput. Archit. Lett., vol. 10, no. 1, pp. 16–19, Jan./Jun. 2011.

Digital Library

[41]

FreePDK45. [Online]. Available: https://www.eda.ncsu.edu/wiki/FreePDK45:Contents

[42]

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.

[43]

T. Tang, L. Xia, B. Li, Y. Wang, and H. Yang, “Binary convolutional neural network on RRAM,” in Proc. ACM/IEEE Asia South Pac. Design Autom. Conf. (ASP-DAC), 2017, pp. 782–787.

[44]

L. Gao, P.-Y. Chen, and S. Yu, “Programming protocol optimization for analog weight tuning in resistive memories,” IEEE Electron Device Lett., vol. 36, no. 11, pp. 1157–1159, Nov. 2015.

Cited By

Lee JLi PDev KYoo JMeinerzhagen P(2024)Systolic Array Acceleration of Spiking Neural Networks with Application-Independent Split-Time Temporal CodingProceedings of the 29th ACM/IEEE International Symposium on Low Power Electronics and Design10.1145/3665314.3672292(1-6)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1145/3665314.3672292
Xu JLiu HDuan ZLiao XJin HYang XLi HLiu CMao FZhang Y(2024)ReHarvest: An ADC Resource-Harvesting Crossbar Architecture for ReRAM-Based DNN AcceleratorsACM Transactions on Architecture and Code Optimization10.1145/365920821:3(1-26)Online publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1145/3659208
Park JKim BSung H(2024)NavCim: Comprehensive Design Space Exploration for Analog Computing-in-Memory ArchitecturesProceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques10.1145/3656019.3676946(168-182)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3656019.3676946
Show More Cited By

Index Terms

NeuroSim: A Circuit-Level Macro Model for Benchmarking Neuro-Inspired Architectures in Online Learning

Index terms have been assigned to the content through auto-classification.

Recommendations

Circuit-Level Performance Benchmarking and Scalability Analysis of Carbon Nanotube Transistor Circuits

Carbon nanotubes (CNTs) show great promise as extensions to silicon CMOS due to their excellent electronic properties and extremely small size. Using a Carbon Nanotube Field Effect Transistor (CNFET) SPICE model, we evaluate circuit-level performance of ...
Circuit-level techniques to control gate leakage for sub-100nm CMOS
ISLPED '02: Proceedings of the 2002 international symposium on Low power electronics and design

Although still negligible for state-of-the-art CMOS, gate leakage will become significant in the future for sub-100nm technologies, due to the scaling of oxide thickness. We propose several circuit techniques to control gate leakage based on the fact ...
Circuit-level NBTI macro-models for collaborative reliability monitoring
GLSVLSI '10: Proceedings of the 20th symposium on Great lakes symposium on VLSI

The increasing significance of Negative Bias Temperature Instability (NBTI) induced device-reliability degradation presents a compelling reason to perform efficient circuit-level reliability tracking. We propose a novel collaborative monitoring frame-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems Volume 37, Issue 12

Dec. 2018

254 pages

ISSN:0278-0070

Issue’s Table of Contents

0278-0070 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 December 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lee JLi PDev KYoo JMeinerzhagen P(2024)Systolic Array Acceleration of Spiking Neural Networks with Application-Independent Split-Time Temporal CodingProceedings of the 29th ACM/IEEE International Symposium on Low Power Electronics and Design10.1145/3665314.3672292(1-6)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1145/3665314.3672292
Xu JLiu HDuan ZLiao XJin HYang XLi HLiu CMao FZhang Y(2024)ReHarvest: An ADC Resource-Harvesting Crossbar Architecture for ReRAM-Based DNN AcceleratorsACM Transactions on Architecture and Code Optimization10.1145/365920821:3(1-26)Online publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1145/3659208
Park JKim BSung H(2024)NavCim: Comprehensive Design Space Exploration for Analog Computing-in-Memory ArchitecturesProceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques10.1145/3656019.3676946(168-182)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3656019.3676946
Bai JZhang HKang W(2024)MixMixQ: Quantization with Mixed Bit-Sparsity and Mixed Bit-Width for CIM AcceleratorsProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3658809(537-540)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3649476.3658809
Fan MChen XYang DJin ZLiu WDe V(2024)ReCG: ReRAM-Accelerated Sparse Conjugate GradientProceedings of the 61st ACM/IEEE Design Automation Conference10.1145/3649329.3656515(1-6)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3649329.3656515
Qu SZhao SLi BHe YCai XZhang LWang YTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)CIM-MLC: A Multi-level Compilation Stack for Computing-In-Memory AcceleratorsProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640359(185-200)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640359
Lu ZWang XArafin MYang HLiu ZZhang JQu G(2024)An RRAM-Based Computing-in-Memory Architecture and Its Application in Accelerating Transformer InferenceIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.334565132:3(485-496)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1109/TVLSI.2023.3345651
Han LPan RZhou ZLu HChen YYang HHuang PSun GLiu XKang J(2024)CoMN: Algorithm-Hardware Co-Design Platform for Nonvolatile Memory-Based Convolutional Neural Network AcceleratorsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2024.335822043:7(2043-2056)Online publication date: 24-Jan-2024
https://dl.acm.org/doi/10.1109/TCAD.2024.3358220
Roohi ATabrizchi SMorsali MPan DAngizi S(2024)PiPSim: A Behavior-Level Modeling Tool for CNN Processing-in-Pixel AcceleratorsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.330557443:1(141-150)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TCAD.2023.3305574
Zhang ZJiang JWang QMao ZJing N(2024)3A-ReRAM: Adaptive Activation Accumulation in ReRAM-Based CNN AcceleratorIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.329796843:1(176-188)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TCAD.2023.3297968
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents