research-article

Open access

Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)

Authors:

Md Abdullah-Al Kaiser,

Sreetama Sarkar,

Peter A. Beerel,

Akhilesh R. JaiswalAuthors Info & Claims

GLSVLSI '23: Proceedings of the Great Lakes Symposium on VLSI 2023

Pages 613 - 618

https://doi.org/10.1145/3583781.3590235

Published: 05 June 2023 Publication History

Abstract

The massive amounts of data generated by camera sensors motivate data processing inside pixel arrays, i.e., at the extreme-edge. Several critical developments have fueled recent interest in the processing-in-pixel-in-memory paradigm for a wide range of visual machine intelligence tasks, including (1) advances in 3D integration technology to enable complex processing inside each pixel in a 3D integrated manner while maintaining pixel density, (2) analog processing circuit techniques for massively parallel low-energy in-pixel computations, and (3) algorithmic techniques to mitigate non-idealities associated with analog processing through hardware-aware training schemes. This article presents a comprehensive technology-circuit-algorithm landscape that connects technology capabilities, circuit design strategies, and algorithmic optimizations to power, performance, area, bandwidth reduction, and application-level accuracy metrics. We present our results using a comprehensive co-design framework incorporating hardware and algorithmic optimizations for various complex real-life visual intelligence tasks mapped onto our P2M paradigm.

References

[1]

2013. WikiChip. https://en.wikichip.org/wiki/WikiChip

[2]

L Bose, P Dudek, Stephen J Chen, Jand C, and Walterio W Mayol-Cuevas. 2020. Fully embedding fast convolutional networks on pixel processor arrays. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16. Springer, 488--503.

Digital Library

[3]

Y Chen, H Dai, and Y Ding. 2022. Pseudo-stereo for monocular 3d object detection in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 887--897.

[4]

G Datta, S Kundu, Z Yin, J Lakkireddy, R Mathai, A Jacob, P Beerel, and A Jaiswal. 2022a. A processing-in-pixel-in-memory paradigm for resource-constrained TinyML applications. Scientific Reports, Vol. 12, 1 (2022), 14396.

[5]

G Datta, S Kundu, Z Yin, J Mathai, Z Liu, Z Wang, M Tian, S Lu, R Lakkireddy, et al. 2022b. P 2 M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking. In 2022 IFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC). IEEE, 1--6.

[6]

R Eki, S Yamada, H Ozawa, H Kai, K Okuike, H Gowtham, H Nakanishi, E Almog, Y Livne, G Yuval, et al. 2021. 9.6 A 1/2.3 inch 12.3 Mpixel with on-chip 4.97 TOPS/W CNN processor back-illuminated stacked CMOS image sensor. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 64. IEEE, 154--156.

[7]

A Ghodrati, Babak E Bejnordi, and A Habibian. 2021. FrameExit: Conditional Early Exiting for Efficient Video Recognition. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 15603--15613.

[8]

T Hsu, G Chen, Y Chen, C Lo, R Liu, M Chang, K Tang, and C Hsieh. 2022. A 0.8 V Intelligent Vision Sensor with Tiny Convolutional Neural Network and Programmable Weights Using Mixed-Mode Processing-in-Sensor Technique for Image Classification. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 65. IEEE, 1--3.

[9]

T Hsu, Y Chen, R Liu, C Lo, K Tang, M Chang, and C Hsieh. 2020. A 0.5-V real-time computational CMOS image sensor with programmable kernel for feature extraction. IEEE Journal of Solid-State Circuits, Vol. 56, 5 (2020), 1588--1596.

[10]

L Jiao, R Zhang, F Liu, S Yang, B Hou, L Li, and X Tang. 2021. New generation deep learning for video object detection: A survey. IEEE Transactions on Neural Networks and Learning Systems (2021).

[11]

Y Kagawa, N Fujii, K Aoyagi, Y Kobayashi, S Nishi, N Todaka, S Takeshita, J Taura, H Takahashi, Y Nishimura, et al. 2016. Novel stacked CMOS image sensor with advanced Cu2Cu hybrid bonding. In 2016 IEEE International Electron Devices Meeting (IEDM). IEEE, 8--4.

[12]

Y Kagawa, H Hashiguchi, T Kamibayashi, M Haneda, N Fujii, S Furuse, T Hirano, and H Iwamoto. 2020. Impacts of Misalignment on 1μm Pitch Cu-Cu Hybrid Bonding. In 2020 IEEE International Interconnect Technology Conference (IITC). IEEE, 148--150.

[13]

Y Kagawa and H Iwamoto. 2019. 3D integration technologies for the stacked CMOS image sensors. In 2019 International 3D Systems Integration Conference (3DIC). IEEE, 1--4.

[14]

S Kuo, M Dunna, D Bharadia, and P Mercier. 2022. A WiFi and Bluetooth backscattering combo chip featuring beam steering via a fully-reflective phased-controlled multi-antenna termination technique enabling operation over 56 meters. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 65. IEEE, 1--3.

[15]

M Lefebvre, L Moreau, R Dekimpe, and D Bol. 2021. 7.7 A 0.2-to-3.6 TOPS/W programmable convolutional imager soc with in-sensor current-domain ternary-weighted MAC operations for feature extraction and region-of-interest detection. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 64. IEEE, 118--120.

[16]

H Nam and B Han. 2016. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4293--4302.

[17]

M Seo, M Chu, H Jung, S Kim, J Song, J Lee, S Kim, J Lee, S Byun, D Bae, et al. 2021. A 2.6 e-rms low-random-noise, 116.2 mW low-power 2-Mp global shutter CMOS image sensor with pixel-level ADC and in-pixel memory. In 2021 Symposium on VLSI Technology. IEEE, 1--2.

[18]

R Song, K Huang, Z Wang, and H Shen. 2022. A reconfigurable convolution-in-pixel CMOS image sensor architecture. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, 10 (2022), 7212--7225.

[19]

S Tabrizchi, A Nezhadi, S Angizi, and A Roohi. 2023. AppCiP: Energy-Efficient Approximate Convolution-in-Pixel Scheme for Neural Network Acceleration. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2023).

[20]

K Teja et al. 2021. Design of 1.8 V LVDS Transmitter in GF 22nm For Associative Memory. In 2021 International Semiconductor Conference (CAS). IEEE, 201--204.

[21]

H Xu, N Lin, L Luo, Q Wei, R Wang, C Zhuo, X Yin, F Qiao, and H Yang. 2021. Senputing: An ultra-low-power always-on vision perception chip featuring the deep fusion of sensing and computing. IEEE Transactions on Circuits and Systems I: Regular Papers, Vol. 69, 1 (2021), 232--243.

[22]

F Yu, H Chen, X Wang, W Xian, F Chen, Yand Liu, V Madhavan, and T Darrell. 2020. BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]

Y Zhang, X Zhang, and M S Bakir. 2018. Benchmarking digital die-to-die channels in 2.5-D and 3-D heterogeneous integration platforms. IEEE Transactions on Electron Devices, Vol. 65, 12 (2018), 5460--5467.

[24]

F Zhou and Y Chai. 2020. Near-sensor and in-sensor computing. Nature Electronics, Vol. 3, 11 (2020), 664--671.

Cited By

Yin ZDatta GKaiser MBeerel PJacob AJaiswal A(2023)Design Considerations for 3D Heterogeneous Integration Driven Analog Processing-in-Pixel for Extreme-Edge Intelligence2023 IEEE International Conference on Rebooting Computing (ICRC)10.1109/ICRC60800.2023.10386206(1-5)Online publication date: 5-Dec-2023
https://doi.org/10.1109/ICRC60800.2023.10386206

Index Terms

Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)
1. Computing methodologies
  1. Machine learning
2. Hardware
  1. Integrated circuits

Recommendations

A low-power 12-bit SAR ADC for remote pressure measurement

A 12-bit low power successive approximation ADC has been designed and fabricated in 0.35 µm, poly---poly capacitor AMS technology. The SAR ADC is targeted for bridge type pressure sensor in wireless sensor node applications. An improved cascaded ...
Low-Voltage CMOS Transconductor-C Filter Design Using Charge-Pump Circuit

A very low voltage transconductor for video frequency range applications and compatible with standard CMOS technology is described. In the proposed transconductor, except the DC level shifter circuit (DCLS), the whole transconductor uses the main supply ...
Design of variable gain low noise amplifier using feedback circuit with memory circuits for 5.2 GHz band

This paper presents a novel adaptive gain control method for Low Noise Amplifiers (LNAs) at the 5.2 GHz band using a feedback circuit, and operating in the baseband signal frequency. A uniform step variable gain can be implemented using a two-stage LNA ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GLSVLSI '23: Proceedings of the Great Lakes Symposium on VLSI 2023

June 2023

731 pages

ISBN:9798400701252

DOI:10.1145/3583781

General Chairs:
Himanshu Thapliyal
University of Tennessee, Knoxville, USA
,
Ronald DeMara
University of Central Florida, USA
,
Program Chairs:
Inna Partin-Vaisband
University of Illinois Chicago, USA
,
Srinivas Katkoori
University of South Florida, USA

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGDA: ACM Special Interest Group on Design Automation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2023

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GLSVLSI '23

Sponsor:

SIGDA

GLSVLSI '23: Great Lakes Symposium on VLSI 2023

June 5 - 7, 2023

TN, Knoxville, USA

Acceptance Rates

Overall Acceptance Rate 312 of 1,156 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
187
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)22

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yin ZDatta GKaiser MBeerel PJacob AJaiswal A(2023)Design Considerations for 3D Heterogeneous Integration Driven Analog Processing-in-Pixel for Extreme-Edge Intelligence2023 IEEE International Conference on Rebooting Computing (ICRC)10.1109/ICRC60800.2023.10386206(1-5)Online publication date: 5-Dec-2023
https://doi.org/10.1109/ICRC60800.2023.10386206

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents