Cited By
View all- Imani MGupta SKim YRosing TManne SHunter HAltman E(2019)FloatPIMProceedings of the 46th International Symposium on Computer Architecture10.1145/3307650.3322237(802-815)Online publication date: 22-Jun-2019
Training low bitwidth convolutional neural network on RRAM
Authors: 
Yi Cai, Tianqi Tang, Lixue Xia, Ming Cheng, Zhenhua Zhu, Yu Wang, Huazhong YangAuthors Info & Claims
Yi Cai, Tianqi Tang, Lixue Xia, Ming Cheng, Zhenhua Zhu, Yu Wang, Huazhong YangAuthors Info & ClaimsPages 117 - 122
Abstract
Convolutional Neural Networks (CNNs) have achieved excellent performance on various artificial intelligence (AI) applications, while a higher demand on energy efficiency is required for future AI. Resistive Random-Access Memory (RRAM)-based computing system provides a promising solution to energy-efficient neural network training. However, it's difficult to support high-precision CNN in RRAM-based hardware systems. Firstly, multi-bit digital-analog interfaces will take up most energy overhead of the whole system. Secondly, it's difficult to write the RRAM to expected resistance states accurately; only low-precision numbers can be represented. To enable CNN training based on RRAM, we propose a low-bitwidth CNN training method, using low-bitwidth convolution outputs (CO), activations (A), weights (W) and gradients (G) to train CNN models based on RRAM. Furthermore, we design a system to implement the training algorithms. We explore the accuracy under different bitwidth combinations of (A,CO,W,G), and propose a practical tradeoff between accuracy and energy overhead. Our experiments demonstrate that the proposed system perform well on low-bitwidth CNN training tasks. For example, training LeNet-5 with 4-bit convolution outputs, 4-bit weights, 4-bit activations and 4-bit gradients on MNIST can still achieve 97.67% accuracy. Moreover, the proposed system can achieve 23.0X higher energy efficiency than GPU when processing the training task of LeNet-5, and 4.4X higher energy efficiency when processing the training task of ResNet-20.
References
[1]
K. Simonyan et al., "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
[2]
J. Fan et al., "Human tracking using convolutional neural networks." IEEE Transactions on Neural Networks, vol. 21, no. 10, pp. 1610--1623, 2010.
[3]
G. Hinton et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82--97, 2012.
[4]
A. Karpathy et al., "Deep visual-semantic alignments for generating image descriptions," in Computer Vision and Pattern Recognition, 2015.
[5]
B. Li et al., "Merging the interface: Power, area and accuracy cooptimization for rram crossbar-based mixed-signal computing system," in DAC, 2015, p. 13.
[6]
L. Xia et al., "Selected by input: Energy efficient structure for rram-based convolutional neural network," in DAC, 2016.
[7]
P. Chi et al., "Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory," in ISCA, vol. 43, 2016.
[8]
A. Shafiee et al., "Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars," in Proc. ISCA, 2016.
[9]
F. Alibart et al., "High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm," Nanotechnology, vol. 23, no. 7, p. 075201, 2012.
[10]
R. Degraeve et al., "Causes and consequences of the stochastic aspect of filamentary rram," Microelectronic Engineering, vol. 147, pp. 171--175, 2015.
[11]
M. Courbariaux et al., "Binarized neural network: Training deep neural networks with weights and activations constrained to+ 1 or-1," arXiv preprint arXiv:1602.02830, 2016.
[12]
M. Rastegari et al., "Xnor-net: Imagenet classification using binary convolutional neural networks," arXiv preprint arXiv:1603.05279, 2016.
[13]
S. Zhou et al., "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients," arXiv preprint arXiv:1606.06160, 2016.
[14]
T. Tang et al., "Binary convolutional neural network on rram," in Design Automation Conference (ASP-DAC), 2017 22nd Asia and South Pacific. IEEE, 2017, pp. 782--787.
[15]
M. Cheng et al., "Time: A training-in-memory architecture for memristor-based deep neural networks," in Proceedings of the 54th Annual Design Automation Conference 2017. ACM, 2017, p. 26.
[16]
Z. Jiang et al., "A compact model for metal-oxide resistive random access memory with experiment verification," IEEE Transactions on Electron Devices, vol. 63, no. 5, pp. 1884--1892, 2016.
[17]
S. Yu et al., "Scaling-up resistive synaptic arrays for neuro-inspired architecture: Challenges and prospect," in Electron Devices Meeting (IEDM), 2015 IEEE International. IEEE, 2015, pp. 17--3.
[18]
Y. LeCun et al., "Comparison of learning algorithms for handwritten digit recognition," in International conference on artificial neural networks, vol. 60. Perth, Australia, 1995, pp. 53--60.
[19]
K. He et al., "Deep residual learning for image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770--778.
[20]
C. M. Bishop, "Training with noise is equivalent to tikhonov regularization," Neural computation, vol. 7, no. 1, pp. 108--116, 1995.
[21]
A. F. Murray and P. J. Edwards, "Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training," IEEE Transactions on neural networks, vol. 5, no. 5, pp. 792--802, 1994.
[22]
X. Dong et al., "Nvsim: A circuit-level performance, energy, and area model for emerging nonvolatile memory," TCAD, vol. 31, no. 7, pp. 994--1007, 2012.
- Training low bitwidth convolutional neural network on RRAM
Recommendations
Training low bitwidth convolutional neural network on RRAM
2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC)Convolutional Neural Networks (CNNs) have achieved excellent performance on various artificial intelligence (AI) applications, while a higher demand on energy efficiency is required for future AI. Resistive Random-Access Memory (RRAM)-based computing ...
Binary convolutional neural network on RRAM
2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC)Recent progress in the machine learning field makes low bit-level Convolutional Neural Networks (CNNs), even CNNs with binary weights and binary neurons, achieve satisfying recognition accuracy on ImageNet dataset. Binary CNNs (BCNNs) make it possible for ...
Towards dropout training for convolutional neural networks
Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper ...
Comments
Information & Contributors
Information
Published In
Sponsors
- IEEE Circuits and Systems Society
- SIGDA: ACM Special Interest Group on Design Automation
- IEEE Council on Electronic Design Automation (CEDA)
Publisher
IEEE Press
Publication History
Published: 22 January 2018
Check for updates
Qualifiers
- Research-article
Conference
ASPDAC '18
Sponsor:
ASPDAC '18: 23rd Asia and South Pacific Design Automation Conference
January 22 - 25, 2018
Jeju, Republic of Korea
Acceptance Rates
Overall Acceptance Rate 466 of 1,454 submissions, 32%
Upcoming Conference
ASPDAC '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 154Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)2
Reflects downloads up to 16 Jan 2025
Other Metrics
Citations
Cited By
View all- Imani MGupta SKim YRosing TManne SHunter HAltman E(2019)FloatPIMProceedings of the 46th International Symposium on Computer Architecture10.1145/3307650.3322237(802-815)Online publication date: 22-Jun-2019
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in