research-article

Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks

Authors:

Kyounghoon Kim,

Kiyoung ChoiAuthors Info & Claims

DAC '16: Proceedings of the 53rd Annual Design Automation Conference

Article No.: 124, Pages 1 - 6

https://doi.org/10.1145/2897937.2898011

Published: 05 June 2016 Publication History

Abstract

This paper presents an efficient DNN design with stochastic computing. Observing that directly adopting stochastic computing to DNN has some challenges including random error fluctuation, range limitation, and overhead in accumulation, we address these problems by removing near-zero weights, applying weight-scaling, and integrating the activation function with the accumulator. The approach allows an easy implementation of early decision termination with a fixed hardware design by exploiting the progressive precision characteristics of stochastic computing, which was not easy with existing approaches. Experimental results show that our approach outperforms the conventional binary logic in terms of gate area, latency, and power consumption.

References

[1]

R. Venkatesan, et al., "Spintastic: spin-based stochastic logic for energy-efficient computing," Proc. DATE, pp. 1575--1578, 2015.

Digital Library

[2]

A. Krizhevsky, et al., "Imagenet classification with deep convolutional neural networks," Proc. NIPS, pp. 1097--1105, 2012.

[3]

G. Hinton, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, pp. 82--97, 2012.

[4]

C. Szegedy, et al., "Going deeper with convolutions," arXiv preprint arXiv:1409.4842, 2014.

[5]

A. Coates, et al., "Deep learning with COTS HPC systems," Proc. ICML, pp. 1337--1345, 2013.

[6]

M. Montemerlo, et al., "Junior: The stanford entry in the urban challenge," J. Field Robot., vol. 25, pp. 569--597, 2008.

Digital Library

[7]

K. H. Lee, et al., "A low-power processor with configurable embedded machine-learning accelerators for high-order and adaptive analysis of medical-sensor signals," IEEE J. Solid-State Circuit, vol. 48, pp. 1625--1637, 2013.

[8]

M. Courbariaux, et al., "Training deep neural networks with low precision multiplications," Proc. workshop contribution at ICLR, 2015.

[9]

H. Geoffrey, et al., "Distilling the knowledge in a neural network," Proc. NIPS workshop, 2014.

[10]

B. D. Brown, et al., "Stochastic neural computation. II. Soft competitive learning," IEEE Trans. Comput., vol. 50, pp. 906--920, 2001.

Digital Library

[11]

K. Sanni, et al., "FPGA implementation of a Deep Belief Network architecture for character recognition using stochastic computation," Proc. CISS, pp. 1--5, 2015.

[12]

Y. A. LeCun, et al., "Efficient backprop," in Neural networks: Tricks of the trade, ed: Springer, 2012, pp. 9--48.

Digital Library

[13]

S. Han, et al., "Learning both weights and connections for efficient neural networks," Proc. NIPS, 2015.

[14]

J. Schmidhuber, "Deep learning in neural networks: An overview," Neural Networks, vol. 61, pp. 85--117, 2015.

Digital Library

[15]

S. Venkataramani, et al., "Scalable-effort classifiers for energyefficient machine learning," Proc. DAC, p. 67, 2015.

Digital Library

[16]

L. Deng, "The MNIST database of handwritten digit images for machine learning research," IEEE Signal Process. Mag., vol. 29, pp. 141--142, 2012.

[17]

B. D. Brown, et al., "Stochastic neural computation. I. Computational elements," IEEE Trans. Comput., vol. 50, pp. 891--905, 2001.

Digital Library

[18]

A. Ardakani, et al., "VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing," arXiv preprint arXiv:1509.08972, 2015.

[19]

O. C. Ibe, Elements of Random Walk and Diffusion Processes: John Wiley & Sons, 2013.

Digital Library

[20]

K. Kim, et al., "Approximate De-randomizer for Stochastic Circuits," Proc. ISOCC, 2015.

[21]

A. Alaghi, et al., "Survey of stochastic computing," ACM Trans. Embed. Comput. Syst., vol. 12, p. 92, 2013.

Digital Library

[22]

K. Kim, et al., "An Energy-Efficient Random Number Generator for Stochastic Circuits," Proc. ASP-DAC, 2016.

Cited By

Abdul Halim ZLee YAb Wahab MAlmohamad T(2024)Stochastic Computing Convolution Neural Network Architecture Reinvented For Highly Efficient Artificial Intelligence Workload on Field Programmable Gate ArrayResearch10.34133/research.0307Online publication date: 8-Jan-2024
https://doi.org/10.34133/research.0307
Xie THu YWei RLi MWang YWang RHuang R(2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546691
ARIMURA YYAMASHITA S(2024)Efficient Realization of an SC Circuit with Feedback and Its ApplicationsIEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences10.1587/transfun.2023EAP1019E107.A:7(958-965)Online publication date: 1-Jul-2024
https://doi.org/10.1587/transfun.2023EAP1019
Show More Cited By

Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Hardware
  1. Integrated circuits
    1. Logic circuits

Recommendations

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks
GLSVLSI '17: Proceedings of the Great Lakes Symposium on VLSI 2017

Recently, Deep Convolutional Neural Networks (DCNNs) have made tremendous advances, achieving close to or even better accuracy than human-level perception in various tasks. Stochastic Computing (SC), as an alternate to the conventional binary computing ...
Cost-effective stochastic MAC circuits for deep neural networks
Abstract
Stochastic computing (SC) is a promising computing paradigm that can help address both the uncertainties of future process technology and the challenges of efficient hardware realization for deep neural networks (DNNs). However the ...
Reversible stochastic computing in ratioed, unsigned extended and signed extended stochastic logic formats
Abstract
Stochastic computing is a promising technique for addressing the problems of increase in leakage current, increased sensitivity to soft errors etc. in today's semiconductor industry. Stochastic computing has been presented in open ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

DAC '16: Proceedings of the 53rd Annual Design Automation Conference

June 2016

1048 pages

ISBN:9781450342360

DOI:10.1145/2897937

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

DAC '16

DAC '16: The 53rd Annual Design Automation Conference 2016

June 5 - 9, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

151
Total Citations
View Citations
1,398
Total Downloads

Downloads (Last 12 months)139
Downloads (Last 6 weeks)15

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abdul Halim ZLee YAb Wahab MAlmohamad T(2024)Stochastic Computing Convolution Neural Network Architecture Reinvented For Highly Efficient Artificial Intelligence Workload on Field Programmable Gate ArrayResearch10.34133/research.0307Online publication date: 8-Jan-2024
https://doi.org/10.34133/research.0307
Xie THu YWei RLi MWang YWang RHuang R(2024)ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546691(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546691
ARIMURA YYAMASHITA S(2024)Efficient Realization of an SC Circuit with Feedback and Its ApplicationsIEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences10.1587/transfun.2023EAP1019E107.A:7(958-965)Online publication date: 1-Jul-2024
https://doi.org/10.1587/transfun.2023EAP1019
Mou DWang BLiu D(2024)SC-CGRA: An Energy-Efficient CGRA Using Stochastic ComputingIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.345331035:11(2023-2038)Online publication date: Nov-2024
https://doi.org/10.1109/TPDS.2024.3453310
Hu ALi WLyu DHe G(2024)Efficient Parallel Stochastic Computing Multiply-Accumulate (MAC) Technique Using Pseudo-Sobol Bit-StreamsIEEE Transactions on Nanotechnology10.1109/TNANO.2024.336862823(170-179)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TNANO.2024.3368628
Liu YYu STasnim MTan S(2024)Fast and Scaled Counting-Based Stochastic Computing Divider DesignIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2024.336542643:8(2277-2287)Online publication date: Aug-2024
https://doi.org/10.1109/TCAD.2024.3365426
Zhong KWang JChen LWang PYou ZZhang YZhang J(2024)A Brief Survey on Randomizer Design and Optimization for Efficient Stochastic Computing2024 IEEE International Test Conference in Asia (ITC-Asia)10.1109/ITC-Asia62534.2024.10661315(1-6)Online publication date: 18-Aug-2024
https://doi.org/10.1109/ITC-Asia62534.2024.10661315
EL-Hady MAbbas MKhanday FSaid LRadwan A(2024)DISH: Digital image steganography using stochastic-computing with high-capacityMultimedia Tools and Applications10.1007/s11042-023-17998-983:25(66033-66048)Online publication date: 20-Jan-2024
https://doi.org/10.1007/s11042-023-17998-9
Guo HZhao YLi ZHao YLiu CSong XLi XDu ZZhang RGuo QChen TXu Z(2023)Cambricon-U: A Systolic Random Increment Memory Architecture for Unary ComputingProceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture10.1145/3613424.3614286(424-437)Online publication date: 28-Oct-2023
https://dl.acm.org/doi/10.1145/3613424.3614286
Hu SHan KHu J(2023)High Performance and Hardware Efficient Stochastic Computing Elements for Deep Neural Network2023 6th World Conference on Computing and Communication Technologies (WCCCT)10.1109/WCCCT56755.2023.10052402(181-186)Online publication date: 6-Jan-2023
https://doi.org/10.1109/WCCCT56755.2023.10052402
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents