Sensitivity-Based Error Resilient Techniques with Heterogeneous Multiply-Accumulate Unit for Voltage Scalable Deep Neural Network Accelerators

Dongyeob Shin, Wonseok Choi, Jongsun Park, Swaroop Ghosh

Research output: Contribution to journalArticle

Abstract

With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we present an error resilient technique to enable aggressive voltage scaling by exploiting the asymmetric error resilience (sensitivity) with respect to DNN layers, filters, and channels. First-order Taylor expansion is used to evaluate the filter/channel-level weight sensitivities of large scale DNNs which accurately approximates weight sensitivities from actual error injection simulations. We also present the heterogeneous multiply-accumulate (MAC) unit based design approach where some of the MAC units are designed larger with shorter critical path delays for robustness to aggressive voltage scaling while other MAC units are designed relatively smaller. The sensitivity variations among filter weights can be leveraged to design DNN accelerator such that the computations with more sensitive weights are assigned to more robust (larger) MAC units while the computations with less sensitive weights are assigned to less robust (smaller) MAC units. Using dynamic programming, the sizes of MAC units are selected to achieve best DNN accuracy under ISO area constraint. As a result, the proposed voltage scalable DNN accelerator can achieve 34% energy savings in post layout simulations using 65 nm CMOS process with ImageNet dataset using ResNet-18 compared to state-of-the-art timing error recovery technique.

Original languageEnglish (US)
Article number8792195
Pages (from-to)520-531
Number of pages12
JournalIEEE Journal on Emerging and Selected Topics in Circuits and Systems
Volume9
Issue number3
DOIs
StatePublished - Sep 1 2019

Fingerprint

Particle accelerators
Electric potential
Network layers
Deep neural networks
Dynamic programming
Energy conservation
Voltage scaling

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering

Cite this

@article{f500ee4279384b34b62af86171ef6c92,
title = "Sensitivity-Based Error Resilient Techniques with Heterogeneous Multiply-Accumulate Unit for Voltage Scalable Deep Neural Network Accelerators",
abstract = "With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we present an error resilient technique to enable aggressive voltage scaling by exploiting the asymmetric error resilience (sensitivity) with respect to DNN layers, filters, and channels. First-order Taylor expansion is used to evaluate the filter/channel-level weight sensitivities of large scale DNNs which accurately approximates weight sensitivities from actual error injection simulations. We also present the heterogeneous multiply-accumulate (MAC) unit based design approach where some of the MAC units are designed larger with shorter critical path delays for robustness to aggressive voltage scaling while other MAC units are designed relatively smaller. The sensitivity variations among filter weights can be leveraged to design DNN accelerator such that the computations with more sensitive weights are assigned to more robust (larger) MAC units while the computations with less sensitive weights are assigned to less robust (smaller) MAC units. Using dynamic programming, the sizes of MAC units are selected to achieve best DNN accuracy under ISO area constraint. As a result, the proposed voltage scalable DNN accelerator can achieve 34{\%} energy savings in post layout simulations using 65 nm CMOS process with ImageNet dataset using ResNet-18 compared to state-of-the-art timing error recovery technique.",
author = "Dongyeob Shin and Wonseok Choi and Jongsun Park and Swaroop Ghosh",
year = "2019",
month = "9",
day = "1",
doi = "10.1109/JETCAS.2019.2933862",
language = "English (US)",
volume = "9",
pages = "520--531",
journal = "IEEE Journal on Emerging and Selected Topics in Circuits and Systems",
issn = "2156-3357",
publisher = "IEEE Circuits and Systems Society",
number = "3",

}

Sensitivity-Based Error Resilient Techniques with Heterogeneous Multiply-Accumulate Unit for Voltage Scalable Deep Neural Network Accelerators. / Shin, Dongyeob; Choi, Wonseok; Park, Jongsun; Ghosh, Swaroop.

In: IEEE Journal on Emerging and Selected Topics in Circuits and Systems, Vol. 9, No. 3, 8792195, 01.09.2019, p. 520-531.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Sensitivity-Based Error Resilient Techniques with Heterogeneous Multiply-Accumulate Unit for Voltage Scalable Deep Neural Network Accelerators

AU - Shin, Dongyeob

AU - Choi, Wonseok

AU - Park, Jongsun

AU - Ghosh, Swaroop

PY - 2019/9/1

Y1 - 2019/9/1

N2 - With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we present an error resilient technique to enable aggressive voltage scaling by exploiting the asymmetric error resilience (sensitivity) with respect to DNN layers, filters, and channels. First-order Taylor expansion is used to evaluate the filter/channel-level weight sensitivities of large scale DNNs which accurately approximates weight sensitivities from actual error injection simulations. We also present the heterogeneous multiply-accumulate (MAC) unit based design approach where some of the MAC units are designed larger with shorter critical path delays for robustness to aggressive voltage scaling while other MAC units are designed relatively smaller. The sensitivity variations among filter weights can be leveraged to design DNN accelerator such that the computations with more sensitive weights are assigned to more robust (larger) MAC units while the computations with less sensitive weights are assigned to less robust (smaller) MAC units. Using dynamic programming, the sizes of MAC units are selected to achieve best DNN accuracy under ISO area constraint. As a result, the proposed voltage scalable DNN accelerator can achieve 34% energy savings in post layout simulations using 65 nm CMOS process with ImageNet dataset using ResNet-18 compared to state-of-the-art timing error recovery technique.

AB - With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we present an error resilient technique to enable aggressive voltage scaling by exploiting the asymmetric error resilience (sensitivity) with respect to DNN layers, filters, and channels. First-order Taylor expansion is used to evaluate the filter/channel-level weight sensitivities of large scale DNNs which accurately approximates weight sensitivities from actual error injection simulations. We also present the heterogeneous multiply-accumulate (MAC) unit based design approach where some of the MAC units are designed larger with shorter critical path delays for robustness to aggressive voltage scaling while other MAC units are designed relatively smaller. The sensitivity variations among filter weights can be leveraged to design DNN accelerator such that the computations with more sensitive weights are assigned to more robust (larger) MAC units while the computations with less sensitive weights are assigned to less robust (smaller) MAC units. Using dynamic programming, the sizes of MAC units are selected to achieve best DNN accuracy under ISO area constraint. As a result, the proposed voltage scalable DNN accelerator can achieve 34% energy savings in post layout simulations using 65 nm CMOS process with ImageNet dataset using ResNet-18 compared to state-of-the-art timing error recovery technique.

UR - http://www.scopus.com/inward/record.url?scp=85070722599&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070722599&partnerID=8YFLogxK

U2 - 10.1109/JETCAS.2019.2933862

DO - 10.1109/JETCAS.2019.2933862

M3 - Article

AN - SCOPUS:85070722599

VL - 9

SP - 520

EP - 531

JO - IEEE Journal on Emerging and Selected Topics in Circuits and Systems

JF - IEEE Journal on Emerging and Selected Topics in Circuits and Systems

SN - 2156-3357

IS - 3

M1 - 8792195

ER -