Sensitivity based error resilient techniques for energy efficient deep neural network accelerators

Wonseok Choi, Dongyeob Shin, Jongsun Park, Swaroop Ghosh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we propose novel error resilient techniques to enable aggressive voltage scaling by exploiting different amount of error resilience (sensitivity) with respect to DNN layers, filters, and channels. First, to rapidly evaluate filter/channel-level weight sensitivities of large scale DNNs, first-order Taylor expansion is used, which accurately approximates weight sensitivity from actual error injection simulation. With measured timing error probability of each multiply-accumulate (MAC) units considering process variations, the sensitivity variation among filter weights can be leveraged to design DNN accelerator, such that the computations with more sensitive weights are assigned to more robust MAC units, while those with less sensitive weights are assigned to less robust MAC units. Based on post-synthesis timing simulations, 51% energy savings has been achieved with CIFAR-10 dataset using VGG-9 compared to state-of-the-art timing error recovery technique with the same constraint of 3% accuracy loss.

Original languageEnglish (US)
Title of host publicationProceedings of the 56th Annual Design Automation Conference 2019, DAC 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781450367257
DOIs
StatePublished - Jun 2 2019
Event56th Annual Design Automation Conference, DAC 2019 - Las Vegas, United States
Duration: Jun 2 2019Jun 6 2019

Publication series

NameProceedings - Design Automation Conference
ISSN (Print)0738-100X

Conference

Conference56th Annual Design Automation Conference, DAC 2019
CountryUnited States
CityLas Vegas
Period6/2/196/6/19

Fingerprint

Accelerator
Energy Efficient
Particle accelerators
Neural Networks
Accumulate
Error Resilience
Timing
Multiplication
Filter
Unit
Voltage
Scaling
Error Recovery
Process Variation
Network layers
Taylor Expansion
Error Probability
Energy Saving
Chemical reactions
Injection

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Control and Systems Engineering
  • Electrical and Electronic Engineering
  • Modeling and Simulation

Cite this

Choi, W., Shin, D., Park, J., & Ghosh, S. (2019). Sensitivity based error resilient techniques for energy efficient deep neural network accelerators. In Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019 [a204] (Proceedings - Design Automation Conference). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1145/3316781.3317908
Choi, Wonseok ; Shin, Dongyeob ; Park, Jongsun ; Ghosh, Swaroop. / Sensitivity based error resilient techniques for energy efficient deep neural network accelerators. Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019. Institute of Electrical and Electronics Engineers Inc., 2019. (Proceedings - Design Automation Conference).
@inproceedings{6e52ca93182c42e9a70b96b69a31819e,
title = "Sensitivity based error resilient techniques for energy efficient deep neural network accelerators",
abstract = "With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we propose novel error resilient techniques to enable aggressive voltage scaling by exploiting different amount of error resilience (sensitivity) with respect to DNN layers, filters, and channels. First, to rapidly evaluate filter/channel-level weight sensitivities of large scale DNNs, first-order Taylor expansion is used, which accurately approximates weight sensitivity from actual error injection simulation. With measured timing error probability of each multiply-accumulate (MAC) units considering process variations, the sensitivity variation among filter weights can be leveraged to design DNN accelerator, such that the computations with more sensitive weights are assigned to more robust MAC units, while those with less sensitive weights are assigned to less robust MAC units. Based on post-synthesis timing simulations, 51{\%} energy savings has been achieved with CIFAR-10 dataset using VGG-9 compared to state-of-the-art timing error recovery technique with the same constraint of 3{\%} accuracy loss.",
author = "Wonseok Choi and Dongyeob Shin and Jongsun Park and Swaroop Ghosh",
year = "2019",
month = "6",
day = "2",
doi = "10.1145/3316781.3317908",
language = "English (US)",
series = "Proceedings - Design Automation Conference",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019",
address = "United States",

}

Choi, W, Shin, D, Park, J & Ghosh, S 2019, Sensitivity based error resilient techniques for energy efficient deep neural network accelerators. in Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019., a204, Proceedings - Design Automation Conference, Institute of Electrical and Electronics Engineers Inc., 56th Annual Design Automation Conference, DAC 2019, Las Vegas, United States, 6/2/19. https://doi.org/10.1145/3316781.3317908

Sensitivity based error resilient techniques for energy efficient deep neural network accelerators. / Choi, Wonseok; Shin, Dongyeob; Park, Jongsun; Ghosh, Swaroop.

Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019. Institute of Electrical and Electronics Engineers Inc., 2019. a204 (Proceedings - Design Automation Conference).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Sensitivity based error resilient techniques for energy efficient deep neural network accelerators

AU - Choi, Wonseok

AU - Shin, Dongyeob

AU - Park, Jongsun

AU - Ghosh, Swaroop

PY - 2019/6/2

Y1 - 2019/6/2

N2 - With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we propose novel error resilient techniques to enable aggressive voltage scaling by exploiting different amount of error resilience (sensitivity) with respect to DNN layers, filters, and channels. First, to rapidly evaluate filter/channel-level weight sensitivities of large scale DNNs, first-order Taylor expansion is used, which accurately approximates weight sensitivity from actual error injection simulation. With measured timing error probability of each multiply-accumulate (MAC) units considering process variations, the sensitivity variation among filter weights can be leveraged to design DNN accelerator, such that the computations with more sensitive weights are assigned to more robust MAC units, while those with less sensitive weights are assigned to less robust MAC units. Based on post-synthesis timing simulations, 51% energy savings has been achieved with CIFAR-10 dataset using VGG-9 compared to state-of-the-art timing error recovery technique with the same constraint of 3% accuracy loss.

AB - With inherent algorithmic error resilience of deep neural networks (DNNs), supply voltage scaling could be a promising technique for energy efficient DNN accelerator design. In this paper, we propose novel error resilient techniques to enable aggressive voltage scaling by exploiting different amount of error resilience (sensitivity) with respect to DNN layers, filters, and channels. First, to rapidly evaluate filter/channel-level weight sensitivities of large scale DNNs, first-order Taylor expansion is used, which accurately approximates weight sensitivity from actual error injection simulation. With measured timing error probability of each multiply-accumulate (MAC) units considering process variations, the sensitivity variation among filter weights can be leveraged to design DNN accelerator, such that the computations with more sensitive weights are assigned to more robust MAC units, while those with less sensitive weights are assigned to less robust MAC units. Based on post-synthesis timing simulations, 51% energy savings has been achieved with CIFAR-10 dataset using VGG-9 compared to state-of-the-art timing error recovery technique with the same constraint of 3% accuracy loss.

UR - http://www.scopus.com/inward/record.url?scp=85067800054&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067800054&partnerID=8YFLogxK

U2 - 10.1145/3316781.3317908

DO - 10.1145/3316781.3317908

M3 - Conference contribution

T3 - Proceedings - Design Automation Conference

BT - Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Choi W, Shin D, Park J, Ghosh S. Sensitivity based error resilient techniques for energy efficient deep neural network accelerators. In Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019. Institute of Electrical and Electronics Engineers Inc. 2019. a204. (Proceedings - Design Automation Conference). https://doi.org/10.1145/3316781.3317908