Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Massive memory requirements and disruptive I/O traffic in Convolutional Neural Networks (CNNs) restrict hardware/software optimization for training and inference tasks using larger networks despite their success in improving the intelligence of systems into which they are integrated. Interestingly, it has been observed that the early CNN layers in vision tasks consistently produce edge-like feature representations that can be mimicked by traditional vision algorithms. In this work, we demonstrate how to exploit this to create common edge-like features for sharing among multiple CNNs, and how to heuristically approximate them in a reduced dimensionality. In our proposed approximation, the feature space of three representative CNNs decreases by 1.6×-5.1×, and the size of training dataset is halved. As a result, we enhance both inference throughput and training speed by 2×, while providing accuracies that are still close to the original versions. We anticipate that this approach will lead to new opportunities in distributed intelligence systems, and the technique for redesigning CNN models based on dimensional reduction of feature space is orthogonal to and compatible with many other existing hardware/software optimizations.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages218-225
Number of pages8
ISBN (Electronic)9781538684771
DOIs
StatePublished - Jan 16 2019
Event36th International Conference on Computer Design, ICCD 2018 - Orlando, United States
Duration: Oct 7 2018Oct 10 2018

Publication series

NameProceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018

Conference

Conference36th International Conference on Computer Design, ICCD 2018
CountryUnited States
CityOrlando
Period10/7/1810/10/18

Fingerprint

Neural networks
Hardware
Network layers
Throughput
Data storage equipment

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Safety, Risk, Reliability and Quality

Cite this

Choi, J., Sampson, J., & Narayanan, V. (2019). Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems. In Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018 (pp. 218-225). [8615691] (Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCD.2018.00041
Choi, Jinhang ; Sampson, Jack ; Narayanan, Vijaykrishnan. / Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems. Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 218-225 (Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018).
@inproceedings{c8a0f10e19d644f6b2b1b511563e692e,
title = "Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems",
abstract = "Massive memory requirements and disruptive I/O traffic in Convolutional Neural Networks (CNNs) restrict hardware/software optimization for training and inference tasks using larger networks despite their success in improving the intelligence of systems into which they are integrated. Interestingly, it has been observed that the early CNN layers in vision tasks consistently produce edge-like feature representations that can be mimicked by traditional vision algorithms. In this work, we demonstrate how to exploit this to create common edge-like features for sharing among multiple CNNs, and how to heuristically approximate them in a reduced dimensionality. In our proposed approximation, the feature space of three representative CNNs decreases by 1.6×-5.1×, and the size of training dataset is halved. As a result, we enhance both inference throughput and training speed by 2×, while providing accuracies that are still close to the original versions. We anticipate that this approach will lead to new opportunities in distributed intelligence systems, and the technique for redesigning CNN models based on dimensional reduction of feature space is orthogonal to and compatible with many other existing hardware/software optimizations.",
author = "Jinhang Choi and Jack Sampson and Vijaykrishnan Narayanan",
year = "2019",
month = "1",
day = "16",
doi = "10.1109/ICCD.2018.00041",
language = "English (US)",
series = "Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "218--225",
booktitle = "Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018",
address = "United States",

}

Choi, J, Sampson, J & Narayanan, V 2019, Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems. in Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018., 8615691, Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018, Institute of Electrical and Electronics Engineers Inc., pp. 218-225, 36th International Conference on Computer Design, ICCD 2018, Orlando, United States, 10/7/18. https://doi.org/10.1109/ICCD.2018.00041

Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems. / Choi, Jinhang; Sampson, Jack; Narayanan, Vijaykrishnan.

Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 218-225 8615691 (Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems

AU - Choi, Jinhang

AU - Sampson, Jack

AU - Narayanan, Vijaykrishnan

PY - 2019/1/16

Y1 - 2019/1/16

N2 - Massive memory requirements and disruptive I/O traffic in Convolutional Neural Networks (CNNs) restrict hardware/software optimization for training and inference tasks using larger networks despite their success in improving the intelligence of systems into which they are integrated. Interestingly, it has been observed that the early CNN layers in vision tasks consistently produce edge-like feature representations that can be mimicked by traditional vision algorithms. In this work, we demonstrate how to exploit this to create common edge-like features for sharing among multiple CNNs, and how to heuristically approximate them in a reduced dimensionality. In our proposed approximation, the feature space of three representative CNNs decreases by 1.6×-5.1×, and the size of training dataset is halved. As a result, we enhance both inference throughput and training speed by 2×, while providing accuracies that are still close to the original versions. We anticipate that this approach will lead to new opportunities in distributed intelligence systems, and the technique for redesigning CNN models based on dimensional reduction of feature space is orthogonal to and compatible with many other existing hardware/software optimizations.

AB - Massive memory requirements and disruptive I/O traffic in Convolutional Neural Networks (CNNs) restrict hardware/software optimization for training and inference tasks using larger networks despite their success in improving the intelligence of systems into which they are integrated. Interestingly, it has been observed that the early CNN layers in vision tasks consistently produce edge-like feature representations that can be mimicked by traditional vision algorithms. In this work, we demonstrate how to exploit this to create common edge-like features for sharing among multiple CNNs, and how to heuristically approximate them in a reduced dimensionality. In our proposed approximation, the feature space of three representative CNNs decreases by 1.6×-5.1×, and the size of training dataset is halved. As a result, we enhance both inference throughput and training speed by 2×, while providing accuracies that are still close to the original versions. We anticipate that this approach will lead to new opportunities in distributed intelligence systems, and the technique for redesigning CNN models based on dimensional reduction of feature space is orthogonal to and compatible with many other existing hardware/software optimizations.

UR - http://www.scopus.com/inward/record.url?scp=85062208772&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062208772&partnerID=8YFLogxK

U2 - 10.1109/ICCD.2018.00041

DO - 10.1109/ICCD.2018.00041

M3 - Conference contribution

AN - SCOPUS:85062208772

T3 - Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018

SP - 218

EP - 225

BT - Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Choi J, Sampson J, Narayanan V. Heuristic Approximation of Early-Stage CNN Data Representation for Vision Intelligence Systems. In Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 218-225. 8615691. (Proceedings - 2018 IEEE 36th International Conference on Computer Design, ICCD 2018). https://doi.org/10.1109/ICCD.2018.00041