TraNNsformer: Clustered Pruning on Crossbar-based Architectures for Energy Efficient Neural Networks

Aayush Ankit, Timur Ibrayev, Abhronil Sengupta, Kaushik Roy

Research output: Contribution to journalArticle

Abstract

Implementation of Neuromorphic Systems using Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning han2015learning is a powerful technique to remove redundant connections for designing optimally connected (maximally sparse) DNNs. However, such pruning techniques induce irregular connections that are incoherent to the crossbar structure. Eventually they produce DNNs with highly inefficient hardware realizations (in terms of area and energy). In this work, we propose TraNNsformer -an integrated training framework that transforms DNNs to enable their efficient realization on MCA-based systems. TraNNsformer first prunes the connectivity matrix while forming clusters with the remaining connections. Subsequently, it retrains the network to fine tune the connections and reinforce the clusters. This is done iteratively to transform the original connectivity into an optimally pruned and maximally clustered mapping. We evaluated the proposed framework by transforming networks of different complexity based on Multi-Layer Perceptron (MLP) and Convolutional Neural Network (CNN) topologies on a wide range of datasets (MNIST, SVHN, CIFAR10, and ImageNet) and executing them on MCA-based systems to analyze the area and energy benefits. Without accuracy loss, TraNNsformer reduces the area (energy) consumption by 28% -55% (49% -67%) of MLP networks and by 28% -48% (3% -39%) of CNN networks with respect to the original network implementations. Compared to network pruning, TraNNsformer achieves 28% -49% (15% -29%) area (energy) savings for MLP networks and 20% -44% (1% -11%) area (energy) saving for CNN networks. Furthermore, TraNNsformer is a technology-aware framework that allows to map a given DNN to any MCA size permissible by the memristive technology for reliable operations.

Fingerprint

Neural networks
Multilayer neural networks
Energy conservation
Energy utilization
Topology
Deep neural networks
Hardware

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design
  • Electrical and Electronic Engineering

Cite this

@article{ccac50f16a234c7796e2adcc1ce11f42,
title = "TraNNsformer: Clustered Pruning on Crossbar-based Architectures for Energy Efficient Neural Networks",
abstract = "Implementation of Neuromorphic Systems using Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning han2015learning is a powerful technique to remove redundant connections for designing optimally connected (maximally sparse) DNNs. However, such pruning techniques induce irregular connections that are incoherent to the crossbar structure. Eventually they produce DNNs with highly inefficient hardware realizations (in terms of area and energy). In this work, we propose TraNNsformer -an integrated training framework that transforms DNNs to enable their efficient realization on MCA-based systems. TraNNsformer first prunes the connectivity matrix while forming clusters with the remaining connections. Subsequently, it retrains the network to fine tune the connections and reinforce the clusters. This is done iteratively to transform the original connectivity into an optimally pruned and maximally clustered mapping. We evaluated the proposed framework by transforming networks of different complexity based on Multi-Layer Perceptron (MLP) and Convolutional Neural Network (CNN) topologies on a wide range of datasets (MNIST, SVHN, CIFAR10, and ImageNet) and executing them on MCA-based systems to analyze the area and energy benefits. Without accuracy loss, TraNNsformer reduces the area (energy) consumption by 28{\%} -55{\%} (49{\%} -67{\%}) of MLP networks and by 28{\%} -48{\%} (3{\%} -39{\%}) of CNN networks with respect to the original network implementations. Compared to network pruning, TraNNsformer achieves 28{\%} -49{\%} (15{\%} -29{\%}) area (energy) savings for MLP networks and 20{\%} -44{\%} (1{\%} -11{\%}) area (energy) saving for CNN networks. Furthermore, TraNNsformer is a technology-aware framework that allows to map a given DNN to any MCA size permissible by the memristive technology for reliable operations.",
author = "Aayush Ankit and Timur Ibrayev and Abhronil Sengupta and Kaushik Roy",
year = "2019",
month = "1",
day = "1",
doi = "10.1109/TCAD.2019.2946820",
language = "English (US)",
journal = "IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems",
issn = "0278-0070",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - TraNNsformer

T2 - Clustered Pruning on Crossbar-based Architectures for Energy Efficient Neural Networks

AU - Ankit, Aayush

AU - Ibrayev, Timur

AU - Sengupta, Abhronil

AU - Roy, Kaushik

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Implementation of Neuromorphic Systems using Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning han2015learning is a powerful technique to remove redundant connections for designing optimally connected (maximally sparse) DNNs. However, such pruning techniques induce irregular connections that are incoherent to the crossbar structure. Eventually they produce DNNs with highly inefficient hardware realizations (in terms of area and energy). In this work, we propose TraNNsformer -an integrated training framework that transforms DNNs to enable their efficient realization on MCA-based systems. TraNNsformer first prunes the connectivity matrix while forming clusters with the remaining connections. Subsequently, it retrains the network to fine tune the connections and reinforce the clusters. This is done iteratively to transform the original connectivity into an optimally pruned and maximally clustered mapping. We evaluated the proposed framework by transforming networks of different complexity based on Multi-Layer Perceptron (MLP) and Convolutional Neural Network (CNN) topologies on a wide range of datasets (MNIST, SVHN, CIFAR10, and ImageNet) and executing them on MCA-based systems to analyze the area and energy benefits. Without accuracy loss, TraNNsformer reduces the area (energy) consumption by 28% -55% (49% -67%) of MLP networks and by 28% -48% (3% -39%) of CNN networks with respect to the original network implementations. Compared to network pruning, TraNNsformer achieves 28% -49% (15% -29%) area (energy) savings for MLP networks and 20% -44% (1% -11%) area (energy) saving for CNN networks. Furthermore, TraNNsformer is a technology-aware framework that allows to map a given DNN to any MCA size permissible by the memristive technology for reliable operations.

AB - Implementation of Neuromorphic Systems using Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning han2015learning is a powerful technique to remove redundant connections for designing optimally connected (maximally sparse) DNNs. However, such pruning techniques induce irregular connections that are incoherent to the crossbar structure. Eventually they produce DNNs with highly inefficient hardware realizations (in terms of area and energy). In this work, we propose TraNNsformer -an integrated training framework that transforms DNNs to enable their efficient realization on MCA-based systems. TraNNsformer first prunes the connectivity matrix while forming clusters with the remaining connections. Subsequently, it retrains the network to fine tune the connections and reinforce the clusters. This is done iteratively to transform the original connectivity into an optimally pruned and maximally clustered mapping. We evaluated the proposed framework by transforming networks of different complexity based on Multi-Layer Perceptron (MLP) and Convolutional Neural Network (CNN) topologies on a wide range of datasets (MNIST, SVHN, CIFAR10, and ImageNet) and executing them on MCA-based systems to analyze the area and energy benefits. Without accuracy loss, TraNNsformer reduces the area (energy) consumption by 28% -55% (49% -67%) of MLP networks and by 28% -48% (3% -39%) of CNN networks with respect to the original network implementations. Compared to network pruning, TraNNsformer achieves 28% -49% (15% -29%) area (energy) savings for MLP networks and 20% -44% (1% -11%) area (energy) saving for CNN networks. Furthermore, TraNNsformer is a technology-aware framework that allows to map a given DNN to any MCA size permissible by the memristive technology for reliable operations.

UR - http://www.scopus.com/inward/record.url?scp=85073570179&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073570179&partnerID=8YFLogxK

U2 - 10.1109/TCAD.2019.2946820

DO - 10.1109/TCAD.2019.2946820

M3 - Article

AN - SCOPUS:85073570179

JO - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

JF - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

SN - 0278-0070

ER -