TY - JOUR

T1 - State-space representations of deep neural networks

AU - Hauser, Michael

AU - Gunn, Sean

AU - Saab, Samer

AU - Ray, Asok

N1 - Funding Information:
S. S. has been supported by the Walker Fellowship from the Applied Research Laboratory at the Pennsylvania State University. The work reported here has been supported in part by the U.S. Air Force Office of Scientific Research under grants FA9550-15-1-0400 and FA9550-18-1-0135 in the area of dynamic data-driven application systems. Any opinions, findings, and conclusions or recommendations expressed in this letter are our own and do not necessarily reflect the views of the sponsoring agencies.
Publisher Copyright:
© 2019 Massachusetts Institute of Technology.

PY - 2019/3/1

Y1 - 2019/3/1

N2 - This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

AB - This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

UR - http://www.scopus.com/inward/record.url?scp=85061596492&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061596492&partnerID=8YFLogxK

U2 - 10.1162/neco_a_01165

DO - 10.1162/neco_a_01165

M3 - Article

C2 - 30645180

AN - SCOPUS:85061596492

SN - 0899-7667

VL - 31

SP - 538

EP - 554

JO - Neural Computation

JF - Neural Computation

IS - 3

ER -