State-space representations of deep neural networks

Michael Hauser, Sean Gunn, Samer Saab, Asok Ray

    Research output: Contribution to journalArticle

    Abstract

    This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

    Original languageEnglish (US)
    Pages (from-to)538-554
    Number of pages17
    JournalNeural Computation
    Volume31
    Issue number3
    DOIs
    StatePublished - Mar 1 2019

    Fingerprint

    Neural Networks
    Equations
    Layer
    Simulation
    Letters
    Dynamical Systems
    Efficacy
    Algebra
    Experiment

    All Science Journal Classification (ASJC) codes

    • Arts and Humanities (miscellaneous)
    • Cognitive Neuroscience

    Cite this

    Hauser, Michael ; Gunn, Sean ; Saab, Samer ; Ray, Asok. / State-space representations of deep neural networks. In: Neural Computation. 2019 ; Vol. 31, No. 3. pp. 538-554.
    @article{d590e03fb3d94b0392de0ec8c6d3df81,
    title = "State-space representations of deep neural networks",
    abstract = "This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.",
    author = "Michael Hauser and Sean Gunn and Samer Saab and Asok Ray",
    year = "2019",
    month = "3",
    day = "1",
    doi = "10.1162/neco_a_01165",
    language = "English (US)",
    volume = "31",
    pages = "538--554",
    journal = "Neural Computation",
    issn = "0899-7667",
    publisher = "MIT Press Journals",
    number = "3",

    }

    State-space representations of deep neural networks. / Hauser, Michael; Gunn, Sean; Saab, Samer; Ray, Asok.

    In: Neural Computation, Vol. 31, No. 3, 01.03.2019, p. 538-554.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - State-space representations of deep neural networks

    AU - Hauser, Michael

    AU - Gunn, Sean

    AU - Saab, Samer

    AU - Ray, Asok

    PY - 2019/3/1

    Y1 - 2019/3/1

    N2 - This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

    AB - This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of k-many skip connections into network architectures, such as residual networks and additive dense networks, defines kth order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general kth order additive dense networks, where the concatenation operation is replaced by addition, as well as kth order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing kth order smoothness on network architectures with d-many nodes per layer increases the state-space dimension by a multiple of k, and so the effective embedding dimension of the data manifold by the neural network is k·d-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of k2 when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

    UR - http://www.scopus.com/inward/record.url?scp=85061596492&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85061596492&partnerID=8YFLogxK

    U2 - 10.1162/neco_a_01165

    DO - 10.1162/neco_a_01165

    M3 - Article

    C2 - 30645180

    AN - SCOPUS:85061596492

    VL - 31

    SP - 538

    EP - 554

    JO - Neural Computation

    JF - Neural Computation

    SN - 0899-7667

    IS - 3

    ER -