Nonlinear system modeling with random matrices: Echo state networks revisited

Bai Zhang, David J. Miller, Yue Wang

    Research output: Contribution to journalArticlepeer-review

    46 Scopus citations

    Abstract

    Echo state networks (ESNs) are a novel form of recurrent neural networks (RNNs) that provide an efficient and powerful computational model approximating nonlinear dynamical systems. A unique feature of an ESN is that a large number of neurons (the 'reservoir') are used, whose synaptic connections are generated randomly, with only the connections from the reservoir to the output modified by learning. Why a large randomly generated fixed RNN gives such excellent performance in approximating nonlinear systems is still not well understood. In this brief, we apply random matrix theory to examine the properties of random reservoirs in ESNs under different topologies (sparse or fully connected) and connection weights (Bernoulli or Gaussian). We quantify the asymptotic gap between the scaling factor bounds for the necessary and sufficient conditions previously proposed for the echo state property. We then show that the state transition mapping is contractive with high probability when only the necessary condition is satisfied, which corroborates and thus analytically explains the observation that in practice one obtains echo states when the spectral radius of the reservoir weight matrix is smaller than 1.

    Original languageEnglish (US)
    Article number6105577
    Pages (from-to)175-182
    Number of pages8
    JournalIEEE Transactions on Neural Networks and Learning Systems
    Volume23
    Issue number1
    DOIs
    StatePublished - 2012

    All Science Journal Classification (ASJC) codes

    • Software
    • Computer Science Applications
    • Computer Networks and Communications
    • Artificial Intelligence

    Fingerprint Dive into the research topics of 'Nonlinear system modeling with random matrices: Echo state networks revisited'. Together they form a unique fingerprint.

    Cite this