Constructive Learning of Recurrent Neural Networks: Limitations of Recurrent Casade Correlation and a Simple Solution

C. Lee Giles, Dong Chen, Guo Zheng Sun, Hsing Hen Chen, Yee Chung Lee, Mark W. Goudreau

Research output: Contribution to journalArticle

51 Citations (Scopus)

Abstract

It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a “preliminary" approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.

Original languageEnglish (US)
Pages (from-to)829-836
Number of pages8
JournalIEEE Transactions on Neural Networks
Volume6
Issue number4
DOIs
StatePublished - Jul 1995

Fingerprint

Recurrent neural networks
Neurons
Correlation methods
Finite automata
Chemical activation
Topology
Neural networks

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Cite this

Giles, C. Lee ; Chen, Dong ; Sun, Guo Zheng ; Chen, Hsing Hen ; Lee, Yee Chung ; Goudreau, Mark W. / Constructive Learning of Recurrent Neural Networks : Limitations of Recurrent Casade Correlation and a Simple Solution. In: IEEE Transactions on Neural Networks. 1995 ; Vol. 6, No. 4. pp. 829-836.
@article{ee47d5fd27134577b74615d58e6b3d5e,
title = "Constructive Learning of Recurrent Neural Networks: Limitations of Recurrent Casade Correlation and a Simple Solution",
abstract = "It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a “preliminary{"} approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.",
author = "Giles, {C. Lee} and Dong Chen and Sun, {Guo Zheng} and Chen, {Hsing Hen} and Lee, {Yee Chung} and Goudreau, {Mark W.}",
year = "1995",
month = "7",
doi = "10.1109/72.392247",
language = "English (US)",
volume = "6",
pages = "829--836",
journal = "IEEE Transactions on Neural Networks and Learning Systems",
issn = "2162-237X",
publisher = "IEEE Computational Intelligence Society",
number = "4",

}

Constructive Learning of Recurrent Neural Networks : Limitations of Recurrent Casade Correlation and a Simple Solution. / Giles, C. Lee; Chen, Dong; Sun, Guo Zheng; Chen, Hsing Hen; Lee, Yee Chung; Goudreau, Mark W.

In: IEEE Transactions on Neural Networks, Vol. 6, No. 4, 07.1995, p. 829-836.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Constructive Learning of Recurrent Neural Networks

T2 - Limitations of Recurrent Casade Correlation and a Simple Solution

AU - Giles, C. Lee

AU - Chen, Dong

AU - Sun, Guo Zheng

AU - Chen, Hsing Hen

AU - Lee, Yee Chung

AU - Goudreau, Mark W.

PY - 1995/7

Y1 - 1995/7

N2 - It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a “preliminary" approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.

AB - It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a “preliminary" approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.

UR - http://www.scopus.com/inward/record.url?scp=0029341578&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029341578&partnerID=8YFLogxK

U2 - 10.1109/72.392247

DO - 10.1109/72.392247

M3 - Article

C2 - 18263373

AN - SCOPUS:0029341578

VL - 6

SP - 829

EP - 836

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

SN - 2162-237X

IS - 4

ER -