Pruning Recurrent Neural Networks for Improved Generalization Performance

C. Lee Giles, Christian W. Omlin

Research output: Contribution to journalArticlepeer-review

61 Scopus citations

Abstract

Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.

Original languageEnglish (US)
Pages (from-to)848-851
Number of pages4
JournalIEEE Transactions on Neural Networks
Volume5
Issue number5
DOIs
StatePublished - Sep 1994

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Pruning Recurrent Neural Networks for Improved Generalization Performance'. Together they form a unique fingerprint.

Cite this