Pruning recurrent neural networks for improved generalization performance

Christian W. Omlin, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

The experimental results in this paper demonstrate that a simple pruning/retraining method effectively improves the generalization performance of recurrent neural networks trained to recognize regular languages. The technique also permits the extraction of symbolic knowledge in the form of deterministic finite-state automata (DFA's) which are more consistent with the rules to be learned. Weight decay has also been shown to improve a network's generalization performance. Simulations with two small DFA's (≤10 states) and a large finite-memory machine (64 states) demonstrate that the performance improvement due to pruning/retraining is generally superior to the improvement due to training with weight decay. In addition, there is no need to guess a `good' decay rate.

Original languageEnglish (US)
Title of host publicationNeural Networks for Signal Processing - Proceedings of the IEEE Workshop
PublisherIEEE
Pages690-699
Number of pages10
StatePublished - 1994
EventProceedings of the 4th IEEE Workshop on Neural Networks for Signal Processing (NNSP'94) - Ermioni, GREECE
Duration: Sep 6 1994Sep 8 1994

Other

OtherProceedings of the 4th IEEE Workshop on Neural Networks for Signal Processing (NNSP'94)
CityErmioni, GREECE
Period9/6/949/8/94

Fingerprint

Recurrent neural networks
Finite automata
Formal languages
Network performance
Data storage equipment

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Software
  • Electrical and Electronic Engineering

Cite this

Omlin, C. W., & Giles, C. L. (1994). Pruning recurrent neural networks for improved generalization performance. In Neural Networks for Signal Processing - Proceedings of the IEEE Workshop (pp. 690-699). IEEE.
Omlin, Christian W. ; Giles, C. Lee. / Pruning recurrent neural networks for improved generalization performance. Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. IEEE, 1994. pp. 690-699
@inproceedings{9a5c55c058d044f5bab496a734fdd211,
title = "Pruning recurrent neural networks for improved generalization performance",
abstract = "The experimental results in this paper demonstrate that a simple pruning/retraining method effectively improves the generalization performance of recurrent neural networks trained to recognize regular languages. The technique also permits the extraction of symbolic knowledge in the form of deterministic finite-state automata (DFA's) which are more consistent with the rules to be learned. Weight decay has also been shown to improve a network's generalization performance. Simulations with two small DFA's (≤10 states) and a large finite-memory machine (64 states) demonstrate that the performance improvement due to pruning/retraining is generally superior to the improvement due to training with weight decay. In addition, there is no need to guess a `good' decay rate.",
author = "Omlin, {Christian W.} and Giles, {C. Lee}",
year = "1994",
language = "English (US)",
pages = "690--699",
booktitle = "Neural Networks for Signal Processing - Proceedings of the IEEE Workshop",
publisher = "IEEE",

}

Omlin, CW & Giles, CL 1994, Pruning recurrent neural networks for improved generalization performance. in Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. IEEE, pp. 690-699, Proceedings of the 4th IEEE Workshop on Neural Networks for Signal Processing (NNSP'94), Ermioni, GREECE, 9/6/94.

Pruning recurrent neural networks for improved generalization performance. / Omlin, Christian W.; Giles, C. Lee.

Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. IEEE, 1994. p. 690-699.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Pruning recurrent neural networks for improved generalization performance

AU - Omlin, Christian W.

AU - Giles, C. Lee

PY - 1994

Y1 - 1994

N2 - The experimental results in this paper demonstrate that a simple pruning/retraining method effectively improves the generalization performance of recurrent neural networks trained to recognize regular languages. The technique also permits the extraction of symbolic knowledge in the form of deterministic finite-state automata (DFA's) which are more consistent with the rules to be learned. Weight decay has also been shown to improve a network's generalization performance. Simulations with two small DFA's (≤10 states) and a large finite-memory machine (64 states) demonstrate that the performance improvement due to pruning/retraining is generally superior to the improvement due to training with weight decay. In addition, there is no need to guess a `good' decay rate.

AB - The experimental results in this paper demonstrate that a simple pruning/retraining method effectively improves the generalization performance of recurrent neural networks trained to recognize regular languages. The technique also permits the extraction of symbolic knowledge in the form of deterministic finite-state automata (DFA's) which are more consistent with the rules to be learned. Weight decay has also been shown to improve a network's generalization performance. Simulations with two small DFA's (≤10 states) and a large finite-memory machine (64 states) demonstrate that the performance improvement due to pruning/retraining is generally superior to the improvement due to training with weight decay. In addition, there is no need to guess a `good' decay rate.

UR - http://www.scopus.com/inward/record.url?scp=0028697762&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0028697762&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0028697762

SP - 690

EP - 699

BT - Neural Networks for Signal Processing - Proceedings of the IEEE Workshop

PB - IEEE

ER -

Omlin CW, Giles CL. Pruning recurrent neural networks for improved generalization performance. In Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. IEEE. 1994. p. 690-699