Constructing deterministic finite-state automata in recurrent neural networks

Christian W. Omlin, C. Lee Giles

Research output: Contribution to journalArticle

91 Citations (Scopus)

Abstract

Recurrent neural networks that are trained to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidal discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n states and m input alphabet symbols, the constructive algorithm generates a "programmed" neural network with O(n) neurons and O(mn) weights. We compare our algorithm to other methods proposed in the literature.

Original languageEnglish (US)
Pages (from-to)937-972
Number of pages36
JournalJournal of the ACM
Volume43
Issue number6
DOIs
StatePublished - Jan 1 1996

Fingerprint

Recurrent neural networks
Finite automata
Neural networks
Neurons
Topology

All Science Journal Classification (ASJC) codes

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Hardware and Architecture
  • Artificial Intelligence

Cite this

@article{95606f0784504e3d9238e36649e3073a,
title = "Constructing deterministic finite-state automata in recurrent neural networks",
abstract = "Recurrent neural networks that are trained to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidal discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n states and m input alphabet symbols, the constructive algorithm generates a {"}programmed{"} neural network with O(n) neurons and O(mn) weights. We compare our algorithm to other methods proposed in the literature.",
author = "Omlin, {Christian W.} and Giles, {C. Lee}",
year = "1996",
month = "1",
day = "1",
doi = "10.1145/235809.235811",
language = "English (US)",
volume = "43",
pages = "937--972",
journal = "Journal of the ACM",
issn = "0004-5411",
publisher = "Association for Computing Machinery (ACM)",
number = "6",

}

Constructing deterministic finite-state automata in recurrent neural networks. / Omlin, Christian W.; Giles, C. Lee.

In: Journal of the ACM, Vol. 43, No. 6, 01.01.1996, p. 937-972.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Constructing deterministic finite-state automata in recurrent neural networks

AU - Omlin, Christian W.

AU - Giles, C. Lee

PY - 1996/1/1

Y1 - 1996/1/1

N2 - Recurrent neural networks that are trained to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidal discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n states and m input alphabet symbols, the constructive algorithm generates a "programmed" neural network with O(n) neurons and O(mn) weights. We compare our algorithm to other methods proposed in the literature.

AB - Recurrent neural networks that are trained to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidal discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n states and m input alphabet symbols, the constructive algorithm generates a "programmed" neural network with O(n) neurons and O(mn) weights. We compare our algorithm to other methods proposed in the literature.

UR - http://www.scopus.com/inward/record.url?scp=0030286473&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030286473&partnerID=8YFLogxK

U2 - 10.1145/235809.235811

DO - 10.1145/235809.235811

M3 - Article

VL - 43

SP - 937

EP - 972

JO - Journal of the ACM

JF - Journal of the ACM

SN - 0004-5411

IS - 6

ER -