Recurrent neural networks, hidden Markov models and stochastic grammars

G. Z. Sun, H. H. Chen, Y. C. Lee, C. L. Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

A discussion is presented of the advantage of using a linear recurrent network to encode and recognize sequential data. The hidden Markov model (HMM) is shown to be a special case of such linear recurrent second-order neural networks. The Baum-Welch reestimation formula, which has proved very useful in training HMM, can also be used to learn a linear recurrent network. As an example, a network has successfully learned the stochastic Reber grammar with only a few hundred sample strings in about 14 iterations. The relative merits and limitations of the Baum-Welch optimal ascent algorithm in comparison with the error correction-gradient descent-learning algorithm are discussed.

Original languageEnglish (US)
Title of host publication90 Int Jt Conf Neural Networks IJCNN 90
PublisherPubl by IEEE
Pages729-734
Number of pages6
StatePublished - 1990
Event1990 International Joint Conference on Neural Networks - IJCNN 90 - San Diego, CA, USA
Duration: Jun 17 1990Jun 21 1990

Other

Other1990 International Joint Conference on Neural Networks - IJCNN 90
CitySan Diego, CA, USA
Period6/17/906/21/90

Fingerprint

Recurrent neural networks
Hidden Markov models
Error correction
Learning algorithms
Neural networks

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Sun, G. Z., Chen, H. H., Lee, Y. C., & Giles, C. L. (1990). Recurrent neural networks, hidden Markov models and stochastic grammars. In 90 Int Jt Conf Neural Networks IJCNN 90 (pp. 729-734). Publ by IEEE.
Sun, G. Z. ; Chen, H. H. ; Lee, Y. C. ; Giles, C. L. / Recurrent neural networks, hidden Markov models and stochastic grammars. 90 Int Jt Conf Neural Networks IJCNN 90. Publ by IEEE, 1990. pp. 729-734
@inproceedings{acb1271a06524f5dafe80c6eda10d18b,
title = "Recurrent neural networks, hidden Markov models and stochastic grammars",
abstract = "A discussion is presented of the advantage of using a linear recurrent network to encode and recognize sequential data. The hidden Markov model (HMM) is shown to be a special case of such linear recurrent second-order neural networks. The Baum-Welch reestimation formula, which has proved very useful in training HMM, can also be used to learn a linear recurrent network. As an example, a network has successfully learned the stochastic Reber grammar with only a few hundred sample strings in about 14 iterations. The relative merits and limitations of the Baum-Welch optimal ascent algorithm in comparison with the error correction-gradient descent-learning algorithm are discussed.",
author = "Sun, {G. Z.} and Chen, {H. H.} and Lee, {Y. C.} and Giles, {C. L.}",
year = "1990",
language = "English (US)",
pages = "729--734",
booktitle = "90 Int Jt Conf Neural Networks IJCNN 90",
publisher = "Publ by IEEE",

}

Sun, GZ, Chen, HH, Lee, YC & Giles, CL 1990, Recurrent neural networks, hidden Markov models and stochastic grammars. in 90 Int Jt Conf Neural Networks IJCNN 90. Publ by IEEE, pp. 729-734, 1990 International Joint Conference on Neural Networks - IJCNN 90, San Diego, CA, USA, 6/17/90.

Recurrent neural networks, hidden Markov models and stochastic grammars. / Sun, G. Z.; Chen, H. H.; Lee, Y. C.; Giles, C. L.

90 Int Jt Conf Neural Networks IJCNN 90. Publ by IEEE, 1990. p. 729-734.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Recurrent neural networks, hidden Markov models and stochastic grammars

AU - Sun, G. Z.

AU - Chen, H. H.

AU - Lee, Y. C.

AU - Giles, C. L.

PY - 1990

Y1 - 1990

N2 - A discussion is presented of the advantage of using a linear recurrent network to encode and recognize sequential data. The hidden Markov model (HMM) is shown to be a special case of such linear recurrent second-order neural networks. The Baum-Welch reestimation formula, which has proved very useful in training HMM, can also be used to learn a linear recurrent network. As an example, a network has successfully learned the stochastic Reber grammar with only a few hundred sample strings in about 14 iterations. The relative merits and limitations of the Baum-Welch optimal ascent algorithm in comparison with the error correction-gradient descent-learning algorithm are discussed.

AB - A discussion is presented of the advantage of using a linear recurrent network to encode and recognize sequential data. The hidden Markov model (HMM) is shown to be a special case of such linear recurrent second-order neural networks. The Baum-Welch reestimation formula, which has proved very useful in training HMM, can also be used to learn a linear recurrent network. As an example, a network has successfully learned the stochastic Reber grammar with only a few hundred sample strings in about 14 iterations. The relative merits and limitations of the Baum-Welch optimal ascent algorithm in comparison with the error correction-gradient descent-learning algorithm are discussed.

UR - http://www.scopus.com/inward/record.url?scp=0025547722&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0025547722&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0025547722

SP - 729

EP - 734

BT - 90 Int Jt Conf Neural Networks IJCNN 90

PB - Publ by IEEE

ER -

Sun GZ, Chen HH, Lee YC, Giles CL. Recurrent neural networks, hidden Markov models and stochastic grammars. In 90 Int Jt Conf Neural Networks IJCNN 90. Publ by IEEE. 1990. p. 729-734