Empirical analysis of the expected source values rule

Richard Spartz, Vasant Honavar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Despite its notoriously slow learning time, back-propagation (BP) is one of the most widely used neural network training algorithms. Two major reasons for this slow convergence are the step size problem and the flat spot problem [Fahlman, 1988]. In [Samad, 1991] a simple modification, the expected source values (ESV) rule, is proposed for speeding up the BP algorithm. We have extended the ESV rule by coupling it with a flat-spot removal strategy presented in [Fahlman, 1988], as well as incorporating a momentum term to combat the step size problem. The resulting rule has shown dramatically improved learning time over standard BP, measured in training epochs. Two versions of the ESV modification are mentioned in [Samad. 1991], on-demand and up-front, but simulation results are given mostly for the on-demand case. Our results indicate that the up-front version works somewhat better than the on-demand version in terms of learning speed. We have also analyzed the interactions between the three modifications as they are used in various combinations.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
EditorsMarry Lou Padgett
PublisherPubl by Society of Photo-Optical Instrumentation Engineers
Pages95-100
Number of pages6
ISBN (Print)1565550072
StatePublished - Dec 1 1993
EventProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense - Alabama, AL, USA
Duration: Feb 10 1992Feb 12 1992

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
Volume1721
ISSN (Print)0277-786X

Other

OtherProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense
CityAlabama, AL, USA
Period2/10/922/12/92

Fingerprint

Backpropagation
learning
Backpropagation algorithms
education
Momentum
combat
Neural networks
time measurement
momentum
simulation
interactions

All Science Journal Classification (ASJC) codes

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering

Cite this

Spartz, R., & Honavar, V. (1993). Empirical analysis of the expected source values rule. In M. L. Padgett (Ed.), Proceedings of SPIE - The International Society for Optical Engineering (pp. 95-100). (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 1721). Publ by Society of Photo-Optical Instrumentation Engineers.
Spartz, Richard ; Honavar, Vasant. / Empirical analysis of the expected source values rule. Proceedings of SPIE - The International Society for Optical Engineering. editor / Marry Lou Padgett. Publ by Society of Photo-Optical Instrumentation Engineers, 1993. pp. 95-100 (Proceedings of SPIE - The International Society for Optical Engineering).
@inproceedings{2741fff484864a68b240ae9715d64a9e,
title = "Empirical analysis of the expected source values rule",
abstract = "Despite its notoriously slow learning time, back-propagation (BP) is one of the most widely used neural network training algorithms. Two major reasons for this slow convergence are the step size problem and the flat spot problem [Fahlman, 1988]. In [Samad, 1991] a simple modification, the expected source values (ESV) rule, is proposed for speeding up the BP algorithm. We have extended the ESV rule by coupling it with a flat-spot removal strategy presented in [Fahlman, 1988], as well as incorporating a momentum term to combat the step size problem. The resulting rule has shown dramatically improved learning time over standard BP, measured in training epochs. Two versions of the ESV modification are mentioned in [Samad. 1991], on-demand and up-front, but simulation results are given mostly for the on-demand case. Our results indicate that the up-front version works somewhat better than the on-demand version in terms of learning speed. We have also analyzed the interactions between the three modifications as they are used in various combinations.",
author = "Richard Spartz and Vasant Honavar",
year = "1993",
month = "12",
day = "1",
language = "English (US)",
isbn = "1565550072",
series = "Proceedings of SPIE - The International Society for Optical Engineering",
publisher = "Publ by Society of Photo-Optical Instrumentation Engineers",
pages = "95--100",
editor = "Padgett, {Marry Lou}",
booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",

}

Spartz, R & Honavar, V 1993, Empirical analysis of the expected source values rule. in ML Padgett (ed.), Proceedings of SPIE - The International Society for Optical Engineering. Proceedings of SPIE - The International Society for Optical Engineering, vol. 1721, Publ by Society of Photo-Optical Instrumentation Engineers, pp. 95-100, Proceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense, Alabama, AL, USA, 2/10/92.

Empirical analysis of the expected source values rule. / Spartz, Richard; Honavar, Vasant.

Proceedings of SPIE - The International Society for Optical Engineering. ed. / Marry Lou Padgett. Publ by Society of Photo-Optical Instrumentation Engineers, 1993. p. 95-100 (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 1721).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Empirical analysis of the expected source values rule

AU - Spartz, Richard

AU - Honavar, Vasant

PY - 1993/12/1

Y1 - 1993/12/1

N2 - Despite its notoriously slow learning time, back-propagation (BP) is one of the most widely used neural network training algorithms. Two major reasons for this slow convergence are the step size problem and the flat spot problem [Fahlman, 1988]. In [Samad, 1991] a simple modification, the expected source values (ESV) rule, is proposed for speeding up the BP algorithm. We have extended the ESV rule by coupling it with a flat-spot removal strategy presented in [Fahlman, 1988], as well as incorporating a momentum term to combat the step size problem. The resulting rule has shown dramatically improved learning time over standard BP, measured in training epochs. Two versions of the ESV modification are mentioned in [Samad. 1991], on-demand and up-front, but simulation results are given mostly for the on-demand case. Our results indicate that the up-front version works somewhat better than the on-demand version in terms of learning speed. We have also analyzed the interactions between the three modifications as they are used in various combinations.

AB - Despite its notoriously slow learning time, back-propagation (BP) is one of the most widely used neural network training algorithms. Two major reasons for this slow convergence are the step size problem and the flat spot problem [Fahlman, 1988]. In [Samad, 1991] a simple modification, the expected source values (ESV) rule, is proposed for speeding up the BP algorithm. We have extended the ESV rule by coupling it with a flat-spot removal strategy presented in [Fahlman, 1988], as well as incorporating a momentum term to combat the step size problem. The resulting rule has shown dramatically improved learning time over standard BP, measured in training epochs. Two versions of the ESV modification are mentioned in [Samad. 1991], on-demand and up-front, but simulation results are given mostly for the on-demand case. Our results indicate that the up-front version works somewhat better than the on-demand version in terms of learning speed. We have also analyzed the interactions between the three modifications as they are used in various combinations.

UR - http://www.scopus.com/inward/record.url?scp=0027871691&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0027871691&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0027871691

SN - 1565550072

T3 - Proceedings of SPIE - The International Society for Optical Engineering

SP - 95

EP - 100

BT - Proceedings of SPIE - The International Society for Optical Engineering

A2 - Padgett, Marry Lou

PB - Publ by Society of Photo-Optical Instrumentation Engineers

ER -

Spartz R, Honavar V. Empirical analysis of the expected source values rule. In Padgett ML, editor, Proceedings of SPIE - The International Society for Optical Engineering. Publ by Society of Photo-Optical Instrumentation Engineers. 1993. p. 95-100. (Proceedings of SPIE - The International Society for Optical Engineering).