Empirical comparison of flat-spot elimination techniques in back-propagation networks

Rajesh Parekh, Karthik Balakrishnan, Vasant Honavar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
EditorsMarry Lou Padgett
PublisherPubl by Society of Photo-Optical Instrumentation Engineers
Pages55-60
Number of pages6
Volume1721
ISBN (Print)1565550072
StatePublished - 1993
EventProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense - Alabama, AL, USA
Duration: Feb 10 1992Feb 12 1992

Other

OtherProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense
CityAlabama, AL, USA
Period2/10/922/12/92

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering
  • Condensed Matter Physics

Fingerprint Dive into the research topics of 'Empirical comparison of flat-spot elimination techniques in back-propagation networks'. Together they form a unique fingerprint.

  • Cite this

    Parekh, R., Balakrishnan, K., & Honavar, V. (1993). Empirical comparison of flat-spot elimination techniques in back-propagation networks. In M. L. Padgett (Ed.), Proceedings of SPIE - The International Society for Optical Engineering (Vol. 1721, pp. 55-60). Publ by Society of Photo-Optical Instrumentation Engineers.