Empirical comparison of flat-spot elimination techniques in back-propagation networks

Rajesh Parekh, Karthik Balakrishnan, Vasant Honavar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Citations (Scopus)

Abstract

Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
EditorsMarry Lou Padgett
PublisherPubl by Society of Photo-Optical Instrumentation Engineers
Pages55-60
Number of pages6
Volume1721
ISBN (Print)1565550072
StatePublished - 1993
EventProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense - Alabama, AL, USA
Duration: Feb 10 1992Feb 12 1992

Other

OtherProceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense
CityAlabama, AL, USA
Period2/10/922/12/92

Fingerprint

Backpropagation
learning
elimination
Chemical activation
activation
self organizing systems
output
complement
Learning algorithms
seats
marking
Learning systems
Multilayers
education
Derivatives
Neural networks
evaluation

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering
  • Condensed Matter Physics

Cite this

Parekh, R., Balakrishnan, K., & Honavar, V. (1993). Empirical comparison of flat-spot elimination techniques in back-propagation networks. In M. L. Padgett (Ed.), Proceedings of SPIE - The International Society for Optical Engineering (Vol. 1721, pp. 55-60). Publ by Society of Photo-Optical Instrumentation Engineers.
Parekh, Rajesh ; Balakrishnan, Karthik ; Honavar, Vasant. / Empirical comparison of flat-spot elimination techniques in back-propagation networks. Proceedings of SPIE - The International Society for Optical Engineering. editor / Marry Lou Padgett. Vol. 1721 Publ by Society of Photo-Optical Instrumentation Engineers, 1993. pp. 55-60
@inproceedings{24815d3899134014ad222d4dc47ad7ec,
title = "Empirical comparison of flat-spot elimination techniques in back-propagation networks",
abstract = "Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.",
author = "Rajesh Parekh and Karthik Balakrishnan and Vasant Honavar",
year = "1993",
language = "English (US)",
isbn = "1565550072",
volume = "1721",
pages = "55--60",
editor = "Padgett, {Marry Lou}",
booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",
publisher = "Publ by Society of Photo-Optical Instrumentation Engineers",

}

Parekh, R, Balakrishnan, K & Honavar, V 1993, Empirical comparison of flat-spot elimination techniques in back-propagation networks. in ML Padgett (ed.), Proceedings of SPIE - The International Society for Optical Engineering. vol. 1721, Publ by Society of Photo-Optical Instrumentation Engineers, pp. 55-60, Proceedings of the 3rd Workshop on Neural Networks: Academic/Industrial/NASA/Defense, Alabama, AL, USA, 2/10/92.

Empirical comparison of flat-spot elimination techniques in back-propagation networks. / Parekh, Rajesh; Balakrishnan, Karthik; Honavar, Vasant.

Proceedings of SPIE - The International Society for Optical Engineering. ed. / Marry Lou Padgett. Vol. 1721 Publ by Society of Photo-Optical Instrumentation Engineers, 1993. p. 55-60.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Empirical comparison of flat-spot elimination techniques in back-propagation networks

AU - Parekh, Rajesh

AU - Balakrishnan, Karthik

AU - Honavar, Vasant

PY - 1993

Y1 - 1993

N2 - Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.

AB - Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.

UR - http://www.scopus.com/inward/record.url?scp=0027878882&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0027878882&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0027878882

SN - 1565550072

VL - 1721

SP - 55

EP - 60

BT - Proceedings of SPIE - The International Society for Optical Engineering

A2 - Padgett, Marry Lou

PB - Publ by Society of Photo-Optical Instrumentation Engineers

ER -

Parekh R, Balakrishnan K, Honavar V. Empirical comparison of flat-spot elimination techniques in back-propagation networks. In Padgett ML, editor, Proceedings of SPIE - The International Society for Optical Engineering. Vol. 1721. Publ by Society of Photo-Optical Instrumentation Engineers. 1993. p. 55-60