LMS learning algorithms: Misconceptions and new results on convergence

Zi Qin Wang, Michael T. Manry, Jeffrey L. Schiano

    Research output: Contribution to journalArticlepeer-review

    56 Scopus citations


    The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.

    Original languageEnglish (US)
    Pages (from-to)47-56
    Number of pages10
    JournalIEEE Transactions on Neural Networks
    Issue number1
    StatePublished - Jan 2000

    All Science Journal Classification (ASJC) codes

    • Software
    • Computer Science Applications
    • Computer Networks and Communications
    • Artificial Intelligence


    Dive into the research topics of 'LMS learning algorithms: Misconceptions and new results on convergence'. Together they form a unique fingerprint.

    Cite this