Optimal habits can develop spontaneously through sensitivity to local cost

Theresa M. Desrochers, Dezhe Z. Jin, Noah D. Goodman, Ann M. Graybiel

Research output: Contribution to journalArticle

21 Scopus citations

Abstract

Habits and rituals are expressed universally across animal species. These behaviors are advantageous in allowing sequential behaviors to be performed without cognitive overload, and appear to rely on neural circuits that are relatively benign but vulnerable to takeover by extreme contexts, neuropsychiatric sequelae, and processes leading to addiction. Reinforcement learning (RL) is thought to underlie the formation of optimal habits. However, this theoretic formulation has principally been tested experimentally in simple stimulus-response tasks with relatively fewavailable responses. We asked whether RL could also account for the emergence of habitual action sequences in realistically complex situations in which no repetitive stimulus-response links were present and in which many response options were present. We exposed naïve macaque monkeys to such experimental conditions by introducing a unique free saccade scan task. Despite the highly uncertain conditions and no instruction, the monkeys developed a succession of stereotypical, self-chosen saccade sequence patterns. Remarkably, these continued to morph for months, long after session-averaged reward and cost (eye movement distance) reached asymptote. Prima facie, these continued behavioral changes appeared to challenge RL. However, trial-by-trial analysis showed that pattern changes on adjacent trials were predicted by lowered cost, and RL simulations that reduced the cost reproduced the monkeys' behavior. Ultimately, the patterns settled into stereotypical saccade sequences that minimized the cost of obtaining the reward on average. These findings suggest that brain mechanisms underlying the emergence of habits, and perhaps unwanted repetitive behaviors in clinical disorders, could follow RL algorithms capturing extremely local explore/exploit tradeoffs.

Original languageEnglish (US)
Pages (from-to)20512-20517
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume107
Issue number47
DOIs
StatePublished - Nov 23 2010

All Science Journal Classification (ASJC) codes

  • General

Fingerprint Dive into the research topics of 'Optimal habits can develop spontaneously through sensitivity to local cost'. Together they form a unique fingerprint.

  • Cite this