N400 brain responses to spoken phrases paired with photographs of scenes: Implications for visual scene displays in AAC systems

Krista M. Wilkinson, Allyson Stutzman, Andrea Seisler

Research output: Contribution to journalArticle

Abstract

Augmentative and alternative communication (AAC) systems are often implemented for individuals whose speech cannot meet their full communication needs. One type of aided display is called a Visual Scene Display (VSD). VSDs consist of integrated scenes (such as photographs) in which language concepts are embedded. Often, the representations of concepts on VSDs are perceptually similar to their referents. Given this physical resemblance, one may ask how well VSDs support development of symbolic functioning. We used brain imaging techniques to examine whether matches and mismatches between the content of spoken messages and photographic images of scenes evoke neural activity similar to activity that occurs to spoken or written words. Electroencephalography (EEG) was recorded from 15 college students who were shown photographs paired with spoken phrases that were either matched or mismatched to the concepts embedded within each photograph. Of interest was the N400 component, a negative deflecting wave 400 ms post-stimulus that is considered to be an index of semantic functioning. An N400 response in the mismatched condition (but not the matched) would replicate brain responses to traditional linguistic symbols. An N400 was found, exclusively in the mismatched condition, suggesting that mismatches between spoken messages and VSD-type representations set the stage for the N400 in ways similar to traditional linguistic symbols.

Original languageEnglish (US)
Pages (from-to)51-62
Number of pages12
JournalAAC: Augmentative and Alternative Communication
Volume31
Issue number1
DOIs
StatePublished - Mar 1 2015

Fingerprint

Linguistics
Communication Aids for Disabled
Communication
Brain
Semantics
Neuroimaging
Electroencephalography
Language
Students

All Science Journal Classification (ASJC) codes

  • Rehabilitation
  • Speech and Hearing

Cite this

@article{1bc02b76431143ddbc8d4c1a84026533,
title = "N400 brain responses to spoken phrases paired with photographs of scenes: Implications for visual scene displays in AAC systems",
abstract = "Augmentative and alternative communication (AAC) systems are often implemented for individuals whose speech cannot meet their full communication needs. One type of aided display is called a Visual Scene Display (VSD). VSDs consist of integrated scenes (such as photographs) in which language concepts are embedded. Often, the representations of concepts on VSDs are perceptually similar to their referents. Given this physical resemblance, one may ask how well VSDs support development of symbolic functioning. We used brain imaging techniques to examine whether matches and mismatches between the content of spoken messages and photographic images of scenes evoke neural activity similar to activity that occurs to spoken or written words. Electroencephalography (EEG) was recorded from 15 college students who were shown photographs paired with spoken phrases that were either matched or mismatched to the concepts embedded within each photograph. Of interest was the N400 component, a negative deflecting wave 400 ms post-stimulus that is considered to be an index of semantic functioning. An N400 response in the mismatched condition (but not the matched) would replicate brain responses to traditional linguistic symbols. An N400 was found, exclusively in the mismatched condition, suggesting that mismatches between spoken messages and VSD-type representations set the stage for the N400 in ways similar to traditional linguistic symbols.",
author = "Wilkinson, {Krista M.} and Allyson Stutzman and Andrea Seisler",
year = "2015",
month = "3",
day = "1",
doi = "10.3109/07434618.2014.965342",
language = "English (US)",
volume = "31",
pages = "51--62",
journal = "AAC: Augmentative and Alternative Communication",
issn = "0743-4618",
publisher = "Informa Healthcare",
number = "1",

}

N400 brain responses to spoken phrases paired with photographs of scenes : Implications for visual scene displays in AAC systems. / Wilkinson, Krista M.; Stutzman, Allyson; Seisler, Andrea.

In: AAC: Augmentative and Alternative Communication, Vol. 31, No. 1, 01.03.2015, p. 51-62.

Research output: Contribution to journalArticle

TY - JOUR

T1 - N400 brain responses to spoken phrases paired with photographs of scenes

T2 - Implications for visual scene displays in AAC systems

AU - Wilkinson, Krista M.

AU - Stutzman, Allyson

AU - Seisler, Andrea

PY - 2015/3/1

Y1 - 2015/3/1

N2 - Augmentative and alternative communication (AAC) systems are often implemented for individuals whose speech cannot meet their full communication needs. One type of aided display is called a Visual Scene Display (VSD). VSDs consist of integrated scenes (such as photographs) in which language concepts are embedded. Often, the representations of concepts on VSDs are perceptually similar to their referents. Given this physical resemblance, one may ask how well VSDs support development of symbolic functioning. We used brain imaging techniques to examine whether matches and mismatches between the content of spoken messages and photographic images of scenes evoke neural activity similar to activity that occurs to spoken or written words. Electroencephalography (EEG) was recorded from 15 college students who were shown photographs paired with spoken phrases that were either matched or mismatched to the concepts embedded within each photograph. Of interest was the N400 component, a negative deflecting wave 400 ms post-stimulus that is considered to be an index of semantic functioning. An N400 response in the mismatched condition (but not the matched) would replicate brain responses to traditional linguistic symbols. An N400 was found, exclusively in the mismatched condition, suggesting that mismatches between spoken messages and VSD-type representations set the stage for the N400 in ways similar to traditional linguistic symbols.

AB - Augmentative and alternative communication (AAC) systems are often implemented for individuals whose speech cannot meet their full communication needs. One type of aided display is called a Visual Scene Display (VSD). VSDs consist of integrated scenes (such as photographs) in which language concepts are embedded. Often, the representations of concepts on VSDs are perceptually similar to their referents. Given this physical resemblance, one may ask how well VSDs support development of symbolic functioning. We used brain imaging techniques to examine whether matches and mismatches between the content of spoken messages and photographic images of scenes evoke neural activity similar to activity that occurs to spoken or written words. Electroencephalography (EEG) was recorded from 15 college students who were shown photographs paired with spoken phrases that were either matched or mismatched to the concepts embedded within each photograph. Of interest was the N400 component, a negative deflecting wave 400 ms post-stimulus that is considered to be an index of semantic functioning. An N400 response in the mismatched condition (but not the matched) would replicate brain responses to traditional linguistic symbols. An N400 was found, exclusively in the mismatched condition, suggesting that mismatches between spoken messages and VSD-type representations set the stage for the N400 in ways similar to traditional linguistic symbols.

UR - http://www.scopus.com/inward/record.url?scp=84923105805&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84923105805&partnerID=8YFLogxK

U2 - 10.3109/07434618.2014.965342

DO - 10.3109/07434618.2014.965342

M3 - Article

C2 - 25521434

AN - SCOPUS:84923105805

VL - 31

SP - 51

EP - 62

JO - AAC: Augmentative and Alternative Communication

JF - AAC: Augmentative and Alternative Communication

SN - 0743-4618

IS - 1

ER -