Direct behavior rating: An evaluation of time-series interpretations as consequential validity

Theodore J. Christ, Peter Marlow Nelson, Ethan R. Van Norman, Sandra M. Chafouleas, T. Chris Riley-Tillman

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three "true" trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67% of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.

Original languageEnglish (US)
Pages (from-to)157-170
Number of pages14
JournalSchool Psychology Quarterly
Volume29
Issue number2
DOIs
StatePublished - Jan 1 2014

Fingerprint

time series
rating
Students
interpretation
trend
evaluation
Linear Models
significance test
Direction compound
student
graduate
classroom
teacher
evidence
experience

All Science Journal Classification (ASJC) codes

  • Education
  • Developmental and Educational Psychology

Cite this

Christ, Theodore J. ; Nelson, Peter Marlow ; Van Norman, Ethan R. ; Chafouleas, Sandra M. ; Riley-Tillman, T. Chris. / Direct behavior rating : An evaluation of time-series interpretations as consequential validity. In: School Psychology Quarterly. 2014 ; Vol. 29, No. 2. pp. 157-170.
@article{a51395fabdf84d5cbb6cc64c1789ccfa,
title = "Direct behavior rating: An evaluation of time-series interpretations as consequential validity",
abstract = "Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three {"}true{"} trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67{\%} of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.",
author = "Christ, {Theodore J.} and Nelson, {Peter Marlow} and {Van Norman}, {Ethan R.} and Chafouleas, {Sandra M.} and Riley-Tillman, {T. Chris}",
year = "2014",
month = "1",
day = "1",
doi = "10.1037/spq0000029",
language = "English (US)",
volume = "29",
pages = "157--170",
journal = "School Psychology",
issn = "2578-4218",
publisher = "American Psychological Association Inc.",
number = "2",

}

Direct behavior rating : An evaluation of time-series interpretations as consequential validity. / Christ, Theodore J.; Nelson, Peter Marlow; Van Norman, Ethan R.; Chafouleas, Sandra M.; Riley-Tillman, T. Chris.

In: School Psychology Quarterly, Vol. 29, No. 2, 01.01.2014, p. 157-170.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Direct behavior rating

T2 - An evaluation of time-series interpretations as consequential validity

AU - Christ, Theodore J.

AU - Nelson, Peter Marlow

AU - Van Norman, Ethan R.

AU - Chafouleas, Sandra M.

AU - Riley-Tillman, T. Chris

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three "true" trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67% of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.

AB - Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three "true" trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67% of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.

UR - http://www.scopus.com/inward/record.url?scp=84902459001&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84902459001&partnerID=8YFLogxK

U2 - 10.1037/spq0000029

DO - 10.1037/spq0000029

M3 - Article

C2 - 23895317

AN - SCOPUS:84902459001

VL - 29

SP - 157

EP - 170

JO - School Psychology

JF - School Psychology

SN - 2578-4218

IS - 2

ER -