Exploring the utility of Bayesian truth serum for assessing design knowledge

Scarlett Rae Miller, Brian P. Bailey, Alex Kirlik

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

Original languageEnglish (US)
Pages (from-to)487-515
Number of pages29
JournalHuman-Computer Interaction
Volume29
Issue number5-6
DOIs
StatePublished - Apr 2 2014

Fingerprint

Serum
Students
Education
Statistical tests
Personnel
Surveys and Questionnaires
Feedback

All Science Journal Classification (ASJC) codes

  • Applied Psychology
  • Human-Computer Interaction

Cite this

Miller, Scarlett Rae ; Bailey, Brian P. ; Kirlik, Alex. / Exploring the utility of Bayesian truth serum for assessing design knowledge. In: Human-Computer Interaction. 2014 ; Vol. 29, No. 5-6. pp. 487-515.
@article{a486f34b6838451f9009752e50cbfe3f,
title = "Exploring the utility of Bayesian truth serum for assessing design knowledge",
abstract = "Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.",
author = "Miller, {Scarlett Rae} and Bailey, {Brian P.} and Alex Kirlik",
year = "2014",
month = "4",
day = "2",
doi = "10.1080/07370024.2013.870393",
language = "English (US)",
volume = "29",
pages = "487--515",
journal = "Human-Computer Interaction",
issn = "0737-0024",
publisher = "Taylor and Francis Ltd.",
number = "5-6",

}

Exploring the utility of Bayesian truth serum for assessing design knowledge. / Miller, Scarlett Rae; Bailey, Brian P.; Kirlik, Alex.

In: Human-Computer Interaction, Vol. 29, No. 5-6, 02.04.2014, p. 487-515.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Exploring the utility of Bayesian truth serum for assessing design knowledge

AU - Miller, Scarlett Rae

AU - Bailey, Brian P.

AU - Kirlik, Alex

PY - 2014/4/2

Y1 - 2014/4/2

N2 - Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

AB - Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

UR - http://www.scopus.com/inward/record.url?scp=84903120993&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84903120993&partnerID=8YFLogxK

U2 - 10.1080/07370024.2013.870393

DO - 10.1080/07370024.2013.870393

M3 - Article

AN - SCOPUS:84903120993

VL - 29

SP - 487

EP - 515

JO - Human-Computer Interaction

JF - Human-Computer Interaction

SN - 0737-0024

IS - 5-6

ER -