Exploring the utility of Bayesian truth serum for assessing design knowledge

Scarlett R. Miller, Brian P. Bailey, Alex Kirlik

Research output: Contribution to journalArticlepeer-review

16 Scopus citations

Abstract

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

Original languageEnglish (US)
Pages (from-to)487-515
Number of pages29
JournalHuman-Computer Interaction
Volume29
Issue number5-6
DOIs
StatePublished - Apr 2 2014

All Science Journal Classification (ASJC) codes

  • Applied Psychology
  • Human-Computer Interaction

Fingerprint Dive into the research topics of 'Exploring the utility of Bayesian truth serum for assessing design knowledge'. Together they form a unique fingerprint.

Cite this