Towards the development of an objective assessment technique for use in engineering design education

Scarlett Rae Miller, Brian P. Bailey, Alex Kirlik

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Design is now considered to be the central or distinguishing activity of engineering and thus is being widely integrated throughout engineering education. However, engineering programs face special challenges in assessing student design abilities because of the subjective nature of design. Research has shown that current evaluation methods are insufficient in evaluating student design performance because quantitative tests of students design skills are not indicators of students design performance and open-ended project evaluations are mudded by the subjective biases of the single human evaluator. The purpose of this paper is to evaluate a new evaluation metric that utilizes a novel application of the Bayesian Truth Serum (BTS) in an effort to increase the accuracy of design skill evaluations and minimize subjective bias. Through a series of experiments our results indicate that BTS is a better indicator of idea generation ability than traditional evaluation methods. Although more work is needed to validate and understand how BTS can be used as an evaluation technique in engineering design, the results from this study are promising and provide a first step at deriving quantitative measures of student design performance in engineering.

Original languageEnglish (US)
Title of host publication119th ASEE Annual Conference and Exposition
PublisherAmerican Society for Engineering Education
ISBN (Print)9780878232413
StatePublished - Jan 1 2012
Event119th ASEE Annual Conference and Exposition - San Antonio, TX, United States
Duration: Jun 10 2012Jun 13 2012

Other

Other119th ASEE Annual Conference and Exposition
CountryUnited States
CitySan Antonio, TX
Period6/10/126/13/12

Fingerprint

Education
Students
Engineering education
Experiments

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Miller, S. R., Bailey, B. P., & Kirlik, A. (2012). Towards the development of an objective assessment technique for use in engineering design education. In 119th ASEE Annual Conference and Exposition American Society for Engineering Education.
Miller, Scarlett Rae ; Bailey, Brian P. ; Kirlik, Alex. / Towards the development of an objective assessment technique for use in engineering design education. 119th ASEE Annual Conference and Exposition. American Society for Engineering Education, 2012.
@inproceedings{3bcb9c2e116e4c4a9e8835529e497e83,
title = "Towards the development of an objective assessment technique for use in engineering design education",
abstract = "Design is now considered to be the central or distinguishing activity of engineering and thus is being widely integrated throughout engineering education. However, engineering programs face special challenges in assessing student design abilities because of the subjective nature of design. Research has shown that current evaluation methods are insufficient in evaluating student design performance because quantitative tests of students design skills are not indicators of students design performance and open-ended project evaluations are mudded by the subjective biases of the single human evaluator. The purpose of this paper is to evaluate a new evaluation metric that utilizes a novel application of the Bayesian Truth Serum (BTS) in an effort to increase the accuracy of design skill evaluations and minimize subjective bias. Through a series of experiments our results indicate that BTS is a better indicator of idea generation ability than traditional evaluation methods. Although more work is needed to validate and understand how BTS can be used as an evaluation technique in engineering design, the results from this study are promising and provide a first step at deriving quantitative measures of student design performance in engineering.",
author = "Miller, {Scarlett Rae} and Bailey, {Brian P.} and Alex Kirlik",
year = "2012",
month = "1",
day = "1",
language = "English (US)",
isbn = "9780878232413",
booktitle = "119th ASEE Annual Conference and Exposition",
publisher = "American Society for Engineering Education",
address = "United States",

}

Miller, SR, Bailey, BP & Kirlik, A 2012, Towards the development of an objective assessment technique for use in engineering design education. in 119th ASEE Annual Conference and Exposition. American Society for Engineering Education, 119th ASEE Annual Conference and Exposition, San Antonio, TX, United States, 6/10/12.

Towards the development of an objective assessment technique for use in engineering design education. / Miller, Scarlett Rae; Bailey, Brian P.; Kirlik, Alex.

119th ASEE Annual Conference and Exposition. American Society for Engineering Education, 2012.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Towards the development of an objective assessment technique for use in engineering design education

AU - Miller, Scarlett Rae

AU - Bailey, Brian P.

AU - Kirlik, Alex

PY - 2012/1/1

Y1 - 2012/1/1

N2 - Design is now considered to be the central or distinguishing activity of engineering and thus is being widely integrated throughout engineering education. However, engineering programs face special challenges in assessing student design abilities because of the subjective nature of design. Research has shown that current evaluation methods are insufficient in evaluating student design performance because quantitative tests of students design skills are not indicators of students design performance and open-ended project evaluations are mudded by the subjective biases of the single human evaluator. The purpose of this paper is to evaluate a new evaluation metric that utilizes a novel application of the Bayesian Truth Serum (BTS) in an effort to increase the accuracy of design skill evaluations and minimize subjective bias. Through a series of experiments our results indicate that BTS is a better indicator of idea generation ability than traditional evaluation methods. Although more work is needed to validate and understand how BTS can be used as an evaluation technique in engineering design, the results from this study are promising and provide a first step at deriving quantitative measures of student design performance in engineering.

AB - Design is now considered to be the central or distinguishing activity of engineering and thus is being widely integrated throughout engineering education. However, engineering programs face special challenges in assessing student design abilities because of the subjective nature of design. Research has shown that current evaluation methods are insufficient in evaluating student design performance because quantitative tests of students design skills are not indicators of students design performance and open-ended project evaluations are mudded by the subjective biases of the single human evaluator. The purpose of this paper is to evaluate a new evaluation metric that utilizes a novel application of the Bayesian Truth Serum (BTS) in an effort to increase the accuracy of design skill evaluations and minimize subjective bias. Through a series of experiments our results indicate that BTS is a better indicator of idea generation ability than traditional evaluation methods. Although more work is needed to validate and understand how BTS can be used as an evaluation technique in engineering design, the results from this study are promising and provide a first step at deriving quantitative measures of student design performance in engineering.

UR - http://www.scopus.com/inward/record.url?scp=85029041140&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029041140&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85029041140

SN - 9780878232413

BT - 119th ASEE Annual Conference and Exposition

PB - American Society for Engineering Education

ER -

Miller SR, Bailey BP, Kirlik A. Towards the development of an objective assessment technique for use in engineering design education. In 119th ASEE Annual Conference and Exposition. American Society for Engineering Education. 2012