Psychometric analysis of residence and MOOC assessments

Eric Loken, Zita Oravecz, Conrad Tucker, Fridolin Jakob Linder

Research output: Contribution to journalConference articlepeer-review

Abstract

Undergraduate STEM programs are faced with the daunting challenge of managing instruction and assessment for classes that enroll thousands of students per year, and the bulk of student assessment is often determined by multiple choice tests. Instructors try to monitor the reliability metrics and diagnostics for item quality, but rarely is there a more formal evaluation of the psychometric properties of these assessments. College assessment strategies seem to be dominated by a common-sense view of testing that is generally unconcerned about precision of measurement. We see an opportunity to have an impact on undergraduate science instruction by incorporating more rigorous measurement models for testing, and using them to assist instructional goals and assessment. We apply item response theory to analyze tests from two undergraduate STEM classes, a resident instruction physics class and a Massive Open Online Course (MOOC) in geography. We evaluate whether the tests are equally informative across levels of student proficiency, and we demonstrate how precision could be improved with adaptive testing. We find that the measurement precision of multiple choice tests appears to be greatest in the lower half of the class distribution, a property that has consequences for assessment of mastery and for evaluating testing interventions.

Original languageEnglish (US)
JournalASEE Annual Conference and Exposition, Conference Proceedings
Volume122nd ASEE Annual Conference and Exposition: Making Value for Society
Issue number122nd ASEE Annual Conference and Exposition: Making Value for...
StatePublished - Jan 1 2015
Event2015 122nd ASEE Annual Conference and Exposition - Seattle, United States
Duration: Jun 14 2015Jun 17 2015

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Fingerprint Dive into the research topics of 'Psychometric analysis of residence and MOOC assessments'. Together they form a unique fingerprint.

Cite this