Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education

Jessica Menold, Kathryn Weed Jablokow, Senay Purzer, Daniel Michael Ferguson, Matthew W. Ohland

Research output: Contribution to journalConference article

2 Citations (Scopus)

Abstract

Many sound methods exist for creating the items or questions that make up educational surveys and assessments. These methods include the use of content experts, reviews of existing instruments, and lists of behaviors and descriptors commonly associated with the construct(s) we wish to assess. Unfortunately, however, item creation sometimes becomes overly dependent upon a researcher's personal attitudes about the construct(s) being tested, or on "borrowing" items from other instruments that may or may not be sound measures of the construct(s) of interest. These risks are particularly likely for new researchers in engineering education, who may have little experience with best practices in social science research. One way to support best practices in the development of new surveys and assessments is to use an instrument blueprint to guide the creation of items, as well as the collection of validity evidence. This paper outlines a process for instrument blueprint creation and content validation to help support best practices in educational assessment. Based on Messick's unified theory of validity1, the instrument blueprint includes a process for item construction that incorporates multiple resources, including: (1) the views of content experts; (2) research from the relevant domain of interest; (3) reviews of existing instruments; and (4) the expertise of the research team. This paper uses the development of a new instrument to measure engineering innovativeness as an illustrative example of the blueprinting process. Our new instrument will assess 20 characteristics of innovative engineers as identified by in-depth studies of expert engineering innovators in previous research2. This work highlights the positive impact of using a systematic process for item construction to transform current methods of assessment in engineering education.

Original languageEnglish (US)
JournalASEE Annual Conference and Exposition, Conference Proceedings
Volume122nd ASEE Annual Conference and Exposition: Making Value for Society
Issue number122nd ASEE Annual Conference and Exposition: Making Value for...
StatePublished - Jan 1 2015
Event2015 122nd ASEE Annual Conference and Exposition - Seattle, United States
Duration: Jun 14 2015Jun 17 2015

Fingerprint

Blueprints
Engineering education
Acoustic waves
Social sciences
Engineers

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Menold, J., Jablokow, K. W., Purzer, S., Ferguson, D. M., & Ohland, M. W. (2015). Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education. ASEE Annual Conference and Exposition, Conference Proceedings, 122nd ASEE Annual Conference and Exposition: Making Value for Society(122nd ASEE Annual Conference and Exposition: Making Value for...).
Menold, Jessica ; Jablokow, Kathryn Weed ; Purzer, Senay ; Ferguson, Daniel Michael ; Ohland, Matthew W. / Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education. In: ASEE Annual Conference and Exposition, Conference Proceedings. 2015 ; Vol. 122nd ASEE Annual Conference and Exposition: Making Value for Society, No. 122nd ASEE Annual Conference and Exposition: Making Value for...
@article{aeb89f2e881e49849b620624cc604f32,
title = "Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education",
abstract = "Many sound methods exist for creating the items or questions that make up educational surveys and assessments. These methods include the use of content experts, reviews of existing instruments, and lists of behaviors and descriptors commonly associated with the construct(s) we wish to assess. Unfortunately, however, item creation sometimes becomes overly dependent upon a researcher's personal attitudes about the construct(s) being tested, or on {"}borrowing{"} items from other instruments that may or may not be sound measures of the construct(s) of interest. These risks are particularly likely for new researchers in engineering education, who may have little experience with best practices in social science research. One way to support best practices in the development of new surveys and assessments is to use an instrument blueprint to guide the creation of items, as well as the collection of validity evidence. This paper outlines a process for instrument blueprint creation and content validation to help support best practices in educational assessment. Based on Messick's unified theory of validity1, the instrument blueprint includes a process for item construction that incorporates multiple resources, including: (1) the views of content experts; (2) research from the relevant domain of interest; (3) reviews of existing instruments; and (4) the expertise of the research team. This paper uses the development of a new instrument to measure engineering innovativeness as an illustrative example of the blueprinting process. Our new instrument will assess 20 characteristics of innovative engineers as identified by in-depth studies of expert engineering innovators in previous research2. This work highlights the positive impact of using a systematic process for item construction to transform current methods of assessment in engineering education.",
author = "Jessica Menold and Jablokow, {Kathryn Weed} and Senay Purzer and Ferguson, {Daniel Michael} and Ohland, {Matthew W.}",
year = "2015",
month = "1",
day = "1",
language = "English (US)",
volume = "122nd ASEE Annual Conference and Exposition: Making Value for Society",
journal = "ASEE Annual Conference and Exposition, Conference Proceedings",
issn = "2153-5965",
number = "122nd ASEE Annual Conference and Exposition: Making Value for...",

}

Menold, J, Jablokow, KW, Purzer, S, Ferguson, DM & Ohland, MW 2015, 'Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education', ASEE Annual Conference and Exposition, Conference Proceedings, vol. 122nd ASEE Annual Conference and Exposition: Making Value for Society, no. 122nd ASEE Annual Conference and Exposition: Making Value for....

Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education. / Menold, Jessica; Jablokow, Kathryn Weed; Purzer, Senay; Ferguson, Daniel Michael; Ohland, Matthew W.

In: ASEE Annual Conference and Exposition, Conference Proceedings, Vol. 122nd ASEE Annual Conference and Exposition: Making Value for Society, No. 122nd ASEE Annual Conference and Exposition: Making Value for..., 01.01.2015.

Research output: Contribution to journalConference article

TY - JOUR

T1 - Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education

AU - Menold, Jessica

AU - Jablokow, Kathryn Weed

AU - Purzer, Senay

AU - Ferguson, Daniel Michael

AU - Ohland, Matthew W.

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Many sound methods exist for creating the items or questions that make up educational surveys and assessments. These methods include the use of content experts, reviews of existing instruments, and lists of behaviors and descriptors commonly associated with the construct(s) we wish to assess. Unfortunately, however, item creation sometimes becomes overly dependent upon a researcher's personal attitudes about the construct(s) being tested, or on "borrowing" items from other instruments that may or may not be sound measures of the construct(s) of interest. These risks are particularly likely for new researchers in engineering education, who may have little experience with best practices in social science research. One way to support best practices in the development of new surveys and assessments is to use an instrument blueprint to guide the creation of items, as well as the collection of validity evidence. This paper outlines a process for instrument blueprint creation and content validation to help support best practices in educational assessment. Based on Messick's unified theory of validity1, the instrument blueprint includes a process for item construction that incorporates multiple resources, including: (1) the views of content experts; (2) research from the relevant domain of interest; (3) reviews of existing instruments; and (4) the expertise of the research team. This paper uses the development of a new instrument to measure engineering innovativeness as an illustrative example of the blueprinting process. Our new instrument will assess 20 characteristics of innovative engineers as identified by in-depth studies of expert engineering innovators in previous research2. This work highlights the positive impact of using a systematic process for item construction to transform current methods of assessment in engineering education.

AB - Many sound methods exist for creating the items or questions that make up educational surveys and assessments. These methods include the use of content experts, reviews of existing instruments, and lists of behaviors and descriptors commonly associated with the construct(s) we wish to assess. Unfortunately, however, item creation sometimes becomes overly dependent upon a researcher's personal attitudes about the construct(s) being tested, or on "borrowing" items from other instruments that may or may not be sound measures of the construct(s) of interest. These risks are particularly likely for new researchers in engineering education, who may have little experience with best practices in social science research. One way to support best practices in the development of new surveys and assessments is to use an instrument blueprint to guide the creation of items, as well as the collection of validity evidence. This paper outlines a process for instrument blueprint creation and content validation to help support best practices in educational assessment. Based on Messick's unified theory of validity1, the instrument blueprint includes a process for item construction that incorporates multiple resources, including: (1) the views of content experts; (2) research from the relevant domain of interest; (3) reviews of existing instruments; and (4) the expertise of the research team. This paper uses the development of a new instrument to measure engineering innovativeness as an illustrative example of the blueprinting process. Our new instrument will assess 20 characteristics of innovative engineers as identified by in-depth studies of expert engineering innovators in previous research2. This work highlights the positive impact of using a systematic process for item construction to transform current methods of assessment in engineering education.

UR - http://www.scopus.com/inward/record.url?scp=84941994496&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84941994496&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:84941994496

VL - 122nd ASEE Annual Conference and Exposition: Making Value for Society

JO - ASEE Annual Conference and Exposition, Conference Proceedings

JF - ASEE Annual Conference and Exposition, Conference Proceedings

SN - 2153-5965

IS - 122nd ASEE Annual Conference and Exposition: Making Value for...

ER -

Menold J, Jablokow KW, Purzer S, Ferguson DM, Ohland MW. Using an instrument blueprint to support the rigorous development of new surveys and assessments in engineering education. ASEE Annual Conference and Exposition, Conference Proceedings. 2015 Jan 1;122nd ASEE Annual Conference and Exposition: Making Value for Society(122nd ASEE Annual Conference and Exposition: Making Value for...).