Studying the reliability and validity of test scores for mathematical and spatial reasoning tasks for engineering students

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

While preliminary, we interpret our results as encouraging as we attempt to measure four different constructs that we believe are keys to academic success in engineering. Two constructs represent mathematical skills or traits. Two constructs represent spatial reasoning strategies or traits. Correlation coefficients indicated there were some moderate associations between pairs of constructs (e.g., M1 and S2), and a preliminary factor analysis showed that part of the relation between constructs may be related to the types of external representations (e.g., diagrams, equations, graphs) presented in the test items or the primary physics concepts (e.g. acceleration, force, motion) for which the items have been designed to assess achievement in engineering. In the future, we will plan studies so that more items are included on each scale. Further, we will attempt to replicate the initial findings reported in this manuscript. If correlation coefficients between scores are at best moderate, then this pattern of results has important implications for assessment, teaching, and research in engineering education. Simply, it implies that the skill set required to succeed in engineering may be multidimensional. As such, a set of various tasks or tests are needed to help students understand their profiles of strengths and weaknesses as they attempt to meet degree requirements in this, sometimes, challenging field. In future research, it might be important to study how students think aloud as they solve problems not only within one construct set, but also, how they compare and contrast the problems they attempt to solve across sets. A mixed-methodological approach where we study both test scores with verbal reports might help us understand more about the complex processing required to complete exercises successfully. Further, verbal protocol summaries might reveal the role of one or more external representations given how students interpret problems in engineering and the processes they use to select or construct solutions for these problems.

Original languageEnglish (US)
JournalASEE Annual Conference and Exposition, Conference Proceedings
StatePublished - 2011

Fingerprint

Students
Factor analysis
Engineering education
Teaching
Physics
Processing

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

@article{57c1b4d82b51414fb5b04fcf49ccaf18,
title = "Studying the reliability and validity of test scores for mathematical and spatial reasoning tasks for engineering students",
abstract = "While preliminary, we interpret our results as encouraging as we attempt to measure four different constructs that we believe are keys to academic success in engineering. Two constructs represent mathematical skills or traits. Two constructs represent spatial reasoning strategies or traits. Correlation coefficients indicated there were some moderate associations between pairs of constructs (e.g., M1 and S2), and a preliminary factor analysis showed that part of the relation between constructs may be related to the types of external representations (e.g., diagrams, equations, graphs) presented in the test items or the primary physics concepts (e.g. acceleration, force, motion) for which the items have been designed to assess achievement in engineering. In the future, we will plan studies so that more items are included on each scale. Further, we will attempt to replicate the initial findings reported in this manuscript. If correlation coefficients between scores are at best moderate, then this pattern of results has important implications for assessment, teaching, and research in engineering education. Simply, it implies that the skill set required to succeed in engineering may be multidimensional. As such, a set of various tasks or tests are needed to help students understand their profiles of strengths and weaknesses as they attempt to meet degree requirements in this, sometimes, challenging field. In future research, it might be important to study how students think aloud as they solve problems not only within one construct set, but also, how they compare and contrast the problems they attempt to solve across sets. A mixed-methodological approach where we study both test scores with verbal reports might help us understand more about the complex processing required to complete exercises successfully. Further, verbal protocol summaries might reveal the role of one or more external representations given how students interpret problems in engineering and the processes they use to select or construct solutions for these problems.",
author = "Laura Pauley and Kulikowich, {Jonna Marie} and Nell Sedransk and Engel, {Renata S.}",
year = "2011",
language = "English (US)",
journal = "ASEE Annual Conference and Exposition, Conference Proceedings",
issn = "2153-5965",

}

TY - JOUR

T1 - Studying the reliability and validity of test scores for mathematical and spatial reasoning tasks for engineering students

AU - Pauley, Laura

AU - Kulikowich, Jonna Marie

AU - Sedransk, Nell

AU - Engel, Renata S.

PY - 2011

Y1 - 2011

N2 - While preliminary, we interpret our results as encouraging as we attempt to measure four different constructs that we believe are keys to academic success in engineering. Two constructs represent mathematical skills or traits. Two constructs represent spatial reasoning strategies or traits. Correlation coefficients indicated there were some moderate associations between pairs of constructs (e.g., M1 and S2), and a preliminary factor analysis showed that part of the relation between constructs may be related to the types of external representations (e.g., diagrams, equations, graphs) presented in the test items or the primary physics concepts (e.g. acceleration, force, motion) for which the items have been designed to assess achievement in engineering. In the future, we will plan studies so that more items are included on each scale. Further, we will attempt to replicate the initial findings reported in this manuscript. If correlation coefficients between scores are at best moderate, then this pattern of results has important implications for assessment, teaching, and research in engineering education. Simply, it implies that the skill set required to succeed in engineering may be multidimensional. As such, a set of various tasks or tests are needed to help students understand their profiles of strengths and weaknesses as they attempt to meet degree requirements in this, sometimes, challenging field. In future research, it might be important to study how students think aloud as they solve problems not only within one construct set, but also, how they compare and contrast the problems they attempt to solve across sets. A mixed-methodological approach where we study both test scores with verbal reports might help us understand more about the complex processing required to complete exercises successfully. Further, verbal protocol summaries might reveal the role of one or more external representations given how students interpret problems in engineering and the processes they use to select or construct solutions for these problems.

AB - While preliminary, we interpret our results as encouraging as we attempt to measure four different constructs that we believe are keys to academic success in engineering. Two constructs represent mathematical skills or traits. Two constructs represent spatial reasoning strategies or traits. Correlation coefficients indicated there were some moderate associations between pairs of constructs (e.g., M1 and S2), and a preliminary factor analysis showed that part of the relation between constructs may be related to the types of external representations (e.g., diagrams, equations, graphs) presented in the test items or the primary physics concepts (e.g. acceleration, force, motion) for which the items have been designed to assess achievement in engineering. In the future, we will plan studies so that more items are included on each scale. Further, we will attempt to replicate the initial findings reported in this manuscript. If correlation coefficients between scores are at best moderate, then this pattern of results has important implications for assessment, teaching, and research in engineering education. Simply, it implies that the skill set required to succeed in engineering may be multidimensional. As such, a set of various tasks or tests are needed to help students understand their profiles of strengths and weaknesses as they attempt to meet degree requirements in this, sometimes, challenging field. In future research, it might be important to study how students think aloud as they solve problems not only within one construct set, but also, how they compare and contrast the problems they attempt to solve across sets. A mixed-methodological approach where we study both test scores with verbal reports might help us understand more about the complex processing required to complete exercises successfully. Further, verbal protocol summaries might reveal the role of one or more external representations given how students interpret problems in engineering and the processes they use to select or construct solutions for these problems.

UR - http://www.scopus.com/inward/record.url?scp=85029069833&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029069833&partnerID=8YFLogxK

M3 - Article

JO - ASEE Annual Conference and Exposition, Conference Proceedings

JF - ASEE Annual Conference and Exposition, Conference Proceedings

SN - 2153-5965

ER -