Assessment of resident surgical skills: Is testing feasible?

Barbara Goff, Lynn Mandel, Gretchen Lentz, Amy VanBlaricom, Anne Marie Amies Oelschlager, David Lee, Andrew Galakatos, Matthew Davies, Peter Nielsen

Research output: Contribution to journalArticle

86 Citations (Scopus)

Abstract

Objective: We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results. Study design: An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models. Results: Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's α were. 97 for overall global rating and. 95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from. 71 to. 97 for individual tasks and was. 95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was $40 to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training. Conclusion: Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.

Original languageEnglish (US)
Pages (from-to)1331-1338
Number of pages8
JournalAmerican journal of obstetrics and gynecology
Volume192
Issue number4
DOIs
StatePublished - Jan 1 2005

Fingerprint

Internship and Residency
Curriculum
Checklist
Costs and Cost Analysis

All Science Journal Classification (ASJC) codes

  • Obstetrics and Gynecology

Cite this

Goff, B., Mandel, L., Lentz, G., VanBlaricom, A., Oelschlager, A. M. A., Lee, D., ... Nielsen, P. (2005). Assessment of resident surgical skills: Is testing feasible? American journal of obstetrics and gynecology, 192(4), 1331-1338. https://doi.org/10.1016/j.ajog.2004.12.068
Goff, Barbara ; Mandel, Lynn ; Lentz, Gretchen ; VanBlaricom, Amy ; Oelschlager, Anne Marie Amies ; Lee, David ; Galakatos, Andrew ; Davies, Matthew ; Nielsen, Peter. / Assessment of resident surgical skills : Is testing feasible?. In: American journal of obstetrics and gynecology. 2005 ; Vol. 192, No. 4. pp. 1331-1338.
@article{55c42359399048cea308c14f0703972c,
title = "Assessment of resident surgical skills: Is testing feasible?",
abstract = "Objective: We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results. Study design: An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models. Results: Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's α were. 97 for overall global rating and. 95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from. 71 to. 97 for individual tasks and was. 95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was $40 to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training. Conclusion: Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.",
author = "Barbara Goff and Lynn Mandel and Gretchen Lentz and Amy VanBlaricom and Oelschlager, {Anne Marie Amies} and David Lee and Andrew Galakatos and Matthew Davies and Peter Nielsen",
year = "2005",
month = "1",
day = "1",
doi = "10.1016/j.ajog.2004.12.068",
language = "English (US)",
volume = "192",
pages = "1331--1338",
journal = "American Journal of Obstetrics and Gynecology",
issn = "0002-9378",
publisher = "Mosby Inc.",
number = "4",

}

Goff, B, Mandel, L, Lentz, G, VanBlaricom, A, Oelschlager, AMA, Lee, D, Galakatos, A, Davies, M & Nielsen, P 2005, 'Assessment of resident surgical skills: Is testing feasible?', American journal of obstetrics and gynecology, vol. 192, no. 4, pp. 1331-1338. https://doi.org/10.1016/j.ajog.2004.12.068

Assessment of resident surgical skills : Is testing feasible? / Goff, Barbara; Mandel, Lynn; Lentz, Gretchen; VanBlaricom, Amy; Oelschlager, Anne Marie Amies; Lee, David; Galakatos, Andrew; Davies, Matthew; Nielsen, Peter.

In: American journal of obstetrics and gynecology, Vol. 192, No. 4, 01.01.2005, p. 1331-1338.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Assessment of resident surgical skills

T2 - Is testing feasible?

AU - Goff, Barbara

AU - Mandel, Lynn

AU - Lentz, Gretchen

AU - VanBlaricom, Amy

AU - Oelschlager, Anne Marie Amies

AU - Lee, David

AU - Galakatos, Andrew

AU - Davies, Matthew

AU - Nielsen, Peter

PY - 2005/1/1

Y1 - 2005/1/1

N2 - Objective: We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results. Study design: An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models. Results: Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's α were. 97 for overall global rating and. 95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from. 71 to. 97 for individual tasks and was. 95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was $40 to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training. Conclusion: Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.

AB - Objective: We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results. Study design: An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models. Results: Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's α were. 97 for overall global rating and. 95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from. 71 to. 97 for individual tasks and was. 95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was $40 to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training. Conclusion: Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.

UR - http://www.scopus.com/inward/record.url?scp=16844379125&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=16844379125&partnerID=8YFLogxK

U2 - 10.1016/j.ajog.2004.12.068

DO - 10.1016/j.ajog.2004.12.068

M3 - Article

C2 - 15846232

AN - SCOPUS:16844379125

VL - 192

SP - 1331

EP - 1338

JO - American Journal of Obstetrics and Gynecology

JF - American Journal of Obstetrics and Gynecology

SN - 0002-9378

IS - 4

ER -

Goff B, Mandel L, Lentz G, VanBlaricom A, Oelschlager AMA, Lee D et al. Assessment of resident surgical skills: Is testing feasible? American journal of obstetrics and gynecology. 2005 Jan 1;192(4):1331-1338. https://doi.org/10.1016/j.ajog.2004.12.068