Evaluating Report Text Variation and Informativeness: Natural Language Processing of CT Chest Imaging for Pulmonary Embolism

Marco Huesch, Rekha Cherian, Sam Labib, Rickhesvar Mahraj

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Objective: The aim of this study was to quantify the variability of language in free text reports of pulmonary embolus (PE) studies and to gauge the informativeness of free text to predict PE diagnosis using machine learning as proxy for human understanding. Materials and Methods: All 1,133 consecutive chest CTs with contrast studies performed under a PE protocol and ordered in the emergency department in 2016 were selected from our departmental electronic workflow system. We used commercial text-mining and predictive analytics software to parse and describe all report text and to generate a suite of machine learning rules that sought to predict the “gold standard” radiological diagnosis of PE. Results: There was extensive variation in the length of Findings section and Impression section texts across the reports, only marginally associated with a positive PE diagnosis. A marked concentration of terms was found: for example, 20 words were used in the Findings section of 93% of the reports, and 896 of 2,296 distinct words were each used in only one report's Impression section. In the validation set, machine learning rules had perfect sensitivity but imperfect specificity, a low positive predictive value of 73%, and a misclassification rate of 3%. Conclusion: Use of free text reporting was associated with extensive variability in report length and report terms used. Interpretation of the free text was a difficult machine learning task and suggests potential difficulty for human recipients in fully understanding such reports. These results support the prospective assessment of the impact of a fully structured report template with at least some mandatory discrete fields on ease of use of reports and their understanding.

Original languageEnglish (US)
Pages (from-to)554-562
Number of pages9
JournalJournal of the American College of Radiology
Volume15
Issue number3
DOIs
StatePublished - Mar 1 2018

Fingerprint

Natural Language Processing
Embolism
Pulmonary Embolism
Thorax
Lung
Data Mining
Workflow
Proxy
Hospital Emergency Service
Language
Software
Sensitivity and Specificity
Machine Learning

All Science Journal Classification (ASJC) codes

  • Radiology Nuclear Medicine and imaging

Cite this

@article{0681339cd39d4f128a5ee52e6fb18200,
title = "Evaluating Report Text Variation and Informativeness: Natural Language Processing of CT Chest Imaging for Pulmonary Embolism",
abstract = "Objective: The aim of this study was to quantify the variability of language in free text reports of pulmonary embolus (PE) studies and to gauge the informativeness of free text to predict PE diagnosis using machine learning as proxy for human understanding. Materials and Methods: All 1,133 consecutive chest CTs with contrast studies performed under a PE protocol and ordered in the emergency department in 2016 were selected from our departmental electronic workflow system. We used commercial text-mining and predictive analytics software to parse and describe all report text and to generate a suite of machine learning rules that sought to predict the “gold standard” radiological diagnosis of PE. Results: There was extensive variation in the length of Findings section and Impression section texts across the reports, only marginally associated with a positive PE diagnosis. A marked concentration of terms was found: for example, 20 words were used in the Findings section of 93{\%} of the reports, and 896 of 2,296 distinct words were each used in only one report's Impression section. In the validation set, machine learning rules had perfect sensitivity but imperfect specificity, a low positive predictive value of 73{\%}, and a misclassification rate of 3{\%}. Conclusion: Use of free text reporting was associated with extensive variability in report length and report terms used. Interpretation of the free text was a difficult machine learning task and suggests potential difficulty for human recipients in fully understanding such reports. These results support the prospective assessment of the impact of a fully structured report template with at least some mandatory discrete fields on ease of use of reports and their understanding.",
author = "Marco Huesch and Rekha Cherian and Sam Labib and Rickhesvar Mahraj",
year = "2018",
month = "3",
day = "1",
doi = "10.1016/j.jacr.2017.12.017",
language = "English (US)",
volume = "15",
pages = "554--562",
journal = "Journal of the American College of Radiology",
issn = "1558-349X",
publisher = "Elsevier BV",
number = "3",

}

Evaluating Report Text Variation and Informativeness : Natural Language Processing of CT Chest Imaging for Pulmonary Embolism. / Huesch, Marco; Cherian, Rekha; Labib, Sam; Mahraj, Rickhesvar.

In: Journal of the American College of Radiology, Vol. 15, No. 3, 01.03.2018, p. 554-562.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Evaluating Report Text Variation and Informativeness

T2 - Natural Language Processing of CT Chest Imaging for Pulmonary Embolism

AU - Huesch, Marco

AU - Cherian, Rekha

AU - Labib, Sam

AU - Mahraj, Rickhesvar

PY - 2018/3/1

Y1 - 2018/3/1

N2 - Objective: The aim of this study was to quantify the variability of language in free text reports of pulmonary embolus (PE) studies and to gauge the informativeness of free text to predict PE diagnosis using machine learning as proxy for human understanding. Materials and Methods: All 1,133 consecutive chest CTs with contrast studies performed under a PE protocol and ordered in the emergency department in 2016 were selected from our departmental electronic workflow system. We used commercial text-mining and predictive analytics software to parse and describe all report text and to generate a suite of machine learning rules that sought to predict the “gold standard” radiological diagnosis of PE. Results: There was extensive variation in the length of Findings section and Impression section texts across the reports, only marginally associated with a positive PE diagnosis. A marked concentration of terms was found: for example, 20 words were used in the Findings section of 93% of the reports, and 896 of 2,296 distinct words were each used in only one report's Impression section. In the validation set, machine learning rules had perfect sensitivity but imperfect specificity, a low positive predictive value of 73%, and a misclassification rate of 3%. Conclusion: Use of free text reporting was associated with extensive variability in report length and report terms used. Interpretation of the free text was a difficult machine learning task and suggests potential difficulty for human recipients in fully understanding such reports. These results support the prospective assessment of the impact of a fully structured report template with at least some mandatory discrete fields on ease of use of reports and their understanding.

AB - Objective: The aim of this study was to quantify the variability of language in free text reports of pulmonary embolus (PE) studies and to gauge the informativeness of free text to predict PE diagnosis using machine learning as proxy for human understanding. Materials and Methods: All 1,133 consecutive chest CTs with contrast studies performed under a PE protocol and ordered in the emergency department in 2016 were selected from our departmental electronic workflow system. We used commercial text-mining and predictive analytics software to parse and describe all report text and to generate a suite of machine learning rules that sought to predict the “gold standard” radiological diagnosis of PE. Results: There was extensive variation in the length of Findings section and Impression section texts across the reports, only marginally associated with a positive PE diagnosis. A marked concentration of terms was found: for example, 20 words were used in the Findings section of 93% of the reports, and 896 of 2,296 distinct words were each used in only one report's Impression section. In the validation set, machine learning rules had perfect sensitivity but imperfect specificity, a low positive predictive value of 73%, and a misclassification rate of 3%. Conclusion: Use of free text reporting was associated with extensive variability in report length and report terms used. Interpretation of the free text was a difficult machine learning task and suggests potential difficulty for human recipients in fully understanding such reports. These results support the prospective assessment of the impact of a fully structured report template with at least some mandatory discrete fields on ease of use of reports and their understanding.

UR - http://www.scopus.com/inward/record.url?scp=85040724288&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040724288&partnerID=8YFLogxK

U2 - 10.1016/j.jacr.2017.12.017

DO - 10.1016/j.jacr.2017.12.017

M3 - Article

C2 - 29396123

AN - SCOPUS:85040724288

VL - 15

SP - 554

EP - 562

JO - Journal of the American College of Radiology

JF - Journal of the American College of Radiology

SN - 1558-349X

IS - 3

ER -