Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Radiology includes using medical images for detection and diagnosis of diseases as well as guiding further interventions. Chest X-rays are commonly used radiological examinations to help spot thoracic abnormalities or diseases, especially lung-related diseases. However, the reporting of chest x-rays requires experienced radiologists who are often in shortage in many regions of the world. In this paper, we first develop an automatic radiology report generation system. Due to the lack of large annotated radiology report datasets and the difficulty of evaluating the generated reports, the clinical value of such systems is often limited. To this end, we train our report generation network on the small IU Chest X-ray dataset then transfer the learned visual features to classification networks trained on the large ChestX-ray14 dataset and use a novel attention guided feature fusion strategy to improve the detection performance of 14 common thoracic diseases. Through learning the correspondences between different types of feature representations, common features learned by both the report generation and the classification model are assigned with higher attention weights and the weighted visual features boost the performance of state-of-the-art baseline thoracic disease classification networks without altering any learned features. Our work not only offers a new way to evaluate the effectiveness of the learned radiology report generation network, but also proves the possibility of transferring different types of visual representations learned on a small dataset for one task to complement features learned on another large dataset for a different task and improve the model performance.

Original languageEnglish (US)
Title of host publicationInformation Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings
EditorsAlbert C.S. Chung, Siqi Bao, James C. Gee, Paul A. Yushkevich
PublisherSpringer Verlag
Pages125-138
Number of pages14
ISBN (Print)9783030203504
DOIs
StatePublished - Jan 1 2019
Event26th International Conference on Information Processing in Medical Imaging, IPMI 2019 - Hong Kong, China
Duration: Jun 2 2019Jun 7 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11492 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference26th International Conference on Information Processing in Medical Imaging, IPMI 2019
CountryChina
CityHong Kong
Period6/2/196/7/19

Fingerprint

Radiology
X rays
Pulmonary diseases
Performance Model
Medical Image
Shortage
Lung
Fusion reactions
Large Data Sets
Baseline
Fusion
Correspondence
Complement
Evaluate
Vision

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Xue, Y., & Huang, X. (2019). Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation. In A. C. S. Chung, S. Bao, J. C. Gee, & P. A. Yushkevich (Eds.), Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings (pp. 125-138). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11492 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-20351-1_10
Xue, Yuan ; Huang, Xiaolei. / Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation. Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings. editor / Albert C.S. Chung ; Siqi Bao ; James C. Gee ; Paul A. Yushkevich. Springer Verlag, 2019. pp. 125-138 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{0e2c81ab92ca426188ed18a06f0d4634,
title = "Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation",
abstract = "Radiology includes using medical images for detection and diagnosis of diseases as well as guiding further interventions. Chest X-rays are commonly used radiological examinations to help spot thoracic abnormalities or diseases, especially lung-related diseases. However, the reporting of chest x-rays requires experienced radiologists who are often in shortage in many regions of the world. In this paper, we first develop an automatic radiology report generation system. Due to the lack of large annotated radiology report datasets and the difficulty of evaluating the generated reports, the clinical value of such systems is often limited. To this end, we train our report generation network on the small IU Chest X-ray dataset then transfer the learned visual features to classification networks trained on the large ChestX-ray14 dataset and use a novel attention guided feature fusion strategy to improve the detection performance of 14 common thoracic diseases. Through learning the correspondences between different types of feature representations, common features learned by both the report generation and the classification model are assigned with higher attention weights and the weighted visual features boost the performance of state-of-the-art baseline thoracic disease classification networks without altering any learned features. Our work not only offers a new way to evaluate the effectiveness of the learned radiology report generation network, but also proves the possibility of transferring different types of visual representations learned on a small dataset for one task to complement features learned on another large dataset for a different task and improve the model performance.",
author = "Yuan Xue and Xiaolei Huang",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-20351-1_10",
language = "English (US)",
isbn = "9783030203504",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "125--138",
editor = "Chung, {Albert C.S.} and Siqi Bao and Gee, {James C.} and Yushkevich, {Paul A.}",
booktitle = "Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings",
address = "Germany",

}

Xue, Y & Huang, X 2019, Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation. in ACS Chung, S Bao, JC Gee & PA Yushkevich (eds), Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11492 LNCS, Springer Verlag, pp. 125-138, 26th International Conference on Information Processing in Medical Imaging, IPMI 2019, Hong Kong, China, 6/2/19. https://doi.org/10.1007/978-3-030-20351-1_10

Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation. / Xue, Yuan; Huang, Xiaolei.

Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings. ed. / Albert C.S. Chung; Siqi Bao; James C. Gee; Paul A. Yushkevich. Springer Verlag, 2019. p. 125-138 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11492 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation

AU - Xue, Yuan

AU - Huang, Xiaolei

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Radiology includes using medical images for detection and diagnosis of diseases as well as guiding further interventions. Chest X-rays are commonly used radiological examinations to help spot thoracic abnormalities or diseases, especially lung-related diseases. However, the reporting of chest x-rays requires experienced radiologists who are often in shortage in many regions of the world. In this paper, we first develop an automatic radiology report generation system. Due to the lack of large annotated radiology report datasets and the difficulty of evaluating the generated reports, the clinical value of such systems is often limited. To this end, we train our report generation network on the small IU Chest X-ray dataset then transfer the learned visual features to classification networks trained on the large ChestX-ray14 dataset and use a novel attention guided feature fusion strategy to improve the detection performance of 14 common thoracic diseases. Through learning the correspondences between different types of feature representations, common features learned by both the report generation and the classification model are assigned with higher attention weights and the weighted visual features boost the performance of state-of-the-art baseline thoracic disease classification networks without altering any learned features. Our work not only offers a new way to evaluate the effectiveness of the learned radiology report generation network, but also proves the possibility of transferring different types of visual representations learned on a small dataset for one task to complement features learned on another large dataset for a different task and improve the model performance.

AB - Radiology includes using medical images for detection and diagnosis of diseases as well as guiding further interventions. Chest X-rays are commonly used radiological examinations to help spot thoracic abnormalities or diseases, especially lung-related diseases. However, the reporting of chest x-rays requires experienced radiologists who are often in shortage in many regions of the world. In this paper, we first develop an automatic radiology report generation system. Due to the lack of large annotated radiology report datasets and the difficulty of evaluating the generated reports, the clinical value of such systems is often limited. To this end, we train our report generation network on the small IU Chest X-ray dataset then transfer the learned visual features to classification networks trained on the large ChestX-ray14 dataset and use a novel attention guided feature fusion strategy to improve the detection performance of 14 common thoracic diseases. Through learning the correspondences between different types of feature representations, common features learned by both the report generation and the classification model are assigned with higher attention weights and the weighted visual features boost the performance of state-of-the-art baseline thoracic disease classification networks without altering any learned features. Our work not only offers a new way to evaluate the effectiveness of the learned radiology report generation network, but also proves the possibility of transferring different types of visual representations learned on a small dataset for one task to complement features learned on another large dataset for a different task and improve the model performance.

UR - http://www.scopus.com/inward/record.url?scp=85066151592&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066151592&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-20351-1_10

DO - 10.1007/978-3-030-20351-1_10

M3 - Conference contribution

AN - SCOPUS:85066151592

SN - 9783030203504

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 125

EP - 138

BT - Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings

A2 - Chung, Albert C.S.

A2 - Bao, Siqi

A2 - Gee, James C.

A2 - Yushkevich, Paul A.

PB - Springer Verlag

ER -

Xue Y, Huang X. Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation. In Chung ACS, Bao S, Gee JC, Yushkevich PA, editors, Information Processing in Medical Imaging - 26th International Conference, IPMI 2019, Proceedings. Springer Verlag. 2019. p. 125-138. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-20351-1_10