Multimodal recurrent model with attention for automated radiology report generation

Yuan Xue, Tao Xu, L. Rodney Long, Zhiyun Xue, Sameer Antani, George R. Thoma, Sharon Xiaolei Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology report generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.

Original languageEnglish (US)
Title of host publicationMedical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings
EditorsJulia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, Gabor Fichtinger, Alejandro F. Frangi
PublisherSpringer Verlag
Pages457-466
Number of pages10
ISBN (Print)9783030009274
DOIs
StatePublished - Jan 1 2018
Event21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 - Granada, Spain
Duration: Sep 16 2018Sep 20 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11070 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018
CountrySpain
CityGranada
Period9/16/189/20/18

Fingerprint

Radiology
Medical Image
Generative Models
Memory Term
X rays
Model
Workload
Baseline
Encoding
Decision Making
Magnetic resonance imaging
Neural Networks
Metric
Decision making
Evaluation
Experimental Results
Neural networks

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Xue, Y., Xu, T., Rodney Long, L., Xue, Z., Antani, S., Thoma, G. R., & Huang, S. X. (2018). Multimodal recurrent model with attention for automated radiology report generation. In J. A. Schnabel, C. Davatzikos, C. Alberola-López, G. Fichtinger, & A. F. Frangi (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings (pp. 457-466). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11070 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-00928-1_52
Xue, Yuan ; Xu, Tao ; Rodney Long, L. ; Xue, Zhiyun ; Antani, Sameer ; Thoma, George R. ; Huang, Sharon Xiaolei. / Multimodal recurrent model with attention for automated radiology report generation. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. editor / Julia A. Schnabel ; Christos Davatzikos ; Carlos Alberola-López ; Gabor Fichtinger ; Alejandro F. Frangi. Springer Verlag, 2018. pp. 457-466 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{3cdeaa9d68cf4ea7826da49cb1a745cf,
title = "Multimodal recurrent model with attention for automated radiology report generation",
abstract = "Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology report generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.",
author = "Yuan Xue and Tao Xu and {Rodney Long}, L. and Zhiyun Xue and Sameer Antani and Thoma, {George R.} and Huang, {Sharon Xiaolei}",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-00928-1_52",
language = "English (US)",
isbn = "9783030009274",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "457--466",
editor = "Schnabel, {Julia A.} and Christos Davatzikos and Carlos Alberola-L{\'o}pez and Gabor Fichtinger and Frangi, {Alejandro F.}",
booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings",
address = "Germany",

}

Xue, Y, Xu, T, Rodney Long, L, Xue, Z, Antani, S, Thoma, GR & Huang, SX 2018, Multimodal recurrent model with attention for automated radiology report generation. in JA Schnabel, C Davatzikos, C Alberola-López, G Fichtinger & AF Frangi (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11070 LNCS, Springer Verlag, pp. 457-466, 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018, Granada, Spain, 9/16/18. https://doi.org/10.1007/978-3-030-00928-1_52

Multimodal recurrent model with attention for automated radiology report generation. / Xue, Yuan; Xu, Tao; Rodney Long, L.; Xue, Zhiyun; Antani, Sameer; Thoma, George R.; Huang, Sharon Xiaolei.

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. ed. / Julia A. Schnabel; Christos Davatzikos; Carlos Alberola-López; Gabor Fichtinger; Alejandro F. Frangi. Springer Verlag, 2018. p. 457-466 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11070 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Multimodal recurrent model with attention for automated radiology report generation

AU - Xue, Yuan

AU - Xu, Tao

AU - Rodney Long, L.

AU - Xue, Zhiyun

AU - Antani, Sameer

AU - Thoma, George R.

AU - Huang, Sharon Xiaolei

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology report generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.

AB - Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology report generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.

UR - http://www.scopus.com/inward/record.url?scp=85054079960&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054079960&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-00928-1_52

DO - 10.1007/978-3-030-00928-1_52

M3 - Conference contribution

AN - SCOPUS:85054079960

SN - 9783030009274

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 457

EP - 466

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings

A2 - Schnabel, Julia A.

A2 - Davatzikos, Christos

A2 - Alberola-López, Carlos

A2 - Fichtinger, Gabor

A2 - Frangi, Alejandro F.

PB - Springer Verlag

ER -

Xue Y, Xu T, Rodney Long L, Xue Z, Antani S, Thoma GR et al. Multimodal recurrent model with attention for automated radiology report generation. In Schnabel JA, Davatzikos C, Alberola-López C, Fichtinger G, Frangi AF, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Springer Verlag. 2018. p. 457-466. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-00928-1_52