Improving image captioning by leveraging knowledge graphs

Yimin Zhou, Yiwei Sun, Vasant Honavar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We explore the use of a knowledge graphs, that capture general or commonsense knowledge, to augment the information extracted from images by the state-of-the-art methods for image captioning. We compare the performance of image captioning systems that as measured by CIDEr-D, a performance measure that is explicitly designed for evaluating image captioning systems, on several benchmark data sets such as MS COCO. The results of our experiments show that the variants of the state-of-the-art methods for image captioning that make use of the information extracted from knowledge graphs can substantially outperform those that rely solely on the information extracted from images.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages283-293
Number of pages11
ISBN (Electronic)9781728119755
DOIs
StatePublished - Mar 4 2019
Event19th IEEE Winter Conference on Applications of Computer Vision, WACV 2019 - Waikoloa Village, United States
Duration: Jan 7 2019Jan 11 2019

Publication series

NameProceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019

Conference

Conference19th IEEE Winter Conference on Applications of Computer Vision, WACV 2019
CountryUnited States
CityWaikoloa Village
Period1/7/191/11/19

Fingerprint

Experiments

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Cite this

Zhou, Y., Sun, Y., & Honavar, V. (2019). Improving image captioning by leveraging knowledge graphs. In Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019 (pp. 283-293). [8658870] (Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/WACV.2019.00036
Zhou, Yimin ; Sun, Yiwei ; Honavar, Vasant. / Improving image captioning by leveraging knowledge graphs. Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 283-293 (Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019).
@inproceedings{a44ad369049e4dbbbdd68341a9231ab2,
title = "Improving image captioning by leveraging knowledge graphs",
abstract = "We explore the use of a knowledge graphs, that capture general or commonsense knowledge, to augment the information extracted from images by the state-of-the-art methods for image captioning. We compare the performance of image captioning systems that as measured by CIDEr-D, a performance measure that is explicitly designed for evaluating image captioning systems, on several benchmark data sets such as MS COCO. The results of our experiments show that the variants of the state-of-the-art methods for image captioning that make use of the information extracted from knowledge graphs can substantially outperform those that rely solely on the information extracted from images.",
author = "Yimin Zhou and Yiwei Sun and Vasant Honavar",
year = "2019",
month = "3",
day = "4",
doi = "10.1109/WACV.2019.00036",
language = "English (US)",
series = "Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "283--293",
booktitle = "Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019",
address = "United States",

}

Zhou, Y, Sun, Y & Honavar, V 2019, Improving image captioning by leveraging knowledge graphs. in Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019., 8658870, Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Institute of Electrical and Electronics Engineers Inc., pp. 283-293, 19th IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, United States, 1/7/19. https://doi.org/10.1109/WACV.2019.00036

Improving image captioning by leveraging knowledge graphs. / Zhou, Yimin; Sun, Yiwei; Honavar, Vasant.

Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 283-293 8658870 (Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Improving image captioning by leveraging knowledge graphs

AU - Zhou, Yimin

AU - Sun, Yiwei

AU - Honavar, Vasant

PY - 2019/3/4

Y1 - 2019/3/4

N2 - We explore the use of a knowledge graphs, that capture general or commonsense knowledge, to augment the information extracted from images by the state-of-the-art methods for image captioning. We compare the performance of image captioning systems that as measured by CIDEr-D, a performance measure that is explicitly designed for evaluating image captioning systems, on several benchmark data sets such as MS COCO. The results of our experiments show that the variants of the state-of-the-art methods for image captioning that make use of the information extracted from knowledge graphs can substantially outperform those that rely solely on the information extracted from images.

AB - We explore the use of a knowledge graphs, that capture general or commonsense knowledge, to augment the information extracted from images by the state-of-the-art methods for image captioning. We compare the performance of image captioning systems that as measured by CIDEr-D, a performance measure that is explicitly designed for evaluating image captioning systems, on several benchmark data sets such as MS COCO. The results of our experiments show that the variants of the state-of-the-art methods for image captioning that make use of the information extracted from knowledge graphs can substantially outperform those that rely solely on the information extracted from images.

UR - http://www.scopus.com/inward/record.url?scp=85063593526&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063593526&partnerID=8YFLogxK

U2 - 10.1109/WACV.2019.00036

DO - 10.1109/WACV.2019.00036

M3 - Conference contribution

AN - SCOPUS:85063593526

T3 - Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019

SP - 283

EP - 293

BT - Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Zhou Y, Sun Y, Honavar V. Improving image captioning by leveraging knowledge graphs. In Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 283-293. 8658870. (Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019). https://doi.org/10.1109/WACV.2019.00036