The story picturing engine: Finding elite images to illustrate a story using mutual reinforcement

Dhiraj Joshi, James Z. Wang, Jia Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

27 Citations (Scopus)

Abstract

In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme.

Original languageEnglish (US)
Title of host publicationMIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval
Pages119-126
Number of pages8
StatePublished - Dec 1 2004
EventMIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval - New York, NY, United States
Duration: Oct 15 2004Oct 16 2004

Publication series

NameMIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval

Other

OtherMIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval
CountryUnited States
CityNew York, NY
Period10/15/0410/16/04

Fingerprint

Reinforcement
Engines
Markov processes
Semantics

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Joshi, D., Wang, J. Z., & Li, J. (2004). The story picturing engine: Finding elite images to illustrate a story using mutual reinforcement. In MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval (pp. 119-126). (MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval).
Joshi, Dhiraj ; Wang, James Z. ; Li, Jia. / The story picturing engine : Finding elite images to illustrate a story using mutual reinforcement. MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval. 2004. pp. 119-126 (MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval).
@inproceedings{d43dcd38d5074dd0b20f4b6684b4b7d8,
title = "The story picturing engine: Finding elite images to illustrate a story using mutual reinforcement",
abstract = "In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme.",
author = "Dhiraj Joshi and Wang, {James Z.} and Jia Li",
year = "2004",
month = "12",
day = "1",
language = "English (US)",
isbn = "1581139403",
series = "MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval",
pages = "119--126",
booktitle = "MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval",

}

Joshi, D, Wang, JZ & Li, J 2004, The story picturing engine: Finding elite images to illustrate a story using mutual reinforcement. in MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval. MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval, pp. 119-126, MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval, New York, NY, United States, 10/15/04.

The story picturing engine : Finding elite images to illustrate a story using mutual reinforcement. / Joshi, Dhiraj; Wang, James Z.; Li, Jia.

MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval. 2004. p. 119-126 (MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - The story picturing engine

T2 - Finding elite images to illustrate a story using mutual reinforcement

AU - Joshi, Dhiraj

AU - Wang, James Z.

AU - Li, Jia

PY - 2004/12/1

Y1 - 2004/12/1

N2 - In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme.

AB - In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme.

UR - http://www.scopus.com/inward/record.url?scp=15344344406&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=15344344406&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:15344344406

SN - 1581139403

SN - 9781581139402

T3 - MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval

SP - 119

EP - 126

BT - MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval

ER -

Joshi D, Wang JZ, Li J. The story picturing engine: Finding elite images to illustrate a story using mutual reinforcement. In MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval. 2004. p. 119-126. (MIR'04 - Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval).