Toward bridging the annotation-retrieval gap in image search by a generative modeling approach

Ritendra Datta, Weina Ge, Jia Li, James Z. Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Citations (Scopus)

Abstract

While automatic image annotation remains an actively pursued research topic, enhancement of image search through its use has not been extensively explored. We propose an annotation-driven image retrieval approach and argue that under a number of different scenarios, this is very effective for semantically meaningful image search. In particular, our system is demonstrated to effectively handle cases of partially tagged and completely untagged image databases, multiple keyword queries, and example based queries with or without tags, all in near-realtime. Because our approach utilizes extra knowledge from a training dataset, it outperforms state-of-the-art visual similarity based retrieval techniques. For this purpose, a novel structure-composition model constructed from Beta distributions is developed to capture the spatial relationship among segmented regions of images. This model combined with the Gaussian mixture model produces scalable categorization of generic images. The categorization results are found to surpass previously reported results in speed and accuracy. Our novel annotation framework utilizes the categorization results to select tags based on term frequency, term saliency, and a WordNet-based measure of congruity, to boost salient tags while penalizing potentially unrelated ones. A bag of words distance measure based on WordNet is used to compute semantic similarity. The effectiveness of our approach is shown through extensive experiments.

Original languageEnglish (US)
Title of host publicationProceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006
Pages977-986
Number of pages10
DOIs
StatePublished - Dec 1 2006
Event14th Annual ACM International Conference on Multimedia, MM 2006 - Santa Barbara, CA, United States
Duration: Oct 23 2006Oct 27 2006

Publication series

NameProceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006

Other

Other14th Annual ACM International Conference on Multimedia, MM 2006
CountryUnited States
CitySanta Barbara, CA
Period10/23/0610/27/06

Fingerprint

Image retrieval
Semantics
Chemical analysis
Experiments

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Media Technology

Cite this

Datta, R., Ge, W., Li, J., & Wang, J. Z. (2006). Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. In Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006 (pp. 977-986). (Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006). https://doi.org/10.1145/1180639.1180856
Datta, Ritendra ; Ge, Weina ; Li, Jia ; Wang, James Z. / Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006. 2006. pp. 977-986 (Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006).
@inproceedings{8941eca8b1e14762ba48b460b536f4f7,
title = "Toward bridging the annotation-retrieval gap in image search by a generative modeling approach",
abstract = "While automatic image annotation remains an actively pursued research topic, enhancement of image search through its use has not been extensively explored. We propose an annotation-driven image retrieval approach and argue that under a number of different scenarios, this is very effective for semantically meaningful image search. In particular, our system is demonstrated to effectively handle cases of partially tagged and completely untagged image databases, multiple keyword queries, and example based queries with or without tags, all in near-realtime. Because our approach utilizes extra knowledge from a training dataset, it outperforms state-of-the-art visual similarity based retrieval techniques. For this purpose, a novel structure-composition model constructed from Beta distributions is developed to capture the spatial relationship among segmented regions of images. This model combined with the Gaussian mixture model produces scalable categorization of generic images. The categorization results are found to surpass previously reported results in speed and accuracy. Our novel annotation framework utilizes the categorization results to select tags based on term frequency, term saliency, and a WordNet-based measure of congruity, to boost salient tags while penalizing potentially unrelated ones. A bag of words distance measure based on WordNet is used to compute semantic similarity. The effectiveness of our approach is shown through extensive experiments.",
author = "Ritendra Datta and Weina Ge and Jia Li and Wang, {James Z.}",
year = "2006",
month = "12",
day = "1",
doi = "10.1145/1180639.1180856",
language = "English (US)",
isbn = "1595934472",
series = "Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006",
pages = "977--986",
booktitle = "Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006",

}

Datta, R, Ge, W, Li, J & Wang, JZ 2006, Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. in Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006. Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006, pp. 977-986, 14th Annual ACM International Conference on Multimedia, MM 2006, Santa Barbara, CA, United States, 10/23/06. https://doi.org/10.1145/1180639.1180856

Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. / Datta, Ritendra; Ge, Weina; Li, Jia; Wang, James Z.

Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006. 2006. p. 977-986 (Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Toward bridging the annotation-retrieval gap in image search by a generative modeling approach

AU - Datta, Ritendra

AU - Ge, Weina

AU - Li, Jia

AU - Wang, James Z.

PY - 2006/12/1

Y1 - 2006/12/1

N2 - While automatic image annotation remains an actively pursued research topic, enhancement of image search through its use has not been extensively explored. We propose an annotation-driven image retrieval approach and argue that under a number of different scenarios, this is very effective for semantically meaningful image search. In particular, our system is demonstrated to effectively handle cases of partially tagged and completely untagged image databases, multiple keyword queries, and example based queries with or without tags, all in near-realtime. Because our approach utilizes extra knowledge from a training dataset, it outperforms state-of-the-art visual similarity based retrieval techniques. For this purpose, a novel structure-composition model constructed from Beta distributions is developed to capture the spatial relationship among segmented regions of images. This model combined with the Gaussian mixture model produces scalable categorization of generic images. The categorization results are found to surpass previously reported results in speed and accuracy. Our novel annotation framework utilizes the categorization results to select tags based on term frequency, term saliency, and a WordNet-based measure of congruity, to boost salient tags while penalizing potentially unrelated ones. A bag of words distance measure based on WordNet is used to compute semantic similarity. The effectiveness of our approach is shown through extensive experiments.

AB - While automatic image annotation remains an actively pursued research topic, enhancement of image search through its use has not been extensively explored. We propose an annotation-driven image retrieval approach and argue that under a number of different scenarios, this is very effective for semantically meaningful image search. In particular, our system is demonstrated to effectively handle cases of partially tagged and completely untagged image databases, multiple keyword queries, and example based queries with or without tags, all in near-realtime. Because our approach utilizes extra knowledge from a training dataset, it outperforms state-of-the-art visual similarity based retrieval techniques. For this purpose, a novel structure-composition model constructed from Beta distributions is developed to capture the spatial relationship among segmented regions of images. This model combined with the Gaussian mixture model produces scalable categorization of generic images. The categorization results are found to surpass previously reported results in speed and accuracy. Our novel annotation framework utilizes the categorization results to select tags based on term frequency, term saliency, and a WordNet-based measure of congruity, to boost salient tags while penalizing potentially unrelated ones. A bag of words distance measure based on WordNet is used to compute semantic similarity. The effectiveness of our approach is shown through extensive experiments.

UR - http://www.scopus.com/inward/record.url?scp=34547151997&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34547151997&partnerID=8YFLogxK

U2 - 10.1145/1180639.1180856

DO - 10.1145/1180639.1180856

M3 - Conference contribution

AN - SCOPUS:34547151997

SN - 1595934472

SN - 9781595934475

T3 - Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006

SP - 977

EP - 986

BT - Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006

ER -

Datta R, Ge W, Li J, Wang JZ. Toward bridging the annotation-retrieval gap in image search by a generative modeling approach. In Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006. 2006. p. 977-986. (Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006). https://doi.org/10.1145/1180639.1180856