CLiMB ToolKit: A case study of iterative evaluation in a multidisciplinary project

Rebecca Jane Passonneau, Roberta Blitz, David Elson, Angela Giral, Judith Klavans

Research output: Contribution to conferencePaper

1 Citation (Scopus)

Abstract

Digital image collections in libraries and other curatorial institutions grow too rapidly to create new descriptive metadata for subject matter search or browsing. CLiMB (Computational Linguistics for Metadata Building) was a project designed to address this dilemma that involved computer scientists, linguists, librarians, and art librarians. The CLiMB project followed an iterative evaluation model: each next phase of the project emerged from the results of an evaluation. After assembling a suite of text processing tools to be used in extracting metada, we conducted a formative evaluation with thirteen participants, using a survey in which we varied the order and type of four conditions under which respondents would propose or select image search terms Results of the formative evaluation led us to conclude that a CLiMB ToolKit would work best if its main function was to propose terms for users to review. After implementing a prototype ToolKit using a browser interface, we conducted an evaluation with ten experts. Users found the ToolKit very habitable, remained consistently satisfied throughout a lengthy evaluation, and selected a large number of terms per image.

Original languageEnglish (US)
Pages2413-2418
Number of pages6
StatePublished - Jan 1 2006
Event5th International Conference on Language Resources and Evaluation, LREC 2006 - Genoa, Italy
Duration: May 22 2006May 28 2006

Other

Other5th International Conference on Language Resources and Evaluation, LREC 2006
CountryItaly
CityGenoa
Period5/22/065/28/06

Fingerprint

computational linguistics
evaluation
librarian
text processing
computer scientist
Toolkit
Evaluation
Metadata
Computational Linguistics
expert
art

All Science Journal Classification (ASJC) codes

  • Education
  • Library and Information Sciences
  • Linguistics and Language
  • Language and Linguistics

Cite this

Passonneau, R. J., Blitz, R., Elson, D., Giral, A., & Klavans, J. (2006). CLiMB ToolKit: A case study of iterative evaluation in a multidisciplinary project. 2413-2418. Paper presented at 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy.
Passonneau, Rebecca Jane ; Blitz, Roberta ; Elson, David ; Giral, Angela ; Klavans, Judith. / CLiMB ToolKit : A case study of iterative evaluation in a multidisciplinary project. Paper presented at 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy.6 p.
@conference{6de6b74d08f24d62868c25dbe784be8e,
title = "CLiMB ToolKit: A case study of iterative evaluation in a multidisciplinary project",
abstract = "Digital image collections in libraries and other curatorial institutions grow too rapidly to create new descriptive metadata for subject matter search or browsing. CLiMB (Computational Linguistics for Metadata Building) was a project designed to address this dilemma that involved computer scientists, linguists, librarians, and art librarians. The CLiMB project followed an iterative evaluation model: each next phase of the project emerged from the results of an evaluation. After assembling a suite of text processing tools to be used in extracting metada, we conducted a formative evaluation with thirteen participants, using a survey in which we varied the order and type of four conditions under which respondents would propose or select image search terms Results of the formative evaluation led us to conclude that a CLiMB ToolKit would work best if its main function was to propose terms for users to review. After implementing a prototype ToolKit using a browser interface, we conducted an evaluation with ten experts. Users found the ToolKit very habitable, remained consistently satisfied throughout a lengthy evaluation, and selected a large number of terms per image.",
author = "Passonneau, {Rebecca Jane} and Roberta Blitz and David Elson and Angela Giral and Judith Klavans",
year = "2006",
month = "1",
day = "1",
language = "English (US)",
pages = "2413--2418",
note = "5th International Conference on Language Resources and Evaluation, LREC 2006 ; Conference date: 22-05-2006 Through 28-05-2006",

}

Passonneau, RJ, Blitz, R, Elson, D, Giral, A & Klavans, J 2006, 'CLiMB ToolKit: A case study of iterative evaluation in a multidisciplinary project' Paper presented at 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy, 5/22/06 - 5/28/06, pp. 2413-2418.

CLiMB ToolKit : A case study of iterative evaluation in a multidisciplinary project. / Passonneau, Rebecca Jane; Blitz, Roberta; Elson, David; Giral, Angela; Klavans, Judith.

2006. 2413-2418 Paper presented at 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy.

Research output: Contribution to conferencePaper

TY - CONF

T1 - CLiMB ToolKit

T2 - A case study of iterative evaluation in a multidisciplinary project

AU - Passonneau, Rebecca Jane

AU - Blitz, Roberta

AU - Elson, David

AU - Giral, Angela

AU - Klavans, Judith

PY - 2006/1/1

Y1 - 2006/1/1

N2 - Digital image collections in libraries and other curatorial institutions grow too rapidly to create new descriptive metadata for subject matter search or browsing. CLiMB (Computational Linguistics for Metadata Building) was a project designed to address this dilemma that involved computer scientists, linguists, librarians, and art librarians. The CLiMB project followed an iterative evaluation model: each next phase of the project emerged from the results of an evaluation. After assembling a suite of text processing tools to be used in extracting metada, we conducted a formative evaluation with thirteen participants, using a survey in which we varied the order and type of four conditions under which respondents would propose or select image search terms Results of the formative evaluation led us to conclude that a CLiMB ToolKit would work best if its main function was to propose terms for users to review. After implementing a prototype ToolKit using a browser interface, we conducted an evaluation with ten experts. Users found the ToolKit very habitable, remained consistently satisfied throughout a lengthy evaluation, and selected a large number of terms per image.

AB - Digital image collections in libraries and other curatorial institutions grow too rapidly to create new descriptive metadata for subject matter search or browsing. CLiMB (Computational Linguistics for Metadata Building) was a project designed to address this dilemma that involved computer scientists, linguists, librarians, and art librarians. The CLiMB project followed an iterative evaluation model: each next phase of the project emerged from the results of an evaluation. After assembling a suite of text processing tools to be used in extracting metada, we conducted a formative evaluation with thirteen participants, using a survey in which we varied the order and type of four conditions under which respondents would propose or select image search terms Results of the formative evaluation led us to conclude that a CLiMB ToolKit would work best if its main function was to propose terms for users to review. After implementing a prototype ToolKit using a browser interface, we conducted an evaluation with ten experts. Users found the ToolKit very habitable, remained consistently satisfied throughout a lengthy evaluation, and selected a large number of terms per image.

UR - http://www.scopus.com/inward/record.url?scp=57349087579&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=57349087579&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:57349087579

SP - 2413

EP - 2418

ER -

Passonneau RJ, Blitz R, Elson D, Giral A, Klavans J. CLiMB ToolKit: A case study of iterative evaluation in a multidisciplinary project. 2006. Paper presented at 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy.