Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty

Faez Ahmed, Mark Fuge, Samuel Todd Hunter, Scarlett Rae Miller

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form "Is idea A more similar to idea B or to idea C?" We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

Original languageEnglish (US)
Title of host publication30th International Conference on Design Theory and Methodology
PublisherAmerican Society of Mechanical Engineers (ASME)
ISBN (Electronic)9780791851845
DOIs
StatePublished - Jan 1 2018
EventASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2018 - Quebec City, Canada
Duration: Aug 26 2018Aug 29 2018

Publication series

NameProceedings of the ASME Design Engineering Technical Conference
Volume7

Other

OtherASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2018
CountryCanada
CityQuebec City
Period8/26/188/29/18

Fingerprint

Excel
Evaluation
Computational methods
Scoring
Experimental Evaluation
Computational Methods
Demonstrate
Polygon
Creativity
Uniqueness
Metric
Unknown
Similarity
Human
Methodology
Design
Judgment
Knowledge
Truth
Form

All Science Journal Classification (ASJC) codes

  • Mechanical Engineering
  • Computer Graphics and Computer-Aided Design
  • Computer Science Applications
  • Modeling and Simulation

Cite this

Ahmed, F., Fuge, M., Hunter, S. T., & Miller, S. R. (2018). Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty. In 30th International Conference on Design Theory and Methodology (Proceedings of the ASME Design Engineering Technical Conference; Vol. 7). American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/DETC2018-85470
Ahmed, Faez ; Fuge, Mark ; Hunter, Samuel Todd ; Miller, Scarlett Rae. / Unpacking subjective creativity ratings : Using embeddings to explain and measure idea novelty. 30th International Conference on Design Theory and Methodology. American Society of Mechanical Engineers (ASME), 2018. (Proceedings of the ASME Design Engineering Technical Conference).
@inproceedings{42a7bbb9b57d48d3999d775efa98c5e6,
title = "Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty",
abstract = "Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form {"}Is idea A more similar to idea B or to idea C?{"} We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.",
author = "Faez Ahmed and Mark Fuge and Hunter, {Samuel Todd} and Miller, {Scarlett Rae}",
year = "2018",
month = "1",
day = "1",
doi = "10.1115/DETC2018-85470",
language = "English (US)",
series = "Proceedings of the ASME Design Engineering Technical Conference",
publisher = "American Society of Mechanical Engineers (ASME)",
booktitle = "30th International Conference on Design Theory and Methodology",

}

Ahmed, F, Fuge, M, Hunter, ST & Miller, SR 2018, Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty. in 30th International Conference on Design Theory and Methodology. Proceedings of the ASME Design Engineering Technical Conference, vol. 7, American Society of Mechanical Engineers (ASME), ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2018, Quebec City, Canada, 8/26/18. https://doi.org/10.1115/DETC2018-85470

Unpacking subjective creativity ratings : Using embeddings to explain and measure idea novelty. / Ahmed, Faez; Fuge, Mark; Hunter, Samuel Todd; Miller, Scarlett Rae.

30th International Conference on Design Theory and Methodology. American Society of Mechanical Engineers (ASME), 2018. (Proceedings of the ASME Design Engineering Technical Conference; Vol. 7).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Unpacking subjective creativity ratings

T2 - Using embeddings to explain and measure idea novelty

AU - Ahmed, Faez

AU - Fuge, Mark

AU - Hunter, Samuel Todd

AU - Miller, Scarlett Rae

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form "Is idea A more similar to idea B or to idea C?" We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

AB - Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form "Is idea A more similar to idea B or to idea C?" We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

UR - http://www.scopus.com/inward/record.url?scp=85056888136&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056888136&partnerID=8YFLogxK

U2 - 10.1115/DETC2018-85470

DO - 10.1115/DETC2018-85470

M3 - Conference contribution

AN - SCOPUS:85056888136

T3 - Proceedings of the ASME Design Engineering Technical Conference

BT - 30th International Conference on Design Theory and Methodology

PB - American Society of Mechanical Engineers (ASME)

ER -

Ahmed F, Fuge M, Hunter ST, Miller SR. Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty. In 30th International Conference on Design Theory and Methodology. American Society of Mechanical Engineers (ASME). 2018. (Proceedings of the ASME Design Engineering Technical Conference). https://doi.org/10.1115/DETC2018-85470