Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty

Faez Ahmed, Mark Fuge, Sam Hunter, Scarlett Miller

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form "Is idea A more similar to idea B or to idea C?" We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

Original languageEnglish (US)
Title of host publication30th International Conference on Design Theory and Methodology
PublisherAmerican Society of Mechanical Engineers (ASME)
ISBN (Electronic)9780791851845
DOIs
StatePublished - Jan 1 2018
EventASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2018 - Quebec City, Canada
Duration: Aug 26 2018Aug 29 2018

Publication series

NameProceedings of the ASME Design Engineering Technical Conference
Volume7

Other

OtherASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2018
CountryCanada
CityQuebec City
Period8/26/188/29/18

All Science Journal Classification (ASJC) codes

  • Mechanical Engineering
  • Computer Graphics and Computer-Aided Design
  • Computer Science Applications
  • Modeling and Simulation

Fingerprint Dive into the research topics of 'Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty'. Together they form a unique fingerprint.

  • Cite this

    Ahmed, F., Fuge, M., Hunter, S., & Miller, S. (2018). Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty. In 30th International Conference on Design Theory and Methodology (Proceedings of the ASME Design Engineering Technical Conference; Vol. 7). American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/DETC2018-85470