Evaluating the discriminatory value and reliability of ideation metrics for their application to concept development and prototyping

Approximately half of new product development projects fail in the market place. Within the product development process, prototyping represents the largest sunk cost; it also remains the least researched and understood. While researchers have recently started to evaluate the impact of formalized prototyping methods and frameworks on end designs, these studies have typically evaluated the success or failure of these methods using binary metrics, and they often evaluate only the design's technical feasibility. Intuitively, we know that a product's success or failure in the marketplace is determined by far more than just the product's technical quality; and yet, we have no clear way of evaluating the design changes and pivots that occur during concept development and prototyping activities, as an explicit set of rigorous and informative metrics to evaluate ideas after concept selection does not exist. The purpose of the current study was to investigate the discriminatory value and reliability of ideation metrics originally developed for concept generation as metrics to evaluate functional prototypes and related concepts developed throughout prototyping activities. Our investigation revealed that new metrics are needed in order to understand the translation of product characteristics, such as originality, novelty, and quality, from original concept through concept development and prototyping to finalized product.