Beyond Likert scales: Exploring designers' perceptions through visual reflection activities

When it comes to the assessment of design behaviors and outcomes, direct observations by external viewers and subjective reflections by the participants themselves can all yield important information. External viewers, for example, may code video evidence or apply design metrics to a designer's solutions, both of which can lead to interesting statistical analyses and detailed insights. In collecting and analyzing designers' personal reflections and perceptions, researchers often utilize Likert-type scales, multiple choice questions, or short open-ended prompts. While these modes of data collection are useful and valid, they also constrain participants' responses to fixed options in the case of Likert-type scales and multiple-choice questions, and to verbal expressions in the case of open-ended prompts. Few examples of other types of reflection activities (e.g., graphing, sketching) have been presented or studied in the engineering education literature. In this project, which is part of a larger investigation into high performance design teams, we explored the use of graphing and other visual techniques for recording designers' perceptions of their design processes and products. Our primary aim was to introduce greater richness into the evaluation of designers' behaviors and outcomes as we posed research questions about their relationship to cognitive variables. In this paper, we will discuss two of these reflection activities-an emotional state plot and a graphical assessment of a design solution's feasibility, usefulness, and novelty-in the context of a team design challenge. While the designers under study worked as teams, each individual designer provided her/his perceptions of the team process and design solutions using these reflection activities. Our analysis of individual perceptions from seven design teams of 4-5 designers each will be reported here. This investigation also includes the cognitive style of each participant (as measured by Kirton's Adaption-Innovation inventory-KAI^®) as a potential mediating variable on the individual perceptions of emotions and outcomes. This paper will provide details of the two reflection activities, our associated methods of analysis, key findings related to the designers' perceptions and cognitive styles, benefits and limitations of these two reflection activities, and implications for design educators.