It is important to understand the older adults prior to the design process. The understanding can better facilitate design conversations between the researchers and the older adults. In this paper, we discussed our experiences of building a relationship with the older adults in a local senior center. However, we encountered challenges when we tried to engaging the older adults, for example, skepticism towards the research purposes, reluctance to talk, and the difficulty of collecting data under a fun and leisure environment. We learned that creating relationships is important to engage older adults in research. Engaging the older adults in interactive dynamics could facilitate richer conversations with the older adults. Based on our lessons from interacting with the older adults, we propose that building long-term relationships and engaging older adults in fun research and design activities can facilitate a more successful participatory design.