Conversational agents (CAs) with voice interfaces are becoming ubiquitous and are routinely used by a wide range of individuals. Users with vision and mobility issues can leverage the voice interfaces of CAs to complete tasks with increasing complexity. However, CAs with voice interfaces pose unique challenges for those with diferent accessibility needs - specifically, older adults with hearing loss. There has been litle work to understand how deaf older adults leverage CAs and the challenges they face while using voice interfaces. To address this gap, we conduct in-depth qualitative interviews with 4 deaf older adults to understand how and why they use CAs. We explore their expectations for their devices and identify common challenges (e.g., default voices used by commercial CAs). We provide suggestions for designing CAs that can beter accommodate a range of hearing abilities and provide the first step forward toward a more inclusive CA.