Public speaking anxiety is one of the most common social phobias. We explore the feasibility of using a conversational agent to reduce this anxiety. We developed a public-speaking tutor on the Amazon Alexa platform that enables users to engage in cognitive reconstruction exercises. We also investigated how the sociability of the agent might affect its performance as a tutor. A user study of 53 college students with fear of public speaking showed that the interaction with the agent served to assuage pre-speech state anxiety. Agent sociability improved the sense of interpersonal closeness, which was associated with lower pre-speech anxiety. Moreover, sociability of the agent increased participants' satisfaction and their willingness to continue engagement. Our findings, thus, have implications not only for addressing public speaking anxiety in a scalable way but also for the design of future conversational agents using smart speaker platforms.