Nowadays, micro-blogging sites such as Twitter have become powerful tools for communicating with others in various situations. Especially in disaster events, these sites can be the best platforms for seeking or providing social support, of which informational support and emotional support are the most important types. Sympathy, a sub-type of emotional support, is an expression of one's compassion or sorrow for a difficult situation that another person is facing. Providing sympathy to people affected by a disaster can help change people's emotional states from negative to positive emotions, and hence, help them feel better. Moreover, detecting sympathy contents in Twitter can potentially be used for finding candidate donors since the emotion “sympathy” is closely related to people who may be willing to donate. Thus, in this paper, as a starting point, we focus on detecting sympathy-related tweets. We address this task using Convolutional Neural Networks (CNNs) with refined word embeddings. Specifically, we propose a refined word embedding technique in terms of various pre-trained word vector models and show great performance of CNNs that use these refined embeddings in the sympathy tweet classification task. We also report experimental results showing that the CNNs with the refined word embeddings outperform not only traditional machine learning techniques, such as Naïve Bayes, Support Vector Machines and AdaBoost with conventional feature sets as bags of words, but also Long Short-Term Memory Networks.