Recent advances in positioning technology have generated massive volume of human mobility data. At the same time, large amount of spatial context data are available and provide us with rich context information. Combining the mobility data with surrounding spatial context enables us to understand the semantics of the mobility records, e.g., what is a user doing at a location (e.g., dining at a restaurant or attending a football game). In this paper, we aim to answer this question by annotating the mobility records with surrounding venues that were actually visited by the user. The problem is non-trivial due to high ambiguity of surrounding contexts. Unlike existing methods that annotate each location record independently, we propose to use all historical mobility records to capture user preferences, which results in more accurate annotations. Our method does not assume the availability to any training data on user preference because of the difficulties to obtain such data in the real-world setting. Instead, we design a Markov random field model to find the best annotations that maximize the consistency of annotated venues. Through extensive experiments on real datasets, we demonstrate that our method significantly outperforms the baseline methods.