The scope of this paper is the challenging task of classifying terrestrial images of buildings, automatically. Straight line segments and their connectivity incorporate significant information about object shapes. Man-made buildings exhibit special generic shapes which are extracted from embedded spatial and angular line segment relationships by cluster analysis. After employing an agglomerative hierarchical cluster analysis we obtain geometrical structure information features on different scales. For the classification process we apply support vector machines (SVM) with polynomial and radial basis function (RBF) kernels to separate the feature space by a hyperplane into 2 classes. The method is applied to an image collection taken from the Corel image database and compared with traditional edge- orientation histogram features. We obtained a 88 % true positive classification rate (recall) with an F-measure value of 81.3 %.