This paper addresses the problem of robust identification of a class of discrete-time affine hybrid systems, switched affine models, in a set membership framework. Given a finite collection of noisy input/output data and some minimal a priori information about the set of admissible plants, the objective is to identify a suitable set of affine models along with a switching sequence that can explain the available experimental information, while optimizing a performance criteria (either minimum number of switches or minimum number of plants). Our main result shows that this problem can be reduced to a sparsification form, where the goal is to maximize sparsity of a given vector sequence. Although in principle this leads to an NP-hard problem, as we show in the paper, efficient convex relaxations can be obtained by exploiting recent results on sparse signal recovery. These results are illustrated using two non-trivial problems arising in computer vision applications: video-shot and dynamic texture segmentation.