This paper addresses the problem of robust identification of a class of discrete-time affine hybrid systems, switched affine models, in a set membership framework. Given a finite collection of noisy input/output data and a bound on the number of subsystems, the objective is to identify a suitable set of affine models along with a switching sequence that can explain the available experimental information. Our method builds upon an algebraic procedure proposed by Vidal et al. for noise free measurements. In the presence of norm bounded noise, this algebraic procedure leads to a very challenging nonconvex polynomial optimization problem. Our main result shows that this problem can be reduced to minimizing the rank of a matrix whose entries are affine in the optimization variables, subject to a convex constraint imposing that these variables are the moments of an (unknown) probability distribution function with finite support. Appealing to well known convex relaxations of rank leads to an overall semi-definite optimization problem that can be efficiently solved. These results are illustrated with two examples showing substantially improved identification performance in the presence of noise.