We propose a quantified asymmetry based method for age estimation. Our method uses machine learning to discover automatically the most discriminative asymmetry feature set from different brain regions and image scales. Applying this regression model on a T1 MR brain image set of 246 healthy individuals (121 females; 125 males, 66 ± 7.5 years old), we achieve a mean absolute error of 5.4 years and a mean signed error of -0.2 years for age estimation on unseen MR images using the stringent leave-15%-out cross validation. Our results show significant changes in asymmetry with aging in the following regions: the posterior horns of the lateral ventricles, the amygdala, the ventral putamen with a nearby region of the anterior inferior caudate nucleus, the basal forebrain, hyppocampus and parahyppocampal regions. We confirm the validity of the age estimation model using permutation test on 30 replicas of the original dataset with randomly permuted ages (with p-value < 0.001). Furthermore, we apply this model to a separate set of MR images containing normal, Alzheimer's disease (AD) and mild cognitive impairment (MCI) subjects. Our results reflect the relative severity of brain pathology between the three subject groups: mean signed age estimation error is 0.6 years for normal controls, 2.2 years for MCI patients, and 4.7 years for AD patients.