The recent detection of X-ray flares during the afterglow phase of gamma-ray bursts (GRBs) suggests an inner-engine origin, at radii inside the forward shock. There must be inverse Compton (IC) emission arising from such flare photons scattered by forward shock afterglow electrons when they are passing through the forward shock. We find that this IC emission produces high energy gamma-ray flares, which may be detected by AGILE, GLAST and ground-based TeV telescopes. The anisotropic IC scattering between flare photons and forward shock electrons does not affect the total IC component intensity, but cause a time delay of the IC component peak relative to the flare peak. We speculate that this IC component may already have been detected by EGRET from a very strong burst - GRB940217. Future observations by GLAST may help to distinguish whether X-ray flares originate from late central engine activity or from external shocks.