Massive X-ray binaries (MXB) in starburst galaxies and reflection-dominated AGN have been proposed as models of the cosmic X-ray background (CXB). We calculate the CXB spectrum to be expected from an evolving population of MXB in starburst galaxies and find that starbursts cannot produce much of the background at low energies (≲30 keV) without significant evolution. We find their contribution at higher energies to be negligible, even with strong evolution. We also investigate the consistency of the MXB and AGN models. If MXB in starburst galaxies do produce much of the low-energy CXB, some other type of source must be responsible for the high-energy background. If reflecting AGN are this other type of source, up to 92% of their emission must be reprocessed. Even without a starburst contribution, the AGN spectrum must still be about 85% reprocessed.