We report a systematic study of polarization and magnetic field effects on the optical response of Fe3O4-silicone elastomer composite. The Fe3O4 particles were aligned in a silicone elastomer matrix with an external static magnetic field. Films of composites containing 5wt% of 20nm ≤ d ≤ 30nm Fe3O4 particles aligned in- and out-of-plane in the elastomer host were prepared. The optical spectra of the films were measured with the Perkin-Elmer Lambda 950 UV/vis/NIR spectrometer. We observed a systematic redshift in the optical response of the out-of-plane composite films with increasing static magnetic field strength, which saturated near 600 Gauss. We obtained a maximum redshift of ∼46 nm at 600 Gauss. The observed redshift in the optical response of the out-of-plane composite film is attributed to the effect of the magnetic field. This facilitated the formation of the highly aligned particles that induced strong electric dipole in the aligned particles. Interestingly, there were no observable shifts with increasing magnetic field strength in the in-plane films, suggesting that the orientation (polarization) of the magnetic dipole and the induced electric dipole play a crucial role in the optical response.