An experimental investigation was conducted to determine the relative propulsive and combustion behavior of various hydroxyl terminated polybutadiene (HTPB)-based solid-fuel formulations containing nano-sized energetic particles. In total, 19 solid fuel formulations were investigated. Seventeen formulations contained 13% additive, one contained 6.5%, and one contained 5.65% boron (by weight). Nano-sized particles were cast in an HTPB solid-fuel grain and burned in the Long Grain Center-Perforated (LGCP) hybrid rocket engine using pure oxygen as the oxidizer injected at the head-end of the engine. The addition of nano-sized particles into the solid fuel significantly enhanced performance due to the short ignition (~4 ns) and combustion times (~50 ns) and higher heat release near the surface. It was found that the addition of 13% energetic powders (such as Viton-A coated Alex®) showed an increase of up to 120% in mass burning rate compared to the pure HTPB fuel for an average oxidizer mass flux of 112 kg/m2-s. Chemical and physical analyses were conducted on the nano-sized energetic particles to determine the oxide layer thickness, active aluminum content, and the average particle size. Alex® particles had the highest active aluminum content of 84.6% while minimizing the oxide layer thickness to 4 nm. Viton-A coated aluminum flakes showed a mass burning rate increase of nearly 42% compared to HTPB and twice the increase compared to the uncoated flakes because fluorine and fluorine-containing compounds produced from the dissociation of Viton-A contributed to the rapid ignition and combustion. Boron-based solid fuels showed significant increase (~44%) in mass burning rate which indicates there could be higher energy feedback from the combustion zone to the regressing fuel surface due to boron's higher heat of oxidation. The effect of pressure on the liner regression rate was investigated and was verified to be independent of pressure. Nano-sized Alex® particles (13% by weight) coated with Viton-A had average ηC* ranging from 88 to 92%, where the solid fuel containing micron-sized aluminum particles exhibited combustion efficiencies from 81 to 85%. The boron-containing (5.65% boron) formulation had a molar equivalence of 13% aluminum (by weight). By comparing the 13% boron formulation and the 5.65% boron formulation, it was evident that the 5.65% boron solid fuel had a higher C* combustion efficiency (83%) than the entire range (78 to 81%) for fuels containing 13% boron. The thrust level for the formulation containing Alex® particles (13% by weight) coated with Viton-A was highest among all formulations.