Due to widespread adoption of no-till, there is considerable interest in methods of incorporating manure into soil with minimal disturbance. Watershed and nutrient trading programs rely upon empirical water quality findings to promote manure application methods and to extrapolate the water quality benefits of those methods. Rainfall simulation studies have been a key source of input into the water quality benefits of manure management alternatives. Rainfall simulation studies offer a means of controlling key storm variables and selecting the timing of events, often with an eye to representing worst case scenarios. However, the control offered by these studies results in limited representation of management, climatic, and hydrologic variables. To better understand the effect of assessment method on management recommendations, we compared edge-of-field findings from controlled rainfall events with those obtained under natural conditions. Eighteen field plot pairs were arranged in a randomized complete block design with replications blocked by contour strip. Pairs consisted of one 2 x 10 m plot receiving artificial rainfall 72 h after application and one 10 x 10 m plot receiving natural rainfall for the entire year. Flow path lengths for both plots were the same (10 m). Effects of artificial versus natural rainfall on dissolved reactive P and total P losses in runoff were compared across five dairy manure application methods: conventional surface application, surface application with incorporation, banded application with aeration, shallow disk injection, and high pressure injection. Runoff assessment methods affected the differences observed between manure application treatments. The artificial rainfall method created a high-volume rainfall event within three days of manure application whereas no large natural rainfall events occurred within months of manure application. For both artificial and natural rainfall assessment methods, plots receiving manure application yielded less runoff, lower total suspended solids levels, and higher dissolved reactive P losses compared with control plots that did not receive manure. Results from the natural rainfall plots indicate that, in the long term under non-extreme rainfall events, the tillage (chisel plowing followed by culti-mulching) and shallow disk injection methods provide the most reduction in pollutant loss across all environmental variables studied, followed closely by the banded and aerated treatment. Because shallow disk injection simultaneously applies and incorporates manure, it requires fewer passes over the field and may be economically preferable.