Ballast aggregate settlement is generally a result of consolidation or rearrangment of ballast particles in the area underneath crossties. Excessive settlement negatively impacts track performance, resulting in increased risk of train derailment. The purpose of this paper is to compare two methods to evaluate ballast aggregate settlement with repeated loading in railroad: discrete element modeling and laboratory tests using "SmartRock". In this study, ballast aggregates are considered as uniformly graded, angular shaped with crushed faces. For the discrete element modeling, digital imaging techniques are utilized to create the ballast aggregates. Aggregate settlement in railroad ballast and the effect of aggregate shape on the dynamic response of ballast are evaluated through the discrete element simulations. A wireless device, "SmartRock" is developed to study the relationship between individual ballast particle behavior and overall ballast performance. It has a shell of a typical ballast particle shape with force cells attached on the surface and embedded with a tri-axial gyroscope, a tri-axial accelerometer, and a tri-axial magnetometer. The device can move under train traffic like a real ballast particle and record inter-particle contact forces and particle motion in real time. For the laboratory tests, a model-scale track section is constructed and subjected to repeated loading similar to train traffic. The developed "SmartRock" are embedded below rail seats and in the track shoulders. The laboratory data using "SmartRock" can be compared with results from the discrete element modeling in the future. These comparisons will validate the discrete element modeling procedure as a means to analyze railroad ballast aggregate behavior and the potential of "SmartRock" in railroad applications.