Torsional vibration signature analysis has shown the potential to detect shaft cracks during normal operation in rotating equipment. The method tracks changes in the natural torsional vibration frequencies associated with shaft crack propagation. Prototype systems have been installed on two Reactor Coolant Pumps (RCP) at Tennessee Valley Authority Sequoyah Power Plant Unit 1 nuclear reactor to monitor for possible shaft crack initiation and growth. The implemented torsional vibration sensing system is a combination of specially designed and commercial off-the-shelf components. A series of specialized data processing routines are then applied to produce the torsional vibration signal and enhance its quality. Since shaft crack growth is directly related to natural frequency changes, it is necessary to determine the smallest statistically significant natural frequency shift that can be detected to provide the earliest possible warning. The integrated nature of the prototype hardware/software system along with the inaccessibility to the equipment inside the reactor containment building makes it difficult to separate and evaluate the precision of each component. Hence, a simulation-based evaluation was performed to determine the cumulative effect of the measurement system on the uncertainty in the torsional natural frequency estimation. A single degree of freedom simulation model was developed which matched the modal response of the RCP. The model inputs were adjusted to match the spectral statistical characteristics (amplitude and variance) of the RCP torsional vibration. Two natural frequency identification algorithms (an optimization based SDOF algorithm and a random decrement/Prony algorithm) were subsequently applied to one hundred sets of simulation runs. A statistical analysis was then performed on the natural frequency estimates to establish high probability (99.9%) tolerance limits and the smallest statistically significant frequency change, for a 99.9% probability, which can be detected with the prototype system.