In safety critical systems, such as air traffic control systems and nuclear power plants, accidents are prevented in part by using system control to move the system from undesirable states to desirable states. This “system control-ability” is a result of the system’s composition and configuration and can potentially be used as a safety indicator. In this paper, we propose a probabilistic metric to evaluate a system’s controllability, Probabilistic System Controllability (PSC). We develop a framework for quantitative safety assessment in complex systems using PSC. To demonstrate the metric’s utility, we apply the framework to three different system configurations for collision avoidance air traffic control systems. The result shows that, in a pending collision scenario, a system with both automation and controller has lower controllability, than a system with automation only. This conclusion reinforces the observation of the Überlingen Mid-Air Collision Accident that redundancy in systems can result in less safety.