Private Browsing Mode (PBM) is widely supported by all major commodity web browsers. However, browser extensions can greatly undermine PBM. In this paper, we propose an approach to comprehensively identify and stop privacy breaches under PBM caused by browser extensions. Our approach is primarily based on run-time behavior tracking. We combine dynamic analysis and symbolic execution to represent extensions' behavior to identify privacy breaches in PBM caused by extensions. Our analysis shows that many extensions have not fulfilled PBM's guidelines on handling private browsing data. To the best of our knowledge, our approach also provides the first work to stop privacy breaches through instrumentation. We implemented a prototype SoPB on top of Firefox and evaluated it with 1,912 extensions. The results show that our approach can effectively identify and stop privacy breaches under PBM caused by extensions, with almost negligible performance impact.