In this paper we highlight the need for privacy of user data used in digital identity management systems. We investigate the issues from the individual, business, and government perspectives. We provide surveys related to the growing problem of identity theft and the sociological concerns of individuals with respect to the privacy of their identity data. We show the privacy concerns, especially with respect to health and biometric data, where the loss of privacy of that data may have serious consequences. Moreover, we also discuss how privacy concerns change according to the individual's disposition to provide the data. Voluntary disclosure of personal information is more acceptable to users than if information disclosure is involuntary, like in the case of surveillance. Finally, we highlight the shortcomings of current identity management systems with respect to the current privacy needs and motivate the need of hardened importance of privacy enabling functionalities in such systems.