Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping

Dongpeng Xu, Jiang Ming, Dinghao Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

19 Citations (Scopus)

Abstract

Cryptographic functions have been commonly abused by malware developers to hide malicious behaviors, disguise destructive payloads, and bypass network-based firewalls. Now-infamous crypto-ransomware even encrypts victim's computer documents until a ransom is paid. Therefore, detecting cryptographic functions in binary code is an appealing approach to complement existing malware defense and forensics. However, pervasive control and data obfuscation schemes make cryptographic function identification a challenging work. Existing detection methods are either brittle to work on obfuscated binaries or ad hoc in that they can only identify specific cryptographic functions. In this paper, we propose a novel technique called bit-precise symbolic loop mapping to identify cryptographic functions in obfuscated binary code. Our trace-based approach captures the semantics of possible cryptographic algorithms with bit-precise symbolic execution in a loop. Then we perform guided fuzzing to efficiently match boolean formulas with known reference implementations. We have developed a prototype called CryptoHunt and evaluated it with a set of obfuscated synthetic examples, well-known cryptographic libraries, and malware. Compared with the existing tools, CryptoHunt is a general approach to detecting commonly used cryptographic functions such as TEA, AES, RC4, MD5, and RSA under different control and data obfuscation scheme combinations.

Original languageEnglish (US)
Title of host publication2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages921-937
Number of pages17
ISBN (Electronic)9781509055326
DOIs
StatePublished - Jun 23 2017
Event2017 IEEE Symposium on Security and Privacy, SP 2017 - San Jose, United States
Duration: May 22 2017May 24 2017

Publication series

NameProceedings - IEEE Symposium on Security and Privacy
ISSN (Print)1081-6011

Other

Other2017 IEEE Symposium on Security and Privacy, SP 2017
CountryUnited States
CitySan Jose
Period5/22/175/24/17

Fingerprint

Binary codes
Computer system firewalls
Semantics
Malware

All Science Journal Classification (ASJC) codes

  • Safety, Risk, Reliability and Quality
  • Software
  • Computer Networks and Communications

Cite this

Xu, D., Ming, J., & Wu, D. (2017). Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping. In 2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings (pp. 921-937). [7958617] (Proceedings - IEEE Symposium on Security and Privacy). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SP.2017.56
Xu, Dongpeng ; Ming, Jiang ; Wu, Dinghao. / Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping. 2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 921-937 (Proceedings - IEEE Symposium on Security and Privacy).
@inproceedings{b2c17c37b2a8444ea1138e8cc85fc9c1,
title = "Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping",
abstract = "Cryptographic functions have been commonly abused by malware developers to hide malicious behaviors, disguise destructive payloads, and bypass network-based firewalls. Now-infamous crypto-ransomware even encrypts victim's computer documents until a ransom is paid. Therefore, detecting cryptographic functions in binary code is an appealing approach to complement existing malware defense and forensics. However, pervasive control and data obfuscation schemes make cryptographic function identification a challenging work. Existing detection methods are either brittle to work on obfuscated binaries or ad hoc in that they can only identify specific cryptographic functions. In this paper, we propose a novel technique called bit-precise symbolic loop mapping to identify cryptographic functions in obfuscated binary code. Our trace-based approach captures the semantics of possible cryptographic algorithms with bit-precise symbolic execution in a loop. Then we perform guided fuzzing to efficiently match boolean formulas with known reference implementations. We have developed a prototype called CryptoHunt and evaluated it with a set of obfuscated synthetic examples, well-known cryptographic libraries, and malware. Compared with the existing tools, CryptoHunt is a general approach to detecting commonly used cryptographic functions such as TEA, AES, RC4, MD5, and RSA under different control and data obfuscation scheme combinations.",
author = "Dongpeng Xu and Jiang Ming and Dinghao Wu",
year = "2017",
month = "6",
day = "23",
doi = "10.1109/SP.2017.56",
language = "English (US)",
series = "Proceedings - IEEE Symposium on Security and Privacy",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "921--937",
booktitle = "2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings",
address = "United States",

}

Xu, D, Ming, J & Wu, D 2017, Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping. in 2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings., 7958617, Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc., pp. 921-937, 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, United States, 5/22/17. https://doi.org/10.1109/SP.2017.56

Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping. / Xu, Dongpeng; Ming, Jiang; Wu, Dinghao.

2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2017. p. 921-937 7958617 (Proceedings - IEEE Symposium on Security and Privacy).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping

AU - Xu, Dongpeng

AU - Ming, Jiang

AU - Wu, Dinghao

PY - 2017/6/23

Y1 - 2017/6/23

N2 - Cryptographic functions have been commonly abused by malware developers to hide malicious behaviors, disguise destructive payloads, and bypass network-based firewalls. Now-infamous crypto-ransomware even encrypts victim's computer documents until a ransom is paid. Therefore, detecting cryptographic functions in binary code is an appealing approach to complement existing malware defense and forensics. However, pervasive control and data obfuscation schemes make cryptographic function identification a challenging work. Existing detection methods are either brittle to work on obfuscated binaries or ad hoc in that they can only identify specific cryptographic functions. In this paper, we propose a novel technique called bit-precise symbolic loop mapping to identify cryptographic functions in obfuscated binary code. Our trace-based approach captures the semantics of possible cryptographic algorithms with bit-precise symbolic execution in a loop. Then we perform guided fuzzing to efficiently match boolean formulas with known reference implementations. We have developed a prototype called CryptoHunt and evaluated it with a set of obfuscated synthetic examples, well-known cryptographic libraries, and malware. Compared with the existing tools, CryptoHunt is a general approach to detecting commonly used cryptographic functions such as TEA, AES, RC4, MD5, and RSA under different control and data obfuscation scheme combinations.

AB - Cryptographic functions have been commonly abused by malware developers to hide malicious behaviors, disguise destructive payloads, and bypass network-based firewalls. Now-infamous crypto-ransomware even encrypts victim's computer documents until a ransom is paid. Therefore, detecting cryptographic functions in binary code is an appealing approach to complement existing malware defense and forensics. However, pervasive control and data obfuscation schemes make cryptographic function identification a challenging work. Existing detection methods are either brittle to work on obfuscated binaries or ad hoc in that they can only identify specific cryptographic functions. In this paper, we propose a novel technique called bit-precise symbolic loop mapping to identify cryptographic functions in obfuscated binary code. Our trace-based approach captures the semantics of possible cryptographic algorithms with bit-precise symbolic execution in a loop. Then we perform guided fuzzing to efficiently match boolean formulas with known reference implementations. We have developed a prototype called CryptoHunt and evaluated it with a set of obfuscated synthetic examples, well-known cryptographic libraries, and malware. Compared with the existing tools, CryptoHunt is a general approach to detecting commonly used cryptographic functions such as TEA, AES, RC4, MD5, and RSA under different control and data obfuscation scheme combinations.

UR - http://www.scopus.com/inward/record.url?scp=85025115790&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85025115790&partnerID=8YFLogxK

U2 - 10.1109/SP.2017.56

DO - 10.1109/SP.2017.56

M3 - Conference contribution

AN - SCOPUS:85025115790

T3 - Proceedings - IEEE Symposium on Security and Privacy

SP - 921

EP - 937

BT - 2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Xu D, Ming J, Wu D. Cryptographic Function Detection in Obfuscated Binaries via Bit-Precise Symbolic Loop Mapping. In 2017 IEEE Symposium on Security and Privacy, SP 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2017. p. 921-937. 7958617. (Proceedings - IEEE Symposium on Security and Privacy). https://doi.org/10.1109/SP.2017.56