Identifying Behavior Dispatchers for Malware Analysis

Kyuhong Park, Burak Sahin, Yongheng Chen, Jisheng Zhao, Evan Downing, Hong Hu, Wenke Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Malware is a major threat to modern computer systems. Malicious behaviors are hidden by a variety of techniques: code obfuscation, message encoding and encryption, etc. Countermeasures have been developed to thwart these techniques in order to expose malicious behaviors. However, these countermeasures rely heavily on identifying specific API calls, which has significant limitations as these calls can be misleading or hidden from the analyst. In this paper, we show that malicious programs share a key component which we call a behavior dispatcher, a code structure which is intercepted between various condition checks and malicious actions. By identifying these behavior dispatchers, a malware analysis can be guided into behavior dispatchers and activate hidden malicious actions more easily. We propose BDHunter, a system that automatically identifies behavior dispatchers to assist triggering malicious behaviors. BDHunter takes advantage of the observation that a dispatcher compares an input with a set of expected values to determine which malicious behaviors to execute next. We evaluate BDHunter on recent malware samples to identify behavior dispatchers and show that these dispatchers can help trigger more malicious behaviors (otherwise hidden). Our experimental results show that BDHunter identifies 77.4% of dispatchers within the top 20 candidates discovered. Furthermore, BDHunter-guided concolic execution successfully triggers 13.0x and 2.6x more malicious behaviors, compared to unguided symbolic and concolic execution, respectively. These demonstrate that BDHunter effectively identifies behavior dispatchers, which are useful for exposing malicious behaviors.

Original languageEnglish (US)
Title of host publicationASIA CCS 2021 - Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security
PublisherAssociation for Computing Machinery, Inc
Pages759-773
Number of pages15
ISBN (Electronic)9781450382878
DOIs
StatePublished - May 24 2021
Event16th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2021 - Virtual, Online, Hong Kong
Duration: Jun 7 2021Jun 11 2021

Publication series

NameASIA CCS 2021 - Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security

Conference

Conference16th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2021
Country/TerritoryHong Kong
CityVirtual, Online
Period6/7/216/11/21

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Computer Science Applications
  • Information Systems
  • Software

Fingerprint

Dive into the research topics of 'Identifying Behavior Dispatchers for Malware Analysis'. Together they form a unique fingerprint.

Cite this