DeepReturn: A deep neural network can learn how to detect previously-unseen ROP payloads without using any heuristics

Xusheng Li, Zhisheng Hu, Haizhou Wang, Yiwei Fu, Ping Chen, Minghui Zhu, Peng Liu

Research output: Contribution to journalArticlepeer-review

Abstract

Return-oriented programming (ROP) is a code reuse attack that chains short snippets of existing code to perform arbitrary operations on target machines. Existing detection methods against ROP exhibit unsatisfactory detection accuracy and/or have high runtime overhead. In this paper, we present DeepReturn, which innovatively combines address space layout guided disassembly and deep neural networks to detect ROP payloads. The disassembler treats application input data as code pointers and aims to find any potential gadget chains, which are then classified by a deep neural network as benign or malicious. Our experiments show that DeepReturn has high detection rate (99.3%) and a very low false positive rate (0.01%). DeepReturn successfully detects all of the 100 real-world ROP exploits that are collected in-the-wild, created manually or created by ROP exploit generation tools. DeepReturn is non-intrusive and does not incur any runtime overhead to the protected program.

Original languageEnglish (US)
Pages (from-to)499-523
Number of pages25
JournalJournal of Computer Security
Volume28
Issue number5
DOIs
StatePublished - 2020

All Science Journal Classification (ASJC) codes

  • Software
  • Safety, Risk, Reliability and Quality
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'DeepReturn: A deep neural network can learn how to detect previously-unseen ROP payloads without using any heuristics'. Together they form a unique fingerprint.

Cite this