SoK: Automatic Deobfuscation of Virtualization-Protected Applications

SoK: Automatic Deobfuscation of Virtualization-Protected Applications

Abstract

Malware authors often rely on code obfuscation to hide the malicious functionality of their software, making detection and analysis more difficult. One of the most advanced techniques for binary obfuscation is virtualization-based obfuscation, which converts the functionality of a program into the bytecode of a randomly generated virtual machine which is embedded into the protected program. To enable the automatic detection and analysis of protected malware, new deobfuscation techniques against virtualization-based obfuscation are constantly being developed and proposed in the literature. In this work, we systematize existing knowledge of automatic deobfuscation of virtualization-protected programs in a novel classification scheme and evaluate where we stand in the arms race between malware authors and code analysts in regards to virtualization-based obfuscation. In addition to a theoretical discussion of different types of deobfuscation methodologies, we present an in-depth practical evaluation that compares state-of-the-art virtualization-based obfuscators with currently available deobfuscation tools. The results clearly indicate the possibility of automatic deobfuscation of virtualization-based obfuscation in specific scenarios. Furthermore, however, the results highlight limitations of existing deobfuscation methods. Multiple challenges still lie ahead on the way towards reliable and resilient automatic deobfuscation of virtualization-based obfuscation.

Grafik Top
Authors
  • Kochberger, Patrick
  • Schrittwieser, Sebastian
  • Schweighofer, Stefan
  • Kieseberg, Peter
  • Weippl, Edgar
Grafik Top
Shortfacts
Category
Paper in Conference Proceedings or in Workshop Proceedings (Paper)
Event Title
The 16th International Conference on Availability, Reliability and Security
Divisions
Security and Privacy
Subjects
Computersicherheit
Angewandte Informatik
Event Location
Virtual Event
Event Type
Conference
Event Dates
17-20 Aug 2021
Series Name
ARES 2021
Publisher
Association for Computing Machinery
Date
2021
Official URL
https://doi.org/10.1145/3465481.3465772
Export
Grafik Top