Structural Feature Based Anomaly Detection for Packed Executable Identification - Computational Intelligence in Security for Information Systems - page 230

Structural Feature Based Anomaly Detection for
Packed Executable Identification

Xabier Ugarte-Pedrero, Igor Santos, and Pablo G. Bringas

S 3 Lab, DeustoTech - Computing, Deusto Institute of Technology
University of Deusto,
Avenida de las Universidades 24, 48007
Bilbao, Spain
{xabier.ugarte,isantos,pablo.garcia.bringas}@deusto.es

Abstract. Malware is any software with malicious intentions. Commer-
cial anti-malware software relies on signature databases. This approach
has proven to be effective when the threats are already known. However,
malware writers employ software encryption tools and code obfuscation
techniques to hide the actual behaviour of their malicious programs.
One of these techniques is executable packing, which consists of encrypt-
ing the real code of the executable so that it is decrypted in its exe-
cution. Commercial solutions to this problem try to identify the packer
and then apply the corresponding unpacking routine for each packing
algorithm. Nevertheless, this approach fails to detect new and custom
packers. Therefore, generic unpacking methods have been proposed which
execute the binary in a contained environment and gather its actual code.
However, these approaches are very time-consuming and, therefore, a fil-
ter step is required that identifies whether an executable is packed or not.
In this paper, we present the first packed executable detector based on
anomaly detection. This approach represents not packed executables as
feature vectors of structural information and heuristic values. Thereby,
an executable is classified as packed or not packed by measuring its de-
viation to the representation of normality (not packed executables). We
show that this method achieves high accuracy rates detecting packed
executables while maintaining a low false positive rate.

Keywords: malware, anomaly detection, computer security, packer.

1 Introduction

Malware (or malicious software) is defined as computer software that has been
explicitly designed to harm computers or networks. The amount, power and va-
riety of malware programs increases every year, as does its ability to overcome
all kinds of security barriers [1]. Currently, malware writers use executable pack-
ing techniques (cyphering or compressing the actual malicious code) to hide the
actual behaviour of their creations. Packed programs have a decryption routine
that extracts the real payload from memory and then executes it. Currently, up
to the 80 % of the detected malware is packed [2].