Structural Feature Based Anomaly Detection for Packed Executable Identification - Computational Intelligence in Security for Information Systems - page 231

Signatures have been applied for the detection of packed executables (e.g.,
PEID [3] and Faster Universal Unpacker (FUU) [4]). However, this approach
has the same shortcoming as signatures for malware detection: it is not effective
with unknown obfuscation techniques or custom packers. Indeed, according to
Morgenstern and Pilz [5], the 35 % of malware is packed by a custom packer.
Dynamic unpacking approaches monitor the execution of a binary in order to
extract its actual code. These methods execute the samples inside an isolated en-
vironment that can be deployed as a virtual machine or an emulator [6]. Several
dynamic unpackers use heuristics to determine the exact point where the exe-
cution jumps from the unpacking routine to the original code and once reached,
bulk the memory content to obtain an unpacked version of the malicious code
(e.g., Universal PE Unpacker [7] and OllyBonE [8]). Notwithstanding, concrete
heuristics cannot be applied to all the packers in the wild, as all of them work
in very different manners. In contrast, not so highly heuristic-dependant ap-
proaches have been proposed for generic dynamic unpacking (e.g., PolyUnpack
[9], Renovo [10], OmniUnpack [11] and Eureka [12]). Nonetheless, these methods
are very tedious and time-consuming, and cannot counter conditional execution
of unpacking routines, a technique used for anti-debugging and anti-monitoring
defense [13,14,15]. Another common approach is using the structural informa-
tion of the PE executables to determine if the sample under analysis is packed
or if it is suspicious of containing malicious code (e.g., PE-Miner [16], PE-Probe
[17] and Perdisci et al. [18]).
In light of this background, we propose here the first method that applies
anomaly detection to packed executable filtering as a previous phase to dynamic
and generic unpacking. This approach is able to determine whether an executable
is packed or not by comparing some structural features with a dataset composed
only of not packed executables. If the executable under inspection presents a
considerable deviation to what it is considered as usual (not packed executables),
it is considered suspicious and is isolated for a further analysis. This method does
not need updated data about packed executables, and thus, it reduces the efforts
of labelling executables manually.
Summarising, our main contributions are: (i) we select a set of structural
characteristics extracted from PE executables to determine whether a sample is
packed or not and provide a relevance measure for each characteristic based on
information gain, (ii) we propose an anomaly-detection-based architecture for
packed executable filtering, by means of weighted comparison against a dataset
composed of only not packed executables and (iii) we evaluate the method using
three different deviation measures.
2 Structural Features of the Portable Executable Files
Given the conclusions obtained in previous work [16,17,18], we selected a set
of 211 structural features from the PE executables. Some of the features were
obtained directly from the PE file header while the rest are calculated values
based on certain heuristics commonly used by the research community. Shafiq
et al. [17] used PE executable structural features were used to determine if an