1. BinaryPig: Scalable Binary Data
Extraction in Hadoop
Created By:
Jason Trost, Telvis Calhoun, Zach Hanif

2. Bringing
data
science
to
cyber
security,
allowing
you
to
sense,
analyze
and
act
in
real
5me.

3. Agenda
• • The Problem
• • BinaryPig Architecture
• • Code and Implementation Details
• • Analysis and Results
• • Demo
• • Wrap-Up
• A

4. Background
2.5 years
20M samples
9.5TB of malware
•

6. Malware data mining is useful
• • Threat intel feeds
• Contextual enrichment on events
• Machine learning models

7. Pre-­‐BinaryPig:
Architecture

8. Pre-­‐BinaryPig:
Storage
Issues
• • We kept running out of disk!
• • We lost samples when NFS nodes failed.

9. Pre-­‐BinaryPig
-­‐
Processing
Issues
• • No Data Locality.
• • Node failures were catastrophic.
• • Hard to add new analysis scripts.

10. Pre-­‐Binary
Pig
-­‐
Data
Explora=on
Issues
• • How can I share my findings for greater fame and glory?
• • Create a table schema for every analysis script?
• • RDBMS failure is worse than zombie apocalypse.

11. We
needed
a
system
that...
• • Scales to our historical data
• • Recovers from failures
• • Grows through scripting
• • Supports dynamic schemas
• • Searchable via the web

12. BinaryPig
FRAMEWORK FOR
PROCESSING SMALL
BINARY FILES USING
APACHE HADOOP AND
APACHE PIG
Bi

13. BinaryPig
• • Simple DSL
• • Pluggable analytics
• • Plays nice with existing tools
• • Enables rapid iteration

14. BinaryPig

15. BinaryPig
-­‐
Storage
• • HDFS, scalable, replicated
• • Aggregate malware samples into sequence files

16. BinaryPig
-­‐
Processing
• • Hadoop - robust, distributed, with data locality
• • Apache Pig - Extensible, simple

17. BinaryPig
-­‐
Results
Explora=on
• • UI - turns your grandma into a data scientist
• • Elasticsearch - schemaless, replicated, awesome

18. Yet
Another
Framework?
• Malware
tools
didn't
scale
• Hadoop
does
not
play
well
with
small
binary
files
• Hadoop
did
not
integrate
exis5ng
malware
analysis
tools

19. Code
and
Implementa5on
Details

20. BinaryPig
is
easy
to
use!

21. BinaryPig
Ingest
Tools
• Generate
sequence
file
from
directory
containing
malware
samples
./bin/dir_to_sequencefile <malwareDir> <hdfsOutputFile>
• Generate
sequence
file
from
archive
./bin/archive_to_sequencefile <archive>
<hdfsOutputFile>

22. BinaryPig
Loaders
• Converts
raw
data
to
a
tuple
• Execu5ng
Loader
o Executes
a
specific
script/program
on
a
file
wriIen
to
a
logical
path
o Example:
Hashing
• Daemon
Loader
o Writes
binaries
to
a
path,
and
provides
those
paths
to
an
already
running
analysis
process
o Example:
Clamd

23. Op=miza=ons
in
BinaryPig
• To
leverage
pre-­‐exis5ng
tools,
we
had
to
write
malware
binaries
to
the
local
filesystem
on
the
worker
nodes
o Note:
local
copy,
not
network
copy
o We
op5mized
this
to
use
/dev/shm/
instead
• Quick
scripts
are
great
for
rapid
itera5on,
but...
o Interpreter
startup
5me
can
dominate
execu5on
5me
o Crea5ng
small,
long
running,
analy5c
daemons
provides
a
huge
speedup
for
frequently
used
tasks
o i.e.
the
clamscand
model
of
execu5on

24. BinaryPig:
Loader
Implementa=ons
• Generic Script Loader
• Generic Daemon Loader
• ClamAV Loader
• Yara Loader
• Hashing Loader

25. strings.sh:
#!/bin/bash
strings "$@"
BinaryPig:
Scrip=ng
strings.pig:
define Loader com.endgame.binarypig.loaders.ExecutingTextLoader;
data = LOAD '$INPUT' USING Loader('strings.sh');
DUMP data;

26. BinaryPig
supports
non-­‐PE32
files
• Handles more than just malware...
o Image analysis
o PDF data extraction
o APK extraction
o Any small binary files

27. Web
Interface

28. Analysis
and
Results

29. Malware
Census
• •
20
Million
unique
binaries
o •
~94%
PE
format
o •
~6%
are
mostly
Android
APK's
• •
5
hours
to
run
historical
set

30. General
Findings

31. Feature
Extrac=on
• Our
core
mo5va5on
was
to
dras5cally
improve
the
experience
of
valida5ng
research.
• Packer
iden5fica5on
o Overall
and
Sec5onal
Entropy
o Kolmogorov
Complexity
o Sec5on
and
resource
names
o Sec5on
flags
• Import
tables
• Func5on
Calls
• Resource
hashes
and
subfeatures

32. Feature
Depth
• PEHeaders
are
shallow
o Easy
to
manipulate
o Less
resolu5on
than
reverse
engineering
features
o File
metadata
is
also
low
resolu5on
• Headers
provide
excellent
fast
features
• Headers
are
oben
ignored
• Work
the
analysis
around
the
feature
resolu5on
o Ignore
5ght
clusters,
go
for
wide
ones
o Triage,
not
true
classifica5on

33. Clustering
Results
• Triage
for
dynamic
analysis
winnowing
• Largest
cluster:
377,882
samples
o Three
malware
families
contained
within
o Second
largest:
124,894
samples
• Valida5on
is
tricky
o Manual
valida5on
cannot
be
en5rely
avoided
o Cluster
meanings
change
with
feature
sets
o Cannot
just
go
off
of
AV
results

34. ICO
Extrac=on

35. Icon
Features
• Pixel
based
features
o Brightness
o Color
values
o Pixel
density
• Cryptographic
and
fuzzy
hashing
o Perceptual
hashes
• Edge
detec5on

36. Icon
Results
• Icon
clustering
o Groups
do
not
just
include
family
lines
o Copycat
malware
is
shown
as
well
o Clear
indica5ons
of
malicious
intent
• Method
of
infec5on
can
be
extrapolated
o Phishing
o Obfuscated
executables
o Adware
(more
than
we
expected)
o False
posi5ves
-­‐
popular
sobware
detec5on

37. Lessons
Learned
• Feature
Selec5on
o Over
500
features
in
PEheader
alone
o Abundance
of
features
requires
pruning
• Interpreta5on
and
Valida5on
of
Results
o Manual
valida5on
is
an
unfortunate
reality
o Care
has
to
be
taken
to
ensure
that
unsupervised
learning
provides
meaningful
results

38. DEMO

39. WRAP
UP

40. • Rapid Iteration
• Feature extraction
• Clustering analysis for rapid malware triage
• Enables weekly AV scans with latest signatures over
previous malware.
• Created binary classifier to improve sample collection
and categorize new samples
So
What?

41. Future
work
• • Compatibility with Pig 0.10.* and 0.11.*
• • EC2 tutorial
• • More examples/starter scripts
o • Inclusion of some of our Mahout tasks
o • Open source process for that is moving forward
• • Better error logging and handling
o Messages should be stored in a separate DB
• • Easier deployments
o • Analytic daemons
o • Dependency libs
o • Fabric/Salt/Puppet/Chef

42. BinaryPig
is
Open
Source!
https://github.com/endgameinc/binarypig
Apache 2 License

43. We
are
hiring!
http://endgame.com/careers

44. QUESTIONS