This presentation was delivered at Art into Science 2017 in Austin, TX. I discuss the ongoing cognitive crisis in information security, and present original research methods and results related to the investigation process.
3. Symptoms of a Cognitive Crisis
1. Demand for expertise greatly outweights
supply
2. Most information cannot be trusted or
validated
3. Inability to mobilize and tackle big systemic
issues
4. Ethnography of the SOC
“An analyst’s job is highly
dynamic and requires dealing
with constantly evolving threats.
Doing the job is more art than
science. Ad hoc, on-the-job
training for new analysts is the
norm."
Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to
studying CSIRTs. Network, 100, 2.
5. Ethnography of the SOC
“The profession [security] is so
nascent that the how-tos have
not been fully realized even by
the people who have the
knowledge…the process
required to connect the dots is
unclear even to analysts.
Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to
studying CSIRTs. Network, 100, 2.
6. Symptoms of a Cognitive Crisis
1. Demand for expertise greatly outweights
supply
2. Most information cannot be trusted or
validated
3. Inability to mobilize and tackle big systemic
issues
7. The Cognitive Revolution
1. Understand the
processes used to draw
conclusions
2. Develop repeatable
methods and techniques
3. Build and advocate
training that teaches
practitioners how to
think
13. Investigations as Mental
Labyrinths
The investigation is
the core construct of
information security.
How do we study
them when everyone
has a different
toolset?
Follow the Data!
Alert
OSINT
Reputation
File Hash
Sandbox
Behaviors
AV Detections
(VT)
Imphash
More File
Hashes
Friendly Host
Network PCAP
Host
Windows
Logs
Security Log
System Log
App LogRegistry
File System
Hostile Host Network
PCAP
Flow
16. What data did analysts look at
first?
72%
16%
12%
Observed
PCAP Flow OSINT
Data Suggests:
Analysts prefer a higher context data set…
…even if other data sets are available
…even if lower context data sets can lead to a
resolution.
17. Did the first move affect analysis
speed?
Data Suggests:
While PCAP provides richer context, it may slow down
the investigation if that’s where you start
Starting with a lower context data source can increase
speed when working with higher context data
16
10
9
PCAP Flow OSINT
Avg Time to Close
18. What happens when Bro data
replaces PCAP?
46%
25%
29%
Observed (Bro)
Bro Flow OSINT
72%
16%
12%
Observed (PCAP)
PCAP Flow OSINT
19. What happens when Bro data
replaces PCAP?
16
10
9
PCAP Flow OSINT
Avg Time to Close (PCAP)
10 10 11
Bro Flow OSINT
Avg Time to Close (Bro)
Data Suggests:
Better organization of high context data sources
can yield improvements in analysts performance
20. What data sources were viewed
most and least frequently?
Data Suggests:
Network data is used more frequently than host data…
…even when host data can be used exclusively to resolve.
…even when easy access is provided to host sources.
Revisting data is more prevalent on higher context data
sources
Data Sources Viewed Data Sources Revisited
PCA
P
84%
Flow
11%
OSIN
T
5%
21. How many steps were taken to
make a disposition judgement?
Data Suggests:
At some point, the number of data sources you
investigate impacts the speed of the investigation
Understanding where data exists and when to use it
can impact analysis speed
6
12
9
3
0
5
10
15
6-10 11-15 16-20 21-25
Number of Steps
9
12
14
24
0
5
10
15
20
25
30
6-10 11-15 16-20 21-25
Avg Time to Close
22. Did analysts investigate friendly or
hostile systems first?
9%
91%
Observed
Friendly Hostile
Data Suggests:
Analysts are more compelled to investigate unknown external
threats than internal systems
Analysts don’t fully understand their own techniques
41%
59%
Friendly
Friendly Hostile
Every town had one doctor and they were also your vet
Many home remedies spawn from this time – milk as a treatment for stomach ulcers is an example
Major health crises were frequent and impossible to control
Anthroplogists Ethnography
Is this an individual thing, or is it a systemic problem?
Every town had one doctor and they were also your vet
Many home remedies spawn from this time – milk as a treatment for stomach ulcers is an example
Major health crises were frequent and impossible to control
We ended up with an investigation game
Sidebar: Analysts looked at the PCAP 100% of the time, even if it wasn’t necessary.
This points to tendencies gained from training. Most shops don’t have easy access to host data.
Anecdotal – Experts I knew took less than 10 steps.
Anecdotal – Novices I knew took > 15.