2. Why You Should(n’t) Listen
Michael Roytman
• Data Scientist, Risk I/O
• MS Operations Research, Georgia Tech
• Fraud Detection, Large Bank
• Naive Grad Student Not Too Long Ago
• Still Plays With Legos
• Barely Passed Regression Analysis
3. Roadmap
• The Struggle
• What’s Bad?
• What’s Good?
• Framework
• Data Driven Insights
• Decision-Making
4. Starting From Scratch
“It is a capital mistake to theorize
before one has data. Insensibly one
begins to twist facts to suit theories,
instead of theories to suit facts.”
-Sir Arthur Conan Doyle, 1887
6. Starting From Scratch
Primary Sources!
Twitter!
InfoSec Blogs!
Academia!
• GScholar!
• JSTOR!
• IEEE!
• ProQuest!
• CISOs
CSIOs!
• Pen Testers!
• Threat Reports!
• SOTI/DBIR!
!
Text
• Thought Leaders (you
know who you are)!
• BlackHats!
• Vuln Researchers!
• MITRE!
• OSVDB!
• NIST CVSS
Committee(s)!
• Internal Message
Boards for ^!
7. Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias !
(http://blog.risk.io/2013/04/data-fundamentalism/) !
Jerico/Sushidude @ BlackHat !
(https://www.blackhat.com/us-13/briefings.html#Martin)!
Luca Allodi - CVSS DDOS !
(http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):!
8. Data Fundamentalism - What’s The Big Deal?
!
”Since 2006 Vulnerabilities have declined by 26 percent.” !
(http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)!
!
!
“The total number of vulnerabilities in 2013 is up 16 percent so far when
compared to what we saw in the same time period in 2012. ”!
(http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)!
!
!
9. What’s Good?
Bad For Vulnerability Statistics:!
!
NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on. !
Good For Vulnerability Statistics:!
!
Vulnerabilities. !
21. Defend Like You’ve Done It Before
Groups,
Motivations
Learning
from
Breaches
Asset
Topology,
Actual Vulns
on System
Vulnerability
Definitions
Exploits
22. Work With What You’ve Got:
Akamai, Safenet
NVD,
MITRE
ExploitDB,
Metasploit
27. Duplication
We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities
We Want: F(Number of Scanners) => Vulnerability Coverage
<---------Good Luck!
Make Decisions At The Margins!
100.0
75.0
50.0
25.0
0.0
0
1
2
3
4
5
6
33. CVSS - A VERY General Guide For Remediation - Yep
Open Vulns With Breaches Occuring By Severity
160000.0
120000.0
80000.0
40000.0
0.0
1
2
3
4
5
6
7
8
9
10
34. The One Billion Dollar Question
Probability(You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities)
1.98%
35. I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.00000
0.01000
0.02000
0.03000
0.04000
37. I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0
0.1
0.2
0.2
0.3
40. I Love It When You Call Me Big Data
Spray and Pray => 2%
!
CVSS 10 => 4%
!
Metasploit + ExploitDB => 30%
!
A Good Model That’s Not Built By One Kid
Without Hadoop => ???!
41. Thank You
Don’t Be A Stranger
Blog: http://blog.risk.io
Twitter: @mroytman