How AI, OpenAI, and ChatGPT impact business and software.
Moneysec - Moneyball for Security
1. 1
Moneysec - Moneyball
for Information Security
Phil Agcaoili
Wednesday March 13, 2013
2. 2
Agenda
1st: Moneyball Overview
2nd: Introduction to Moneysec
3rd: Evidence-based Data for Moneysec
Home Run: What can you do with this?
Moneysec Jared Pfost Brian Keefer
Chief Executive Officer Security Architect
ideas borrowed from: Third Defense Leading SaaS Security Company
3. 3
Moneyball
• Reshaping a laggard team into a world-class winner–
with one of the lowest budgets
• “Willingness to re-think baseball: how it is managed,
how it is played, who is best suited to play it, and
why. “– Michael Lewis, author
– Which statistics are most correlated to winning games
– Focus on these success metrics and actually use them for
recruiting, player development, and game-time decisions.
– Full management commitment
4. 4
Correlation
The degree that two measurements or variables
show a tendency to vary together
negative Car Fuel Efficiency
zero positive
# of Car Accidents
Car Value
Car Mileage Car Body Color Car Insurance Cost
5. 5
The Moneyball Formula
(hits + walks) x total bases
runs created =
(at bats + walks)
Bill James’ formula:
“…it implied, specifically, that (professional baseball people) didn’t place enough
value on walks and extra base hits . . . And placed too much value on batting
average and stolen bases.”
“…The details of James’s equation didn’t matter all that much…What mattered
was (a) it was a rational, testable hypothesis; and (b) James made it so clear
and interesting that it provoked a lot of intelligent people to join the
conversation.” p. 78
Metric: On-base Percentage instead of Batting Average
Source: Michael Lewis - Moneyball
7. 7
Moneyball Formula in Action
(hits + walks) x total bases
runs created =
(at bats + walks)
2011
(1,556 + 571) x 2,360
runs created = = 794
(5,635 + 571)
787 actual
9. 9
Oakland A’s
• Teams bid for players in Free Agent market
• Start of 2002 A’s had payroll ~$40M*
• NY Yankees payroll ~$126M*
• So poor teams have no shot at winning, right?
*From “Moneyball”
1999-2001
Team Wins Losses Est Payroll**
NYY 280 203 $257M
OAK 280 205 $70M
**Estimate from baseball-reference.com
10. 10
Billy Beane
• GM Billy Beane defied convention
– He didn’t follow “best practices”
– He made data-drive decisions
– Hired Paul DePodesta
“The evaluation of young baseball players had been taken out
of the hands of old baseball men and placed in the hands
of people who had what Billy valued most . . . a degree in
something other than baseball.” p. 41
“What you don’t do is what the Yankees do. If we do . . . We
lose every time, because they’re doing it with 3 times more
money than we are . . . The poor team was forced to find
bargains: young players and whatever older guys the
market had undervalued.” p. 119
11. 11
Traditional Baseball
• Talent is evaluated by scouts
• Scouts are usually washed-up players
• i.e. “Industry veterans” or “experts”
• Value statements are largely subjective
12. 12
A few word about scouts…
Should we say
outdated?
13. 13
Next-gen Baseball
• Started in 1977
• Bill James wanted to see what influenced
game outcome
– Realized stats created in 1859 didn’t properly
attribute events
14. 14
Key lessons
• Don’t make emotional decisions
– Recognize your bias
• Collect the “right” data
– Look for correlations
• Set reasonable criteria for success
– Don’t overspend
15. 15
Moneyball Metrics
• Not all measures are equally important (80/20)
• Watch out for “analysis paralysis”.
• What are the most meaningful measures?
• Less is more. Allows focus.
• What processes are intuitively managed that could
be better run based on statistical facts?
• How would this affect your culture and how you hire,
develop, promote, and field a security team based
on these insights?
16. 16
Moneyball Metrics
• Key to the value of Moneyball Metrics
True potential = not just measure, but also:
– Track and trend performance over time
– Benchmark performance vs. self (and peers)
– Identify strengths and weaknesses
– Diagnose - understand the interrelationships and
underlying drivers of performance
– Prescribe actions to improve performance
– Establish performance goals for both individuals
and overall security team
– Become “World-Class”
All metrics are worthless – unless you do
something with them.
17. 17
What can we learn from
Moneyball?
“Why didn’t anybody do this before?”
Moneysec
A practical approach to
security investments
19. Problem Statement
• Every organization is competing with
attackers
• Most don’t have Fortune 50 budget
• How can you be effective?
20. 20
Conventional “Wisdom”
• “Everyone knows” that you need
– Firewall
– Anti-virus
– Change passwords frequently
– Prohibit social networking
– Etc.
21. 21
Do they work?
• Port 80 goes through the firewall
• Anti-virus misses custom malware
• Stolen passwords used quickly
• Social networking key to marketing and
employee satisfaction
22. 22
Clearly this is not working
• Do we actually want a new strategy?
• What does winning look like?
• How do we get started?
• Applying new model
• Use the security stats that are out there
– Verizon Data Breach Investigations Report
– Trustwave Global Security Report
– Ponemon Institute Cost of Data Breach Report and Research Studies
– Manidant M-Trends Report
– Symantec reports
– CSO Magazine Global State of Information Security Survey
– Metricon
23. 23
Models for Research and Investment
theory development theory
interpretation hypothesis
data hypothesis testing
• Inductive • Deductive
– starts with data available – starts with theoretical
– concludes with possible framework
hypotheses – concludes with logical
– bottom up data driven deductions
approach – theory driven approach
24. 24
Trends
Motivating
Event
Fix what’s broken
• Hacks and compromise
– Fix what’s already been hacked at your company
• Understand security trends for your industry
– Small and Medium Business beware
– Banks – DDOS, fraud, botnets, and web authentication attacks
– Hospitality – Credit cards, point of sale systems, Wifi, and admin
accounts
– DIB – RSA hack, Adobe/Microsoft 0days, remote access, and phishing
– News – NYT/WSJ, phishing, Oracle Java 0days
– Retail – Open Wifi, POS
– LEA – 0day, social engineering and phishing
– Credit card processors – Phishing and egress traffic
– Websites – Sony (SQL Injection) and exclusion from core security
• Evaluate your threat landscape to prioritize your treatment
strategy
25. 25
2012 Verizon Data Breach Investigations Report (DBIR)
• 5th year of public releases
– Starting in 2008
– 7 total reports (mid-year
supplementals in 2008 and
2009)
• Dataset now contains:
– 8 years of data
26. 26
2012 Verizon Data Breach Investigations Report (DBIR)
2012 Trustwave Global Security Report
In those cases in which an external entity was
necessary for detection, analysis found that attackers
had an average of 173.5 days within the victim’s
environment before detection occurred.
Conversely, organizations that relied on self-detection
were able to identify attackers within their systems an
average of 43 days after initial compromise.
29. 29
2011 Verizon Data Breach Investigations Report (DBIR)
• Eastern Europe takes a
commanding lead
Who are the (external) bad guys?
30. 30
2011 Verizon Data Breach Investigations Report (DBIR)
• Quite a jump in regular
users (was 51% last year)
• % of breaches involving
Finance staff doubled
• % of breaches involving
executives increased from
7% to 11%
Who are the (internal) bad guys?
33. 33
2011 Verizon Data Breach Investigations Report (DBIR)
• 11% of breaches employed some level of social engineering (down from 28% last year)
Social Engineering and Physical Security Trends
34. 34
2011 VZ Data Breach Investigations Report and Moneysec
Device Patch & Config Monitoring
36. 36
Confidential Documents at Risk Study
Key Findings
• The negligent insider seems to pose the greatest risk because of poor
internal controls and improper accessing and transferring confidential
documents.
• Sensitive documents are most at risk at the document and file level.
• Governance tasks or procedures for privilege and access to sensitive
documents need improvement.
• Budget and compliance monitoring procedures are the critical success
factors to achieving good internal controls and governance procedures.
37. 37
Information Leakage
• Ex-employees, partners, and customers
• Over 1/3 due to negligence
• Increasing loss from external collaboration
Percentage cause of data breach
Estimated sources of data breach
2010 CSO
Cost of Data Breach report Global State of Information Security Survey
Ponemon Institute 2010
38. 38
Evidence-based Investments
• Don’t protect everything
– Protect most important data and services
• Small, targeted investments
– Pass the Red Face Test – Reduce Investments through integration
• Antivirus - Forefront
• Full Disk Encryption – Bitlocker
– Patch and harden configs
– Change default credentials and restrict/monitor privileged accounts
– Increase focus on closing the detection and response gap
– Secure development through application testing and code reviews
– Increase awareness and change culture
• Social engineering and phishing
• Destroy what you don’t need
• Treat all endpoints as hostile
• Collapse to cores
– Protect cores really, really well
• Collect your own metrics and apply security as necessary
39. 39
Moneysec Metrics
• You can measure anything! Even intangibles.
• You don’t always need to be exact.
• Reducing uncertainty adds value.
• Having just some data can go a long way to help a decision maker.
Source: Douglas Hubbard
• Use industry data
– You’re not a beautiful snowflake
– Apply what correlates
• Moneysec metrics
– Measure what’s easy
– Set targets
– Justify as needed
– Optimize Cost vs. Target
• More metrics:
– Moneysec Evolved
– Metricon
40. 40
Conclusion
Quick attempt to grossly oversimplify the theme:
•Billy Beane developed new metrics to win for less.
•Use industry security performance data to help prioritize
spending to win for less.
•Internal information security metrics can be applied along with
industry security performance data.
•Measure well. Gather evidence.
•Determine if you need more or less.
•Apply inductive and deductive approaches to make better
investments.
•Use the evidence you have for better investments.
41. 41
Questions & Answers
Phil Agcaoili
Co-Founder & Board Member, Southern CISO Security Council
Distinguished Fellow and Fellows Chairman, Ponemon Institute
Co-Chair, Communication Sector Coordinating Council (CSCC),
Cybersecurity Committee – Technology Sub-Committee
Founding Member, Cloud Security Alliance (CSA)
Inventor & Co-Author, CSA Cloud Controls Matrix, GRC Stack,
Security, Trust and Assurance Registry (STAR), and
Open Certification Framework (OCF)
@hacksec
https://www.linkedin.com/in/philA