Security risk management metrics and methodology. Includes examples of good and bad security metrics using a risk-based approach.

1. Risk Management
Metrics that Matter

2. Ed Bellis
• Co-founder and CTO at Kenna Security, an
automated risk & vulnerability intelligence platform
• Orbitz CISO for 6 years
• 20+ years Info Security experience including
Bank of America, CSC, E&Y
• Contributing Author Beautiful Security
• Frequent speaker at events such as…
About Me

3. Warning
This presentation contains large amounts of data used
for the purpose of proving an information security
theory. No marketers were harmed during the making of
this presentation.

4. You Are What You Measure

5. JET FUEL X PEANUT BUTTER = SHINY
-ALEX HUTTON

6. Inherent Risk Residual Risk
Know & Measure the Difference
vs.
Hint: This is NOT a math formula

7. Inherent Risk: 80
Please Don’t Do This!
Control Effectiveness: 50%
X
Residual Risk: 40

8. JET FUEL X PEANUT BUTTER = SHINY
-ALEX HUTTON

9. Do This Instead
1. Calculate Risk
2. Identify Potential Key Controls
3. ReCalculate Risk

10. The Language Barrier
*source: Cyber Balance Sheet - The Cyentia Institute

11. The Language Barrier
*source: Cyber Balance Sheet -
The Cyentia Institute
What the CISO perceives
as important versus what
the BoD believes is
important often don’t
match and often neither
are actually given.

12. The Language Barrier
*source: Cyber Balance Sheet -
The Cyentia Institute

13. But First…
Threats, Vulnerabilities
& Risks.. oh my!

14. But First… Some Definitions
Threat: A negative scenario you want to avoid.
Threat Actor: the agent that makes the threat happen.
Vulnerabilities: a weakness that can be exploited.
Risk: a negative scenario you want to avoid combined
with its probability & impact.

15. FAIR Example: Risk Taxonomy

16. Integrate or Die

17. Operationalizing Security Risk Management
Measurement + Integration

18. Risk Management Decision Making

19. Selecting the Right Metrics for Risk Management
Risks > Counts
Results > Work
Quantitative Where Possible

20. Know Your Assets
Some Useful Metrics
1.External Asset Coverage
2.Internal Asset Coverage
3.Time to Discover

21. Know Your Business
Some useful metrics here include:
1. System Susceptibility
1. Value to Attackers
2. Vulnerabilities
2. Time to Compromise: How long would it take to compromise any of the key controls for
these assets and applications?
3. Threat Accessibility
1. Access Points and Attack Surface
4. Threat Actor Capability
1. Tools
2. Resources c.
3. Techniques
Does Your Threat Model
Include Alexa Ratings?

22. Know Your Risk
Some Useful Metrics
1.Risk by Asset
2.Risk by Business Unit
3.Trending Risk over Time
4.Mean Time to Risk Reduction
*use targets/goals and mature to SLAs

23. Know Your Resources
Some Useful Metrics
1.Budget Spent on Security Remediation
2.Risk Carried Above Tolerance Level
3.Hours spent per Security Solution

24. Know Your Direction
Some Useful Metrics
1.Risk Reduction by Group Over Time
2.Risk Goal/SLA by Group
3.Cumulative Risk Accepted Over Time

25. Some Not So Useful Metrics
1. Measuring Work AKA “atta boy metrics”
Number of Vulnerabilities Closed
Number of Patches Deployed
Number of Incidents Responded to

26. Some Not So Useful Metrics
2. Measuring Counts “vanity metrics”
Number of Packets Dropped
Number of Malware Detections
Number of IDS Alerts

27. Some Not So Useful Metrics
3. Averages can be a Fool’s Errand
Average Age of Vulnerability
Average Time to Discover
Average Time to Respond
Hint: Averages are skewed by outliers. Medians are your friend.

28. Aging Can Incent Wrong Behavior

29. Remember This?

30. Your Coworkers Have Day Jobs Too
Leverage Existing Tools
• Bug Trackers
• Trouble Ticketing
• Configuration Management
• Continuous Integration & Deployment
Bonus Points: Leverage Existing Tools for Security Purposes

31. Your Coworkers Have Day Jobs Too
Leverage Existing Processes
• Change Management
• Bug Fixing
• Design Reviews
• QA Testing
• Continuous Integration

32. The Payoff
Operationalizing Security Risk Management
Security Teams
Operations Teams
Development Teams
Executive Management
Common Language
Distinct Objectives
Efficiency
Effectiveness

33. References
FAIR Risk Taxonomy: http://www.opengroup.org/subjectareas/security/risk
Cyber Balance Sheet: https://go.focal-point.com/cyber-balance-sheet-report
Risk Management Metrics That Matter: https://blog.kennasecurity.com/
2017/03/creating-risk-management-metrics-that-matter/

34. Q&A