Navigating the Unknowable: Resilience through Security Chaos Engineering When applied to Cyber Security, Chaos Engineering is advancing our ability to reveal objective information about the effectiveness of operational security measures proactively through empirical experimentation. In this session we will introduce the core concepts behind this new technique and how you can get started in building and applying it.

1. Navigating the
Unknowable: Creating
Resilience with Security
Chaos Engineering
Aaron Rinehart, CTO and Founder, Verica
Jamie Dicken, Manager of Applied Security, Cardinal Health

2. About Us

3. Agenda
Agenda
• System Engineering is Messy
• A New Approach to Learning
• Intro to Chaos Engineering
• Security Chaos Engineering
• Use Cases
• Case Study – Cardinal Health
• Case Study – UnitedHealth Group
• Getting Started
© 2020 RSA Conference. All Rights Reserved. 3

4. Problem
System Engineering is Messy

5. In the Beginning…
© 2020 RSA Conference. All Rights Reserved. 5

6. After a few months…
© 2020 RSA Conference. All Rights Reserved. 6
Hard Coded Passwords
Identity Conflicts
Lead Software Engineer
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution Errors
Expired Certificate
Regulatory
Audit
Rolling Sev1 Outage on
Portal
Code Freeze
Bolt-on batch jobs
Portal Retry Storm Outage

7. Years?
© 2020 RSA Conference. All Rights Reserved. 7
Hard Coded Passwords
Identity Conflicts
Lead Software Engineer
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outag
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution Errors
Expired Certificate
Regulatory
Audit
Rolling Sev1 Outage on
Portal
Code Freeze
Orphaned documentation
Migration to new CSP
Exposed secrets
FW Rule Misconfiguration
Bolt-on batch jobs
Portal Retry Storm Outage
3000 Microservices Δ-> 4000 Microservices
WAF Outage -> Disabled
Large Customer
Outage
DNS Resolution Errors
Portal Retry Storm Outage
Scalability Issues
Delayed Features
Expired Certificate
Code Freeze
Delayed Features
Rolling Sev1 Outage on
Portal
Code Freeze
Bolt-on batch jobs
WAF Outage -> Disabled
Large Customer
Outage
Lead Software Engineer
finds a new job at Google
Scalability Issues
Stability Issues
Full-Scale Customer
Outage
Autoscaling Keeps
Breaking
Bolt-on batch jobs
Bolt-on batch jobs

8. The
Design-Oriented
Mindset is
Old-School

9. Forget Something?
© 2020 RSA Conference. All Rights Reserved. 9
•
•
•
•
•

10. The Reality
“The only way to
understand complex
system is to interact
with it.”
- Dave Snowden
© 2020 RSA Conference. All Rights Reserved. 10

11. A New Approach
to Learning

12. What is
Learning?
Continuous Learning
!=
Continuous Fixing
© 2020 RSA Conference. All Rights Reserved. 12

13. Think
About It…
How does a System
become stable?
© 2020 RSA Conference. All Rights Reserved. 13

14. A Change
in Mindset
People operate
differently when they
expect things to fail.
© 2020 RSA Conference. All Rights Reserved. 14

15. Instrumenting
Chaos
© 2020 RSA Conference. All Rights Reserved. 15
Testing
vs
Experimentation

16. Chaos
Engineering
It’s about establishing order from Chaos

17. Security Chaos
Engineering

18. Why Security
Chaos
Engineering?
Understand your
system and where
its security gaps are
before an adversary
does.
© 2020 RSA Conference. All Rights Reserved. 18

19. Why Security
Chaos
Engineering?
We often
misremember what
our systems really
are. As a result, the
opportunity for
accidents and
mistakes increases.
© 2020 RSA Conference. All Rights Reserved. 19

20. Continuous
Security
Verification
Reduce uncertainty
by building
confidence in how
the system actually
functions.
© 2020 RSA Conference. All Rights Reserved. 20

21. Security Chaos
Engineering
Use Cases

22. Use Cases
•Incident Response
•Security Control Validation
•Security Observability
•Compliance Monitoring
© 2020 RSA Conference. All Rights Reserved. 22

23. Security Chaos
Engineering in
Action
Applied Security at Cardinal Health
ChaoSlingr at UnitedHealth Group

24. Cardinal
Health:
Applied
Security
Assumption: The tools, technical designs,
and technology standards we have keep us
secure.
Reality:
© 2020 RSA Conference. All Rights Reserved.
48% of Data
Breaches are due
to failures of those
controls
IBM Security 2020 Cost of a Data Breach Report

25. Cardinal Health: Applied Security
© 2020 RSA Conference. All Rights Reserved. 25

26. Applied Security Goals for
Identifying Security Gaps
© 2020 RSA Conference. All Rights Reserved. 26
Identify
indisputable,
critical security
gaps
Illustrate the
“big picture” of
security gaps
Ensure security
gaps weren’t
re-opened later

27. Continuous
Verification
and
Validation
© 2020 RSA Conference. All Rights Reserved. 27

28. © 2020 RSA Conference. All Rights Reserved. 28
Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately detect,
block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

29. Getting Started
with Security Chaos Engineering

30. Released!
© 2020 RSA Conference. All Rights Reserved. 30

31. Test your security before someone
else does!
31
Software
Testing
Security
Chaos
Engineering
Validates
functional
requirements
Validates
security
requirements
Automated or
Manual
Continuous Execution
Test-Driven Development
opportunities
Ideal pre- and post-deployment
validation

32. © 2020 RSA Conference. All Rights Reserved. 32
Test Opportunities
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

33. Conclusion
The case for Security Chaos Engineering to improve Resilience

34. Thank You!