Presentation delivered at UK Azure User Group (22nd Feb 2018)

1. Using Security to drive
Chaos Engineering
Dinis Cruz, CISO
22 Feb 2018

2. What is chaos
engineering

3. Chaos Engineering
Building Confidence
in System Behaviour
through experiments

5. Chaos Engineering is

7. Evolution of Testing
https://www.slideshare.net/NoraJones1/choose-your-own-adventure-qcon-2017-1

8. Chaos Engineering is
about trying controlled
changes to observe system
availability deviation

9. Chaos Engineering is
specifically about Availability

10. Chaos Engineering is
carefully injecting harm into
our systems to test the system’s
ability to respond to it.

11. Chaos Engineering
is the discipline of experimenting on a
distributed system in order to build
confidence in the system’s capability
to withstand turbulent conditions in
production.

12. Chaos Engineering is
Limited scope, continuous,
disaster recovery

13. Back to Chaos
Engineering

14. 1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.
2. Hypothesize that this steady state will continue in both the control group and
the experimental group.
3. Introduce variables that reflect real world events like servers that crash, hard
drives that malfunction, network connections that are severed, etc.
4. Try to disprove the hypothesis by looking for a difference in steady state
between the control group and the experimental group.
Chaos in practice - 4 experiments
http://principlesofchaos.org/

15. 1.Build a Hypothesis around Steady State Behavior
2.Vary Real-world Events
3.Run Experiments in Production
4.Automate Experiments to Run Continuously
5.Minimize Blast Radius
Advanced Principles
http://principlesofchaos.org/

16. the idea that “Chaos
engineering is not Testing”
is caused by
the TDD tragedy

17. Chaos Engineering
is testing
the different are the test abstractions
and the extra random layer

18. Security

19. Cyber Security is a ‘change
generation factory’

20. Security testing are chaos
creators

21. I have been called
Director of chaos :)

22. Do you understand what is
going on in your network?

23. Biggest threat is not the issue,
but is not having visibility

24. When do you know about
security incidents?
(or changes)

25. You need to know what the
attackers are doing on your
system

26. If you don’t know what is on
the pentest report, you have a
big problem
(your SOC team should be able to tell you)

27. Best Security model is one
based on the attacker making
a mistake (i.e. a change)

28. Use risks to understand reality
and to make the business
owners responsible for their
decisions

29. Use Threat Models to
understand how your system
works and to document it

30. Use tests to replicate known
behaviours and simulate
changes
(with and without random events)

31. Properties of resilient
and secure systems

32. Availability

33. Plan for Failure
Ability to sustain failures

34. Validate and Sanitise
all requests

35. Authenticate and Authorise
all requests

36. Reduce capabilities and
features gracefully

37. Hostile to
insecure traffic
and
insecure code

38. Have error budgets
(from Google SRE)

39. Are easy to change

40. Are easy to refactor
(make changes with confidence)

41. Pushes to production
happen minutes
(fully tested and 100x a day (if needed))

42. The bigger they
get the faster they go
(it is smooth and safe to make changes)

43. Have 99% change coverage

44. Change coverage

45. It is not about
test code coverage

46. What matters is
change coverage

47. If you make changes
and they are not detected

48. you are just making
random changes

49. you are an
agent of chaos

50. Every change you make
has to have a respective
test change
(much better pair programming model)

51. Some scenarios

52. If a server or app misbehaves
when do you know about it?

53. If your servers start running
30% slower what happens to
your application?

54. If your servers fail to reboot
after a patch, what happens
to your system?

56. Malicious or api breaking
NuGet package

57. If a server on your cloud is
mining bitcoins would you
know about it

58. is chaos = change?

59. Chaos Engineering
is modern change
management

60. Chaos Engineering
is the programatic
introduction of changes

61. https://en.wikipedia.org/wiki/Change_management_(engineering)

62. should it then be called
Digital Change Engineering?

63. one more thing

64. We are recruiting at Photobox
Group Security

67. Any questions
@DinisCruz