Video of this talk is here: http://www.opscode.com/blog/2012/02/14/automate-all-the-things/

1. “Rebooting a Cloud”
Jesse Robbins
Cofounder, Opscode
@jesserobbins
http://www.flickr.com/photos/mwichary/2355832413/

2. Jesse Robbins
Cofounder, Opscode
@jesserobbins
jesse@opscode.com

3. “Rebooting a Cloud”
http://www.flickr.com/photos/mwichary/2355832413/

4. Opscode is Hiring!

5. Press button,
Receive candy!
(who here wants candy?)
(there is always someone.)
http://www.flickr.com/photos/mwichary/2355832413/

6. “Rebooting a Cloud”
http://www.flickr.com/photos/mwichary/2355832413/

8. Complexity Increases,
Latent Defects Accumulate,
&
FAILURE HAPPENS!

9. define:
The Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds
99.99999% 3 Seconds

10. 99.9% *
99.9% *
99.9%
=
99.7%
(oops!)

11. Everyone goes through
the same thing...

12. Denial
This isn’t happening.

13. ANGER
“HOW THE $^@* COULD
THIS BE HAPPENING!!!”

14. (more anger)
“#@(* #$#$ %@%!!#$@ #*
$#@$ @$*&#($*&@#(*$(”

15. Bargaining
If we do this one thing, it will
stop happening.

16. Depression
What’s the point if this is
just going to happen?

17. Acceptance
“Failure Happens. We’ll just
have to design for it.”

18. Successful companies say:
“Failure Happens”
“Embrace Failure”
“Design For Failure”
“Healthy attitude about Failure”
“Resilient (to Failure)”
THE OUTAGES WILL CONTINUE
UNTIL THE APPROACH
IMPROVES ;-)

19. GameDay
Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
http://www.flickr.com/photos/dnorman/2678090600

20. define:
GameDay
An exercise designed to increase
Resilience through large-scale fault
injection across critical systems.
Part of a larger discipline called
Resilience Engineering.
Not new, just new to us ;-)

21. Resilience is a product of
People & Technology

22. GameDay increases Resilience in 3 ways
Preparation
‣ Identification and mitigation of risks and impact from
failure
‣ Reduces frequency of failure (MTBF)
‣ Reduces duration of recovery (MTTR)
Participation
‣ Builds confidence & competence responding to failure
and under stress.
‣ Strengthens individual and cultural ability to anticipate,
mitigate, respond to, and recover from failures of all
types.
Exercises
‣ Trigger and expose “latent defects”
‣ Choose when discover them, instead of letting that be
determined by the next real disaster.

23. start small...
http://www.flickr.com/photos/oakleyoriginals/5674150237

24. increase awareness
http://www.flickr.com/photos/maunzy/5099921731

25. build confidence
http://www.flickr.com/photos/skevbo/4864249944

26. full scale, live fire exercises
http://tacomafiredepartment.blogspot.com/2010/05/west-slope-training-burn.html

27. safety standards &
“building codes”
http://www.flickr.com/photos/peregrinari/3801964067

28. no substitutes for experience...
Failure free operations require
experience with failure.
Ana Grillo © Ana Grillo Photography

29. Lessons Learned:
Every Post-mortem, Ever...

30. Root Cause:
“a perfect storm of
impossible events”

31. Lesson #1
“we have a bunch of
manual processes
which we need to
automate”

32. (thanks to Hyperbole & A Half))

33. Infrastructure as Code:
Enable the reconstruction of the
business from nothing but a
source code repository, an
application data backup, and
bare resources.

34. Lesson #1.5
“Golden Images don’t
work in dynamic (ie:
Cloud) Environments”

35. Golden Images are not the answer
• Gold is heavy & expensive
• Hard to transport
• Hard to mold
• Easy to lose configuration detail
http://www.flickr.com/photos/garysoup/2977173063/

36. When this
Varnish
Jboss App
Memcache
Postgres Slaves
Postgres Master

37. Becomes this...
Varnish
Jboss App
Memcache
Postgres Slaves
Postgres Master

38. • Load balancer
Change is only Constant config
• Nagios host ping
Varnish
• Nagios host ssh
• Nagios host HTTP
• Nagios host app
Jboss App
health
• Graphite CPU
• Graphite
Memcache
Memory
• Graphite Disk
Postgres Slaves
• Graphite SNMP
• Memcache
• Postgres Master
12+ resource changes for 1 node addition
firewall
• Postgres firewall

39. Scale & Complexity increase!
AZ2
DC1
AZ3

40. CLONING CANNOT COPE
WITH THIS
http://www.flickr.com/photos/evelynishere/2798236471/

41. (thanks to Hyperbole & A Half )

42. Lesson #2
“We need better
Incident Management”

44. http://www.flickr.com/photos/mrwilleeumm/1102333740/sizes/o/in/photostream/

45. ♥ Dev+Ops
Culture
Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

46. make it fun!

47. Lesson #3
“(load balancing|website|DNS|
database|deployment|storage|
provisioning|cloud|etc)
failover didn’t work.
We need to test & maintain our
emergency tools & processes”

48. use it or lose it.

49. Give yourself lots of knobs
& levers & use them as part
of regular process.

50. MTTR > MTBF

51. I ❤ MTTR

52. (thanks to Hyperbole & A Half )

53. GameDay
Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
http://www.flickr.com/photos/dnorman/2678090600

55. Thank You!!!
Jesse Robbins
Cofounder, Opscode
@jesserobbins
jesse@opscode.com