Planning Application Resilience

1. Planning Application Resilience
Jennifer Davis
2/26/15

2. Goal: Communication
Jennifer Davis
Solutions Engineer
Twitter: @sigje
Hashtag: #getchef
Email: sigje@chef.io

3. What is Resilience?

4. Resilience
• Elasticity – Spring back into shape
• Recoverability – Quick to recover/rebuild

5. Enduring Resilience
• State of Elasticity
• State of Recoverability

6. Resilience Minimum Goal!

7. Fear
Depression
Fear
Event
Degradation
Failure
Event

8. Resilience
Recognition
Event Recovery

9. Post Traumatic Growth
Recognition
Event
Growth

10. Applications reflect Quality of Organizations
Conway’s law
Organizations which design systems … are constrained to
produce designs which are copies of the communication
structures of these organizations.

11. Applications reflect Quality of Organizations
The resilience of an organization will be reflected in the
resilience of the applications and services built by the
organization.

12. 5 Critical Metrics of Resilient Organizations
• Willingness to tackle challenges.
• Sense of Agency.
• Adaptability.
• Diversity.
• Rope Factor

13. Red Shirt Syndrome
• Tackling challenges.
• Learned Helplessness.

14. Stormtrooper Syndrome
• Agency.
• Adaptability: Role Adherence.

15. Borg Syndrome
• Embrace Diversity.
• Recognize differences in
perspectives.
• Eliminate system
blindness.

16. Rope Factor
Enough rope to get things done, not enough for cowboys.
Rope too short
Excessive time in meetings
Death march for each sprint
Rope too long
Cowboy behavior.

17. Resilience is ordinary.
• Intentional behaviors, thoughts, and actions.
• Reflection of the organization.

18. Don’t build organizational
systems that encourage the
wrong behaviors.

19. Resilience isn’t managed through limiting change.
• Security Patches?
• Over Engineering Delays in Schedule
• Under Engineering – Rewrite required to scale
Stability is a myth.

20. Qualities of Resilient Software
• Elasticity
• Recoverability

21. Qualities of Resilient Software
• Elasticity
• Recoverability
Automation Resilience

22. Resilient Automation Platform
• Complex dependency handling between nodes.
• Fault tolerance.
• Security.
• Multi-Platform.
• Flexibility.

23. Chef is a Resilient Automation Platform.

24. Chef is a language.
• Describe infrastructure as
code.
• Programmatically provision and
configure servers.
• Versioning, artifacts

25. Chef is a toolset
• Collection of tools that allow you to model, measure, and improve workflows.

26. chef is a command line utility
• Generate skeleton for application, cookbook, recipes, attributes, files, templates,
and custom resources.
• Prep environment with correct ruby gems.
• Verifies environment is configured and installed correctly.

27. Chef is a community.
• Mailing lists
• https://supermarket.chef.io/
• Chef Conf 3/31 – 4/2 Santa Clara
• Chef Summit
• IRC #chef
• Twitter @chef
CODE:
BUILDITBETTER

28. Configuration as Code
Elastic
Configurable
Responsive
Elasticity
Recoverability

29. Infrastructure Automation is creating
control systems that reduce the
burden on people to manage services
and increase the quality, accuracy and
precision of a service to the
consumers of the service.

30. Infrastructure Elements to Resources
file
package
cron
user
File
Package
Cron Job
User

31. Resources
• Fundamental building blocks
• Describes piece of system and it’s desired state
• Chef DSL is ruby.

32. Example of describing a resource
Recipe: (chef-apply cookbook)::(chef-apply
recipe)
* package[nano] action install
- install version 2.0.9-7.el6 of package
nano
sudo chef-apply -e "package 'nano'"

33. Test and Repair
Resources follow a test and
repair model
• package ”nano"
Is nano installed?
Done Install it
Yes No

34. Recipe
• A recipe is an ordered list of resources.

35. Recipe
package “httpd”
template “/var/www/html/index.html” do
source “index.html.erb”
end
service “httpd” do
action [:enable, :start]
end

36. Cookbook
• A collection of recipes (and other elements like files and templates).
• Map 1-1 to a piece of software or functionality.
• Distribution unit
• Versioned
• Modular and re-usable.

37. Chef Provisioning – Part of Chef DK
https://flic.kr/p/knDPjc
• Describe multiple tier applications.
• Deploy many copies of your
application cluster.
• Spread cluster across different clouds/
machines.
• Orchestrate deployment.
• Parallelize machine deployment.

38. Configuration as Code
Elastic
Configurable
Responsive
Elasticity
Recoverability

39. Chef Provisioning
machine ‘web1’ do
recipe ‘webserver’
end

40. Multi-platform
• AWS
• Azure
• Fog
• Vagrant
• Docker
• LXC
• .. more
.. We’ll use AWS in this example
https://github.com/chef/chef-provisioning-aws
http://aws.amazon.com/start-ups/loft/

41. AWS
• SQS Queues
• SNS Topics
• Elastic Load Balancers
• VPCs
• Security Groups
• Instances
• Images
• Autoscaling Groups
• SSH Key pairs
• Launch configs

42. AWS Config: ~/.aws/config
[default]
region=us-­‐west-­‐2
aws_access_key_id
=
aws_secret_access_key
=

43. Cookbook Setup
$ chef generate cookbook webserver

44. Provision Recipe
$ cd webserver
$ chef generate recipe provision

45. Edit Provision Recipe
$ vi recipes/provision.rb

46. Edit Provision Recipe
require “chef/provisioning/aws_driver”
with_driver “aws”
machine ‘web1’ do
recipe ‘webserver’
converge true
end

48. ..but I need multiple webservers
require “chef/provisioning/aws_driver”
with_driver “aws”
num_webservers = 3
(0… num_webservers).each do |i|
machine “web_0#{i}” do
recipe ‘apache’
end
end

49. …add security
aws_security_group "#{name}-http" do
inbound_rules [{:ports => 80, :protocol => :tcp, :sources =>
['0.0.0.0/0']}]
end

50. …add security
with_machine_options({
:bootstrap_options
=>
{
:security_groups
=>
[
"#{name}-­‐http”]
}
})

51. ..add load balancing
load_balancer
"#{name}-­‐webserver-­‐lb"
do
load_balancer_options({
:availability_zones
=>
["us-­‐west-­‐2a",
"us-­‐west-­‐2b",
“us-­‐
west-­‐2c"],
:listeners
=>
[{:port
=>
80,
:protocol
=>
:http,
:instance_port
=>
80,
:instance_protocol
=>
:http
}],
:security_group_name
=>
“#{name}-­‐http”
})
machines
elb_instances
end

52. Bulkhead Pattern
• Compartmentalization to limit failure.
• Repeatable Clusters
• … across platforms.

53. Configuration as Code
Elastic
Configurable
Responsive
Elasticity
Recoverability

54. Responsive
• Chef-Client
• Chef Handlers
• Jenkins with Test Kitchen
• Collaborate with Source Control
• Share your stories

55. Responsive – chef-client
• Agent that runs on node applies policy

56. Responsive - Jenkins with Test Kitchen
• Write tests to minimize risk
• Push change regularly

57. Responsive – Collaborate with Source Control
• Don’t let role adherence get in the way of collaboration.
• Pull requests

58. Responsive – Chef Handlers
• Start
• Exception
• Report

59. Share your stories
• Blameless Postmortems are really useful.
• Knowledge sharing across teams.
• Share across companies – DevOpsDays

60. Don’t build tools that create
systems that encourage the
wrong behaviors.

61. Jumpstart Learning
• The LearnChef Site
• Guided Tutorials
• Chef Fundamentals intro
http://learnchef.com
• How-To’s, Conference Talks, Webinars, more
http://youtube.com/user/getchef
• Attend a Chef Fundamentals Class (HELLO-CHEF code)

62. Further Resources
• http://chef.io
• http://docs.chef.io
• http://supermarket.chef.io
• http://lists.opscode.com
• irc.freenode.net #chef
• Twitter @chef #getchef, @learnchef #learnchef

63. Thank you!
Jennifer Davis
Twitter: @sigje
Hashtag: #getchef
Email: sigje@chef.io