SlideShare ist ein Scribd-Unternehmen logo
1 von 50
Downloaden Sie, um offline zu lesen
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Image Service Outage
postrotate
  /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null`
  2> /dev/null || true
NEVER TEST IN
PRODUCTION!
It only takes
one tiny
mistake
How Do You Enforce This?
•   Documented standards and communicated best practices

•   Robust testing workflow

    •   Environments

    •   Knife Plugins

•   Linting with rules derived from standards

    •   Foodcritic
Testing Workflow
How We Use Environments

•   Three environments: production, development, testing

    •   Testing is unconstrained

    •   Test nodes are depooled and “flipped” to the testing environment,
        then repooled and analyzed

    •   Test nodes are then flipped back to production
Working with Environments
•    knife-flip by Etsy engineer Jon Cowie
     (https://github.com/jonlives/knife-flip)

    % knife node flip somenode.etsy.com testing

    % knife role flip SomeRole testing

•    knife-bulkchangeenvironment (https://github.com/jonlives/knife-
     bulkchangeenvironment)

    % knife node bulk_change_environment testing production
Keeping Environments in Sync
•   knife-env-diff by Etsy engineer John Goulah

    •   Get it at https://github.com/jgoulah/knife-env-diff

% knife environment diff development production

diffing environment development against production

cookbook: hadoop
 development version: = 0.1.0
 production version: = 0.1.8

cookbook: mysql
 development version: = 0.2.4
 production version: = 0.2.5
Introducing Knife Spork


•   Knife plugin providing a testing/versioning workflow

•   Authored by Jon Cowie

•   Get it at https://github.com/jonlives/knife-spork
Spork Features

•   Four stage process

    •   Check: Look at versioning info for a cookbook

    •   Bump: Automatically increment the cookbook’s version number

    •   Upload: Knife upload and freeze

    •   Promote: Set environment constraints equal to specified version
git:
                              enabled: true
                            irccat:
                              enabled: true
                              server: irccat.mycompany.com
                              port: 12345

Spork Config                   channel: "#chef"
                            graphite:
                              enabled: true
                              server: graphite.mycompany.com
• /path/to/chef-repo          port: 2003
  /config/spork-config.yml    gist:
                              enabled: true
• /etc/spork-config.yml        in_chef: true
                              chef_path: cookbooks/gist/files/default/gist
• ~/.chef/spork-config.yml     path: /usr/bin/gist
                            foodcritic:
                              enabled: true
                              fail_tags: [any]
                              tags: [foo]
                              include_rules: [/home/me/myrules]
                            default_environments: [ production, development ]
% knife spork check foodcritic

Checking versions for cookbook foodcritic...

Current local version: 0.0.4

Remote versions (Max. 5 most recent only):
*0.0.4, frozen
0.0.3, frozen
0.0.2, unfrozen
0.0.1, frozen

DANGER: Your local cookbook has same version number as the
starred version above!

Please bump your local version or you won't be able to
upload.
% knife spork bump foodcritic

Loaded config file /home/pmcdonnell/git/chef-repo/config/
spork-config.yml...

Loaded config file /etc/spork-config.yml...

Pulling latest changes from git

Pulling latest changes from git submodules (if any)

Bumping patch level of the foodcritic cookbook from 0.0.4 to
0.0.5

Git add'ing /home/pmcdonnell/git/chef-repo/cookbooks/
foodcritic/metadata.rb
% knife spork upload foodcritic

Loaded config file /home/pmcdonnell/git/chef-repo/config/
spork-config.yml...

Loaded config file /etc/spork-config.yml...

Uploading and freezing foodcritic             [0.0.5]

upload complete
% knife spork promote foodcritic --remote

Pulling latest changes from git

Checking that foodcritic version 0.0.5 exists on the server
before promoting (any error means it hasn't been uploaded
yet)...

foodcritic version 0.0.5 found on server!

Environment: production
Adding version constraint foodcritic = 0.0.5
Saving changes into production.json

Git add'ing /home/pmcdonnell/git/chef-repo/environments/
production.json

Uploading production to server
WARNING: You're about to promote changes to several
cookbooks:
logrotate: = 0.1.24 changed to = 0.1.23
foodcritic: = 0.0.4 changed to = 0.0.5

Are you sure you want to continue? (Y/N) n

You said no, so I'm done here.

Would you like to reset your local production.json to match
the server?? (Y/N) y

Git add'ing /home/pmcdonnell/git/chef-repo/environments/
production.json

production.json reset.
Spork’s Logging Mechanisms
 •   Irccat: Logs to IRC channel (https://github.com/RJ/irccat)
[11:35:33] <irccat> CHEF: pmcdonnell uploaded and froze cookbook ldap version 0.1.27
[11:35:43] <irccat> CHEF: pmcdonnell uploaded environment production
https://github.etsycorp.com/gist/376967
[11:35:43] <irccat> CHEF: pmcdonnell uploaded environment development
https://github.etsycorp.com/gist/376968



 •   Graphite: promote --remote sends to deploys.chef metric

 •   Gist: Added to irccat notifications on promote --remote
Environment production uploaded at 2012-05-15 18:35:42 UTC by pmcdonnell

Constraints updated on server in this version:
ldap: = 0.1.26 changed to = 0.1.27
Linting
Foodcritic
•   A lint tool for Chef cookbooks written by Andrew Crump
    (http://acrmp.github.com/foodcritic/)

•   Comes with a good set of default rules and is very easily extensible

•   To enable in spork config:

               foodcritic:
                 enabled: true
                 fail_tags: [any]
                 tags: [foo]
                 include_rules: [/home/me/myrules]
Etsy’s Rules
•       A work in progress, but newly open-sourced at
        https://github.com/etsy/foodcritic-rules

•       Our rules are “style”-tagged rules that serve to enforce what we
        consider to be best practices in our environment

    •    ETSY001 - Package or yum_package resource used with :upgrade action

    •    ETSY002 - Execute resource used to run git commands
    •    ETSY003 - Execute resource used to run curl or wget commands

    •    ETSY004 - Execute resource defined without conditional or action :nothing

    •    ETSY005 - Action :restart sent to a core service
Rule Resulting from Image Outage
•   ETSY005 - Action :restart sent to a core service

    •   Trippable services include httpd, mysql, memcached, postgresql-server


% foodcritic -t etsy -I ~/git/chef-repo/config/rules.rb ~/
git/chef-repo/cookbooks/apache

ETSY005: Action :restart sent to a core service:
/home/pmcdonnell/git/chef-repo/cookbooks/apache/recipes/
default.rb:39
Rule Resulting from Image Outage
30 template "/etc/httpd/conf/httpd.conf" do
31   source "httpd-conf.erb"
32   owner "root"
33   group "root"
34   mode 00644
35   variables(
36     :fqdn => node[:fqdn],
37     :port => "80"
38   )
39   notifies :restart, resources(:service => "httpd")
40 end
Memcache Outage
02:27 < jallspaw> [Sat, 10 Jul 2010 01:45:01 +0000]
INFO: Upgrading package[memcached] version from
               1.4.2-1.fc10 to 1.4.5-1.el5
Don’t leave
“known unknowns”
lying in wait
Resulting Foodcritic Rule
•   ETSY001 - Package or yum_package resource used with :upgrade action

    •   Enforces always using :install


% foodcritic -t etsy -I ~/git/chef-repo/config/rules.rb ~/
git/chef-repo/cookbooks/memcache

ETSY001: Package or yum_package resource used with :upgrade
action: /home/pmcdonnell/git/chef-repo/cookbooks/memcache/
recipes/default.rb:20
Resulting Foodcritic Rule
20 package "memcached" do
21   action :upgrade
22 end


                      Changed to:
20 package "memcached" do
21   version "1.4.2-1.fc10"
22   action :install
23 end
Reporting and Monitoring
Using Handlers
 •   Etsy’s handlers (https://github.com/etsy/chef-handlers)

     •   Log failures to IRC
[10:52:03] <irccat> Chef run failed on dev-dbtasks01.ny4dev.etsy.com
[10:52:03] <irccat> https://github.etsycorp.com/gist/371229



     •   Graph aggregated metrics with Graphite

     •   Graph chef “deploys”
Graph with Graphite
•   Metrics reporting made possible by knife-lastrun, authored by
    John Goulah (https://github.com/jgoulah/knife-lastrun)

    •   Provides a handler and knife plugin for reporting on the most recent
        chef run, storing data as node attributes

        •   Elapsed, starting, and ending time

        •   Exit code status

        •   Backtrace/exception information
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
% dsh -g all -c -M 'grep "Chef Run complete in" /var/log/
chef/client.log | head -n 3' 2>&1 | tee /tmp/tee && grep
'Chef Run complete' /tmp/tee | sort -n -k +13 | tail -5

dn0035.doop:   [Mon, 14 May 2012 03:21:07   +0000] INFO: Chef
Run complete   in 512.936813012 seconds
dn0004.doop:   [Mon, 14 May 2012 04:28:03   +0000] INFO: Chef
Run complete   in 677.423964906 seconds
dn0006.doop:   [Mon, 14 May 2012 04:29:51   +0000] INFO: Chef
Run complete   in 770.231469266 seconds
dn0025.doop:   [Mon, 14 May 2012 04:26:13   +0000] INFO: Chef
Run complete   in 787.183615612 seconds
dn0030.doop:   [Mon, 14 May 2012 04:30:42   +0000] INFO: Chef
Run complete   in 848.586507872 seconds
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012
Finding Run Time Outliers

•   Knife doesn’t currently support Lucene’s NumericRangeQuery

    •   Elapsed time is a floating point number, but we can only match it as a
        string due to query limitations in knife

    •   Work around it with knife search -a
% knife search node 'elapsed:[200 TO 225]' -a
lastrun.runtimes.elapsed

4 items found

id:                         cent6-vmtemplate.ny4dev.etsy.com
lastrun.runtimes.elapsed:   21.642378406

id:                         sandboxmisc01.ny4.etsy.com
lastrun.runtimes.elapsed:   211.749555

id:                         smardenfeld.vm.ny4dev.etsy.com
lastrun.runtimes.elapsed:   22.184596

id:                         bob0120.vm.ny4dev.etsy.com
lastrun.runtimes.elapsed:   21.348335354
% knife node lastrun sandboxmisc01.ny4.etsy.com

Status                    failed
Elapsed Time              211.78604
Start Time                2012-05-15 07:43:18 +0000
End Time                  2012-05-15 07:46:50 +0000

Backtrace
Omitted for brevity

Exception
Chef::Exceptions::Package: package[diffutils]
(installerz::diffutils line 1) had an error: Yum failed -
#<Process::Status: pid 21293 exit 1> - returns: ["yum-dump
Repository Error: Cannot retrieve repository metadata
(repomd.xml) for repository: PostgreSQL-8.3-x86_64. Please
verify its path and try againn"]
What Did Chef Just Do?
•   chefrecentupdates by Etsy engineer Laurie Denness
    (https://github.com/lozzd/ChefScripts)


% chefrecentupdates
...
1 resources updated in /var/log/chef/client.log-20120505.gz:
[Fri, 04 May 2012 17:49:42 +0000]
INFO: cookbook_file[/usr/bin/gist]
...
Preventative Measures
Knife Preflight
•   By Jon Cowie (https://github.com/jonlives/knife-preflight)
% knife preflight memcache::datacache
Searching for nodes containing memcache::datacache in their expanded
run_list...
4 Nodes found

datacache03.ny4.etsy.com
datacache04.ny4.etsy.com
datacache01.ny4.etsy.com
datacache02.ny4.etsy.com

Searching for roles containing memcache::datacache in their run_list...
1 Roles found

Datacache

Found 4 nodes and 1 roles using the specified search criteria
Continuous Chef
•   Using Jenkins and base virtual machine images
“Out-of-Band” Management


•   dsh (distributed shell) works even if Chef server is down

    •   Etsy’s dsh groups are managed by Chef and generated from the list of
        nodes corresponding to each role
Configs Bundled with Packages


•   Be careful with configs distributed with packages overwriting Chef
    configs

    •   They must be replaced by Chef before restarting services, so watch
        out for resource order
Jon will be at Velocity!
•   Workshop: Michelin Starred Cooking with Chef

    •   11:00am Monday, 06/25/2012

    •   Topics

        •   Team-wide familiarity and understanding

        •   Critical approach and experimentation with workflows

        •   Plugin writing 101
We’re Hiring!

        http://www.etsy.com/careers

•   TONS of engineering positions open!

•   Especially looking for a talented network engineer; referrals welcome!

Weitere ähnliche Inhalte

Was ist angesagt?

Chef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation SetupChef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation SetupChef Software, Inc.
 
Opscode Webinar: Managing Your VMware Infrastructure with Chef
Opscode Webinar: Managing Your VMware Infrastructure with ChefOpscode Webinar: Managing Your VMware Infrastructure with Chef
Opscode Webinar: Managing Your VMware Infrastructure with ChefChef Software, Inc.
 
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...Chef Software, Inc.
 
Common configuration with Data Bags - Fundamentals Webinar Series Part 4
Common configuration with Data Bags - Fundamentals Webinar Series Part 4Common configuration with Data Bags - Fundamentals Webinar Series Part 4
Common configuration with Data Bags - Fundamentals Webinar Series Part 4Chef
 
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2Chef
 
Introduction to Chef
Introduction to ChefIntroduction to Chef
Introduction to ChefKnoldus Inc.
 
Chef, Devops, and You
Chef, Devops, and YouChef, Devops, and You
Chef, Devops, and YouBryan Berry
 
Test Driven Development with Chef
Test Driven Development with ChefTest Driven Development with Chef
Test Driven Development with ChefSimone Soldateschi
 
CLUG 2014-10 - Cookbook CI with Jenkins
CLUG 2014-10 - Cookbook CI with JenkinsCLUG 2014-10 - Cookbook CI with Jenkins
CLUG 2014-10 - Cookbook CI with JenkinsZachary Stevens
 
Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3Chef
 
Test-Driven Infrastructure with Chef
Test-Driven Infrastructure with ChefTest-Driven Infrastructure with Chef
Test-Driven Infrastructure with ChefMichael Lihs
 
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...Chef Software, Inc.
 
Testing for infra code using test-kitchen,docker,chef
Testing for infra code using  test-kitchen,docker,chefTesting for infra code using  test-kitchen,docker,chef
Testing for infra code using test-kitchen,docker,chefkamalikamj
 
Automating your infrastructure with Chef
Automating your infrastructure with ChefAutomating your infrastructure with Chef
Automating your infrastructure with ChefJohn Ewart
 
Infrastructure Automation with Chef
Infrastructure Automation with ChefInfrastructure Automation with Chef
Infrastructure Automation with ChefJonathan Weiss
 
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Software, Inc.
 
Automating Large Applications on Modular and Structured Form with Gulp
Automating Large Applications on Modular and Structured Form with GulpAutomating Large Applications on Modular and Structured Form with Gulp
Automating Large Applications on Modular and Structured Form with GulpAnderson Aguiar
 
Cloud Automation with Opscode Chef
Cloud Automation with Opscode ChefCloud Automation with Opscode Chef
Cloud Automation with Opscode ChefSri Ram
 
Infrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & AnsibleInfrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & Ansiblewajrcs
 

Was ist angesagt? (20)

Chef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation SetupChef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation Setup
 
Opscode Webinar: Managing Your VMware Infrastructure with Chef
Opscode Webinar: Managing Your VMware Infrastructure with ChefOpscode Webinar: Managing Your VMware Infrastructure with Chef
Opscode Webinar: Managing Your VMware Infrastructure with Chef
 
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
 
Common configuration with Data Bags - Fundamentals Webinar Series Part 4
Common configuration with Data Bags - Fundamentals Webinar Series Part 4Common configuration with Data Bags - Fundamentals Webinar Series Part 4
Common configuration with Data Bags - Fundamentals Webinar Series Part 4
 
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
 
Introduction to Chef
Introduction to ChefIntroduction to Chef
Introduction to Chef
 
Chef, Devops, and You
Chef, Devops, and YouChef, Devops, and You
Chef, Devops, and You
 
Test Driven Development with Chef
Test Driven Development with ChefTest Driven Development with Chef
Test Driven Development with Chef
 
The unintended benefits of Chef
The unintended benefits of ChefThe unintended benefits of Chef
The unintended benefits of Chef
 
CLUG 2014-10 - Cookbook CI with Jenkins
CLUG 2014-10 - Cookbook CI with JenkinsCLUG 2014-10 - Cookbook CI with Jenkins
CLUG 2014-10 - Cookbook CI with Jenkins
 
Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3
 
Test-Driven Infrastructure with Chef
Test-Driven Infrastructure with ChefTest-Driven Infrastructure with Chef
Test-Driven Infrastructure with Chef
 
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
 
Testing for infra code using test-kitchen,docker,chef
Testing for infra code using  test-kitchen,docker,chefTesting for infra code using  test-kitchen,docker,chef
Testing for infra code using test-kitchen,docker,chef
 
Automating your infrastructure with Chef
Automating your infrastructure with ChefAutomating your infrastructure with Chef
Automating your infrastructure with Chef
 
Infrastructure Automation with Chef
Infrastructure Automation with ChefInfrastructure Automation with Chef
Infrastructure Automation with Chef
 
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
 
Automating Large Applications on Modular and Structured Form with Gulp
Automating Large Applications on Modular and Structured Form with GulpAutomating Large Applications on Modular and Structured Form with Gulp
Automating Large Applications on Modular and Structured Form with Gulp
 
Cloud Automation with Opscode Chef
Cloud Automation with Opscode ChefCloud Automation with Opscode Chef
Cloud Automation with Opscode Chef
 
Infrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & AnsibleInfrastructure Automation with Chef & Ansible
Infrastructure Automation with Chef & Ansible
 

Ähnlich wie Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012

Testing Your Automation Code (Vagrant Version)
Testing Your Automation Code (Vagrant Version)Testing Your Automation Code (Vagrant Version)
Testing Your Automation Code (Vagrant Version)Mischa Taylor
 
Testing your-automation-code (vagrant version) v0.2
Testing your-automation-code (vagrant version) v0.2Testing your-automation-code (vagrant version) v0.2
Testing your-automation-code (vagrant version) v0.2Sylvain Tissot
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflowTomas Doran
 
Open Source Recipes for Chef Deployments of Hadoop
Open Source Recipes for Chef Deployments of HadoopOpen Source Recipes for Chef Deployments of Hadoop
Open Source Recipes for Chef Deployments of HadoopDataWorks Summit
 
Cooking Perl with Chef: Real World Tutorial with Jitterbug
Cooking Perl with Chef: Real World Tutorial with JitterbugCooking Perl with Chef: Real World Tutorial with Jitterbug
Cooking Perl with Chef: Real World Tutorial with JitterbugDavid Golden
 
Using Test Kitchen for testing Chef cookbooks
Using Test Kitchen for testing Chef cookbooksUsing Test Kitchen for testing Chef cookbooks
Using Test Kitchen for testing Chef cookbooksTimur Batyrshin
 
Chef introduction
Chef introductionChef introduction
Chef introductioncawamata
 
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.Mandi Walls
 
Introduction to Chef - April 22 2015
Introduction to Chef - April 22 2015Introduction to Chef - April 22 2015
Introduction to Chef - April 22 2015Jennifer Davis
 
Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Mandi Walls
 
Chef infrastructure as code - paris.rb
Chef infrastructure as code - paris.rbChef infrastructure as code - paris.rb
Chef infrastructure as code - paris.rbNicolas Ledez
 
Emerging chef patterns and practices
Emerging chef patterns and practicesEmerging chef patterns and practices
Emerging chef patterns and practicesOwain Perry
 
Atmosphere 2014: Really large scale systems configuration - Phil Dibowitz
Atmosphere 2014: Really large scale systems configuration - Phil DibowitzAtmosphere 2014: Really large scale systems configuration - Phil Dibowitz
Atmosphere 2014: Really large scale systems configuration - Phil DibowitzPROIDEA
 
Introduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to ChefIntroduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to ChefAll Things Open
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmersmrsabo
 
Cfg mgmtcamp c-dwithchef
Cfg mgmtcamp c-dwithchefCfg mgmtcamp c-dwithchef
Cfg mgmtcamp c-dwithchefGeorge Miranda
 

Ähnlich wie Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012 (20)

Testing Your Automation Code (Vagrant Version)
Testing Your Automation Code (Vagrant Version)Testing Your Automation Code (Vagrant Version)
Testing Your Automation Code (Vagrant Version)
 
Testing your-automation-code (vagrant version) v0.2
Testing your-automation-code (vagrant version) v0.2Testing your-automation-code (vagrant version) v0.2
Testing your-automation-code (vagrant version) v0.2
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
 
Chef training Day4
Chef training Day4Chef training Day4
Chef training Day4
 
Open Source Recipes for Chef Deployments of Hadoop
Open Source Recipes for Chef Deployments of HadoopOpen Source Recipes for Chef Deployments of Hadoop
Open Source Recipes for Chef Deployments of Hadoop
 
Cooking Perl with Chef: Real World Tutorial with Jitterbug
Cooking Perl with Chef: Real World Tutorial with JitterbugCooking Perl with Chef: Real World Tutorial with Jitterbug
Cooking Perl with Chef: Real World Tutorial with Jitterbug
 
Using Test Kitchen for testing Chef cookbooks
Using Test Kitchen for testing Chef cookbooksUsing Test Kitchen for testing Chef cookbooks
Using Test Kitchen for testing Chef cookbooks
 
Chef introduction
Chef introductionChef introduction
Chef introduction
 
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.
Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.
 
Introduction to Chef - April 22 2015
Introduction to Chef - April 22 2015Introduction to Chef - April 22 2015
Introduction to Chef - April 22 2015
 
The Environment Restaurant
The Environment RestaurantThe Environment Restaurant
The Environment Restaurant
 
Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014
 
Chef infrastructure as code - paris.rb
Chef infrastructure as code - paris.rbChef infrastructure as code - paris.rb
Chef infrastructure as code - paris.rb
 
Emerging chef patterns and practices
Emerging chef patterns and practicesEmerging chef patterns and practices
Emerging chef patterns and practices
 
Atmosphere 2014: Really large scale systems configuration - Phil Dibowitz
Atmosphere 2014: Really large scale systems configuration - Phil DibowitzAtmosphere 2014: Really large scale systems configuration - Phil Dibowitz
Atmosphere 2014: Really large scale systems configuration - Phil Dibowitz
 
IT Automation with Chef
IT Automation with ChefIT Automation with Chef
IT Automation with Chef
 
Introduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to ChefIntroduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to Chef
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmers
 
Cfg mgmtcamp c-dwithchef
Cfg mgmtcamp c-dwithchefCfg mgmtcamp c-dwithchef
Cfg mgmtcamp c-dwithchef
 

Kürzlich hochgeladen

Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 

Kürzlich hochgeladen (20)

Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 

Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012

  • 3. postrotate /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
  • 5. It only takes one tiny mistake
  • 6. How Do You Enforce This? • Documented standards and communicated best practices • Robust testing workflow • Environments • Knife Plugins • Linting with rules derived from standards • Foodcritic
  • 8. How We Use Environments • Three environments: production, development, testing • Testing is unconstrained • Test nodes are depooled and “flipped” to the testing environment, then repooled and analyzed • Test nodes are then flipped back to production
  • 9. Working with Environments • knife-flip by Etsy engineer Jon Cowie (https://github.com/jonlives/knife-flip) % knife node flip somenode.etsy.com testing % knife role flip SomeRole testing • knife-bulkchangeenvironment (https://github.com/jonlives/knife- bulkchangeenvironment) % knife node bulk_change_environment testing production
  • 10. Keeping Environments in Sync • knife-env-diff by Etsy engineer John Goulah • Get it at https://github.com/jgoulah/knife-env-diff % knife environment diff development production diffing environment development against production cookbook: hadoop development version: = 0.1.0 production version: = 0.1.8 cookbook: mysql development version: = 0.2.4 production version: = 0.2.5
  • 11. Introducing Knife Spork • Knife plugin providing a testing/versioning workflow • Authored by Jon Cowie • Get it at https://github.com/jonlives/knife-spork
  • 12. Spork Features • Four stage process • Check: Look at versioning info for a cookbook • Bump: Automatically increment the cookbook’s version number • Upload: Knife upload and freeze • Promote: Set environment constraints equal to specified version
  • 13. git: enabled: true irccat: enabled: true server: irccat.mycompany.com port: 12345 Spork Config channel: "#chef" graphite: enabled: true server: graphite.mycompany.com • /path/to/chef-repo port: 2003 /config/spork-config.yml gist: enabled: true • /etc/spork-config.yml in_chef: true chef_path: cookbooks/gist/files/default/gist • ~/.chef/spork-config.yml path: /usr/bin/gist foodcritic: enabled: true fail_tags: [any] tags: [foo] include_rules: [/home/me/myrules] default_environments: [ production, development ]
  • 14. % knife spork check foodcritic Checking versions for cookbook foodcritic... Current local version: 0.0.4 Remote versions (Max. 5 most recent only): *0.0.4, frozen 0.0.3, frozen 0.0.2, unfrozen 0.0.1, frozen DANGER: Your local cookbook has same version number as the starred version above! Please bump your local version or you won't be able to upload.
  • 15. % knife spork bump foodcritic Loaded config file /home/pmcdonnell/git/chef-repo/config/ spork-config.yml... Loaded config file /etc/spork-config.yml... Pulling latest changes from git Pulling latest changes from git submodules (if any) Bumping patch level of the foodcritic cookbook from 0.0.4 to 0.0.5 Git add'ing /home/pmcdonnell/git/chef-repo/cookbooks/ foodcritic/metadata.rb
  • 16. % knife spork upload foodcritic Loaded config file /home/pmcdonnell/git/chef-repo/config/ spork-config.yml... Loaded config file /etc/spork-config.yml... Uploading and freezing foodcritic [0.0.5] upload complete
  • 17. % knife spork promote foodcritic --remote Pulling latest changes from git Checking that foodcritic version 0.0.5 exists on the server before promoting (any error means it hasn't been uploaded yet)... foodcritic version 0.0.5 found on server! Environment: production Adding version constraint foodcritic = 0.0.5 Saving changes into production.json Git add'ing /home/pmcdonnell/git/chef-repo/environments/ production.json Uploading production to server
  • 18. WARNING: You're about to promote changes to several cookbooks: logrotate: = 0.1.24 changed to = 0.1.23 foodcritic: = 0.0.4 changed to = 0.0.5 Are you sure you want to continue? (Y/N) n You said no, so I'm done here. Would you like to reset your local production.json to match the server?? (Y/N) y Git add'ing /home/pmcdonnell/git/chef-repo/environments/ production.json production.json reset.
  • 19. Spork’s Logging Mechanisms • Irccat: Logs to IRC channel (https://github.com/RJ/irccat) [11:35:33] <irccat> CHEF: pmcdonnell uploaded and froze cookbook ldap version 0.1.27 [11:35:43] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/376967 [11:35:43] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/376968 • Graphite: promote --remote sends to deploys.chef metric • Gist: Added to irccat notifications on promote --remote Environment production uploaded at 2012-05-15 18:35:42 UTC by pmcdonnell Constraints updated on server in this version: ldap: = 0.1.26 changed to = 0.1.27
  • 21. Foodcritic • A lint tool for Chef cookbooks written by Andrew Crump (http://acrmp.github.com/foodcritic/) • Comes with a good set of default rules and is very easily extensible • To enable in spork config: foodcritic: enabled: true fail_tags: [any] tags: [foo] include_rules: [/home/me/myrules]
  • 22. Etsy’s Rules • A work in progress, but newly open-sourced at https://github.com/etsy/foodcritic-rules • Our rules are “style”-tagged rules that serve to enforce what we consider to be best practices in our environment • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing • ETSY005 - Action :restart sent to a core service
  • 23. Rule Resulting from Image Outage • ETSY005 - Action :restart sent to a core service • Trippable services include httpd, mysql, memcached, postgresql-server % foodcritic -t etsy -I ~/git/chef-repo/config/rules.rb ~/ git/chef-repo/cookbooks/apache ETSY005: Action :restart sent to a core service: /home/pmcdonnell/git/chef-repo/cookbooks/apache/recipes/ default.rb:39
  • 24. Rule Resulting from Image Outage 30 template "/etc/httpd/conf/httpd.conf" do 31 source "httpd-conf.erb" 32 owner "root" 33 group "root" 34 mode 00644 35 variables( 36 :fqdn => node[:fqdn], 37 :port => "80" 38 ) 39 notifies :restart, resources(:service => "httpd") 40 end
  • 26. 02:27 < jallspaw> [Sat, 10 Jul 2010 01:45:01 +0000] INFO: Upgrading package[memcached] version from 1.4.2-1.fc10 to 1.4.5-1.el5
  • 28. Resulting Foodcritic Rule • ETSY001 - Package or yum_package resource used with :upgrade action • Enforces always using :install % foodcritic -t etsy -I ~/git/chef-repo/config/rules.rb ~/ git/chef-repo/cookbooks/memcache ETSY001: Package or yum_package resource used with :upgrade action: /home/pmcdonnell/git/chef-repo/cookbooks/memcache/ recipes/default.rb:20
  • 29. Resulting Foodcritic Rule 20 package "memcached" do 21 action :upgrade 22 end Changed to: 20 package "memcached" do 21 version "1.4.2-1.fc10" 22 action :install 23 end
  • 31. Using Handlers • Etsy’s handlers (https://github.com/etsy/chef-handlers) • Log failures to IRC [10:52:03] <irccat> Chef run failed on dev-dbtasks01.ny4dev.etsy.com [10:52:03] <irccat> https://github.etsycorp.com/gist/371229 • Graph aggregated metrics with Graphite • Graph chef “deploys”
  • 32. Graph with Graphite • Metrics reporting made possible by knife-lastrun, authored by John Goulah (https://github.com/jgoulah/knife-lastrun) • Provides a handler and knife plugin for reporting on the most recent chef run, storing data as node attributes • Elapsed, starting, and ending time • Exit code status • Backtrace/exception information
  • 35. % dsh -g all -c -M 'grep "Chef Run complete in" /var/log/ chef/client.log | head -n 3' 2>&1 | tee /tmp/tee && grep 'Chef Run complete' /tmp/tee | sort -n -k +13 | tail -5 dn0035.doop: [Mon, 14 May 2012 03:21:07 +0000] INFO: Chef Run complete in 512.936813012 seconds dn0004.doop: [Mon, 14 May 2012 04:28:03 +0000] INFO: Chef Run complete in 677.423964906 seconds dn0006.doop: [Mon, 14 May 2012 04:29:51 +0000] INFO: Chef Run complete in 770.231469266 seconds dn0025.doop: [Mon, 14 May 2012 04:26:13 +0000] INFO: Chef Run complete in 787.183615612 seconds dn0030.doop: [Mon, 14 May 2012 04:30:42 +0000] INFO: Chef Run complete in 848.586507872 seconds
  • 40. Finding Run Time Outliers • Knife doesn’t currently support Lucene’s NumericRangeQuery • Elapsed time is a floating point number, but we can only match it as a string due to query limitations in knife • Work around it with knife search -a
  • 41. % knife search node 'elapsed:[200 TO 225]' -a lastrun.runtimes.elapsed 4 items found id: cent6-vmtemplate.ny4dev.etsy.com lastrun.runtimes.elapsed: 21.642378406 id: sandboxmisc01.ny4.etsy.com lastrun.runtimes.elapsed: 211.749555 id: smardenfeld.vm.ny4dev.etsy.com lastrun.runtimes.elapsed: 22.184596 id: bob0120.vm.ny4dev.etsy.com lastrun.runtimes.elapsed: 21.348335354
  • 42. % knife node lastrun sandboxmisc01.ny4.etsy.com Status failed Elapsed Time 211.78604 Start Time 2012-05-15 07:43:18 +0000 End Time 2012-05-15 07:46:50 +0000 Backtrace Omitted for brevity Exception Chef::Exceptions::Package: package[diffutils] (installerz::diffutils line 1) had an error: Yum failed - #<Process::Status: pid 21293 exit 1> - returns: ["yum-dump Repository Error: Cannot retrieve repository metadata (repomd.xml) for repository: PostgreSQL-8.3-x86_64. Please verify its path and try againn"]
  • 43. What Did Chef Just Do? • chefrecentupdates by Etsy engineer Laurie Denness (https://github.com/lozzd/ChefScripts) % chefrecentupdates ... 1 resources updated in /var/log/chef/client.log-20120505.gz: [Fri, 04 May 2012 17:49:42 +0000] INFO: cookbook_file[/usr/bin/gist] ...
  • 45. Knife Preflight • By Jon Cowie (https://github.com/jonlives/knife-preflight) % knife preflight memcache::datacache Searching for nodes containing memcache::datacache in their expanded run_list... 4 Nodes found datacache03.ny4.etsy.com datacache04.ny4.etsy.com datacache01.ny4.etsy.com datacache02.ny4.etsy.com Searching for roles containing memcache::datacache in their run_list... 1 Roles found Datacache Found 4 nodes and 1 roles using the specified search criteria
  • 46. Continuous Chef • Using Jenkins and base virtual machine images
  • 47. “Out-of-Band” Management • dsh (distributed shell) works even if Chef server is down • Etsy’s dsh groups are managed by Chef and generated from the list of nodes corresponding to each role
  • 48. Configs Bundled with Packages • Be careful with configs distributed with packages overwriting Chef configs • They must be replaced by Chef before restarting services, so watch out for resource order
  • 49. Jon will be at Velocity! • Workshop: Michelin Starred Cooking with Chef • 11:00am Monday, 06/25/2012 • Topics • Team-wide familiarity and understanding • Critical approach and experimentation with workflows • Plugin writing 101
  • 50. We’re Hiring! http://www.etsy.com/careers • TONS of engineering positions open! • Especially looking for a talented network engineer; referrals welcome!