SlideShare ist ein Scribd-Unternehmen logo
1 von 36
Downloaden Sie, um offline zu lesen
Learning Lessons Scaling to
                      5000 Agents
                                    Russ Johnson
                               rjohnson@stubhub.com
                                   @professoruss




Confidential                                   Slide 1
#whoami

    Started out in the mid 90’s
    Recovering Windows Admin
    Storage Guy
    Datacenter Monkey
    Once upon a time network guy
    At StubHub since December 2006
    Working on puppet adoption in a crazy infrastructure
    Puppet certified


April 9, 2013                                              2
4 Puppet Masters




                DEV/QA   PROD




                 DR      CORP
April 9, 2013                   3
The road to sanity




April 9, 2013            4
Set up your master properly

            Apache/Passenger
            Tune Passenger
                PassengerMaxPoolSize 32
                PassengerMinInstances 4
                PassengerMaxRequests 10000
                PassengerStatThrottleRate 30


            16 cores, 32GB
            load average: 4.03, 3.71, 3.45
            4000+ agents



April 9, 2013                                  5
Thundering herds




April 9, 2013          6
Build your hosts the same way!

            Old way:
                systemimager, vmware clones, manual installs
            Results:
                INCONSISTENCY!




April 9, 2013                                                  7
Build your hosts the same way!

            New way:
                Cobbler < 5m bare metal to on the network
            Results:
                Same results every time! No drift between base




April 9, 2013                                                    8
Set up your working environment properly

            Geppetto – eclipse based IDE
                  http://cloudsmith.github.com/geppetto/index.html
            VIM
            •    Pathogen – For autoloading vim plugins
            •    Snipmate – Snippets
            •    Tabular – Text filtering and alignment
            •    Syntastic – Syntax checking
            •    mv-vim-puppet – Make vim puppet friendly
            •    puppet-lint – Syntax checker (gem)

April 9, 2013                                                        9
Syntastic/puppet-lint




April 9, 2013               10
Set up your working environment properly



                            +




April 9, 2013                                  11
Version Control is not enough

            Ever do a 4 way diff across 60 modules to find most of
            them different?




April 9, 2013                                                        12
What to do?

 How do I stop manual edits?

                        How do I deal with 80+ Dev/QA Environments?


                Versioning?                              Librarian?

                                                    Pulp?
        Branching?

                                                   Internal Forge?

 Dynamic Environments?
                                            Puppet Module Tool?
                    What does PuppetLabs do?
April 9, 2013                                                     13
PuppetLabs seems to know what to do




                         Let’s investigate
                        puppet module tool




http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html



April 9, 2013                                                        14
Generate a module




http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html

April 9, 2013                                                        15
Edit Modulefile




http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013                                                        16
Document the manifest




                http://rdoc.sourceforge.net/
April 9, 2013                                  17
Write Documentation?




April 9, 2013              18
Free Docs!

        puppet doc -a -o /var/www/html/puppetdocs --mode rdoc




April 9, 2013                                                   19
What’s actually installed?




April 9, 2013                    20
Catching live edits and preventing them



                Splunk -> puppet module changes -> alerting




                       The NOC will hunt you down!


April 9, 2013                                                 21
Build and install the module




     tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub-
     puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/
     staging/modules/puppetserver




April 9, 2013                                                 22
Releasing like that?




April 9, 2013              23
Internal Forge

    mod_rewrite:
           Simulate the api – redirect to json metadata files
           $htmlroot/api/v1/releases.json?module=user/module



    ruby script:
           Generate metadata files for each module release and
           all modules.
           Similar to createrepo (yum)


April 9, 2013                                                    24
Internal Forge - Search




April 9, 2013                 25
Internal Forge - install




April 9, 2013                  26
Internal Forge - upgrade




April 9, 2013                  27
Case statements? How bout Hiera?




April 9, 2013                          28
Avoid case statement insanity

    case $::system_role {
         ‘browse’, ‘search’: {
                …do some stuff…
          }
         ‘db’: {
                …other stuff…
         }
         ‘otherrole’: {
                …please make it stop!!!!
         }
    }
April 9, 2013                              29
hieradata

    $hieradata/browse.yaml:
           ---
           module::parameter: ‘foo’

    $hieradata/search.yaml:
           ---
           module::parameter: ‘bar’

    $hieradata/defaults.yaml:
           ---
           module::parameter: ‘I want this everywhere unless there are overrides’




April 9, 2013                                                                       30
Case -> variables -> hiera

    §    9000 lines of case statements


    §    1000 lines with case/variables


    §    ~20 lines with defined type


                     Code compression FTW!


April 9, 2013                                31
Dynamic Environments

    §    puppet.conf:
    modulepath = /etc/puppet/environments/$environment/modules
    manifest       = /etc/puppet/environments/$environment/manifests/site.pp
    manifestdir = /etc/puppet/environments/$environment/manifests




    §    hiera.yaml:
    :datadir: '/etc/puppet/environments/%{environment}/hieradata'




April 9, 2013                                                              32
Release process

 §    Syntax check/validate
 §    Test on VMs
 §    Build module package
 §    Release to internal forge
 §    puppet module install to staging environment
 §    Test again!
 §    puppet module install to production environment


April 9, 2013                                            33
The road to yesop

    §    Staging
    §    Process
    §    Repeatability
    §    Consistency
    §    Document everything
    §    Breaking things where it’s cheap
    §    Test everything!


April 9, 2013                                34
Then VS now

 §    Environment build time:
         –      Then: 3+ weeks
                 •  It was wrong

                 •  It didn’t work

                 •  Nobody knew what to expect

         –      Now: < 1 day
                 •  It’s the same every time

                 •  We know exactly what’s installed

                 •  Internal consumers get what they expect

                 •  Less outages from human error


April 9, 2013                                                 35
Questions?




April 9, 2013                36

Weitere ähnliche Inhalte

Was ist angesagt?

Chef Conf 2015: Package Management & Chef
Chef Conf 2015: Package Management & ChefChef Conf 2015: Package Management & Chef
Chef Conf 2015: Package Management & Chefice799
 
Puppet Development Workflow
Puppet Development WorkflowPuppet Development Workflow
Puppet Development WorkflowJeffery Smith
 
Puppet Camp Paris 2016 Data in Modules
Puppet Camp Paris 2016 Data in ModulesPuppet Camp Paris 2016 Data in Modules
Puppet Camp Paris 2016 Data in ModulesMartin Alfke
 
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet
 
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
Bootstrapping Puppet and Application Deployment - PuppetConf 2013Bootstrapping Puppet and Application Deployment - PuppetConf 2013
Bootstrapping Puppet and Application Deployment - PuppetConf 2013Puppet
 
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Puppet
 
Learning Puppet Chapter 1
Learning Puppet Chapter 1Learning Puppet Chapter 1
Learning Puppet Chapter 1Vishal Biyani
 
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...Puppet
 
Custom Non-RDS Multi-AZ Mysql Replication
Custom Non-RDS Multi-AZ Mysql ReplicationCustom Non-RDS Multi-AZ Mysql Replication
Custom Non-RDS Multi-AZ Mysql ReplicationMichael H. Oshita
 
Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015ice799
 
Puppet getting started by Dirk Götz
Puppet getting started by Dirk GötzPuppet getting started by Dirk Götz
Puppet getting started by Dirk GötzNETWAYS
 
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014Puppet
 
Best practices for ansible
Best practices for ansibleBest practices for ansible
Best practices for ansibleGeorge Shuklin
 
Introduction to puppet
Introduction to puppetIntroduction to puppet
Introduction to puppetHabeeb Rahman
 
Towards Continuous Deployment with Django
Towards Continuous Deployment with DjangoTowards Continuous Deployment with Django
Towards Continuous Deployment with DjangoRoger Barnes
 
Django deployment best practices
Django deployment best practicesDjango deployment best practices
Django deployment best practicesErik LaBianca
 
Learn Puppet : Quest Guide for the Learning VM
Learn Puppet : Quest Guide for the Learning VMLearn Puppet : Quest Guide for the Learning VM
Learn Puppet : Quest Guide for the Learning VMKumaran Balachandran
 
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopPuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopWalter Heck
 

Was ist angesagt? (20)

Puppet fundamentals
Puppet fundamentalsPuppet fundamentals
Puppet fundamentals
 
Chef Conf 2015: Package Management & Chef
Chef Conf 2015: Package Management & ChefChef Conf 2015: Package Management & Chef
Chef Conf 2015: Package Management & Chef
 
Puppet Development Workflow
Puppet Development WorkflowPuppet Development Workflow
Puppet Development Workflow
 
Puppet Camp Paris 2016 Data in Modules
Puppet Camp Paris 2016 Data in ModulesPuppet Camp Paris 2016 Data in Modules
Puppet Camp Paris 2016 Data in Modules
 
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
 
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
Bootstrapping Puppet and Application Deployment - PuppetConf 2013Bootstrapping Puppet and Application Deployment - PuppetConf 2013
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
 
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013
 
Puppet: From 0 to 100 in 30 minutes
Puppet: From 0 to 100 in 30 minutesPuppet: From 0 to 100 in 30 minutes
Puppet: From 0 to 100 in 30 minutes
 
Learning Puppet Chapter 1
Learning Puppet Chapter 1Learning Puppet Chapter 1
Learning Puppet Chapter 1
 
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...
Building and Testing from Scratch a Puppet Environment with Docker - PuppetCo...
 
Custom Non-RDS Multi-AZ Mysql Replication
Custom Non-RDS Multi-AZ Mysql ReplicationCustom Non-RDS Multi-AZ Mysql Replication
Custom Non-RDS Multi-AZ Mysql Replication
 
Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015
 
Puppet getting started by Dirk Götz
Puppet getting started by Dirk GötzPuppet getting started by Dirk Götz
Puppet getting started by Dirk Götz
 
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014
Beaker: Automated, Cloud-Based Acceptance Testing - PuppetConf 2014
 
Best practices for ansible
Best practices for ansibleBest practices for ansible
Best practices for ansible
 
Introduction to puppet
Introduction to puppetIntroduction to puppet
Introduction to puppet
 
Towards Continuous Deployment with Django
Towards Continuous Deployment with DjangoTowards Continuous Deployment with Django
Towards Continuous Deployment with Django
 
Django deployment best practices
Django deployment best practicesDjango deployment best practices
Django deployment best practices
 
Learn Puppet : Quest Guide for the Learning VM
Learn Puppet : Quest Guide for the Learning VMLearn Puppet : Quest Guide for the Learning VM
Learn Puppet : Quest Guide for the Learning VM
 
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopPuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
 

Andere mochten auch

Monitis: All-in-One Systems Monitoring from the Cloud
Monitis: All-in-One Systems Monitoring from the CloudMonitis: All-in-One Systems Monitoring from the Cloud
Monitis: All-in-One Systems Monitoring from the CloudHovhannes Avoyan
 
Managing and Scaling Puppet - PuppetConf 2014
Managing and Scaling Puppet - PuppetConf 2014Managing and Scaling Puppet - PuppetConf 2014
Managing and Scaling Puppet - PuppetConf 2014Puppet
 
Using puppet, foreman and git to develop and operate a large scale internet s...
Using puppet, foreman and git to develop and operate a large scale internet s...Using puppet, foreman and git to develop and operate a large scale internet s...
Using puppet, foreman and git to develop and operate a large scale internet s...techblog
 
Mapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked DataMapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked DataPeter Haase
 
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic EnvironmentContinuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic EnvironmentPuppet
 
Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Colin Charles
 
Ruby application based on http
Ruby application based on httpRuby application based on http
Ruby application based on httpRichard Huang
 
vSphere APIs for performance monitoring
vSphere APIs for performance monitoringvSphere APIs for performance monitoring
vSphere APIs for performance monitoringAlan Renouf
 
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Gleicon Moraes
 
Advanced Strategic Management: Midterm
Advanced Strategic Management: MidtermAdvanced Strategic Management: Midterm
Advanced Strategic Management: MidtermAglazer1
 
Experiences from Running Masterless Puppet - PuppetConf 2014
Experiences from Running Masterless Puppet - PuppetConf 2014Experiences from Running Masterless Puppet - PuppetConf 2014
Experiences from Running Masterless Puppet - PuppetConf 2014Puppet
 
PostgreSQL Materialized Views with Active Record
PostgreSQL Materialized Views with Active RecordPostgreSQL Materialized Views with Active Record
PostgreSQL Materialized Views with Active RecordDavid Roberts
 
The Complete MariaDB Server Tutorial - Percona Live 2015
The Complete MariaDB Server Tutorial - Percona Live 2015The Complete MariaDB Server Tutorial - Percona Live 2015
The Complete MariaDB Server Tutorial - Percona Live 2015Colin Charles
 
Redis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational DatabasesRedis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational DatabasesKarel Minarik
 
Taking Control of Chaos with Docker and Puppet
Taking Control of Chaos with Docker and PuppetTaking Control of Chaos with Docker and Puppet
Taking Control of Chaos with Docker and PuppetPuppet
 
Detecting headless browsers
Detecting headless browsersDetecting headless browsers
Detecting headless browsersSergey Shekyan
 
Monitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code AgeMonitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code AgePuppet
 
How to make keynote like presentation with markdown
How to make keynote like presentation with markdownHow to make keynote like presentation with markdown
How to make keynote like presentation with markdownHiroaki NAKADA
 

Andere mochten auch (20)

Monitis: All-in-One Systems Monitoring from the Cloud
Monitis: All-in-One Systems Monitoring from the CloudMonitis: All-in-One Systems Monitoring from the Cloud
Monitis: All-in-One Systems Monitoring from the Cloud
 
Managing and Scaling Puppet - PuppetConf 2014
Managing and Scaling Puppet - PuppetConf 2014Managing and Scaling Puppet - PuppetConf 2014
Managing and Scaling Puppet - PuppetConf 2014
 
Using puppet, foreman and git to develop and operate a large scale internet s...
Using puppet, foreman and git to develop and operate a large scale internet s...Using puppet, foreman and git to develop and operate a large scale internet s...
Using puppet, foreman and git to develop and operate a large scale internet s...
 
Mapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked DataMapping, Interlinking and Exposing MusicBrainz as Linked Data
Mapping, Interlinking and Exposing MusicBrainz as Linked Data
 
Continuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic EnvironmentContinuously-Integrated Puppet in a Dynamic Environment
Continuously-Integrated Puppet in a Dynamic Environment
 
JSON and the APInauts
JSON and the APInautsJSON and the APInauts
JSON and the APInauts
 
Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7
 
Ruby application based on http
Ruby application based on httpRuby application based on http
Ruby application based on http
 
Sensu
SensuSensu
Sensu
 
vSphere APIs for performance monitoring
vSphere APIs for performance monitoringvSphere APIs for performance monitoring
vSphere APIs for performance monitoring
 
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
 
Advanced Strategic Management: Midterm
Advanced Strategic Management: MidtermAdvanced Strategic Management: Midterm
Advanced Strategic Management: Midterm
 
Experiences from Running Masterless Puppet - PuppetConf 2014
Experiences from Running Masterless Puppet - PuppetConf 2014Experiences from Running Masterless Puppet - PuppetConf 2014
Experiences from Running Masterless Puppet - PuppetConf 2014
 
PostgreSQL Materialized Views with Active Record
PostgreSQL Materialized Views with Active RecordPostgreSQL Materialized Views with Active Record
PostgreSQL Materialized Views with Active Record
 
The Complete MariaDB Server Tutorial - Percona Live 2015
The Complete MariaDB Server Tutorial - Percona Live 2015The Complete MariaDB Server Tutorial - Percona Live 2015
The Complete MariaDB Server Tutorial - Percona Live 2015
 
Redis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational DatabasesRedis — The AK-47 of Post-relational Databases
Redis — The AK-47 of Post-relational Databases
 
Taking Control of Chaos with Docker and Puppet
Taking Control of Chaos with Docker and PuppetTaking Control of Chaos with Docker and Puppet
Taking Control of Chaos with Docker and Puppet
 
Detecting headless browsers
Detecting headless browsersDetecting headless browsers
Detecting headless browsers
 
Monitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code AgeMonitoring in an Infrastructure as Code Age
Monitoring in an Infrastructure as Code Age
 
How to make keynote like presentation with markdown
How to make keynote like presentation with markdownHow to make keynote like presentation with markdown
How to make keynote like presentation with markdown
 

Ähnlich wie Lessons I Learned While Scaling to 5000 Puppet Agents

Recommender Systems with Ruby (adding machine learning, statistics, etc)
Recommender Systems with Ruby (adding machine learning, statistics, etc)Recommender Systems with Ruby (adding machine learning, statistics, etc)
Recommender Systems with Ruby (adding machine learning, statistics, etc)Marcel Caraciolo
 
Engineering culture
Engineering cultureEngineering culture
Engineering culturePamela Fox
 
Everything You Were Taught About Java Is Wrong
Everything You Were Taught About Java Is WrongEverything You Were Taught About Java Is Wrong
Everything You Were Taught About Java Is WrongTim Boudreau
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationPuppet
 
PHP Mega Meetup, Sep, 2020, Anti patterns in php
PHP Mega Meetup, Sep, 2020, Anti patterns in phpPHP Mega Meetup, Sep, 2020, Anti patterns in php
PHP Mega Meetup, Sep, 2020, Anti patterns in phpAhmed Abdou
 
Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1Max De Marzi
 
Off the Treadmill: Building a Drupal Platform for Your Organization
Off the Treadmill: Building a Drupal Platform for Your OrganizationOff the Treadmill: Building a Drupal Platform for Your Organization
Off the Treadmill: Building a Drupal Platform for Your OrganizationRick Vugteveen
 
The State of Puppet
The State of PuppetThe State of Puppet
The State of PuppetPuppet
 
Introduction to Express and Grunt
Introduction to Express and GruntIntroduction to Express and Grunt
Introduction to Express and GruntPeter deHaan
 
Introduction into Drupal site building
Introduction into Drupal site buildingIntroduction into Drupal site building
Introduction into Drupal site buildingIztok Smolic
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingDmitry Spodarets
 
Exploring composer in drupal 8 with drupal project - salva molina
Exploring composer in drupal 8 with drupal project - salva molinaExploring composer in drupal 8 with drupal project - salva molina
Exploring composer in drupal 8 with drupal project - salva molinaSalvador Molina (Slv_)
 
Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Puppet
 
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...AMD Developer Central
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on ContainersCorwin Brown
 
Bay Area Drupal Camp Efficiency
Bay Area Drupal Camp EfficiencyBay Area Drupal Camp Efficiency
Bay Area Drupal Camp Efficiencysmattoon
 
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.Peter Hurford
 
Atlaskickin' the Plugin SDK, AtlasCamp US 2012
Atlaskickin' the Plugin SDK, AtlasCamp US 2012Atlaskickin' the Plugin SDK, AtlasCamp US 2012
Atlaskickin' the Plugin SDK, AtlasCamp US 2012Atlassian
 

Ähnlich wie Lessons I Learned While Scaling to 5000 Puppet Agents (20)

Recommender Systems with Ruby (adding machine learning, statistics, etc)
Recommender Systems with Ruby (adding machine learning, statistics, etc)Recommender Systems with Ruby (adding machine learning, statistics, etc)
Recommender Systems with Ruby (adding machine learning, statistics, etc)
 
Engineering culture
Engineering cultureEngineering culture
Engineering culture
 
Everything You Were Taught About Java Is Wrong
Everything You Were Taught About Java Is WrongEverything You Were Taught About Java Is Wrong
Everything You Were Taught About Java Is Wrong
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global Organization
 
PHP Mega Meetup, Sep, 2020, Anti patterns in php
PHP Mega Meetup, Sep, 2020, Anti patterns in phpPHP Mega Meetup, Sep, 2020, Anti patterns in php
PHP Mega Meetup, Sep, 2020, Anti patterns in php
 
Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1
 
Off the Treadmill: Building a Drupal Platform for Your Organization
Off the Treadmill: Building a Drupal Platform for Your OrganizationOff the Treadmill: Building a Drupal Platform for Your Organization
Off the Treadmill: Building a Drupal Platform for Your Organization
 
The State of Puppet
The State of PuppetThe State of Puppet
The State of Puppet
 
CloudInit Introduction
CloudInit IntroductionCloudInit Introduction
CloudInit Introduction
 
Introduction to Express and Grunt
Introduction to Express and GruntIntroduction to Express and Grunt
Introduction to Express and Grunt
 
Introduction into Drupal site building
Introduction into Drupal site buildingIntroduction into Drupal site building
Introduction into Drupal site building
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance Computing
 
Exploring composer in drupal 8 with drupal project - salva molina
Exploring composer in drupal 8 with drupal project - salva molinaExploring composer in drupal 8 with drupal project - salva molina
Exploring composer in drupal 8 with drupal project - salva molina
 
Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...
 
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...
PL-4047, Big Data Workload Analysis Using SWAT and Ipython Notebooks, by Moni...
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on Containers
 
Bay Area Drupal Camp Efficiency
Bay Area Drupal Camp EfficiencyBay Area Drupal Camp Efficiency
Bay Area Drupal Camp Efficiency
 
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.
Vowpal Platypus: Very Fast Multi-Core Machine Learning in Python.
 
Drupal Development Tips
Drupal Development TipsDrupal Development Tips
Drupal Development Tips
 
Atlaskickin' the Plugin SDK, AtlasCamp US 2012
Atlaskickin' the Plugin SDK, AtlasCamp US 2012Atlaskickin' the Plugin SDK, AtlasCamp US 2012
Atlaskickin' the Plugin SDK, AtlasCamp US 2012
 

Mehr von Puppet

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyamlPuppet
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)Puppet
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscodePuppet
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twentiesPuppet
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codePuppet
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approachPuppet
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationPuppet
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliancePuppet
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowPuppet
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Puppet
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppetPuppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkPuppet
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping groundPuppet
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy SoftwarePuppet
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User GroupPuppet
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsPuppet
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyPuppet
 

Mehr von Puppet (20)

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepo
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyaml
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscode
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twenties
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance code
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approach
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliance
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNow
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden Windows
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael Pinson
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin Reeuwijk
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping ground
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User Group
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOps
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
 

Kürzlich hochgeladen

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 

Kürzlich hochgeladen (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 

Lessons I Learned While Scaling to 5000 Puppet Agents

  • 1. Learning Lessons Scaling to 5000 Agents Russ Johnson rjohnson@stubhub.com @professoruss Confidential Slide 1
  • 2. #whoami Started out in the mid 90’s Recovering Windows Admin Storage Guy Datacenter Monkey Once upon a time network guy At StubHub since December 2006 Working on puppet adoption in a crazy infrastructure Puppet certified April 9, 2013 2
  • 3. 4 Puppet Masters DEV/QA PROD DR CORP April 9, 2013 3
  • 4. The road to sanity April 9, 2013 4
  • 5. Set up your master properly Apache/Passenger Tune Passenger PassengerMaxPoolSize 32 PassengerMinInstances 4 PassengerMaxRequests 10000 PassengerStatThrottleRate 30 16 cores, 32GB load average: 4.03, 3.71, 3.45 4000+ agents April 9, 2013 5
  • 7. Build your hosts the same way! Old way: systemimager, vmware clones, manual installs Results: INCONSISTENCY! April 9, 2013 7
  • 8. Build your hosts the same way! New way: Cobbler < 5m bare metal to on the network Results: Same results every time! No drift between base April 9, 2013 8
  • 9. Set up your working environment properly Geppetto – eclipse based IDE http://cloudsmith.github.com/geppetto/index.html VIM •  Pathogen – For autoloading vim plugins •  Snipmate – Snippets •  Tabular – Text filtering and alignment •  Syntastic – Syntax checking •  mv-vim-puppet – Make vim puppet friendly •  puppet-lint – Syntax checker (gem) April 9, 2013 9
  • 11. Set up your working environment properly + April 9, 2013 11
  • 12. Version Control is not enough Ever do a 4 way diff across 60 modules to find most of them different? April 9, 2013 12
  • 13. What to do? How do I stop manual edits? How do I deal with 80+ Dev/QA Environments? Versioning? Librarian? Pulp? Branching? Internal Forge? Dynamic Environments? Puppet Module Tool? What does PuppetLabs do? April 9, 2013 13
  • 14. PuppetLabs seems to know what to do Let’s investigate puppet module tool http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html April 9, 2013 14
  • 17. Document the manifest http://rdoc.sourceforge.net/ April 9, 2013 17
  • 19. Free Docs! puppet doc -a -o /var/www/html/puppetdocs --mode rdoc April 9, 2013 19
  • 21. Catching live edits and preventing them Splunk -> puppet module changes -> alerting The NOC will hunt you down! April 9, 2013 21
  • 22. Build and install the module tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub- puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/ staging/modules/puppetserver April 9, 2013 22
  • 24. Internal Forge mod_rewrite: Simulate the api – redirect to json metadata files $htmlroot/api/v1/releases.json?module=user/module ruby script: Generate metadata files for each module release and all modules. Similar to createrepo (yum) April 9, 2013 24
  • 25. Internal Forge - Search April 9, 2013 25
  • 26. Internal Forge - install April 9, 2013 26
  • 27. Internal Forge - upgrade April 9, 2013 27
  • 28. Case statements? How bout Hiera? April 9, 2013 28
  • 29. Avoid case statement insanity case $::system_role { ‘browse’, ‘search’: { …do some stuff… } ‘db’: { …other stuff… } ‘otherrole’: { …please make it stop!!!! } } April 9, 2013 29
  • 30. hieradata $hieradata/browse.yaml: --- module::parameter: ‘foo’ $hieradata/search.yaml: --- module::parameter: ‘bar’ $hieradata/defaults.yaml: --- module::parameter: ‘I want this everywhere unless there are overrides’ April 9, 2013 30
  • 31. Case -> variables -> hiera §  9000 lines of case statements §  1000 lines with case/variables §  ~20 lines with defined type Code compression FTW! April 9, 2013 31
  • 32. Dynamic Environments §  puppet.conf: modulepath = /etc/puppet/environments/$environment/modules manifest = /etc/puppet/environments/$environment/manifests/site.pp manifestdir = /etc/puppet/environments/$environment/manifests §  hiera.yaml: :datadir: '/etc/puppet/environments/%{environment}/hieradata' April 9, 2013 32
  • 33. Release process §  Syntax check/validate §  Test on VMs §  Build module package §  Release to internal forge §  puppet module install to staging environment §  Test again! §  puppet module install to production environment April 9, 2013 33
  • 34. The road to yesop §  Staging §  Process §  Repeatability §  Consistency §  Document everything §  Breaking things where it’s cheap §  Test everything! April 9, 2013 34
  • 35. Then VS now §  Environment build time: –  Then: 3+ weeks •  It was wrong •  It didn’t work •  Nobody knew what to expect –  Now: < 1 day •  It’s the same every time •  We know exactly what’s installed •  Internal consumers get what they expect •  Less outages from human error April 9, 2013 35