SlideShare a Scribd company logo
1 of 13
Deploying Puppet Code At Light Speed
Tomas Doran
@bobtfish
tdoran@yelp.com
2014-01-27
Key lessons
•Speed of light is (still) slow
– West coast => EU slow
– East coast => Asia slow

•git <3
– Shipping minimal changes = fast

•Eventual consistency FTW
– Centrally orchestrating a global network = LOL
– Parts of the network will be down

•mcollective <3
– Yes, it’s a chainsaw
– Decoupling good!
What we did
•New puppet deployment system
• svn => git
• ssh for loop => parallel mcollective
• All users sudo root => mcollective policies
• push => pull

•Winning!
– 4m deployment
– 4s deployment
– 32 puppetmasters globally
Why?
•Pull models > Push models
– Eventual consistency FTW
– In a large network, you will have failure - don’t rely on
100% up to ship new code
– Just cron updates to environments you care about

•Environments
•
•
•
•

Personal branches
Demo/test your code with —environment
Different teams can own different parts of the code
Staged rollouts by merging changes between
branches
How?

• Environments in puppet.conf
• For modules
• modulepath = /etc/puppet/environments/
$environment/modules
• For site.pp
• manifest = /etc/puppet/environments/
$environment/manifests/site.pp
• For hiera data
• datadir = /etc/puppet/environments
• %{::environment}/common.yaml
• For manifests/ - refactor!!!
Segue - ‘refactor’
•Originally it meant…
• Change form
• But not function
• incrementally
• Supported by tests

•I’m giving up on that meaning…
• So abused by everyone, all the time
• It just means ‘change shit’
• See also hacker/cracker…
• We won that one, right?
How?
•Custom mcollective agent
–Every git branch => puppet environment
– 205 lines of code
– 215 lines of tests ;)

•Cron job
–
–
–
–

mcollective agent also ships a CLI (local) client
Just cron updates to all the branches you care about
(or all branches)
Eventual consistency!
Detached work trees
•/etc/puppet/puppet.git
git clone —bare
git —git-dir=/etc/puppet/puppet.git
—worktree=/etc/puppet/environments/my
_example checkout --detach
--force master
Gitolite
•Allows role users
– Generate ‘puppetupdate’ ssh key.
– Allow this to READ the puppet code.
– Distribute to puppet masters

•All the access controls
–
–
–
–

Multiple puppetupdate ssh keys.
Allow different keys different branches
dev/stage/prod
I don’t need this _yet_, but it’s there!
Gotchas
•ssh concurrency limits
– Defaults are conservative
– Limit number of processes in ‘preauth’

•Scaling MOAR
– If you have 100s of puppet masters
– You’re gonna want to have multiple git servers
– Still easy, just 2 step orchestration:
– Pull to all slave git servers
– puppetupdate all the masters
TODOs
•Documentation not on slideshare
– I did fix the README, it’s still not awesome.
– Only 250 lines of code, just reading it isn’t hard ;)

•Better application
– Nicer user display of status

•Extend mcollective
– mco plugin package only bundles mco bits
– You need to ship /usr/local/sbin/puppetupdate
yourself
Link me?
• https://github.com/Yelp/puppetupdate
• Slides: http://slideshare.net/bobtfish
• Tweet me @bobtfish
• Guess what?
• We’re hiring!!!
• SF + Palo Alto
• (+ London, + Hamburg, + Dublin)

More Related Content

What's hot

Tips on High Performance Server Programming
Tips on High Performance Server ProgrammingTips on High Performance Server Programming
Tips on High Performance Server ProgrammingJoshua Zhu
 
DCSF 19 eBPF Superpowers
DCSF 19 eBPF SuperpowersDCSF 19 eBPF Superpowers
DCSF 19 eBPF SuperpowersDocker, Inc.
 
Measuring directly from cpu hardware performance counters
Measuring directly from cpu  hardware performance countersMeasuring directly from cpu  hardware performance counters
Measuring directly from cpu hardware performance countersJean-Philippe BEMPEL
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
Performance optimization 101 - Erlang Factory SF 2014
Performance optimization 101 - Erlang Factory SF 2014Performance optimization 101 - Erlang Factory SF 2014
Performance optimization 101 - Erlang Factory SF 2014lpgauth
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing LandscapeKernel TLV
 
Understanding of linux kernel memory model
Understanding of linux kernel memory modelUnderstanding of linux kernel memory model
Understanding of linux kernel memory modelSeongJae Park
 
Get Lower Latency and Higher Throughput for Java Applications
Get Lower Latency and Higher Throughput for Java ApplicationsGet Lower Latency and Higher Throughput for Java Applications
Get Lower Latency and Higher Throughput for Java ApplicationsScyllaDB
 
Epoll - from the kernel side
Epoll -  from the kernel sideEpoll -  from the kernel side
Epoll - from the kernel sidellj098
 
Troubleshooting common oslo.messaging and RabbitMQ issues
Troubleshooting common oslo.messaging and RabbitMQ issuesTroubleshooting common oslo.messaging and RabbitMQ issues
Troubleshooting common oslo.messaging and RabbitMQ issuesMichael Klishin
 
Intel processor trace - What are Recorded?
Intel processor trace - What are Recorded?Intel processor trace - What are Recorded?
Intel processor trace - What are Recorded?Pipat Methavanitpong
 
Nerves Project Intro to ErlangDC
Nerves Project Intro to ErlangDCNerves Project Intro to ErlangDC
Nerves Project Intro to ErlangDCFrank Hunleth
 
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...Continuent
 
Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)Jyrki Pulliainen
 
Windows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonWindows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonSeungmo Koo
 
Hermes Reliable Replication Protocol - ASPLOS'20 Presentation
Hermes Reliable Replication Protocol -  ASPLOS'20 PresentationHermes Reliable Replication Protocol -  ASPLOS'20 Presentation
Hermes Reliable Replication Protocol - ASPLOS'20 PresentationAntonios Katsarakis
 
Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka
Kafka Summit NYC 2017 - Deep Dive Into Apache KafkaKafka Summit NYC 2017 - Deep Dive Into Apache Kafka
Kafka Summit NYC 2017 - Deep Dive Into Apache Kafkaconfluent
 
Guider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLGuider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLPeace Lee
 
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...ScyllaDB
 

What's hot (20)

Tips on High Performance Server Programming
Tips on High Performance Server ProgrammingTips on High Performance Server Programming
Tips on High Performance Server Programming
 
DCSF 19 eBPF Superpowers
DCSF 19 eBPF SuperpowersDCSF 19 eBPF Superpowers
DCSF 19 eBPF Superpowers
 
Measuring directly from cpu hardware performance counters
Measuring directly from cpu  hardware performance countersMeasuring directly from cpu  hardware performance counters
Measuring directly from cpu hardware performance counters
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
Network
NetworkNetwork
Network
 
Performance optimization 101 - Erlang Factory SF 2014
Performance optimization 101 - Erlang Factory SF 2014Performance optimization 101 - Erlang Factory SF 2014
Performance optimization 101 - Erlang Factory SF 2014
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
Understanding of linux kernel memory model
Understanding of linux kernel memory modelUnderstanding of linux kernel memory model
Understanding of linux kernel memory model
 
Get Lower Latency and Higher Throughput for Java Applications
Get Lower Latency and Higher Throughput for Java ApplicationsGet Lower Latency and Higher Throughput for Java Applications
Get Lower Latency and Higher Throughput for Java Applications
 
Epoll - from the kernel side
Epoll -  from the kernel sideEpoll -  from the kernel side
Epoll - from the kernel side
 
Troubleshooting common oslo.messaging and RabbitMQ issues
Troubleshooting common oslo.messaging and RabbitMQ issuesTroubleshooting common oslo.messaging and RabbitMQ issues
Troubleshooting common oslo.messaging and RabbitMQ issues
 
Intel processor trace - What are Recorded?
Intel processor trace - What are Recorded?Intel processor trace - What are Recorded?
Intel processor trace - What are Recorded?
 
Nerves Project Intro to ErlangDC
Nerves Project Intro to ErlangDCNerves Project Intro to ErlangDC
Nerves Project Intro to ErlangDC
 
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...
Training Slides: Intermediate 204: Identifying and Resolving Issues with Tung...
 
Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)
 
Windows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonWindows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance Comparison
 
Hermes Reliable Replication Protocol - ASPLOS'20 Presentation
Hermes Reliable Replication Protocol -  ASPLOS'20 PresentationHermes Reliable Replication Protocol -  ASPLOS'20 Presentation
Hermes Reliable Replication Protocol - ASPLOS'20 Presentation
 
Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka
Kafka Summit NYC 2017 - Deep Dive Into Apache KafkaKafka Summit NYC 2017 - Deep Dive Into Apache Kafka
Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka
 
Guider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLGuider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGL
 
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
 

Viewers also liked

Slide1 Introduction
Slide1  IntroductionSlide1  Introduction
Slide1 Introductiondianechanmp
 
Portfolio Jonas Becker
Portfolio Jonas BeckerPortfolio Jonas Becker
Portfolio Jonas BeckerIndianajonas
 
Building an Enthusiastic Club
Building an Enthusiastic ClubBuilding an Enthusiastic Club
Building an Enthusiastic Clubguest7c6875
 
Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Puppet
 
Building On Puppet and Puppet Forge
Building On Puppet and Puppet ForgeBuilding On Puppet and Puppet Forge
Building On Puppet and Puppet ForgePuppet
 
Cakoi Stall Final97
Cakoi Stall Final97Cakoi Stall Final97
Cakoi Stall Final97dianechanmp
 
SMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar ChartSMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar Chartdianechanmp
 
Cakoi Stall Final97
Cakoi Stall Final97Cakoi Stall Final97
Cakoi Stall Final97dianechanmp
 
Google Presentation2
Google Presentation2Google Presentation2
Google Presentation2dianechanmp
 
SMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar ChartSMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar Chartdianechanmp
 
Wrangling 3rd Party Installers from Puppet
Wrangling 3rd Party Installers from PuppetWrangling 3rd Party Installers from Puppet
Wrangling 3rd Party Installers from PuppetPuppet
 
Ppt psv 11 kump.2
Ppt psv 11 kump.2Ppt psv 11 kump.2
Ppt psv 11 kump.2ninasweet
 
Puppet Module Best Practices - Puppet Camp Silicon Valley
Puppet Module Best Practices - Puppet Camp Silicon ValleyPuppet Module Best Practices - Puppet Camp Silicon Valley
Puppet Module Best Practices - Puppet Camp Silicon ValleyPuppet
 
Cisco Router As A Vpn Server
Cisco Router As A Vpn ServerCisco Router As A Vpn Server
Cisco Router As A Vpn Servermmoizuddin
 
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunselingninasweet
 
SMS Presentation
SMS PresentationSMS Presentation
SMS Presentationdianechanmp
 

Viewers also liked (19)

Kuliah1
Kuliah1Kuliah1
Kuliah1
 
Slide1 Introduction
Slide1  IntroductionSlide1  Introduction
Slide1 Introduction
 
2010-10 WCR
2010-10 WCR2010-10 WCR
2010-10 WCR
 
Portfolio Jonas Becker
Portfolio Jonas BeckerPortfolio Jonas Becker
Portfolio Jonas Becker
 
Building an Enthusiastic Club
Building an Enthusiastic ClubBuilding an Enthusiastic Club
Building an Enthusiastic Club
 
Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...Building scalable applications while scaling your infrastructure by rhommel l...
Building scalable applications while scaling your infrastructure by rhommel l...
 
Building On Puppet and Puppet Forge
Building On Puppet and Puppet ForgeBuilding On Puppet and Puppet Forge
Building On Puppet and Puppet Forge
 
Cakoi Stall Final97
Cakoi Stall Final97Cakoi Stall Final97
Cakoi Stall Final97
 
SMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar ChartSMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar Chart
 
Ciscovpn
CiscovpnCiscovpn
Ciscovpn
 
Cakoi Stall Final97
Cakoi Stall Final97Cakoi Stall Final97
Cakoi Stall Final97
 
Google Presentation2
Google Presentation2Google Presentation2
Google Presentation2
 
SMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar ChartSMS Presentation with New Proposal Of Bar Chart
SMS Presentation with New Proposal Of Bar Chart
 
Wrangling 3rd Party Installers from Puppet
Wrangling 3rd Party Installers from PuppetWrangling 3rd Party Installers from Puppet
Wrangling 3rd Party Installers from Puppet
 
Ppt psv 11 kump.2
Ppt psv 11 kump.2Ppt psv 11 kump.2
Ppt psv 11 kump.2
 
Puppet Module Best Practices - Puppet Camp Silicon Valley
Puppet Module Best Practices - Puppet Camp Silicon ValleyPuppet Module Best Practices - Puppet Camp Silicon Valley
Puppet Module Best Practices - Puppet Camp Silicon Valley
 
Cisco Router As A Vpn Server
Cisco Router As A Vpn ServerCisco Router As A Vpn Server
Cisco Router As A Vpn Server
 
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling
122256451 fatin-azimah-jusoh-perbandingan-teori-teori-kaunseling
 
SMS Presentation
SMS PresentationSMS Presentation
SMS Presentation
 

Similar to Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley

Puppet Release Workflows at Jive Software
Puppet Release Workflows at Jive SoftwarePuppet Release Workflows at Jive Software
Puppet Release Workflows at Jive SoftwarePuppet
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflowTomas Doran
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scalePierre Souchay
 
Puppet Development Workflow
Puppet Development WorkflowPuppet Development Workflow
Puppet Development WorkflowJeffery Smith
 
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...Ihor Banadiga
 
Understanding salt modular sub-systems and customization
Understanding salt   modular sub-systems and customizationUnderstanding salt   modular sub-systems and customization
Understanding salt modular sub-systems and customizationjasondenning
 
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docx
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docxCIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docx
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docxclarebernice
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
Cooking a rabbit pie
Cooking a rabbit pieCooking a rabbit pie
Cooking a rabbit pieTomas Doran
 
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like ServersPuppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like ServersPuppet
 
systemd @ Facebook -- a year later
systemd @ Facebook -- a year latersystemd @ Facebook -- a year later
systemd @ Facebook -- a year laterDavide Cavalca
 
systemd @ Facebook in 2019
systemd @ Facebook in 2019systemd @ Facebook in 2019
systemd @ Facebook in 2019Davide Cavalca
 
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!Puppet
 
Splunk: Forward me the REST of those shells
Splunk: Forward me the REST of those shellsSplunk: Forward me the REST of those shells
Splunk: Forward me the REST of those shellsAnthony D Hendricks
 
PHP Backends for Real-Time User Interaction using Apache Storm.
PHP Backends for Real-Time User Interaction using Apache Storm.PHP Backends for Real-Time User Interaction using Apache Storm.
PHP Backends for Real-Time User Interaction using Apache Storm.DECK36
 

Similar to Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley (20)

Puppet Release Workflows at Jive Software
Puppet Release Workflows at Jive SoftwarePuppet Release Workflows at Jive Software
Puppet Release Workflows at Jive Software
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
 
Puppet Development Workflow
Puppet Development WorkflowPuppet Development Workflow
Puppet Development Workflow
 
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...
Ansible for Configuration Management for Lohika DevOps training 2018 @ Lohika...
 
Understanding salt modular sub-systems and customization
Understanding salt   modular sub-systems and customizationUnderstanding salt   modular sub-systems and customization
Understanding salt modular sub-systems and customization
 
Who Broke My Crypto
Who Broke My CryptoWho Broke My Crypto
Who Broke My Crypto
 
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docx
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docxCIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docx
CIS3110 Winter 2016CIS3110 (Operating Systems) Assig.docx
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
Cooking a rabbit pie
Cooking a rabbit pieCooking a rabbit pie
Cooking a rabbit pie
 
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like ServersPuppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
 
Introducing E-Cell 3.2
Introducing E-Cell 3.2Introducing E-Cell 3.2
Introducing E-Cell 3.2
 
systemd @ Facebook -- a year later
systemd @ Facebook -- a year latersystemd @ Facebook -- a year later
systemd @ Facebook -- a year later
 
systemd @ Facebook in 2019
systemd @ Facebook in 2019systemd @ Facebook in 2019
systemd @ Facebook in 2019
 
UNIX Notes
UNIX NotesUNIX Notes
UNIX Notes
 
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
 
Ansible - A 'crowd' introduction
Ansible - A 'crowd' introductionAnsible - A 'crowd' introduction
Ansible - A 'crowd' introduction
 
Splunk: Forward me the REST of those shells
Splunk: Forward me the REST of those shellsSplunk: Forward me the REST of those shells
Splunk: Forward me the REST of those shells
 
PHP Backends for Real-Time User Interaction using Apache Storm.
PHP Backends for Real-Time User Interaction using Apache Storm.PHP Backends for Real-Time User Interaction using Apache Storm.
PHP Backends for Real-Time User Interaction using Apache Storm.
 

More from Puppet

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyamlPuppet
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)Puppet
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscodePuppet
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twentiesPuppet
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codePuppet
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approachPuppet
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationPuppet
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliancePuppet
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowPuppet
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Puppet
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppetPuppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkPuppet
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping groundPuppet
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy SoftwarePuppet
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User GroupPuppet
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsPuppet
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyPuppet
 

More from Puppet (20)

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepo
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyaml
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscode
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twenties
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance code
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approach
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliance
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNow
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden Windows
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael Pinson
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin Reeuwijk
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping ground
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User Group
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOps
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
 

Recently uploaded

My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfRankYa
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embeddingZilliz
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 

Recently uploaded (20)

My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embedding
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 

Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley

  • 1.
  • 2. Deploying Puppet Code At Light Speed Tomas Doran @bobtfish tdoran@yelp.com 2014-01-27
  • 3. Key lessons •Speed of light is (still) slow – West coast => EU slow – East coast => Asia slow •git <3 – Shipping minimal changes = fast •Eventual consistency FTW – Centrally orchestrating a global network = LOL – Parts of the network will be down •mcollective <3 – Yes, it’s a chainsaw – Decoupling good!
  • 4. What we did •New puppet deployment system • svn => git • ssh for loop => parallel mcollective • All users sudo root => mcollective policies • push => pull •Winning! – 4m deployment – 4s deployment – 32 puppetmasters globally
  • 5. Why? •Pull models > Push models – Eventual consistency FTW – In a large network, you will have failure - don’t rely on 100% up to ship new code – Just cron updates to environments you care about •Environments • • • • Personal branches Demo/test your code with —environment Different teams can own different parts of the code Staged rollouts by merging changes between branches
  • 6. How? • Environments in puppet.conf • For modules • modulepath = /etc/puppet/environments/ $environment/modules • For site.pp • manifest = /etc/puppet/environments/ $environment/manifests/site.pp • For hiera data • datadir = /etc/puppet/environments • %{::environment}/common.yaml • For manifests/ - refactor!!!
  • 7. Segue - ‘refactor’ •Originally it meant… • Change form • But not function • incrementally • Supported by tests •I’m giving up on that meaning… • So abused by everyone, all the time • It just means ‘change shit’ • See also hacker/cracker… • We won that one, right?
  • 8. How? •Custom mcollective agent –Every git branch => puppet environment – 205 lines of code – 215 lines of tests ;) •Cron job – – – – mcollective agent also ships a CLI (local) client Just cron updates to all the branches you care about (or all branches) Eventual consistency!
  • 9. Detached work trees •/etc/puppet/puppet.git git clone —bare git —git-dir=/etc/puppet/puppet.git —worktree=/etc/puppet/environments/my _example checkout --detach --force master
  • 10. Gitolite •Allows role users – Generate ‘puppetupdate’ ssh key. – Allow this to READ the puppet code. – Distribute to puppet masters •All the access controls – – – – Multiple puppetupdate ssh keys. Allow different keys different branches dev/stage/prod I don’t need this _yet_, but it’s there!
  • 11. Gotchas •ssh concurrency limits – Defaults are conservative – Limit number of processes in ‘preauth’ •Scaling MOAR – If you have 100s of puppet masters – You’re gonna want to have multiple git servers – Still easy, just 2 step orchestration: – Pull to all slave git servers – puppetupdate all the masters
  • 12. TODOs •Documentation not on slideshare – I did fix the README, it’s still not awesome. – Only 250 lines of code, just reading it isn’t hard ;) •Better application – Nicer user display of status •Extend mcollective – mco plugin package only bundles mco bits – You need to ship /usr/local/sbin/puppetupdate yourself
  • 13. Link me? • https://github.com/Yelp/puppetupdate • Slides: http://slideshare.net/bobtfish • Tweet me @bobtfish • Guess what? • We’re hiring!!! • SF + Palo Alto • (+ London, + Hamburg, + Dublin)