Jasper Poppe

1. PuppetConf Portland 2011
Automated deployment With Seedbank and Puppet
Puppetconf 2011 by Jasper Poppe

2. Introduction
- Who is Jasper Poppe
- Current position at the eBay Classifieds Group
- eBay Classifieds Group
- eBay Classifieds Group platforms
Puppetconf 2011 by Jasper Poppe

3. The Talk
- Automated live installation of an infrastructure from scratch
- Explanation and demo of Infrastucture Anywhere
- Explanation of seedBank
- Explanation of our Puppet Structure
- Open source seedBank, Infrastructure anywhere and a selection of our Puppet Modules
- There will be time for questions at the half and at the end of the talk
Puppetconf 2011 by Jasper Poppe

4. Infrastructure Anywhere
CLouds are so 2011
Puppetconf 2011 by Jasper Poppe

5. Prologue
- Infrastructure from Scratch: Live Installation
Puppetconf 2011 by Jasper Poppe

6. Terminilogy
- Overlord - An Overlord is a server which is running the core applications needed for a
fully automated infrastructure
- Minion - The Overlord deploys automatically to those servers
- Seed file - A file which provides answers to the Debian installer, seeding is the core
method used for Debian package configuration
- seedBank - This is a simple CLI tool/daemon to manage Debian and Ubuntu network in-
stallations. Since it is based on seed files it will provide the cleanest installations possible
by just using the standard Debian tools. seedBank is written by me in Python.
- Fabric - A deployment tool based on Python, in my own words it is BASH in Python with
some additional features to make deployment tasks easier.
- Infrastructure Anywhere - Some simple Fabric scripts which are the glue between seed-
Bank and Puppet, it provides the components to do fully automated deployments of a
core infrastructure from scratch in a production, virtual, cloud based or any other
environment.
Puppetconf 2011 by Jasper Poppe

7. Infrastructure Anywhere
fabfile.py
chicken.py
powerdns.py
dnsmasq.py
vmware.py
libvirt.py
virtualbox.py
Puppetconf 2011 by Jasper Poppe

8. Infrastructure Anywhere
Debian ISO
Infrastructure
ISO Modified for
Anywhere
Overlord Bootstrap
Overlay Directory
rc.local
Seed File
SSH Key
Puppet Repository
Puppetconf 2011 by Jasper Poppe

9. Infrastructure Anywhere
Debian ISO
Overlord
Infrastructure
ISO Modified for
Anywhere
Overlord Bootstrap
Overlay Directory
rc.local
Seed File
SSH Key
Puppet Repository
Minions
Puppetconf 2011 by Jasper Poppe

10. Infrastructure Anywhere
Infrastructure
Anywhere
Overlord
seedBank
Gateway
TFTP
DNS
DHCP
Puppetmaster
Minions
Local Network
Internet
Puppetconf 2011 by Jasper Poppe

11. InfraStructure Anywhere
overlord001
a
c
m
e
.
.
.
.
Hostname
Platform
TLD
Zone
Location
- Zones and locations are used in our platform base modules
- Platform base modules configure about 95% of the machine
- Makes it easy to override settings from a production to a QA environment while still
using exactly the same Puppet code
- For now extlookup is used
Puppetconf 2011 by Jasper Poppe

12. seedbank
The Cleanest way of automated Debian and Ubuntu
Network Installations Possible
Puppetconf 2011 by Jasper Poppe

13. seedbank
- Simplicity
- Easy netboot file management
- Automatically integrates ‘Debian non free firmware’ to the netboot image
- Carefully hand crafted ready to use templated preseed files
- Custom enable and disable hooks for easy integration with external tools
- Support for stand alone puppet manifests which will be applied during the first boot
- Configuration management database hooks
- Support for templated file overlays
- Makes it easy to do installations over serial consoles
- Template based management of seed files
- Separation between seed files and disk recipes
- No NFS mounts needed, it only needs a Debian repository
- No database backend, everything is stored in ‘pxelinux.cfg’ files
Puppetconf 2011 by Jasper Poppe

14. seedbank
- seedbank_setup
Download and manage Debian/Ubuntu netboot image(s) which are required to do a PXE
boot from the network
- seedbank
Seedbank prepares the ‘pxelinux.cfg’ files which will start the Debian installation process,
when the system boots via from the network
- seedbank_daemon
A simple multithreaded HTTP daemon which generates seed files on the fly based on
information which is stored in the ‘pxelinux.cfg’ files
After a successful installation it disables the related ‘pxelinux.cfg’ file
Puppetconf 2011 by Jasper Poppe

15. seedbank
Minion
Network
Debian Installer
Puppetconf 2011 by Jasper Poppe

16. seedbank
Minion
Network
Debian Installer
seedBank Daemon
Pxelinux.cfg
Seed File
Seed Template
Disk Recipe
Puppet Manifests
Overlay
Puppetconf 2011 by Jasper Poppe

17. seedbank
Configure DNS
Configure DHCP
Configure TFTP
wget http://infrastructureanywhere.com/seedbank.deb
sudo dpkg -i seedbank.deb
sudo vi /etc/seedbank/settings.py
sudo /etc/init.d/seedbank restart
sudo seedbank_setup debian-squeeze-amd64
sudo seedbank -r default minion001.a.c.m.e debian-squeeze-amd64
reboot minion001.a.c.m.e
Puppetconf 2011 by Jasper Poppe

18. seedbank
seedbank
-M 00:11:22:33:44:55
-o minion
-r default
-m network
minion001.a.c.m.e
ubuntu-natty-amd64
Puppetconf 2011 by Jasper Poppe

19. Questions Until Now
Remember... Stupid questions do not exist
Puppetconf 2011 by Jasper Poppe

20. Showtime
ebay Classifieds Group
Datacenter suite Amsterdam
Puppetconf 2011 by Jasper Poppe

21. Structuring Puppet
Just Because you want to avoid the evil faces of puppet!
Puppetconf 2011 by Jasper Poppe

22. Structuring Puppet
After we discovered you could use multiple module paths in Puppet I came up with a top
down structure which is using multiple module paths
In other words
- It is not allowed to include, inherit, etc. from other modules which reside in the same
path, so every module will be isolated
- It is only allowed to include, inherit, override etc. from a higher level than the path you
are working from
Puppetconf 2011 by Jasper Poppe

23. Structuring Puppet
Problems I have taken in account with the design
- Be able to share code between different platforms in a responsible way
- Keep code as isolated as possible
- Keep dependencies as clear as possible
- Keep it as simple as simple as possible
- Give Puppet newcomers the possibility to start developing without any direct danger
Puppetconf 2011 by Jasper Poppe

24. Structuring Puppet
Shared
Puppetconf 2011 by Jasper Poppe

25. Structuring Puppet
Shared
Users
Puppetconf 2011 by Jasper Poppe

26. Structuring Puppet
Shared
Users
Platform
Puppetconf 2011 by Jasper Poppe

27. Structuring Puppet
Shared
Users
Platform
Configs
Puppetconf 2011 by Jasper Poppe

28. The SOurce Code
- seedBank, Infrastruture Anywhere and a selection of our Puppet modules have just been
made public
- Everything has been released under the Apache 2 license
- Currently I am the only maintainer and author of seedBank
- Still lots of room for code/documentation improvements so you are welcome to submit
patches via GitHub
- Two Git repositories at GitHub
One with the seedBank code (Infrastructure Anywhere is included in the addons directory)
The other will contain all Puppet modules used in this talk
Puppetconf 2011 by Jasper Poppe

29. The source Code
http://infrastructureanywhere.com/
Puppetconf 2011 by Jasper Poppe

30. Questions
Please ask your questions
Puppetconf 2011 by Jasper Poppe

31. Thanks to ...
Roalt Zijlstra - For being a mentor with his endless patience years ago while I still was a
linux noob
GJ Moed - For supporting me through the whole Pupppet process, and his bright ideas/
discussions
Danny van der Meulen - For being Danny and his faith in me becoming a senior Linux
system engineer
Armand Verstappen - For having the faith and giving the recognition to whatever I am
doing
Lex van Roon - For giving me the energy with his enthusiastic attitude, and co developing
Infrastructure Anywhere
Puppetconf 2011 by Jasper Poppe