1. Puppet at scale
Rhommel Lamas
November 2013
Thursday, November 28, 13

2. Who am I?
Thursday, November 28, 13

3. Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Thursday, November 28, 13

4. Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Config Management
• Worked with CFEngine2 for 2 years.
• Puppet user since 2009.
Thursday, November 28, 13

5. Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Config Management
• Worked with CFEngine2 for 2 years.
• Puppet user since 2009.
Obsessed about config management and I use Emacs.
Thursday, November 28, 13

6. What is 3scale?
3scale provides tools such as traffic
management, API administration, traffic
reports, and more, to help companies
publicize their APIs easily and power new
types of connected applications.
Thursday, November 28, 13

7. What does scaling stands for?
Thursday, November 28, 13

8. What does scaling stands for?
Scalability is the ability of a system, network,
or process to handle a growing amount of
work in a capable manner or its ability to be
enlarged to accommodate that growth.
Wikipedia
Thursday, November 28, 13

9. What are our scaling needs?
Thursday, November 28, 13

10. What are our scaling needs?
• How many people do we need to scale our
system infrastructure?
Thursday, November 28, 13

11. What are our scaling needs?
• How many people do we need to scale our
system infrastructure?
• How many environments and systems can
our team maintain?
Thursday, November 28, 13

12. What are our scaling needs?
• How many people do we need to scale our
system infrastructure?
• How many environments and systems can
our team maintain?
• Can we delegate configurations easily to our
team members?
Thursday, November 28, 13

13. Puppet dynamic environments
Thursday, November 28, 13

14. Puppet dynamic environments
• Purpose of Puppet environments.
Thursday, November 28, 13

15. Puppet dynamic environments
• Purpose of Puppet environments.
• Puppet base environments at 3scale.
Thursday, November 28, 13

16. Puppet environments at 3scale
environments
/etc/puppet
auth.conf
production
Capfile
hieradata
autosign.conf
common.yaml
production.yaml
config.ru
preview.yaml
hiera.yaml
manifests
production.pp
hipchat.yaml
preview.pp
fileserver.conf
nodes
production
unicorn.rb
webserver.pp
environments
preview
webserver.pp
modules
nginx
preview
staging
Thursday, November 28, 13

17. [main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest
= $confdir/environments/$environment/manifests/site.pp
modulepath = $confdir/environments/$environment/modules
Thursday, November 28, 13

18. [main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest
= $confdir/environments/$environment/manifests/$environment.pp
modulepath = $confdir/environments/$environment/modules
# production.pp
import 'nodes/production/nodes.pp'
import 'nodes/production/*.pp
Thursday, November 28, 13

19. :hierarchy:
- '%{::osfamily}'
- '%{::environment}'
- common
:backends:
- yaml
- json
:yaml:
:datadir: "/etc/puppet/environments/%{environment}/hieradata"
Thursday, November 28, 13

20. Thursday, November 28, 13

21. Puppet dynamic environments
• Purpose of Puppet environments.
• Puppet environments at 3scale.
• Puppet development workflow (Git).
• Puppet-lint
• Puppet-rspec
• Code Review
Thursday, November 28, 13

22. Thursday, November 28, 13

23. Thursday, November 28, 13

24. Thursday, November 28, 13

25. Puppet at 3scale.
Thursday, November 28, 13

26. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
Thursday, November 28, 13

27. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
Thursday, November 28, 13

28. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
Thursday, November 28, 13

29. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit $runinterval/2.
Thursday, November 28, 13

30. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit $runinterval/2.
• Using DNS SRV records.
Thursday, November 28, 13

31. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit $runinterval/2.
• Using DNS SRV records.
• The foreman as our ENC.
Thursday, November 28, 13

32. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit $runinterval/2.
• Using DNS SRV records.
• The foreman as our ENC.
• Puppetdb and hiera.
Thursday, November 28, 13

33. Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit $runinterval/2.
• Using DNS SRV records.
• The foreman as our ENC.
• Puppetdb and hiera.
•MCollective with Rabbitmq connector.
Thursday, November 28, 13

34. ROUTE
53
_x-puppet._tcp.3scale.net
_x-puppet-ca._tcp.3scale.net
Puppet
Master
+
RabbitMQ
Cluster
+
MCollective
Client
+
PuppetDB
+
The
Foreman
Puppet
Master
+
RabbitMQ
Cluster
+
MCollective
Client
+
PuppetDB
+
The
Foreman
Puppet
Master
+
RabbitMQ
Cluster
+
MCollective
Client
+
PuppetDB
+
The
Foreman
RabbitMQ Load Balancer
Puppet
Agent
+
Mcollective
Server
Thursday, November 28, 13
Puppet
Agent
+
Mcollective
Server
Puppet
Agent
+
Mcollective
Server
PostgreSQL
RDS
MultiAZ

35. Thursday, November 28, 13

36. What nobody will tell you about
puppet.
Thursday, November 28, 13

37. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
Thursday, November 28, 13

38. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find errors on early stages so
TEST your modules.
Thursday, November 28, 13

39. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find errors on early stages so
TEST your modules.
• Don’t code remotely on your servers, use your local
development environment and git.
Thursday, November 28, 13

40. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find errors on early stages so
TEST your modules.
• Don’t code remotely on your servers, use your local
development environment and git.
• Read puppet documentation on every version.
Thursday, November 28, 13

41. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find errors on early stages so
TEST your modules.
• Don’t code remotely on your servers, use your local
development environment and git.
• Read puppet documentation on every version.
• Be careful about dependencies.
Thursday, November 28, 13

42. What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find errors on early stages so
TEST your modules.
• Don’t code remotely on your servers, use your local
development environment and git.
• Read puppet documentation on every version.
• Be careful about dependencies.
• Once you start to puppetize you become an Addict.
Thursday, November 28, 13

43. Questions?
at 3scale
we’re hiring
http://3scale.net/jobs
Rhommel Lamas
Twitter: @rhoml
Thursday, November 28, 13

44. Thank you.
at 3scale
we’re hiring
http://3scale.net/jobs
Rhommel Lamas
Twitter: @rhoml
Thursday, November 28, 13

45. Thursday, November 28, 13

46. Useful links
Thursday, November 28, 13

47. Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
Thursday, November 28, 13

48. Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html
Thursday, November 28, 13

49. Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html
• http://puppetlabs.com/blog/git-workflow-and-puppet-environments
Thursday, November 28, 13

50. Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html
• http://puppetlabs.com/blog/git-workflow-and-puppet-environments
• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn
Thursday, November 28, 13

51. Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html
• http://puppetlabs.com/blog/git-workflow-and-puppet-environments
• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn
• https://github.com/cernops/puppetdb_foreman
Thursday, November 28, 13