"Build a Modern Infrastructure in 45 min!" presented by Matthew Barr, HERE at Puppet Camp NYC 2014

1. Prepared by
Build a modern infrastructure in 45 min!
Matthew Barr
Sr. Systems Engineer

2. Is your
infrastructure a
mess?
Let’s fix it :)

3. What we’re going to do:
• Define a modern infrastructure
• Glance at their architectures
• Demonstrate how to do this yourselves
• … And then the details..

4. What is a modern infrastructure?

5. It includes:
• Centralized logging
• Monitoring
• Orchestration
• CI (continuous integration)
• Metrics*

6. What we’ll do today: Setup
• Mcollective
• Sensu (ideal for cloud infra)
• Logstash + ElasticSearch + Kibana
• Jenkins

7. MCollective (mco)
• Orchestration
• Uses ActiveMQ or RabbitMQ
• Maintained by Puppet Labs
• http://puppetlabs.com/mcollective

9. • Distributed monitoring system
• Uses RabbitMQ
• has a easy API
• Adding/remove servers without restarting or changing config files
on server
• http://sensuapp.org
Sensu!

11. Logstash
http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash

12. Elastic Search & Kibana
• Elasticsearch (http://www.elasticsearch.com) is a “distributed
restful search and analytics tool”
• It’s used as a datastore for Logstash. (it’s not the only one, but
one of the most used.)
• Kibana is a dashboard for use with Elasticsearch & Logstash.

14. What we’re actually doing:
• Show how to use a set of forge modules to build an infrastructure
out.
• using the mbarr/moderninfra as an opinionated profile module
• download the necessary modules using librarian-puppet

15. We’ll:
• Build a RabbitMQ server + sensu server
• the admin host (has the mco client)
• Build a logstash server
• Build a Jenkins host

16. Each server will also:
• be sending logs via logstash-forwarder
• run Sensu client checks
• run a mco server

17. Moderninfra module

18. A forge module just for you!
• Sets up the basics of each service
• Sets up the requirements correctly to all work together
• Has… opinions.

19. Install from the forge:
puppet module install mbarr-moderninfra

20. The code!
---!
moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'!
moderninfra::mco_password: 'shhhh..its.a.secret.'!
moderninfra::sensu_password: 'whatsupdoc'!
moderninfra::logstash_server: 'logstash.aws.mbarr.net'
Hiera data, to make life easier:
class moderninfra (!
$rmqserver,!
$logstash_server,!
$rmq=false,!
$mco_client=false,!
$mco_server=false,!
$sensu_client=false,!
$sensu_server=false,!
$logstash=false,!
$logstash_forwarder=true,!
$mco_password=undef,!
$sensu_password=undef,!
) {...}

21. node default {!
if $role == "mco" {!
class {'moderninfra':!
rmq => true,!
mco_client => true,!
sensu_server => true,!
}!
include profiles::sensuchecks !
}!
!
if $role == "puppet" {!
class {'moderninfra':!
mco_server => true,!
sensu_client => true,!
}!
}
if $role == "logstash" {!
class {'moderninfra':!
logstash => true,!
mco_server => true,!
sensu_client => true,!
}!
include profiles::logstash!
}!
!
if $role == "jenkins" {!
class {'moderninfra':!
mco_server => true,!
sensu_client => true,!
}!
include jenkins!
}!
}
Site.pp

22. RabbitMQ, Sensu & Mcollective

23. RabbitMQ
• This is the middle ware that is used by both mco & sensu.
• Our module uses the Puppet SSL certs for connections
• Adds a second cert for the host, via the puppet-certificate
module.

24. Code
class {'moderninfra':!
rmq => true,!
mco_client => true,!
sensu_server => true,!
}!
include profiles::sensuchecks !
}

26. RMQ Note
• To be fair: Sensu isn’t running w/ SSL certs
• I’ve used other self signed certs before without issue
• Looks like there’s a bug that hopefully is actually fixed in Erlang
OTP 17.1

27. Mcollective
• Using SSL to secure PSK connections between mco & RabbitMQ
• Installs the package, service & puppet agents.

28. root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet
!
* [ ============================================================> ] 4 / 4
!
puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged.
rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
!
Summary of Arch:
!
No aggregate summary could be computed
!
Summary of Ensure:
!
3.6.2-1puppetlabs1 = 3
purged = 1
!
!
Finished processing 4 / 4 hosts in 1172.09 ms

29. Sensu
• Client on all 4 hosts
• Server on RMQ box
• Distributed checks
• Dashboard on 8080
• profiles::sensuchecks installs various checks. (not in module)

33. Actually making sensu GO: (on server)
class profiles::sensuchecks {!
sensu::check { 'check_ntp':!
command => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H
pool.ntp.org -w 20 -c 40',!
handlers => 'default',!
subscribers => 'general',!
standalone => false,!
custom => { occurrences => 2 },!
}!
sensu::check { 'check_cron':!
command => '/etc/sensu/plugins/check-procs.rb -p cron -C 1 -c 10 -w 10 ',!
handlers => 'default',!
subscribers => 'general',!
interval => 60,!
standalone => false,!
custom => { occurrences => 2 },!
}!
}!

34. Logstash

35. • Centralized logging system
• Inputs, Outputs, Filters
• Inputs: syslog, files, redis..
• Outputs:elasticsearch, etc
• Filters: Grok, many others

37. Logstash profile
class profiles::logstash {!
!
logstash::configfile { 'basic_config':!
source => 'puppet:///modules/profiles/logstash/basic_config',!
order => 10!
}!
!
include kibana3!
!
}!

38. Logstash config
input {
lumberjack {
port => 12345
ssl_certificate => "/etc/logstash/ssl/cert.pem"
ssl_key => "/etc/logstash/ssl/key.pem"
type => "lumberjack"
}
}
!
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
!
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}

39. Logstash-forwarder
• Data is sent from logs on client to Logstash server via SSL
• Keeps track of log positions and what’s been sent
• Server listens on 12345, for now.

40. Elasticsearch & Kibana
• This is what Kibana looks like with data from logstash fed into
elasticsearch
• (It’s zoomed a bit, so you can see the good parts.)

43. Jenkins

44. Jenkins
• Continuous integration tool
• There is code to set up slaves in the Jenkins module.
• https://forge.puppetlabs.com/rtyler/jenkins

45. include jenkins

46. Things this module doesn’t do:
• Build your puppet master
• DNS names for Puppet master, RMQ, Logstash, etc
• Although the cloud formation templates do!

47. But it might let
you sleep at
night…

48. Appendix:!
Puppet Master

49. • Built w/ CloudFormations template
• Sorry, not vagrant. Might be added soon.
• uses cloud-init to provision puppet & code base
• Uses puppet 3.6.2
• Librarian-puppet

50. Puppet Master
• Set host name & domain
• Install puppet
• rm -rf /etc/puppet
• git clone REPO /etc/puppet

51. Appendix: !
Librarian-puppet

52. Librarian Puppet
• Lets you take a Puppetfile, and manage modules & dependencies
• can use forge or git repos
• Takes over your modules directory, though.
• adds to .gitignore & regenerates the directory from the
Puppetfile
• I’ve used a pattern of a second directory (modules-local) to allow
a slow migration & local files to stay in your existing repo

53. Modules-local pattern
Old:
modulepath = $confdir/modules:$confdir/modules-local
!
3.6+ directory environments: environment.conf
modulepath = modules:modules-local

54. Puppetfile
forge "https://forgeapi.puppetlabs.com"
!
mod "reidmv/puppet_certificate"
mod "elasticsearch/logstash"
mod "elasticsearch/elasticsearch"
mod "sensu/sensu"
!
mod "rtyler/jenkins"
!
mod "puppetlabs/mcollective"
!
mod "thejandroman/kibana3", "0.0.3"
!
# mod "mbarr/moderninfra",
# :git => "git://github.com/matthewbarr/moderninfra.git"
!
#mod "garethr/graphite"

55. modules
├── activemq
├── apache
├── apt
├── concat
├── datacat
├── elasticsearch
├── epel
├── erlang
├── file_concat
├── git
├── java
├── java_ks
├── jenkins
├── kibana3
├── logstash
├── mcollective
├── puppet_certificate
├── rabbitmq
├── sensu
├── staging
├── stdlib
├── vcsrepo
└── zypprepo
modules-local
├── moderninfra
└── profiles

56. We’re hiring! (in Boston)!
!
!
Matthew Barr!
@matthewbarr (github & twitter)!
matthew.barr@here.com!
mbarr@mbarr.net!
http://github.com/matthewbarr/build-modern-infra