Everybody loves Prometheus. Many exporters are available to gather specific data. You can download the binaries from GitHub, start them and they will expose data via plain HTTP, without any firewalling or authentication. That would just complicate the whole setup! This is how the usual installation guide looks. It works, but it doesn’t cover all the important parts a systems engineer loves. – What about authentication? – What about firewalling? – How can we make the Prometheus server aware of the exporters? – How do we integrate custom applications that also provide prometheus compatible metrics? A secure and automated rollout of exporters isn’t easy. Also an authenticated connection from the prometheus server to the exporters requires some preparation. This talk will cover a proper concept and all details to rollout multiple exporters to many systems, completely automated with Puppet.

1. Rollout allyour Prometheus exporters withRollout allyour Prometheus exporters with
Puppet!Puppet!
(ProudlypoweredbyVoxPupuli Puppet modules)(ProudlypoweredbyVoxPupuli Puppet modules)
1 / 251 / 25

2. $ whoami
Tim 'bastelfreak' Meusel
Puppet Contributor since 2012
Merging stuff on Vox Pupuli (Puppet Community) since 2015
Vox Pupuli Project Management Committee member
2 / 25

3. Let us go on amonitoring journeyLet us go on amonitoring journey
3 / 253 / 25

4. Starting Point Assume you maintain a few thousand servers
@bastelsblog for @voxpupuliorg
4 / 25

5. Starting Point Assume you maintain a few thousand servers
Assume you have at least one coworker that develops software
@bastelsblog for @voxpupuliorg
4 / 25

6. Starting Point Assume you maintain a few thousand servers
Assume you have at least one coworker that develops software
You have to maintain that software in production
@bastelsblog for @voxpupuliorg
4 / 25

7. Starting Point Assume you maintain a few thousand servers
Assume you have at least one coworker that develops software
You have to maintain that software in production
All monitoring you have is really bad
It has false positives, it isn't efficient, it's complex
@bastelsblog for @voxpupuliorg
4 / 25

8. Starting Point Assume you maintain a few thousand servers
Assume you have at least one coworker that develops software
You have to maintain that software in production
All monitoring you have is really bad
It has false positives, it isn't efficient, it's complex
Or you have no monitoring at all
@bastelsblog for @voxpupuliorg
4 / 25

9. Chapter 1: ExplorationChapter 1: Exploration
Or:TheHappyPhaseOr:TheHappyPhase
5 / 255 / 25

10. Starting Point
Exploration
You attend your most favourite monitoring conference
@bastelsblog for @voxpupuliorg
6 / 25

11. Starting Point
Exploration
You attend your most favourite monitoring conference
Somebody recommends you node_exporter
"Just download the binary, run it as root and it works" they said...
@bastelsblog for @voxpupuliorg
6 / 25

12. Starting Point
Exploration
Prometheus Exporters:
Reads many metrics from your box, expose them via HTTP
No HTTPS, no authentication
@bastelsblog for @voxpupuliorg
7 / 25

13. Starting Point
Exploration
Prometheus Exporters:
Reads many metrics from your box, expose them via HTTP
No HTTPS, no authentication
Has many collectors, that can be enabled/disabled
@bastelsblog for @voxpupuliorg
7 / 25

14. Starting Point
Exploration
Prometheus Exporters:
Reads many metrics from your box, expose them via HTTP
No HTTPS, no authentication
Has many collectors, that can be enabled/disabled
Can read values from 3rd parties from a txt file
@bastelsblog for @voxpupuliorg
7 / 25

15. Starting Point
Exploration
Prometheus Exporters:
Reads many metrics from your box, expose them via HTTP
No HTTPS, no authentication
Has many collectors, that can be enabled/disabled
Can read values from 3rd parties from a txt file
Single binary, written in go
Nice for dependency management, but please consider building
packages
Do not download random binaries from the internet...
@bastelsblog for @voxpupuliorg
7 / 25

16. Starting Point
Exploration
Prometheus Exporters:
# HELP Total number of scrapes by HTTP status code.
# TYPE counter
promhttp_metric_handler_requests_total{code="200"} 5
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
Each metric has a name and a value
There should be a HELP
There should be a type
A metric can have labels (code="XXX" in this case)
@bastelsblog for @voxpupuliorg
8 / 25

17. Starting Point
Exploration
Tell your developers to expose data as prometheus metrics
@bastelsblog for @voxpupuliorg
9 / 25

18. Starting Point
Exploration
Tell your developers to expose data as prometheus metrics
Enrich your exporter with 3rd party data
@bastelsblog for @voxpupuliorg
9 / 25

19. Starting Point
Exploration
Tell your developers to expose data as prometheus metrics
Enrich your exporter with 3rd party data
Many applications already provide prometheus-style metrics
@bastelsblog for @voxpupuliorg
9 / 25

20. Starting Point
Exploration
Tell your developers to expose data as prometheus metrics
Enrich your exporter with 3rd party data
Many applications already provide prometheus-style metrics
There are hundreds of exporters available
@bastelsblog for @voxpupuliorg
9 / 25

21. Chapter 2: IntegrationChapter 2: Integration
Or:Millionways toshoot yourselfinthefootOr:Millionways toshoot yourselfinthefoot
10 / 2510 / 25

22. Starting Point
Exploration
Integration
With the puppet/prometheus module:
include prometheus::node_exporter
@bastelsblog for @voxpupuliorg
11 / 25

23. Starting Point
Exploration
Integration
Prometheus database scrapes all exporters
Everybody can make backups of your metrics o/
(Because they are available to the whole internet)
@bastelsblog for @voxpupuliorg
12 / 25

24. Starting Point
Exploration
Integration
Discovery
Prometheus database needs to know the targets
Register each exporter in Consul as a service
Feed services into Prometheus
Use KyleAnderson/consul Puppet module to deploy consul
@bastelsblog for @voxpupuliorg
13 / 25

25. Starting Point
Exploration
Integration
Discovery
@bastelsblog for @voxpupuliorg
14 / 25

26. Starting Point
Exploration
Integration
Discovery
Take a look at ./examples/node_exporter_consul.pp
@bastelsblog for @voxpupuliorg
15 / 25

27. Starting Point
Exploration
Integration
Discovery
Authentication
Puppet Agent/Server use TLS client certificates
Deploy nginx in front of node_exporter
Require a valid client certificate from prometheus database
Vox Pupuli provides puppet/nginx o/
@bastelsblog for @voxpupuliorg
16 / 25

28. Starting Point
Exploration
Integration
Discovery
Authentication
Take a look at ./examples/node_exporter_profile.pp
@bastelsblog for @voxpupuliorg
17 / 25

29. Starting Point
Exploration
Integration
Discovery
Authentication
Consul:
No authentication by default
Can also use TLS client certificates / a secret key
Puppet certificates can be reused
@bastelsblog for @voxpupuliorg
18 / 25

30. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
Only Prometheus database should be able to access exporters
@bastelsblog for @voxpupuliorg
19 / 25

31. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
Only Prometheus database should be able to access exporters
Firewalling should be automated
@bastelsblog for @voxpupuliorg
19 / 25

32. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
Only Prometheus database should be able to access exporters
Firewalling should be automated
IP addresses might change often
@bastelsblog for @voxpupuliorg
19 / 25

33. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
class profiles::prometheus {
@@ferm::rule{"prometheus2$trusted['certname']":
ensure => 'present',
chain => 'INPUT',
saddr => facts['networking']['ip6'],
dport => 9100,
proto => 'tcp',
policy => 'ACCEPT',
tag => 'prometheus2node_exporter',
}
}
class profiles::node_exporter {
Ferm::Rule <<| tag == 'prometheus2node_exporter' |>>
}
Exported resources to the rescue
Vox Pupuli provides puppet/ferm
@bastelsblog for @voxpupuliorg
20 / 25

34. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
Consul:
Consul creates a meshed network
iptables doesn't scale with thousands of entries
filtering pakets and changing rules will be delayed
@bastelsblog for @voxpupuliorg
21 / 25

35. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
ipsets and separate chains to the rescue!
handle consul traffic in a separate iptables chain
ip sets are a hashmap of CIDR ranges
In memory, scale very good
iptables can check against this hashmap
You cannot mix IPv4/IPv6 in one hashmap
Updating the ip set is independet from iptables rules
Vox Pupuli provides puppet/ipset
@bastelsblog for @voxpupuliorg
22 / 25

36. Starting Point
Exploration
Integration
Discovery
Authentication
Firewalling
Take a look at ./examples/ipsets.pp
@bastelsblog for @voxpupuliorg
23 / 25

37. Chapter 3: ConclusionChapter 3: Conclusion
Or:o/?Or:o/?
24 / 2524 / 25

38. Starting Point
Exploration
Integration
Conclusion
Rolling out an exporter with Puppet is easy
1 line of code! (for a dev environment)
Required config management code for a production setup is still
reasonable
@bastelsblog for @voxpupuliorg
25 / 25

39. Starting Point
Exploration
Integration
Conclusion
Rolling out an exporter with Puppet is easy
puppet/prometheus supports 20 famous exporters
You can easily add support for another one
Ping the Vox Pupuli people and ask for help
@bastelsblog for @voxpupuliorg
25 / 25

40. Starting Point
Exploration
Integration
Conclusion
Rolling out an exporter with Puppet is easy
puppet/prometheus supports 20 famous exporters
Consul as service discovery
+ Probably the solution that scales best
+ Since you now have it, you can use it for even more stuff
- More configuration effort than an actual exporter
@bastelsblog for @voxpupuliorg
25 / 25

41. Starting Point
Exploration
Integration
Conclusion
Rolling out an exporter with Puppet is easy
puppet/prometheus supports 20 famous exporters
Consul as service discovery
github.com/bastelfreak/osmc2019 repo with slides + all examples +
Vagrantfile to show your coworkers
Contact: tim@bastelfreak.de or bastelfreak on freenode
Collection of related talks
Thanks foryourattention!
@bastelsblog for @voxpupuliorg
25 / 25