HashiConf '19 Hallway track presenting Advanced Consul Templates with consul-templaterb

1. Pierre Souchay
Discovery Team @Criteo
Twitter: @vizionr
Github: pierresouchay
Consul
templates
on steroids
Leading Discovery Team @Criteo (SDKs + Consul)
Dealing with 240k+ services, 38k+ Consul nodes in 9 DCs
1st external contributor to Consul
Author of consul-templaterb

2. Why consul-templaterb?
Easier for complex templates
• ERB is nicer, more powerful than Golang templating
• Structured output formats (JSON, XML, YAML…)
• Hot-Reload, includes, lazy evaluation
• You can create a full featured GUI!
• Handle Consul compatibility for features (ex: meta)
• OSS with examples: https://github.com/criteo/consul-templaterb/

3. Why consul-templaterb?
Very High performance on large clusters
• More optimized for performance on large clusters: 800M/s -> 12k/s
• Bandwidth usage / endpoint (debug)
• Optimizations built-in per-endpoints

4. Why consul-templaterb?
Good compatibility with Consul-Template
• Most of features are present (process baby-sitting), reload commands…
• Many of the flags are compatible (using the same syntax, eg: --template)

5. How it works internally…
while (true) {
data = compute_template()
ready = download_update_data_async()
if (ready) {
write_output_to_file(data)
}
}

6. Well documented API
• API is fully documented
• https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md
• Includes links to real world examples
• API includes regression tests
• All samples are always tested at build time and all working!
• Output of APIs is unit-tested
• Many samples (HTML, YAML, JSON, text, full HTML5 GUI)
• https://github.com/criteo/consul-templaterb/tree/master/samples

7. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
service(srv).each do |snode|
result[srv][snode.status] += 1
end
end
%><%= JSON.pretty_generate(result) %>

8. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
service(srv).each do |snode|
result[srv][snode.status] += 1
end
end
%><%= JSON.pretty_generate(result) %>
{
"chef-server": {
"passing": 11,
"warning": 0,
"critical": 0
},
"chef-webscale": {
"passing": 3,
"warning": 0,
"critical": 0
},
"chef-webscale-backend": {
"passing": 1,
"warning": 0,
"critical": 0
},
"host-arbitrage": {
"passing": 388,
"warning": 0,
"critical": 2
[…]

9. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
service(srv).each do |snode|
result[srv][snode.status] += 1
end
end
%><%= YAML.dump(result) %>

10. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
service(srv).each do |snode|
result[srv][snode.status] += 1
end
end
%><%= YAML.dump(result) %>
---
chef-server:
passing: 11
warning: 0
critical: 0
chef-webscale:
passing: 3
warning: 0
critical: 0
chef-webscale-backend:
passing: 1
warning: 0
critical: 0
host-arbitrage:
passing: 388
warning: 0
critical: 2
host-arbitrage-admin:
passing: 388
warning: 0
[…]

11. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
datacenters.each do |dc|
service(srv, dc:dc).each do |snode|
result[srv][snode.status] += 1
end
end
end
%><%= YAML.dump(result) %>

12. A nice API with sensitive defaults
<%
result = {}
services(tag:"webscale").each do |srv, tags|
result[srv] = {'passing' => 0, 'warning' => 0, 'critical' => 0}
datacenters.each do |dc|
service(srv, dc:dc).each do |snode|
result[srv][snode.status] += 1
end
end
end
%><%= YAML.dump(result) %>
---
chef-server:
passing: 25
warning: 0
critical: 0
chef-webscale:
passing: 3
warning: 0
critical: 0
chef-webscale-backend:
passing: 1
warning: 0
critical: 0
host-arbitrage:
passing: 4493
warning: 0
critical: 23
host-arbitrage-admin:
passing: 4493
warning: 0
[…]

13. Demo Time!
• Using ruby gems
gem install consul-templaterb
consul-templaterb --help
• Using docker for the GUI:
https://hub.docker.com/r/discoverycriteo/consul-templaterb

14. History of Services in a large Consul Cluster

15. Highly scalable UI (1000s of users)

16. See how we are using it to build new Stuff
Tomorrow at Sep 11 | 3:20 PM - 3:55 PM at Elwha A | Level 5
Inversion Of Control with Consul
https://hashiconf.hashicorp.com/schedule/inversion-of-control-with-consul
Slides: https://www.slideshare.net/PierreSouchay/2019-hashiconf-seattleconsulioc
Other resources:
https://www.hashicorp.com/resources/operating-consul-at-scale
https://www.hashicorp.com/resources/criteo-containers-consul-connect