Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven't nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably. We all love infrastructure as code, we automate everything ™. However making sure all of our infrastructure assets are monitored effectively can be slow and resource intensive multi stage process. During this talk we will investigate how we can setup and observe a service mesh platform using HashiCorp's Consul Connect by recording its metrics. logs and traces. This talk will focus on configuring and analysing the metrics, logs and traces Consul Connect produces using Prometheus, Loki, Tempo and Grafana.

1. Observability with
Consul Connect
a
HashiTalks 2021

2. ➔Used to be a Molecular Biologist,
Used to be a Molecular Biologist,
➔Then became a Dev,
Then became a Dev,
➔Now an Ops.
Now an Ops.
➔Currently
Currently CTO @ Hot Potatoes
CTO @ Hot Potatoes
➔Today Proud Dad
Today Proud Dad

3. Short History of
scaling applications
Highly biased

4. Vertical Scaling

5. Horizontal Scaling

6. Load Balancers

7. Service Discovery

8. service {
name = "counting"
port = 9003
check {
id = "counting-check"
http = "http://localhost:9003/health"
interval = "1s"
timeout = "1s"
}
}
Consul Service Definition

9. dig @127.0.0.1 -p 8600 counting.service.consul. ANY
zone "consul" IN {
type forward;
forward only;
forwarders { 127.0.0.1 port 8600; };
};
Consul Service Usage

10. Service Mesh

11. Service {
name = "counting"
port = 9003
connect {
sidecar_service {
proxy {
}
}
}
check {
id = "counting-check"
http = "http://localhost:9003/health"
interval = "1s"
timeout = "1s"
}
}
Consul Connect Declaration

12. service {
name = "dashboard"
port = 9002
connect {
sidecar_service {
proxy {
upstreams = [
{
destination_name = "counting"
local_bind_port = 5000
}
]
}
}
}
}
Connect to a service

13. consul connect proxy -sidecar-for counting > counting-proxy.log &
consul connect proxy -sidecar-for dashboard > dashboard-proxy.log &
Fire up those connections

14. Power of the service mesh
https://learn.hashicorp.com/tutorials/consul/service-mesh-with-envoy-proxy
consul intention create dashboard counting

15. Observability

16. Observability

17. Consul Service Mesh is using Envoy

18. Metrics
https://prometheus.io/

19. service {
name = "counting"
port = 9003
connect {
sidecar_service {
proxy {
config {
envoy_prometheus_bind_addr = "192.168.1.20:8888"
}
}
}
}
Expose Envoy metrics

20. enable_central_service_config = true
config_entries {
bootstrap = [
{
kind = "proxy-defaults"
name = "global"
config {
envoy_prometheus_bind_addr = "192.168.1.20:8888"
}
}
]
}
telemetry {
disable_hostname = true,
prometheus_retention_time = "72h"
}
Expose Envoy metrics by default

21. global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'consul-connect-envoy'
static_configs:
- targets: ['192.168.1.20:8888']
- job_name: consul-connect-envoy
scrape_interval: 15s
honor_labels: true
consul_sd_configs:
- server: localhost:8500
services: [consul-connect-envoy]
Collect metrics

22. ui_config {
enabled = true
metrics_provider = "prometheus"
metrics_proxy {
base_url = "http://192.168.1.20:9090"
}
}
Expose in Consul UI

23. Exploratory usage

24. Dashboards
https://grafana.com/oss/grafana/

25. datasources:
- name: Prometheus
type: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
basicAuth: false
isDefault: false
version: 1
editable: false
Data Sources

26. Dashboards

27. Logs
https://grafana.com/oss/loki/

28. prometheus:
image: prom/prometheus:latest
volumes:
- ./etc/prometheus.yaml:/etc/prometheus.yaml
entrypoint:
- /bin/prometheus
- --config.file=/etc/prometheus.yaml
ports:
- "9090:9090"
logging:
driver: loki
options:
loki-url: 'http://localhost:3100/api/prom/push'
Docker / K8S
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

29. consul connect envoy -sidecar-for counting-1 -- --log-path counting-proxy.log
scrape_configs:
- job_name: service
pipeline_stages:
- docker:
static_configs:
- targets:
- localhost
labels:
job: service
host: $FQDN
__path__: /var/log/*-proxy.log
VM / Baremetal using promtail
https://grafana.com/docs/loki/latest/clients/promtail/

30. - name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: false
version: 1
editable: false
apiVersion: 1
Data Sources

31. Exploratory usage

32. Traces
https://grafana.com/oss/tempo/

33. envoy_tracing_json = <<EOL
{
"http": {
"name": "envoy.zipkin",
"config": {
"collector_cluster": "zipkin",
"collector_endpoint": "/api/v1/spans"
}
}
}
EOL
Data Sources

34. Config {
envoy_extra_static_clusters_json = <<EOL
{
...
"load_assignment": {
"cluster_name": "zipkin",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "192.168.1.20",
"port_value": 9411,
"protocol": "TCP"
}
Data Sources

35. - name: Tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo-query:16686
basicAuth: false
isDefault: false
version: 1
editable: false
apiVersion: 1
uid: tempo
Data Sources

36. - name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: false
version: 1
editable: false
apiVersion: 1
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: (?:traceID|trace_id)=(w+)
name: TraceID
url: $${__value.raw}
Data Sources

37. Exploratory usage

38. bram@attachmentgenie.com
@attachmentgenie
slideshare.net/attachmentgenie
Thank You