Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure, but we have not nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably? We all love infrastructure as code, we automate everything ™. However, making sure all of our infrastructure assets are monitored effectively can be slow and resource intensive multi stage process. During this talk we will investigate how we can setup nomad cluster that can automatically scale our infrastructure both horizontally as vertically to be able to cope with increased demand by users/ This talk will focus on making sure we on configuring Nomad and its new autoscaler component to be able to make data driven decisions about scaling nomad jobs in or out to fit current customers usage.

1. Autoscaling your
Nomad jobs
stackconf online 2021

2. ➔Used to be a Molecular Biologist,
Used to be a Molecular Biologist,
➔Then became a Dev,
Then became a Dev,
➔Now an Ops.
Now an Ops.
➔Currently
Currently CTO @ Hot Potatoes
CTO @ Hot Potatoes

3. Moving it all to the cloud

4. Vertical Scaling

5. Horizontal Scaling / Load Balancers

6. And than stuff got complicated….

7. Nomad

8. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
network {
port "http" {
to = 80
}
}
task "nginx" {
driver = "docker"
config {
image = "${PRIVATE}.dkr.ecr.us-east-1.amazonaws.com/blog:19"
ports = ["http"]
Deploy the blog

9. job "blog" {
group "hugo" {
count = 2
service {
name = "blog"
tags = ["traefik.enable=true"]
port = "http"
check {
type = "tcp"
interval = "10s"
timeout = "2s"
}
}
1 == None

10. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}q
Force onto different hardware

11. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${meta.rack}"
target "his" {
percent = 50
}
target "her" {
percent = 50
}
}
Suggest onto different hardware

12. /etc/nomad.d/config.hcl
Client {
Enabled = true
Meta {
"rack" = "his"
}
}
Based on custom meta-data

13. ●
Introduced in/with Nomad 0.11
●
(Currently) independent release cycle
●
Gaining new functionality every release
●
Build in Functionality for horizontal and vertical scaling
●
Extendable by your own (community) plugins
Nomad-Autoscaler

14. ●
Makes decisions based on a checks
●
Checks are a combination of
– Data queried from an APM
– Defined STRATEGY
– Attempt to approach TARGET value
●
Multiple Checks can be combined
●
Answer with the most resources will win!
●
ScaleOut and ScaleIn => ScaleOut
●
ScaleOut and ScaleNone => ScaleOut
●
ScaleOut(10) and ScaleOut(9) => ScaleOut(10)
Nomad-Autoscaler TLDR

15. job "autoscaler" {
type = "service"
group "autoscaler" {
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.3"
command = "nomad-autoscaler"
args = [
"agent",
"-config",
"${NOMAD_TASK_DIR}/config.hcl",
"-http-bind-address",
"0.0.0.0",
]
Deploy the autoscaler

16. /etc/nomad.d/config.hcl
nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
Config for the autoscaler

17. Metrics
https://
prometheus.io/

18. group "hugo" {
count = 3
scaling {
enabled = true
min = 1
max = 20
policy {
cooldown = "20s"
check "avg_instance_sessions" {
source = "prometheus"
query = "scalar(avg(traefik_service_open_connections{service="blog@consulcatalog"}))"
strategy "target-value" {
target = 5
}
Enable autoscaling for the blog

19. Dashboards
https://grafana.com/oss/grafana/

20. Enable autoscaling

21. Observe scaling down event

22. Observe the autoscaler
agent: querying APM: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value
target=local-nomad
agent: calculating new count: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value
target=local-nomad
agent: next count outside limits: policy_id=248f6157-ca37-f868-a0ab-
cabbc67fec1d source=prometheus strategy=target-value target=local-nomad
from=3 to=0 min=1 max=10
agent: updated count to be within limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus
strategy=target-value target=local-nomad from=3 to=1 min=1 max=10
agent: scaling target: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d
source=prometheus strategy=target-value target=local-nomad
target_config="map[group:demo job_id:webapp]" from=3 to=1 reason="capping
count to min value of 1"

23. Apply load
hey -z 1m -c 30 http://127.0.0.1:8000

24. Remove load

25. Logs
https://grafana.com/oss/loki/

26. group "autoscaler" {
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.3"
command = "nomad-autoscaler"
logging {
type = "loki"
config {
loki-url = 'http://loki.service.consul:3100/api/prom/push'
tag = "loki"
Directly to loki
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

27. task "promtail" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = true
}
config {
image = "grafana/promtail:2.2.1"
args = [
"-config.file",
"${NOMAD_TASK_DIR}/promtail.yaml",
]
Promtail sidecar

28. ${NOMAD_TASK_DIR}/promtail.yaml
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
task: autoscaler
__path__: /alloc/logs/autoscaler*
pipeline_stages:
- match:
selector: '{task="autoscaler"}'
stages:
- json:
expressions:
policy_id: '"@policy_id"'
source: '"@source"'
strategy: '"@strategy"'
target: '"@target"'
group: '"@group"'
job: '"@job"'
namespace: '"@namespace"'
Promtail sidecar
https://grafana.com/docs/loki/latest/clients/promtail/

29. Annotate your graphs

30. Correlate events with metrics

31. https://learn.hashicorp.com/tutorials/
nomad/autoscaler-vagrant-demo?
in=nomad/ecosystem
Try it yourself

32. Moving it all to the cloud *

33. apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
target "aws-asg" {
driver = "aws-asg"
config = {
aws_region = "{{ $x := env "attr.platform.aws.placement.availability-zone" }}{{ $length := len $x |subtract 1 }}
{{ slice $x 0 $length}}"
}
}
Grow into your platform

34. scaling "cluster_policy" {
policy {
cooldown = "2m"
evaluation_interval = "1m"
check "cpu_allocated_percentage" {
source = "prometheus"
query = "scalar(sum(nomad_client_allocated_cpu{node_class="hashistack"}*100/
(nomad_client_unallocated_cpu{node_class="hashistack"}+nomad_client_allocated_cpu{node_class="hashistack"}))/
count(nomad_client_allocated_cpu{node_class="hashistack"}))"
strategy "target-value" {
target = 70
}
}
target "aws-asg" {
dry-run = "false"
aws_asg_name = "${client_asg_name}"
node_class = "hashistack"
node_drain_deadline = "5m”
Grow into your platform

35. Observe the autoscaler again
agent.worker.check_handler: querying source: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-
bbe71b5544f7 source=prometheus strategy=target-value target=aws-asg
query=scalar(sum(nomad_client_allocated_memory{node_class="hashistack"}*100/
(nomad_client_unallocated_memory{node_class="hashistack"}+nomad_client_allocated_memory{node_class="hashistack"}))/
count(nomad_client_allocated_memory))
agent.worker.check_handler: calculating new count: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-
bbe71b5544f7 source=prometheus strategy=target-value target=aws-asg count=1 metric=95.17948717948718
agent.worker.check_handler: scaling target:
check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-
bbe71b5544f7 source=prometheus strategy=target-value target=aws-asg
from=1 to=2 reason="scaling up because factor is 1.359707" meta=map[]
internal_plugin.aws-asg: successfully performed and verified scaling out:
action=scale_out asg_name=hashistack-nomad_client desired_count=2
agent.worker.check_handler: successfully submitted scaling action to target: check=mem_allocated_percentage
policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus strategy=target-value target=aws-asg desired_count=2

36. https://github.com/hashicorp/nomad-
autoscaler/tree/master/demo/remote
Try it yourself

37. Moving it all to the cloud – QED

38. bram@attachmentgenie.com
@attachmentgenie
slideshare.net/attachmentgenie
Thank You