Nomad is an open source tool for dynamic workload scheduling that can be used to easily transform applications to be cloud native. The document discusses how Nomad was used to deploy a personal blog to AWS, including writing a Nomad job file and using features like spreading tasks across hosts and auto-scaling. It also covers integrating Nomad with other HashiCorp tools like Consul for service discovery, Vault for access control, and the Nomad autoscaler for auto-scaling. Examples are provided of monitoring metrics with Prometheus and visualizing with Grafana.

1. Easy Cloud Native Transformation
with HashiCorp Nomad
Bram Vogelaar
@attachmentgenie

2. $ whoami
• Used to be a Molecular Biologist
• Then became a Dev
• Now an Ops
• Currently Cloud Engineer @ The Factory
• Amsterdam HUG organizer

3. Moving it all to the cloud

4. Vertical Scaling

5. Horizontal Scaling / Load Balancers

6. And than stuff got complicated….

7. The story starts with my personal website

8. Nomad
l Open Source tool for dynamic workload scheduling
l Batch, containerized, and non-containerized applications.
l Has native Consul and Vault integrations.
l Has token based access setup.
l Jobs written in (H)ashiCorp (C)onfiguration (L)anguage
https://www.nomadproject.io/

9. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
network {
port "http" {
to = 80
}
}
task "nginx" {
driver = "docker"
config {
image = "${PRIVATE}.dkr.ecr.us-east-1.amazonaws.com/blog:19"
ports = ["http"]
Deploy the blog

10. 1 == None
job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2

11. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}
Force onto different hardware

12. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${node.datacenter}"
}
Suggest onto different hardware

13. /etc/nomad.d/config.hcl
Client {
Enabled = true
Meta {
"rack" = "his"
}
}
Based on custom meta-data

14. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 2
Spread {
attribute = "${meta.rack}"
target "his" {
percent = 50
}
target "her" {
percent = 50
}
}
Based on custom meta-data

15. service {
name = ”blog"
provider = "nomad"
port = ”http"
}
Service Definition

16. template {
data = <<EOH
http {
server {
listen 80;
location / {
{{ range nomadService ”blog" }}
proxy_pass http://{{ .Address }}:{{ .Port }};
{{ end }}
}
}
}
EOH
destination = "local/api-servers"
Service Usage

17. Nomad Pack
• Levant
• Templating and packaging tool
• Easily deploy popular applications to Nomad
• Re-use common patterns across internal applications
• Find and share job specifications with the Nomad community
• Nightlies only right now!
https://github.com/hashicorp/nomad-pack-community-registry

18. Nomad Pack
• nomad-pack registry list
• nomad-pack run hello_world
• nomad-pack run hello_world --var message=hola
https://github.com/hashicorp/nomad-pack

19. https://github.com/attachmentgenie/vagrant-scheduler
Try it yourself

20. Consul
l Open-Source Service Discovery Tool
l Build-in KV store
l Service Mesh tool
https://www.consul.io/

21. service {
name = "blog"
port = "http"
check {
type = "tcp"
interval = "10s"
timeout = "2s"
}
}
Service Definition

22. Check {
type = "tcp"
interval = "10s"
timeout = "2s"
check_restart {
limit = 3
grace = "10s"
ignore_warnings = false
}
}
Stampeding herd

23. group "hugo" {
Restart {
interval = "10m"
attempts = 2
delay = "15s"
mode = "fail"
}
task "nginx" {
Restart failed jobs

24. group "hugo" {
Count = 2
reschedule {
delay = "30s"
delay_function = "constant" #constant, exponential, fibonacci
unlimited = true # or max_delay = “1h”
}
task "nginx" {
Reschedule a job

25. • https://github.com/ThomasObenaus/dummy-services
• https://github.com/Shopify/toxiproxy
Testing your assumptions

26. group "hugo" {
Count = 10
update {
max_parallel = 2
min_healthy_time = "30s"
healthy_deadline = "5m"
}
task "nginx" {
Updates

27. group "hugo" {
Count = 10
Update {
max_parallel = 1
canary = 10
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = false
}
task "nginx" {
Blue/Green Release

28. group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = true
}
task "nginx" {
Canary Release

29. service {
name = "blog"
tags = ["v2"]
}
$version++

30. group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = false
}
task "nginx" {
Canary Release++

31. kind = "service-router"
name = "blog"
routes = [
{
match {
http {
header = [
{
name = "group"
exact = "test"
}, ] } }
destination {
service = "blog"
service_subset = "v2"
} },]
Consul to the rescue

32. ● Introduced in/with Nomad 0.11
● (Currently) independently release cycle
● Gaining new functionality every release
● Build in Functionality for horizontal and vertical scaling
● But extendable by your own (community) plugins
Nomad autoscaler

33. ● Makes decisions based on a checks
● Checks are a combination of
• Data queried from an APM
• Defined STRATEGY
• Attempt to approach TARGET value
● Multiple Checks can be combined
• Answer with the most resources will win!
• ScaleOut and ScaleIn => ScaleOut
• ScaleOut and ScaleNone => ScaleOut
• ScaleIn and ScaleNone => ScaleNone
• ScaleOut(10) and ScaleOut(9) => ScaleOut(10)
• ScaleIn(3) and ScaleIn(4) => ScaleIn(4)
Auto-scaling TLDR

34. • job "autoscaler" {
type = "service"
datacenters = ["aws"]
group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.6"
command = "nomad-autoscaler"
args = [
"agent",
"-config",
"${NOMAD_TASK_DIR}/config.hcl",
"-http-bind-address",
"0.0.0.0",
]
Deploy the autoscaler

35. • /etc/nomad.d/config.hcl
• nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
Config for the autoscaler

36. Metrics
https://prometheus.io/

37. • job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 3
scaling {
enabled = true
min = 1
max = 20
policy {
cooldown = "20s"
check "avg_instance_sessions" {
source = "prometheus"
query = "scalar(avg(traefik_service_open_connections{service="blog@consulcatalog"}))"
strategy "target-value" {
target = 5
}
Enable autoscaling for the blog

38. Dashboards
https://grafana.com/oss/grafana/

39. Enable autoscaling

40. Observe scaling down event

41. agent: querying APM: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad
agent: calculating new count: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad
agent: next count outside limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=
agent: updated count to be within limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value target=local-nomad from=3 to=1 min=1 max=10
agent: scaling target: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus strategy=target-value
Observe the autoscaler

42. hey -z 1m -c 30 http://127.0.0.1:8000
Apply load

43. Remove load

44. Logs
https://grafana.com/oss/loki/

45. group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.6"
command = "nomad-autoscaler"
logging {
type = "loki"
config {
loki-url = 'http://loki.service.consul:3100/api/prom/push'
tag = "loki"
}
}
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
Direct to loki

46. task "promtail" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = true
}
config {
image = "grafana/promtail:2.5.0"
args = [
"-config.file",
"local/promtail.yaml",
]
Promtail sidecar

47. • scrape_configs:
- job_name: system
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
task: autoscaler
__path__: /alloc/logs/autoscaler*
pipeline_stages:
- match:
selector: '{task="autoscaler"}'
stages:
- regex:
expression: '.*policy_id=(?P<policy_id>[a-zA-Z0-9_-]+).*source=(?P<source>[a-zA-Z0-9_-
]+).*strategy=(?P<strategy>[a-zA-Z0-9_-]+).*target=(?P<target>[a-zA-Z0-9_-]+).*Group:(?P<group>[a-zA-Z0-
9]+).*Job:(?P<job>[a-zA-Z0-9_-]+).*Namespace:(?P<namespace>[a-zA-Z0-9_-]+)'
https://grafana.com/docs/loki/latest/clients/promtail/
Promtail sidecar

48. Annotate your graphs

49. Correlate events with metrics

50. • apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
• target "aws-asg" {
driver = "aws-asg"
config = {
aws_region = "{{ $x := env "attr.platform.aws.placement.availability-zone" }}{{ $length := len $x |subtract 1 }}{{ slice $x 0 $length}}"
}
}
Grow into your platform

51. • scaling "cluster_policy" {
policy {
cooldown = "2m"
evaluation_interval = "1m"
check "cpu_allocated_percentage" {
source = "prometheus"
query =
"scalar(sum(nomad_client_allocated_cpu{node_class="hashistack"}*100/(nomad_client_unallocated_cpu{node_class="hashistack"}+nomad_client_allocated_cpu{n
ode_class="hashistack"}))/count(nomad_client_allocated_cpu{node_class="hashistack"}))"
strategy "target-value" {
target = 70
}
}
target "aws-asg" {
dry-run = "false"
aws_asg_name = "${client_asg_name}"
node_class = "hashistack"
node_drain_deadline = "5m”
Grow into your platform

52. agent.worker.check_handler: querying source: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus strategy=target-value target=a
agent.worker.check_handler: calculating new count: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus strategy=target-value ta
agent.worker.check_handler: scaling target: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-36
internal_plugin.aws-asg: successfully performed and verified scaling out: action=scale_out asg_name=hashistack-n
agent.worker.check_handler: successfully submitted scaling action to target: check=mem_allocated_percentage policy_id=bf68649a-d087-2e69-362e-bbe71b5544f7 source=prometheus
Observe the autoscaler again

53. https://github.com/hashicorp/nomad-
autoscaler/tree/master/demo/remote
Try it yourself

54. Moving it all to the cloud – QED

55. Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie
https://hashiconf.com/europe/ <= Running Trusted Payloads With Nomad and Waypoint

56. Questions ?
The Floor is yours…