Nomad is popular as an efficient, lightweight container orchestrator. But a truly efficient, lightweight deployment environment can only be built on a minimal Linux that is designed specifically for running containers. In this talk, we introduce Flatcar Container Linux to the Hashicorp/Nomad community. Already well known and widely deployed by Kubernetes users, Flatcar works just as well – or perhaps even better! – for Nomad. Flatcar Container Linux is a secure, immutable, auto-updating, lightweight Linux operating system. This makes Flatcar a perfect match for Linux containers running on Nomad: nodes will update automatically and stay secure in a simple way, without the administrator having to do the heavy lifting. This talk will explain best practices for deploying Nomad on Flatcar and demonstrate a Nomad cluster running on Flatcar.

1. Nomad + Flatcar: a harmonious
marriage of lightweights
Hashitalks 2022 | 2022-02-17

2. Hi, I'm Iago
Iago López Galeiras
Principal SWE Manager
Microsoft
Github: iaguis
Twitter: iaguis
Email: iagol@microsoft.com

3. What is Flatcar?
https://flatcar.org

4. flatcar
noun
/ˈflætkɑr/
a car on a train without a roof or sides,
used for carrying goods

5. What is Flatcar?
https://flatcar.org

6. Automated,
streamlined
updates
Easily apply all latest security
patches
Atomic updates and rollbacks
Minimal distribution
for containers
Reduced dependencies
Less base software to manage
Reduced attack surface area
What is a “Container Linux”?
Secure, immutable
file system
Read-only /usr partition
No package installation or
modification of core OS files
Removes entire category of
security threats - e.g. runc
vulnerability CVE-2019-5736*
Declarative
provisioning
First boot setup from declarative
configuration
Immutable infrastructure (no
custom per-node changes
during production)
Repeatable deployment
Operational simplicity for security and management at scale
*kinvolk.io/blog/2019/02/runc-breakout-vulnerability-mitigated-on-flatcar-linux

7. runc "breakout" vulnerability mitigated
kinvolk.io/blog/2019/02/runc-breakout-vulnerability-mitigated-on-flatcar-linux

8. Flatcar Heritage
minimal
set of
packages
update
mechanism
container
optimizatio
n
Gentoo ChromeOS
CoreOS Container Linux
Flatcar Container Linux

9. Flatcar Distribution
❏ Four channels:
❏ Alpha → Beta → Stable
❏ LTS
❏ Publicly available
❏ Images on AWS, Azure, Packet, GCE, …
❏ https://<channel>.release.flatcar-linux.net
❏ Update server
❏ Public: public.update.flatcar-linux.net
❏ Nebraska

10. How Flatcar fits Nomad nicely
❏ Both emphasize simplicity
❏ Containers
❏ Forget about your OS
❏ Always updated
❏ Latest security fixes applied automatically
❏ Repeatable deployments

11. How is Flatcar configured

12. Ignition Config
❏ JSON format
❏ Declaration of files, systemd units, networks, users,
filesystems, and partitions
❏ Referencing data from external resources
❏ Applied from initramfs (first-boot flag file for GRUB
sets kernel parameter)
❏ Contrast to cloud-init which runs after the
initramfs, and on every boot

13. Container Linux Config (CLC)
❏ Friendlier YAML format with extras (octal
permissions, variables for metadata)
❏ Transpiled to Ignition JSON through transpiler “ct”
cat cl.yaml | docker run --rm -i quay.io/coreos/ct:latest-dev > ignition.json
./flatcar_production_qemu.sh -i ignition.json
❏ Spec: flatcar.org/docs/latest/provisioning/config-transpiler/configuration/

14. Container Linux Config (CLC)

15. Container Linux Config Example
storage:
files:
- path: /etc/some.conf
filesystem: root
mode: 0644
contents:
inline: |
A=a
B=b
Or with remote instead of 'inline' content:
remote:
url: …

16. "Installing" Nomad on Flatcar
❏ No package manager
❏ No official container
❏ And running the Nomad agent on a container is not supported
❏ Immutable infrastructure
❏ Use ignition to deploy and extract binaries

17. "Installing" Nomad on Flatcar
...
files:
- path: /opt/bin/nomad_1.2.6_linux_amd64.zip
filesystem: root
mode: 0644
contents:
remote:
url: https://releases.hashicorp.com/nomad/1.2.6/nomad_1.2.6_linux_amd64.zip
verification:
hash:
function: sha512
sum: c134198bd79b1e0c98d12262a7450cd0eee941f70118cfd196955f064073d312e395585a034d5ce32aa4c4763cb5a20a75d6764
07347f1ee515865f8d206cab5
...

18. "Installing" Nomad on Flatcar
...
systemd:
units:
- name: "prepare-nomad-binary.service"
enabled: true
contents: |
[Unit]
Description=Unpack Nomad binary to /opt/bin
ConditionPathExists=!/opt/bin/nomad
[Service]
Type=oneshot
Restart=on-failure
RemainAfterExit=yes
Environment=NOMAD_VERSION=1.2.6
ExecStart=/usr/bin/unzip "/opt/bin/nomad_${NOMAD_VERSION}_linux_amd64.zip" -d /opt/bin
ExecStart=/usr/bin/rm "/opt/bin/nomad_${NOMAD_VERSION}_linux_amd64.zip"
[Install]
WantedBy=multi-user.target
...

19. "Installing" Nomad on Flatcar
❏ That's pretty much it

20. What's missing?

21. Missing piece: update orchestration
❏ Ideally you want to orchestrate updates
❏ So you don't end up with all your nodes updating at the same time
❏ You can accomplish this with Locksmith

22. Flatcar Update Process (without Kubernetes)
Partition
B
Partition
A
update_
engine
service
Update
Server
1
Check for
updates
2 Download new
image
locksmith
service
5 Boot with
partition B
Flatcar Node
❏ Based on Google Omaha protocol (from ChromeOS)
❏ Dual immutable partitions enable atomic updates and easy rollback
❏ User control over updates 📖https://docs.flatcar-linux.org/os/update-strategies/
etcd
3
Check
for
lock
4 Lock
availabl
e

23. Missing piece: update orchestration
❏ Ideally you want to orchestrate updates
❏ So you don't end up with all your nodes updating at the same time
❏ You can accomplish this with Locksmith
❏ Even better, you want your workloads to be moved to other
nodes while you're updating
❏ On Kubernetes this is done with FLUO (Flatcar Linux Update Operator)

24. Flatcar
update
operator
Flatcar Update Process (with Kubernetes)
Partition
B
Partition
A
update_
engine
service
Update
Server
1 Check for
updates
2 Download new
image
7 Boot with
partition B
Flatcar Node
flatcar
update
agent
4 Drain
node 3 Set reboot-
needed
annotation
5
Set reboot-
ok
annotation
6
Send
reboot
signal

25. Missing piece: update orchestration
❏ Ideally you want to orchestrate updates
❏ So you don't end up with all your nodes updating at the same time
❏ You can accomplish this with Locksmith
❏ Even better, you want your workloads to be moved to other
nodes while you're updating
❏ On Kubernetes this is accomplished with FLUO (Flatcar Linux Update Operator)
❏ It would be great to have something similar for Nomad

26. DEMO
https://github.com/kinvolk/nomad-on-flatcar

27. Appendix
❏ Flatcar docs
❏ https://www.flatcar.org/docs/latest/
❏ Using Flatcar with Terraform
❏ Flatcar Terraform examples: https://github.com/flatcar-linux/flatcar-terraform
❏ "Immutable Infrastructure with Flatcar Container Linux"
talk: https://fosdem.org/2022/schedule/event/flatcar_container_linux/

28. Thank you!