How to write webscale applications with consul, nomad and terraform

1. To the cloud and beyond...

2. Doner Hub
The best doner marketplace

3. Doner Hub
Marketplace for doners
Ratings and reviews
Search by dietary preferences
Social network

4. Old-school monolith
Internet DBShop app

5. Old-school monolith with LB
Internet DB
Shop app
Shop appLoadbalancer

6. Monolithic architecture problems
● Large codebase
● Hard to scale
● Redeploy the entire application on each update
● Bug in any module can potentially bring down the entire process
● Has a barrier to adopting new technologies
● Continuous deployment is difficult

8. NEED
TO SCALE!!!

9. Service oriented architecture
Internet DBShop app
Loadbalancer
Orders
Search
Reviews

10. Service oriented architecture
Internet
DB
Shop app
Loadbalancer
Orders
Search
Reviews
DB
DB

11. success?

12. Service dependency graph
Available
Unstable
Down

13. SOA
architecture
support

14. SOA architecture problems
● Adds a complexity to the project
● Partitioned database architecture
● Difficult to implement changes that span multiple services
● Complex testing process
● Service discovery
● Configuration
● Deployment

15. SOA architecture problems
● Adds a complexity to the project
● Partitioned database architecture
● Difficult to implement changes that span multiple services
● Complex testing process
● Service discovery
● Configuration
● Deployment

16. Problem: Load Balancing
Shop app
Node 1
Node 2
Node 3
Orders service
Ensure that requests are balanced across instances

17. Solution: Load Balancer for internal services
Shop app
Node 1
Node 2
Node 3
Orders service
Loadbalancer

18. Problem: health checking
Shop app
Node 1
Node 2
Node 3
Services
Loadbalancer
Route traffic only to alive instances

19. Problem: add new nodes
Shop app
Node 1
Node 2
Node 3
Services
Loadbalancer
Dynamically add new nodes Node 4

20. Problems with internal load balancers
Yet another infrastructure to run and maintain
Load grows with the numbers of service instances
Single point of failure
LB configuration management and reload
Require health probing

21. Solution: Consul
Consul is a distributed service mesh
to connect, secure, and configure
services across any runtime platform
and public or private cloud

22. Consul features
Service discovery
Distributed locks and semaphores
Distributed key-value storage
Cross DC sync

24. Consul service registration
Register services (IP + port + checks) via config file or HTTP
Discover services via DNS or HTTP
Load balancing

25. Service registration
$ curl -x PUT -d
'{
"name": "orders",
"port": 8000,
"Tags": [
"V1"
],
"checks": [{
"HTTP": "http://localhost:8000/health",
"Interval": "10s",
"TTL": "15s"
}]
}' http://localhost:8500/v1/agent/service/register

26. Consul: DNS interface
$ dig @127.0.0.1 -p 8600 orders.service.consul
...
;; QUESTION SECTION:
;orders.service.consul. IN A
;; ANSWER SECTION:
orders.service.consul. 0 IN A 172.20.20.10
orders.service.consul. 0 IN A 172.20.20.11
orders.service.consul. 0 IN A 172.20.20.12

27. Consul: DNS interface
$ dig @127.0.0.1 -p 8600 orders.service.consul SRV
...
;; QUESTION SECTION:
;orders.service.consul. IN SRV
;; ANSWER SECTION:
orders.service.consul. SRV 1 1 8000 foobar-0.node.dc1.consul.
orders.service.consul. SRV 1 1 8001 foobar-2.node.dc1.consul.
orders.service.consul. SRV 1 1 8002 foobar-3.node.dc1.consul.
;; ADDITIONAL SECTION:
foobar-0.node.dc1.consul. 0 IN A 172.20.20.10
foobar-1.node.dc1.consul. 0 IN A 172.20.20.11
foobar-2.node.dc1.consul. 0 IN A 172.20.20.12

28. Consul: HTTP API
$ curl http://localhost:8500/v1/catalog/service/orders
[{
"ID": "1a6010e0-518c-2338-89f5-d7b8e4031a67",
"Node": "node-1",
"Address": "172.20.20.10",
"Datacenter": "dc1",
"TaggedAddresses": {
"lan": "172.20.20.10",
"wan": "10.0.10.10"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceID": "orders-node-1-8000",
"ServiceName": "orders",
"ServiceTags": [],
"ServiceAddress": "172.20.20.10",
"ServiceMeta": {},
"ServicePort": 8000,
"ServiceEnableTagOverride": false,
"CreateIndex": 580989,
"ModifyIndex": 580989
}]

29. Service discovery and DNS
Zero touch integration with nearly any existing software
Look up as “orders.service.consul”
Shuffled results - ditch the load balancer

30. Failure detection
State of the world is as fresh as possible
Automatically pull failed instances out of service
Immediately put healthy new instances into service

31. Service oriented architecture
Shop app
Orders
Search
Reviews
Loadbalancer
Consul
Get service IP:port
/orders
/reviews
/search

32. Fabio
Zero configuration TCP/HTTP(S) reverse proxy
First class integration with Consul
Weighted routing
Manual overrides

33. Fabio tag registration
$ curl -x PUT -d
'{
"name": "orders",
"port": 8000,
"Tags": [
"V1",
"utlprefix-donerhub.com/orders"
],
"checks": [{
"HTTP": "http://localhost:8000/health",
"Interval": "10s",
"TTL": "15s"
}]
}' http://localhost:8500/v1/agent/service/register

34. Load balancing with Fabio
Fabio
Orders
Search
Reviews
Consul
Get IP:port
/orders
/reviews
/search
Internet

36. Risky features
Large refactoring
Switch storage layer
New payment API

37. Consul KV as a feature toggle
Write flag value to consul
$ consul kv put search/elastic true

38. Consul KV as a feature toggle
Read flag value from application
if consul.Get("search/elastic") == "true" {
// use elastic search
}

39. Consul KV as a feature toggle
Enables “dark mode” features
Easy way to disable without needing a code deploy

41. Solution: Consul template
Retrieves keys and services from Consul and renders them into a template
$ consul-template -template "in.tpl:out.txt:command"

42. Consul template
CONSUL
key = {{ key “keyName” }}
CONSUL TEMPLATE
key = value
FILE.CONF
1
2
3

43. How it works
Consul template performs blocking query against the KV store
Stores the value in the file when key is updated
Runs the specified command after the update

44. Availability around the world
frontend
orders reviews
frontend
orders reviews

45. Availability around the world
frontend
orders reviews
frontend
orders reviews

48. Availability around the world
frontend
orders reviews
frontend
orders reviews

49. Manual overrides
Override service address to “orders.service.eu.consul” or “172.30.20.10”
Not transparent
Need config fiddling and may need app restarts

50. Consul prepared queries
Centrally managed policies that allow dynamic behaviour for service lookups
without any knowledge from applications
Define which service to look up, and rules for what to do if none are not
available in the local datacenter
Can be executed via DNS or HTTP

51. Consul prepared query
$ curl -x POST -d
'{
"Name": "",
"Template": {
"Type": "name_prefix_match",
},
"Service": {
"Service": "${name.full}",
"OnlyPassing": true,
"Failover": {
"Datacenters": ["US", "EU"]
}
}
}' http://localhost:8500/v1/query

52. Consul prepared query
$ dig @127.0.0.1 -p 8600 orders.query.consul
...
;; QUESTION SECTION:
;orders.query.consul. IN A
;; ANSWER SECTION:
orders.query.consul. 0 IN A 172.20.20.10
orders.query.consul. 0 IN A 172.20.20.11
orders.query.consul. 0 IN A 172.20.20.12

53. Consul prepared query: Name + tags
$ curl -x POST -d
'{
"Name": "",
"Template": {
"Type": "name_prefix_match",
"Regexp": "^(([^.]+).)?([^.]+)$",
"RemoveEmptyTags": true
},
"Service": {
"Service": "${match(3)}",
"Tags": ["${match(2)}"],
"OnlyPassing": true,
"Failover": {
"NearestN": 3
}
}
}' http://localhost:8500/v1/query

54. Consul prepared query: Name + tags
$ dig @127.0.0.1 -p 8600 master.db.query.consul
...
;; QUESTION SECTION:
;master.db.query.consul. IN A
;; ANSWER SECTION:
master.db.query.consul. 0 IN A 172.30.30.10

55. Single line config for geo failover
Local service discovery: “orders.service.consul”
Local + geo failover: “orders.query.consul”

56. Availability around the world
frontend
orders reviews
frontend
orders reviews

57. Deployment

58. Deployment
Manual ?!
Bash script
System package managers (apt, rpm, yum, etc...)
Ansible, Puppet, Salt

59. Deployment problems
Complex deployment scripts
Infrastructure configuration and maintenance
Rollback to stable version
Resources utilization
Scalability

61. Solution: Nomad
Nomad is a single binary tool for schedule and
run applications on Linux, Windows, and Mac
across multiple datacenters

62. Nomad
● Declarative

63. Nomad
● Declarative
● Scalable

64. Nomad
● Declarative
● Scalable
● Predictable

65. Nomad
● Declarative
● Scalable
● Predictable
● Infrastructure agnostic

66. Nomad features
Docker Support
Operationally Simple
Built for Scale
Canary deployments
Handle server failures gracefully
UI interface

67. Nomad architecture

68. Nomad: multi DC architecture

69. Scheduler Types
Service - long running jobs
Batch - short lived and periodic jobs
System - runs on all nodes (logs collector, metrics, etc...)

70. job "orders" {
datacenters = ["eu1", "eu2"]
type = "service"
group "orders" {
count = 3
task "orders" {
driver = "docker"
config {
image = "registry.cloud/orders"
port_map { http = 80 }
}
resources {
cpu = 500
memory = 100
network { port "http" {} }
}
service {
name = "orders"
tags = ["urlprefix-donerhub.com/orders", "urlprefix-www.donerhub.com/orders"]
port = "http"
check {
name = "alive"
type = "http"
interval = "10s"
timeout = "2s"
path = "/health-check"
}
}
}
}
}

71. Nomad job scheduling
Schedule the job contained in the file job1.nomad, monitoring placement:
$ nomad job run job1.nomad
==> Monitoring evaluation "52dee78a"
Evaluation triggered by job "example"
Evaluation within deployment: "62eb607c"
Allocation "5e0b39f0" created: node "3e84d3d2", group "group1"
Allocation "5e0b39f0" status changed: "pending" -> "running"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "52dee78a" finished with status "complete"

73. Solved issues
Service discovery
Configuration management
Deployment

74. Problem: Infrastructure management
Service discovery
Configuration management
Deployment
Infrastructure management

75. Solution: Terraform
Terraform enables you to safely and
predictably create, change, and
improve infrastructure as code to
increase operator productivity and
transparency.

76. Terraform features
Infrastructure as Code
Execution Plans
Resource Graph
Change Automation
Multi-vendor api

77. Terraform features
Configuration can be stored in version control, shared, and collaborated on by
teams of operators
Track the complete history of infrastructure versions
Manage resources across all major infrastructure providers (AWS, GCP, Azure,
DigitalOcean, OpenStack, VMware, and more)

78. Terraform
Typical terraform project files:
$ ls
main.tf variables.tf output.tf terraform.tfstate

79. main.tf
provider "aws" {
access_key = "${var.access_token}"
secret_key = "${var.secret_key}"
region = "us-east-1"
}
resource "aws_instance" "web" {
count = ${var.backend_count}
ami = "ami-2757f631"
instance_type = "t2.micro"
}

80. variables.tf
variable "access_token" {
description = "AWS access token"
default = "123456789ABCDEF"
}
variable "secret_key" {
description = "AWS secret key"
default = "123456789ABCDEF"
}
variable "backend_count" {
description = "Backend instances count"
default = 5
}

81. output.tf
output "web servers IPs" {
value = ["${aws_instance.web.*.public_ip}"]
description = "Web servers public IPs"
}

82. $ terraform apply
# ...
+ aws_instance.web
ami: "ami-2757f631"
availability_zone: "<computed>"
instance_state: "<computed>"
instance_type: "t2.micro"
key_name: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
# ...
aws_instance.web: Creation complete
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
# ...

83. $ terraform show
aws_instance.web:
id = i-32cf65a8
ami = ami-2757f631
availability_zone = us-east-1a
instance_state = running
instance_type = t2.micro
private_ip = 172.31.30.244
public_dns = ec2-52-90-212-55.compute-1.amazonaws.com
public_ip = 52.90.212.55
subnet_id = subnet-1497024d
vpc_security_group_ids.# = 1
vpc_security_group_ids.3348721628 = sg-67652003

86. Thank you!
Oleg Lobanov
eBay
@o1egl
https://github.com/o1egl
HashiCorp Tools