from my meetup. You can also watch the video of the meetup: https://www.youtube.com/watch?v=4n5pF_vEQE4

1. Swarm:Native
DockerClustering
27/7/16

2. AboutMe:
● docker run -d --name=royeetager royee-tageraboutme:latest
● docker inspect royeetager
[
{
“Name": “Royee Tager",
“Age": “34",
“Role": “DevOps Engineer @ eXelate, A Nielsen Company",
"State": {
"Status": “Have been working on Docker for the last 2 years",
“Emails”: [
“royeectu@gmail.com”,
“royee.tager@nielsen.com"
]
“Mobile”: “050-6531133”,
“Linkedin": “https://il.linkedin.com/in/royeetager",
“Comment": “My #1 meetup"
}
}
]

3. http://www.exelate.com
Aboutexelate:
● Founded in 2007 - New York (~100 employees) & Israel (~60 employees)
● Big Data company
● Has been acquired by Nielsen in March 2015
● Products:
○ DMP (Data Management Platform) ⇒ NMC (Nielsen Marketing Cloud)
○ Data Marketplace
○ Analytics

4. WhoreliesontheeXelatecustomerdatacloud?
46 of the top 50 internet ad brands:
Directly and via partnerships with 125 of the world’s top media companies
Reaching 90% of US Households and 5B Global Unique Devices and its Users

5. NielsenMarketingCloud(NMC):
Most of the product’s components are actually Docker containers

6. Agenda&Objectivesofthismeetup:
● Share our knowledge & experience
● Why do we use Docker?
● What were our goals?
● Terms:
○ Orchestration & Scheduling
○ Service Discovery & Service Registration
○ Networking
● How did we choose our stack?
● Problems we encountered
● How to build an entire environment from scratch
● Tips & Tricks
● Q&A

7. Goals:
● Monolithic ⇒ Microservices
● Scale up infrastructure & applications
● Easy & quick deployments
● Easy to maintain
● Cluster orchestration, management & scheduling
● HA
● Monitoring the infrastructure & applications healthchecks

8. Howtochoose:
● Make a list of requirements
● Make a list of the technologies stack
● Read articles & forums
● Compare between the various solutions
● POC
● Attend meetups
● Don’t be afraid to change things

9. EverythingisbasedonDocker1.11andnoton1.12
Docker 1.12 has been introduced for the 1st time @ the Dockercon16

10. WhatisacontainerScheduling:
Scheduler Interface Scheduler
Distributed
Information Store
Return Host Density
Schedule App F
On Host 2
HOST 1
App
A
App
B
HOST 2
App
C
App
F
HOST 3
App
D
App
E
Example: Scheduler APP F
● Requires data store (Consul, Etcd, etc)

11. ServiceRegistration:
Automatically registers/deregisters services for docker containers

12. Connection Request
ServiceDiscovery(example):
Discovery Service
DISCOVERY FLOW
3
App
A
HOST 1
192.168.1.2
App
B
HOST 2
192.168.1.3
Registered App A
IP: 192.168.1.2
Where is APP A?
App A is at
IP: 192.168.1.2

13. TechnologyStack-Containers:
● LXC
● Docker
● Rocket (rkt)

14. TechnologyStack-cluster/schedulers:
● Cloud oriented:
○ Amazon ECS (EC2 container service)
○ Tutum
● Cloud, bare metal, VMs:
○ Docker Swarm
○ Kubernetes
○ Mesos

15. TechnologyStack-Servicediscovery:
● Hosted @ Docker Hub
● Etcd
● Consul
● ZooKeeper
● Nomad

16. TechnologyStack-relationsdiagram:
Kubernetes
Nomad
Docker
Consul
Etcd
ZooKeeper Marathon
Mesos
Tutum
Amazon
ECS
Docker
Swarm
Hosted
Discovery
Service

17. TechnologyStack-relationsdiagram:
Docker
Tutum Amazon ECS

18. TechnologyStack-relationsdiagram:
Nomad
Docker Consul

19. TechnologyStack-relationsdiagram:
Docker
ZooKeeper Marathon
Mesos

20. TechnologyStack-relationsdiagram:
Kubernetes
Docker Etcd

21. TechnologyStack-relationsdiagram:
Docker
Consul
Etcd
Docker
Swarm
Hosted
Discovery
Service
ZooKeeper

22. Whatwechose&why:
● We chose to work with:
○ Docker
○ Swarm
○ Consul
● Swarm uses the standard Docker API = the same docker CLI
● Swarm supports bare metal, VMS, cloud
● Swarm supports many discovery services (Consul, Etcd, ZookKeeper, Hosted)
● Consul service discovery via: DNS & HTTP
● Consul supports healthchecks

23. What’sinourplate:
● At least 2 (HA) Docker hosts (EC2 instances)
● At least 2 (HA) Swarm hosts (EC2 instances)
● At least 3 (quorum) Consul servers (EC2 instances)

24. OurProductionEnvironment:consul01
consul02
consul03
docker01 docker02
swarm01 swarm02

25. Consul:
● Same binary for server / client (depends on the configuration)
● Install client on each docker host
● At least 3 (quorum) Consul servers (EC2 instances)
● DNS interface
● Web GUI

26. Consul-Serverconfiguration:

27. Consul-Clientconfiguration:

28. Consul-usefulcommands:

29. Consul-usefulcommands:

30. Consul-uI:

31. Consul-uI:

32. docker-server:
Docker daemon configuration:

33. Swarm-server:
The command for running swarm server:
Swarm server is running:

34. Swarm-agent:
The command for running swarm agent:
Must run on each Docker host of the cluster:

35. containerSchedulinginswarm:● Supported data stores: Consul, Etcd, ZooKeeper, Hosted (not sutiable for
production)
● Creating / starting containers in the cluster based on 2 steps:
○ 1 - Apply filters to exclude nodes:
■ Node Filters:
● Health - Prevents running containers on unhealthy nodes
● Constraint - Container on a host with a matching label or
metadata
■ Container Configuration Filters:
● Affinity - “Attractions” (name, id, image, label) between
containers
● Dependency - Container on the same host as another given
container (volume / network)
● Port - Selects a node where a particular port is available
○ 2 - Use a strategy to pick a node:
■ Random (not for production use)
■ Spread (default) - balance number of containers evenly across all
nodes
■ Binpack - Swarm optimizes for the node which is most packed

36. containerSchedulinginswarm-Spread:
Spread (default) - balance number of containers evenly across all nodes =
Swarm optimizes for the node with the least number of containers
Spread

37. containerSchedulinginswarm-binpack:
Binpack - Swarm optimizes for the node which is most packed
Binpack

38. Registrator:
● Runs on each Docker host (it is also a container)
● Reads events from Docker socket (/var/run/docker.sock)
● Supported registry backends:
○ Consul
○ Etcd
○ SkyDNS 2
○ Zookeeper
http://gliderlabs.com/registrator/latest

39. Registrator:
The command for running registrator:
Must run on each Docker host of the cluster:

40. ServiceRegistration-Registrator&Consul:

41. Networking(overlay)inswarm:● The default is the bridge device
● Requires (prerequisite) a valid key-value store service
● Needs to configure the Docker daemon on each Docker host
● docker network create --driver overlay myNetwork
● docker run -d --name=example --net=myNetwork
● Gives us isolation
● You can attach containers to multiple networks
VSBridge Network: Overlay Network:

42. Howwebuilttheenvironment(toolsset):
● DevOps => Ansible playbooks - for Docker, Swarm & Consul
● Developers / DevOps => Docker Beta / Toolbox
● Developers => Jenkins jobs:
○ Creates Docker images ⇒ Pushes images to private repositories (Docker
Hub)
○ CI (scale 1)
● Developers => docker-compose

43. Monitoring:
● We use DataDog
● dd-agent (container) on each Docker host
● We use the following integrations:
○ Docker
○ AWS
○ Consul
○ Slack
https://www.datadoghq.com

44. Monitoring:

45. Logs:
● Json-file is the default
● Logs driver: https://docs.docker.com/engine/admin/logging/overview
● Supported formats: syslog, journald, gelf, splunk, json, awslogs, etc
● We use Graylog (GELF = Graylog Extended Log Format)
● https://www.graylog.org

46. Problemsweencountered:
● Service registration
● Consul healthchecks
● Nginx => HAProxy => Nginx

47. FuturePlans:
● Docker 1.11 ⇒ Docker 1.12
○ Get rid of registrator
○ Get rid of Consul

48. Q&A

49. Wearehiring:
● DevOps (CI / CD) engineer
● Web application team leader
● Frontend developer
● Java & Node.js developer
● Java & performance engineer
● Java developer & machine learning
● Node.js developer
exelatejobsil@nielsen.com
http://exelate.com/about-us/careers