The talk presented at MySQL & Friends devroom at FOSDEM 2016 in Brussels: https://fosdem.org/2016/schedule/event/clusternaut/ Devroom: https://fosdem.org/2016/schedule/track/mysql_and_friends/

1. Clusternaut:
Orchestrating Percona XtraDB Cluster
with Kubernetes.
Raghavendra Prabhu
FOSDEM ’16
rprabhu@yelp.com/me@rdprabhu.com/@randomsurfer

2. Yelp’s Mission:
Connecting people with great
local businesses.

3. Yelp Stats:
As of Q3 2015
89M 3271%90M

4. Me
Raghavendra Prabhu
Software Engineer, Distributed Systems @ Yelp
rprabhu@yelp.com / me@rdprabhu.com

5. Galera - “The Oar boat”

6. κυβερνήτης “The Helmsman”

8. DCOS
➔ Mesosphere
➔ Paasta
➔ GCE
◆ Reference
➔ ECS
➔ Smartcloud
➔ Tectonic*

9. Rationale
➔ Nodes v/s Hosts
➔ Reusable components
◆ Monitoring
◆ DNS
◆ Logging
◆ Metrics
◆ Scheduler
➔ Agnostic
➔ Roles

10. The Fit
➔ Layered
◆ Client - Server
◆ Multi-layered
➔ Scaling
◆ Horizontal and Vertical
● Preferred?
➔ Statelessness in databases
◆ Planes of logic: Control/Data
➔ Elasticity
◆ Elastic Scalability

11. Declarative vs Imperative
➔ Configurable mgmt
◆ Puppet, Nix, Terraform
➔ Microservices
◆ What runs on my laptop
● What runs on server
● Reproducibility
➔ 12-factor app
➔ Composability
➔ Immutable deployment artifact

12. Containers
● Wth is a container and why should I care
○ Operating system virtualization
● Isolation
○ Hierarchies of isolation - application, cgroups,
namespaces, seccomp…
● Unikernels and VMs
○ Role?
● Some - LXC, Docker*, Rocket*, runc, jails, solaris
zones, lmcty, systemd-nspawn

13. Galera - really short intro!
➔ MySQL and WSREP api
➔ Galera plugin
➔ Group communication
➔ Synchronous replication
◆ ‘Virtually’
➔ EVS
➔ Certification-based
◆ Optimistic Concurrency
➔ Automatic Node Provisioning

14. Galera - really short intro!
➔ CAP theorem and Galera
◆ CP
➔ How does it fit
◆ Others
◆ Idempotency
➔ Stateless?
◆ Symmetric
◆ Replicas - Cassandra et.al.
◆ MySQL Cluster
➔ Maintenance of Quorum

15. Orchestration
● SOA def
○ ‘Stitching’
○ ‘Composing’
● Automation?
● Choreography
● Best of both worlds

16. Kubernetes
➔ Constituents:
◆ Kubelet
◆ Pods
● Main service & helpers
◆ Services
● The gcomm:// URL.
◆ Replication Controller
◆ Labels and Selectors
● MCollective

18. Kubernetes
➔ Others:
◆ Mesos
● Aurora, Chronos, Marathon
◆ Docker Swarm
◆ Fleet
➔ Key Differences

19. Pods

20. Pods
● Herd..
● What should they contain - containers!
● How is the grouping done
● Pods and nodes
○ Colocation
● Pod communication
● Labels

21. Services

22. Services
● Don’t commingle with `microservices`
● Think of endpoints.
● Layering architecture
○ Logical address of subset of pods
● Communication
○ Environment
■ Ordering requirement
■ Discovery
○ DNS
■ Issues with DNS

23. Replication Controller
● “Herd Management”
● ASG
● Pod template
○ Pattern
○ Anti-pattern
● Role
○ Init for cluster
○ Rolling updates
○ Multi-version

24. Networking
● Docker-style linking
● Proxy for Pods
● Types
○ Pod to Pod
○ Pod to Service
○ Intra-Pod
○ External to Service
● Providers:
○ OpenVSwitch / Flannel / Calico / Weave / Google

25. External components
● Flannel / Others
● Etcd
● Fluentd
● Skydns
● Container Registry
● Scheduler
● REST server
● Proxy
● cAdvisor

26. PAAS: PaaSTA
● Docker
● Mesos
○ Chronos
○ Marathon
● Sensu
● Smartstack
○ Zookeeper
● Jenkins
● Splunk / Signalfx
● Why

27. Deployment
● Declare and build individual Galera/PXC nodes.
○ Keep it minimal and simple
○ No assumptions
● Without Kubernetes
○ Docker-compose
■ Possible issues
● Galera node ⇔ Pod
○ Haproxy
○ xinetd

28. Deployment
● Basic Steps:
○ Create a ‘flat’ network - 10.0.0.0/24
○ Create a ‘cluster’ - zone
○ Create a service endpoint.
■ Internal service - 3306/4567/4568.
■ External service - 3306/3306(?).
■ Expose the external.
■ Session affinities.

29. Deployment
● Next:
○ Bootstrap a node Pod from a template.
■ Query existing with selector.
○ Start rest of nodes from template.
■ Point to Service with selector.
■ Replication controller
○ Volumes

30. Deployment - Implications
● Load balancing in state transfers
● Respawning of nodes on timeout
○ May not be same nodes.
● kubectl to manage
● Separation of client and cluster traffic

31. Service Definition

32. Dockerfile

33. ```
... | kubectl create -f -
kubectl expose service eclient --
port=3306 --target-port=3306 --
name=loadbl --type='LoadBalancer’
```

35. ```
.. | kubectl create -f -
kubectl scale --replicas=8
replicationcontrollers
controller
```
```
kubectl get --no-headers
pods -l 'name=pxc' | wc -l
2
kubectl stop …
kubectl get --no-headers
pods -l 'name=pxc' | wc -l
2
```

36. Credits!
● https://www.pinterest.com/duanejohnson851/star-trek-tng/
● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_-
_dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG
● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png
● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png
● https://camo.githubusercontent.
com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f67
2e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67
● http://galeracluster.com/documentation-webpages/_images/replicationapi.png

37. Further reading!
● http://kubernetes.io
● https://open.mesosphere.com/frameworks/
● https://coreos.com/kubernetes/docs/latest/kubernetes-networking.
html
● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html
● http://12factor.net/
● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html

38. Contact
Raghavendra Prabhu
rprabhu@yelp.com / me@rdprabhu.com
Twitter: @randomsurfer
Linkedin: rdprabhu
Github: ronin13
http://rdprabhu.com
http://about.me/raghavendra.prabhu
http://rprabhu.yelp.com

39. @YelpEngineering
YelpEngineers
engineeringblog.yelp.com
github.com/yelp