This talk is about orchestration of Cassandra on Kubernetes with Cassandra Operator and Yelp's Platform-as-a-Service: PaaSTA. The talk focusses specifically on the internals of cassandra operator and its core reconcile loop for reconciliation of cluster state and on-disk configuration.

1. Cassandra Operator with
Yelp PaaSTA
Raghavendra Prabhu & Matthew Mead-Briggs
JetStack Connect Nov 2019

2. Yelp’s Mission
Connecting people with great
local businesses.

3. Powered by Cassandra
Your Reviews
1
2
1
Waitlist for diners2

4. Overview
● Why Kubernetes
● Status quo with Cassandra
● Cassandra Operator
● Pain Points
● Future

5. What is PaaSTA
github.com/yelp/paasta github.com/yelp/clusterman

6. Why K8s
● Why Kubernetes
○ Why not persist with plain EC2
● Why DIY the operator

7. Status Quo of Cassandra @ Yelp
● What is Cassandra
● Roughly a hundred clusters on AWS ASG
● New cluster launches with k8s
○ Batteries included: Good defaults, TLS
● Migration strategy in place
○ Backward compatible discovery mechanism
● K8s clusters deployed on spot fleet as well.
● Local k8s development cluster with
https://github.com/kubernetes-sigs/kind

8. Cassandra
Operator

9. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft
Multi-region Cluster

10. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft + Operator
Multi-region Cluster

11. State of Cassandra Operator at Yelp
● Cassandra cluster specification
● What is in a Cassandra Pod
● Storage aka State
● Reconciliation
○ StatefulSet
○ Core event loop
● Deployment

12. The Recipe aka the Cluster Spec

13. Smartstack Seed Provider
Synapse
Client
Service
HAProxy
Nerve
Service
ZK

15. Cassandra Pod
● Cassandra container
● Sidecars
○ Hacheck for Nerve (Smartstack)
○ Cron Jobs
○ Sensu alerting
● Node: metrics collection
○ Puppet

16. To sidecar or not to sidecar
● Emit data to host/external
service
● Collect data from process in
hosts namespace
● Sidecar collects data

17. Storage aka State
● Dynamic Provisioning
○ StorageClass per cluster
○ “Compute follows Data”
■ Immediate Volume Binding Mode
■ Stripe cluster across AZs
● EBS for Cassandra
○ Clear separation of stateful and stateless
○ Makes it easy to delete statefulsets
○ Bouncing the cluster is also quite fast

18. Storage aka State

23. Hash-based reconciliation
● Compute hash of Pod Template
● Attach as label to the StatefulSet
● Compare label on existing StatefulSet to newly computed

24. Cluster Readiness
● Cluster ready = AND(pod readiness) over all
○ Service Readiness
● Readiness per pod: UN in Cassandra
● Liveness check: U for Cassandra
● Hooks
○ Draining

25. Locking
● Clusters are multi-region
● Operators are per-region
● Non-federated setup
● Coordination with etcd leases
● LeaseID stored in Custom Resource Status

26. IAM roles
● For cassandra we need access to S3 and dynamoDB for
backups
● https://github.com/uswitch/kiam
● Proxies the EC2 metadata service for Pods
● Allows us to lift and shift IAM profiles from EC2

27. PaaSTA Secrets Support
● PaaSTA on Mesos already supports secrets
● User friendly cli to “create” secrets
● Use Vault’s transit endpoint to encrypt
● Sync these secrets into kubernetes Secrets
● Cassandra is using these for TLS secrets
$ echo "SOMETHINGSECRET" | paasta secret add -s cassandra_k8s -n
secret-name-here -c norcal-devc

28. Deployment / PaaSTA integration

31. Migration
● Launching new clusters in k8s is easy
● Migration of existing clusters without downtime is hard
● Unified discovery with smartstack
● How: Add k8s nodes to existing Cassandra cluster
● We have migrated a few already!

32. Pain Points

33. Pain points
● Client-side Validation
● Statefulset inflexibility with changes
○ Manual intervention for stuck statefulset deployments
○ Resizing the Persistent Volume
○ Orphaned EBS volumes
● Unready/Dead Nodes, Spot fleet and garbage collection

35. Heading towards >
● Load-based autoscaling for Cassandra pods
● EBS snapshotting automation
● Fleet autoscaling with Clusterman
● Better integration tests for the operator and for the clusters!
● Production clusters on AWS spot fleet
● More workloads on kubernetes (just started our Kafka operator)

36. Conclusion

37. We're Hiring!
www.yelp.com/careers/

39. Questions?

40. Credits
● Apache cassandra logo
● https://kubernetes.io/
● https://etcd.io/
● https://aws.amazon.com/architecture/icons/
● https://www.yelp.com/brand
● https://thenounproject.com/
● https://commons.wikimedia.org/wiki/File:Back-to-the-future-logo.svg
● https://www.writeups.org/star-trek-brent-spiner-data/

41. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp