Containers have become a key component of modern distributed application design. Similarly, Apache Cassandra has emerged as a clear choice for a scalable, high available and performant database in a highly distributed cloud computing architecture. In this session, we'll walk you through a number of patterns that enabled us to run Cassandra successfully with Docker in production. We will also share the lessons learned in running Cassandra with a very small set of resources that are applicable to both your local development environment and larger, less constrained production deployments. We'll then introduce you to Amazon ECS, a highly scalable, high-performance service to run and manage distributed applications using Docker. Finally, we will discuss a few best practices used by our customers for running their Cassandra clusters on ECS.

1. Running Cassandra on
Amazon’s ECS
Anirvan Chakraborty
@anirvan_c

2. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Agenda
• Motivation
• Docker
• Cassandra
• Cassandra on Docker best practices
• EC2 Container Service ECS
• Cassandra on ECS

3. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Motivation

4. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Motivation
• Ease of development
• Support polyglot languages, frameworks and
components
• Operational simplicity
• Quick feedback loop

5. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker

6. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker history
• Came out of dotCloud, a PaaS company
• Was originally written in Python
• Got re-written in Golang in Feb, 2013
• Docker 0.1 was released on Mar, 2013
• Docker 1.11.1 is the latest release

7. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker tag line
Build, ship and run any app, anywhere

8. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker tag line
• Build: package your application in a container
• Ship: move it between machines
• Run: execute that container with your application
• Any application: as long as it runs on Linux
• Anywhere: local VM, bare metal, cloud instances

9. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Why Docker?
• Deploy reliably & consistently
• Execution is fast and light weight
• Simplicity
• Developer friendly workflow
• Fantastic community

10. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Apache Cassandra

11. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is Apache Cassandra?
• Fast distributed database
• High Availability
• Linear Scalability
• Predictable performance
• No single point of failure
• Multi-DC
• Easy to manage
• Can use commodity hardware
• Not a drop in replacement for RDBMS

12. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Hash ring
• Data is partitioned around the ring
• Location of data in ring is determined by
partition key
• Data is replicated to N servers based on RF
• All nodes hold data and can answer read or
write queries
source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview

13. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
CAP Tradeoff
• During network partition it is impossible to be
both consistent and highly available
• Latency between data centres also makes
consistency impractical
• Cassandra chooses Availability & Partition tolerance
over Consistency

14. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Replication
• Choose “replication factor” or RF
• Data is always replicated to each replica
• If node is down, missing data is replayed via hinted
handoff
source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview

15. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Consistency level
source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview
• Per query consistency
• ALL, QUORUM, ONE
• How many replicas to respond OK for query to
succeed

16. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cassandra on Docker

17. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* Dev Environment
• Make it run as slow, but as stable as possible!
• Super low memory settings in cassandra-env.sh
• MAX_HEAP_SIZE=“128M”
• HEAP_NEWSIZE=“24M”
• Remove caches in dev mode in cassandra.yml
• key_cache_size_in_mb: 0
• reduce_cache_sizes_at: 0
• reduce_cache_capacity_to: 0

18. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* Production
• Use host networking (—net=host) for better
network performance
• Put data, commitlog and saved_caches in volume
mount folders to the underlying host
• Run cassandra on the foreground using (-f)
• Tune JVM heap for optimal size
• Tune JVM garbage collector for your workload

19. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Amazon EC2
Container Service

20. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is ECS?
…is a highly scalable, fast, container
management service that makes it easy to
run, stop, and manage Docker containers
on a cluster of Amazon EC2 instances.
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

21. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is ECS?
Amazon Docker as a Service
https://www.expeditedssl.com/aws-in-plain-english

22. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
How does ECS work?
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

23. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cluster
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

24. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Container Instance
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

25. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
ECS Agent
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

26. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Task
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

27. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Task
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
{
"containerDefinitions": [
{
"name": "wordpress",
"links": [
"mysql"
],
"image": "wordpress",
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"memory": 500,
"cpu": 10
},
{
"name": "mysql",
"image": "mysql",
"cpu": 10,
"memory": 500,
"essential": true
}
],
"family": "hello_world"
}Task definition
Docker Container
Docker Container

28. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Service
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

29. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
ECS Service
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

30. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Typical ECS workflow
• Build Docker image using whatever you want
• Push image to registry
• Create JSON file describing your task definition
• Register this task definition with ECS
• Make sure that your cluster has enough resources
• Start a new task from the task definition

31. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cassandra on ECS

32. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* on ECS
• Simple service discovery using ECS API
• Custom Dockerfile and entry-point script to
control Cassandra configuration
• Cleanup downed node and repair cluster on
node startup

33. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Service Discovery
Using
• AWS EC2 API
• AWS ECS API
• AWS Metadata URL
1. Find all running tasks for a cluster
2.Find EC2 instance ids for the tasks
3.Find private IP addresses for the container
instances

34. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Custom Dockerfile
• Expose Cassandra configuration parameters as
Docker environment variables
• Update entry-point script to use Docker
environment variables to configure Cassandra

35. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Node Startup
1. Find the IPs of all running seed nodes
2.Check and identify any downed nodes in the
cluster nodetool status
3.Remove the downed nodes nodetool remove
4.Repair the cluster nodetool repair
5.Start a new Cassandra node

36. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Work in progress

37. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Where to next?
• Open source C* Dockerfile & entry-point script
• Use Consul for Service Discovery in ECS
• Consider Flocker for managing volumes with
Dockerized C*
• Consider Weave for forming C* cluster across
network domains

38. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Thanks!
Twitter: @cakesolutions
Tel: 0845 617 1200
Email: enquiries@cakesolutions.net
Jobs: http://www.cakesolutions.net/
careers

39. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Resources
https://academy.datastax.com/courses/ds101-introduction-cassandra
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-
container-service.html
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html
https://www.expeditedssl.com/aws-in-plain-english
https://github.com/aws/amazon-ecs-agent
https://blog.docker.com/2015/03/why-i-love-docker-and-why-youll-love-it-too/
http://www.datastax.com/resources/whitepapers/best-practices-running-datastax-
enterprise-within-docker