In this talk Ben will walk you through running Cassandra in a docker environment to give you a flexible development environment that uses only a very small set of resources, both locally and with your favorite cloud provider. Lessons learned running Cassandra with a very small set of resources are applicable to both your local development environment and larger, less constrained production deployments.

1. Cassandra and Docker
Two buzzwords or a match made in heaven?
instaclustr.com
@Instaclustr

2. Who am I and what do I do?
• Ben Bromhead
• Co-founder and CTO of Instaclustr -> www.instaclustr.com
<sales>
• Instaclustr provides Cassandra-as-a-Service in the cloud.
• Currently in AWS, Azure and Google Cloud in private beta with more to come.
• We currently manage 50+ nodes for various customers, who do various things with
it.
</sales>

3. Objectives
• A quick intro on docker.
• Why docker matters and how it works.
• Working with Cassandra and docker.
• Running C* in a constrained env w/ docker

4. The Challenge
Static website
Web frontend
DB
Queue
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
Java + Cassandra + Spark
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs +
phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
DevelopmentVM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
Mul$plicity*of*Stacks*
Mul$plicity*of*
hardware*
environments*
Production Cluster
Customer Data Center
Do*services*and*apps*
interact*
appropriately?*
Can*I*migrate*
smoothly*and*
quickly?*

5. Static website
Web frontend
Background workers
DB
Analytics
Queue
Developmen
t VM
QA Server
Single Prod
Server
Onsite
Cluster
Public Cloud
Contributor’
s laptop
Customer
Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Dependency madness

6. Pre 1960’s transportMul$plicity*of*Goods*
Mul$pilicity*of*
methods*for*
transpor$ng/storing*
Do*I*worry*about*
how*goods*interact*
(e.g.*coffee*beans*
next*to*spices)*
Can*I*transport*quickly*
and*smoothly*
(e.g.*from*boat*to*
train*to*truck)*

7. Also a dependency mess
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?

8. Solution: Intermodal Shipping containers
Mul$plicity*of*Goods*
Mul$plicity*of*
methods*for*
transpor$ng/storing*
Do*I*worry*about*
how*goods*interact*
(e.g.*coffee*beans*
next*to*spices)*
Can*I*transport*
quickly*and*smoothly*
(e.g.*from*boat*to*
train*to*truck)*
…in between, can be loaded and
unloaded, stacked, transported
efficiently over long distances,
and transferred from one mode
of transport to another!
A standard container that is
loaded with virtually any
goods, and stays sealed until
it reaches final delivery.!

9. Docker, shipping containers for code
Static website Web frontendUser DB Queue Analytics DB
Development
VM
QA server Public Cloud Contributor’s
laptop
Mul$plicity*of*Stacks*
Mul$plicity*of*
hardware*
environments*
Production Cluster
Customer Data
Center
Do*services*and*apps*
interact*
appropriately?*
Can*I*migrate*
smoothly*and*quickly*
…that can be manipulated using
standard operations and run
consistently on virtually any
hardware platform !
An engine that enables any
payload to be encapsulated
as a lightweight, portable,
self-sufficient container…!

10. Solves the deployment dependency matrix
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developmen
t VM
QA Server
Single Prod
Server
Onsite
Cluster
Public Cloud
Contributor’
s laptop
Customer
Servers

11. Why docker matters
• Finally Developers have a solution to build once and deploy
anywhere
• Finally Ops/Admin has a solution to configure anywhere
• Finally DevOps is easy
• Dev == Test == Staging == Production
• Move with speed

12. Docker, how it works.
• Runs anywhere (Linux kernel 2.6.32+)
• Uses lightweight VMs:
• Own process space (namespace)
• Process isolation and resource control (cgroups)
• Own network adapter
• Own filesystem (chroot)
• Linux Analog to Solaris Zones, *BSD jails

13. Docker, how it works.
• Difference between a container and a VM
Virtual Machine Container

14. Docker, how it works.
• What about the packaging component?
• Uses Union filesystem to create a git like workflow around your deployed code:
!
!
Docker!
Container!
Image!
Registry!
Push%
!
!
!
!
Bins/!
Libs!
!
!
!
!
App!
A!
App!Δ!!
!
!
!
!
Bins/!
Docker'Engine' Docker'Engine'
Update'
Host'is'now'running'A’’'
'
App'Δ''
'
'
'
'
Bins/'
'
'
'
'
Bins/'
Libs'
'
'
'
'
App'
A'
'
'
'
'
Bins/'
'
'
'
'
Bins/'
Libs'
'
'
'
'
App'
A’’'
Host'running'A'wants'to'upgrade'to'A’’.'
Requests'update.'Gets'only'diffs'
'

15. Docker is it production ready?

16. Cassandra and Docker
• So how do we get on board the hype train? Without killing
performance or stability?
• Build Cassandra in a docker container, run it, then test.
• Run in dev to get comfortable with it.
• Talk to others who use it in production
• https://github.com/docker/docker/issues - You will spend a lot of
time here

17. Docker + Networking
• 1st attempt, throughput dropped in half!
• Writes sucked, streaming sucked, what was going on?
• Quick check with iperf showed a 50% hit in throughput
• Docker uses Linux Ethernet Bridges for basic software defined
routing. This will hose your network throughput.
• Use the host network stack instead (—net=host), only saw a ~10%
hit on network performance

18. Docker + Networking
• Docker uses Linux Ethernet Bridges for basic software defined
routing. This will hose your network throughput.
• Use the host network stack instead (—net=host), only saw a ~10%
hit on network performance
• Also solves NAT issues in an AWS like networking environment.

19. Docker + Filesystem
• Don’t want to throw it out when you upgrade/stop container.
• Use volume mount folders to the underlying host!

20. Docker + Filesystem
• The filesystems (AUFS, BTRFS etc) that bring great benefits to
Dockers workflow around building and snapshoting containers are
not very good for databases.
• UnionFS (AUFS) is terrible for writing lots of big files.
• BTRFS is a pain to use from an ops point of view.
• Hooray volume mounts use the underlying filesystem.

21. Docker + Process Capabilities
• Mlockall permission denied? A process needs CAP_IPC_LOCK &
RLIMIT_MEMLOCK in order to perform this operation. By default
docker doesn't assign this to a running container…
• Can’t use native memory. Cassandra becomes slooow.
• Can use --privileged and be done with it. Kind of lazy though
• Use --cap-add instead

22. Docker + SIGTERM propagation
• When stopping the process docker will send a SIGTERM.
• PID 1 does not have default signal handlers!
• Bad if you use a bash script to launch Cassandra
Images shameless copied from https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

23. Docker + SIGTERM propagation
• Java to the rescue!
• Make sure you run the cassandra bash script with -f (foreground)
• exec causes the JVM to replace the bash process… making the
world a happier place

24. Docker + SIGTERM propagation
• Tools like OpsCenter Server will have trouble with this.
• Can be fixed using a wacky combination of trap and wait stanzas in
your OpsCenter Server script (see http://veithen.github.io/
2014/11/16/sigterm-propagation.html)
• But now you have a bash script that duplicates init/systemd/
supervisord
• The debate rages on…

25. Docker + CoreOS
• Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis
etc
• We chose CoreOS (Appeared to be sane, etcd is cool, systemd if
you are into that kind of thing)

26. Docker + CoreOS
• Disable automatic updates + restarts (seriously do this)
• Fix logging, otherwise you will log to 3 locations (/var/log/
cassandra, journalctl and dockers json based log
• JVM will exit with error 143 (128 + 15 for SIGTERM). Need to ignore
that in your systemd service definition.

27. Docker + Dev Env
• Docker relies on Linux kernel capabilites… so no native docker in
OS X
• We use OSX for dev, so we run vagrant and the CoreOS vagrant file
https://github.com/coreos/coreos-vagrant
• Look at https://github.com/tobert/cassandra-docker for something
more off the shelf

28. Docker + C* + Dev Env
• How do I run lots of C* instances on a VM or my dev laptop without
it falling over?
• Make it run as slowly, but as stable as possible!
• This is actually a great learning exercise as you discover a lot about
how Cassandra works under the hood.

29. Docker + C* + Dev Env
• Set Memory to be super low, edit your cassandra-env.sh:
MAX_HEAP_SIZE="128M"
HEAP_NEWSIZE="24M"

30. Docker + C* + Dev Env
• Tune compaction to have free reign and to smash the disk
concurrent_compactors:
1
in_memory_compaction_limit_in_mb:
2
compaction_throughput_mb_per_sec:
0

31. Docker + C* + Dev Env
• Let’s use HSHA thrift server as it reduces the memory per thread
used.
rpc_server_type:
hsha

32. Docker + C* + Dev Env
• The HSHA server also lets us limit the number of threads serving in
flight requests, but still have a large number of clients connected.
concurrent_reads:
4
concurrent_writes:
4
rpc_min_threads:
2
rpc_max_threads:
2
• You can play with these to get the right numbers based on how your
clients connect, but keep them low.

33. Docker + C* + Dev Env
• This is Dev! Caches have no power here!
key_cache_size_in_mb:
0
reduce_cache_sizes_at:
0
reduce_cache_capacity_to:
0

34. Docker + C* + Dev Env
• How well does this work?!?!
• Will survive running the insane workload in the c* 2.1 new stresstest
tool.
• We run this on AWS t1.micro instances
• Sign up at https://www.instaclustr.com and give our new Developer
nodes a spin!

35. Go forth and conquer!
Questions?