A lightning talk on tips and general recommendations for troubleshooting Apache Cassandra.

1. #CASSANDRA13
Jeremy Hanna | Support Engineer at DataStax | @jeromatron
Troubleshooting Cassandra

2. #CASSANDRA13
Java
*Latest 1.6 Oracle JDK (not OpenJDK)
*Make sure JNA is enabled “JNA mlockall successful”

3. #CASSANDRA13
Garbage Collection
*Look for warnings:
GC for [ParNew|ConcurrentMarkSweep]: 3000 ms
Heap is 0.93 full. You may need to ...
*Enable GC logging in cassandra-env.sh
correlate with system.log pauses
*Heap dump
*OpsCenter GC graphs

4. #CASSANDRA13
Memory
*MAX_HEAPSIZE
Keep as low as you can (ideally ≤ 8G)
*HEAP_NEWSIZE
No more than 800M
*Disable swap
*Trend towards off-heap
*Be careful with cache (even off-heap cache)

5. #CASSANDRA13
Troubleshooting
*Narrow down bottlenecks
*Query tracing in 1.2+
*What has changed recently? (conf, data size, deletions, etc.)
*OpsCenter
Relevant graphs
Diagnostics in 3.x

6. #CASSANDRA13
Special note on SSD performance
*Look at your tpstats
High pending or blocked on flush writer
Coincides with relatively idle disk usage
*Update device queue settings
echo noop > /sys/block/sda/queue/scheduler
echo 0 > /sys/block/sda/queue/read_ahead_kb
echo 0 > /sys/block/sda/queue/rotational
*Change mount options
/dev/sda /var/commitlog ext4 noatime,nodiratime,discard,errors=remount-ro 1 2