The document summarizes an incident where a Cassandra cluster experienced performance issues and nodes going down due to high load. It describes how the event loop threads became blocked waiting on the inbound request queue, causing high latency and client retries. The solution presented applies backpressure when the inbound queue becomes full by either returning an overloaded exception or applying throttling depending on a new startup option. Performance tests showed that with this backpressure patch applied, the cluster maintained higher throughput and lower latencies under heavy traffic loads compared to a cluster without the patch. Going forward, the backpressure mechanism could be improved to use more dynamic outlier detection rather than fixed thresholds. Applying the patch to production then prevented further blocked threads and out of memory errors under heavy loads
4. The Incident
Cluster metrics
#1 #2 #3
#4 #5 #6
T0
T1
T2
T3
T4
T5
Sequence of events
High ops
Blocked thread pool
High latency
Client retries
High heap
Nodes down
24. ● Do not block eventloop threads
● Define an Overloaded State
○ Configurable thresholds per client IP and Global (node-level)
native_transport_max_concurrent_requests_in_bytes_per_ip
native_transport_max_concurrent_requests_in_bytes
25. ● What should Cassandra do while in Overloaded State?
○ Based on a new client connection startup option THROW_ON_OVERLOAD
○ By default, apply backpressure
○ Otherwise (if set to true), return “Overloaded Exception” response message
back to the client.
40. ● Backpressure is based on memory usage on Incoming queue - biased towards
writes
● Depends on fixed thresholds. Instead, have dynamic outlier detection
