In this talk, we will share useful tools and techniques that we are using in the field to understand Scylla clusters. Users will learn how to use those same tools to better understand their deployment. Some of the questions that will be answered are: - how to find out which queries are the slowest and why - how we go about understanding the impact of the data model in a node's performance - how to check which resources are the bottlenecks in the cluster

1. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Scylla
Performance Toolbox
ScyllaDB
Avi Kivity

2. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Understanding environment
and application impact
on performance
CTO, ScyllaDB
Avi Kivity

3. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Avi Kivity
3
KVM hypervisor author and ex-maintainer
ScyllaDB co-founder and CTO

4. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Agenda
4
▪ Environment
▪ Tracing
▪ Metrics

5. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Environment

6. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Environment
▪ Networking
▪ Disk interrupts
▪ Disk write cache
▪ Virtualization and containers
6

7. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Networking model (multiqueue)
7
NIC
OS/HW
Core Core Core Core Core Core
Rx Queue

8. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Networking model (singlequeue)
8
NIC
OS/HW
Core Core Core Core Core Core
Rx Queue
S/W Rx Queue

9. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Networking model (hybrid)
▪ Each core group is assigned a single hardware queue
▪ One core in core group handles networking
▪ Useful when too few hardware queues
▪ Too difficult to draw
9

10. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
How is the networking model configured?
▪ Determined by scylla_setup based on the hardware
▪ Stored in /etc/scylla.d/perftune.yaml
10
$ cat /etc/scylla.d/perftune.yaml
cpu_mask: '0x000000ff'
mode: mq
nic: eth0
tune:
- net

11. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Unbalanced networking
top - 11:40:29 up 3 min, 1 user, load average: 4.48, 4.36, 3.16
Tasks: 152 total, 8 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu0 : 34.3 us, 17.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 6.1 hi, 42.6 si, 0.0 st
%Cpu1 : 33.0 us, 5.0 sy, 0.0 ni, 59.1 id, 0.0 wa, 0.6 hi, 2.3 si, 0.0 st
%Cpu2 : 40.3 us, 4.3 sy, 0.0 ni, 52.2 id, 0.0 wa, 0.1 hi, 3.1 si, 0.0 st
%Cpu3 : 37.3 us, 5.7 sy, 0.0 ni, 54.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu4 : 31.0 us, 4.3 sy, 0.0 ni, 61.8 id, 0.0 wa, 0.2 hi, 2.7 si, 0.0 st
%Cpu5 : 41.3 us, 5.3 sy, 0.0 ni, 49.8 id, 0.0 wa, 0.1 hi, 3.5 si, 0.0 st
%Cpu6 : 31.0 us, 4.3 sy, 0.0 ni, 62.7 id, 0.0 wa, 0.0 hi, 2.0 si, 0.0 st
%Cpu7 : 34.0 us, 2.3 sy, 0.0 ni, 59.4 id, 0.0 wa, 0.2 hi, 4.1 si, 0.0 st
KiB Mem : 62882836 total, 61356464 free, 1129072 used, 397300 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 61124456 avail Mem
11

12. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back cache
Write-back cache
▪ Scylla writes to disk
▪ Disk places data in DRAM cache, and acknowledges
▪ Disk initiates data write to actual SSD in background
▪ Scylla asks disk to verify that the data made it to non-volatile
storage
▪ Disk waits until background write completes
o Potential stall
12

13. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
STALL
Disk write cache - write back
13
Scylla
Disk controller
Media
Write
Media
access
FlushACK
Media
access
complete
ACK

14. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back cache
Write-back cache
▪ Scylla writes to disk
▪ Disk places data in DRAM cache, and acknowledges
▪ Disk initiates data write to actual SSD in background
▪ Scylla asks disk to verify that the data made it to non-volatile
storage
▪ Disk does not wait until background write completes
o No stall
14

15. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back
15
Scylla
Disk controller
Media
Write
Media
access
Flush
ACK
Media
access
complete
ACK

16. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Beware of iowait
▪ iowait caused by pushing XFS out of its comfort zone
16
top - 11:40:29 up 3 min, 1 user, load average: 4.48, 4.36, 3.16
Tasks: 152 total, 8 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu0 : 34.1 us, 10.2 sy, 0.0 ni, 0.0 id, 47.0 wa, 6.1 hi, 2.6 si, 0.0 st

17. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Tracing

18. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Types of tracing
▪ Single-shot
▪ Probabilistic
▪ Slow query
18

19. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Single-shot tracing
▪ Useful for gaining an understanding of a query during
development
▪ Issue from cqlsh
19

20. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Probabilistic tracing
▪ Useful to gain an insight about what the application is doing
▪ Controlled by nodetool
▪ Start with very low probability to avoid disturbing the workload
20
$ nodetool settraceprobability 0.000001

21. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Slow-query logging
▪ Catch that long (and slow) tail
▪ Caution: a slow query can interfere with fast queries
21

22. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Metrics

23. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Metrics overview
▪ Aggregated vs. Shard metrics
▪ CPU metrics
▪ I/O metrics
▪ Coordinator-side metrics
▪ Replica-side metrics
23

24. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Zooming into aggregated metrics
▪ Start with cluster-level view
▪ Look at individual nodes
o Cluster runs at speed of slowest node
▪ Look at individual shards
o Node runs at speed of slowest shard
24

25. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
CPU metrics
▪ Utilization / load
o For throughput load, should achieve 100%
o If not
• Does one shard reach 100% and the others don’t?
– Hot partition
– Check networking environment
• Sufficient client concurrency?
25

26. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
I/O Queue metrics
I/O by type of operation: query, compaction, commitlog
▪ Bandwidth, IOPS (and average size)
▪ Delay
▪ Correlates with iostat command output
26

27. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Coordinator-side metrics
▪ CQL requests per second
▪ CQL connections and their distribution
o High connection open rate?
o Sufficient connections per shard?
o Bad connection distribution?
▪ Statements prepared
o Is the client using prepared statements correctly?
▪ Foreground reads and writes
▪ Background reads and writes
▪ Reconciliation
27

28. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Replica-side metrics
▪ Reads and writes - hot shard, hot node
▪ Cache hits/misses - compare with expectations
▪ Cache total memory - watch for sudden drops
▪ Active SSTable reads - high value indicates weak I/O
▪ Queued SSTable reads - high value indicates weak I/O
▪ Current compactions
28

29. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Summary

30. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Summary
▪ Many moving parts
▪ Despite automation, things can go wrong
▪ Application may get things wrong
▪ Need combination of methodical approach and intuition
▪ Engage the developers so we can improve things
30

31. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
THANK YOU
avi@scylladb.com
@AviKivity
Please stay in touch
Any questions?