26 May presentation for Ceph Tech Talks. http://ceph.com/ceph-tech-talks/

1. Ceph Benchmarking Tool (CBT)
Kyle BaderCeph Tech Talk
May 26, 2016

2. INTRO TO CBT

3. • Benchmarking framework written in python
• Began as a engineering benchmark tool for upstream developlment
• Adopted for downstream performance and sizing
• Used by many people in Ceph community
• Red Hat
• Intel / Samsung / SanDisk
• Quanta QCT / Supermicro / Dell
WHAT IS IT?

4. CBT PERSONALITIES
HEAD
• CBT checkout
• Key based authentication to all other hosts
• Including itself..
• PDSH packages
• Space to store results archives
• YAML testplans

5. CBT PERSONALITIES
CLIENT
• Generates load against the SUT
• Ceph admin keyring readable by cbt user
• Needs loadgen tools installed
• FIO
• COSbench
• Should be a VM for kvmrbdfio
• Can be containerized (good for rbdfio)

6. CBT PERSONALITIES
MON • Nodes to setup monitors on
OSD • Nodes to setup OSDs

7. • RADOS Bench
• FIO with RBD engine
• FIO on KRBD on EXT4
• FIO on KVM (vdb) on EXT4
• COSBench for S3/Swift against RGW
CBT BENCHMARKS

8. • Cluster creation ( optional, use_existing: true )
• Cache tier configuration
• Replicated and Erasure coded pools
• Collects monitoring information from every node
• Collectl – cpu/disk/net/etc.
CBT EXTRAS

9. • SSH Key on head
• Pub key in all hosts authorized_keys (including head)
• Ceph packages on all hosts
• PDSH packages on all hosts (for pdcp)
• Collectl installed on all hosts
BASIC SETUP

10. • Test network beforehand, bad network easily impairs performance
• All-to-All iperf
• Check network routes, interfaces
• Bonding
• Switches should use 5-tuple-hashing for LACP
• Nodes should use LACP xmit_hash_policy=layer3+4
TEST METHODOLOGY

11. • Use multiple iterations for micro benchmarks
• Use client sweeps to establish point of contention / max throughput
• Client sweeps should always start with X(1) ~ 1 client
• Should have 4-6 different increments of clients
• Eg. client1, client[1-2], client[1-3], client[1-4]
TEST METHODOLOGY

12. Testplan Examples

13. CBT CLUSTER CONFIGURATION
cluster:
head: "ceph@head”
clients: ["ceph@client"]
osds: ["ceph@osd"]
mons: ["ceph@mon"]
osds_per_node: 1
fs: xfs mkfs_opts: -f -i size=2048
mount_opts: -o inode64,noatime,logbsize=256k
conf_file: /etc/ceph.conf
ceph.conf: /etc/ceph/cepf.conf
iterations: 3
rebuild_every_test: False
tmp_dir: "/tmp/cbt"
pool_profiles:
replicated:
pg_size: 4096
pgp_size: 4096
replication: 'replicated'

14. CLIENT SWEEPS
cluster:
head: "ceph@head”
clients: ["ceph@client1"]
osds: ["ceph@osd"]
mons: ["ceph@mon"]
cluster:
head: "ceph@head”
clients: ["ceph@client1”,”ceph@client2”]
osds: ["ceph@osd"]
mons: ["ceph@mon"]
cluster:
head: "ceph@head”
clients: ["ceph@client1”,”ceph@client2”,
”ceph@client3”]
osds: ["ceph@osd"]
mons: ["ceph@mon"]
cluster:
head: "ceph@head”
clients: ["ceph@client1”,”ceph@client2”,
“ceph@client3”,”ceph@client4”]
osds: ["ceph@osd"]
mons: ["ceph@mon"]

15. • Spawns RADOS bench processes
on each client
• Establish raw RADOS throughput
• Works against replicated or EC pools
RADOS BENCH
benchmarks:
radosbench:
op_size: [ 4194304, 524288, 4096 ]
write_only: False
time: 300
concurrent_ops: [ 128 ]
concurrent_procs: 1
use_existing: True
pool_profile: replicated
osd_ra: [256]

16. • Spawns FIO proccesses on each client
• Uses RBD ioengine
• Establish raw librbd performance
• No VM / container setup required
FIO WITH RBD IO ENGINE
benchmarks:
librbdfio:
time: 900
vol_size: 65536
mode: [ randwrite, randread, randrw ]
rwmixread: 70
op_size: [ 4096, 16384 ]
procs_per_volume: [ 1 ]
volumes_per_client: [ 1 ]
iodepth: [ 16 ]
osd_ra: [ 128 ]
cmd_path: '/home/ceph-admin/fio/fio’
pool_profile: 'rbd’
log_avg_msec: 100
use_existing_volumes: true

17. • Maps KRBD volume to each client
• Creates EXT4 filesystem on KRBD
• Mounts filesystem
• Spawns FIO process per client
• Uses AIO IO Engine on filesystem
• Client can be container or bare metal
• Establishes KRBD performance potential
FIO WITH KRBD ON EXT4
benchmarks:
rbdfio:
time: 900
vol_size: 65536
mode: [ randwrite, randread, randrw ]
rwmixread: 70
op_size: [ 4096, 16384 ]
concurrnet_procs: [ 1 ]
iodepth: [ 16 ]
osd_ra: [ 128 ]
cmd_path: '/home/ceph-admin/fio/fio’
pool_profile: 'rbd’
log_avg_msec: 100

18. • Create KVM instances outside CBT
• KVM instances listed as clients
• Creates EXT4 filesystem on /dev/vdb
• Mounts filesystem
• Spanws FIO process per client
• Uses AIO IO Engine
• Establish RBD performance with QEMU
IO susbsystems
FIO WITH KVM (VDB) ON EXT4
benchmarks:
kvmrbdfio:
time: 900
vol_size: 65536
mode: [ randwrite, randread, randrw ]
rwmixread: 70
op_size: [ 4096, 16384 ]
concurrnet_procs: [ 1 ]
iodepth: [ 16 ]
osd_ra: [ 128 ]
cmd_path: '/home/ceph-admin/fio/fio’
pool_profile: 'rbd’
log_avg_msec: 100

19. • Install COSBench on
head/clients outside CBT
• Install / Configure RGW
outside CBT
• Translates CBT YAML to
COSBench XML
• Runs COSBench
COSBENCH
benchmarks:
cosbench:
cosbench_dir: /root/0.4.1.0
cosbench_xml_dir: /home/ceph-admin/plugin/cbt/conf/cosbench/
controller: client01
auth:
config: username=cosbench:operator;password=intel2012;url=…
obj_size: [128KB]
template: [default]
mode: [write]
ratio: [100]
….
Example at cbt/docs/cosbench.README

20. Running CBT
# Loop through each test plan
for clients in $(seq 1 6);do
cbt/cbt –archive=/tmp/${clients}-clients-results path/to/test.yaml
done

21. ANALYZING DATA

22. • No robust tools for analysis
• Nested archive directory based on YAML options
• Archive/000000/Librbdfio/osd_ra-00000128…
• Usually awk/grep/cut-fu to csv
• Plot charts with gnplot, Excel, R
ANALYZING DATA

23. THANK YOU!