This document discusses Ceph, an open-source distributed storage system. It provides concise overviews of key Ceph components and features, including RADOS, which provides a reliable, autonomic distributed object store; RADOS Gateway, which enables S3 and Swift compatibility; RBD for virtual block devices; and tiered storage. CRUSH is described as Ceph's algorithm for determining data placement across clusters in a way that avoids failures and enables high parallel recovery. Benchmark results are shown for a Dell hardware configuration using Ceph.

1. Block and Object Store
Thursday, 21 April 2016
Senior Technical Specialist
Computing Platforms – Technical Support Services
University of Cape Town
Stefan Coetzee

2. Ceph
Robust Services built on RADOS

3. Ceph Components

4. RADOS Gateway

5. RADOS Gateway Features
• REST-based object storage proxy
• Uses RADOS to store objects
• Stripes large RESTful objects across many RADOS objects
• API supports buckets & accounts
• Usage accounting for billing
• Compatible with S3 and Swift applications
• Zones and regions
• Topologies similar to S3 and others
• Global bucket and user/account namespace
• Cross data center synchronization
• Asynchronously replicate buckets between regions
• Read affinity
• Serve local data from local DC
• Dynamic DNS to send clients to closest DC

6. RBD Virtual Disks

7. RBD Kernel Module

8. RBD Features
• Stripe images across entire cluster (pool)
• Read-only snapshots
• Copy-on-write clones
• Broad integration
• Qemu
• Linux Kernel
• iSCSI (STGT, LIO)
• Openstack, CloudStack, Nebula, Ganeti, Proxmox
• Incremental backup (relative to snapshots)

9. Ceph
RADOS – Reliable Autonomic Distributed Object Store

10. RADOS
• Flat object namespace within each pool
• Rich object API (librados)
• Strong consistency
• Infrastructure aware, dynamic topology
• Hash-based placement (CRUSH)
• Direct client to server data path

11. RADOS Cluster

12. RADOS Components
OSDs
• 10s to 1000s in a cluster
• One per disk ( or per SSD, RAID group …)
• Serve stored objects to clients
• Intelligently peer for replication & recovery
Monitors
• Maintain cluster membership and state
• Provide consensus for distributed
decision-making
• Small, odd number (eg 5)
• Not part of data path

13. Object Storage Daemons

14. Ceph
Data Placement – CRUSH (Controller Replication
Under Scalable Hashing)

15. Where do objects live?

16. A metadata server?

17. Calculated placement

18. CRUSH

19. CRUSH is a quick calculation

20. CRUSH avoids failed devices

21. CRUSH: Declustered placement
• Each PG independently maps to a
pseudorandom set of OSDs
• PGs that map to the same OSD generally
have replicas that do not
• When an OSD fails, each PG it stored will
generally be re-replicated by a different OSD
• Highly parallel recovery
• Avoid single-disk recovery bottleneck

22. CRUSH: Dynamic Data Placement
• Pseudo-random placement algorithm
• Fast calculation, no lookup
• Repeatable, deterministic
• Statically uniform distribution
• Stable mapping
• Limited data migration on change
• Rule-based configuration
• Infrastructure topology aware
• Adjustable replication
• Weighted devices (different sizes)

23. Ceph
Tiered Storage

24. 2 ways to cache
• Within each OSD
• Combing SSD & HDD under each OSD
• Make localized promote/demote decisions
• Leverage existing tools
• dm-cache, bcache, FlashCache
• Controller based caching
• Cache on separate devices/nodes
• Different hardware for different tiers
• Slow nodes for cold data
• High performance nodes for hot data
• Scale each tier independently

25. Tiered Storage

26. RADOS Tiering Principles
• Each tier is a RADOS pool
• May be replicated or erasure coded
• Tiers are durable
• eg replicated across SSDs in multiple hosts
• Each tier has it's own CRUSH policy
• eg map cache pool to SSDs devices/hosts only
• librados adapts to tiering topology
• Transparently direct requests accordingly
• No changes to RBD, RGW, CephFS, etc.

27. Read (Cache Hit)

28. Read (Cache Miss)

29. Read (Cache Miss with Promote)

30. Write (Cache Hit)

31. Write (Cache Miss)

32. Tiering Agent
• Each PG has an internal tiering agent
• Manages PG based on administrator defined policy
• Flush dirty objects
• When pool reaches target dirty ratio
• Tries to select cold objects
• Marks objects clean when they have been written back to the
base pool
• Evict (delete) clean objects
• Greater "effort" as cache pool approaches target size

33. Ceph
ARC Implementation

34. 01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
E S T
PowerEdge R620 6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
6
7
5
3
4
2
1
0
E S T
PowerEdge R620 6
7
5
3
4
2
1
0
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0
2
1
3
5
4
6
8
7
9
11
10
Front
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
Dell R730xd
• 2x Intel Xeon E5-2630 (8 Core)
• 384GB DDR4
• 2x 3.5" 400GB SSD
• 6GB LSI SAS
• 4x 10GB Intel Ethernet
Dell MD3060e
• 40x 6TB SATA

35. Infrastructure Costs
• Dell R730xd
• R 115 000
• MD 3060e
• R 310 000

36. Benchmarks
• Write
• 16 writes per second
• 4MB Blocks
• 600 MB/s sustained throughput
• Rand Read
• 16 Reads per second
• 4MB Blocks
• 2600 MB/s sustained throughput

37. //TODO:
• Introduce caching layer
• Better align PG to OSD count

38. Thank You

40. Section header
Subhead