Introduction to Ceph, an open-source, massively scalable distributed file system. This document explains the architecture of Ceph and integration with OpenStack.

1. What you need to
know about Ceph
Gluster Community Day, 20 May 2014
Haruka Iwao

2. Index
What is Ceph?
Ceph architecture
Ceph and OpenStack
Wrap-up

3. What is Ceph?

4. Ceph
The name "Ceph" is a
common nickname given to
pet octopuses, short for
cephalopod.

5. Cephalopod?

7. Ceph is...
{ }
object storage and file system
Open-source
Massively scalable
Software-defined

8. History of Ceph
2003 Project born at UCSC
2006 Open sourced
Papers published
2012 Inktank founded
“Argonaut” released

9. In April 2014

10. Yesterday
Red Hat acquires me
I joined Red Hat as an
architect of storage systems
This is just a coincidence.
Red Hat acquires me

11. Ceph releases
Major release every 3 months
Argonaut
Bobtail
Cuttlefish
Dumpling
Emperor
Firefly
Giant (coming in July)

12. Ceph architecture

13. Ceph at a glance

14. Layers in Ceph
RADOS = /dev/sda
Ceph FS = ext4
/dev/sda
ext4

15. RADOS
Reliable
Replicated to avoid data loss
Autonomic
Communicate each other to
detect failures
Replication done transparently
Distributed
Object Store

16. RADOS (2)
Fundamentals of Ceph
Everything is stored in
RADOS
Including Ceph FS metadata
Two components: mon, osd
CRUSH algorithm

17. OSD
Object storage daemon
One OSD per disk
Uses xfs/btrfs as backend
Btrfs is experimental!
Write-ahead journal for
integrity and performance
3 to 10000s OSDs in a cluster

18. OSD (2)
DISK
FS
DISK DISK
OSD
DISK DISK
OSD OSD OSD OSD
FS FS FSFS
btrfs
xfs
ext4

19. MON
Monitoring daemon
Maintain cluster map and
state
Small, odd number

20. Locating objects
RADOS uses an algorithm
“CRUSH” to locate objects
Location is decided through
pure “calculation”
No central “metadata” server
No SPoF
Massive scalability

21. CRUSH
1. Assign a placement group
pg = Hash(object name) % num pg
2. CRUSH(pg, cluster map, rule)
1
2

22. Cluster map
Hierarchical OSD map
Replicating across failure domains
Avoiding network congestion

23. Object locations computed
0100111010100111011 Name: abc, Pool: test
Hash(“abc”) % 256 = 0x23
“test” = 3
Placement Group: 3.23

24. PG to OSD
Placement Group: 3.23
CRUSH(PG 3.23, Cluster Map, Rule)
→ osd.1, osd.5, osd.9
1
5
9

25. Synchronous Replication
Replication is synchronous
to maintain strong consistency

26. When OSD fails
OSD marked “down”
5mins later, marked “out”
Cluster map updated
CRUSH(PG 3.23, Cluster Map #1, Rule)
→ osd.1, osd.5, osd.9
CRUSH(PG 3.23, Cluster Map #2, Rule)
→ osd.1, osd.3, osd.9

27. Wrap-up: CRUSH
Object name + cluster map
→ object locations
Deterministic
No metadata at all
Calculation done on clients
Cluster map reflects network
hierarchy

28. RADOSGW
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
RBD
A reliable and fully-
distributed block
device, with a Linux
kernel client and a
QEMU/KVM driver
CEPH FS
A POSIX-compliant
distributed file
system, with a Linux
kernel client and
support for FUSE
RADOSGW
A bucket-based REST
gateway, compatible
with S3 and Swift

29. RADOSGW
S3 / Swift compatible
gateway to RADOS

30. RBD
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
RBD
A reliable and fully-
distributed block
device, with a Linux
kernel client and a
QEMU/KVM driver
CEPH FS
A POSIX-compliant
distributed file
system, with a Linux
kernel client and
support for FUSE
RADOSGW
A bucket-based REST
gateway, compatible
with S3 and Swift

31. RBD
RADOS Block Devices

32. RBD
Directly mountable
rbd map foo --pool rbd
mkfs -t ext4 /dev/rbd/rbd/foo
OpenStack integration
Cinder & Glance
Will explain later

33. Ceph FS
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
RBD
A reliable and fully-
distributed block
device, with a Linux
kernel client and a
QEMU/KVM driver
CEPH FS
A POSIX-compliant
distributed file
system, with a Linux
kernel client and
support for FUSE
RADOSGW
A bucket-based REST
gateway, compatible
with S3 and Swift

34. Ceph FS
POSIX compliant file system
build on top of RADOS
Can mount with Linux native
kernel driver (cephfs) or FUSE
Metadata servers (mds)
manages metadata of the
file system tree

35. Ceph FS is reliable
MDS writes journal to RADOS
so that metadata doesn’t
lose by MDS failures
Multiple MDS can run for HA
and load balancing

36. Ceph FS and OSD
MDS
OSDOSDOSD
POSIX Metadata
(directory, time, owner, etc)
MDS
Write metadata journal
Data I/O
Metadata held in-memory

41. DYNAMIC SUBTREE PARTITIONING

42. Ceph FS is experimental

43. Other features
Rolling upgrades
Erasure Coding
Cache tiering
Key-value OSD backend
Separate backend network

44. Rolling upgrades
No interruption to the
service when upgrading
Stop/Start daemons one by
one
mon → osd → mds →
radowgw

45. Erasure coding
Use erasure coding instead
of parity for data durability
Suitable for rarely modified
or accessed objects
Erasure Coding Replication
Space overhead
(survive 2 fails)
Approx 40% 200%
CPU High Low
Latency High Low

46. Cache tiering
Cache tier
ex. SSD
Base tier
ex. HDD,
erasure coded
librados
transparent to clients
read/write
read when miss
fetch when miss
flush to base tier

47. Key-value OSD backend
Use LevelDB for OSD
backend (instead of xfs)
Better performance esp for
small objects
Plans to support RocksDB,
NVMKV, etc

48. Separate backend network
Backend network
for replication
Clients
Frontend network
for service
OSDs
1. Write
2. Replicate

49. OpenStack Integration

50. OpenStack with Ceph

51. RADOSGW and Keystone
Keystone Server
RADOSGW
RESTful Object Store
Query token
Access with token
Grant/revoke

52. Glance Integration
RB
D
Glance Server
/etc/glance/glance-api.conf
default_store=rbd
rbd_store_user=glance
rbd_store_pool=images
Store, Download
Need just 3 lines!
Image

53. Cinder/Nova Integration
RB
D
Cinder Server
qemu
VM
librbd
nova-compute
Boot from volume
Management
Volume Image
Copy-on-write clone

54. Benefits of using with
Unified storage for both
images and volumes
Copy-on-write cloning and
snapshot support
Native qemu / KVM support
for better performance

56. Wrap-up

57. Ceph is
Massively scalable storage
Unified architecture for
object / block / POSIX FS
OpenStack integration is
ready to use & awesome

58. Ceph and GlusterFS
Ceph GlusterFS
Distribution Object based File based
File location Deterministic
algorithm
(CRUSH)
Distributed
hash table,
stored in xattr
Replication Server side Client side
Primary usage Object / block
storage
POSIX-like file
system
Challenge POSIX file
system needs
improvement
Object / block
storage needs
improvement

59. Further readings
Ceph Documents
https://ceph.com/docs/master/
Well documented.
Sébastien Han
http://www.sebastien-han.fr/blog/
An awesome blog.
CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data
http://ceph.com/papers/weil-crush-sc06.pdf
CRUSH algorithm paper
Ceph: A Scalable, High-Performance Distributed File System
http://www.ssrc.ucsc.edu/Papers/weil-osdi06.pdf
Ceph paper
Ceph の覚え書きのインデックス
http://www.nminoru.jp/~nminoru/unix/ceph/
Well written introduction in Japanese

60. One more thing

61. Calamari will be open sourced
“Calamari, the monitoring
and diagnostics tool that
Inktank has developed as
part of the Inktank Ceph
Enterprise product, will soon
be open sourced.”
http://ceph.com/community/red-hat-to-acquire-inktank/#sthash.1rB0kfRS.dpuf

62. Calamari screens

63. Thank you!