inwinSTACK - ceph integrate with kubernetes

1. Ceph integrate with Kubernetes
Speaker : Thor Chin

2. Agenda
Ceph Luminous important
Features update
Why Kubernetes and Ceph
Ceph integrate with Kubernetes
Ceph
integrate
with
Kubernetes

3. Ceph Luminous important
Features update

4. Ceph Version Timeline
2017
2016
2015
2014
Firefly
Hammer
Jewel
Luminous

5. Ceph Luminous important Features update
※ Bluestore
※ Erasure Coding Overwrite
※ Multiple MDS
※ Ceph MGR Dashboard

6. Bluestore

7. FileStore vs BlueStore
FileStore BlueStore
Filesystem Device
Journal Direct Write / WAL
Disk Disk, SSD, NVME

8. Erasure Coding

9. Replica vs Erasure Coding
Replica Erasure Coding
space used 3x (n=3) 1.5x (k+m=3)
CPU
overhead
Normal High
support RBD, RGW, CephFS RGW only

10. Replica vs EC Overwrite
Replica EC Overwrite
space used 3x (n=3) 1.50x (k+m=3)
CPU
overhead
Normal High
support ALL ALL

11. Single MDS
Ceph Storage Cluster
MDS.
0
MDS.
1
MDS.
2
MDS.
3
Active
Standby

12. Multiple MDS
Ceph Storage Cluster
MDS.
0
MDS.
1
MDS.
2
MDS.
3
Active
Standby

13. Directory Fragmentation
M M M
M M M
M M M
M M M
M M M
M M M
M M M
M M M
MDS.0 MDS.1MDS.0

14. Ceph MGR Dashboard Architecture
MGRDashboard

15. Ceph Health

16. Pool Usage

17. Host Monitor

18. Logging

19. CephFS Monitor

20. Ceph Monitoring Tools Comparison
Calamari Grafana Nagios OpenATTIC Ceph MGR
Dashboard
Supported Version Hammer ALL ALL ALL Luminous
Cluster Status ✓ ✓ ✓ ✓ ✓
Pool Status ✓ ✓ ✓ ✓ ✓
Pool Mgmt ✓ ✓
OSD Status ✓ ✓ ✓ ✓ ✓
OSD Mgmt ✓
IOPS / Throughput ✓ ✓ ✓
MDS Status ✓ ✓ ✓
CephFS Status ✓
Extra Customized and
Time Range
Email Alert

21. Why Kubernetes and Ceph

22. Why Kubernetes?
 Powerful Community
 Open Source – no vendor lock-in
 Large Scale
 Rolling Upgrade
 Self-healing
 Federation and Hybrid
 Easy to Management
 Easy to integrate with monitoring system
– Heapster, InfluxDB, Grafana

23. Why Ceph ?
 Distributed storage system
 Fault tolerant , no SPoF
 X86 Commodity hardware
 Saving you costs, giving you flexibility
 Large scale – Incremental expansion
 10s to 1000s of nodes
 Unified storage platform
 Scalable object , Block , File system
 Open source – No vendor lock-in
 Automatically balance the file system
 Data security
 with 2 or more copies in different physical store media
23

24. Ceph integrate with
Kubernetes

25. CephFS or CephRBD?
CephFS with K8S Ceph RBD with K8S
Multiple Read Yes Yes
Multiple Write Yes No
Quota No Yes (Image Size)
HA support Yes Yes
Performance Good enough Better
Isolated volume No (shared) Yes

26. Hyper Converged VS Hyper Scale Architecture

27. K8S
master
etcd
kub-ctl
scheduler
K8S
master
etcd
kub-ctl
scheduler
K8S
master
etcd
kub-ctl
scheduler
K8S
node
ceph-mon
ceph-mds
ceph-rgw
K8S
node
ceph-mon
ceph-mds
ceph-rgw
Storage
node
ceph-osd
ceph-osd
ceph-osd
Storage
node
ceph-osd
ceph-osd
ceph-osd
Storage
node
ceph-osd
ceph-osd
ceph-osd
10g
Hyper Scale Architecture

28. Hyper Converged and Containerized Architecture
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
master
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
10g
master master
Admin
node
Auto-
discovery
Auto-
deploy

29. Mixed Architecture
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
master
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
K8S
Ceph-Mon
Ceph-MDS
Ceph-OSD
10g
master master
Admin
node
Auto-
discovery
Auto-
deploy
Storage
node
ceph-osd
ceph-osd
ceph-osd
Storage
node
ceph-osd
ceph-osd
ceph-osd
Storage
node
ceph-osd
ceph-osd
ceph-osd
K8S
node
POD A
POD B
POD C
K8S
node
POD A
POD B
POD C

30. www.inwinstack.com
Thank You!
迎 棧 科 技 股 份 有 限 公 司