Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Ceph - A distributed storage system

1.115 Aufrufe

Veröffentlicht am

Presentation held at Locaweb Tech Talks

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Ceph - A distributed storage system

  1. 1. A distributed storage system
  2. 2. whoami ● Italo Santos ● @ Locaweb since 2007 ● Sysadmin @ Storage Team
  3. 3. Introduction ● Single Storage System ● Scalable ● Reliable ● Self-healing ● Fault Tolerant ● NO single point of failure
  4. 4. Architecture
  5. 5. RADOS A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
  6. 6. Ceph Storage Cluster
  7. 7. OSDs M Monitors MDS
  8. 8. OSDs
  9. 9. OSDs ● One per disk ● Store data ● Replication ● Recovery ● Backfilling ● Rebalancing ● OSDs heartbeat
  10. 10. DISK FS DISK DISK OSD DISK DISK OSD OSD OSD OSD FS FS FSFS btrfs xfs ext4 MMM
  11. 11. M Ceph Monitors
  12. 12. Monitors ● Cluster map ● Monitors map ● OSDs map ● Placement Group map ● CRUSH map
  13. 13. Metadata Server (MDS)
  14. 14. MDS ● Used only by CephFS ● POSIX-compliant shared filesystem ● Manage metadata ○ Directory hierarchy ○ File metadata ● Stores metadata on RADOS
  15. 15. CRUSH
  16. 16. CRUSH ● Pseudo-random placement algorithm ○ Fast calculation ○ Deterministic ● Statistically uniform distribution ● Limited data migration on change ● Rule-based configuration
  17. 17. 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 10 CRUSH(pg, cluster state, rule set) hash(object name) % num pg
  18. 18. CLIENT ??
  19. 19. Placement Groups (PGs)
  20. 20. Placement Groups ● Logical collection of objects ● Maps PGs to OSDs dynamically ● Computationally less expensive ○ Reduce number of process ○ Less of per-object metadata ● Dynamically rebalance
  21. 21. Placement Groups
  22. 22. Placement Groups ● Increase PGs reduces per-osd load ● ~100 PGs per OSD (i.e., OSD per object = Number of replicas) ● Defined on pool creation ● PGs with multiple pools ○ Balance PGs per pool with PGs per OSD
  23. 23. Pools
  24. 24. Pools ● Replicated ○ Object replicated N times (i.e., default size = 3) ○ Object + 2 protection replicas ● Erasure Coded ○ Stores objects as K+M chunks (i.e., size = K+M) ○ Divided into K data chunks and M coding chunks
  25. 25. Ceph Clients
  26. 26. RADOS A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
  27. 27. RadosGW Ceph Object Gateway
  28. 28. RadosGW ● Object Storage Interface ● Apache + FastCGI ● S3-compatible ● Swift-compatible ● Common namespace ● Store data on Ceph cluster
  29. 29. RBD Rados Block Device
  30. 30. RBD ● Block device interface ● Data striped on ceph cluster ● Thin-provisioned ● Snapshot support ● Linux Kernel-based (librbd) ● Cloud native support
  31. 31. CephFS Ceph File System
  32. 32. CephFS ● POSIX-compliant filesystem ● Shared filesystem ● Directory hierarchy ● File metadata (owner, timestamps, mode, etc.) ● Ceph MDS required ● NOT production ready!
  33. 33. Thanks Italo Santos @ Storage Team

×