This document discusses distributed storage technologies and challenges. It covers persistent memory, distributed storage case studies like GlusterFS and Ceph, and challenges such as network latency. It also addresses how distributed storage systems scale performance and capacity through techniques like horizontal scaling across commodity nodes and maintaining a single namespace.

3. ● Technologies
○ Persistent memory, aka storage class memory (SCM)
○ Distributed storage
● Case studies
○ GlusterFS, Ceph
● Challenges
○ Network latency
○ Accelerating parts of the system with SCM
○ CPU latency

4. ● Near DRAM speeds
● Wearability better than SSDs (claims Intel)
● API available
○ Crash-proof transactions
○ Byte or block addressable
● Likely to be at least as expensive as SSDs
● Fast random access
● Has support in Linux
What do we know / expect?

5. ● Single server scales poorly
○ Horizontal scaling expensive
● Multiple servers in distributed storage scale well
○ Maintain single namespace
● Commodity nodes
○ Easy expansion by adding nodes
○ Good fit for low cost hardware
○ Minimal impact on node failure
How to scale performance and capacity?

6. ●
●
●
○

7. ●
●
○
○

9. ●
○
○
●
○
○
○
○

11. ● “Primary copy” : update replicas in parallel,
○ processes reads and writes
○ Ceph’s choice, also Gluster’s “journal based replication” (under development)
● Other design options
○ Read at “tail” - the data there is always committed
Latency cost to replicate across nodes
client
Primary
server
Replica 1
Replica 2

12. ●
●
●
client
Replica 1
Replica 1
Replica 2

13. ●
○
○
●
○
○
○
○
○

14. ●
●
○
○
●
○
○
○
○

15. Biggest gain with reads, little gain for small I/O.
Sequential I/O (1024 block size)
Random I/O (1024 block size)
1024 bytes transfers

16. ●
●
○
●
○

18. DM-cache Ceph tiering

19. ●
●
●
OSD
bluestore
RocksDB
WAL
OSD
filestore
XFS
Journal
Journal

20. ●
○
○
○
○
●
○
○

22. ●
●
○
○
●
●
○

23. ●
○
○
○
○
d1
d2
d3
f1

24. ●
○
○
○
○
●
○
●

25. d2 d3 f1
S1 S2 S3 S4
d1/d2/d3/f1
Four LOOKUPs
Four servers
16 LOOKUPs total in worse case
d2 VFS layer
Gluster client
Gluster server
Client

26. ●
○
●
○
○
Red is
cached

28. ●
○
○
○
○

29. ●
●
●
●
●
session_dispatch _lock
PG::map_lock
pg_map_lock
OpWQ lock

30. ●
○
○
○
○
●
○
○
○

31. ●
●
○
○
○
●
○
○

32. ●
●
○

35. ●
○
○
○
○
○
●
B

37. ●
○
○
○
●
○
○
●
○
○