By Rick Branson of Instagram

1. CASSANDRA & SOLID
STATE DRIVES
Rick Branson, DataStax

2. FACT
CASSANDRA’S STORAGE
ENGINE WAS OPTIMIZED
FOR SPINNING DISKS

3. LSM-TREES

4. WRITE PATH

5. Client Cassandra
On-Disk Node Commit Log
{ cf1: { row1: { col1: abc } } }
{ cf1: { row1: { col2: def } } }
{ cf1: { row1: { col1: <del> } } }
{ cf1: { row2: { col1: xyz } } }
{ cf1: { row1: { col3: foo } } }
insert({ cf1: { row1: { col3: foo } } })
In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”
row1
row2
col1: [del] col2: “def” col3: “foo”
col1: “xyz”
COMMIT

6. SSTableSSTableSSTableSSTableSSTableSSTableSSTable
1
SSTableSSTableSSTableSSTableSSTableSSTableSSTable
2
SSTableSSTableSSTableSSTableSSTableSSTableSSTable
3
SSTableSSTableSSTableSSTableSSTableSSTableSSTable
4
In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”In-Memory Memtable for “cf1”
row1
row2
col1: [del] col2: “def” col3: “foo”
col1: “xyz”
FLUSH

7. SSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTable SSTableSSTableSSTableSSTableSSTable SSTableSSTableSSTableSSTableSSTable
31 2 4
SSTables are merged to maintain read performance
COMPACT
SSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTable

8. SSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTable SSTableSSTableSSTableSSTableSSTable SSTableSSTableSSTableSSTableSSTable
SSTableSSTableSSTableSSTableSSTableSSTableSSTableSSTable
New SSTable is streamed
to disk and old SSTables
are erased
X X X X

9. TAKEAWAYS
• All disk writes are sequential, append-
only operations
• On-disk tables (SSTables) are written in
sorted order, so compaction is linear
complexity O(N)
• SSTables are completely immutable

10. TAKEAWAYS
• All disk writes are sequential, append-
only operations
• On-disk tables (SSTables) are written in
sorted order, so compaction is linear
complexity O(N)
• SSTables are completely immutable
IMPORTANT

11. COMPARED
• Most popular data storage engines
rewrite modified data in-place: MySQL
(InnoDB), PostgreSQL, Oracle,
MongoDB, Membase, BerkeleyDB, etc
• Most perform similar buffering of
writes before flushing to disk
• ... but flushes are RANDOM writes

12. SPINNING DISKS
• Dirt cheap: $0.08/GB
• Seek time limited by time it takes for drive
to rotate: IOPS = RPM/60
• 7,200 RPM = ~120 IOPS
• 15,000 RPM has been the max for decades
• Sequential operations are best: 125MB/
sec for modern SATA drives

13. THAT WAS THE WORLD
IN WHICH CASSANDRA
WAS BUILT

14. 2012: MLC NAND FLASH*
• Affordable: ~$1.75/GB street
• Massive IOPS: 39,500/sec read, 23,000/
sec write
• Latency of less than 100µs
• Good sequential throughput: 270MB/
sec read, 205MB/sec write
• Way cheaper per IOPS: $0.02 vs $1.25
* based on specifications provided by Intel for 300GB Intel 320 drive

15. WITH RANDOM ACCESS
STORAGE, ARE CASSANDRA’S
LSM-TREES OBSOLETE?

17. SOLID STATE HAS
SOME MAJOR BUTS...

18. ... BUT
• Cannot overwrite directly: must erase
first, then write
• Can write in small increments (4KB),
but only erase in ~512KB blocks
• Latency: write is ~100µs, erase is ~2ms
• Limited durability: ~5,000 cycles (MLC)
for each erase block

19. WEAR LEVELING is used
to reduce the number of
total erase operations

20. WEAR LEVELING

21. WEAR LEVELING
Erase Block

22. WEAR LEVELING

23. WEAR LEVELING

24. WEAR LEVELING
Disk Page

25. WEAR LEVELING
Write 1

26. WEAR LEVELING
Write 1
Write 2

27. WEAR LEVELING
Write 1
Write 2
Write 3

28. Write 1
Write 2
Write 3
How is data from only
Write 2 modified?
Remember: the whole block must be erased

29. Mark Garbage

30. Mark Garbage Append
Modified
Data
Empty Block

31. Wait... GARBAGE?

32. THAT MEANS...

33. ... fragmentation,
WHICH MEANS...

34. Garbage Collection!

35. GARBAGE COLLECTION
• Compacts fragmented disk blocks
• Erase operations drag on performance
• Modern SSDs do this in the
background... as much as possible
• If no empty blocks are available, GC
must be done before ANY writes can
complete

36. WRITE AMPLIFICATION
• When only a few kilobytes are written,
but fragmentation causes a whole
block to be rewritten
• The smaller & more random the writes,
the worse this gets
• Modern “mark and sweep” GC reduces
it, but cannot eliminate it

37. Torture test shows massive
write performance drop-off
for heavily fragmented drive
Source: http://www.anandtech.com/show/4712/the-crucial-m4-ssd-update-faster-with-fw0009/6

38. Some poorly designed drives
COMPLETELY fall apart
Source: http://www.anandtech.com/show/5272/ocz-octane-128gb-ssd-review/6

39. Even a well-behaved drive
suffers significantly from the
torture test
Source: http://www.anandtech.com/show/4244/intel-ssd-320-review/11

40. Post-torture, all disk blocks
were marked empty, and the
“fast” comes back...
Source: http://www.anandtech.com/show/4244/intel-ssd-320-review/11

42. “TRIM”
• Filesystems don’t typically immediately
erase data when files are deleted, they just
mark them as deleted and erase later
• TRIM allows the OS to actively tell the drive
when a region of disk is no longer used
• If an entire erase block is marked as
unused, GC is avoided, otherwise TRIM
just hastens the collection process

43. TRIM only reduces the
write amplification effect,
it can’t eliminate it.

44. THEN THERE’S
LIFETIME...

47. AnandTech estimates that modern MLC SSDs
only last about 1.5 years under heavy MySQL load,
which causes around 10x write amplification

48. REMEMBER THIS?

49. TAKEAWAYS
• All disk writes are sequential, append-
only operations
• On-disk tables (SSTables) are written in
sorted order, so compaction is linear
complexity O(N)
• SSTables are completely immutable

50. CASSANDRA
ONLY WRITES
SEQUENTIALLY

51. “For a sequential write workload,
write amplification is equal to 1,
i.e., there is no write
amplification.”
Source: Hu, X.-Y., and R. Haas, “The Fundamental Limitations of Flash Random Write
Performance: Understanding, Analysis, and Performance Modeling”

52. THANK YOU.
~ @rbranson