Our VP, Community Ross Turk's slides from his Structure talk in July 2012.

1. Inktank
Delivering the Future of Storage

2. ME ME ME ME ME ME.
I made a slide today. It’s all about me.
2

3. ME ME ME ME ME ME.
I made a slide today. It’s all about me.
Ross Turk
VP Community, Inktank
ross@inktank.com
@rossturk
inktank.com | ceph.com
3

4. 4

5. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
5

6. Let’s Start With a Good, Old-Fashioned Origin Story
JD Hancock, Flickr / CC BY 2.0 6

7. The Evolution of Storage
A brief history of information storage technology
7

8. Cave Paintings: The Earliest Form (maybe) of Information Storage
Chico.Ferreira, Flickr / CC BY 2.0 8

9. Technology Review: Cave Painting
The good The bad
• Low cost per smudge • Limited storage capacity
• Multitouch • 10 caveman ideas per wall
• No support for CIFS
9

10. HUMAN + = WRITING
==
x1000 x1
10

11. Technology Review: Books and Libraries
The good The bad
• Cost per scroll is high • No automatic replication
• Can be eased w/slave labor • Must complete backups before
Caesar’s invasion of Egypt!
11

12. Books (Strahov, Prague Library)
Moyan_Brenn, Flickr / CC BY-ND 2.0 12

13. Printing Press
FateDenied, Flickr / CC BY 2.0 13

14. magnet + tape = magnetic tape
==
x1000 x1
14

15. IBM System 360 Tape Drives
Erik Pitti, Wikipedia / CC BY-ND 2.0 15

16. HUMAN ROCK
INK
HUMAN
PAPER
HUMAN COMPUTER TAPE
16

17. 11101011 10110110
10110101 10101001
00100100 01001001
== 10100100 10100101
01011010 01101010
10101010 10101010
01010110 01010011
17

18. Tape Is Stupid
Mrs. Gemstone, Flickr / CC BY-SA 2.0 18

19. Computers Need Programmers (and Operators)
USDAgov, Flickr / CC BY 2.0 19

20. HUMAN COMPUTER TAPE
20

21. Throughput Becomes Important
rfduck, Flickr / CC BY-ND 2.0 21

22. Hard Drive
Jeff Kubina, Flickr / CC-BY-SA 2.0 22

23. Hard Drives Are Totally Better
amazing spinny hard drives sucky stupid tape
23

24. aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 011 010 000 dc
000 110 001
24

25. file
owner: rturk
created: aug12
last viewed: aug17
size: 42025
11101011 10110110 10110101 perms: 644
10101001 00100100 01001001
10100100 10100101 01011010
01101010 10101010 10101010
25

26. aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 01 010 000 dc
10
000 110 001
26

27. Humanity Outgrows the Hard Drive
Mr. T in DC, Flickr / CC BY 2.0 27

28. DISK
DISK
DISK
COMPUTE DISK
HUMAN
R
DISK
DISK
DISK
28

29. What Happens When Two HUMANs Need Access to the Same Resource?
wFourier, Flickr / CC BY 2.0 29

30. DISK
DISK
HUMAN
DISK
COMPUTE DISK
HUMAN
R
DISK
HUMAN
DISK
DISK
30

31. HUMAN HUMAN
HUMAN
HUMAN DISK
HUMAN
HUMAN DISK
HUMAN
HUMAN DISK
DISK
HUMAN
DISK
HUMAN
HUMAN DISK
(COMPUTER)
HUMAN
DISK
HUMAN HUMAN
DISK
HUMAN
HUMAN DISK
HUMAN DISK
HUMAN DISK
HUMAN
HUMAN DISK
HUMAN
HUMAN
(actually more like this…)
31

32. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
32

33. object
texture: crunch
flavor: smoke, salt
nutrition: none
color: bacon
11101011 10110110 10110101
10101001 00100100 01001001
10100100 10100101 01011010
01101010 10101010 10101010
33

34. X
aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 011 010 000 dc
000 110 001
34

35. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
APP R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
35

36. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
COMPUTE R
DISK
COMPUTE
R R
DISK
DISK COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
36

37. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
VM COMPUTE
DISK
R
COMPUTE
DISK
R
VM COMPUTE
DISK
R
COMPUTE
DISK
VM R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
37

38. The Current State of Storage
How people store information today, and why it’s still not perfect yet
38

39. Ceph
Cloud computing
Distributed storage
Shared storage
Computers
Writing
Painting
How Much Store Things All Human History!!
Time-scale: Roughly logarithmic. Content: Whatever the opposite of “scientific” is. 39

40. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
40

41. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
41

42. C D
C D
C D
C D
C D
C D
C D
C D
C D
C D
C D
C D
42

43. C D
C D
C D
HUMAN
C D
C D
C D
HUMAN C D
C D
C D
HUMAN C D
C D
C D
43

44. Storage Hardware
Michael Moll, Wikipedia / CC BY-SA 2.0 44

45. 6.4 Million Square Feet of Expensive Factory Buildings
Dude94111, Flickr / CC BY 2.0 45

46. Storage Hardware Vendors Have Bills to Pay
CarbonNYC, Flickr / CC BY 2.0 46

47. …Which Means That Customers Do Too
401K 2012, Flickr / CC BY-SA 2.0 47

48. Technology Is Becoming a Commodity
RaeAllen, Flickr / CC-BY 2.0 48

49. Commodity Prices Fluctuate
May-07 May-08 May-09 May-10 May-11 May-12
49

50. Growing With Hardware Appliances
First PB Second PB
C D • Proprietary C D • Proprietary storage
C D
storage hardware C D
hardware
C D • Well-known C D • Same storage
C D
storage vendor C D
vendor
C D C D
C D $14 b’zillion C D Another
C D C D
$14 b’zillion
C D C D
C D C D
C D C D
C D C D
C D C D
50

51. Dedicated Hardware Appliances Are OLD TECHNOLOGY
Paul Keller, Flickr / CC BY 2.0 51

52. Source: http://www.cpubenchmark.net/high_end_cpus.html
52

53. FLAGSHIP
PRODUCT
53

54. “I'm sick of paying for hardware with a three-year-old proc in it!”
Mel B., Flickr / CC BY 2.0 54

55. Hardware Appliances are Mysterious Black Boxes
Abode of Chaos, Flickr / CC BY 2.0 55

56. C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
56

57. X
C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
57

58. C D
C D
C D
C D
C D
HUMAN !! C D
[DEVELOPER] C D
C D
C D
C D
C D
C D
58

59. Give More Money To The Big Proprietary Vendors
It will make them very, very happy. 59

60. Storage Should Be Better
People need storage solutions that…
• …are open
• …are easy to manage
• …satisfy their requirements
• performance
• functional
• financial
60

61. The Birth of a New Storage Solution
We think our roots are showing
61

62. DreamHost
62

63. Sage Weil
Co-founder of
DreamHost
Inventor of Ceph
CEO of Inktank
63

64. DreamHost
DreamHost is staffed by extraordinarily hip people 64

65. +
65

66. New Monthly Code Commits
700
600
500
400
300
200
100
0
2004-06 2005-07 2006-07 2007-07 2008-07 2009-07 2010-07 2011-07
66

67. Ceph Starts Popping Up
67

68. philosophy design
OPEN
SOURCE
68

69. Open Source is the Best Way to Spread Ideas
orchidgalore, Flickr / CC BY 2.0 69

70. philosophy design
OPEN
SOURCE
COMMUNITY-
FOCUSED
70

71. All of Us Are Smarter Than Some of Us
rturk, Linkedin Inmap 71

72. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY-
FOCUSED
72

73. Ceph
Too much for a room
Too much for a computer
Too much for a drive
Too much for a book
Too much for a cave
Ceph is Built to Scale
Time-scale: Roughly logarithmic. Content: Whatever the opposite of “scientific” is. 73

74. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE
74

75. Ariolimax Californicus
aroid, Flickr / CC BY 2.0 75

76. single point
of failure
replicated replicated
The Octopus (A Metaphor)
I love speaking in metaphors. 76

77. The Beehive (A Better Metaphor)
blumenbiene, Flickr / CC BY 2.0 77

78. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE
SOFTWARE
BASED
78

79. C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
79

80. C D
C D
✔
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
80

81. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE
SOFTWARE
BASED
SELF-
MANAGING
81

82. Hard Drives Are Tiny Record Players and They Fail Often
jon_a_ross, Flickr / CC BY 2.0 82

83. D D
D D
D D =
D D
x 1 MILLION
55 times / day
83

84. Enter: Ceph
An architectural and functional overview of the Ceph system
84

85. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
85

86. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
86

87. OSD OSD OSD OSD OSD
btrfs
FS FS FS FS FS
xfs
ext4
DISK DISK DISK DISK DISK
M M M
87

88. HUMAN
M
M M
88

89. Monitors:
M
• Maintain cluster map
• Provide consensus for
distributed decision-making
• Must have an odd number
• These do not serve stored
objects to clients
OSDs:
• One per disk (recommended)
• At least three in a cluster
• Serve stored objects to
clients
• Intelligently peer to perform
replication tasks
• Supports object classes
89

90. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
90

91. APP
LIBRADOS
socket
M
M M
91

92. LIBRADOS
L
• Provides direct access to
RADOS for applications
• C, C++, Python, PHP, Java
• No HTTP overhead

93. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
93

94. APP
REST
RADOSGW
LIBRADOS
socket
M
M M
94

95. RADOS Gateway:
• REST-based interface to
RADOS
• Supports buckets,
accounting
• Compatible with S3 and
Swift applications
95

96. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
96

97. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M
97

98. CONTAINER VM CONTAINER
LIBRBD LIBRBD
LIBRADOS LIBRADOS
M
M M
98

99. HOST
KRBD (KERNEL MODULE)
LIBRADOS
M
M M
99

100. RADOS Block Device:
• Storage of virtual disks in
RADOS
• Allows decoupling of VMs and
containers
• Live migration!
• Images are striped across the
cluster
• Boot support in QEMU, KVM,
and OpenStack Nova
• Mount support in the Linux
kernel
100

101. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
101

102. CLIENT
metadata 01 data
10
M
M M
102

103. Metadata Server
• Manages metadata for a
POSIX-compliant shared
filesystem
• Directory hierarchy
• File metadata (owner,
timestamps, mode, etc.)
• Stores metadata in RADOS
• Does not serve file data to
clients
• Only required for shared
filesystem
103

104. What Makes Ceph Unique?
Part one: CRUSH
104

105. C D
C D
C D
C D
C D
??
APP C D
C D
C D
C D
C D
C D
C D
105

106. C D
C D
C D
C D
C D
APP C D
C D
C D
C D
C D
C D
C D
106

107. How Long Did It Take You To Find Your Keys This Morning?
azmeen, Flickr / CC BY 2.0 107

108. Dear Diary: Today I Put My Keys on the Kitchen Counter
Barnaby, Flickr / CC BY 2.0 108

109. C D
C D A-G
C D
C D
C D H-N
APP F* C D
C D
C D O-T
C D
C D
C D U-Z
C D
109

110. I Always Put My Keys on the Hook By the Door
vitamindave, Flickr / CC BY 2.0 110

111. HOW DO YOU
FIND YOUR KEYS
WHEN YOUR HOUSE
IS
INFINITELY BIG
AND
ALWAYS CHANGING?
111

112. The Answer: CRUSH!!!!!
pasukaru76, Flickr / CC SA 2.0 112

113. 10 10 01 01 10 10 01 11 01 10
hash(object name) % num pg
10 10 01 01 10 10 01 11 01 10
CRUSH(pg, cluster state, rule set)
113

114. 10 10 01 01 10 10 01 11 01 10
10 10 01 01 10 10 01 11 01 10
114

115. CRUSH
• Pseudo-random placement
algorithm
• Ensures even distribution
• Repeatable, deterministic
• Rule-based configuration
• Replica count
• Infrastructure topology
• Weighting
115

116. CLIENT
??
116

117. 117

118. 118

119. CLIENT
??
119

120. What Makes Ceph Unique
Part two: thin provisioning
120

121. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M
121

122. HOW DO YOU
SPIN UP
THOUSANDS OF VMs
INSTANTLY
AND
EFFICIENTLY?
122

123. instant copy
144 0 0 0 0 = 144
123

124. write
CLIENT
write
write
write
144 4 = 148
124

125. read
read
CLIENT
read
144 4 = 148
125

126. What Makes Ceph Unique?
Part three: clustered metadata
126

127. Metadata for a POSIX-Compliant Filesystem
Barnaby, Flickr / CC BY 2.0 127

128. CLIENT
01
10
M
M M
128

129. M
M M
129

130. one tree
three metadata servers
??
130

131. 131

132. 132

133. 133

134. 134

135. DYNAMIC SUBTREE PARTITIONING
135

136. And Now: Backpedaling
136

137. ALMOST
EVERYTHING
WORKS
137

138. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP AWESOME AWESOME
NEARLY
AWESOME AWESOME
RADOS AWESOME
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
138

139. *
LAN SCALE!!
* OR REALLY REALLY SCARY FAST WAN
139

140. What is Inktank?
I really like your polo shirt, please tell me what it means!
140

141. 141

142. Who?
• Ceph’s inventor and (most) developers
142

143. Why?
• To ensure the long-term success of Ceph
• To help companies adopt Ceph through
services, support, training, and consulting
143

144. When?
• Founded: December 28, 2011
• Brand Launched: April 2012
144

145. What do we want from you??
• Try Ceph! Tell us what you think. Ask if
you need help. Help others if you can!
• Are you a company? Consider dedicating
dev resources to the project.
145

146. Questions?
Ross Turk
VP Community, Inktank
ross@inktank.com
@rossturk
inktank.com | ceph.com
146