Our VP, Community Ross Turk's slides from his Virtual Cloud Day talk in Nov 2012.

1. S C A L I N G
S T O R A G E
W I T H
C E P H
Ross
Turk,
Inktank

2. WHO?
Ross Turk
VP Community, Inktank
§ ross@inktank.com
§ @rossturk
inktank.com | ceph.com

5. me

6. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

7. I N
T H E
B E G I N N I N G
Magic Madzik, Flickr / CC BY 2.0

8. E A R L Y
I N F O R M A T I O N
S T O R A G E
Chico.Ferreira, Flickr / CC BY 2.0

9. W R I T I N G
>
C A V E
P A I N T I N G S
kevingessner, Flickr / CC BY-SA 2.0

10. ==
x1000 x1

11. P E O P L E
B E G I N
W R I T I N G
A
L O T
Moyan_Brenn, Flickr / CC BY-ND 2.0

12. W R I T I N G
I S
T I M E -­‐ C O N S U M I N G
trekkyandy, Flickr / CC BY 2.0

13. T H E
I N D U S T R I A L I Z A T I O N
O F
W R I T I N G
FateDenied, Flickr / CC BY 2.0

14. magnet + tape = magnetic tape
==
x1000 x1

15. S T O R A G E
B E C O M E S
M E C H A N I C A L
Erik Pitti, Wikipedia / CC BY-ND 2.0

16. HUMAN ROCK
INK
HUMAN
PAPER
HUMAN COMPUTER TAPE

17. C O M P U T E R S
N E E D
P E O P L E
T O
W O R K
USDAgov, Flickr / CC BY 2.0

18. HUMAN COMPUTER TAPE

19. 11101011 10110110
10110101 10101001
00100100 01001001
== 10100100 10100101
01011010 01101010
10101010 10101010
01010110 01010011

20. T H R O U G H P U T
B E C O M E S
I M P O R T A N T
Zane Luke, Flickr / CC BY-ND 2.0

21. L A Z 0 R
B 3 A M S
C H A N G E
E V E R Y T H I N G ! !
Jeff Kubina, Flickr / CC-BY-SA 2.0

22. H A R D
D R I V E S
A R E
T O T A L L Y
B E T T E R
amazing spinny hard drives sucky stupid tape
slow

23. E V E R Y T H I N G
G E T S
M E S S Y
Rob!, Flickr / CC BY 2.0

24. aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 011 010 000 dc
000 110 001

25. file
owner: rturk
created: aug12
last viewed: aug17
size: 42025
11101011 10110110 10110101 perms: 644
10101001 00100100 01001001
10100100 10100101 01011010
01101010 10101010 10101010

26. aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 01 010 000 dc
10
000 110 001

27. W E
O U T G R O W
T H E
H A R D
D R I V E
Mr. T in DC, Flickr / CC BY 2.0

28. DISK
DISK
HUMAN
DISK
COMPUTE DISK
HUMAN
R
DISK
HUMAN
DISK
DISK

29. HUMAN HUMAN
HUMAN
HUMAN DISK
HUMAN
HUMAN DISK
HUMAN
HUMAN DISK
DISK
HUMAN
DISK
HUMAN
HUMAN DISK
(COMPUTER)
HUMAN
DISK
HUMAN HUMAN
DISK
HUMAN
HUMAN DISK
HUMAN DISK
HUMAN DISK
HUMAN
HUMAN DISK
HUMAN
HUMAN
(actually more like this…)

30. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

31. X
aa
ab 111010 ac
101 ba bb bc 111 010
da 110 db 011 010 000 dc
000 110 001

32. object
pace: quick
driver: frog
license: expired
expression: agog
11101011 10110110 10110101
10101001 00100100 01001001
10100100 10100101 01011010
01101010 10101010 10101010

33. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
APP R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

34. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
COMPUTE R
DISK
COMPUTE
R R
DISK
DISK COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

35. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
VM COMPUTE
DISK
R
COMPUTE
DISK
R
VM COMPUTE
DISK
R
COMPUTE
DISK
VM R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

36. Ceph
Cloud computing
Distributed storage
Shared storage
Computers
Writing
Painting
S T O R A G E
T H R O U G H O U T
H I S T O R Y
Time-scale: Roughly logarithmic. Content: Whatever the opposite of “scientific” is.

37. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
HUMAN R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

38. COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R
COMPUTE
DISK
R

39. C D
C D
C D
C D
C D
C D
C D
C D
C D
C D
C D
C D

40. C D
C D
C D
HUMAN
C D
C D
C D
HUMAN C D
C D
C D
HUMAN C D
C D
C D

41. S T O R A G E
A P P L I A N C E S
Michael Moll, Wikipedia / CC BY-SA 2.0

42. 6 . 4
M I L L I O N
S Q F T
O F
F A C T O R I E S
Dude94111, Flickr / CC BY 2.0

43. T E C H N O L O G Y
I S
A
C O M M O D I T Y
RaeAllen, Flickr / CC-BY 2.0

44. C O M M O D I T Y
P R I C E S
F L U C T U A T E
May-07 May-08 May-09 May-10 May-11 May-12

45. Hardware Appliances are Mysterious Black Boxes
Abode of Chaos, Flickr / CC BY 2.0

46. C D
C D
C D
C D
C D
HUMAN !! C D
[DEVELOPER] C D
C D
C D
C D
C D
C D

47. C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D

48. X
C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D

49. THE WORLD
NEEDS
AN OPEN STORAGE
TECHNOLOGY
THAT
SCALES

50. S A G E
W E I L
§ Co-founder of DreamHost
§ Inventor of Ceph
§ CEO of Inktank

51. philosophy design
OPEN
SOURCE

52. O P E N
S O U R C E
S P R E A D S
I D E A S
orchidgalore, Flickr / CC BY 2.0

53. philosophy design
OPEN
SOURCE
COMMUNITY-
FOCUSED

54. W E
A R E
S M A R T E R
T O G E T H E R
rturk, Linkedin Inmap

55. C E P H
B E L O N G S
T O
A L L
O F
U S
wackybadger, Flickr / CC BY 2.0

56. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY-
FOCUSED

57. Ceph
Too much for a room
Too much for a computer
Too much for a drive
Too much for a book
Too much for a cave
C E P H
I S
B U I L T
T O
S C A L E

58. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE

59. A R I L O M A X
C A L I F O R N I C U S
aroid, Flickr / CC BY 2.0

60. single point
of failure
highly-available
replicated
T H E
O C T O P U S
( A
M E T A P H O R )
I love speaking in metaphors.

61. T H E
B E E H I V E
( A N O T H E R
M E T A P H O R )
blumenbiene, Flickr / CC BY 2.0

62. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE
SOFTWARE
BASED

63. C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D

64. C D
C D
✔
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D

65. philosophy design
OPEN SCALABL
SOURCE E
COMMUNITY- NO SINGLE POINT OF
FOCUSED FAILURE
SOFTWARE
BASED
SELF-
MANAGING

66. D I S K S
=
J U S T
T I N Y
R E C O R D
P L A Y E R S
jon_a_ross, Flickr / CC BY 2.0

67. D D
D D
D D =
D D
x 1 MILLION
55 times / day

69. I T
A L L
S T A R T E D
W I T H
A
D R E A M

70. +

71. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

72. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

73. OSD OSD OSD OSD OSD
btrfs
FS FS FS FS FS
xfs
ext4
DISK DISK DISK DISK DISK
M M M

74. HUMAN
M
M M

75. Monitors:
§ Maintain cluster map
M § Provide consensus for
distributed decision-
making
§ Must have an odd number
§ These do not serve stored
objects to clients
OSDs:
§ One per disk
(recommended)
§ At least three in a cluster
§ Serve stored objects to
clients
§ Intelligently peer to perform
replication tasks
§ Supports object classes

76. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

77. APP
LIBRADOS
native
M
M M

78. LIBRADOS
L
§ Provides direct access to
RADOS for applications
§ C, C++, Python, PHP,
Java
§ No HTTP overhead

79. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

80. APP APP
REST
RADOSGW RADOSGW
LIBRADOS LIBRADOS
native
M
M M

81. RADOS Gateway:
§ REST-based interface to
RADOS
§ Supports buckets,
accounting
§ Compatible with S3 and
Swift applications

82. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

83. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M

84. CONTAINER VM CONTAINER
LIBRBD LIBRBD
LIBRADOS LIBRADOS
M
M M

85. HOST
KRBD (KERNEL MODULE)
LIBRADOS
M
M M

86. RADOS Block Device:
§ Storage of virtual disks in
RADOS
§ Allows decoupling of VMs
and containers
§ Live migration!
§ Images are striped across
the cluster
§ Boot support in QEMU,
KVM, and OpenStack Nova
§ Mount support in the Linux
kernel

87. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

88. CLIENT
metadata 01 data
10
M
M M

89. Metadata Server
§ Manages metadata for a
POSIX-compliant shared
filesystem
§ Directory hierarchy
§ File metadata (owner,
timestamps, mode, etc.)
§ Stores metadata in RADOS
§ Does not serve file data to
clients
§ Only required for shared
filesystem

90. WHAT MAKES
CEPH
UNIQUE?

91. H O W
D O
Y O U
F I N D
Y O U R
K E Y S ?
azmeen, Flickr / CC BY 2.0

92. C D
C D
C D
C D
C D
??
APP C D
C D
C D
C D
C D
C D
C D

93. C D
C D A-G
C D
C D
C D H-N
APP F* C D
C D
C D O-T
C D
C D
C D U-Z
C D

94. I
A L W A Y S
P U T
M Y
K E Y S
O N
T H E
H O O K
vitamindave, Flickr / CC BY 2.0

95. C D
C D
C D
C D
C D
APP C D
C D
C D
C D
C D
C D
C D

96. D E A R
D I A R Y :
K E Y S
=
I N
T H E
K I T C H E N
Barnaby, Flickr / CC BY 2.0

97. HOW DO YOU
FIND YOUR KEYS
WHEN YOUR HOUSE
IS
INFINITELY BIG
AND
ALWAYS CHANGING?

98. T H E
A N S W E R :
C R U S H ! !
pasukaru76, Flickr / CC SA 2.0

99. 10 10 01 01 10 10 01 11 01 10
hash(object name) % num pg
10 10 01 01 10 10 01 11 01 10
CRUSH(pg, cluster state, rule set)

100. 10 10 01 01 10 10 01 11 01 10
10 10 01 01 10 10 01 11 01 10

101. CRUSH
§ Pseudo-random placement
algorithm
§ Ensures even distribution
§ Repeatable, deterministic
§ Rule-based configuration
§ Replica count
§ Infrastructure topology
§ Weighting

102. CLIENT
??

105. CLIENT
??

106. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M

107. HOW DO YOU
SPIN UP
THOUSANDS OF VMs
INSTANTLY
AND
EFFICIENTLY?

108. instant copy
144 0 0 0 0 = 144

109. write
CLIENT
write
write
write
144 4 = 148

110. read
read
CLIENT
read
144 4 = 148

111. HOW DO YOU
MANAGE
DIRECTORY HEIRARCHY
WITHOUT
A
SINGLE POINT OF
FAILURE?

112. F I L E S Y S T E M S
R E Q U I R E
M E T A D A T A
Barnaby, Flickr / CC BY 2.0

113. CLIENT
01
10
M
M M

114. M
M M

115. one tree
three metadata servers
??

120. DYNAMIC SUBTREE PARTITIONING

121. AND NOW
BACKPEDALING

122. ALMOST
EVERYTHING
WORKS

123. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP AWESOME AWESOME
NEARLY
AWESOME AWESOME
RADOS AWESOME
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes

124. *
LAN SCALE!!
* OR REALLY REALLY SCARY FAST WAN

125. C E P H
A N D
C L O U D S T A C K
tableatny, Flickr / CC BY 2.0

126. R B D
S U P P O R T
I N
C L O U D S T A C K
§ Allows storage of virtual disks inside RADOS
§ Works with KVM only right now
§ No snapshots yet
§ Upcoming in CloudStack 4
§ More information can be found on the mailing list:
§ ceph-devel / incubator-cloudstack-dev:
http://article.gmane.org/gmane.comp.file-systems.ceph.devel/7505

127. QUESTIONS?
Ross Turk
VP Community, Inktank
§ ross@inktank.com
§ @rossturk
inktank.com | ceph.com