1. Cluster
Management
and
Containerization
Benjamin
Hindman,
@benh
Twitter,
Inc.

2. $
whoami
2006
-­‐
2011
2009
-­‐
2010
-­‐

3. cluster management

4. cluster management
(server/IT automation)

5. cluster
management
① configuration/package
management
② deployment
③ naming

6. cluster
management
① configuration/package
management
② deployment
③ naming
④ monitoring

7. cluster
management
① configuration/package
management
② deployment
③ naming
④ monitoring
ops

8. cluster
management
① configuration/package
management
② deployment
③ naming
④ monitoring
ops
developers

9. cluster
management
configuration/package
management
naming
deployment

10. configuration/package
management
“what/how
do
things
get
installed?”
(10’s
of
machines)
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com
$
ssh
host
./configure
&&
make
install

11. configuration/package
management
“what/how
do
things
get
installed?”
(10’s
of
machines)
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com
$
ssh
host
rpm
-­‐ivh
pkg-­‐x.y.z.rpm

12. deployment
“what
should
run
where?”
“how
should
it
be
started/stopped?”
$
ssh
host
nohup
myapp
(10’s
of
machines)
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com

13. deployment
“what
should
run
where?”
“how
should
it
be
started/stopped?”
$
ssh
host
monit
start
myapp
(10’s
of
machines)
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com

14. deployment
“what
should
run
where?”
“how
should
it
be
started/stopped?”
$
scp
myapp
host
$
ssh
host
monit
myapp
(10’s
of
machines)
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com

15. deployment
“what
should
run
where?”
“how
should
it
be
started/stopped?”
(10’s
of
machines)
$
ssh
host
git
pull
&&
monit
myapp
hosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com

16. “how
should
apps
find
each
other?”
naming
webhosts.txt
web1.twttr.com
web2.twttr.com
web3.twttr.com
web4.twttr.com
dbhosts.txt
db1.twttr.com
db2.twttr.com
db3.twttr.com
db4.twttr.com
(10’s
of
machines)

17. (10’s
of
machines)
(100’s
-­‐>
1000’s
of
machines)
to
scale,
need
more
automation

18. Twitter,
circa
2010
webhosts.txt
dbhosts.txt
$
ssh
host
…
(configuration/package
management)
(deployment)

19. MySQL
Cassandra
Rails
Hadoop
memcached

20. challenges

21. challenges
① failures

22. failures

23. MySQL
Cassandra
Rails
Hadoop
memcached

24. MySQL
Cassandra
Rails
Hadoop
memcached

25. types
of
failure:
fault
domains
machine
(disk,
memory,
CPU,
etc)
rack
(switch,
PDU)
datacenter

26. challenges
②
maintenance
(aka
“planned
failures”)

27. maintenance
① upgrading
software
(i.e.,
installing
and
uninstalling
packages)
ops
developers

28. maintenance
① upgrading
software
(i.e.,
installing
and
uninstalling
packages)
② replacing
machines,
switches,
PDUs,
etc

29. MySQL
Cassandra
Rails
Hadoop
memcached

30. MySQL
Cassandra
Rails
Hadoop
memcached

31. challenges
③
utilization

32. Rails
Hadoop
memcached
utilization

33. utilization
Rails
Hadoop
memcached

34. utilization
Rails
Hadoop
memcached
buy
less
machines
or
run
more
applications!

35. challenges
① failures
② maintenance
③ utilization

36. challenges
① failures
② maintenance
③ utilization
planning
for
failure?

37. planning
for
failure

38. challenges
① failures
② maintenance
③ utilization
planning
for
utilization?

39. planning
for
utilization
intra-­‐machine
resource
sharing:
share
a
single
machine’s
resources
between
multiple
applications
(multi-­‐tenancy)
intra-­‐datacenter
resource
sharing:
share
multiple
machine’s
resources
between
multiple
applications

40. Twitter,
circa
2010
what
software
can
help
me!?

41. cluster
management
industry
academia

42. different
software
academia
industry
•
MPI
(Message
Passing
Interface)
•
Apache
(mod_perl,
mod_php)
•
web
services
(Java,
Ruby,
…)

43. different
scale
(at
first)
academia
industry
•
1,000’s
of
machines
•
10’s
of
machines

44. cluster
management
academia
industry
•
PBS
(Portable
Batch
System)
•
TORQUE
•
SGE
(Sun
Grid
Engine)
•
ssh
•
Puppet/Chef
•
Capistrano/Ansible
cluster
managers

45. cluster
manager
provides
a
level-­‐of-­‐indirection
between
hardware
resources
(machines)
and
applications/jobs

46. MySQL
Cassandra
Rails
Hadoop
memcached

47. cluster
manager
Rails
Hadoop
memcached
…
…

48. cluster
management
academia
industry
•
PBS
(Portable
Batch
System)
•
TORQUE
•
SGE
(Sun
Grid
Engine)
•
ssh
•
Puppet/Chef
•
Capistrano/Ansible
batch
computation!

49. Mesos
is
a
modern
general
purpose
cluster
manager
(i.e.,
not
just
focused
on
batch
scheduling)

50. Mesos
service
batch
storage
…
…
streaming
support
many
different
types
of
computation/scheduling

51. Mesos
service
batch
storage
…
streaming
(1)
coordinate
for
resources

52. Mesos
service
batch
storage
…
streaming
(2)
launch
tasks

53. Mesos
service
batch
storage
…
streaming
(3)
launch
tasks

54. Mesos
service
batch
storage
…
streaming

55. Mesos
service
batch
storage
…
streaming

56. Mesos
service
batch
storage
…
streaming
(4)
task
termination

57. Mesos
service
batch
storage
…
streaming
(5)
task
status
update

58. challenges revisited
① failures
② maintenance
③ utilization

59. challenges revisited
① failures
② maintenance
③ utilization

60. Mesos
service
batch
storage
…
streaming

61. Mesos
service
batch
storage
…
streaming

62. Mesos
service
batch
storage
…
streaming

63. Mesos
service
batch
storage
…
streaming
(5)
task
status
update

64. challenges revisited
① failures
② maintenance
③ utilization

65. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers

66. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers

67. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers

68. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers

69. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers

70. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers
(2) multi-­‐tenancy
on
individual
machines

71. Mesos
service
batch
storage
…
streaming
(1) when
resources
become
idle,
can
be
scheduled
and
reused
by
other
schedulers
(2) multi-­‐tenancy
on
individual
machines

72. multi-­‐tenancy
task!
task!
containers
task!

73. containerization
started
leveraging
containerization
technology
in
~2011
2011
LXC
2012
cgroups
2013
Docker
(preliminary)
2014

74. how
Mesos
has
changed
cluster
management
at
Twitter
today

75. configuration/package
management
developers
(1)
bundle
services
as
jar,
tar/gzip
(2)
upload
to
HDFS

76. configuration/package
management
(planning)
developers
(1)
bundle
services
as
jar,
tar/gzip,
and
using
Docker
(2)
upload
to
HDFS
(or
use
registry)

77. deployment
Apache
Aurora
(incubating),
a
scheduler
for
running
stateless
services
written
in
any
language
(but
primarily
used
at
Twitter
for
JVM
services)

78. deployment
(via
Aurora)
developers
(1) describe
service
using
Python
based
DSL
(2)
submit
service
to
Aurora
using
CLI

79. deployment
(via
Marathon)
developers
(1) describe
services
using
JSON
(2)
submit
service
to
Marathon
via
REST

80. naming
Apache
ZooKeeper
using
Apache
ZooKeeper
and
server
sets
(github.com/twitter/commons)

81. naming
(1) task
gets
launched
on
machine
Apache
ZooKeeper
using
Apache
ZooKeeper
and
server
sets
(github.com/twitter/commons)

82. naming
(2)
service
gets
registered
in
a
server
set
in
ZooKeeper
(1) task
gets
launched
on
machine
Apache
ZooKeeper
using
Apache
ZooKeeper
and
server
sets
(github.com/twitter/commons)

83. naming
(2)
service
gets
registered
in
a
server
set
in
ZooKeeper
(1) task
gets
launched
on
machine
(3)
other
services
use
ZooKeeper
to
find
services
they
need
Apache
ZooKeeper
using
Apache
ZooKeeper
and
server
sets
(github.com/twitter/commons)

84. naming
(2)
service
gets
registered
in
a
server
set
in
ZooKeeper
(1) task
gets
launched
on
machine
(3)
other
services
use
ZooKeeper
to
find
services
they
need
(4)
services
connect
directly
with
one
another
Apache
ZooKeeper
using
Apache
ZooKeeper
and
server
sets
(github.com/twitter/commons)

85. naming
alternative
(2)
update
HAProxy
with
new
service
location
(1) task
gets
launched
on
machine
(3)
other
services
send
traffic
through
HAProxy
ZooKeeper/server sets requires injecting code into your clients!

86. where
are
we
today?
ops
developers

87. where
are
we
today?
ops
developers
deploys
decoupled
from
ops
(many
deploys
per
day,
per
service)
maintenance
consists
of
“draining”
hosts,
getting
tasks
rescheduled,
then
pulling
the
cord

88. wait
…
don’t
virtual
machines
solve
my
cluster
management
challenges?

89. wait
…
don’t
virtual
machines
solve
my
cluster
management
challenges?
No.
VMs
are
neither
sufficient
nor
necessary!

90. big
computers
small applications

91. IaaS
public private

94. challenges revisited
① failures
② maintenance
③ utilization

95. challenges revisited
① failures
② maintenance
③ utilization
public
or
private
IaaS,
failures
still
occur
(on
EC2,
instead
of
racks,
have
availability
zones,
instead
of
datacenters,
have
regions)

96. challenges revisited
① failures
② maintenance
③ utilization
provider
wins
with
public
IaaS,
better
resource
sharing
with
private
IaaS,
but
a
static
partition
of
VMs
is
still
a
static
partition!

97. MySQL
Cassandra
Rails
Hadoop
memcached

98. cluster
manager
Rails
Hadoop
memcached
…
…

99. Mesos:
level
of
abstraction
Mesos
build
and
run
using
resources

100. Mesos:
level
of
abstraction
IaaS
Mesos
provision
and
manage
machines
build
and
run
using
resources

101. Mesos
on
IaaS
IaaS
Mesos
use
OpenStack
or
EC2
to
run
Mesos

102. Mesos
on
IaaS/hardware
IaaS
Mesos
hardware
use
OpenStack
or
EC2
or
physical
machines
to
run
Mesos

103. physical machines virtual machines
aggregation
not
virtualization

104. physical machines datacenter computer
aggregation
not
virtualization

105. small
computers
?
big applications

106. "small" computers?

107. power wall
time
complex
single core
simple
many core

108. "big" applications?

109. applications
don’t
fit
on
a
single
computer
anymore

111. "BIG"

113. (1)
lots
of
data
…
(2)
lots
of
users
…
growing
everyday

114. these
applications
need
lots
of
resources
(CPUs,
memory,
I/O)

115. these
applications
need
datacenters

116. the
datacenter
is
the
new
computer

117. desktop
computer
server datacenter
OS
OS
OS
the
datacenter
computer
needs
an
OS

118. operating
system
“a
collection
of
software
that
manages
the
computer
hardware
resources
and
provides
common
services
for
computer
programs”
- Wikipedia

119. datacenter
operating
system
“a
collection
of
software
that
manages
the
datacenter
computer
hardware
resources
and
provides
common
services
for
computer
programs”
- Wikipedia

120. datacenter
operating
system
“a
collection
of
software
that
manages
the
datacenter
computer
hardware
resources
and
provides
common
services
for
computer
programs”
- Wikipedia

121. today
Your App
API
tomorrow
datacenter
OS
provides
common
functionality
every
new
distributed
system
re-­‐
implements:
•
failure
detection
•
package
distribution
•
task
starting
•
resource
isolation
•
resource
monitoring
•
task
killing,
cleanup
•
…

122. today
Your App
API
tomorrow
provides
common
functionality
every
new
distributed
system
re-­‐
implements:
•
failure
detection
•
package
distribution
•
task
starting
•
resource
isolation
•
resource
monitoring
•
task
killing,
cleanup
•
…
datacenter
OS
Don’t
reinvent
the
wheel!

123. case
studies
distributed
in-­‐memory
analytics
framework
distributed
cron
scheduler
(with
dependencies)
github.com/apache/spark
github.com/airbnb/chronos

124. ~Presentation()

125. cluster
management
w/
Docker
+
Mesos
① configuration/package
management
② deployment
③ naming

126. datacenter
OS
IaaS
Mesos
hardware
provide
common
functionality
via
an
API
(kernel)
your
distributed
system

127. Mesos
0.19.0
released
today!
mesos.apache.org
mesos.apache.org/blog
@ApacheMesos