Talk I gave at ComSys2013 (http://www.ipsj.or.jp/sig/os/index.php?ComSys2013) about challenges of running LXC containers at Heroku.

1. LINUX
CONTAINERS
AT SCALE
CHALLENGES IN A VERY DENSE ENVIRONMENT
fabio.kung@gmail.com
fabio@heroku.com

2. こんにちは COMSYS 2013
FABIO KUNG
Technical Lead, Runtime Systems at Heroku

4. heroku scale web=3 worker=2

5. lxc-at-scale.herokuapp.com

6. millions of (web) applications

7. one of the biggest deployments of Linux containers (LXC)

8. 12FACTOR.NET
portable
modern cloud platforms
scale up and down

9. two regions in production: us-east and eu-west

10. many different Availability Zones

14. CHALLENGES

15. MILLIONS OF APPS

16. IDLING

17. STATE MANAGEMENT AND
CONVERGENCE

18. HRK SAEWB3WRE=
EOU CL E= OKR5

19. HRK SAEWB3WRE=
EOU CL E= OKR5

20. ARBITRARY CODE
EXECUTION

21. GIT PUSH HEROKU MASTER
$gtrmt i eoe v
hrk
eou
gthrk.o:yp.i (ec)
i@eoucmmapgt fth
hrk
eou
gthrk.o:yp.i (uh
i@eoucmmapgt ps)
$gtps hrk mse
i uh eou atr
Cutn ojcs 1 dn.
onig bet: , oe
Wiigojcs 10 (/) 11bts|0btss dn.
rtn bet: 0% 11, 8 ye
ye/, oe
Ttl1(et 0,rue 0(et 0
oa
dla ) esd
dla )
--- Rb apdtce
--> uy p eetd
--- CmiigRb
--> opln uy
..
.
T gthrk.o:yp.i
o i@eoucmmapgt
9deb.21a mse - mse
1f0.f5b7 atr > atr

22. CONTAINERS ARE JUST LINUX
$hrk rnbs
eou u ah
Rnig`ah atce t tria..u,rn1
unn bs` tahd o emnl. p u.
~$hsnm
otae
2df7-724d-713599c6
a1bc52-9dbb-0917e4
~$p ax
s uf
UE
SR
PD%P %E
I CU MM
VZ RSTY
S
S T
nbd
ooy
1 00 00 238 15 ?
.
.
06
46
u50
407
2 00 00 146 21 ?
.
.
91
06
u50
407
4 00 00 136 11 ?
.
.
51
12
SA SAT
TT TR
S
0:2
21
S
0:2
21
R
+ 0:2
21
TM CMAD
IE OMN
00 p-u
:0 srn
00 bs
:0 ah
00 p ax
:0 _ s uf
~$l /
s
ap bn dv ec hm lb lb4 ls+on po si tp ur vr
p
i
e
t
oe i
i6
otfud rc bn m
s
a
~$

23. ATTACHED PROCESSES
TTY, attach/detach (re-entry)

24. MIXED WORKLOADS
CPU bound vs I/O bound tasks [1]

25. RESOURCE MANAGEMENT

26. INSTANCE SIZE

27. DENSITY
STATIC VS DYNAMIC

28. LIMITS
CPU, MEMORY, NETWORKING, OS

29. LINUX WASN'T BORN WITH
CONTAINERS

30. COMPARE

31. CPU
GOOD
shares/CFS, hard caps, pin

32. MEMORY
GOOD
limits, overcommit, swap usage

33. TOOLS
BAD/HARD
cgroup/namespace awareness
/ r cvs / g o p
po
cru
Tracing and troubleshooting:
perftools
/rc<I>shd
po/PD/ce
...

34. I/O
BAD/HARD
blkio
filesystem quotas
statistics
throttling

35. NETWORKING
BAD/HARD
isolation
abuse/fraud
limits
statistics
L3 routing, L2 switching

36. USER MANAGEMENT

37. ROOT IN CONTAINERS
apt-get install libmy-dev

38. USER NAMESPACES

39. UNIQUE UIDS
often required by non-cgroup aware tools

40. PID=1 SPECIAL MEANING

41. can not die
signals are filtered
upstart vs systemd vs in-house

42. DISK IMAGES

43. HOW CONTAINERS ARE LAUNCHED
1.
2.
3.
4.
5.
download
extract
make it safe
set up the container filesystem
run

44. DOWNLOAD IS A PROBLEM
1-10GB

45. SOLUTION: SPLIT
base image, root filesystem (stack image): large, shared
application files (slug): smaller, private to each container

46. FORMATS
directory based, tarballs, squashfs, ...
filesystem w/ incremental snapshots:
AUFS, btrfs, zfs, ...
block level: dm, lvm, ...
VM image formats: qcow2, vmdk, vdi, ...

47. STANDARD?

48. https://github.com/containers/container-rfc
We lost the standards game for virtual
machine images, but it feels like this
community is tight nit enough we might be
able to do something for Linux Containers.
-- Alex Polvi (coreos.com)

49. ROUTING

50. containers are constantly moving

51. STATE REPLICATION

54. VIRTUAL NETWORKING

55. 1. Ongaro, Diego, Alan L. Cox and Scortt Rixner. "Scheduling I/O in
Machine Monitors". Rice University. 2 Aug 2012
<http://pages.cs.wisc.edu/~remzi/Classes/838/Spring2013/Pape
vee08.pdf>

56. THANK YOU!
ありがとうございました
fabio.kung@gmail.com
fabio@heroku.com