A year of problems running JVMs inside Docker containers.

1. Docker and the JVM,
a good idea?
Christopher Batey
@chbatey

2. @chbatey
Overview
● Motivation
● How they work
● What you need to know
○ Tools
○ Problems by example
■ Thread per request (Dropwizard/Jetty)
■ Thread per core (Ratpack/Netty)

3. @chbatey

4. @chbatey

5. @chbatey
rat-pack
dropwizard
wiremock
● 12 cores w/ hyper-threading
● 64GB RAM
Apache Benchmark

6. @chbatey
Motivation
5

7. @chbatey
Deployment models
Host
App

8. @chbatey
So why?

9. @chbatey
Multiple JVMs on the same machine?

10. @chbatey
JVM ergonomics
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.html

11. @chbatey
What
9

12. @chbatey
Docker
● Namespaces
○ Process
○ Mounts
○ Networking
● CGroups
○ CPU
○ Memory

13. Tools & Examples
12.5

14. @chbatey
Tools
● Ps, Htop and Top
● Systemd-cg{ls, top}
● Dstat
● Cadvisor
● Heapster
● Jcmd
● dmesg

15. @chbatey
Images

16. @chbatey
Images

17. @chbatey
PID namespace

18. @chbatey
Tool: Htop

19. @chbatey
Tool: Top

20. @chbatey
Tool: systemd-cgls

21. @chbatey
Wtf?

22. @chbatey
Control Groups (cgroups)
● CPU - shares
● Memory - limit
https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.
html
https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-
memory.html

23. @chbatey
Memory
20

24. @chbatey
Limiting memory

25. Free inside a container
/proc/* is not control group aware

26. @chbatey
Limiting memory

27. @chbatey
Tool: Dmesg

28. @chbatey
More memory

29. @chbatey
Survives o/

30. @chbatey
Survives o/ but is slow :(

31. @chbatey
Tool: systemd-cgtop

32. @chbatey
Tool: DStat

33. @chbatey
Tool: systemd-cgtop

34. @chbatey
Tool: Htop

35. @chbatey
Disabling swap

36. @chbatey
Disabling swap

37. @chbatey
Sizing the memory for a JVM
● Heap
● Threads
● Metaspace
● Code
● GC
● Native (direct ByteBuffers and sun.misc.Unsafe)
30

38. @chbatey
Native Memory Tracking
-XX:NativeMemoryTracking=summary

39. @chbatey
Tool: jcmd
● JCmd allows us to see a JVMs memory
usage

40. @chbatey
Tool: jcmd
● Jcmd VM.native_memory
● 100+ Threads?

41. @chbatey
Tool: Jcmd
Jcmd Thread.print

42. @chbatey
It isn’t just one thread pool in Jetty!
jcmd 26 Thread.print | grep "dw" | wc -l
65
jcmd 26 Thread.print | grep "selector" | wc -l
24
jcmd 26 Thread.print | grep "acceptor" | wc -l
12

43. @chbatey
Wtf?

44. @chbatey
Overriding

45. @chbatey
Solution?

46. @chbatey
Solution?

47. @chbatey
Ratpack threads
jcmd 26 Thread.print | grep "ratpack-compute" | wc -l
24

48. @chbatey
It isn’t just your web framework
jcmd 26 Thread.print | grep G1 | wc -l
15
jcmd 26 Thread.print | grep "C1|C2" | wc -l
4

49. @chbatey
It isn’t just your web framework

50. @chbatey
CPU
40

51. @chbatey

52. @chbatey

53. @chbatey

54. @chbatey

55. @chbatey
Limiting CPU - shares

56. @chbatey

57. @chbatey
Tool: systemd-cgtop

58. @chbatey
Understanding CPU shares
1024 512 512
CPU 0 CPU 1

59. @chbatey
Understanding CPU shares
1024 512 512

60. CPU quota
Cpu_period = 100000
One cores = 100000
Two cores = 200000
etc

61. @chbatey
That is more like it! Or is it?

62. @chbatey

63. @chbatey
JVM inside containers
● Get an appropriate number of threads
● Size memory and test

64. @chbatey
Thanks for listening
Github: https://github.com/chbatey/docker-jvm-talk
Twitter: @chbatey
Want to work (in London) with Docker, Rkt, Kubernetes, Go, Java, Python,
Cassandra, MySQL? christopher.batey@gmail.com

65. @chbatey
Why I care
● Accidently become a Platform Engineer / Site Reliability Engineer
○ Team of three that builds an internal PaaS for 100s of applications
● Tech:
○ AWS
○ Docker
○ Kubernetes (custom build)
○ Cassandra
○ MySQL
○ Prometheus