It is our experience that 60-70% of all the applications we analyze incur a performance penalty due to how it consumes and/or retains memory. Yet if you ask people what is their biggest performance headache it’s unlikely that they’d recognize memory as being #1. I have been involved in analyzing numerous applications for just these kinds of issues and in this session, I will cover the tell tale signs and symptoms that your application is one of those suffering from some sort of memory issue. I'll discuss what steps you can take to cure this problem and will also cover how the JVM can both help reduce the memory ~strength of your application. What should you do?"

1. Copyright 2018 Kirk Pepperdine
MEMORY
THE TROUBLE WITH

2. Copyright 2018 Kirk Pepperdine
jClarity
▸ Kirk Pepperdine
▸ Author of jPDM, a performance diagnostic model
▸ Author of the original Java Performance Tuning workshop
▸ Co-founded jClarity
▸ Building the smart generation of performance diagnostic tooling
▸ Bring predictability into the diagnostic process
▸ Co-founded JCrete
▸ The hottest unconference on the planet
▸ Java Champion(s)
OUR MARKETING SLIDE

3. Copyright 2018 Kirk Pepperdine
jClarity
What is your performance trouble spot

4. Copyright 2018 Kirk Pepperdine
jClarity
> 70% of all applications are bottlenecked
on memory

5. Copyright 2018 Kirk Pepperdine
jClarity
and no,
Garbage Collection
is not a fault!!!!

6. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity

7. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Spring Boot

8. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Cassandra

9. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Cassandra
or any big nosql solution

10. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Apache Spark

11. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Apache Spark
or any big data framework

12. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Log4J

13. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
Log4J
or any Java logging framework

14. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
JSON

15. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
JSON
With almost any Marshalling protocol

16. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
ECom caching products

17. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
ECom caching products
Hibernate

18. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
ECom caching products
Hibernate
and so on

19. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
ECom caching products
Hibernate
and so on
and so on

20. Copyright 2018 Kirk Pepperdine
DO YOU USE
jClarity
ECom caching products
Hibernate
and so on
and so on
and so on

21. Copyright 2018 Kirk Pepperdine
jClarity
then you are very likely in this 70%

22. Copyright 2018 Kirk Pepperdine

23. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
▸ Large live data set size
▸inflated live data set size
▸ loitering
▸ Unstable live data set size
▸memory leak
jClarity

24. Copyright 2018 Kirk Pepperdine
WAR STORIES
▸ Reduced allocation rates from 1.8gb/sec to 0
▸tps jumped from 400,000 to 25,000,000!!!
▸ Stripped all logging our of a transactional engine
▸Throughput jumped by a factor of 4x
▸ Wrapped 2 logging statements in a web socket framework
▸Memory churn reduced by a factor of 2
jClarity

25. Copyright 2018 Kirk Pepperdine
ALLOCATION SITE
jClarity
Foo foo = new Foo();
forms an allocation site
0: new #2 // class java/lang/Object
2: dup
4: invokespecial #1 // Method java/lang/Object."<init>":()V
▸Allocation will (mostly) occur in Java heap
▸ fast path
▸ slow path
▸ small objects maybe optimized to an on-stack allocation

26. Copyright 2018 Kirk Pepperdine
JAVA HEAP
jClarity
Eden Survivor (to) Tenured
Java Heap
▸ Java Heap is made of;
▸ Eden - nursery
▸ Survivor - intermediate pool designed to delay promotion
▸ Tenured - to hold long lived data
▸ Each space contributes to a different set of problems
▸ All affect GC overhead

27. Copyright 2018 Kirk Pepperdine
EDEN ALLOCATIONS
jClarity
top of heap pointer

28. Copyright 2018 Kirk Pepperdine
OBJECT ALLOCATION
jClarity
top of heap pointer
Foo foo = new Foo();
Bar bar = new Bar();
byte[] array = new byte[N];

29. Copyright 2018 Kirk Pepperdine
OBJECT ALLOCATION
jClarity
top of heap pointer
Foo foo = new Foo();
Bar bar = new Bar();
byte[] array = new byte[N];
Foo

30. Copyright 2018 Kirk Pepperdine
OBJECT ALLOCATION
jClarity
top of heap pointer
Foo foo = new Foo();
Bar bar = new Bar();
byte[] array = new byte[N];
Foo Bar

31. Copyright 2018 Kirk Pepperdine
OBJECT ALLOCATION
jClarity
top of heap pointer
Foo foo = new Foo();
Bar bar = new Bar();
byte[] array = new byte[N];
Foo Bar byte[]

32. Copyright 2018 Kirk Pepperdine
OBJECT ALLOCATION
jClarity
top of heap pointer
Foo Bar byte[]
▸ In multi-threaded apps, top of heap pointer must be surrounded by barriers
▸single threads allocation
▸hot memory address
▸ solved by stripping (Thread local allocation blocks)

33. Copyright 2018 Kirk Pepperdine
TLAB ALLOCATION
jClarity
top of heap pointer
▸ Assume 2 threads
▸each thread will have it’s own (set of) TLAB(s)
TLAB TLAB
TLAB pointerTLAB pointer

34. Copyright 2018 Kirk Pepperdine
TLAB ALLOCATIONS
jClarity
▸ Thread 1 -> Foo foo = new Foo(); byte[] array = new byte[N];
▸byte[] doesn’t fit in a TLAB
▸ Thread 2 -> Bar bar = new Bar();
byte[]TLAB TLAB
Foo Bar
TLAB pointerTLAB pointer
top of heap pointer

35. Copyright 2018 Kirk Pepperdine
TLAB WASTE %
jClarity
▸ Allocation failure to prevent buffer overflow
▸ waste up to 1% of a TLAB
FooFoo Foo Foo Foo Foo Foo Foo

36. Copyright 2018 Kirk Pepperdine
TLAB WASTE %
jClarity
▸ Allocation failure to prevent buffer overflow
▸ waste up to 1% of a TLAB
Foo Foo Foo Foo Foo Foo Foo Foo
Foo

37. Copyright 2018 Kirk Pepperdine
TENURED SPACE
jClarity
▸ Allocations in tenured make use of a free list
▸free list allocation is ~10x the cost of bump and run
▸ Data in tenured tends to be long lived
▸amount of data in tenured do affect GC pause times
Bar Bar Foo Foo
Free List

38. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
jClarity

39. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
jClarity
▸Quickly fill Eden
▸ frequent young gc cycles
▸ speeds up aging
▸ premature promotion
▸more frequent tenured cycles
▸increased copy costs
▸increased heap fragmentation
▸Allocation is quick
▸ quick * large number = slow

40. Copyright 2018 Kirk Pepperdine
REDUCING ALLOCATIONS
jClarity
> 1gb/sec
< 300mb/sec
sizeofgain

41. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
▸Large live data set size
▸ inflated live data set size
▸loitering
jClarity

42. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
▸Large live data set size
▸ inflated live data set size
▸loitering
jClarity
▸inflated scan for root times
▸reduced page locality
▸Inflated compaction times
▸ increase copy costs
▸ likely less space to copy too

43. Copyright 2018 Kirk Pepperdine
PAUSE VS OCCUPANCY
jClarity

44. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
▸Large live data set size
▸ inflated live data set size
▸loitering
▸Unstable live data set size
▸ memory leak
jClarity

45. Copyright 2018 Kirk Pepperdine
PROBLEMS
▸ High memory churn rates
▸many temporary objects
▸Large live data set size
▸ inflated live data set size
▸loitering
▸Unstable live data set size
▸ memory leak
jClarity
▸Eventually you run out of heap
space
▸ each app thread throws an
OutOfMemoryError and
terminates
▸JVM will shutdown with all non-
daemon threads terminate

46. Copyright 2018 Kirk Pepperdine
jClarity
Escape Analysis

47. Copyright 2018 Kirk Pepperdine
Demo time
jClarity

48. Copyright 2018 Kirk Pepperdine
TEXT
TITLE TEXT
▸ Body Level One
▸ Body Level Two
▸ Body Level Three
▸ Body Level Four
▸ Body Level Five
Ask out my Java Performance Tuning Workshop
Send us a Java 11 GC log or
tweet about @jclarity
and #censum
and
receive a free Censum License