This slide deck will be removed from here in the future. It has been moved to : https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx

1. @YourTwitterHandle#DV14 #YourTag @MilenDyankov#DV14 #beyondXmx
Web application performance tuning beyond Xmx
Milen Dyankov

2. This talk is not about JVM and GC tuning !
Disappointed? You shouldn't be, there are plenty of other talks out there!

3. This talk is not about Liferay either !
However you may want to check it out anyway! It is a great platform!

4. This is not a university courseThis is not a university course
on performance tuning!on performance tuning!
I'm just sharing my personalI'm just sharing my personal
experience with you!experience with you!

5. performance
tuning
expert

7. Customer's statement:
“It is very slow!
You need to improve
the performance!”
What would you do?

8. Customer's statement:
“It is very slow!
You need to improve
the performance!”
What would you do?
Check the CPU andCheck the CPU and
memory usagememory usage
Check theCheck the
JVM parametersJVM parameters
Check the databaseCheck the database
for slow queriesfor slow queries

9. Check the CPU andCheck the CPU and
memory usagememory usage
Check theCheck the
JVM parametersJVM parameters
Check the databaseCheck the database
for slow queriesfor slow queries
While those are valid points, they are all the WRONG answer!
Customer's statement:
“It is very slow!
You need to improve
the performance!”
What would you do?

10. Ask to provideAsk to provide
numbers!numbers!
Ask to provideAsk to provide
expectations!expectations!
Ask to provideAsk to provide
units!units!
Customer's statement:
“It is very slow!
You need to improve
the performance!”
What would you do?

11. Ask to provideAsk to provide
numbers!numbers!
Ask to provideAsk to provide
expectations!expectations!
Ask to provideAsk to provide
units!units!
Customer's statement:
“It is very slow!
You need to improve
the performance!”
What would you do?
Make sure everyone uses the same definitions and units
to describe both the status and the expectations!

12. “It can only serve N users,
we expect it to serve MxN users”

13. “It can only serve N users,
we expect it to serve MxN users”
N users accessing the systemN users accessing the system
at random times during given period!at random times during given period!
N users accessing the systemN users accessing the system
as a result of marketing campaign!as a result of marketing campaign!
N users accessing the systemN users accessing the system
every day between 8:00 and 8:15!every day between 8:00 and 8:15!
consider these

14. “It can only serve N users,
we expect it to serve MxN users”
N users accessing the systemN users accessing the system
at random times during given period!at random times during given period!
N users accessing the systemN users accessing the system
as a result of marketing campaign!as a result of marketing campaign!
N users accessing the systemN users accessing the system
every day between 8:00 and 8:15!every day between 8:00 and 8:15!
consider these
Don't confuse
ability to scale with
performance tuning!
Use unambiguous
or less ambiguous
units!

15. Unambiguous
Ambiguous
Business Technical
usersusers
concurrent usersconcurrent users
concurrent sessionsconcurrent sessions
concurrent requestsconcurrent requests
requests per secondrequests per second
(M/K)bytes per second(M/K)bytes per second
But keep in mind:
sometimes being too technical
leads to the wrong decisions!
Use unambiguous
or less ambiguous
units!

16. Consider this statement:Consider this statement:
“N needs to travel from town A to town B.
What is the fastest possible way?”
But keep in mind:
sometimes being too technical
leads to the wrong decisions!

17. Consider this statement:
“N needs to travel from town A to town B.
What is the fastest possible way?”
M x
But keep in mind:
sometimes being too technical
leads to the wrong decisions!
Refinement:
N is a group of M passengers

18. Refinement:
N is a group of M passengers
Consider this statement:
“N needs to travel from town A to town B.
What is the fastest possible way?”
But keep in mind:
sometimes being too technical
leads to the wrong decisions!

19. How to do performance tuning?

20. It's kind of like …
… drawing an Owl!
How to do performance tuning?

21. ●
First draw 2 ellipses!
It's kind of like …
… drawing an Owl!
How to do performance tuning?

22. ●
First draw 2 ellipses!
●
Then draw the rest
of the damn Owl!
It's kind of like …
… drawing an Owl!
How to do performance tuning?

23. units = defineUnits();
expectations = defineExpectations(units);
results = measurePerformance();
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
results = measurePerformance();
}
Enough analogies! Let's try with an algorithm:
How to do performance tuning?

24. units = defineUnits();
expectations = defineExpectations(units);
results = measurePerformance();
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
results = measurePerformance();
}
Enough analogies! Let's try with an algorithm:
How to do performance tuning?

25. units = defineUnits();
expectations = defineExpectations(units);
results = measurePerformance();
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
results = measurePerformance();
}
Enough analogies! Let's try with an algorithm:
How to do performance tuning?

26. Application
measurePerformance();measurePerformance();

27. Application
measurePerformance();measurePerformance();

28. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Let's refine the algorithm:
Application
measurePerformance();measurePerformance();

29. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Let's refine the algorithm:
Application
measurePerformance();measurePerformance();

30. Browser Network Server JVM Container Application BrowserNetwork. . .
collectData();collectData();

31. Browser Network Server JVM Container Application BrowserNetwork. . .
Take into account your target group!
Respect “Cache-Control” and “Expires” headers !
Home page is not the main entrance anymore!
Consider search results and referrals!
Take ONLY what you REALLY need!
Measure the impact of 3rd party code (analytics, ads, …)!
collectData();collectData();

32. Browser Network Server JVM Container Application BrowserNetwork. . .
You are NOT the only creature in the space and time!
Understand who else is using the same network, what for and when!
A network is, what the name says – a net, not a line!
There may be many networks and interfaces involved!
Network traffic goes in two directions!
No problem on the way in, does NOT mean no problem on the way out!
(or vice-versa)
collectData();collectData();

33. Browser Network Server JVM Container Application BrowserNetwork. . .
“Server” is one of the most ambiguous words !
Amazon, Jelastic, … are servers as well … but different!
Virtual resources are … well, VIRTUAL!
“… 3 VMs (8GB of RAM each) … oh the server has 16GB of RAM”
Operating System will only tell you what you ask for !
nice, top, htop, atop, dstat, vmstat, ifstat, iostat, …
collectData();collectData();

34. Browser Network Server JVM Container Application BrowserNetwork. . .
Know what your workers are doing!
Regular thread dumps help find long running threads!
Garbage collection of course! Everyone knows that!
-verbose:gc -Xloggc:... -XX:+PrintGCDetails
Tools are nice but they also consume resources!
Don't measure having the profiler attached!
collectData();collectData();

35. Browser Network Server JVM Container Application BrowserNetwork. . .
Policies, politics, preferences and common sense!
Don't fight against windmills!
Sessions tend to grow, last forever and …
… replicate!
Security, audits, transactions … ,Security, audits, transactions … ,
…… and interceptors in general!and interceptors in general!
collectData();collectData();

36. Browser Network Server JVM Container Application BrowserNetwork. . .
Text Object XML Object JSON … !→ → → → →
Simultaneous translation is not an easy job!
Many a mickle makes a muckle!Many a mickle makes a muckle!
Number of calls is as important as the time the call takes!Number of calls is as important as the time the call takes!
Cache stats – hits, misses, invalidation, overflows, ...
Not caching anything? Great! You just found the bottleneck!
collectData();collectData();

37. Browser Network Server JVM Container Application BrowserNetwork. . .
Databases
External storages
Frameworks
3rd
party services
. . .
Last, but not least,
pay attention to all those
little tiny things that makes
an Owl look like an Owl!
Oh and here is something that is valid
for about everything we covered so far:
You can really make use of
another pair of eyes!
collectData();collectData();

38. “It doesn't matter how beautiful your theory is,
it doesn't matter how smart you are.
If it doesn't agree with experiment, it's wrong.”
Richard P. Feynman
Do I really have to do all that?
You better do! Here is why:

39. Do I really have to do all that?
You better do! Here is why:

40. Even the guys behind BIG names
sometimes don't do it right!
2014, Louvre Museum, Paris, France
Do I really have to do all that?
You better do! Here is why:

41. THE USUAL SUSPECTS
is a movie title not a problem solving strategy!
Do I really have to do all that?
You better do! Here is why:

42. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Back to the algorithm:

43. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Back to the algorithm:

44. 1
2
3
analyze(analyze(datadata););

45. The Base Rate Fallacy
When people have both
generic and specific information,
they tend to make judgments of probability
based entirely upon specific information,
leaving out the base rate.
analyze(analyze(datadata););

46. Congruence bias
A hypothesis is tested only
by thinking of results that would be found
if that hypothesis is true.
This heuristic, which many people seem to use,
ignores alternative hypotheses
analyze(analyze(datadata););

47. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Back to the algorithm:

48. units = defineUnits();
expectations = defineExpectations(units);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
while (!results.match(expectations)) {
component = findLowestPerforming(results);
bottleneck = findBottleneck (component);
remove(bottleneck);
- results = measurePerformance()
+ data = collectData();
+ results = analyze(data)
}
Back to the algorithm:

49. I'm sorry! I don't have any universal solution yet!
But I will be happy to give you some
links, tips and warnings
if you don't mind!
remove(remove(bottleneckbottleneck););

50. Browser Network Server JVM Container Application BrowserNetwork. . .
“I sometimes go into a client site and got
3X improvements on performance
just with some tuning of the operating system
and the ways VMs are running.”
Martin Thompson
http://www.infoq.com/presentations/top-10-performance-myths
Step out of the comfort zone!
remove(remove(bottleneckbottleneck););

51. Browser Network Server JVM Container Application BrowserNetwork. . .
Avoid tuning by folklore!
remove(remove(bottleneckbottleneck););
Don't copy JVM settings from various blogs!
They are not meant to be for you!

52. Browser Network Server JVM Container Application BrowserNetwork. . .
Blocking vs Nonblocking Cache
Data structures
( the famoust ArrayList vs LinkedList interview question is for a reason )
Method complexity
( StringBuilder vs String.concat )
Logging
Know what the code you use actually do!
remove(remove(bottleneckbottleneck););

53. Browser Network Server JVM Container Application BrowserNetwork. . .
Going multi-thread !
Going asynchronous !
Going modular!
Going multi-vm !
Going distributed!
Going off-heap!
Get (more) things done!
remove(remove(bottleneckbottleneck););

55. SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS
Tuning the unknown!
“A common mistake that people make
when trying to design something
completely foolproof is to underestimate
the ingenuity of complete fools.”
The Hitchhiker's Guide to the Galaxy
Douglas Adams

56. There are 2382 configuration properties
in Liferay 6.2
Make it configurable!
SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS
Tuning the unknown!

57. Document it !
There are 6896 lines of comments
documenting Liferay 6.2 properties!
SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS
Tuning the unknown!

58. Provide reasonable defaults !
Example:
#
# If the user can unzip compressed HTTP content,
# the GZip filter will zip up the HTTP content
# before sending it to the user. This will speed up
# page rendering for users that are on dial up.
#
com.liferay.portal.servlet.filters.gzip.GZipFilter=true
SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS
Tuning the unknown!

59. Provide performance results
for most common cases!
SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS
Tuning the unknown!

60. @YourTwitterHandle#DV14 #YourTag @MilenDyankov#DV14 #beyondXmx
Photo: http://www.flickr.com/photos/wwworks/4759535950/
http://www.liferay.com
@Liferay
http://www.facebook.com/Liferay
milen.dyankov@liferay.com
http://www.liferay.com/web/milen.dyankov/
@milendyankov
@LiferayPL