Covering the top performance landmines that we see in live production systems and how to avoid them with proper testing

1. Top Performance Land Mines
And how to address them
Andreas Grabner, Lead of Center of Excellence @ dynaTrace

2. WHY do WE care?

3. OnlineGraduatePrograms.com
“Instant America”

4. OnlineGraduatePrograms.com
“Instant America”

5. WHAT do WE miss?

6. Performance Land Mines found in Production
• Bloated Web Frontends • Deployment mistakes
• Too many Database • Traffic estimates incorrect
Statements
• Poorly provisioned hardware
• Too heavy logging and tracing
• No consideration of end user
• Synchronized to Death devices or latency
• Too chatty on remoting • Misconfigured CDN
channels
• Load Balancing Issues
• Wrong use of O/R Mappers
• Networking/infrastructure
• Memory Leaks issues
• Problematic 3rd parties

7. Problem #1: Size

8. Watch your Weight
(Content Size)
Avg Bytes # of Objects Avg GPSI
400,000 50
45
350,000
40
300,000
# of Objects/Avg GPSI
35
250,000
30
Bytes
200,000 25
20
150,000
15
100,000
10
50,000
5
0 0
May Jun Jul '11 Aug Sep Oct Nov Dec Jan Feb Mar Apr
'11 '11 '11 '11 '11 '11 '11 '12 '12 '12 '12

9. Where did all of this stuff come from?
(Application Complexity)
Average number of hosts accessed by the browser across the
Gomez US Sports Web Performance Benchmark 29

10. Where did all of this stuff come from?
(Application Complexity)
Average number of hosts accessed by the browser across the Gomez US
Sports Mobile Website Performance Benchmark 12

11. Cost of Complexity – 1
As the page gets larger, it
gets slower and more
complex

12. Cost of Complexity – 2
Response Time and Page Size by Hosts
16 3000000
14
2500000
12
2000000
10 As the page gets more
Seconds
complex, it gets slower
Bytes
8 1500000
6 and larger
1000000
4
500000
2
0 0
1
3
5
7
9
25
39
11
13
15
17
19
21
23
27
29
31
33
35
37
41
43
45
47
49
51
53
Hosts
Geometric Mean BYTES

13. Tips for Testing: Analyzing Page KPIs
• Analyze Key Performance Indicators (KPIs) for every tested page
Tested Pages
KPIs

14. Tips for Testing: Comparing Test Runs

15. Tips for Testing
• Free Tooling
– dynaTrace AJAX Edition for IE and FF
– YSlow for FF
– SpeedTracer for Chrome
– ShowSlow as performance repository

16. Problem #2:
Deployment Issues

17. Top Deployment Problems
• Missing Files
• Incorrect Access Settings
• Slow Web Server Modules

18. (1) Missing Files: Impact User Experience
Enter Here
PopUp Here

19. (1) Missing Files: Lead to JavaScript Errors
Errors caused by missing files

20. (1) Missing Files: How to identify
• Watch out for HTTP 4xx

21. (1) Missing Files: Tips for Testing
• Test with real browsers
– HTTP-only testing might not download dynamically loaded JS files
– Catch browser-specific implementations
• Test from the In- and Out-side
– Files might be deployed on AppServer but blocked on Web Server or
Load Balancer
– Certain files might come from a CDN. Testing the CDN settings is
important

22. (2) Incorrect Access Settings: Impact User Experience
and Performance
14k Errors in 5 minutes
after deploying in Production

23. (2) Incorrect Access Settings: Impact User Experience
and Performance
Deployment causes extra roundtrips
for each restricted resource

24. (2) Incorrect Access Settings: Tips for Testing
• Test with real browsers
– HTTP-only testing might not download dynamically loaded JS files
– Catch browser-specific implementations
• Test from the In- and Out-side
– Files might be accessible on AppServer but blocked on Web Server
– Watch out for missing JS, CSS and Image files

25. (3) Slow Web Server Modules: Performance Impact
• 90% of ASP.NET Request Time spent in IIS
90%

26. (3) Slow Web Server Modules: Tips for Testing
• Test with all Modules used in Production
• Analyze Time spent in Modules vs. Application Code

27. Problem #3: Overhead
through
Logging/Tracing

28. Logging is important!!
• As otherwise we are really blind – but – it comes with a cost

29. (1) Excessive use of Exceptions
• Tomcat Connection Pool with logAbandoned=true
A debug flag causes many exception
objects to be created

30. (1) Excessive use of Exceptions
• HUNDREDS of exceptions per request

31. (1) Excessive use of Exceptions
• Getting Stack Trace that many times is a performance risk

32. (2) Too granular logging
• Too much logging doesn’t provide much value

33. Tips for Testing
• What to watch out for during a test
– Number of log entries written
– Size and growth of log files
– Usage of Stack Traces in Logs -> Usage of Exception Objects
• Test Production Readiness by
– Watch out for DEBUG, FINE, FINEST, … Log Messages -> should not be
logged
– Show produced log files to developers and ask whether they are useful

34. Other Problems:
Excessive Database

35. Common Database Access Problem Patterns
Application Connection
getConnection()
Application Connection locked
select ... where id= 1
select ... where id= 1 closeConnection()
Application Connection
getConnection()
Code requiring DB access
Connection locked
select ... where id= 100 closeConnection()
Code NOT requiring DB access
Database
N+1 Query/Too many SQLs
Unnecessary Resource Usage
Application select ... from master
select ... from detail
select ... from detail
select ... from detail
Database
Application
select ... from master,
detail where ..
Database
Lazy vs. Eager Loading Loading too much data

36. (1) N+1 Query Problem

37. (2) Loading too much data
• Example from SharePoint
for (int itemIx=0;itemIx< SPContext.Current.List.Items.Count;itemIx++) {
SPListItem listItem = SPContext.Current.List.Items[itemIx];
// do something ...
}
Every access to Count and Items Property queries the whole SharePoint list
We end up with 202 SQL Executions with a total exec time of > 1s

38. Tips for Testing
• Watch out for
– Slow SQL Statements
– Number of SQL Statements per Request
– Same SQL executed many times per request
– Prepared vs. Unprepared Statements
– Connection Pool Usage

39. Other Problems:
Synchronization

40. Identifying Synchronization Problems
Response Time
BAD
CPU
GOOD

41. Tips for Testing
• Use Performance Management Tool
• Take Continuous Thread Dumps

42. Other Problems: 3rd
Party Content

43. 3rd Party Content is important

44. Testing 3rd Party Content
• Watch out for Size and Number of Resources
# of Domains # of Resources Total Bytes DNS [ms] Connect [ms]
With Third
26 176 2856 Kb 1286,82 1176,09
Party Content
Without Third
2 59 897 Kb 0,91 22,25
Party Content
• Remember: Watch this over time

45. Testing 3rd Party Content Performance Impact

46. Testing 3rd Party Content Performance Impact

47. Resources
• Free Online Java Enterprise Performance Book
– http://book.dynatrace.com
• Our Performance Blog
– http://blog.dynatrace.com
• Contact
– andreas.grabner@compuware.com
– @grabnerandi

48. THANK YOU