Top .NET, Java & Web Performance Problems. Why these apps failed, how to avoid it and which metrics to look at, e.g: # of Busy vs. Idle Worker Threads, Connection Pool Acquisition Time, # Exceptions Thrown, ...

1. 1 @Dynatrace
- More on http://blog.dynatrace.com
- Dynatrace Free Trial: http://bit.ly/dttrial
Andreas Grabner - @grabnerandi
Biggest Performance
Mistakes (.NET, Java, Web …)

2. 2 @Dynatrace
That’s why I ended up talking about performance

3. 3 @Dynatrace
Nobody wants this …

4. 4 @Dynatrace
Unless you work for
Google or Microsoft 

5. 5 @Dynatrace
Nor this …

6. 6 @Dynatrace

7. 7 @Dynatrace
As it leads to this …

8. 8 @Dynatrace
The “War Room”
Facebook – December 2012

9. 9 @Dynatrace
And potentially to this …

10. 10 @Dynatrace

11. 11 @Dynatrace
And this …

12. 12 @Dynatrace

13. 13 @Dynatrace
And that’s why Business
doesn’t like it either …

14. 14 @Dynatrace
~80% of problems
caused by ~20% patterns
YES we know this
80%Dev Time in Bug Fixing
$60BDefect Costs
BUT

15. 15 @Dynatrace
6 Situations on
WHY this happened,
HOW to avoid it,
METRICS to look at

16. 16 @Dynatrace

17. 17 @Dynatrace
Incorrect Sizing
of Pools and
Queues

18. 18 @Dynatrace
Online Banking: Slow Balance Check
1.69s (=101s!) To
Check Balance!
87% spent in IIS 600! SQL Executions

19. 19 @Dynatrace
#1 Time really spent in IIS?
Tip: Elapsed Time tells us WHEN a
Method was executed!
Finding: Thread 32 in IIS waited 87s to
pass control to Thread 30 in ASP.NET
Tip: Thread# gives us insight on
Thread Queues / Switches

20. 20 @Dynatrace
#2 What about these SQL Executions?
Finding: EVERY SQL
statement is executed on
ITS OWN Connection!
Tip: Look at
“GetConnection”

21. 21 @Dynatrace
#2 SQL Executions! continued …
#1: Same SQL is executed 67! times
#2: NO PREPARATION
because everything
executed on new
Connection

22. 22 @Dynatrace
Lessons Learned!
ASP.NET Worker
Thread Pool Sizing!
DB Connection Pools
More Efficient SQL

23. 23 @Dynatrace
Idle vs. Busy Threads
# SQLs / Request
# GetConnection

24. 24 @Dynatrace

25. 25 @Dynatrace
“Blindly” (Re)use
Existing
Components

26. 26 @Dynatrace
Requirement: We need a report

27. 27 @Dynatrace
Using Hibernate results in 4k+ SQL Statements to
display 3 items!
Hibernate Executes
4k+ Statements
Individual Execution
VERY FAST
But Total SUM
takes 6s

28. 28 @Dynatrace
• Proper Sizing Specifications
• Understand the Frameworks you are using
Lessons Learned

29. 29 @Dynatrace
# SQLs / Request
# Same SQL /
Request

30. 30 @Dynatrace

31. 31 @Dynatrace
Wrong Tools for
the Job

32. 32 @Dynatrace
#1 Banking: Transaction History CSV Download!
Remember: High GC doesn’t
necessarily mean that this request
has a memory problem – BUT - …
Root Cause:
Heavy use of String.Concat to build the CSV output in Memory
Recommendations:
#1: Use System.Text.StringBuilder instead of String.Concat
#2: Write directly back to OutputStream
Problem: Takes 207s! To download.
87% of Time spent in Garbage
Collection

33. 33 @Dynatrace
#2 Store: Rendering Search Result
Problem: 4.4s to render
result page
Root Cause: Custom RegEx
Library with performance
issues on large strings

34. 34 @Dynatrace
#3 Store: Occasional Performance Spikes
Problem: Occasional Spikes!
Never seen in Development

35. 35 @Dynatrace
First Analysis: His own code 

36. 36 @Dynatrace
DNS Lookup Issue
during Login

37. 37 @Dynatrace
Reverse Lookup by
IP fails on these IPs
Root Cause:
Issues from Data Center Hoster

38. 38 @Dynatrace
• CSV Export Example
• Don’t build large Strings in Memory with String.Concat
• Write to output stream and use StringBuilder
• Search Result Page
• Don’t implement custom RegEx Library – existing Tools are better
• Test with large results and not just small test data
• Geo Lookup
• Don’t trust your Hosting Provider
• E.g: Use GeoIP Databases for accurate reverse lookups
Lessons Learned – Use Better Tools / Libraries

39. 39 @Dynatrace
Memory Usage
Time Spent in APIs

40. 40 @Dynatrace

41. 41 @Dynatrace
Not every
Architect makes
good decisions

42. 42 @Dynatrace
• Symptoms
• HTML takes between 60 and 120s to render
• High GC Time
• Assumptions
• Bad GC Tuning
• Probably bad Database Performance as rendering was simple
Project: Online Room Reservation System

43. 43 @Dynatrace
Developers built own monitoring
void roomreservationReport(int roomid)
{
long startTime = System.currentTimeMillis();
Object data = loadDataForRoom(roomid);
long dataLoadTime = System.currentTimeMillis() - startTime;
generateReport(data, roomid);
}
Result:
Avg. Data Load Time: 45s!
DB Tool says:
Avg. SQL Query: <1ms!

44. 44 @Dynatrace
#1: Loading too much data
24889! Calls to the
Database API!
High CPU and High
Memory Usage to keep all
data in Memory

45. 45 @Dynatrace
#2: On individual connections 12444!
individual
connections
Classical N+1
Query Problem
Individual SQL
really <1ms

46. 46 @Dynatrace
#3: Putting all data in temp Hashtable
Lots of time
spent in
Hashtable.get
Called from their
Entity Objects

47. 47 @Dynatrace
• Custom Measuring
• Was impacted by Garbage Collection
• Just measured overall time but not # SQL Executions
• Learn SQL and don’t use Hashtables as Workaround
Lesson Learned
void roomreservationReport(int roomid)
{
long startTime = System.currentTimeMillis();
Object data = loadDataForRoom(roomid);
long dataLoadTime = System.currentTimeMillis() - startTime;
generateReport(data, roomid);
}

48. 48 @Dynatrace
# SQL Executions
# of SAME SQLs
Conn. Acquisition
Time

49. 49 @Dynatrace

50. 50 @Dynatrace
#Push
without a Plan

51. 51 @Dynatrace
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~
20MB

52. 52 @Dynatrace
Fifa.com during Worldcup
http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

53. 53 @Dynatrace
# Images
# Redirects
Size of Resources

54. 54 @Dynatrace

55. 55 @Dynatrace
#No “Agile”
Deployment

56. 56 @Dynatrace
Load Spike resulted in Unavailability
Adonair

57. 57 @Dynatrace
Alternative: “GoDaddy goes DevOps”
1h before
SuperBowl KickOff
1h after
Game ended

58. 58 @Dynatrace
# of Domains
Total Size

59. 59 @Dynatrace
Waiting for
Bug Reports

60. 60 @Dynatrace
Look behind the scenes

61. 61 @Dynatrace
•# Images
•# Redirects
•Size of Resources
•# SQL Executions
•# of SAME SQLs
•# Items per Page
•# AJAX per Page
Look at these Metrics Before Checking In Code
•Time Spent in API
•# Calls into API
•# Functional Errors
•3rd Party calls
•# of Domains
•Total Size
•…

62. 62 @Dynatrace
Commit Stage
• Compile
• Execute Unit Test
• Code Analysis
• Build installers
Automated
Acceptance
Testing
Automated
Capacity
Testing
Manual testing
• Key showcases
• Exploratory testing Release
Unit & Integration Tests
Functional Tests
Performance Tests
Production
Monitoring
Functional Tests
(R)Evolutionize Performance Optimization

63. 63 @Dynatrace

64. 64 @Dynatrace
Example from Web Diagnostics 282! Objects
on that page9.68MB Page Size
8.8s Page Load
Time
Most objects are images
delivered from your main
domain
Very long Connect time
(1.8s) to your CDN

65. 65 @Dynatrace
Example from Server-Side Diagnostics
526s to render that
report
1 SQL running
210s!
Lots of time spent
in logging to Log4J
Lots of time spent
in rendering

66. 66 @Dynatrace
Online Performance Clinics
Every Other week @
bit.ly/onlineperfclinic

67. 67 @Dynatrace
Your Benefits
• Free Performance Review
• Extended Dynatrace License
“Share Your PurePath”
bit.ly/sharepurepath
My Benefits
• More blog material for next year 
• Gratification that I could help you

68. 68 @Dynatrace
Questions and/or Demo
Slides: slideshare.net/grabnerandi
Get Tools: bit.ly/dttrial
YouTube Tutorials: bit.ly/dttutorials
Contact Me: agrabner@dynatrace.com
Follow Me: @grabnerandi
Read More: blog.dynatrace.com

69. 69 @Dynatrace
Andreas Grabner
Dynatrace Developer Advocate
@grabnerandi
http://blog.dynatrace.com