#1 Putting the application in production - The presenter moved the Spring Pet Clinic sample application to a production environment by removing debugging logs and using a real database. #2 Creating a JMeter test - The presenter used JMeter to simulate 500 users loading the application and found it resulted in many errors and crashes with only 250 requests per second. #3 Profiling - Profiling with VisualVM and JProfiler identified a memory leak in the Dandelion library and the use of HTTP sessions as initial performance issues.

1. © 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.
Performance-tuning the «Spring Pet Clinic»
sample application
By Julien Dubois

2. About me
• Julien Dubois
• From Paris, France
• Co-authored «Spring par la pratique» (French
book on Spring)
• Former SpringSource France director
• Director of Consulting at Ippon Technologies
• Follow me @juliendubois
2

3. Goals of this presentation
• Make a specific application run faster
• Learn classic issues with Spring Web
applications
• Learn our methodology to solve
performance issues
3

4. More information on this presentation
• This presentation started with a series of blog posts:
http://blog.ippon.fr/tag/spring-petclinic/
• You can run your own tests, every step is cleanly commited
on Github:
https://github.com/jdubois/spring-petclinic
4

5. What’s new since the blog entries?
• Java 7
• I got a brand new MacBook :-)
–Does a SSD really matters?
–Does the new core i7 matters?
5

6. 6
Introducing
the Spring
Pet Clinic
sample
application

7. Do you remember the Sun Java Pet Store?
• Used to explain how you
could build an incredibly
complex application
• On a very expensive
application server
• With an horrible-looking
result
• When your customer only
wanted a simple Web app
7

8. The Spring Pet Clinic
• Is here to demonstrate Spring
features and best practices
• Is a very classical MVC-style
application (no REST + funky JS
library)
• Is still a good excuse to use cute
dogs in your presentation
8

9. More information on the Spring Pet Clinic
• Code:
https://github.com/SpringSource/spring-petclinic/
• Short presentation:
https://speakerdeck.com/michaelisvy/spring-petclinic-sample-
application
9

10. 10
#1
Putting the
application
in
production

11. Moving to production
• Be as close as possible to a «production environment»
–Remove «debugging» logs [ Code ]
–Use a «real» database [ Code ]
• For this presentation our environment isn’t perfect
–Everything runs on the same machine
–Even the presentation itself :-)
11

12. Memory tuning
• We’re running the application from the Maven Tomcat plugin
–Of course Maven takes some resources
• Still the application runs with a small memory footprint
–export MAVEN_OPTS="-XX:MaxPermSize=64m -Xms128m -
Xmx128m"
• Why not use more memory?
–It’s easier to spot memory problems
–We don’t need it (we’re not using a big fat Java EE server!)
12

13. 13
#2
Creating a
JMeter test

14. Simulate load
• JMeter will simulate users
–500 users
–In a «realistic» way
• Use a timer
• Check «clear cookies each iteration?»
14

15. Results
• 250 req/sec
• Lots of errors
• Quickly ends with a crash...
15

16. 16
#3
Profiling

17. Does profiling distort results?
• Short answer : not a lot
–We’ll see this when the application runs better
• We have used JProfiler and VisualVM in production many
times
–Often caused less trouble than official «production monitoring
tools» like HP Diagnostics
–Only way to find problems and understand what’s going on
–Easy to install
17

18. Choose your weapon
• VisualVM
–Free & lightweight
–Included in the JDK
• JProfiler
–Best tool we’ve used
• YourKit
–Easy to use & cheap
18

19. First culprit : Dandelion
• More specifically, Dandelion module for
DataTables
–https://github.com/dandelion/dandelion-datatables
• Nice & fun JSP taglib for rendering JQuery
DataTables
–Has a memory leak in this version
–Is now corrected (extremely quick response time
from the Dandelion team)
• So we switched to «plain old JQuery
DataTables» [ Code ]
19

20. Second culprit : the HTTP Session
• Using HTTP Sessions can be helpful
–Developer productivity
–Data security
• Going stateless is often better
–Uses less RAM
–More scalable
20

21. Going stateless
• Example : the PetController.processUpdateForm() method
–Use Spring MVC REST to get more information from the URL
–Fetch again the missing data
• We said more scalability, not more performance (for the moment)
• With some work, the whole application can become stateless
[ Code ]
21

22. Results
• 500 req/sec
• A few errors
• That’s a lot better than before!
22

23. 23
#4
Tomcat
tuning

24. Tomcat tuning
• This is the easiest step
–Migrate to the latest version of the Tomcat plugin
–Use the new NIO connector
• org.apache.coyote.http11.Http11NioProtocol
• [ Code ]
24

25. 25
• 950 req/sec
• No errors
• Victory :-)
Results

26. 26
#5
Removing
locks

27. Locks
• Locks are blocked threads
–Aka «there’s a synchronized somewhere»
• They obviously cause scalability problems
• Let’s launch our profiler again!
27

28. First culprit : commons DBCP
• Commons DBCP does a lot of synchronization
• We replaced it with Tomcat JDBC
–New JDBC pool for Tomcat, for highly concurrent systems
• But we have kept the pool size at 8
–So obviously we will still have locks
–Making it bigger hinders performance
• If the database was on a separate server, we would use a bigger number
• The database is always the problem
• [ Code ]
28

29. Second culprit : Webjars
• WebJars are client-side libraries (JQuery, etc...) packaged as Jar
files
–http://www.webjars.org/
–Those libraries are loaded from the classloader
• Which does a big lock each time it is accessed
• So if you have a lot of libraries this can be a real problem
• There are many solutions to this issue
–Cache those resources with a proxy server
–Use another solution for loading those libraries (Bower ?)
–[ Code ]
29

30. Third culprit : the monitoring Aspect
• This aspect was introduced on purpose
• We should simply disable it [ Code ]
30

31. Results
• 1020 req/sec, still without any error
• Only 10% performance increase
31

32. 32
#6
JDBC
vs
JPA

33. 3 profiles are available
• Do you use Spring profiles?
• Choose between
–JDBC
–JPA
–Spring Data
33

34. Switch to JPA
• Change the profile in the web.xml file
• The application is a little bit slower
–904 req/sec
–After an initial peak, we see the application hanging and then
starting again -> looks like a GC issue
34

35. First problem : Memory
• JPA uses more PermGen, and more memory in general than
JDBC
• Let’s boost the application memory
–export MAVEN_OPTS="-XX:MaxPermSize=128m -Xms256m -
Xmx256m"
–1103 req/sec ! Better than JDBC !
• That’s a different result from our original blog entry
–Looks like we need more memory with JDK 7...
35

36. Second problem : SQL requests
• Run MySQL Workbench
–With JDBC we had more than 30 000 SQL requests per second
–With JPA we have 15 000 SQL requests per second
• Why isn’t performance better when we have only half as
many requests?
36

37. 37
#7
Lazy
loading

38. Lazy loading & Open Session In View
• Pet.visits is an EAGER collection
–This is often a bad idea to eagerly fetch collections
• Let’s configure it with the default, lazy-loaded mode
• To prevent the dreaded LazyInitializationException we’ll use
the Open Session In View pattern
–Not the «perfect» solution, which would require more refactoring
–Good enough in this situation, and easy to implement
–[ Code ]
38

39. Results
• 1165 req/sec, still without any error
• 10 000 SQL req/sec
• We have beaten the JDBC version!
39

40. 40
#8
Adding
cache

41. Hibernate Caching
• We have used Ehcache
• We have put Hibernate’s @Cache annotation everywhere
–Owners, Pets, Visits, etc...
41

42. Spring caching
• For more complex queries, we used Spring’s cache
abstraction
–Adds a caching aspect around slow methods
• Allowed us to remove the very slow «LIKE» in
JpaOwnerRepositoryImpl.findByLastName()
• We could also have used Hibernate’s query cache for the
same result
• [ Code ]
42

43. Results
• 1240 req/sec, still without any
error
• But can we do better?
43

44. 44
#9
Extreme
testing

45. Extreme testing
• If we look at the JMeter report, we have reached JMeter’s
limits
–Adding more users (=threads) isn’t doing a big difference
• The solution is to do more with the existing threads
–Removed the counter
–Decreased the ramp-up period to 5 seconds
–Run without any listeners (or in command-line)
45

46. Results
• We achieved 3000 req/sec
• Memory stayed stable
–We even let it run until we had 100 000 user sessions
• We had to increase the JDBC pool size to 30
–Still a small number of connexions
–Pushing it to 40 decreases performance
46

47. 47
Final
notes

48. Does JDK 7 helps?
• Tested the new G1 Garbage Collector
–export MAVEN_OPTS="-XX:MaxPermSize=128m -Xms256m -
Xmx256m -XX:+UseG1GC"
–3% to 5% slower
–Needs a bigger heap to work correctly
–Doesn’t do any miracle
48

49. Does the new MacBook Pro helps?
• Compared to my original blog posts, with my old MacBook
–We still have the memory issues (until solved)
–We still have the HTTP errors (until solved)
–Results are 10% to 20% better until the last,
«extreme testing» phase
• If you write inefficient code, a faster
CPU and a SSD won’t save
you
49

50. 50
#1 Put the
application in
production
#2 Create a
JMeter test
#3
Profile
#4 Tune the
application
server
#5 Remove
locks
#7 Lazy loading
#6 JDBC vs JPA
#8 Add cache

51. 51
Questions
&
Answers
@juliendubois
Late question?