Are you tired of java.lang.OutOfMemoryError: PermGen space? Then this talk is for you! We'll begin with a crash course in the Java memory model in order to understand what the error message means. Then we'll look at different causes of the error and how to avoid them. We may glance at a few interesting mistakes from the Open Source world. Last but not least you'll learn how you can get rid of java.lang.OutOfMemoryError: PermGen space once and for all.

1. Mattias Jiderhamn – java.jiderhamn.se
Join the war on
Mattias Jiderhamn
ClassLoader leaks

2. Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError:
PermGen space
:-(
Metaspace

3. Mattias Jiderhamn – java.jiderhamn.se
Local dev env

4. Mattias Jiderhamn – java.jiderhamn.se
Continuous Deploy
.war
Crashing

5. Mattias Jiderhamn – java.jiderhamn.se
Agenda
What does it mean?
Why does it happen?
Where does it leak?
How to avoid?
OSS examples
Training!

6. Mattias Jiderhamn – java.jiderhamn.se
What?

7. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Stack
Heap
PermGen /
Metaspace

8. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Per thread
Local variables and
method parameters
Stack

9. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Heap
Young generation
Old generation /
tenured space
Eden
space
Survivor
space
Object instances

10. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Permanent Generation
java.lang.Class instances
etc
Named before JEE and
class unloading
Renamed Metaspace in Java 8
Aka Method area
PermGen /
Metaspace

11. Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError:
PermGen space / Metaspace
= Too many classes are loaded
What?

12. Mattias Jiderhamn – java.jiderhamn.se
Why?

13. Mattias Jiderhamn – java.jiderhamn.se
Reason for OOME
1. Your application is too large
-XX:MaxPermSize=256M
Java 8: Metaspace auto increase by default
2. java.lang.Class instances could not
be garbage collected after redeploy
java.lang.OutOfMemoryError:
PermGen space / Metaspace

14. Mattias Jiderhamn – java.jiderhamn.se
Reference types
• Strong (i.e. normal) reference
–Never GC:ed if reachable
• Soft reference
–GC:ed before OutOfMemoryError
• Weak reference (WeakHashMap)
–GC:ed when no Strong or Soft refs
• Phantom reference
–… won’t prevent GC

15. Mattias Jiderhamn – java.jiderhamn.se
Example
Foo
WeakHashMap
Map m = new WeakHashMap();
Foo myFoo = new Foo();
m.put(myFoo, ”bar”);
myFoo = null;

16. Mattias Jiderhamn – java.jiderhamn.se
GC reachability
GC roots

17. Mattias Jiderhamn – java.jiderhamn.se
Redeploy
Application Server
ClassLoader
app.war
ClassLoader
app.war’
ClassLoader
app.war’’

18. Mattias Jiderhamn – java.jiderhamn.se
ClassLoader refs
ClassLoader
Class ClassClass
Instance

19. Mattias Jiderhamn – java.jiderhamn.se
How leaks happen
Application Server
ClassLoader
Class ClassClass
Instance
GC root

20. Mattias Jiderhamn – java.jiderhamn.se
Where?

21. Mattias Jiderhamn – java.jiderhamn.se
Application Server
• Application Server bugs
• Logging frameworks
– Apache Commons Logging
Unless LogFactory.release()
– Log4j - some configs
Unless LogManager.shutdown()
– java.util.logging custom Level
• Bean Validation API (JSR 303)
• Unified Expression Language (javax.el)
?

22. Mattias Jiderhamn – java.jiderhamn.se
GC roots
• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)
• Live thread
– Stack – local vars, method params
–java.lang.Thread instance
• Object held as synchronization monitor
• JNI references
• JVM specials…

23. Mattias Jiderhamn – java.jiderhamn.se
System classes
• java.sql.DriverManager
• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom
security Provider, custom MBeans, custom
ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller

24. Mattias Jiderhamn – java.jiderhamn.se
DriverManager

25. Mattias Jiderhamn – java.jiderhamn.se
DriverManager
Application Server
app.war
mysql-jdbc.jar
JRE
DriverManager
ClassLoader
com.mysql.jdbc.Driver
1) …
… or 2)
DriverManager.deregisterDriver(…)

26. Mattias Jiderhamn – java.jiderhamn.se
Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}

27. Mattias Jiderhamn – java.jiderhamn.se
Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer

28. Mattias Jiderhamn – java.jiderhamn.se
Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}

29. Mattias Jiderhamn – java.jiderhamn.se
Stopping threads
public class MyThread extends Thread {
public void run() {
while(true) { // Bad idea!
// Do something
}
}
}

30. Mattias Jiderhamn – java.jiderhamn.se
Stopping threads
public class MyThread extends Thread {
private boolean running = true;
public void run() {
while(running) { // Until stopped
// Do something
}
}
public void shutdown() {
running = false;
}
}
private volatile boolean running = true;
Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot

31. Mattias Jiderhamn – java.jiderhamn.se
Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal

32. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
WeakHashMap<Thread, ?>
ThreadLocal
Thread Foo

33. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocal JavaDoc:
”Each thread holds an implicit
reference to its copy of a thread-
local variable as long as the thread
is alive and the ThreadLocal
instance is accessible; …”

34. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
put()
set()

35. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
Pooled threads:
•Threads may outlive ClassLoader
•ThreadLocal → ThreadGlobal!
(?)

36. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
ClassClassLoader
static
Class

37. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
ClassClassLoader
Stale entry

38. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap JavaDoc:
”However, since reference queues
are not used, stale entries are
guaranteed to be removed only
when the table starts running out
of space”

39. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
•Custom value (incl references)
–static ThreadLocal = leak
–otherwise = unpredicted GC
•Custom ThreadLocal = no leak

40. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
try {
myThreadLocal.set(foo);
…
}
finally {
myThreadLocal.remove();
}
Always clear your ThreadLocals!

41. Mattias Jiderhamn – java.jiderhamn.se
Known offenders
Apache ActiveMQ
Apache Axis
Apache Batik
Apache Commons Pool / DBCP
Apache CXF
AWT Toolkit
Bean Validation API / JSR 303
CGLIB (Hibernate / Spring / JBoss /
Apache Geronimo)
dom4j
EclipseLink
GeoTools
Google Guice
Groovy
GWT / javax.imageio
Hessian
iCal4J
Infinispan
IntrospectionUtils
JarURLConnection
Java Advanced Imaging (JAI)
Javassist
Java Cryptography Architecture (JCA)
Java Server Faces 2
javax.security.auth.Policy/login.Configuration
JGroups
Logback
JAXB
Mozilla Rhino
MVEL
OpenOffice Java Uno RunTime (JURT)
Oracle JDBC
RMI
Serialization (<= JDK 1.4.2)
Spring framework
Unified Expression Language / javax.el
URLConnection + HTTP 1.1
XML parsing (DocumentBuilderFactory)

42. Mattias Jiderhamn – java.jiderhamn.se
How?

43. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
• Application server independent
– Version 2 core module is pure Java SE!
• Covers a lot more than Tomcat
– System class references avoided/cleared
– Threads are stopped
– ThreadLocals are cleared
– Known offenders handled
• Logs warnings
• Apache 2 licensed
– You may modify and redistribute

44. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
Zero runtime dependencies
No required config
Maven pom.xml (Servlet 3.0+):
<dependency>
<groupId>se.jiderhamn.classloader-leak-prevention</groupId>
<artifactId>classloader-leak-prevention-servlet3</artifactId>
<version>2.0.2</version>
</dependency>

45. Mattias Jiderhamn – java.jiderhamn.se
Tomcat
Bugzilla #48895
ThreadLocalMap
Thread
Cleanup
Thread
remove()
Unsafe!
get()
set()
remove()
Removed in 6.0.27
Tomcat 7.0.6+ renews the thread pool
whenever an application is redeployed.
Disposable threads for lifecycle events.
Bugzilla #49159

46. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
set(null) ThreadLocalMap
Entry
Thread
ThreadLocal Foo
Stale
Cleanup
Thread

47. Mattias Jiderhamn – java.jiderhamn.se
Open Source
examples

48. Mattias Jiderhamn – java.jiderhamn.se
Bean Validation API
Version 1.0.0.GA
javax.validation.Validation
Application Server
validation-api-1.0.0.GA.jar
app.war
hibernate-validator.jar

49. Mattias Jiderhamn – java.jiderhamn.se
javax.el API
javax.el.BeanELResolver

50. Mattias Jiderhamn – java.jiderhamn.se
Apache CXF
org.apache.cxf.transport.http.CXFAuthenticator
CXF-5442

51. Mattias Jiderhamn – java.jiderhamn.se
OpenOffice JURT
com.sun.star.lib.util.AsynchronousFinalizer

52. Mattias Jiderhamn – java.jiderhamn.se
Training!
Gear up!

53. Mattias Jiderhamn – java.jiderhamn.se
Free tool
Eclipse Memory Analyzer
aka MAT
eclipse.org/mat

54. Mattias Jiderhamn – java.jiderhamn.se
Heap dump
Run application server with
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/temp

55. Mattias Jiderhamn – java.jiderhamn.se
Links
Join the fight!
java.jiderhamn.se @mjiderhamn
github.com/mjiderhamn
• ClassLoader Leak Prevention Library
• Step by step guide: acquire heap
dump and analyze for leaks using
Eclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports
• Submit reports and pull requests!
???