This document discusses various techniques for optimizing CPU and memory usage in Zope and Plone applications. It begins by outlining some "golden rules" like using efficient algorithms, C code when possible, and avoiding duplicate operations. Later sections provide tips for profiling code to locate inefficient areas, using caches to improve performance, and dealing with potential conflicts from concurrent transactions. The goal is to remind developers of optimization best practices and provide actionable advice for writing faster Plone code.

1. The golden rules
Locating CPU usage
CPU optimization
Memory optimization
Massive sites deployment
How to minimize CPU and memory usage of
Zope and Plone applications
Gaël LE MIGNOT – Pilot Systems
October 2007
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

2. The golden rules
Locating CPU usage
CPU optimization
Memory optimization
Massive sites deployment
Plan
Use catalog and product
The golden rules
1
code
Use efficient algorithm
Using caches
Use C code
The hidden devil:
Do things once
conflict errors
The 90-10 or 80-20 law
Memory optimization
4
Locating CPU usage
2
Locating memory usage
Using decorators
Optimizing memory
Issues and tips about
benchmarks Massive sites deployment
5
Using ZEO pro and cons
DeadlockDebugger Squid caching overview
CPU optimization Site exporting strategies
3
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

3. The golden rules
Locating CPU usage
CPU optimization
Memory optimization
Massive sites deployment
Why this talk ?
The current situation
Plone is very powerful
But Plone is slow !
High-level applications developers tend to forget about
optimization
The goals of this talk
Remind you the golden rules
Provide tips, tricks and way to write faster Plone code
Consider both CPU and memory usage
A quick coverage of deployment solutions, the work-around
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

4. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
Oh, no, not maths !
Algorithm is the key
Efficient algorithm is the key to fast programs
The gains for efficient algorithms are in powers of ten
It is not obsoleted by high-level languages !
Program complexity
A general approximation of speed for large datasets
Counts the number of times you execute the core loop
Loop twice: O(n2 ), loop thrice: O(n3 )
Remember: sorting is O(n ln(n)), list lookup O(n), btree
lookup O(ln(n)), dict between O(1) and O(n), usually
considered as O(ln(n))
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

5. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
The test case
Presenting the test case
We have two lists of words
We want words in both lists
I used Debian’s french and english wordlists
Comparable uses in Plone
Finding users in two groups
Handling tags (for tag clouds, for example)
Automatically finding related documents
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

6. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
The test program
The naive version
french = open(quot;frenchquot;)
english = open(quot;englishquot;)
french = list(french)
english = list(english)
words = [ word for word in french
if word in english ]
print len(french), len(english), len(words)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

7. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
The first results
Big dataset
French words: 139704, English words: 98326
Program took 7m52s; replace list with set: 0.17s
Speed-up factor: 2700x
More reasonable dataset
At 1000 words: 140ms and 4.1ms; speed-up: 34x
At 100 words: 1.7ms and 0.32ms; speed-up: 5x
Morality
Use an efficient algorithm; correct datatype is often enough
Don’t forget Python’s powerful set module !
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

8. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
Use C code: trust Python
The theory
On the other side, some people like to hand-optimize code
In addition to being less readable, it’s often slower
Use C code when you can; trust Python’s standard library
Applying to the previous program
Our lists are already sorted
We can save time, and have a O(n) algorithm
But the sets version is still faster: 0.17s, compared to 0.21s
for my manually optimized one (that’s 23% faster)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

9. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
My beautiful hand-crafted code
french = list(open(quot;frenchquot;))
english = list(open(quot;englishquot;))
pos = 0
maxpos = len(english) - 1
words = []
for word in french:
while (word > english[pos]) and (pos < maxpos):
pos += 1
if word == english[pos]:
words.append(word)
print len(french), len(english), len(words)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

10. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
Do things once
The theory
Avoid doing the same thing many times
Use local variables to store results
In a Zope context, it’s even more true with acquisition
And in restricted code, with security checks, even more !
Applying to the previous program
Move the maxpos from before to inside the loop
We go from 0.21s to 0.23s, that’s almost 10% slower.
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

11. The golden rules
Use efficient algorithm
Locating CPU usage
Use C code
CPU optimization
Do things once
Memory optimization
The 90-10 or 80-20 law
Massive sites deployment
The last golden rule
The last golden rule
It states that 10% of the code takes 90% of the ressources
The lighter version is 20% takes 80% of the ressources
It’s already used in Zope: some parts, like ZODB or
security, are implemented in C
But it applies to your code too !
This rule is also a nice transition. . .
How to detect those ?
This will be our next part !
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

12. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
The easy way: using decorators
The idea
Write a simple decorator printing the time taken by its
function execution
Use it around functions you want to test
How to get more accurate values
On Unix systems, you can use resource.getrusage
It’ll give you user-space and kernel-space CPU usage
The limits
You need to decorate all your functions
It’ll consider all threads, not just your own
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

13. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
System, user or real ? ... I’ll say blue !
System and user time
Only account for CPU time used by your Zope process
System is the time spent in kernel-mode, like IO, thread
handling, . . .
System doesn’t account for IO waiting
User is the time spent in your code, Python, Zope, Plone
Real-life clock
Is disturbed by other processes
But accounts for time spent waiting for IO (disk spinning,
ZEO server answer, swaping pages in and out, . . . )
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

14. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
Schroedinger cat will bite us !
Benchmarking affects your application
You can’t observe a process (be it a cat or a Zope) without
disturbing it
getrlimit, time and logging uses CPU time
But if you don’t abuse, the cost stays very low
Benchmarking moves you away from real-life
Unix doesn’t provide per-thread accounting
Benchmarking a full multi-threaded system is unreliable
Running a single thread is going away from real-life
It can change swapping or ZEO access
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

15. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
Beware the power of chaos
Zope is a complex system
It contains cache and lot of other stateful devices
Running a bench twice can lead to different results
The fresh load strategy
Do all your bench on a freshly started Zope
That’ll be the worse case, when all caches are empty
The second test strategy
Start a fresh Zope
1
Run the code once, discard the result
2
Run it again, once caches are full
3
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

16. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
The quick and dirty profiler
What is DeadlockDebugger ?
A useful package listing the current state of all threads
Its goal was to locate deadlocks
But you can use it to find where your code spent its time
It is Free Software, while the pofiler module is not
The magical formula
~ $ while true; do
time >> bench ;
lynx -dump $DEADLOCK_DEBUG_URL >> bench ;
done
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

17. The golden rules
Locating CPU usage Using decorators
CPU optimization Issues and tips about benchmarks
Memory optimization Using DeadlockDebugger
Massive sites deployment
Final remarks
Don’t disturb the beast
When you do benchmarks, try to keep your computer
otherwise idle
That includes between two bench !
Running Firefox or OpenOffice will screw up your memory
cache
Don’t mix chemicals
The DeadlockDebugger trick is highly intrusive
It’ll give you a good overall view, but is not very accurate
Don’t mix it with serious benchmarking
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

18. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
The ABC of efficient Zope code
Use the catalog
It prevents loading objects from the ZODB
Unpickling is slow, even more when using ZEO
But don’t abuse from it, either
Use product code
Zope security checks have a cost
Page Templates and Python Scripts are slow
Content types, tools, Zope3 code and external methods
are fast
But be careful not to create security holes !
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

19. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
A good cache will save your life
How to do it ?
Cache the results of (moderatly) slow methods
You can write a cache decorator
Cache tricks and tips
Don’t store Zope objects, but simple Python types
You can use the parameters as the key
Don’t forget the context
Consider wisely what to use as the key
For example, roles or userid ?
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

20. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
The evil twin of caches: expiry
Automatic expiry
Real-life time based
Maximal number of objects
LRU or scoring ?
Manual expiry
Easy and ugly: restarting the Zope
Including a flush button somewhere
Semi-automatic expiry
Quite complex to handle
But the most efficient
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

21. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
The advertising page
GenericCache
A GPLed pure Python module
It provides a decorator and a class
Supports custom key marshallers
Supports several expiry schemes (time, LRU, manual)
Can easily support semi-automatic expiry
Using it in Zope
You need to write your own key marshallers
You can integrate the flushing in your UI
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

22. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
Why and how do they happen ?
Threads and transactions
Nothing is written to the ZODB until the end
But two threads can change the same object at the same
time
That is a conflict: Zope will raise an error
Zope default behaviour
Zope will then restart the transaction
The user will not see anything
But the processing will be done twice...
Zope will give up after three tries
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

23. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
How to limit them
Make changes local
Try to not change too many objects in the same transaction
This is a design decision, you must think about it before
It is not always possible
Commit your transaction
You can manually commit your transaction
This will lower the risk of collision
But it may also break the consistency
Use it on heavy processing, on safe points
Example: reindexing a catalog, processing a daily batch
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

24. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
How to resolve them
Some conflicts may be resolved
Most recent access date is the max of both
Counters are easily resolved
Container conflicts can sometimes be resolved too
Zope allows you to do so
With the _p_resolveConflict method
It takes three objects: old state, save state, current state
It’s up to you to do a three-way merge
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

25. The golden rules
Locating CPU usage Use catalog and product code
CPU optimization Using caches
Memory optimization The hidden devil: conflict errors
Massive sites deployment
Please draw me a sheep
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

26. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
Peeling onion’s layers
Base idea
Many layers can delay memory release
It’s hard to know how much memory is really used
Layers
Python’s garbage collector may delay release of objects
1
Libraries may recycle objects (like linked-lists items)
2
Your libc’s free may delay release to the operating system
3
The operating system may delay unmap or sbrk actual
4
operation
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

27. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
A bit of theory about swap
Swap and memory
In modern operating systems, the memory is just a cache
for disk
Swapping, unmapping binaries or releasing cache is the
same
You can have swap used without it being a problem
Working set and trashing
The problem is in frequency, not amount
During an operation, a set of pages is used, called the
“working set”
When it doesn’t fit in memory, your system is “trashing”
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

28. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
So, do I have a memory problem ?
The vmstat command
Gives regular information about virtual memory system
Exists on most Unixes, including GNU/Linux and FreeBSD
Typical invocation is vmstat 2
Most interesting columns are si and so on Linux, pi and
po on FreeBSD
More informations
It is a global command, accounting for the whole system
It also contains “normal” I/O to discs and CPU state
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

29. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
A few tips
Using the gc module
You can check for uncollectable cycles
You can ask for manual collection
You can get a list of all objects
Warning, basic types like string are not returned, you need
to walk in containers
Monkey-patching malloc
You can track malloc/free calls with ltrace
On the GNU C library, you can write hooks to malloc in C
Tools like dmalloc can provide debug logs
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

30. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
Sometimes, you get both at once
Use the catalog
Using the catalog instead of awakening objects will reduce
memory usage
It’ll also reduce disk usage, and therefore the cost of
swapping
Store big files in the filesystem
It’ll avoid polluting ZODB cache
Even better: use Apache to serve them directly
You can use products like tramline
You can just serve with Apache the BlobFile directory
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

31. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
Reference counting and big memory chunks
The simple theory
Objects are only destroyed when there is no reference
Local variables are destroyed at the end of the function
You can use the del operator to force it
Use it on big objects before any slow operation
But sometimes it’s not that simple
A reference can held in a calling function
You cannot destroy it then
You need to pass by reference
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

32. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
Let’s find a perfect culprit
The code
def process(target):
what = generate_data()
send(target, what)
def send(target, what):
encoded = encode(what)
del what
target.write(encoded)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

33. The golden rules
Locating CPU usage
Locating memory usage
CPU optimization
Optimizing memory
Memory optimization
Massive sites deployment
A solution
The code
def process(target):
what = generate_data()
params = { quot;dataquot;: what }
del what
send(target, params)
def send(target, params):
encoded = encode(params[quot;dataquot;])
del params[quot;dataquot;]
target.write(encoded)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

34. The golden rules
Locating CPU usage ZEO pro and cons
CPU optimization Squid caching overview
Memory optimization Site exporting strategies
Massive sites deployment
The two faces of ZEO
The good face of ZEO
It is very easy to set up
It makes Zope parellizable on many CPUs
It supports networking, and can allow clustering
It can do both failover and load-balancing
The bad face of ZEO
It’ll slow any ZODB access
The database itself is centralised and can be a bottleneck
Session objects will need to go through ZODB marshalling
too
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

35. The golden rules
Locating CPU usage ZEO pro and cons
CPU optimization Squid caching overview
Memory optimization Site exporting strategies
Massive sites deployment
Using a proxy-cache
Do things only once
The rendering a Plone page is complex
But many visitors will have exactly the same results
Using a proxy-cache like Squid (or Apache) can avoid
doing it many times
But that only covers anonymous users
Squid can do varying
You can use multilingual site behind a Squid
It’ll detect the headers are the serve the correct version
from the cache
Example: EADS Astrium
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

36. The golden rules
Locating CPU usage ZEO pro and cons
CPU optimization Squid caching overview
Memory optimization Site exporting strategies
Massive sites deployment
If one ZODB is not enough...
Static HTML export
Not too hard to implement, but still tricky
Removes all dynamic features like search
Exporting a site
You can copy a ZODB from a running Zope
Or you can use zexp and XML-RPC or Post
Resyncing back
It requires some hacks
But we did it on forum (PloneGossip) and event
subscription (SignableEvent)
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app

37. The golden rules
Locating CPU usage ZEO pro and cons
CPU optimization Squid caching overview
Memory optimization Site exporting strategies
Massive sites deployment
Conclusion
Plone is not doomed to be slow !
Optimization has to be part of all steps, from design to
code
Even with fast CPUs, it’s still an important issue
Thanks
To Pilot Systems team for feedback
To the Free Software community for all the tools available
The usual...
Any question ?
Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app