This talk uses targeted testing to analyse the performance of WSGI servers used to host Python web applications.

1. Monitoring the
performance of Python
web applications
Graham Dumpleton
PyCon HK - November 2015

3. http://newrelic.com

4. http://newrelic.com

7. http://blog.newrelic.com/wp-content/uploads/rum_timeline_diagram_aligned_web_res.jpg

8. http://newrelic.com

9. http://newrelic.com

10. http://www.modwsgi.org

11. Why it matters
• Reduce transaction response times so users are
happier.
• Reduce costs by making better use of the
resources you have available.

12. Visualising traffic

13. Concurrent requests
3
2
1
Processes
Threads
1
2
3

14. Capacity utilisation
3
2
1
Processes
Threads
1
2
3

15. CPU burn (request)

16. I/O Bound - 1 Client

17. I/O Bound - 4 Clients

18. CPU Bound - 1 Client

19. CPU Bound - 2 Clients

20. CPU Bound - 4 Clients (1)
4
1
2

21. CPU burn calculation
CPU usage
CPU burn = ————
request time
CPU usage = user CPU time + system CPU time

22. Increasing concurrency
0%
25%
50%
75%
100%
0 secs
3 secs
6 secs
9 secs
12 secs
1 2 3 4 5 6 7 8 9 10
Request time CPU time (request) CPU burn (request)
Concurrent requests

23. CPU burn (process)

24. CPU Bound - 4 Clients (2)
4
1
2

25. 100% CPU burn
0%
40%
80%
120%
160%
0 secs
3 secs
6 secs
9 secs
12 secs
1 2 3 4 5 6 7 8 9 10
Request time CPU time (request) CPU burn (request)
CPU burn (process)
Concurrent requests

26. 25% CPU burn
0%
40%
80%
120%
160%
0 secs
0.75 secs
1.5 secs
2.25 secs
3 secs
1 2 3 4 5 6 7 8 9 10
Request time CPU time (request) CPU burn (request)
CPU burn (process)
Concurrent requests

27. Global interpreter lock

28. Poor mans threading
Waiting for I/O (thread is blocked)
Running (thread active)
Waiting for GIL
Thread 1
Thread 2
1
2
3
4
5
6

29. 100% CPU burn
4 Processes / 1 Thread
0%
40%
80%
120%
160%
0 secs
0.25 secs
0.5 secs
0.75 secs
1 secs
1 2 3 4 5 6 7 8 9 10
Request time CPU time (request) CPU burn (request)
CPU burn (process)
Concurrent requests

30. 100% CPU burn + Queue time
4 Processes / 1 Thread
0%
40%
80%
120%
160%
0 secs
0.5 secs
1 secs
1.5 secs
2 secs
1 2 3 4 5 6 7 8 9 10
Request time CPU time (request) CPU burn (request)
CPU burn (process) Queue time (max)
Concurrent requests

31. Reaching capacity
4 Clients ==> 4 Processes / 1 Thread
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4

32. 5 clients ==> 4 Processes / 1 Thread
Capacity reached
Delayed
1
1
1
1
1
2
2
2
2
2
3
3
3
3
3 4

33. All requests are not the same

34. Don’t trust benchmarks

35. Is there an answer?

36. I/O vs CPU
• I/O bound request handlers.
• Okay to use multiple threads.
• CPU bound request handlers.
• Better to use multiple processes.
• Restrict processes to single threads, or at most
two if requests have very short response time.

37. I/O and CPU
• Use no more than 3 to 5 threads per process.
• Use a small number of processes.
• Watch the CPU utilisation of processes.
• Be prepared to scale out to more hosts.

38. Partitioning
Proxy
CPU
I/O
CPU
Multiple threads.
Single threaded.
Multiple processes./cpu-tasks
/io-tasks

39. Daemon mode
WSGIDaemonProcess mixed
processes=3 threads=5
WSGIDaemonProcess io
processes=1 threads=25
WSGIDaemonProcess cpu
processes=5 threads=1
WSGIProcessGroup mixed

40. Request Delegation
WSGIScriptAlias / /some/path/app.wsgi
application-group=%{GLOBAL}
<Location /io-tasks>
WSGIProcessGroup io
</Location>
<Location /cpu-tasks>
WSGIProcessGroup cpu
</Location>

41. DEMO TIME
(If we have enough time)

42. Contact me
Graham.Dumpleton@gmail.com
@GrahamDumpleton
http://blog.dscpl.com.au
http://blog.openshift.com