I gave this presentation at YAPC::NA 2012 in Madison, WI.

1. Introduction to
Performance Tuning
Perl Web Applications
Perrin Harkins

2. Find someone to
blame

3. Performance vs scalability
● Not really the same thing
● Can look the same
● We're mostly going to talk about
performance

4. Slowness
● Maybe it's the designer's fault!
● Chrome Developer Tools

5. Network

6. PageSpeed

7. Narrow it down
further

8. Create a repeatable test
● Measure progress
● Metrics
○ Requests/second
○ Time for n requests
○ Concurrency

9. ab
ab -c1 -n100 http://127.0.0.1:8080/bugzilla/buglist.cgi

10. Benchmarking 127.0.0.1 (be patient).....done
Server Software: Apache/2.2.22
Server Hostname: 127.0.0.1
Server Port: 8080
Document Path: /bugzilla/buglist.cgi
Document Length: 14562 bytes
Concurrency Level: 1
Time taken for tests: 33.816 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 1494100 bytes
HTML transferred: 1456200 bytes
Requests per second: 2.96 [#/sec] (mean)
Time per request: 338.163 [ms] (mean)
Time per request: 338.163 [ms] (mean, across all concurrent
requests)
Transfer rate: 43.15 [Kbytes/sec] received

11. Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 313 338 20.7 333 448
Waiting: 312 337 20.6 332 448
Total: 313 338 20.7 333 448
Percentage of the requests served within a certain time (ms)
50% 333
66% 339
75% 343
80% 345
90% 357
95% 374
98% 430
99% 448
100% 448 (longest request)

12. httperf
httperf --wsess=10,5,2 --rate=1
--server=localhost --port=8080
--uri=/bugzilla/buglist.cgi

13. Total: connections 50 requests 90 replies 50 test-duration 21.680 s
Connection rate: 2.3 conn/s (433.6 ms/conn, <=11 concurrent
connections)
Connection time [ms]: min 321.5 avg 2487.2 max 3884.7 median 2537.5
stddev 1002.1
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000
Request rate: 4.2 req/s (240.9 ms/req)
Request size [B]: 203.0
Reply rate [replies/s]: min 1.6 avg 2.4 max 3.0 stddev 0.6 (4
samples)
Reply time [ms]: response 883.0 transfer 3.4
Reply size [B]: header 388.0 content 14562.0 footer 2.0 (total
14952.0)
Reply status: 1xx=0 2xx=50 3xx=0 4xx=0 5xx=0
Session rate [sess/s]: min 0.00 avg 0.46 max 1.00 stddev 0.49 (10/10)
Session: avg 5.00 connections/session
Session lifetime [s]: 12.4

14. HTTP::Recorder
● Acts as HTTP proxy
● Generates Mechanize script
● HTTP::Recorder::Httperf

15. $agent->get('http://127.0.0.1:8080/bugzilla/');
$agent->follow_link(text => 'Search', n => '1');
$agent->form_name('queryform');
$agent->field('bug_status', '__open__');
$agent->field('content', '');
$agent->field('product', 'TestProduct');
$agent->click();

16. /bugzilla/ method=GET
/bugzilla/skins/standard/global.css method=GET
/bugzilla/skins/standard/index.css method=GET
/bugzilla/skins/contrib/Dusk/global.css method=GET
/bugzilla/skins/contrib/Dusk/index.css method=GET
/bugzilla/js/yui/yahoo-dom-event/yahoo-dom-event.js method=GET
/bugzilla/js/yui/cookie/cookie-min.js method=GET
/bugzilla/js/global.js method=GET
/bugzilla/skins/standard/index/file-a-bug.png method=GET
/bugzilla/skins/standard/index/search.png method=GET
/bugzilla/skins/standard/index/new-account.png method=GET
/bugzilla/query.cgi method=GET think=4
/bugzilla/skins/standard/search_form.css method=GET
/bugzilla/buglist.cgi?
query_format=specific&order=relevance+desc&bug_status=__open__&produc
t=TestProduct&content= method=GET think=6
/bugzilla/js/yui/assets/skins/sam/autocomplete.css method=GET
/bugzilla/js/yui/assets/skins/sam/calendar.css method=GET
/bugzilla/skins/standard/buglist.css method=GET
/bugzilla/skins/contrib/Dusk/buglist.css method=GET

17. Autobench

18. Profile to find out where the
time is going
● Devel::NYTProf
● Wall clock time vs. CPU time
● Use your real environment
● Multiple runs and warmup avoid
skewed results
● Let's look at one...

19. Ten bucks says it's
your database

20. Sure, every now and then you
find
● A bad regex
● A string being parsed over and over
● Massive object instantiation
● Network operations
● Disk reads

21. But mostly it's the database
● What’s slow in modern computers?
● What does most of the I/O in a typical
web app?
● Either fix your queries or cache them

22. Profiling DBI
● DBI_PROFILE=2/DBI::ProfileDumper::Apache
● Let's look at one...

23. Speeding up queries
● EXPLAIN plans
○ pt-query-advisor
● SQL generation is not for everything
● A little bit of database server tuning
○ pt-variable-advisor

24. Speeding up DBI
● Cache connections and statements
● Manage commits
● Use native bulk loading tools

25. The last resort: caching
● Cache management is a hard problem
● Code complexity
○ invalidation calls
○ dependency tracking
● Your content creators will hate it
● When you do cache, use CHI

26. A brief word about runtime
environments
● Webserver choice has a minimal effect
on performance
● Persistent daemon: mod_perl, Plack,
FastCGI
● Buffer your output
● Size-limiting or auto restarts

27. Real-world pressures

28. Flailing
● Changing things based on guesses
rather than data
● No QA
● Lots of collateral damage
● Emergency profiling

29. Benchmark::Stopwatch
my $stopwatch = Benchmark::Stopwatch->new->start;
...
$stopwatch->lap('load objects');
...
$stopwatch->lap('render template');
...
$stopwatch->lap('send response');
print $stopwatch->stop->summary;

30. Benchmark::Stopwatch
NAME TIME CUMULATIVE PERCENTAGE
load objects 1.000 1.000 14.289%
render template 3.000 4.001 42.853%
send response 2.000 6.001 28.572%
_stop_ 1.000 7.001 14.287%

31. Doubts about infrastructure
● “You're still using Foobar 1.5?! That’s
so slow!”
● Keep your head
● Get help: FAQ, Google, mailing list/IRC

32. Buying hardware
● Good idea!
● A boatload of RAM hides a multitude of
sins
● Make sure you know what the
bottleneck is

33. Further reading
● Tim Bunce's Advanced DBI slides on
CPAN
● Percona Toolkit

34. Thank you!