12. Analyze Your Logs
⢠Look for slowest requests
⢠Verbose Logging
⢠Detailed DB logging (a small script)
Completed in 0.06763 (14 reqs/sec) | Rendering:
0.01377 (20%) |
DB: 0.00587 (8%) 2.6k, 54 rows, 7 queries |
200 OK [http://url]
13. Database Enhancements
(Reads)
⢠Index all of the keys you are going to use to
ďŹnd your data
⢠Turn on debugging and check the slowest
queries
⢠Use EXPLAIN on your SQL statements
(Some MySQL-speciďŹc enhancements)
15. Database Enhancements
(Writes)
⢠Only save the data that has changed
⢠Rails 2.1: 2x faster on some actions, due to:
⢠Partial Updates (automatic
performance)
⢠Dirty Tracking
16. Database Enhancements
(Writes)
⢠Check your model validations
⢠Transactions w/InnoDB
17. Database Enhancements
(Writes)
⢠Bulk Writes with AR-Extensions
⢠8-12x faster on InnoDB
⢠8-40x faster on MyISAM*
⢠Master/Slave Replication
⢠Master DB for writes with replication to
read slaves
⢠Distributes load to multiple machines
*40x if you donât use validations
18. Useful Plugins
⢠ActiveRecord Extensions
⢠Bulk writes
⢠Dr. Nicâs magic_multi_connections
⢠OfďŹoad reads/writes to another DB
⢠Single-Write Multi-Read (SWMR)
⢠Our plugin to write to one DB & use
multiple read slaves
19. Hardware
⢠Cache everything, so lots of RAM
⢠Make your reads/writes fast
⢠RAID helps:
RAID 10 - faster writes
RAID 5 - faster reads
20. Hardware
⢠Minimize hits to your Rails Stack
⢠Cache images or use a fast web server
⢠Move all your routes.rb routing logic to
dedicated hardware (load balancer)
21. Hardware
⢠Keep App & DB on separate servers
⢠Infrastructure speciďŹc to itâs purpose
⢠Easy to isolate & debug bottlenecks:
Graph from Cacti
22. Load Testing
⢠Test the path a user will take
⢠One Tool: JMeter
⢠Session and full hardware stack testing
⢠jmeter-server
⢠Distributed testing
⢠Test from any computer @ ~1500
threads per box
23.
24. Load Testing
⢠Youâll always ďŹnd new problems under
load
⢠We discovered issues with lots of
new users: checking login twice
⢠Simple queries (especially joins)
became slow
⢠Tweaking how often we wrote scores
to the database, and how many at a
time
25. Load Testing
⢠The type and number of web servers*
is critical. Quick comparison:
Mongrel Ebb Thin
â˘Stable â˘Some bugs â˘Almost as stable
â˘RAM & CPU heavy â˘RAM & CPU light as Mongrel (it uses
â˘Slowest per action â˘Fastest per Mongrelâs parser)
â˘Lightest hardware
action (uses C for
heavy lifting) load
â˘Almost as fast as
Ebb
*Yeah, you still need a web server like Lighttpd or Nginx to handle your static ďŹles and proxy to your Rails backends
26. Load Testing
Using 18 Mongrels
# of questions
1 2 3 4 5 6 7 8 9 10
minutes
Gruff graphs from our
custom user actions
27. Load Testing
Using 18 Ebbs
# of questions
1 2 3 4 5 6 7 8 9 10
minutes
949 questions / minute: 40% faster than mongrel!
28. When You Need a
Little Help
⢠After all of that performance work...
⢠Servers were crashing spontaneously
⢠Nothing was apparent in the logs
⢠Sleepless nights...
31. What Was the
Problem?
⢠Post requests on older browsers (Firefox
1.5, Navigator 7)
⢠Killed our backends, and as each failed, it
passed on the request to another server
⢠Solution: switch from Nginx to Lighttpd
32. A Summary
⢠200 (scaling) 5000 simultaneous users
a 25x performance gain
⢠Achieved by adding plugins, custom
Memcached methods, optimizing the DB,
changing to ebbs, and numerous other code
tweaks (and a slight hardware upgrade)
33.
34. Any Questions:
Josh Schwartzman
Lead Developer, Newzwag
josh@newzwag.com
35. Resources
â˘Active Record Extensions:
http://www.continuousthinking.com/are/activerecord-extensions-0-0-5
â˘Detailed DB Logging: http://gurge.com/blog/2006/11/09/rails-sql-logging-improvements/
â˘Ebb: http://ebb.rubyforge.org/
â˘God (Monitoring): http://god.rubyforge.org/
â˘Gruff Graphs: http://nubyonrails.com/pages/gruff
â˘Jmeter: http://jakarta.apache.org/jmeter/
â˘Magic Multi Connections: http://magicmodels.rubyforge.org/magic_multi_connections/
â˘Seattle.rbâs Memcache Client: http://github.com/ďŹveruns/memcache-client/tree/master
â˘Thin: http://code.macournoyer.com/thin/