Scaling Twitter - Slides for a talk presented at the SDForum Silicon Valley Ruby Conference 2007 on Twitter's challenges scaling Rails.

1. Big Bird.
(scaling twitter)

2. Rails Scales.
(but not out of the box)

3. First, Some Facts
• 600 requests per second. Growing fast.
• 180 Rails Instances (Mongrel). Growing fast.
• 1 Database Server (MySQL) + 1 Slave.
• 30-odd Processes for Misc. Jobs
• 8 Sun X4100s
• Many users, many updates.

7. Joy Pain
Oct Nov Dec Jan Feb March Apr

8. IM IN UR RAILZ
MAKIN EM GO FAST

9. It’s Easy, Really.
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse

10. It’s Easy, Really.
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse
6. Profit

11. the
more
you
know
{ Part the First }

12. We Failed at This.

13. Don’t Be Like Us
• Munin
• Nagios
• AWStats & Google Analytics
• Exception Notifier / Exception Logger
• Immediately add reporting to track problems.

14. Test Everything
• Start Before You Start
• No Need To Be Fancy
• Tests Will Save Your Life
• Agile Becomes
Important When Your
Site Is Down

15. <!-- served to you through a copper wire by sampaati at 22 Apr
15:02 in 343 ms (d 102 / r 217). thank you, come again. -->
<!-- served to you through a copper wire by kolea.twitter.com at
22 Apr 15:02 in 235 ms (d 87 / r 130). thank you, come again. -->
<!-- served to you through a copper wire by raven.twitter.com at
22 Apr 15:01 in 450 ms (d 96 / r 337). thank you, come again. -->
Benchmarks?
let your users do it.
<!-- served to you through a copper wire by kolea.twitter.com at
22 Apr 15:00 in 409 ms (d 88 / r 307). thank you, come again. -->
<!-- served to you through a copper wire by firebird at 22 Apr
15:03 in 2094 ms (d 643 / r 1445). thank you, come again. -->
<!-- served to you through a copper wire by quetzal at 22 Apr
15:01 in 384 ms (d 70 / r 297). thank you, come again. -->

16. The Database
{ Part the Second }

17. “The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield

18. “The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield

19. “The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield

20. Too Late.

21. Index Everything

22. class AddIndex < ActiveRecord::Migration
def self.up
add_index :users, :email
end
def self.down
remove_index :users, :email
end
end
Repeat for any column that appears in a WHERE clause
Rails won’t do this for you.

23. Denormalize A Lot

24. class DenormalizeFriendsIds < ActiveRecord::Migration
def self.up
add_column "users", "friends_ids", :text
end
def self.down
remove_column "users", "friends_ids"
end
end

25. class Friendship < ActiveRecord::Base
belongs_to :user
belongs_to :friend
after_create :add_to_denormalized_friends
after_destroy :remove_from_denormalized_friends
def add_to_denormalized_friends
user.friends_ids << friend.id
user.friends_ids.uniq!
user.save_without_validation
end
def remove_from_denormalized_friends
user.friends_ids.delete(friend.id)
user.save_without_validation
end
end

26. Don’t be Stupid

27. bob.friends.map(&:email)
Status.count()
“email like ‘%#{search}%’”

28. That’s where we are.
Seriously.
If your Rails application is doing anything more
complex than that, you’re doing something wrong*.
* or you observed the First Rule of Butterfield.

29. Partitioning Comes Later.
(we’ll let you know how it goes)

30. The Cache
{ Part the Third }

31. MemCache

32. MemCache

33. MemCache

34. !

35. class Status < ActiveRecord::Base
class << self
def count_with_memcache(*args)
return count_without_memcache unless args.empty?
count = CACHE.get(“status_count”)
if count.nil?
count = count_without_memcache
CACHE.set(“status_count”, count)
end
count
end
alias_method_chain :count, :memcache
end
after_create :increment_memcache_count
after_destroy :decrement_memcache_count
...
end

36. class User < ActiveRecord::Base
def friends_statuses
ids = CACHE.get(“friends_statuses:#{id}”)
Status.find(:all, :conditions => [“id IN (?)”, ids])
end
end
class Status < ActiveRecord::Base
after_create :update_caches
def update_caches
user.friends_ids.each do |friend_id|
ids = CACHE.get(“friends_statuses:#{friend_id}”)
ids.pop
ids.unshift(id)
CACHE.set(“friends_statuses:#{friend_id}”, ids)
end
end
end

37. The Future
ve d
ti r
co
Ac
e
R

38. 90% API Requests
Cache Them!

39. “There are only two hard things in CS:
cache invalidation and naming things.”
– Phil Karlton, via Tim Bray

40. Messaging
{ Part the Fourth }

41. You Already Knew All
That Other Stuff, Right?

42. Producer Consumer
Message
Producer Consumer
Queue
Producer Consumer

43. DRb
• The Good:
• Stupid Easy
• Reasonably Fast
• The Bad:
• Kinda Flaky
• Zero Redundancy
• Tightly Coupled

44. ejabberd
Jabber Client
(drb)
Incoming Outgoing
Presence
Messages Messages
MySQL

45. Server
DRb.start_service ‘druby://localhost:10000’, myobject
Client
myobject = DRbObject.new_with_uri(‘druby://localhost:10000’)

46. Rinda
• Shared Queue (TupleSpace)
• Built with DRb
• RingyDingy makes it stupid easy
• See Eric Hodel’s documentation
• O(N) for take(). Sigh.

47. Timestamp: 12/22/06 01:53:14 (4 months ago)
Author: lattice
Message: Fugly. Seriously. Fugly.
SELECT * FROM messages WHERE
substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}

48. It Scales.
(except it stopped on Tuesday)

49. Options
• ActiveMQ (Java)
• RabbitMQ (erlang)
• MySQL + Lightweight Locking
• Something Else?

50. erlang?
What are you doing?
Stabbing my eyes out with a fork.

51. Starling
• Ruby, will be ported to something faster
• 4000 transactional msgs/s
• First pass written in 4 hours
• Speaks MemCache (set, get)

52. Use Messages to
Invalidate Cache
(it’s really not that hard)

53. Abuse
{ Part the Fifth }

54. The Italians

55. 9000 friends in 24 hours
(doesn’t scale)

56. http://flickr.com/photos/heather/464504545/
http://flickr.com/photos/curiouskiwi/165229284/
http://flickr.com/photo_zoom.gne?id=42914103&size=l
http://flickr.com/photos/madstillz/354596905/
http://flickr.com/photos/laughingsquid/382242677/
http://flickr.com/photos/bng/46678227/