An updated version of the previous IPW2008 My Opera scalability talk, presented at the Nordic Perl Workshop in Oslo, April 16th, 2009.

1. my.opera.com scalability v2
Nordic Perl Workshop ~ Oslo 2009
cosimo streppone <cosimo@cpan.org>

4. Users (k)
2.500
1.640
887
430
257 205
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

5. Users (k)
Servers
Dyn req/s
2.500
1.640
887
257 205 430
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

7. espenao> proxys all over. MOC is dying?
hkmoen> did MOC just die?
...
fred joined #myopera
fred> lots of proxys...
fred left #myopera
...
cosimo> WTF? ok, looking into it

8. static avatars
(or “put your http servers at work”)

10. new storage subsystem
pools, servers
fault tolerance, redundancy
webdav, http, ftp,
scp, mogilefs?, ...

11. user uploads use case
# Create resource object for avatar
my $res = MyOpera::Storage::Resource::Avatar->new(
owner => '{userid}',
content => '{binary data}',
);
# Main storage subsystem handle
my $storage = MyOpera::Storage->new();
# Upload on pools of servers all at once
my $ok = $storage->upload($res);

12. resources
(user uploads, binary blobs, ...)
pools, servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

13. package HTTP::DAV;
...
#
# Retrieves info about a DAV url
#
# depth=0 avoids descending into dirs
#
sub propfind {
my ($self, $url, $depth) = @_;
$depth ||= 1;
# ...
}

14. new cool static URLs
http://static.myopera.com/
/avatars/pool1/
/dL/rLA/vdUxYn14kfe342QEM/
/<user-id>_s.jpg
crappy CGI-style URLs
http://my.opera.com/<user>/avatar.pl

15. /* results */
saved ~500k backend req/day
browser cache used!

16. shared-nothing
architecture
(“zero tolerance for sharing”)

17. lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store

18. dogpile effect
(or the cache “storms”)

19. nfs-based file cache
/var/cache/myopera/
/var/cache/myopera/feeds/
/var/cache/myopera/blogs/
...
/var/cache/myopera/user/
/var/cache/myopera/user/1
/var/cache/myopera/user/10
/var/cache/myopera/user/11
/var/cache/myopera/user/2
...
(another 2 million entries)
...
/var/cache/myopera/user/1999999
(on ext3)

20. soft counters
use MyOpera::Counter;
my $counter = MyOpera::Counter->new({
key => “blog.access-$entry”,
sync => sub {
my ($value) = @_;
# Real table update code
...
}
});
$counter->inc();

21. dml statistics
[DML] time=1237308152, user=,
url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,
name=XWA::User, variable=active, type=module,
elapsed=0.068473, host=my.opera.com
[DML] time=1237308152, user=, url=/community/,
name=XWA::User, variable=, type=module, elapsed=0.015935,
host=my.opera.com
[DML] ...

22. top time-intensive modules
XWA::XMLProd::User::Sidebar 2024.919s (27.2%, 0.28 s/call)
XWA::User 1778.445s (23.9%, 0.09 s/call)
XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)
XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)
XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)
XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)
XWA::Skins 180.486s ( 2.4%, 0.49 s/call)
XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)
XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)
XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)
XWA::Albums 93.740s ( 1.3%, 2.04 s/call)
XWA::Journals 92.390s ( 1.2%, 2.37 s/call)

23. most called modules
XWA::User 18784 calls 28.9%
XWA::XMLProd::User::Sidebar 7186 calls 11.1%
XWA::User::Spotlight 6439 calls 9.9%
XWA::Util::OneTimeKey 5576 calls 8.6%
XWA::User::Journal 4681 calls 7.2%
XWA::OIVWrapper 3974 calls 6.1%
XWA::User::Comments 3890 calls 6.0%
XWA::User::Album 1881 calls 2.9%
XWA::User::Ratings 1500 calls 2.3%
XWA::User::Picture 1485 calls 2.3%
XWA::XMLProd::MemberCount 611 calls 0.9%
XWA::XMLProd::Queue 589 calls 0.9%
XWA::Spotlight 569 calls 0.9%
MyOpera::DML::FastFacts 567 calls 0.9%
XWA::XMLProd::Variables 562 calls 0.9%

24. slowest modules
XWA::User::Journal::Search 20.32s (11 calls)
XWA::User::JournalArchive 4.43s (36 calls)
XWA::User::GeneratePassword 3.76s (3 calls)
XWA::XMLProd::Link::Bookmarks 3.45s (12 calls)
XWA::Fans 3.37s (10 calls)
XWA::User::ForumLatestFriends 3.36s (11 calls)
XWA::User::ForumNew 2.42s (3 calls)
XWA::Journals 2.37s (39 calls)
XWA::Albums 2.04s (46 calls)
XWA::XMLProd::UserSearch 1.87s (2 calls)
XWA::Jahoo 1.83s (46 calls)
XWA::Forum 1.64s (49 calls)
XWA::XMLProd::Link::Trash 0.86s (2 calls)
XWA::XMLProd::LocationSearch 0.72s (43 calls)
XWA::User::ForumSearch 0.67s (78 calls)

25. ?questions?

26. low level TCP/IP “throttling”
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- x.x.x.x/x anywhere
limit: avg 20/sec burst 5
ACCEPT all -- y.y.y.y/x anywhere
limit: avg 20/sec burst 5
DROP all -- x.x.x.x/x anywhere
DROP all -- y.y.y.y/y anywhere

27. full-page caching
- limiting “barrier”
- url hotlist

28. mod_cband

29. Web site monitoring,
in case the live demo fails
:-)