Weitere ähnliche Inhalte Ähnlich wie 豆瓣 Qcon2009 Beijing Ähnlich wie 豆瓣 Qcon2009 Beijing (20) Kürzlich hochgeladen (20) 豆瓣 Qcon2009 Beijing1. Text
2009.4
hongqn@douban.com
3. • 2.8M 1/4
•
• 20M / 500~600/sec
• 23 PC (1U*15/2U*8)
• 12
• 38G memcached
4. • 1U (frodo)
• AMD Athlon 64 1.8GHz
• 1G 160G SATA*2
• Gentoo Linux
• MySQL 5
• Quixote (a Python web framework)
• Lighttpd + SCGI (shire)
• Memcached (!)
5. Internet
Lighttpd
SCGI
App
FS
MySQL Memcache Static Files
6. Gentoo Linux
•
• emerge mysql
• ebuild patch
•
•
• GLSA(Gentoo Linux Security Advisories)
7. MySQL
• The world’s most popular open source database
• / ==> MyISAM
• ==> InnoDB
• Replicate for backup
8. Python
•
• Battery Included
•
•
• CPUG: http://python.cn/
9. Quixote
• REST URL
• Django, TurboGears, Pylons
ZOPE
• http://www.douban.com/subject/1000001
# luz/subject/__init__.py
def _q_lookup(request, name):
subject = get_subject(name)
return lambda req: subject_ui(req, subject)
# luz/subject/subject_ui.ptl
def subject_ui [html] (request, subject):
site_header(request)
“<h1>%s</h1>” % subject.title
site_footer(request)
10. Lighttpd
•
• SCGI
• SCGI: FastCGI
Quixote
• 80 lighttpd
SCGI localhost
Quixote
11. Memcache
• MySQL
• libmemcache python Pyrex
python 3x+
def get_subject(subject_id):
subject = mc.get(‘s:’+subject_id)
if subject is None:
store.farm.execute(“select xxx, xxx from subject where id=%s”,
subject_id)
subject = Subject(*store.farm.fetchone())
mc.set(‘s:’+subject_id, subject)
return subject
13. • 1U
• pippin meriadoc ( merry)
• , 4G 250G SATA*3
•
• IP DNS
IP -_-b
• frodo
(subversion, trac, etc...)
14. Internet
Lighttpd (#$) DNS
HTTP
SCGI Proxy
App Lighttpd (!")
FS
Static Files Memcache MySQL
15. •
• innodb_buffer_pool_size
•
• IP
17. • IP(BGP)
• (arwen)
• 74G 1w SATA * 3
•
•
18. Internet
Data
Lighttpd Mining
SCGI write read
App Replicate
MySQL MySQL
Master Slave
Static Files Memcache
19. • 2M /
• IO
•
, etc...
•
20. • 4G 250G SATA*3
• 10000
• mod_rewrite URL
• lighttpd mod_memcache
• IO
• web
•
•
21. Internet
store.farm
App
SCGI MySQL
Master
Lighttpd Memcache Replicate
store.farmr
HTTP Proxy
WebDAV Web Service
Replicate
Lighttpd
(w/ mod_memcache)
MySQL write
Spiders Slave
Lighttpd
WebDAV
Memcache
Static Files
!"#$%
read
Data
Mining
MySQL
Slave
22. • store farmr
• replicate delay
•
•
• cache
•
•
• ...... but it works
23. replicate delay
def get_subject(sid):
sbj = mc.get(‘s:’+sid)
if sbj is None:
sbj = flush_subject(sid, store.farmr)
return sbj
def flush_subject(sid, cursor=None):
cursor = cursor or store.farm
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
def update_subject(subject, props):
store.farm.execute(“update subject ...”)
store.farm.connection.commit()
flush_subject(subject.id, store.farm)
25. • Scale Up 1U
• 16G 147G SCSI *2 + 500G SATA
• SCSI RAID-0
• MySQL Slave
• memcached
• MyISAM InnoDB
•
• Sphinx
26. Internet
Sphinx
MySQL
Web Service store.farm Master
Replicate
Lighttpd
SCGI
HTTP Proxy App
Memcache
Lighttpd
(w/ mod_memcache) WebDAV
store.farmr
Memcache
Lighttpd
Web Service
WebDAV
Static Files MySQL
Slave
Spiders
Memcache
Memcache
28. • :)
•
•
•
• 3 1U 4 32G 1T SATA * 3
• otho.douban.com lotho.douban.com
• lighttpd 1.5 with aio support
• LVS
• Scale Up: 4G -> 8G
29. Internet
www.douban.com otho.douban.com
LVS LB
(Master)
Lighttpd
Lighttpd 1.5
(w/ mod_cache)
HTTP Proxy
Lighttpd Keepalived
LVS LB
(backup)
Lighttpd
WebDAV Lighttpd 1.5
(w/ mod_cache)
Static Files
30. write
replicate !"#$%
read Data
Mining
MySQL MySQL
Master Slave
Replicate read
write
!"#$%
Data
Mining
MySQL
Slave
31. • 6.4M / (5M PV)
•
•
• CPU memcache /
32. •
• lighttpd mod_scgi round-robin
• lighttpd 1.5
• mod_proxy
• proxy.balance = fair (load based, passive
balancing)
•
• spread
33. Internet
Lighttpd
SCGI
HTTP Proxy
App Memcache
Lighttpd
HTTP Proxy
spread
HTTP Proxy
Lighttpd
Lighttpd
Log
Aggregator SCGI
spread
Lighttpd
WebDAV App Memcache
Static Files
34. • 11M / 3
•
• Sphinx
•
•
load
35. •
•
•
• Sphinx -> Xapian
• MogileFS
36. • libmemcache -> libmemcached consistent
hash memcache
• libmemcached consistent hash bug
• CPU
• libmemcached failover bug
• nginx lighttpd load balance
• spread
• nginx
37. !"
Master
replicate
replicate
%&'() !"
Slave
#$
Master read
replicate
#$ read
Data
Slave Mining
write
!" *+,-
Slave Master
replicate
*+,-
Slave write
replicate
%&'()Data
Mining
#$
Slave
38. •
• store.farm[r] -> store.get_cursor(table=‘xxx’,
ro=True/False)
def flush_subject(sid, ro=False):
cursor = store.get_cursor(table=‘subject’, ro=ro)
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
•
39. Internet
upload.douban.com otho.douban.com
signed POST form Lighttpd 1.5
Uploader
(w/ mod_cache)
HTTP redirect
FileStorage
MogileFS Tracker
Gateway
App
MogileFS MogileFS
Node Node
MogileFS
Master
40. Internet
Lighttpd SCGI App
Lighttpd HTTP Proxy
HTTP Proxy
HTTP Proxy
Lighttpd SCGI App
HTTP Proxy
Nginx
Lighttpd
HTTP Proxy
Lighttpd Lighttpd SCGI App
WebDAV
Static Files
41. • 13M /
• MogileFS
• Tracker DB
•
42. • 8
• 32G CPU
• (300G SCSI×2 + 1T SATA) × 3
• (1T SATA × 3) × 5
•6 2
• DoubanFS
43. DoubanFS
• hash
• hash hash
• Merkle Tree
• consistent hash
• WebDAV
• MogileFS 3 50
46. Internet
upload.douban.com otho.douban.com
signed POST form Lighttpd 1.5
Uploader
(w/ mod_cache)
HTTP redirect
FileStorage
Gateway
App
DoubanFS DoubanFS
Node Node
47. • 16M /
•
• DoubanFS IO
•
48. • DoubanDB
•
• MySQL
• MySQL Master
• failover
• replicate delay
49. DoubanDB
• Key-Value
• Amazon Dynamo
• set(key, value), get(key), delete(key)
• memcache
• Merkle Tree
• Consistent Hash
• TokyoCabinet
• DoubanDB
• DoubanFS 2.0 DoubanDB
51. !"
Master1
replicate
replicate
%&'() !"
Slave
#$
Master1 read
replicate
#$ read
Data
Slave Mining
write
!" *+,-
Master2 Master
replicate
*+,-
Slave write
replicate
%&'()Data
Mining
#$
Master2
52. • DoubanFS
•
•
• ngnix
• www.douban.com LVS
• RabbitMQ spread
53. •
• profile
• memcache cache
• join
•
•