My PyCon Ireland 2012 talk with an analysis and description of some of the various in production parts of our stack. IE, Linux, MySQL, Memcache, Django, Apache, Solr, Django-sentry, Buildout/Fabric, statsd etc. Some discussion also of alternative choices, up and coming technology, things to watch.

1. Lessons Learnt.
An In Production,
Scaleable Django Stack
PyCon ireland - Dublin, Oct 13th-14th 2012
john.moylan@rte.ie / john@8t8.eu / @redsnapper88

2. Brief Intro to RTE.ie
Website of Ireland's national broadcaster

3. Problem
Mostly W.O.R.M, but some stuff can't be cached :(
Lot's of content
(100's GB's of HTML)
10's TB of Audio and Video
Lots of strands (News, Sport, Business, Ent., TV, Radio,
Teletext, Archives, Young Peoples, Performing Groups...)
Small team.
Try to keep things as simple and efficient as possible.
...massive growth

4. Traffic Growth
Oct 2010 V's Oct 2011
Traffic nearly doubled.

5. Flashmob Traffic
36Gb/s of Traffic. 41K Concurrent streams.
Playlists were not cacheable........
But what a great fight.

6. Static?
Legacy system published to static files using
XML/XSL Publisher.
Multiple versions per type of view.
SSI's inside SSI's inside SSI's
Republishing a strand could take days.

7. Dynamic Requirement
Evaluated some systems:
Wicket, Rails, Shale....Django
Hmmm... nice
All in package, sensible defaults. Similar
problem domain.

8. Try Before you buy
2008
Wrote prototype video upload sharing site...
(Django 0.96) and then decided to make it live.
http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/

9. Try Before You Buy
2009
Original RTÉ Player - Written in Django.
2011
Let's use Django to make the whole site
dynamic!

10. ..continued
2012
Sport, About, RTÉ Player v.2, Radio, News?,
Archives, playlists, lifestyle, SSO......and on
and on
2013..
More

11. Django Delivery
Linux
Apache with Event MPM and mod_wsgi
Mysql and MHA (tip Innodb plugin)
Solr for search and 'NoSQL'
Varnish (used to use Squid)
LVS-NAT and LVS-DR Load-balancers.
Memcache
Deploying Glusterfs

12. Code Management
Git
Redmine
Gitosis

13. Deployment
Cobbler
Fabric
Buildout

14. Monitoring
Monitor & Measure everything!
Zenoss
RRDTool
MRTG
Graphite
Statsd - mozilla's Django-statsd
Sentry
Lot's of custom monitoring

15. In House Monitoring projects
Pyvarnish
https://github.com/redsnapper8t8/pyvarnish

16. In House Monitoring Projects
Digital Dash - (open sourced soon)

17. Benchmarking
Tool agnostic but seem to use these alot.
Bonnie++
Netperf
#HTTP
Httperf
Siege

18. Performance Tips
1. Memory, memory, memory
2. Caching - in memory
3. Does it scale horizontally?
4. KISS

19. Lessons Learnt.
Read from memory whenever possible
Cache, learned this one a long time ago but
whenever we have issues it can usually be tracked
down bad design and lack of caching. Don't hit
your web servers unless you have to.
Memcache, Varnish
Long live Rest - GET's should not have side
effects. Varnish
KISS - and no resume driven development.

20. Thank you
Questions?
Also, did I mention that we are hiring?