SlideShare a Scribd company logo
1 of 18
Speeding Up Your Rails Application With MongoDB Phil Cowans CTO, Songkick.com MongoUK – June 18th 2010
SONGKICK: THE WORLD’S LARGEST CONCERT DATABASE Current version launched in June 2009 Data from over 85 ticket vendors. Lots of data: Over 100,000 upcoming events worldwide Over 1.5 million events in total. Over 100,000 setlists. Thousands of photos, posters and videos. Built using Ruby and Rails.
Thanks To: Dan Lucraft (@danlucraft) Matt Wynne (@mattwynne) ...and the rest of the Songkick engineering team.
Phil Cowans phil@songkick.com @philcowans http://www.songkick.com/users/phil

More Related Content

More from Skills Matter

Patterns for slick database applications
Patterns for slick database applicationsPatterns for slick database applications
Patterns for slick database applicationsSkills Matter
 
Scala e xchange 2013 haoyi li on metascala a tiny diy jvm
Scala e xchange 2013 haoyi li on metascala a tiny diy jvmScala e xchange 2013 haoyi li on metascala a tiny diy jvm
Scala e xchange 2013 haoyi li on metascala a tiny diy jvmSkills Matter
 
Oscar reiken jr on our success at manheim
Oscar reiken jr on our success at manheimOscar reiken jr on our success at manheim
Oscar reiken jr on our success at manheimSkills Matter
 
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...Skills Matter
 
Cukeup nyc ian dees on elixir, erlang, and cucumberl
Cukeup nyc ian dees on elixir, erlang, and cucumberlCukeup nyc ian dees on elixir, erlang, and cucumberl
Cukeup nyc ian dees on elixir, erlang, and cucumberlSkills Matter
 
Cukeup nyc peter bell on getting started with cucumber.js
Cukeup nyc peter bell on getting started with cucumber.jsCukeup nyc peter bell on getting started with cucumber.js
Cukeup nyc peter bell on getting started with cucumber.jsSkills Matter
 
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...Skills Matter
 
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...Skills Matter
 
Progressive f# tutorials nyc don syme on keynote f# in the open source world
Progressive f# tutorials nyc don syme on keynote f# in the open source worldProgressive f# tutorials nyc don syme on keynote f# in the open source world
Progressive f# tutorials nyc don syme on keynote f# in the open source worldSkills Matter
 
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...Skills Matter
 
Dmitry mozorov on code quotations code as-data for f#
Dmitry mozorov on code quotations code as-data for f#Dmitry mozorov on code quotations code as-data for f#
Dmitry mozorov on code quotations code as-data for f#Skills Matter
 
A poet's guide_to_acceptance_testing
A poet's guide_to_acceptance_testingA poet's guide_to_acceptance_testing
A poet's guide_to_acceptance_testingSkills Matter
 
Russ miles-cloudfoundry-deep-dive
Russ miles-cloudfoundry-deep-diveRuss miles-cloudfoundry-deep-dive
Russ miles-cloudfoundry-deep-diveSkills Matter
 
Simon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelismSimon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelismSkills Matter
 
I went to_a_communications_workshop_and_they_t
I went to_a_communications_workshop_and_they_tI went to_a_communications_workshop_and_they_t
I went to_a_communications_workshop_and_they_tSkills Matter
 

More from Skills Matter (20)

Patterns for slick database applications
Patterns for slick database applicationsPatterns for slick database applications
Patterns for slick database applications
 
Scala e xchange 2013 haoyi li on metascala a tiny diy jvm
Scala e xchange 2013 haoyi li on metascala a tiny diy jvmScala e xchange 2013 haoyi li on metascala a tiny diy jvm
Scala e xchange 2013 haoyi li on metascala a tiny diy jvm
 
Oscar reiken jr on our success at manheim
Oscar reiken jr on our success at manheimOscar reiken jr on our success at manheim
Oscar reiken jr on our success at manheim
 
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...
Progressive f# tutorials nyc dmitry mozorov & jack pappas on code quotations ...
 
Cukeup nyc ian dees on elixir, erlang, and cucumberl
Cukeup nyc ian dees on elixir, erlang, and cucumberlCukeup nyc ian dees on elixir, erlang, and cucumberl
Cukeup nyc ian dees on elixir, erlang, and cucumberl
 
Cukeup nyc peter bell on getting started with cucumber.js
Cukeup nyc peter bell on getting started with cucumber.jsCukeup nyc peter bell on getting started with cucumber.js
Cukeup nyc peter bell on getting started with cucumber.js
 
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...
Agile testing & bdd e xchange nyc 2013 jeffrey davidson & lav pathak & sam ho...
 
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...
Progressive f# tutorials nyc rachel reese & phil trelford on try f# from zero...
 
Progressive f# tutorials nyc don syme on keynote f# in the open source world
Progressive f# tutorials nyc don syme on keynote f# in the open source worldProgressive f# tutorials nyc don syme on keynote f# in the open source world
Progressive f# tutorials nyc don syme on keynote f# in the open source world
 
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...
Agile testing & bdd e xchange nyc 2013 gojko adzic on bond villain guide to s...
 
Dmitry mozorov on code quotations code as-data for f#
Dmitry mozorov on code quotations code as-data for f#Dmitry mozorov on code quotations code as-data for f#
Dmitry mozorov on code quotations code as-data for f#
 
A poet's guide_to_acceptance_testing
A poet's guide_to_acceptance_testingA poet's guide_to_acceptance_testing
A poet's guide_to_acceptance_testing
 
Russ miles-cloudfoundry-deep-dive
Russ miles-cloudfoundry-deep-diveRuss miles-cloudfoundry-deep-dive
Russ miles-cloudfoundry-deep-dive
 
Serendipity-neo4j
Serendipity-neo4jSerendipity-neo4j
Serendipity-neo4j
 
Simon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelismSimon Peyton Jones: Managing parallelism
Simon Peyton Jones: Managing parallelism
 
Plug 20110217
Plug   20110217Plug   20110217
Plug 20110217
 
Lug presentation
Lug presentationLug presentation
Lug presentation
 
I went to_a_communications_workshop_and_they_t
I went to_a_communications_workshop_and_they_tI went to_a_communications_workshop_and_they_t
I went to_a_communications_workshop_and_they_t
 
Plug saiku
Plug   saikuPlug   saiku
Plug saiku
 
Huguk lily
Huguk lilyHuguk lily
Huguk lily
 

Recently uploaded

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideStefan Dietze
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jNeo4j
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandIES VE
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Hiroshi SHIBATA
 

Recently uploaded (20)

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 

Mongo uk speeding up your rails application with mongodb

  • 1. Speeding Up Your Rails Application With MongoDB Phil Cowans CTO, Songkick.com MongoUK – June 18th 2010
  • 2. SONGKICK: THE WORLD’S LARGEST CONCERT DATABASE Current version launched in June 2009 Data from over 85 ticket vendors. Lots of data: Over 100,000 upcoming events worldwide Over 1.5 million events in total. Over 100,000 setlists. Thousands of photos, posters and videos. Built using Ruby and Rails.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. Thanks To: Dan Lucraft (@danlucraft) Matt Wynne (@mattwynne) ...and the rest of the Songkick engineering team.
  • 18. Phil Cowans phil@songkick.com @philcowans http://www.songkick.com/users/phil

Editor's Notes

  1. Thank you for inviting me to speak at this event.My name is Phil Cowans, and I work for a company called Songkick where I’m chief technical officer. We’re a small company based near Old Street, and this talk is about how we use MongoDB in our production systems. Before I go on, I should say that I actually had very little direct involvement in writing the software I’m about to describe – please do feel free to ask questions, but if they get too in-depth I’ll probably have to refer you to the talented members of my development team who actually did the hard work of putting this together. Our use of MongoDB is fairly simple, but it’s practical and we’ve built some interesting software to support our use case.
  2. Before we get into the tech, I’ll just give you a quick overview of what we do as a company. Songkick is a website about live music. We help our users find out about upcoming concerts by sending them personalised email alerts, and maintain an archive of information about what’s happened in the past, including photos, videos, reviews and setlists, going back to the 1950s. The site has existed in its present form for almost exactly a year, and we’ve now got well over 1.5 million concerts and festivals in the database.Our technology platform is Ruby on Rails, running on Linux and using the standard combination of MySQL, Apache etc.
  3. Here’s a typical page on Songkick (show demo)– the artist page for Sonic Youth. You can see there’s a lot of information here: Upcoming events, similar artists, photos, videos, past events and so on. I can also see similar pages for individual events, venues, cities etc., and user profiles. There’s a lot of data on the site.We want to give our users the best experience possible, which means as well as being well designed and easy to use, the pages need to be fast. We internally set ourselves the goal of making things fast enough so that the user sees something happen, i.e. The page starts to be rendered, no more than 1 second after clicking the link. This means we need to spend as little time as possible computing the HTML that we send to the user.
  4. Every event has a row of information like this.You can see that we try very hard to make the process of finding gigs as rich as possible. Every gig has headliners, supporting artists, venues, attendees, reviews, photos, videos, posters, setlists and tickets!This is great, all of this information is really useful to our users. But all this metadata comes at a cost.
  5. This is the underlying representation of the data in the raw, normalised form we store in MySQL. There are 12 tables here (in fact this is simplified, so data from more tables goes into the event row), so populating the event row from this format involves several multi-table joins. It also requires a fair amount of business logic to decide exactly what to display, and in what format.This sort of thing happens dozens of times per page, so it’s simply not possible to hit the MySQL database every time the page is rendered. The nice thing is we don’t have to – this data is presented in almost exactly the same way for all users who see it (give or take a few modifications such as highlighting which of their friends are going), and appears in the same format on the artist page, the venue page etc. We can therefore precompute the exact data required to display this fragment and cache it ready for when the page needs to be displayed.
  6. Like many frameworks, Rails uses a Model-View-Controller paradigm. Typically this means that the view accesses model objects, which encapsulate rows from a relational database.Rails does support fragment caching – i.e. it can check for a cached version of a bit of markup before executing the code to render it. Out of the box this isn’t quite what we want because:We wanted to be able to pre-populate the cache.We didn’t want to be constrained to cache fragments of completed markup for a number of reasons.Rail’s built in cache expiry is inflexible, which was a big problem.
  7. Consider a ‘document’ view of the above fragment – this is all of the data needed to render the fragment we saw before.Exactly what goes here is flexible:It can be pre-computed HTML – this is clearly fastest to render.Or, it can be a data view as shown here, which needs more work to transform it to HTML, but has the advantage that it can be reused in places where there are similar but not identical representations of the same object, and allows some customisation based on parameters only known at render time (such as the user’s location). It also has the advantage that it doesn’t need to be expired as often as the visual design changes, which does happen fairly frequently.
  8. This is exactly the sort of thing that MongoDB is good at, specifically: Schema-less which is great for our denormalized data which is changing a lot. (Schema less databases are a great fit with dynamic languages.) Pretty quick. Stores most/all of our db in RAM. Supports sharding (or close to supporting it anyway). Seems more mature than some.... Fully supported Ruby driver. (With responsive IRC and developers.)We’d also used it for some internal apps (to store and analyse web traffic stats), so were familiar with it.
  9. Architecturally, it looks like this. The Ruby classes wrapping the document representation of the page or page fragment are called presenters. The idea is that the view should be able to take the output of the presenter and construct the markup with little or no transformation logic.The presenters pull data directly from Mongo, only falling back to the models and the MySQL database if no precomputed version is available.We aren’t fully there yet, but the advantage of this approach is that we can use a mixture of models and presenters in the views as appropriate – building presenters where the code is more stable and there’s more need for high performance.
  10. This is a schematic view of the presenter beind our event row. There’s a method for each of the pieces of data shown in the HTML, so the view can just call these and substitute in the result. These call the models, so may result in complex SQL.
  11. We call the MongoDB collections which hold this data ‘silos’, and we’ve built a library to make it really easy to convert an existing class to use them. Ruby’s flexibility when it comes to metaprogramming really helps here.At the top of the class we include Silo::Store, which brings in the utility functions, and define the collection name and a lamba which constructs the ID for a given instance of the presenter (here the primary key for the underlying event object). This is used as the document’s key in MongoDB.Once we’ve done this it’s simply a matter of adding silo_method statements to indicate which of the presenter’s methods should be persisted in the silo. When we call the title or image_count methods, we’ll now look in MongoDB first, and only execute the method itself if no answer is returned. If the method is executed, the result will be stored in the silo for next time.
  12. Getting stuff into and out of the cache is of course the easy bit – the hard bit is knowing when to expire the cache.This is the list of reasons why the silo for an event row may become invalid – there are a lot of reasons such as adding media, users saying they’re going, venue’s changing name, Artist changing name, new artists being added to the lineup and so on, some of which are several steps removed from the event object itself.All of this is quite complex, so we tried to find a way to make it easy to express the expiry rules.
  13. Rails supports ‘observers’ – hooks into the ActiveRecord ORM to trigger actions when database objects are created, updated or destroyed. We could use this to expire the cache and repopulate it with new values as appropriate.The silo helper methods are able to reflect on the presenter classes, so know which methods need to be run – this helps keep things clean. There are however two big problems.Firstly, this encourages us to organise the expiry rules by the model which triggers the change, rather than by presenter being expired. This seems counter-intuitive to us.Secondly, and most importantly, Rails’ observers are synchronous – we don’t want to speed up page rendering just to slow down requests which mutate the data (and these can be very slow indeed if a lot of things have to be regenerated as a result).
  14. Fortunately, we’d already implemented asynchronous observers for other reasons. Every Create, Update and Destroy event for specific models generates an event which is published to a RabbitMQ message queue. Multiple consumers can listen to these events and act as appropriate. We use this for all sorts of things – sending welcome emails, updating activity feeds, populating users’ calendars, resizing uploaded images and so on. We have a nice Domain Specific Language, again built using Ruby metaprogramming, which makes it really easy to define message consumers in an expressive way.
  15. The daemons which handle silo expiry and regeneration are called ‘silovators’. This is a schematic of part of the code representing the expiry rules for the event listing presenter. Each block listens for a particular action on a particular object, so for example, when an Attendance is created, which happens when a user says he or she is going to a concert, we know to regenerate the silo for that event to update the list of attending users. These can be specialised to look for changes to specific fields as appropriate.
  16. That’s basically it – we observe changes to the underlying MySQL database and via RabbitMQ and the silovators pre-generate the appropriate data and put it in MongoDB. At render time, the presenter grabs the data and does a minimal amount of work to convert it to the HTML which is sent to the user.There are a few more bits and pieces. We’ve had to deal with issues such as locking to prevent concurrency issues between multiple silovator back-ends, and bulk expiration when design changes make cached HTML fragments invalid. We’ve also moved more towards caching HTML more and data less, so have built out support for post-processing of that after retrieval from the silo to customise for a specific user. Finally, we’re big on test driven development, so we’ve put together some tools to make it easy to test the expiry rules.
  17. Thanks to Dan and Matt, who built most of this software and put together an earlier version of this talk which I’ve plagerised, and everyone else on the Songkick engineering team.
  18. I’m Phil Cowans and you can contact me at these places. The code I’ve just described isn’t quite ready for widespread distribution, but if you’re interested get in touch and we should be able to share it with you.Thanks you very much for listening, and I’d be happy to attempt to answer any questions.