Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
MongoDB at fotopedia
         Timeline storage
   Our Swiss Army Knife Database
MongoDB at fotopedia

•   Context

•   Wikipedia data storage

•   Metacache
Fotopedia
•   Fotonauts, an American/French company

•   Photo — Encyclopedia

•   Heavily interconnected system : flickr,
...
First contact


•   Wikipedia imported data
Wikipedia queries

•   wikilinks from one article

•   links to one article

•   geo coordinates

•   redirect

•   why no...
Download
~ 5.7GB gzip
XML



        Geo
        Redirect
        Backlink
        Related
    ~12GB tabular data
Problem


Load ~12GB into a K/V store
CouchDB 0.9 attempt


•   CouchDB had no dedicated import tool

•   need to go through HTTP / Rest API
“DATA LOADING”



   LOADING!




   (obviously hijacked from xkcd.com)
Problem, rephrased


  Load ~12GB into any K/V store
        in hours, not days
Hadoop HBase ?

•   as we were already using Hadoop Map/
    Reduce for preparation

•   bulk load was just emerging at th...
photo by neural.it on Flickr
Problem, rerephrased
      Load ~12GB into any K/V store
            in hours, not days
  without wasting a week on develo...
MongoDB attempt
•   Transforming the tabular data into a JSON
    form : about half an hour or code, 45 minutes
    of had...
Batch   Synchronous


  Download
  ~ 5.7GB gzip



    Geo
    Redirect
    Backlink      Ruby on Rails
    Related
~12GB,...
Hot swap ?
•   Indexing was locking everything.

•   Just run two instances of MongoDB.
    •   One instance is servicing ...
We loved:
•   JSON import format

•   efficiency of mongoimport

•   simple and flexible installation
    •   just one cumbe...
Second contact

•   itʼs just all about graphes, anyway.
    •   wikilinks

    •   people following people

    •   relat...
all about graphes...

•   ... and itʼs also all about cache.

•   The application needs to “feel” faster, letʼs
    cache ...
Page fragment caching
                                                 photo by Mykl Roventine on Flickr


    Nginx SSI

...
Haiku ?

There are only two hard things
in Computer Science:
cache invalidation and naming things.



                    ...
Naming things


•   REST have been a strong design principle in
    fotopedia since the early days, and the efforts
    ar...
/en/2nd_arrondissement_of_Paris




  /en/Paris/fragment/left_col

 /users/john/fragment/contrib




  /en/Paris/fragment/...
Invalidating


•   Rest allows us to invalidate by URL prefix.

•   When the Paris album changes, we have to
    invalidate...
Varnish invalidation

•   Varnish built-in regexp based invalidation is
    not designed for intensive, fine grained
    in...
/en/Paris.*

/en/Paris

/en/Paris/fragment/left_col

/en/Paris/photos.json?skip=0&number=20

/en/Paris/photos.json?skip=13...
Metacache workflow
                                           /en/Paris.*


                    invalidation worker

    Ng...
Waw.

•   This time we are actually using MongoDB as a
    BTree. Impressive.

•   The metacache has been running fine for
...
Invalidate less
•   We need to be more specific as to what we
    invalidate.

•   Today, if somebody votes on a photo in t...
Metacache reloaded
•   Pub/Sub metacache

•   Have the backend send a specific header to
    be caught by the metacache-fee...
/en/Paris

      /en/Paris/fragment/left_col

      /en/Paris/photos.json?skip=0&number=20

      /en/Paris/photos.json?sk...
when somebody votes
       { url:/en/Paris.*, observe:photos }

               when the summary changes
       { url:/en/P...
Other uses cases

•   Timeline activities storage: just one more BTree
    usage.

•   Moderation workflow data: tiny datas...
Current situation

•   Mysql: main data store

•   CouchDB: old timelines (+ chef)

•   MongoDB: metacache, wikipedia, mod...
What about the main store ?


  •   albums are good fit for documents

  •   votes and score may be more tricky

  •   rece...
In short
•   Simple, fast.

•   Hackable: in a language most can read.

•   Clear roadmap.

•   Very helpful and efficient ...
Mongodb, our Swiss Army Knife Database
Mongodb, our Swiss Army Knife Database
Mongodb, our Swiss Army Knife Database
Mongodb, our Swiss Army Knife Database
Mongodb, our Swiss Army Knife Database
Mongodb, our Swiss Army Knife Database
Nächste SlideShare
Wird geladen in …5
×

Mongodb, our Swiss Army Knife Database

Experience feedback on 10 monthes of happy mongodb usage at fotopedia.

You may also checkout: http://www.slideshare.net/octplane/mongodb-vs-mysql-a-devops-point-of-view

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Mongodb, our Swiss Army Knife Database

  1. 1. MongoDB at fotopedia Timeline storage Our Swiss Army Knife Database
  2. 2. MongoDB at fotopedia • Context • Wikipedia data storage • Metacache
  3. 3. Fotopedia • Fotonauts, an American/French company • Photo — Encyclopedia • Heavily interconnected system : flickr, facebook, wikipedia, picassa, twitter… • MongoDB in production since last october • main store lives in MySQL… for now
  4. 4. First contact • Wikipedia imported data
  5. 5. Wikipedia queries • wikilinks from one article • links to one article • geo coordinates • redirect • why not use wikipedia API ?
  6. 6. Download ~ 5.7GB gzip XML Geo Redirect Backlink Related ~12GB tabular data
  7. 7. Problem Load ~12GB into a K/V store
  8. 8. CouchDB 0.9 attempt • CouchDB had no dedicated import tool • need to go through HTTP / Rest API
  9. 9. “DATA LOADING” LOADING! (obviously hijacked from xkcd.com)
  10. 10. Problem, rephrased Load ~12GB into any K/V store in hours, not days
  11. 11. Hadoop HBase ? • as we were already using Hadoop Map/ Reduce for preparation • bulk load was just emerging at that time, requiring to code against HBase private APIs, generate the data in an ad-hoc binary format, ...
  12. 12. photo by neural.it on Flickr
  13. 13. Problem, rerephrased Load ~12GB into any K/V store in hours, not days without wasting a week on development and another week on setup and several months on tuning please ?
  14. 14. MongoDB attempt • Transforming the tabular data into a JSON form : about half an hour or code, 45 minutes of hadoop parallel processing • setup mongo server : 15 minutes • mongoimport : 3 minutes to start it, 90 minutes to run • plug RoR app on mongo : minutes • prototype was done in a day
  15. 15. Batch Synchronous Download ~ 5.7GB gzip Geo Redirect Backlink Ruby on Rails Related ~12GB, 12M docs
  16. 16. Hot swap ? • Indexing was locking everything. • Just run two instances of MongoDB. • One instance is servicing the web app • One instance is asleep or loading data • One third instance knows the status of the two instances.
  17. 17. We loved: • JSON import format • efficiency of mongoimport • simple and flexible installation • just one cumbersome dependency • easy to start (we use runit) • easy to have several instances on one box
  18. 18. Second contact • itʼs just all about graphes, anyway. • wikilinks • people following people • related community albums • and soon, interlanguage links
  19. 19. all about graphes... • ... and itʼs also all about cache. • The application needs to “feel” faster, letʼs cache more. • The application needs to “feel” right, so letʼs cache less. • or — big sigh — invalidate.
  20. 20. Page fragment caching photo by Mykl Roventine on Flickr Nginx SSI photo by Aires Dos Santos Varnish HTTP cache RoR application photo by Leslie Chatfield on Flickr
  21. 21. Haiku ? There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton
  22. 22. Naming things • REST have been a strong design principle in fotopedia since the early days, and the efforts are paying.
  23. 23. /en/2nd_arrondissement_of_Paris /en/Paris/fragment/left_col /users/john/fragment/contrib /en/Paris/fragment/related
  24. 24. Invalidating • Rest allows us to invalidate by URL prefix. • When the Paris album changes, we have to invalidate /en/Paris.*
  25. 25. Varnish invalidation • Varnish built-in regexp based invalidation is not designed for intensive, fine grained invalidation. • We need to invalidate URLs individually.
  26. 26. /en/Paris.* /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27
  27. 27. Metacache workflow /en/Paris.* invalidation worker Nginx SSI /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 Varnish HTTP cache varnish log /en/Paris/fragment/left_col RoR application metacache feeder
  28. 28. Waw. • This time we are actually using MongoDB as a BTree. Impressive. • The metacache has been running fine for several months, and we want to go further.
  29. 29. Invalidate less • We need to be more specific as to what we invalidate. • Today, if somebody votes on a photo in the Paris album, we invalidate all /en/Paris prefix, and most of it is unchanged. • We will move towards a more clever metacache.
  30. 30. Metacache reloaded • Pub/Sub metacache • Have the backend send a specific header to be caught by the metacache-feeder, conaining “subscribe” message. • This header will be a JSON document, to be pushed to the metacache. • The purge commands will be mongo search queries.
  31. 31. /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  32. 32. when somebody votes { url:/en/Paris.*, observe:photos } when the summary changes { url:/en/Paris.*, observe:summary } when the a new link is created { url:/en/Paris.*, observe:links } {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  33. 33. Other uses cases • Timeline activities storage: just one more BTree usage. • Moderation workflow data: tiny dataset, but more complex queries, map/reduce. • Suspended experimentation around log collection and analysis
  34. 34. Current situation • Mysql: main data store • CouchDB: old timelines (+ chef) • MongoDB: metacache, wikipedia, moderation, new timelines • Redis: raw data cache for counters, recent activity (+ resque)
  35. 35. What about the main store ? • albums are good fit for documents • votes and score may be more tricky • recent introduction of resque
  36. 36. In short • Simple, fast. • Hackable: in a language most can read. • Clear roadmap. • Very helpful and efficient team. • Designed with application developer needs in mind.

×