Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
rogue:a scala dsl for mongodbScalaDays 2011 - 6/2/2011Jason Liszka (@jliszka)Jorge Ortiz (@jorgeortiz85)
what is foursquare?location-based social network - “check-in” to bars,restaurants, museums, parks, etc  friend-finder (whe...
foursquare: the numbers>9M users~3M checkins/day>15M venues>400k merchants>60 employees
foursquare: the tech Nginx, HAProxy Scala, Lift MongoDB, PostgreSQL (legacy) (Kestrel, Munin, Ganglia, Python, Memcache, ....
what is mongodb?fast, schema-less document storeindexes & rich queries on any fieldsharding, auto-balancingreplicationgeo-...
mongodb: our numbers8 clusters  some sharded, some not  some master/slave, some replica set~40 machines (68.4GB, m2.4xl on...
mongodb: bsonjson++binary wire formatmore built-in types:  objectid  date  regex  bina
mongodb: bson example{    “_id” :      { “oid” : “4ddbd194686148110d5c1ccc” },    “venuename” : “Starbucks”,    “mayorid” ...
mongodb: query example{    “mayorid” : { “$lte” : 100 },    “venuename” : “Starbucks”,    “tags” : “wifi”,    “latlng” : {...
mongodb: query exampleval query =  (BasicDBOBjectBuilder    .start    .push(“mayorid”)      .add(“$lte”, 100)    .pop    ....
rogue: a scala dsl for mongo type-safe all mongo query features logging & validation hooks pagination index-aware cursors ...
rogue: schema exampleclass Venue extends MongoRecord[Venue] {  object _id extends ObjectIdField(this)  object venuename ex...
rogue: code exampleval vs: List[Venue] =  (Venue where (_.mayorid <= 100)          and (_.venuename eqs “Starbucks”)      ...
rogue: BaseQueryclass BaseQuery[M <: MongoRecord[M], R,                Ord, Sel, Lim, Sk](...) {    def where[F](clause: M...
rogue: QueryFieldclass QueryField[V, M <: MongoRecord[M]]                (val field: Field[V, M]) {  def eqs(v: V) = new E...
rogue: implicitsField[F, M] => QueryField[F, M]Field[LatLong, M] => GeoQueryField[M]Field[List[F], M] => ListQueryField[F,...
rogue: selectval vs: List[(Long, Int)] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorChec...
rogue: selectclass BaseQuery[M <: MongoRecord[M], R,                Ord, Sel, Lim, Sk](...) {    def select[F1, F2](      ...
rogue: select problemsval vs: List[???] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorChe...
rogue: limit problemsval vs: List[Venue] =  (Venue where (_.venuename eqs “Starbucks”)         limit (5)         limit (10...
rogue: more limit problems(Venue where (_.venuename eqs “Starbucks”)       orderAsc (_.total_checkins)       limit (5)    ...
rogue: phantom types     Sel =:= Selected, Unselected     Ord =:= Ordered, Unordered     Lim =:= Limited, Unlimited     Sk...
rogue: phantom typesabstract   sealed   class   Orderedabstract   sealed   class   Unorderedabstract   sealed   class   Se...
rogue: initializing phantomsclass BaseQuery[M <: MongoRecord[M], R,                Ord, Sel, Lim, Sk](...) {  ...}def dflt...
rogue: select phantomclass BaseQuery[M <: MongoRecord[M], R,                Ord, Sel, Lim, Sk](...) {    def select[F1, F2...
rogue: =:=sealed abstract class =:=[From, To] ...implicit def tpEquals[A]: A =:= A = ...
rogue: select solutionsval vs: List[???] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorCh...
rogue: limit solutions(Venue where (_.venuename eqs “Starbucks”)       orderAsc (_.total_checkins)       limit (5)       b...
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646)      and (_.venueid eqs vid)    fetch ())
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646)      and (_.venueid eqs vid) // hidden sca...
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646) // known index     scan (_.venueid eqs vid...
rogue: logging & validation logging:   slf4j   Tracer validation:   radius, $in size   index checks
rogue: paginationval query: Query[Venue] = ...val vs: List[Venue] =  (query    .countPerPage(20)    .setPage(5)    .fetch())
rogue: cursorsval query: Query[Checkin] = ...for (checkin <- query) {  ... f(checkin) ...}
rogue: future directions iteratees for cursors compile-time index checking select partial objects generate full javascript...
we’re hiring    (nyc & sf)http://foursquare.com/jobs  @jliszka, @jorgeortiz85
Nächste SlideShare
Wird geladen in …5
×

Scala Days 2011 - Rogue: A Type-Safe DSL for MongoDB

15.615 Aufrufe

Veröffentlicht am

Talk at Scala Days 2011

Scala Days 2011 - Rogue: A Type-Safe DSL for MongoDB

  1. rogue:a scala dsl for mongodbScalaDays 2011 - 6/2/2011Jason Liszka (@jliszka)Jorge Ortiz (@jorgeortiz85)
  2. what is foursquare?location-based social network - “check-in” to bars,restaurants, museums, parks, etc friend-finder (where are my friends right now?) virtual game (badges, points, mayorships) city guide (local, personalized recommendations) location diary + stats engine (where was I a year ago?) specials (get rewards at your favorite restaurant)
  3. foursquare: the numbers>9M users~3M checkins/day>15M venues>400k merchants>60 employees
  4. foursquare: the tech Nginx, HAProxy Scala, Lift MongoDB, PostgreSQL (legacy) (Kestrel, Munin, Ganglia, Python, Memcache, ...) All on EC2
  5. what is mongodb?fast, schema-less document storeindexes & rich queries on any fieldsharding, auto-balancingreplicationgeo-indexes
  6. mongodb: our numbers8 clusters some sharded, some not some master/slave, some replica set~40 machines (68.4GB, m2.4xl on EC2)2.3 billion records~15k QPS
  7. mongodb: bsonjson++binary wire formatmore built-in types: objectid date regex bina
  8. mongodb: bson example{ “_id” : { “oid” : “4ddbd194686148110d5c1ccc” }, “venuename” : “Starbucks”, “mayorid” : 464, “tags” : [“coffee”, “wifi”, “snacks”], “latlng” : [39.0, -74.0]}
  9. mongodb: query example{ “mayorid” : { “$lte” : 100 }, “venuename” : “Starbucks”, “tags” : “wifi”, “latlng” : { “$near” : [39.0, -74.0] }}{ “_id” : -1 }
  10. mongodb: query exampleval query = (BasicDBOBjectBuilder .start .push(“mayorid”) .add(“$lte”, 100) .pop .add(“veneuname”, “Starbucks”) ... .get)
  11. rogue: a scala dsl for mongo type-safe all mongo query features logging & validation hooks pagination index-aware cursors http://github.com/foursquare/rogue
  12. rogue: schema exampleclass Venue extends MongoRecord[Venue] { object _id extends ObjectIdField(this) object venuename extends StringField(this) object mayorid extends LongField(this) object tags extends ListField[String](this) object latlng extends LatLngField(this)}object Venue extends Venue with MongoMetaRecord[Venue]
  13. rogue: code exampleval vs: List[Venue] = (Venue where (_.mayorid <= 100) and (_.venuename eqs “Starbucks”) and (_.tags contains “wifi”) and (_.latlng near (39.0, -74.0, Degrees(0.2)) orderDesc (_._id) fetch (5))
  14. rogue: BaseQueryclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def where[F](clause: M => QueryClause[F]): ... ...}
  15. rogue: QueryFieldclass QueryField[V, M <: MongoRecord[M]] (val field: Field[V, M]) { def eqs(v: V) = new EqClause(...) def neqs(v: V) = ... def in[L <% List[V]](vs: L) = ... def nin[L <% List[V]](vs: L) = ... def exists(b: Boolean) = ...}
  16. rogue: implicitsField[F, M] => QueryField[F, M]Field[LatLong, M] => GeoQueryField[M]Field[List[F], M] => ListQueryField[F, M]Field[String, M] => StringQueryField[M]Field[Int, M] => NumericQueryField[Int, M]Field[Double, M] => NumericQueryField[Double, M]...
  17. rogue: selectval vs: List[(Long, Int)] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) fetch ())
  18. rogue: selectclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def select[F1, F2]( f1: M => Field[F1, M], f2: M => Field[F2, M]): BaseQuery[M, (F1, F2), ...] = ... ...}
  19. rogue: select problemsval vs: List[???] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ())
  20. rogue: limit problemsval vs: List[Venue] = (Venue where (_.venuename eqs “Starbucks”) limit (5) limit (100) // (???) fetch ())
  21. rogue: more limit problems(Venue where (_.venuename eqs “Starbucks”) orderAsc (_.total_checkins) limit (5) bulkDelete_!!)// How many Venues got deleted?
  22. rogue: phantom types Sel =:= Selected, Unselected Ord =:= Ordered, Unordered Lim =:= Limited, Unlimited Sk =:= Skipped, Unskippedhttp://james-iry.blogspot.com/2010/10/phantom-types-in-haskell-and-scala.html
  23. rogue: phantom typesabstract sealed class Orderedabstract sealed class Unorderedabstract sealed class Selectedabstract sealed class Unselectedabstract sealed class Limitedabstract sealed class Unlimitedabstract sealed class Skippedabstract sealed class Unskipped
  24. rogue: initializing phantomsclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { ...}def dfltQuery[M <: MongoRecord[M]](rec: M) = new BaseQuery[M, M, Unordered, Unselected, Unlimited, Unskipped](...)
  25. rogue: select phantomclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def select[F1, F2]( f1: M => Field[F1, M], f2: M => Field[F2, M]) (implicit ev: Sel =:= Unselected): BaseQuery[M, (F1, F2), Ord, Selected, Lim, Sk] = ... ...}
  26. rogue: =:=sealed abstract class =:=[From, To] ...implicit def tpEquals[A]: A =:= A = ...
  27. rogue: select solutionsval vs: List[???] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ())// won’t compile!
  28. rogue: limit solutions(Venue where (_.venuename eqs “Starbucks”) orderAsc (_.total_checkins) limit (5) bulkDelete_!!)// won’t compile!
  29. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) fetch ())
  30. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) // hidden scan! fetch ())
  31. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) // known index scan (_.venueid eqs vid) // known scan fetch ())
  32. rogue: logging & validation logging: slf4j Tracer validation: radius, $in size index checks
  33. rogue: paginationval query: Query[Venue] = ...val vs: List[Venue] = (query .countPerPage(20) .setPage(5) .fetch())
  34. rogue: cursorsval query: Query[Checkin] = ...for (checkin <- query) { ... f(checkin) ...}
  35. rogue: future directions iteratees for cursors compile-time index checking select partial objects generate full javascript for mapreduce
  36. we’re hiring (nyc & sf)http://foursquare.com/jobs @jliszka, @jorgeortiz85

×