2. DISCLAIMER: MongoDB's product
plans are for informational purposes
only. MongoDB's plans may change
and you should not rely on them for
delivery of a specific feature at a
specific time.
3.
4. Agenda
Document vs. Relational Model
Analytics on MongoDB data
60,000 feet – what is the aggregation pipeline
Aggregation pipeline operators
$lookup (Left Outer Equi Joins) in MongoDB
3.2
Other aggregation enhancements
Worked examples
6. Existing Alternatives to Joins
{ "_id": 10000,
"items": [
{
"productName": "laptop",
"unitPrice": 1000,
"weight": 1.2,
"remainingStock": 23
},
{
"productName": "mouse",
"unitPrice": 20,
"weight": 0.2,
"remainingStock": 276
}
],
…
}
• Option 1: Include all data for an order in
the same document
– Fast reads
• One find delivers all the required data
– Captures full description at the time of the
event
– Consumes extra space
• Details of each product stored in many order
documents
– Complex to maintain
• A change to any product attribute must be
propagated to all affected orders
orders
7. Existing Alternatives to Joins
{
"_id": 10000,
"items": [
12345,
54321
],
...
}
• Option 2: Order document
references product documents
– Slower reads
• Multiple trips to the database
– Space efficient
• Product details stored once
– Lose point-in-time snapshot of full
record
– Extra application logic
• Must iterate over product IDs in
the order document and find the
product documents
• RDBMS would automate through
a JOIN
orders
{
"_id": 12345,
"productName": "laptop",
"unitPrice": 1000,
"weight": 1.2,
"remainingStock": 23
}
{
"_id": 54321,
"productName": "mouse",
"unitPrice": 20,
"weight": 0.2,
"remainingStock": 276
}
products
8. The Winner?
• In general, Option 1 wins
– Performance and containment of everything in same place beats space
efficiency of normalization
– There are exceptions
• e.g. Comments in a blog post -> unbounded size
• However, analytics benefit from combining data from multiple collections
– Keep listening...
17. Aggregation Pipeline Stages
• $match
Filter documents
• $geoNear
Geospherical query
• $project
Reshape documents
• $lookup
New – Left-outer equi joins
• $unwind
Expand documents
• $group
Summarize documents
• $sample
New – Randomly selects a subset
of documents
• $sort
Order documents
• $skip
Jump over a number of documents
• $limit
Limit number of documents
• $redact
Restrict documents
• $out
Sends results to a new collection
18. $lookup
• Left-outer join
– Includes all documents from the
left collection
– For each document in the left
collection, find the matching
documents from the right
collection and embed them
Left Collection Right Collection
31. Aggregation With a Sharded Database
• Workload split between shards
– Client works through mongos as with
any query
– Shards execute pipeline up to a point
– A single shard merges cursors and
continues processing
– Use explain to analyze pipeline split
– Early $match on shard key may
exclude shards
– Potential CPU and memory
implications for primary shard host
– $lookup & $out performed within
Primary shard for the database
?
33. Restrictions
• $lookup only support equality for the match
• $lookup can only be used in the aggregation pipeline (e.g. not for find)
• The pipeline is linear; no forks. Can remove data at each stage and can only add new
raw data through $lookup
• Right collection for $lookup cannot be sharded
• Indexes are only used at the beginning of the pipeline (and right tables in subsequent
$lookups), before any data transformations
• $out can only be used in the final stage of the pipeline
• $geoNear can only be the first stage in the pipeline
• The BI Connector for MongoDB is part of MongoDB Enterprise Advanced
– Not in community
34. Next Steps
• Documentation
– https://docs.mongodb.org/manual/release-notes/3.2/#aggregation-framework-enhancements
• Not yet ready for production but download and try!
– https://www.mongodb.org/downloads#development
• Detailed blog
– https://www.mongodb.com/blog/post/joins-and-other-aggregation-enhancements-coming-in-mongodb-3-2-
part-1-of-3-introduction
• Webinars
– Tomorrow: What's New in MongoDB 3.2 https://www.mongodb.com/webinar/whats-new-in-mongodb-3-2
– Replay: 3.2 $lookup & aggregation https://www.mongodb.com/presentations/webinar-joins-and-other-
aggregation-enhancements-coming-in-mongodb-3-2
• Feedback
– MongoDB 3.2 Bug Hunt
• https://www.mongodb.com/blog/post/announcing-the-mongodb-3-2-bug-hunt
– https://jira.mongodb.org/
DISCLAIMER: MongoDB's product plans are for informational purposes only. MongoDB's plans may change and you
should not rely on them for delivery of a specific feature at a specific time.
35. MongoDB Days 2015
October 6, 2015
October 20, 2015
November 5, 2015
December 2, 2015
France
Germany
UK
Silicon Valley