MongoDB’s basic unit of storage is a document. Documents can represent rich, schema-free data structures, meaning that we have several viable alternatives to the normalized, relational model. In this talk, we’ll discuss the tradeoff of various data modeling strategies in MongoDB using a library as a sample application. You will learn how to work with documents, evolve your schema, and common schema design patterns.

1. Real Time Analytics
Chad Tindel
chad.tindel@10gen.com

2. The goal
Real Time
Analytics Engine
Real Time
Analytics Engine
Data
Sourc
e
Data
Sourc
e
Data
Sourc
e

3. Solution goals

4. Simple log storage
Design Pattern

5. Aggregation - PipelinesAggregation - Pipelines
• Aggregation requests specify a pipeline
• A pipeline is a series of operations
• Conceptually, the members of a collection
are passed through a pipeline to produce
a result
– Similar to a Unix command-line pipe

6. Aggregation PipelineAggregation Pipeline

7. Aggregation - PipelinesAggregation - Pipelines
db.collection.aggregate(
[ {$match: … },
{$group: … },
{$limit: …}, etc
]

8. Pipeline OperationsPipeline Operations
• $match
– Uses a query predicate (like .find({…})) as a
filter
{ $match : { author : "dave" } }
{ $match : { score : { $gt : 50, $lte : 90 } } }

9. Pipeline OperationsPipeline Operations
• $project
– Uses a sample document to determine the
shape of the result (similar to .find()’s 2nd
optional argument)
• Include or exclude fields
• Compute new fields
– Arithmetic expressions, including built-in functions
– Pull fields from nested documents to the top
– Push fields from the top down into new virtual documents

10. Pipeline OperationsPipeline Operations
• $unwind
– Hands out array elements one at a time
{ $unwind : {"$myarray" } }
• $unwind “streams” arrays
– Array values are doled out one at time in the
context of their surrounding document
– Makes it possible to filter out elements before
returning

11. Pipeline OperationsPipeline Operations
• $group
– Aggregates items into buckets defined by a
key

12. GroupingGrouping
• $group aggregation expressions
– Define a grouping key as the _id of the result
– Total grouped column values: $sum
– Average grouped column values: $avg
– Collect grouped column values in an array or
set: $push, $addToSet
– Other functions
• $min, $max, $first, $last

13. Pipeline OperationsPipeline Operations
• $sort
– Sort documents
– Sort specifications are the same as today,
e.g., $sort:{ key1: 1, key2: -1, …}
{ $sort : {“total”:-1} }

14. Pipeline OperationsPipeline Operations
• $limit
– Only allow the specified number of documents
to pass
{ $limit : 20 }

15. Pipeline OperationsPipeline Operations
• $skip
– Skip over the specified number of documents
{ $skip : 10 }

16. Computed ExpressionsComputed Expressions
• Available in $project operations
• Prefix expression language
– Add two fields: $add:[“$field1”, “$field2”]
– Provide a value for a missing field: $ifNull:
[“$field1”, “$field2”]
– Nesting: $add:[“$field1”, $ifNull:[“$field2”,
“$field3”]]
(continued)

17. Computed ExpressionsComputed Expressions
(continued)(continued)
• String functions
– toUpper, toLower, substr
• Date field extraction
– Get year, month, day, hour, etc, from ISODate
• Date arithmetic
• Null value substitution (like MySQL ifnull(),
Oracle nvl())
• Ternary conditional
– Return one of two values based on a predicate
• Other functions….
– And we can easily add more as required

18. Sample data
Original
Event
Data
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326 “http://www.example.com/start.html" "Mozilla/4.08
[en] (Win98; I ;Nav)”
As JSON doc = {
_id: ObjectId('4f442120eb03305789000000'),
host: "127.0.0.1",
time: ISODate("2000-10-10T20:55:36Z"),
path: "/apache_pb.gif",
referer: “http://www.example.com/start.html",
user_agent: "Mozilla/4.08 [en] (Win98; I ;Nav)”
}
Insert to
MongoDB
db.logs.insert( doc )

19. Dynamic Queries
Find all
logs for
a URL
db.logs.find( { ‘path’ : ‘/index.html’ } )
Find all
logs for
a time
range
db.logs.find( { ‘time’ :
{ ‘$gte’ : new Date(2012,0),
‘$lt’ : new Date(2012,1) } } );
Find all
logs for
a host
over a
range of
dates
db.logs.find( {
‘host’ : ‘127.0.0.1’,
‘time’ : { ‘$gte’ : new Date(2012,0),
‘$lt’ : new Date(2012, 1) } } );

20. Aggregation Framework
Request
s per
day by
URL
db.logs.aggregate( [
{ '$match': {
'time': {
'$gte': new Date(2012,0),
'$lt': new Date(2012,1) } } },
{ '$project': {
'path': 1,
'date': {
'y': { '$year': '$time' },
'm': { '$month': '$time' },
'd': { '$dayOfMonth': '$time' } } } },
{ '$group': {
'_id': {
'p':'$path’,
'y': '$date.y',
'm': '$date.m',
'd': '$date.d' },
'hits': { '$sum': 1 } } },
])

21. Aggregation Framework
{
‘ok’: 1,
‘result’: [
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 1 },'hits’: 124 } },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 2 },'hits’: 245} },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 3 },'hits’: 322} },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 4 },'hits’: 175} },
{ '_id': {'p':’/index.html’,'y': 2012,'m': 1,'d': 5 },'hits’: 94} }
]
}

22. Roll-ups with map-
reduce
Design Pattern

23. Map Reduce – Map Phase
Generat
e hourly
rollups
from log
data
var map = function() {
var key = {
p: this.path,
d: new Date(
this.ts.getFullYear(),
this.ts.getMonth(),
this.ts.getDate(),
this.ts.getHours(),
0, 0, 0) };
emit( key, { hits: 1 } );
}

24. Map Reduce – Reduce Phase
Generat
e hourly
rollups
from log
data
var reduce = function(key, values) {
var r = { hits: 0 };
values.forEach(function(v) {
r.hits += v.hits;
});
return r;
}
)

25. Map Reduce
Generat
e hourly
rollups
from log
data
cutoff = new Date(2012,0,1)
query = { 'ts': { '$gt': last_run, '$lt': cutoff } }
db.logs.mapReduce( map, reduce, {
‘query’: query,
‘out’: { ‘reduce’ : ‘stats.hourly’ } } )
last_run = cutoff

26. Map Reduce Output
> db.stats.hourly.find()
{ '_id': {'p':’/index.html’,’d’:ISODate(“2012-0-1 00:00:00”) },
’value': { ’hits’: 124 } },
{ '_id': {'p':’/index.html’,’d’:ISODate(“2012-0-1 01:00:00”) },
’value': { ’hits’: 245} },
{ '_id': {'p':’/index.html’,’d’:ISODate(“2012-0-1 02:00:00”) },
’value': { ’hits’: 322} },
{ '_id': {'p':’/index.html’,’d’:ISODate(“2012-0-1 03:00:00”) },
’value': { ’hits’: 175} },
... More ...

27. Chained Map Reduce
Collection 1 :
Raw Logs
Collection 1 :
Raw Logs
Map
Reduce
Map
Reduce
Collection 2:
Hourly Stats
Collection 2:
Hourly Stats
Collection 3:
Daily Stats
Collection 3:
Daily Stats
Map
Reduce
Map
Reduce
Runs
every hour
Runs
every day

28. Pre-aggregated
documents
Design Pattern

29. Pre-Aggregation
Data for
URL /
Date
{
_id: "20101010/site-1/apache_pb.gif",
metadata: {
date: ISODate("2000-10-10T00:00:00Z"),
site: "site-1",
page: "/apache_pb.gif" },
daily: 5468426,
hourly: {
"0": 227850,
"1": 210231,
...
"23": 20457 },
minute: {
"0": 3612,
"1": 3241,
...
"1439": 2819 }
}

30. Pre-Aggregation
Data for
URL /
Date
id_daily = dt_utc.strftime('%Y%m%d/') + site + page
hour = dt_utc.hour
minute = dt_utc.minute
# Get a datetime that only includes date info
d = datetime.combine(dt_utc.date(), time.min)
query = {
'_id': id_daily,
'metadata': { 'date': d, 'site': site, 'page': page } }
update = { '$inc': {
‘daily’ : 1,
'hourly.%d' % (hour,): 1,
'minute.%d.%d' % (hour,minute): 1 } }
db.stats.daily.update(query, update, upsert=True)

31. Pre-Aggregation
Data for
URL /
Date
db.stats.daily.findOne(
{'metadata': {'date':dt,
'site':'site-1',
'page':'/index.html'}},
{ 'minute': 1 }
);

32. Solution Architect, 10gen