There are many possible approaches to storing and querying relationships between users in social networks. This section will dive into the details of storing a social user graph in MongoDB. It will cover the various schema designs for storing the follower networks of users and propose an optimal design for insert and query performance, as well as looking at performance differences between them.
20. Operational issues
• Updates of embedded arrays
– grow non-linearly with number of indexed array
elements
• Updating edge collection => inserts
– grows close to linearly with existing number of
edges/user
23. Finding Followers
Consider our single followercollection :
> db.followers.find({from : "djw"}, {_id:0, to:1})
{
"to" : "jsr"
}
Using index :
{
"v" : 1,
"key" : { "from" : 1, "to" : 1 },
"unique" : true,
"ns" : "socialite.followers",
"name" : "from_1_to_1"
}
Covered index when
searching on "from"
for all followers
Specify only if
multiple edges cannot
exist
24. Finding Following
What about who a user is following?
Can use a reverse covered index :
{
"v" : 1,
"key" : { "from" : 1, "to" : 1 },
"unique" : true,
"ns" : "socialite.followers",
"name" : "from_1_to_1"
}
{
"v" : 1,
"key" : { "to" : 1, "from" : 1 },
"unique" : true,
"ns" : "socialite.followers",
"name" : "to_1_from_1"
}
Notice the flipped
field order here
25. Finding Following
Wait ! There is an issue with the reverse index…..
SHARDING !
{
"v" : 1,
"key" : { "from" : 1, "to" : 1 },
"unique" : true,
"ns" : "socialite.followers",
"name" : "from_1_to_1"
}
{
"v" : 1,
"key" : { "to" : 1, "from" : 1 },
"unique" : true,
"ns" : "socialite.followers",
"name" : "to_1_from_1"
}
If we shard this collection by
"from", looking up followers
for a specific user is
"targeted" to a shard
To find who the user is
following however, it must
scatter-gather the query to
all shards
27. Dual Edge Collections
When "following" queries are common
– Not always the case
– Consider overhead carefully
Can use dual collections storing
– One for each direction
– Edges are duplicated reversed
– Can be sharded independently
28. Edge Query Rate Comparison
Number of shards
vs
Number of queries
Followers collection
with forward and
reverse indexes
Two collections,
followers, following
one index each
1 10,000 10,000
3 90,000 30,000
6 360,000 60,000
12 1,440,000 120,000
29. Follower Counts
Can use the edge indexes :
How to determine these counts ?
> db.followers.find({_f : "djw"}).count()
> db.following.find({_f : "djw"}).count()
However this can be heavy weight
- Especially for rendering landing page
- Consider maintaining counts on user document
30. Socialite User Service
• Manages user profiles and the follower graph
• Supports arbitrary user data passthrough
• Options for graph storage
– Uses edge collections (can shard by _f)
– Options for maintaining separate follower/ing graphs
– Storing counts vs counting
{
"_id" : ObjectId("52cd1d32a0ee9a1a76d369bb"),
"_f" : "jsr",
"_t" : "djw"
}
{
"v" : 1,
"key" : {"_f" : 1, "_t" : 1},
"unique" : true,
}
31. Next up @ 11:50am :
Scaling the Data Feed
• Delivering user content to followers
• Comparing fanout models
• Caching user timelines for fast retrieval
• Embedding vs Linking Content
32. Building a Social Platform
with MongoDB
MongoDB Inc
Darren Wood & Asya Kamsky
#MongoDBWorld
Hinweis der Redaktion
Scaling the delivery of posts and content to the follower networks of millions of users has many challenges. In this section we look at the various approaches to fanning out posts and look at a performance comparison between them. We will highlight some tricks for caching the recent timeline of active users to drive down read latency.
image at https://dropwizard.github.io/dropwizard of the hat
Tempting to focus on scaling content
Follow requests rival message send rates
Twitter enforces per day follow limits
Single Collection
How to test, show how growing documents are very painful to update.
Add the MTV or appmetrics mtools plot showing what happens to outliers.
actual performance – show how inserting million users was easy – no point even trying to update embedded documents...
side-point of
NEED TO GENERATE FOR broadcast (scatter gather) for following,
direct for followers.
Number of total queries by number of shards...
TO GET WHOM THE USER IS FOLLOWING