This document summarizes Skye Book's presentation on how she refactored the data model and queries at her company Ultravisual to improve the performance of the user feed. Some key points: - The original data model used multiple queries to fetch posts, collections, and social activity for user feeds, which was slow and didn't scale well. - The new model treats the feed as a log of user activity, storing all "stories" like posts, follows, etc. as a single row per user for fast retrieval with one query. - Additional optimizations included caching the story JSON, handling deletions, and modeling onboarding flows as sequenced feed entries. - Production experiences included moving to more robust

1. CASSANDRA
@
ULTRAVISUAL
Cassandra Day New York 2014
Skye Book
Lead Systems Architect

2. ULTRAVISUA
L
A visual network for
inspiration, expression,
and collaboration

3. The Feed
• A user’s first taste of UV
• More than just posts
• Constantly being
tweaked and re-thought

4. SELECT
DISTINCT _post.*
FROM
_post
JOIN
_collection_post cp ON _post.uuid=cp.post_uuid
JOIN
_collection_follow cf ON cp.c_uuid=cf.collection_uuid
WHERE
cf.user_id = ?
ORDER BY _post.created_at DESC
LIMIT 20 OFFSET 0
The Old Way
Started Simple
!
“Show me recent posts in
collections I follow”

5. SELECT
a.*
FROM
_user_follow a, _user_follow b
WHERE
b.follower=12345
AND
a.follower=b.followed
ORDER BY a.followed_at DESC
LIMIT 20 OFFSET 0
The Old Way
Added Complexity
!
“Show me people recently
followed by my connections”

6. The Old Way
Every new feature needs
another query
!
Feed requests generate a
disproportionate amount of
load to normal CRUD ops

7. Reframing the Problem
From This:
A place for posts, new
collections, social activity, and
anything else interesting
nitro404.com/computers/knex.php

8. Reframing the Problem
To This:
A list of items interesting to
the user

9. The New Way
Model First
• With an SQL background, this can be
misleading.
• Essential Question: “How do I need to access
this data?”

10. –Rick Branson, Instagram
Cassandra Summit 2013
“Try to model data as a log of user intent”
The New Way

11. }
The New Way
user statu
s
created_a
t
story json
2 0 61b97280 user_follow:3:5 {“foo”:”bar”}
2 1 5daa04c0 post:bfbd0a39 {“foo”:”bar”}
2 1 565752e0 collection_follow:
5:d70961c1
{“foo”:”bar”}
2 1 4a8189e0 user_follow:3:5 {“foo”:”bar”}
Primary Key Cached story JSON
Model for user feeds
• Fast to fetch user stories
• Cached JSON means almost zero SQL requests

12. Fast.
Response times cut from
over 100’s ms to 30ms
range

13. Launch Week
Featured by Apple!
Cluster Disk Usage
26%
74%

14. Don’t be too cute
cqlsh:ultravisual> ALTER TABLE latest_feed DROP json;

15. Handling Deletions
• Data is only appended,
never deleted from user
feeds
• Adapted Instagram’s ‘Anti-
Column’ solution
• Avoids missed deletions
for nodes down longer
than GCGraceSeconds
• Avoids race condition
where deletion arrives
before write.
Sam follows Sandy
use
r
created_a
t
statu
s
story
2 4a8189e0 1 user_follow:
3:5
Sam unfollows Sandy
use
r
created_a
t
statu
s
story
2 61b97280 0 user_follow:
3:5
2 4a8189e0 1 user_follow:
3:5

16. Negated Entries
use
r
created_a
t
statu
s
story
2 61b97280 0 user_follow:
3:5
2 4a8189e0 1 user_follow:
3:5
use
r
statu
s
created_a
t
story
2 0 61b97280 user_follow:
3:5
2 1 4a8189e0 user_follow:
3:5
Keeps all entries in a single
time series
First page can usually be
populated by a single read
Splits user’s row into two lists,
live and undo
Will always require at least
two reads

17. Further Uses
• User Notifications
• User Onboarding
• Reshare Statistics
• User & Content Reports
• API Statistics

18. User Onboarding
user created_a
t
sequence step content
2 61b97280 onboaring_v2 1 rec_collections_1
3 5daa04c0 onboaring_v2 2 rec_collections_2
5 565752e0 onboaring_v3 1 find_friends
6 4a8189e0 onboaring_v3 1 find_friends
Sequenced feed entries
for users on signup

19. Production Experiences
Drivers
• Java: Started with Astyanax, moved to Datastax
v2
• Node.js: node-cassandra-cql

20. Cryptic message with large batch updates in pre-release versions of
2.0 driver
DS Driver Issue 229
com.datastax.driver.core.exceptions.DriverInternalError: An
unexpected protocol error occured. This is a bug in this library,
please report: Unknown code 256 for a consistency level
As of 2.0, batches with more than 64k statements throw a better
exception:
java.lang.IllagalStateException: Batch statement cannot contain
more than 65536 statements.

21. Just use LZ4
Compression

22. Cassandra-4851
Unfortunate truth in Cassandra 2.0.5
!
cqlsh:test> SELECT *
FROM user_feed
WHERE user = 2
AND created_at > :some_uuid
AND status=0;
!
cqlsh:test> Bad Request: PRIMARY KEY part status cannot be
restricted (preceding part created_at is either not
restricted or by a non-EQ relation)

23. Cassandra-4851
Adds CQL3 support for vector
comparison syntax
!
cqlsh:test> SELECT *
FROM timeline
WHERE day = ’21 Jun 2014’
AND (hour,min) >= (3,50)
AND (hour,min,sec) <= (4,37,30);
Available in 2.0.6

24. Production Experiences
Upgrades
• Manual package installs (dsc20 from Datastax)
• One node at a time
• Upgrade, wait for healthy status &
operations, move on
• OpsCenter provides good overview

25. Production Experiences
Speaking of OpsCenter…
• Don’t be alarmed if nodes appear but agent
data does not
• opscenterd often needs a restart after cluster
upgrade to see agents again

26. Production Experiences
Service Discovery
• Running on AWS using EC2MultiRegionSnitch
• Using OpsWorks (Amazon’s Chef service) for
seed config

27. Chef Cookbook
github.com/skyebook/cassandra-opsworks-chef-
cookbook
• Forked from Michael Klishin’s awesome C* cookbook
• Added integration with OpsWorks’ stack.json
# Add this node as the first seed
# If using the multi-region snitch, we must use the public IP address
if node["cassandra"]["snitch"] == "Ec2MultiRegionSnitch"
seed_array << node["opsworks"]["instance"]["ip"]
else
seed_array << node["opsworks"]["instance"]["private_ip"]
end
!
node["opsworks"]["layers"]["cassandra"]["instances"].each do |instance_name, values|
if node["cassandra"]["snitch"] == "Ec2MultiRegionSnitch"
seed_array << values["ip"]
else
seed_array << values["private_ip"]
end
end
set[:cassandra][:seeds] = seed_array

28. Questions