This talk will walk through the journey of Cassandra at Netflix. It will go into 3-4 specific use cases where Cassandra stands out than the rest of the data-stores and is being used in Netflix, bringing great viewing experience to all customers globally. Roopa will go into the specifics of the data model being used and where Cassandra stands out with its strengths and which places where they learnt the hard way. Roopa will then share some of the best practices and self service platform being used for Cassandra to cater to their developer needs.

1. 2007

5. Courtesy New York Times

8. Cassandra’s
Odyssey @ Netflix
Roopa Tangirala
Engineering Manager

9. Contents
● Brief History
● Cassandra Use Cases
● Supporting Infrastructure
● What’s Next?
● Q & A

12. SCALABILITY

13. EVERY TWO WEEKS!!

14. 2008

15. MOVE TO CLOUD

16. ▪ HIGHLY AVAILABLE
▪ MULTI DATACENTER SUPPORT
▪ PREDICTABLE PERFORMANCE AT SCALE
REQUIREMENTS

17. ▪Massively scalable architecture
▪Multi-datacenter, multi-directional replication
▪Linear scale performance
▪Transparent fault detection and recovery
▪Flexible, dynamic schema data
▪Guaranteed data safety
▪Tunable data consistency
WHY CASSANDRA?

18. Us-west-2
Oregon
Us-east-1
North Virginia
Eu-west-1
Ireland

19. AWS S3
BACKEND NOW

21. DOWNTIME

24. Cassandra Use Cases

25. Membership

27. Current Membership data model
Current [soon to be old model]
1. Thrift based
2. Schema less data model
3. One account info - results in multiple reads
[from multiple nodes]

28. Membership data model - evolved
New data model
1. CQL based
2. Well defined schema
3. A few UDTs for schema flexibility
4. One account with multiple profiles - just one call

29. Membership data model - evolved
New data model
1. Primary key definition
a. Account_id -> part key and
b. profile_id -> clustering column
2. Partition size <= 64k for all the profile
CQL - Table definition

30. Viewing History

33. Growth of Viewing History
33

34. Old Data Model
34

35. New Data Model cont’d
35

36. 36

37. Results
37

38. Global Ratings

40. Global Ratings - I
Two main usage -
1. Full sweep of advisories and rating
countrywide
2. Get advisories and rating for a
given movie_id, country_code

41. ● Full sweep countrywide
Global Ratings - II

42. ● Get advisories and rating for a given movie_id, country_code
Global Ratings - III

43. Observability Tools

44. Observability
● Analysis of logs
● Traces
● Metrics

47. ● Slow Rate of ingestion
● Slow reads
● On ingestion need to parse message and build
index
● Data for each request not continuous on disk
Elasticsearch Issues

48. blah
data

49. Supporting
Infrastructure

50. CDE Service
“Empowering CDE to provide datastores as a
service”

51. Motivation
So. Many. Tools.
...and more

52. A Central Hub for Persistence
Self Service
Management
Insights

53. Self Service

54. Before:
1. Users sent requests via email, Slack, or JIRA tickets.
2. CDE on-call translated their requirements into appropriate resource
requirements, and kicked off automation to create clusters.
3. Once created, CDE on-call notified users via email, including information
about how to access the cluster, set up security group rules, useful links,
etc.
Elasticsearch Cluster Creation

55. * Cost shown is not real, for illustration only.

57. Cluster
Management

58. Goals:
1. Allow Repairs to be enabled/disabled quickly for a cluster.
2. Allow fine-tuning of table-specific settings for subrange repairs,
parallelism, etc.
Cassandra Repairs

61. Goals:
1. Allow Backups to be enabled/disabled quickly for a cluster.
2. Allow Backup schedules, retention periods, S3 buckets/locations, etc. to
be customized.
Cluster Backups

64. Insights

65. Goal:
Provide cluster owners insight into the costs of running their clusters so that
they can make more informed choices/tradeoffs with respect to efficiency in
resource usage.
Cluster Costs

66. * Costs shown are not real, for illustration only.

67. Goal:
Aggregate and display on-instance stats/metadata for all nodes in a cluster,
including:
○ Datastore versions
○ Sidecar versions
○ OS versions
○ AWS instance types
○ etc.
Node Inventory

69. Architecture

70. Web UI
CDE
Service
API
C* Store
(multiregion)
Bolt
Bolt
Datastore Nodes
Bolt
CDE
Service
Backend
(eu-west-1)
CDE
Service
Backend
(us-east-1)
CDE
Service
Backend
(us-west-2)
Datastore Nodes
Datastore Nodes

71. CDE Service Data Model
CDE service -
● Hierarchical model
● Manages hundreds of clusters
● Manages tens of thousands of nodes

72. Sample inserts -
INSERT INTO "eunomia"."eunomia_cass_app_node_info"( "appname", "env", "region",
"nodeuid", "jason_val",)
VALUES('cass_app', 'NA', 'NA', 'NA','app_level_info');
INSERT INTO "eunomia"."eunomia_cass_app_node_info"( "appname", "env", "region",
"nodeuid", "jason_val",)
VALUES('cass_app', 'PROD', 'NA', 'NA','app_env_level_info');
INSERT INTO "eunomia"."eunomia_cass_app_node_info"( "appname", "env", "region",
"nodeuid", "jason_val",)
VALUES('cass_app', 'PROD', 'US-EAST-1', 'NA','app_env_region_level_info');
INSERT INTO "eunomia"."eunomia_cass_app_node_info"( "appname", "env", "region",
"nodeuid", "jason_val",)
VALUES('cass_app', 'PROD', 'US-EAST-1','NODE1','app_env_region_node_level_info');
PRIMARY KEY Definition -
PRIMARY KEY (appname, env, region, nodeuid)
) WITH CLUSTERING ORDER BY (env ASC, region ASC);

74. Sample Select - Get all the data for an app

76. Get data only at app level

77. Cluster-level data

78. Get all the data at app and ENV level

79. Environment-level data (PROD vs TEST)

80. Get data only app and ENV level

81. Region-level data (within an environment)

82. What’s Next?

83. ▪ Unify access to our customer’s persisted data
▪ Codify the CDE team’s best-practices
▪ Provide cluster-level ACLs to avoid unnecessary
access
▪ Provide a paved path for common database
operations
▪ Work at Netflix-scale
Netflix Data Explorer

84. Single Entrypoint for Clusters

88. How do we expand beyond three
regions without increasing the
replicas or cost dramatically?

89. Bounded eventual consistency

90. Cassandra CDC to get data in
sync between cassandra and
derived stores

91. Materialized views production
ready

92. Q&A