http://www.oreilly.com/pub/e/3764
Keystone processes over 700 billion events per day (1 peta byte) with at-least-once processing semantics in the cloud. Monal Daxini details how they used Kafka, Samza, Docker, and Linux at scale to implement a multi-tenant pipeline in AWS cloud within a year. He'll also share plans on offering a Stream Processing as a Service for all of Netflix use.
15. ● Support at-least-once processing
● Scale, Multi-tenancy, Ease of Operations
● Enable future value adds - Stream Processing As a Service
● Replace dormant open source software - Chukwa
Why a new pipeline?
16. Goal - Migrate 1.3 PB of event data to a new Pipeline
in flight, while not losing more that 0.1% of them
33. Kafka (prod) Footprint
Fronting
Kafka
Front Standby Kafka Consumer Kafka
Number of Clusters 24 24 8
Number of Instances 3000+ 72 1000+
Retention Period 8 to 24 hrs 1 hr 2 to 4 hrs
34. ● Independent zookeeper cluster per Kafka cluster
● 5 nodes per ensemble - 160 zookeeper nodes
● 3 ASGs per cluster, 1 ASG per zone
Kafka (prod) Footprint
35. ● Pioneer Tax
● Started with 0.7, went live with 0.8.2
● Done moving to 0.9 & VPC
● Work closely with Confluent to get patches through
○ OSS contributions
Kafka in the Cloud
36. ● No dynamic topic creation
● Two copies
● Rack / Zone aware partition assignment
● Per Cluster Stay under 10k partitions & 200 brokers
● Leave approx. 40% free disk space on each broker
Fronting Kafka Topics
37. Want to know more...
Netflix Tech Blog - Kafka in Keystone Pipeline
40. Samza Job Deployment
● Multiple Samza jobs for one Kafka source topic
● Each job processes messages for one sink
● Each job processes partitions only from one topic
● One checkpoint topic per Kafka source topic and
multiple samza jobs
● Job starts with Immutable Config
41. ● Over 13,000+ docker containers (samza jobs)
● 1,300+ AWS C3-4XL instances
Routing Service Footprint
S3 ElasticSearch Consumer Kafka
Number of containers 7000+ 1500+ 4500+
46. Logs ZFS Volume
Snapshots
Custom Go
Executor
.
/runJo
b
.
/runJo
b
.
/runJo
b
Go Tools Server
Client
Tools
Stream Logs
Browse through
rotated logs by date
Ksnode Tooling
48. Samza Tweaks to ver 0.9.1
● Using ThreadJobFactory - Simplifies deployment and reduces overhead
● SAMZA-41 - range based static partition range assignment
● SAMZA-775- size based Prefetch buffer
○ Default was count based (OOM), and not bytes based
○ Set per topic per job to 60 * peak bytes / sec over the past week.
49. Samza Tweaks to ver 0.9.1
● Backported from 0.10
○ SAMZA-655 - environment variable configuration rewriter
■ Pass config from RDS to executor to Docker to Samza Job
○ SAMZA-540 - expose latency related metrics in OffsetManager
■ checkpointed offset gauge
50. More Info - Monal’s Samza Meetup Slides (10/2015)
Netflix Samza ver 0.9.1 Contributions
56. A True Story
● 80% Loss over 6 hour period
● Large Kafka clusters were impacts, smaller ones were fine
57. ● At times things go wrong, and there’s no turning back
● Reduce complexity
● Minimize blast radius
● Find a way to start over fresh
Lessons Learned
59. ● Cold standby Kafka cluster with 3 instances and different instance type
● Different ZooKeeper cluster with no state
● When failover occur
○ Scaling up cluster
○ Creating topic
○ Creating new routing jobs for failover cluster
○ Switch producer traffic!
Failover
66. Open source and community participation is
an integral part of our strategy and culture
67. Not DevOps, but move towards NoOps
You build it! You run it!
68. My team has
● No dedicated product or project managers
● No separate devops or operations team
We build and run what you saw today!
69. ● This does not mean we are constantly overworked
○ we make wise and simple choices and
○ lean towards automation & self-healing systems
We build and run what you saw today!
70. We built a pipeline in a year with
A very small team,
Relevant new technology,
Contributed back to OSS, and
Processed over 1 Trillion messages / day
Culture Impact
78. ● Starting out Proof Of Concept with Apache Flink
● Exploring Apache Beam
SPaaS - init( )
BETA
79. Why Apache Beam?
○ Portable API layer for building sophisticated data processing
applications
○ Unified APi for processing bounded and unbounded data sources
○ Google lineage - Dataflow model, and reflects Google’s current work
SPaaS - “Beam Me Up, Scotty ! "
80. Why Flink?
○ Flink implements the dataflow model
○ Correctness of results and powerful features for reasoning about time
○ Checkpoints, exactly-once processing
○ Event time, processing time, watermarks, triggers, aligned windows
(fixed, sliding), unaligned windows (dynamic or session windows)
○ Flink’s core is a streaming engine
SPaaS - Flink