Netflix Keystone Pipeline processing 600 billion events a day, and detailed treatise on the modification of and use of Samza for real time routing of events including docker.
25. Mind bender - Sink Isolation
â Multiple Samza jobs for one Kafka source topic
â Each job processes messages for one sink
â E.g. separate job for each S3 & ElasticSearch cluster sinks
â Tradeoff
â Sink isolation for extra load on Kafka source topic cluster
â Initial release
â Each job processes partitions only from one topic
26. Samza Job Details
â Use window function to implement health check
â task.window.ms=30000
â Batch requests to sinks
â Explicit offset commits only
â automatic commits disabled - task.commit.ms=-1
31. 1 checkpoint topic per kafka cluster, sink, source topic
â Change the number of samza jobs for a topic
â Easily redistribute the partitions across jobs
â Add new partitions seamlessly
â Our naming scheme facilitates migrating topics to other clusters
32. Job Startup Delays reading Checkpoint
Causing health check failures - timeout 5 min
What to do?
34. Checkpoint topic Samza Job Configuration
Replication factor is hard coded to 3
task.checkpoint.system=checkpoint
task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory
task.checkpoint.segment.bytes=3145728
35. Add. Checkpoint Information
â About 300 bytes per offset commit
â Change log topic logs into the same checkpoint offset topic
â Even if not enabled, one time large message with system-stream-partition
inserted into the same checkpoint offset topic
42. SAMZA-41 - static partition range assignment
job.systemstreampartition.matcher.class=
org.apache.samza.system.RegexSystemStreamPartitionMatcher
job.systemstreampartition.matcher.config.ranges=[8-10]
^8&|^9$|^10$
you need
44. Prefetch Buffer - When is it going to OOM?
â Default count based per Samza container
â (50,000 / # partitions) per topic
â systems.source.samza.fetch.threshold=50000
â Hard to get it right and avoid OOM
â changing message size
45. SAMZA-775- size based Prefetch buffer
â How much of heap should I use for prefetching?
â systems.source.samza.fetch.threshold.bytes=200000000 (200MB)
â per system / stream / partition
â if > 0 precedence over systems.source.samza.fetch.threshold
46. SAMZA-775- size based Prefetch buffer
â systems.source.samza.fetch.threshold.bytes is a Soft limit
â bytes limit + size of last max message stream
â I donât get it, where is the example?
47. SAMZA-775- size based Prefetch buffer
â systems.source.samza.fetch.threshold.bytes=100000 (100K)
â 50 SystemStreamPartitions
â per system-stream-partition threshold is (100000 / 2) / 50 = 1000 bytes.
â Enforced limit would be
â 1000 bytes + size of last message from the partition
48. SAMZA-775- size based Prefetch buffer
â Value of systems.source.samza.fetch.threshold.bytes based on
â Incoming traffic Bps into source Kafka
â 60 seconds of buffer with region failover traffic
â Samza in memory data structures (2 x message size)
49. SAMZA-775- size based Prefetch buffer
â How does it perform?
â Per message overhead within 0.02% of computed heuristics in the patch
â Actual footprint of systems.source.samza.fetch.threshold.bytes is 10-15% at
the most in worst case.
â Example: If set to 200MB, worst case observed 230MB
50. SAMZA-775- size based Prefetch buffer
â Con
â Implementation to enforce systems.source.samza.fetch.threshold.bytes is very
dependent on the implementation version of Samza
â Hence, higher maintenance when code changes. However,
Well Worth It! Ergonmic Config! Adds Stability!
51. SAMZA-655 & SAMZA-540
â Backported from 0.10
â environment variable configuration rewriter
â Pass config from RDS to executor to Docker to Samza Job
â expose latency related metrics in OffsetManager
â checkpointed offset guage
65. End to End metrics
â Producer to Router latency
â Avg. about 2.5 seconds
â 90 percentile topics under 2 sec
â Kafka to Router consumer lag (estimated time to catch up)
â 65 percentile under 500ms
â 90 percentile under 5 seconds
â Producer event timestamp to Samza job router avg latency - 6 seconds
68. Wait thereâs more in the pipeline...
â Self service tools
â Multi-tenant Stream Processing as a Service - SPaaS
â probably add spark streaming to the mix
â Event traceability - on demand and sampled
â As number of jobs increase checkpoint topic may give way to Cassandra
â Optimization & Automation
72. Fronting Kafka Instances
â 2700 d2.xl AWS instances across 3 regions for regular & failover traffic
â d2.xl
â Large disk (6TB) - 450-475MB/s of sequential I/O throughput
â 30GB memory, 700 Mbps medium network capability
â Replication lag above 18MB/second per broker with thousands of partitions
â cons: multiple instances on same physical host - increases failures
73. Kafka Capacity Planning
1. Stay under 20k partitions per cluster (14K)
2. Leave â 40% free disk space on each broker for growth & movement
3. Throughput per partition based on 1, 2, # of brokers, and the retention
period
74. Partition Assignment
â All assignments Zone / Rack aware
â Strategy 1 - Multiple of brokers
â Stategy 2 - Stateful Round Robin
75. Kafka Auditor as a Service
â Broker monitoring
â Consumer monitoring
â Heart-beat & Continuous message latency
â On-demand Broker performance testing
â Built as a service deployable on single or multiple instances