Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Top Ten Kafka® Configs

806 Aufrufe

Veröffentlicht am

Bill Bejeck, Software Engineer, Confluent

Veröffentlicht in: Technologie
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Top Ten Kafka® Configs

  1. 1. 1 Top Ten Kafka Configs Important Configurations for Optimum Kafka Performance and Robustness
  2. 2. 2 Brief Introduction • Worked at Confluent (Streams Team) 2 years • Apache Kafka Committer • Author Kafka Streams in Action
  3. 3. 3 Special Thanks • Dustin Cote • Justin Manchester • Koelli Mungee • Ryan Prigeon
  4. 4. 4 Agenda • Quick Intro to Kafka • Ground Rules • Broker Configs • Client Configs • Consumer • Producer
  5. 5. 5 Intro to Kafka Broker 1 Broker 2 Broker 3
  6. 6. 6 Intro to Kafka Broker 1 Broker 2 Broker 3 Topic
  7. 7. 7 Intro to Kafka Topic 0 1 2 3 Next record
  8. 8. 8 Intro to Kafka 0 1 2 3 Producer
  9. 9. 9 Intro to Kafka 0 1 2 3 Producer Consumer
  10. 10. 10 Intro to Kafka 0 1 2 3 Producer Consumer Consumer reads available records then commits the offsets
  11. 11. 11 Configuration Ground Rules • Kafka is the central nervous system of your Event based platform • Proper configuration is important
  12. 12. 12 Configuration Ground Rules Not All Configurations need to be tuned!
  13. 13. 13 Configuration Ground Rules • Make sure you understand the behavior and what you’re trying to achieve before changing configs • Focus on what’s essential for your needs
  14. 14. 14 Broker Configs Producer Broker Consumer
  15. 15. 15 Broker Configs • JMX metrics • UncleanLeaderElection • Retention (Topic and Broker) • Min InSync Replicas (discussed in Producer section)
  16. 16. 16 JMX Metrics • JMX metrics not exposed remotely by default • No insight into broker activity without them
  17. 17. 17 JMX Metrics • Set the environment variable JMX_PORT to enable remote JMX metrics
  18. 18. 18 JMX Metrics • With environment variable JMX_PORT set Kafka uses these default properties for JMX -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
  19. 19. 19 JMX Metrics • To override/customize the JMX settings, i.e. provide authentication set the KAFKA_JMX_OPTS env variable
  20. 20. 20 JMX Metrics Viewing JConsole https://docs.oracle.com/javase/8/docs/technotes/guid es/management/jconsole.html
  21. 21. 21 JMX Metrics Connecting ssh -fN -D 7777 -i ~/.ssh/<key name> <instance-public-ip> jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 -J-DsocksNonProxyHosts= service:jmx:rmi:///jndi/rmi://localhost:<JMX_PORT>/jmxrmi
  22. 22. 22 JMX Metrics BrokerTopicMetrics • MessagesInPerSec • BytesInPerSec • BytesOutPerSec
  23. 23. 23 JMX Metrics RequestMetrics • Requests per second • Produce - requests from producer to send records • FetchConsumer – requests from consumer to retrieve records • FetchFollower – requests from follower broker to keep in sync with leader for given topic-partition
  24. 24. 24 JMX Metrics RequestChannel • ReqeustQueueSize Special thanks to Dustin Cote for the Broker Load metrics
  25. 25. 25 UncleanLeaderElection Leader Broker Follower Broker Leader has records up to offset 300 Follower is only up to 150
  26. 26. 26 UncleanLeaderElection Leader Broker Follower Broker Leader has records up to offset 300 Follower is only up to 150
  27. 27. UncleanLeaderElection Allow Follower to become leader • Almost immediately available • Potential for missing records
  28. 28. UncleanLeaderElection Wait to bring back old leader broker • Not highly available • Downtime until old leader recovered • No potential for missing records
  29. 29. UncleanLeaderElection • Default setting is false • Use with caution
  30. 30. 30 Retention • Size • Age of records • now – largest timestamp > retention time
  31. 31. 31 Timestamps - CreateTime Producer sets timestamp Producer Broker Topic
  32. 32. 32 Timestamps -LogAppendTime Producer sets timestamp Producer Broker Topic
  33. 33. 33 Retention Topic A retention 1 week Topic B retention 2 days Kafka Streams Application
  34. 34. 34 Consumer Configs Producer Broker Consumer
  35. 35. 35 Consumer Configurations • Max Poll Interval • Committing
  36. 36. 36 Max Poll Interval max.poll.interval.ms • max time between poll calls • If exceeded then consumer kicked out of group, rebalancing while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //do something with recrods } }
  37. 37. 37 Max Poll Interval max.poll.interval.ms = 30000/30 seconds while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
  38. 38. 38 Max Poll Interval Processing time for fetched records on consumer takes 45 seconds while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //Processing in this loop takes 45 seconds this consumer kicked out of group, and a rebalance is triggered } }
  39. 39. 39 Max Poll Interval max.poll.interval.ms solutions - • Increase the poll interval time – maybe • Possibly decrease max.poll.records while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
  40. 40. 40 Max Poll Interval max.poll.interval.ms solutions - • Decrease processing time • Take processing out of loop while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5)); for (ConsumerRecord<String, String> record : records) { //This loop must complete in under 30 seconds } }
  41. 41. 41 Committing Partition A Consumer By default auto commit enabled every 5 seconds
  42. 42. 42 Committing Partition A Consumer Only commit last record processed
  43. 43. 43 Committing 1 2 3 4 5 6 7 8 9 10 6 last processed offset 10 Records in batch
  44. 44. 44 Committing 1 2 3 4 5 6 7 8 9 10 Committing offset 11 means 7, 8, and 9 committed as well!
  45. 45. 45 Producer Configs Producer Broker Consumer
  46. 46. 46 Producer Configs • Linger • Acks • Retries and Delivery Timeouts • Keeping Records in Order
  47. 47. 47 Linger Producer Record Producer Record Buffer
  48. 48. 48 Linger – Normal Traffic Producer Record Buffer
  49. 49. 49 Linger – Lighter Traffic Producer Record Buffer
  50. 50. 50 Linger Trade Offs • Wait for more records • Waiting to send a batch • Decreased number of requests to broker
  51. 51. 51 Linger Trade Offs • Set linger.ms to zero • Send batches faster • More requests to broker
  52. 52. 52 Acks • Number of acknowledgements from broker that it has persisted the message • Zero • One • All
  53. 53. 53 Acks Zero Producer “Fire” and forget don’t wait for any response Broker (Leader) Broker (Follower) Broker (Follower)
  54. 54. 54 Acks One Producer Produce records and wait for leader broker to acknowledge Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  55. 55. 55 Acks All Producer Produce waits for leader broker to acknowledge after all followers have acknowledged Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  56. 56. 56 Acks All Producer Produce waits for leader broker to acknowledge after all followers have acknowledged Broker (Leader) Broker (Follower) Broker (Follower) Got it! Got it!
  57. 57. 57 InSync Replicas • acks = “all” • Leader broker is considered a replica!
  58. 58. 58 InSync Replicas Producer acks = “all” min.insync.replicas = 1 Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  59. 59. 59 InSync Replicas Producer acks = “all” min.insync.replicas = 1 possible data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  60. 60. 60 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  61. 61. 61 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it! Got it!
  62. 62. 62 InSync Replicas Producer acks = “all” min.insync.replicas = 2 no data loss Broker (Leader) Broker (Follower) Broker (Follower) Got it!
  63. 63. 63 Retries Topic A Partition 0 Producer Retriable Exception
  64. 64. 64 Retries Topic A Partition 0 Producer Retriable Exception 0.. Max Retries
  65. 65. 65 Retries Topic A Partition 0 Producer Retriable Exception 0.. Max Retries or
  66. 66. 66 Retries Topic A Partition 0 Producer Retriable Exception Use delivery.timeout.ms for limit of time from sending to receiving acknowledgement see KIP-91 for details
  67. 67. 67 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1
  68. 68. 68 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2
  69. 69. 69 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1
  70. 70. 70 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1 Batch 2 processed before Batch 1 !
  71. 71. 71 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 2 Batch 1 max.in.flight.requests.per.connection = 1
  72. 72. 72 Keeping Records in Order Topic A Partition 0 Producer Retriable Exception Batch 1 Batch 1 max.in.flight.requests.per.connection = 1 Batch 2
  73. 73. 73 Summary • Configuration is important to the health of a Kafka Cluster/ • Need to know which knobs to turn • Measure the before and after • Try it in development first
  74. 74. 74 Summary • Kafka The Definitive Guide - http://shop.oreilly.com/product/0636920044123.do • Apache Kafka docs - https://kafka.apache.org/documentation/
  75. 75. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE
  76. 76. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
  77. 77. 77 Thanks! Stay in Touch! • https://slackpass.io/confluentcommunity • https://www.confluent.io/blog/ • Twitter @bbejeck • We are hiring! https://www.confluent.io/careers/

×