Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

5 lessons learned for successful migration to Confluent cloud | Natan Silinitsky, wix.com

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 20 Anzeige

5 lessons learned for successful migration to Confluent cloud | Natan Silinitsky, wix.com

Herunterladen, um offline zu lesen

Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (19)

Ähnlich wie 5 lessons learned for successful migration to Confluent cloud | Natan Silinitsky, wix.com (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

5 lessons learned for successful migration to Confluent cloud | Natan Silinitsky, wix.com

  1. 1. 5 lessons learned for Successful Migration to Confluent Cloud Natan Silnitsky Backend Infra Developer, Wix.com natans@wix.com twitter@NSilnitsky linkedin/natansilnitsky natansil.com
  2. 2. >200M Registered users (website builders) from 190 countries @NSilnitsky At Wix >2,000M Kafka messages produced / Day 2,000 Microservices
  3. 3. * easily manage
  4. 4. Data Center 2 Data Center 1 @NSilnitsky Confluent Cloud Kafka Cluster On premise Kafka Cluster I Kafka Cluster II Kafka Cluster III Kafka Cluster IV * AWS/GCP Microservice Microservice 2000
  5. 5. Automation, Automation, Automation 01
  6. 6. A Scala/Java high-level SDK for Apache Kafka. Greyhound @NSilnitsky
  7. 7. Site Apps Service Kafka Consumer Kafka Producer Greyhound wraps Kafka Ecom Catalog Service Kafka Broker Greyhound Producer Greyhound Consumer @NSilnitsky
  8. 8. Data Center 2 Data Center 1 Microservice @NSilnitsky Confluent Cloud Kafka Cluster On premise Kafka Cluster I Kafka Cluster II Kafka Cluster III Kafka Cluster IV * all at once?
  9. 9. Gradual over All at once 02
  10. 10. Data Center 2 Data Center 1 @NSilnitsky Kafka Cluster On premise Confluent Cloud Microservice HTTP HTTP Kafka Cluster I Kafka Cluster II Kafka Cluster III Kafka Cluster IV Option B Replicator Topic 5 Topic 4 Topic 3 Topic 2 Topic 1 Topic 6 Migration Orchestrator * self service - no downtime
  11. 11. First migrate test topics with relayed real traffic 03
  12. 12. Data Center 2 @NSilnitsky Kafka Cluster On premise Confluent Cloud Test app Real Topic->Test Topic Test Topics Kafka Cluster I Kafka Cluster II Kafka Cluster III Kafka Cluster IV Migration Orchestrator Replicator
  13. 13. Only needed topics/partitions 04
  14. 14. Data Center 2 Data Center 1 Microservice @NSilnitsky 0 Bytes 100 MB 170 MB Confluent Cloud
  15. 15. Adapt to Confluent Cloud APIs 05
  16. 16. Data Center 2 Data Center 1 Offset lag Monitoring @NSilnitsky Confluent Cloud Kafka Cluster On premise __consumer_offsets __consumer_offsets Kafka Admin SDK Kafka Admin SDK Unavailable Available
  17. 17. So, We used Greyhound and dedicated orchestration services to have automatic, safe, and gradual migration to Confluent Cloud @NSilnitsky
  18. 18. A Scala/Java high-level SDK for Apache Kafka. 0.1.5 is out! github.com/wix/greyhound
  19. 19. Thank You natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
  20. 20. Slides & More slideshare.net/NatanSilnitsky medium.com/@natansil twitter.com/NSilnitsky natansil.com

×