Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 30 Anzeige

Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex

Herunterladen, um offline zu lesen

This session will dive into our most successful (and unsuccessful!) multi-cluster event replication patterns.

An x-ray of the cross cluster distribution model that powers our globally distributed APIs will touch on the benefits that this model has provided in terms of client API experience, delivery agility and developer experience.

We will focus on recipes for effective use of Mirror Maker event replication to power platform distribution including the challenges of managing a 'fan in' event replication workflow - pulling events created in satellite clusters back to a mothership cluster for processing.

We will introduce the elegant technique of replication event multiplexing - which can be used to simplify the burden of managing a 'fan-in' replication topology by eliminating regional awareness from the application domain and improving replication health monitoring & observability.

This session will dive into our most successful (and unsuccessful!) multi-cluster event replication patterns.

An x-ray of the cross cluster distribution model that powers our globally distributed APIs will touch on the benefits that this model has provided in terms of client API experience, delivery agility and developer experience.

We will focus on recipes for effective use of Mirror Maker event replication to power platform distribution including the challenges of managing a 'fan in' event replication workflow - pulling events created in satellite clusters back to a mothership cluster for processing.

We will introduce the elegant technique of replication event multiplexing - which can be used to simplify the burden of managing a 'fan-in' replication topology by eliminating regional awareness from the application domain and improving replication health monitoring & observability.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex

  1. 1. Fan-out, fan-in & the multiplexer Replication recipes for global platform distribution
  2. 2. Introduction Alex Hilton Engineering Manager Foreign Exchange Team Passions - ! Family - 🍹 Mixology - 🎲 Online Catan - 🚀 Building the worlds next gen financial infrastructure + +
  3. 3. Global Platform Distribution Challenge How best to centrally manage events generated in remote clusters? HQ Distribution across remote clusters yields great portability
  4. 4. Global Platform Distribution Challenge How best to centrally manage events generated in remote clusters? • Why does this challenge exist? • Kafka distribution best practices & mistakes • Mirror Maker replication recipes • Topic Multiplexing • Separate global distribution concerns from application domain • Improve observability & monitoring 1 2 3
  5. 5. Information Distribution Challenge Foreign Exchange Platform FX Partners (Centralized) Globally distributed APIs: • Payment • Transfer • Card • Wallet
  6. 6. Information Distribution Challenge Foreign Exchange Platform Desired Experience: 🌏 Local 🚀 Fast ⭐ High availability Our Constraints: 🎯 Central connections Aggregation opportunities
  7. 7. Key Learning – Kafka promotes decomposition Isolation benefits • Single responsibility • Single writer From monolith… …to pluggable microservice pipeline
  8. 8. Why is Kafka a good fit? Pluggability Kafka is a universal application dependency Workflows modelled as chain of microservices Pluggability benefits - Unburden critical path - Support innovation - Composable capability - Portability
  9. 9. API on Kafka Pattern Price distribution API on Kafka Pattern • Push state & all decision making to very edge of system • Fast API response time 🚀 Fast (API latency) • Read API driven from in-memory state • Write API publish instructions as events ⭐ High availability • Convenient horizonal scalability • Reduce dependencies on critical path 🌏 Local • How to solve…? Instruction execution
  10. 10. Learnings – Simplify the critical path for API responses • Burden Client with Kafka RPC • Exactly once semantics Do Don't • Embrace eventual consistency • Enforce idempotency
  11. 11. Global Distribution Pattern 🌏 Global Distribution Pattern • Remote Kafka clusters power global API distribution • Mirror maker (v2) used to synchronize clusters • Fan out vs Fan in 🚀 Fast (API latency) • Internalize the cost of WAN distribution ⭐ High availability • Opportunities for DR / cross region fallback (cross cluster) 🌏 Local • We can deploy APIs near customers • Kafka yields application portability & composability M irror m aker M irror m aker
  12. 12. Multi-cluster Challenges - Fan out Fan out replication Pattern • Mirror maker pull model • Avoid monolithic replication processor • MM instance deployed to each remote region • Topic data is cloned (but not offsets). • Topic name remains the same • Offsets vary
  13. 13. Mirror Maker Best Practice - Isolation MM Isolation Best Practice • Separate MM instances for each region • Isolate meta data per source-destination combination • Version meta data to support case reset / remodel ⭐ Benefits • Limit blast radius of cluster stability issues • Flexibility to shutdown / maintain regions • Migration pathway to support MM upgrades MM Kafka cluster naming convention: version.source.dest.[dest,src] v5 . HQ . US . dest
  14. 14. Sample Mirror Maker Metadata
  15. 15. Fan in… and the multiplexer
  16. 16. Multi-cluster Challenges - Fan in Fan in challenge Compromised isolation on merged topic • Harder to reconcile / monitor • Violate single writer principal
  17. 17. Multi-cluster Challenges - Fan in Fan in – better isolation Isolation ✅ • Region, topic, mirror maker Portability ❌ • Application & env burden • API & topic region • Processor consumption Scalability 😐 • Multiple workflows Monitoring 😐 • How to monitor replication during off peak periods?
  18. 18. Fan in – the naïve way
  19. 19. Multiplexing to the rescue! a method by which multiple analog or digital signals are combined into one signal over a shared medium. The aim is to share a scarce resource.
  20. 20. Multiplexing to the rescue! Could multiplexing techniques help us to abstract distribution / replication concerns away from application / business domain?
  21. 21. Apply multiplex concept to fan-in challenge We can solve isolation, distribution, portability, replication monitoring concerns without burdening our application • Multiplex service can interlace all source topics into a single channel • Simpler replication, easier to isolate, easier to monitor • Demux service can deinterlace multiplexed byte stream
  22. 22. Multiplexing unburdens application domain from distribution concern Isolation ✅ Portability ✅ Scalability ✅ Monitoring ❓
  23. 23. Generating Mux Stream & Partitioning Multiplexing is not difficult – entire mux service is 300 lines of Kotlin Respect source partition sequence. PartitionKey = srcTopic + srcPartition Can use ByteArray serdes to copy record key & value to mux topic. No need to use original schema Kafka headers provide convenient way to carry original context to demux (the source partition & topic name)
  24. 24. Kafka Header Metadata
  25. 25. Improving observability & monitoring Throughput and latency metrics • Kafka mux headers carry source region to help improve event observability • Demultiplexer exposes latency / throughput metrics to power rich dashboard
  26. 26. Improving observability & monitoring
  27. 27. Improving observability & monitoring Dead replication alerts • Synthetic ‘keepalive’ events generated to keep replication & monitor warm during off-peak time • Synthetic events internalized to multiplexer / demultiplexer & do not affect business workflow
  28. 28. Mux Dashboard
  29. 29. Kafka Distribution Patterns – Fan in Fan out & the Multiplexer ✅ Isolation ✅ Portability ✅ Scalability ✅ Monitoring 🌏 Local 🚀 Fast ⭐ High availability Fan In + Multiplexer Fan Out Conclusions - Install Kafka cluster in each customer region - Keep API processing simple - Strive for strong isolation between regions - Separate application & distribution concerns
  30. 30. What Next…? Support Multiple HQ Regions Share The Multiplexer Code Read more on our Blog! medium.com/airwallex-engineering Feedback / questions: alex.hilton@airwallex.com airwallex.com/careers

×