1. STREAMING
INFRA @ WISE
Levani Kokhreidze

2. 👋 I’m Levani

3. Agenda
1.
2.
3.
4.
A bit about Wise
💻 Engineering @ Wise
🔧 Stream Processing Platform
❓ Q/A

4. Wise is a global technology
company, building the best
way to move and manage
money around the world.
About Wise

5. 💻
Engineering
600>
Engineers
~100 Teams
6 Cities
3 Continents

6. 🔧 Stream
Processing Platform

7. 📰 History
Started building
streaming platform ~5
years ago
Covering the need of
real-time aggregations

8. ⏰ Today
We run >400 stream
processing jobs
~50TB of state
Mostly Kafka Streams
(but also Flink)

9. 🚀 Impact
Around 55% of
transactions are instant
Real-time aggregations
and decision making
scales well with growth
Stream Processing has
become a default choice

10. 🧩 How it all works

15. 🚜 Data Collection

16. Inhouse library for transactional
messaging to Kafka
Library for a consistent data
backfill. Can do enrichment, and
enforces best practices
(timestamps, partitioning, etc)
No CDC
Product teams are the responsible
for providing data for Stream
processing

17. 🧱 Storage

18. Cruise control for
balancing data across
the brokers
We run Streaming
Kafka cluster on
Kubernetes
One StatefulSet has
multiple disks

19. https://tinyurl.com/wise-kafka-k8s

20. ➕ ➕

21. Client-Broker mTLS with SPIRE

22. Client-Broker mTLS with SPIRE

23. Client-Broker mTLS with SPIRE

24. Client-Broker mTLS with SPIRE

25. Client-Broker mTLS with SPIRE

26. Client-Broker mTLS with SPIRE

27. https://tinyurl.com/wise-kafka-security

28. topic-name: "StreamingEngine.account.ledger"
owner-team: "account"
partition-count: 20
producers:
- name: account-ledger
consumers:
- name: streaming-server-profile-ledger
config:
cleanup.policy: "compact"
min.cleanable.dirty.ratio: "0.01"
segment.bytes: "1073741824"
Managing ~6K topics and
hundreds of principal
declarations
Inhouse library for managing
Kafka topic configurations as
yaml files

29. Broker enforces a default
quotas
Quotas are essential for a
self-service platform as
product teams can manage
their own stream processing
jobs

30. 🛠 Processing

31. Internal framework for
developing stream processing
jobs on top of Kafka Streams or
Flink
Mostly optimised for running
stateful workloads
Adds observability, tracing,
self-service capabilities and
more…

32. We provide a capability to
product teams to bootstrap a
new Stream processing job
Gradle plugins for building
docker images
Each Streaming job is deployed
in its own namespace in K8s as
StatefulSets.

33. streaming-pipelines:
ks-demo:
stream-processing-execution-engine: KAFKA_STREAMS
configuration:
auto.offset.reset: earliest
topology.optimization: all
num.stream.threads: ${ENV_KS_DEMO_PIPELINE_STREAM_THREADS:1}
num.standby.replicas: ${ENV_KS_DEMO_PIPELINE_STANDBY_REPLICAS:1}
flink-demo:
stream-processing-execution-engine: FLINK
configuration:
checkpoint.mode: ${ENV_FLINK_DEMO_CHECKPOINT_MODE:EXACTLY_ONCE}

34. @StreamProcessor("ks-demo")
public void ksDemo(final StreamsBuilder ksStreamsBuilder,
@StreamInput("KSDemo.events") final DataStream<Integer, Event> inputStream) {
...
}
@StreamProcessor("flink-demo")
public void flinkDemo(final StreamExecutionEnvironment see,
@StreamInput("FlinkDemo.events") final DataStream<Integer, Event> inputStream) {
...
}

35. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(6)).withGrace(Duration.ofMonths(8)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

36. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

37. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

38. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

39. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

40. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

41. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

42. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

43. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

44. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

45. inputStream
.groupBy((readOnlyKey, event) -> event.getUserId(), "user-id-repartition")
.withTime((key, event) -> event.getTimestamp())
.window(StateWindow.ofSize(Duration.ofMonths(1)).withGrace(Duration.ofMonths(2)))
.toSketchedStream()
.field(
new SketchSpec<>(
SketchKey.of("distinctTransfersCount"),
(key, value) -> value.getTransferId(),
(key, sketchKey, value) -> isIncrease(key, value) ? INCLUDE : REMOVE
)
)
.computeSketches()
.toStream()
.map(mapToYourDomain())
.sink(To.externalCluster("my-sketch-results-topic"));

46. Disk space usage dropped by ~33x

47. // SEI (Streaming Engine Images)
sei {
notifySlack = true
images {
image {
pipeline {
name = "ks-demo"
}
container {
// Optionally add environment variables to the container
environment = [
ENV_KS_DEMO_PIPELINE_STREAM_THREADS: "2"
]
}
}
image {
pipeline {
name = "flink-demo"
}
}
}
}

51. ⏫ High Availability

52. ⏫ HA
KIP-345: Static membership
protocol
Clustered deployments for
critical stream processing jobs
KIP-708: Rack awareness for
Kafka Streams

53. cluster2-ks-demo
cluster1-ks-demo

54. cluster2-ks-demo
cluster1-ks-demo

55. cluster2-ks-demo
cluster1-ks-demo

56. cluster2-ks-demo
cluster1-ks-demo

57. 📚 Summary