Inspired by the Jepsen series of database test suites I created kafka-mirror-tester, a cross-Atlantic automated test suite for Kafka mirroring using Golang and Kubernetes. There, I said k8s, need I say more? Join me to learn how k8s solves database automation tasks and Go drives those tests.

1. Infrastructure
testing using
Kubernetes
@rantav

2. OR:
Testing Kafka
replication over the
Atlantic using
Golang, Kubernetes
and friends

3. Hello!
I am Ran Tavory
I work on multi-region at
appsflyer
You can find me @rantav
(twitter/gmail/facebook)
3

4. What is
AppsFlyer?
Data collection and
analytics system
4

5. The Goal:
Test Kafka
Replication
⊙ Replicate 100MB per second
⊙ Validate correctness
⊙ Measure latency
⊙ Run test cases (failure scenarios)
○ Broker crash, cluster resize,
packet loss...
5

6. Does this
remind
anyone
anything?
⊙ Run test cases (failure scenarios)
○ Broker crash
○ cluster resize
○ packet loss
○ etc...
6

7. “ Jepsen: Call me maybe
// Kyle Kingsbury
7

8. Kafka ?
Queueing System
Stream Processing
Brokers
Topics
Partitions
8

9. HOW ?
High level design
9

10. Kafka Replication
From https://github.com/AppsFlyer/kafka-mirror-tester
10

11. With a home-grown producer and consumer
written in Golang
11

12. WHAT DO WE
WANT?
12
DOCUMENTATION
HOW DO WE DO IT? With
Code

13. WHAT DO WE
WANT?
13
REPRODUCIBILITY
HOW DO WE DO IT? With
Code

14. WHAT DO WE
WANT?
14
MONITORING
HOW DO WE DO IT? With
Code

15. KUBERNETES
15

16. A brief
introduction to
Kubernetes
16
Do we need
an intro to
Kubernetes?

17. CLUSTER MANAGER
Kubernetes is a
17

18. NODES
The host running a kubelet and a
set of Pods
18
It manages

19. PODS
Pods consist of 1 or more containers
19
Nodes contain

20. DEPLOYMENTS
A set of stateless pods
20
Pods run

21. STATEFUL SET
A set of stateful pods
21
Pods also run

22. KUBECTL
The k8s CLI
22
You control k8s using

23. $ make k8s-all
23

24. What does
that do?
⊙ Provision VMs in AWS
⊙ In two regions. Ireland and Virginia.
⊙ Setup VPCs, Subnets, Routing Tables
⊙ Create Security Groups
⊙ Setup Load Balancers
⊙ Install Kubernetes (etcd, masters, nodes)
⊙ Setup Monitoring (Prometheus & Grafana and
install dashboard)
⊙ Install Weave Scope
⊙ Install Kafkas (and test them)
⊙ Install uReplicator (and test it)
⊙ Install test programs (written by my in Go)
⊙ And more… (ASGs, DHCP etc)
24

25. $ kops create cluster
--zones us-east-1a,us-east-1b,us-east-1c
--node-count 40
--node-size i3.large
--master-size m4.large
--master-zones us-east-1a
--networking calico
--cloud aws
25

26. What are we
building?
us-east-1
26
eu-west-1

27. What are we
building?
us-east-1
40 nodes
k8s cluster
27
eu-west-1
48 nodes
k8s cluster

28. What are we
building?
us-east-1
40 nodes
k8s cluster
30 brokers
kafka cluster
28
eu-west-1
48 nodes
k8s cluster
30 brokers
kafka cluster

29. What are we
building?
us-east-1
40 nodes
k8s cluster
30 brokers
kafka cluster
29
eu-west-1
48 nodes
k8s cluster
30 brokers
kafka cluster
8 workers
uReplicator

30. What are we
building?
us-east-1
40 nodes
k8s cluster
30 brokers
kafka cluster
10 producers
30
eu-west-1
48 nodes
k8s cluster
30 brokers
kafka cluster
8 workers
uReplicator
4 consumers

31. 31
End
Result

32. 32
Grafana
Dashboard

33. 33
Weave
Scope

34. 34
Weave
Scope

35. 35
kubectl
example

36. TEST IT.
Let’s put all this to test now
36

37. $ kubectl
--context us-east-1.k8s.local
-n kafka-source
delete pod kafka-source-2
37
KILL A BROKER

38. 38
KILL A BROKER

39. $ kubectl
--context us-east-1.k8s.local
-n kafka-source
scale
statefulset kafka-source
--replicas 29
39
RESIZE KAFKA CLUSTER

40. 40
RESIZE KAFKA CLUSTER

41. $ kubectl
--context eu-west-1.k8s.local
-n ureplicator
scale deployment
ureplicator-worker
--replicas 9
41
ADD uREPLICATOR WORKER

42. 42
ADD uREPLICATOR WORKER

43. $ make k8s-redeploy-tests
43
ADD NEW TOPIC

44. 44
ADD NEW TOPIC

45. $ make k8s-kafka-shell-source
$ bin/kafka-topics.sh --zookeeper
zookeeper:2181 --alter --topic
topic5 --partitions 300
45
ADD PARTITIONS

46. $ kubectl --context eu-west-1.k8s.local
-n ureplicator port-forward
ureplicator-controller-76ff85b889-l9mzl 9000
$ curl -X DELETE http://localhost:9000/topics/topic5
$ curl -X POST -d
'{"topic":"topic5", "numPartitions":"300"}'
http://localhost:9000/topics
46
ADD PARTITIONS (cont)

47. 47
PACKET LOSS

48. 48
PACKET LOSS

49. This is so much
FUN!!!
But how do we
actually do it?
49

50. CODE DEEP DIVE
Golang Producer/Consumer
(just a few bites)
50

51. 51
Producer
Main loop

52. 52
Consumer
Main loop

53. 53
Consumer
Process
message

54. “ Question: How can multiple
consumers validate message
arrival order?
54
Redis?
DynamoDB?
No! There’s a trick!

55. 55
Message format

56. 56
Producer
Sequence
numbers
messageKey = seq % partitions
perKeySeq = seq / partitions

57. CODE DEEP DIVE
Kubernetes, head first
(just a few bites)
57

58. 58
Kafka
kind: StatefulSet
replicas: 30

59. 59
Metrics
sidecar
container

60. 60
Kafka
anti-
affinity

61. 61
uReplicator
deployment

62. 62
Service
monitoring

63. $ make k8s-delete-all
63

64. 64

65. Thanks!
Any questions?
You can find me at @rantav &
rantav@appsflyer.com
65
This presentation:
https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes-
and-go
The project: https://github.com/AppsFlyer/kafka-mirror-tester