Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

1. Securing Kafka At
Zendesk
24th
Aug, 2020

2. Prasenjit Nag (Joy)
Senior Software Engineer, Foundation, Zendesk

3. Assumptions
● basics of Kafka
● basics of mutual TLS (Transport Layer
Security)
● basic kubernetes (K8S) terms

4. OVERVIEW - KAFKA INSIDE A ZENDESK PARTITION
Plaintext
communication

5. OVERVIEW - GLOBAL EVENT BUS (GEB)
Propagate
events between
partitions

6. We need to make inter-pod
communications secure
OVERVIEW

7. OVERVIEW
➔ Authenticated
➔ Encrypted
➔ Authorized

8. OVERVIEW
● only authenticated clients
● encrypted connections
● access to authorised topics
Future kafka clusters

9. mTLS for Kafka
OVERVIEW

10. OVERVIEW - PUBLIC KEY INFRASTRUCTURE

11. Kafka had TLS support
OVERVIEW

12. We needed a
Certificate Authority
OVERVIEW

13. HashiCorp’s Vault can be the
Certificate Authority
OVERVIEW

14. Expectations

15. 𝤿 Certificate revocation
𝤿 CA (Certificate Authority) rotation
𝤿 Logging/Metrics collection/Monitoring
𝤿 Easy client onboarding
EXPECTATIONS
𝤿 Automated certificate generation/regeneration

16. Solution
Components

17. SOLUTION ARCHITECTURE - PREPARING VAULT

18. SOLUTION ARCHITECTURE - CLIENT ARCHITECTURE

19. SOLUTION ARCHITECTURE - BROKER ARCHITECTURE

20. SOLUTION ARCHITECTURE - AUTH MANAGER

21. SOLUTION ARCHITECTURE - AUTH MANAGER
PKI auth manager is a wrapper around consul-template
https://github.com/hashicorp/consul-template
● one-off script or a daemon
● render templates using consul and vault apis
● regenerate secrets based on TTL (Time To Live)
● script execution on updates

22. SOLUTION ARCHITECTURE - CONSUL TEMPLATE

23. LESSONS LEARNT - CONSUL TEMPLATE INVOCATION
consul-template …
-template "...:..:/script"

24. SOLUTION ARCHITECTURE - CA ROTATION
𝤿 global record of current root
𝤿 certificate regeneration on
root changes
𝤿 broadcasting root changes
𝤿 reasonably fast rotation
𝤿 reload regenerated
certificates

25. SOLUTION ARCHITECTURE - CA ROTATION

26. SOLUTION ARCHITECTURE - CA ROTATION - CONSUL KEY

27. SOLUTION ARCHITECTURE - CA ROTATION - BLOCKING QUERIES
GET /v1/kv/path-to-consul-key?index=10&wait=1m0s

28. SOLUTION ARCHITECTURE - CA ROTATION
To do CA rotation, we need:
✅ global record of current root
✅ certificate regeneration on root changes
✅ broadcasting root changes
✅ reasonably fast rotation
𝤿 reload regenerated certificates

29. SOLUTION ARCHITECTURE - CA ROTATION

30. SOLUTION ARCHITECTURE - CA ROTATION
To do CA rotation, we need:
✅ global record of current root
✅ certificate regeneration on root changes
✅ broadcasting root changes
✅ reasonably fast rotation
✅ reload regenerated certificates

31. SOLUTION ARCHITECTURE - CA ROTATION
Lets simulate a CA rotation

32. SOLUTION ARCHITECTURE - CA ROTATION - VALUE OF CONSUL KEY
Certificate Root: root-a
Additional Trust Roots: []
Before rotation

33. SOLUTION ARCHITECTURE - CA ROTATION - VALUE OF CONSUL KEY
Certificate Root: root-a
Additional Trust Roots: [root-b]
During rotation

34. SOLUTION ARCHITECTURE - CA ROTATION - VALUE OF CONSUL KEY
Certificate Root: root-b
Additional Trust Roots: [root-a]
Swap Roots

35. SOLUTION ARCHITECTURE - CA ROTATION - VALUE OF CONSUL KEY
Certificate Root: root-b
Additional Trust Roots: []
Complete Rotation

36. Performance
impact

37. 10%-15% increase in CPU
usage on a test environment
PERFORMANCE IMPACT

38. Some reasons behind the performance impact?
PERFORMANCE IMPACT - REASONS
● mTLS network traffic overhead
● encryption and decryption are CPU intensive
● no ZERO COPY optimisation
● performance of JVM SSL engine

39. PERFORMANCE IMPACT
What is ZERO COPY optimisation?

40. PERFORMANCE IMPACT
What is ZERO COPY optimisation?

41. PERFORMANCE IMPACT
Why we lose ZERO COPY optimisation with TLS?
what is in the disk is exactly
what gets sent over the wire

42. PERFORMANCE IMPACT
Kernel TLS to the rescue? Maybe?
https://www.kernel.org/doc/html/latest/networking/tls.html

43. Client
onboarding

44. ✅ onboarding guides
✅ local Kafka cluster with TLS
✅ example clients configured with TLS
✅ pairing with early adopters
𝤿 tighter K8S integration
CLIENT ONBOARDING
How are we driving adoption?

45. CLIENT ONBOARDING - TIGHTER INTEGRATION WITH K8S

46. Lessons

47. LESSONS LEARNT
Revocation check of public SSL certificates is widely broken
● Neglected tooling
● Kafka doesn’t support custom revocation check
Two ways of checking Revocation:
● Certificate Revocation List (CRL)
● Online Certificate Status Protocol (OCSP)

48. LESSONS LEARNT - REVOCATION
https://somedomain.com/kafka-pki/crl
-Dcom.sun.security.enableCRLDP=true -Dcom.sun.net.ssl.checkRevocation=true
Certificate chain of *.somedomain.com
0 .. Sectigo Limited….
1 .. The USERTRUST Network/CN=USERTrust RSA
2 .. CN=AddTrust External CA Root
3 .. CN=AddTrust External CA Root

49. LESSONS LEARNT
Automating CA rotation of a live
system is hard

50. ✅ Automated certificate generation/regeneration
❌ Certificate revocation
✅ CA (Certificate Authority) rotation
✅ Logging/Metrics collection/Monitoring
✅ Easy client onboarding
EXPECTATIONS

51. Extras
● GEB consumer/producer rate
limiting
● mTLS tooling/infrastructure reuse

52. git.io/ca-rotation

53. Demo

54. Thank you
Github: joycse06
Twitter: joycse06
Email: pnag@zendesk.com