Do you wonder how to cope with the right to be forgotten? Do you wonder how to only process the events of individuals who have given their consent for processing their data? Do you wonder how to protect PII data of your users? Or do you wonder how to implement these across all your heterogeneous languages, clients and processing frameworks without having to re-implement all your streaming services? This talk is for you!
In this talk, we will answer these questions and show you
1) how transparent end-to-end encryption can be implemented on top of Apache Kafka;
2) how crypto-shredding can be used to forget individuals; and
3) how record based access control can be implemented on top of Apache Kafka.
Above all, we will show how this can be done without touching any applications by using an out-of-process architecture (Ã la service-mesh).
stackconf 2021 | Why you should take care of infrastructure driftNETWAYS
Similar to Handling GDPR with Apache Kafka: How to Comply Without Freaking Out? (David Jacot, Independent (formally Swisscom) Kafka Summit London 2019 (20)
3. @davidjacot
Kafka: What are the challenges?
3
Encryption
Right to be Forgotten
Consent
Kafka does not have an
real encryption story
Kafka only provides
Topics authorization
Kafka only knows how to
expire or compact events
! ?
4. @davidjacot
Fortunately, solutions exist!
4
Encryption
Right to be Forgotten
Consent
Kafka does not have an
real encryption story
Kafka only provides
Topics authorization
Kafka only knows how to
expire or compact events
E2E Encryption
Crypto-Shredding
Record based ACLs
! ? ✓
16. @davidjacot
Checkpoint
16
Encryption
Right to be Forgotten
Consent
Kafka does not have an
real encryption story
Kafka only provides
Topics authorization
Kafka only knows how to
expire or compact events
E2E Encryption
Crypto-Shredding
Record based ACLs
! ? ✓
25. @davidjacot
Kafka
Transparent L7 Proxy for Apache Kafka
25
Pod / VM
App Proxy Broker 1
Broker 2
Broker 3
KafkaClient
Interceptors
1
2
3
1 2 3
TCP connections going to Kafka
are redirected to the proxy
Each connection is proxied to its
real destination
Requests & Responses are
intercepted and possibly altered
Clear
mTLS
26. @davidjacot
What is intercepted by the proxy?
ApisRequest / ApisResponse
ProduceRequest / ProduceResponse
FetchRequest / FetchResponse
26
30. @davidjacot
Summary
31
Encryption
Right to be Forgotten
Consent
Kafka does not have an
real encryption story
Kafka only provides
Topics authorization
Kafka only knows how to
expire or compact events
E2E Encryption
Crypto-Shredding
Record based ACLs
! ? ✓