We have introduced Cilium at DockerCon US 2017 this year. Cilium provides application-aware network connectivity, security, and load-balancing for containers. This talk will follow up on the introduction and deep dive into recent kernel developments that address two fundamental questions: How can I provide application-aware security and routing efficiently without overhead embedded into every service? How can container hosts protect themselves from internal and external DDoS attacks? The solutions include: kproxy: a kernel-based socket proxy which allows for application-aware routing and security enforcement with minimal overhead. XDP: A lightning-fast packet processing datapath using BPF. The technology is intended for DDoS mitigation, load-balancing, and forwarding. This talk will deep dive into these exciting technologies and show how Cilium makes BPF and these kernel features available on Linux for your Docker containers.

1. Cilium – Kernel Native Security & DDOS
Mitigation for Microservices with BPF
Cynthia Thomas
Technology Evangelist, Covalent
@_techcet_

2. Gordon is back!
And he got a job at Lego.

3. Robot Competition
● Local marketing launches an
Robot competition for Danish
residents
● Upcoming deadline means tons
of uploaded media

4. Robot Competition
User Data
Mongo
DB
Data
store
Inventory
Image
Upload
Service
Web
Front-
End
Kafka
Broker
Kafka
Broker
Kafka
Broker
Image
processing
Image
processing
Image
processing
ZookeeperZookeeperZookeeper
End-
User
Admin
task
Audit

5. Gordon’s Task List
● Deploy Cilium HTTP-aware security
for microservices
● Isolate Kafka resources for old & new services
● DDOS mitigation via XDP/BPF

6. Gordon’s 1st Task
Update HTTP Security for Microservices

7. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server App
Yearly
Low
Evolution of Application Design & Delivery Frequency

8. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server App
Yearly
Low
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency

9. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server App
Yearly
Low
Distributed
Microservices
10-100 x’s / day
Extreme
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency

10. The world still runs on iptables
matching IPs and ports:
$ iptables -A INPUT -p tcp -s 15.15.15.3 --dport 80
-m conntrack --ctstate NEW -j ACCEPT

11. Robot Competition
User Data
Mongo
DB
Data
store
Inventory
Image
Upload
Service
Web
Front-
End
Kafka
Broker
Image
processing
Zookeeper
End-
User
Admin
task
Audit
Example: HTTP calls

12. API
Web
Front-End
Security for Microservices
Image
Upload
Service
GET /image
POST /image
GET /image/flagged
PUT /image/id

13. API
POST /image
Web
Front-End
Security for Microservices
Image
Upload
Service
GET /image
GET /image/flagged
PUT /image/id
POST /image

14. L3/L4
API
POST /image
Web
Front-End
Security for Microservices
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT
Image
Upload
Service
GET /image
POST /image
GET /image/flagged
PUT /image/id

15. L3/L4
API
POST /image
Web
Front-End
Security for Microservices
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT
exposed
exposed
exposed
Image
Upload
Service
GET /image
POST /image
GET /image/flagged
PUT /image/id

16. Network Security has not
evolved for microservices…
…until now.

17. API
POST /image
Web
Front-End
Cilium API-Aware Security
Image
Upload
Service
GET /image
GET /image/flagged
PUT /image/id
POST /image

18. L7
API
POST /image
Web
Front-End
Cilium API-Aware Security
Image
Upload
Service
GET /image
POST /image
GET /image/flagged
PUT /image/id
FROM Web Front-End
ALLOW POST /image

20. Under the Hood: BPF
Linux
Superpowers
Unleashed

21. BPF instruction set
struct bpf_insn prog[] = {
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
BPF_LD_ABS(BPF_B, ETH_HLEN + offsetof(struct iphdr, protocol) /* R0 =
ip->proto */),
BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4), /* *(u32 *)(fp - 4) = r0
*/ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), /* r2 = fp - 4 */
BPF_LD_MAP_FD(BPF_REG_1, map_fd),
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
BPF_MOV64_IMM(BPF_REG_1, 1), /* r1 = 1 */
BPF_RAW_INSN(BPF_STX | BPF_XADD | BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
/* xadd r0 += r1 */
BPF_MOV64_IMM(BPF_REG_0, 0), /* r0 = 0 */
BPF_EXIT_INSN(),
};
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/ast/bpf.git/tree/samples/bpf/sock_example.c

22. Cilium
Cilium
Agent

23. User
space
Kernel
space
Cilium Architecture
Cilium
Agent

24. API Calls
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

25. BPF
API Calls
BPF Code & Maps
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

26. BPF
API Calls
BPF Code & Maps
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent
Extremely scalable,
highly customized
network filtering, load-
balancing, and
monitoring

27. BPF BPF
API Calls
BPF Code & Maps
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

28. BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

29. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

30. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
Plugins
Cilium Architecture
Cilium
Agent

31. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
CLI,
Monitoring,
Policies
Plugins
Cilium Architecture
Cilium
Agent

32. Gordon’s 2nd Task
Isolate Kafka Topics

33. Kafka?
Used for building real-time
pipelines and streaming apps.
- Horizontally scalable
- Fault-tolerant
- “Wicked fast”
Defined by its own protocol.
1/3 of all Fortune 500
companies use Kafka[1]

34. Kafka Concepts
Topic 1 Producers
Kafka Broker
Topic 1
Topic N
Topic 1 Consumer Group A
Topic 1 Consumer Group B

35. Security for
Microservices
Gordon worries
about open Kafka
topics
The API is too open!
How do I bring on
new services?

36. Robot Competition
User Data
Mongo
DB
Data
store
Inventory
Image
Upload
Service
Web
Front-
End
Kafka
Broker
Kafka
Broker
Kafka
Broker
Image
processing
Image
processing
Image
processing
ZookeeperZookeeperZookeeper
End-
User
Admin
task
Audit
Example: Kafka calls

37. Kafka.
API
Image
processing
Security for Microservices
Kafka
Broker
Fetch inventory
Fetch image
Fetch userid
Fetch analytics
Produce inventory
Produce image
Produce userid
Produce analytics

38. Kafka.
API
Fetch image
Image
processing
Security for Microservices
Kafka
Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid

39. L3/L4
Kafka.
API
Fetch image
Image
processing
Security for Microservices
iptables -s 10.1.1.1
-p tcp --dport 9092
-j ACCEPT
Kafka
Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid

40. L3/L4
Kafka.
API
Fetch image
Image
processing
Security for Microservices
iptables -s 10.1.1.1
-p tcp --dport 9092
-j ACCEPT
exposed
exposed
exposed
Kafka
Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid

41. Kafka.
API
Fetch image
Image
processing
Security for Microservices
Kafka
Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid

42. L7
Kafka.
API
Fetch image
Image
processing
Security for Microservices
Kafka
Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
FROM Image processing
ALLOW Fetch image

43. Demo!
Get real with Gordon

44. Demo: Kafka API Filtering
app1-
producer
app1-
consumer
app2-
producer
app2-
consumer
kafka

45. What just
happened?

46. L7
Kafka.
API
Produce imageprocessing
app1-
producer
Demo: Kafka API Filtering
Kafka
Broker
Produce imageprocessing
Produce userdata
FROM app1-producer
ALLOW Produce imageprocessing

47. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
CLI,
Monitoring,
Policies
Plugins
Sidecar Proxy
Cilium
Agent
Sidecar
Proxy

48. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
CLI,
Monitoring,
Policies
Plugins
In-Kernel Kafka Parsing
Cilium
Agent
kernel proxy
BPF

49. Gordon’s 3rd Task
DDOS Mitigation

50. Robot Competition
● Local marketing launches
competition for Danish residents
only
● DDOS anticipated from other
countries and anti-Lego activists

51. Robot Competition
User Data
Mongo
DB
Data
store
Inventory
Image
Upload
Service
Web
Front-
End
Kafka
Broker
Image
processing
Zookeeper
End-
User
Admin
task
Audit
Example: DDOS attack!

53. BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
User
space
Kernel
space
CLI,
Monitoring,
Policies
Plugins
XDP/BPF handled in NIC driver
Cilium
Agent
XDP

54. Source: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
Facebook published BPF/XDP numbers
for L3/L4 LB at Netdev 2.1
BPF/XDP throughput
IPVS throughput

55. BPF with XDP Setup
pktgen attack:
~11.6 Mpps
randomly in 10.0.0.0/8
legit traffic:
netperf tests on
10.192.1.0/24
Blacklist
16M rules
All /32s in
10.0.0.0/8

56. BPF with XDP for DDoS mitigation
Metric iptables / ipset XDP
DDoS rate [packets/s] 11.6M 11.6M
Drop rate [packets/s] 7.1M 11.6M
Time to load rules [time] 3 min 20 sec 31 sec
Latency under load [ms] 2.3ms 0.1ms
Throughput under DDoS [Gbit/s] 0.014 6.5
Requests/s under DDoS [kReq/s] 0.28 82.8
Sender: Send 64B packets as fast as possible è Receiver: Drop as fast as possible
Source: Daniel Borkmann’s presentation:
http://schd.ws/hosted_files/ossna2017/da/BPFandXDP.pdf

57. ● Cilium deployment for microservices
successfully secured HTTP traffic
● Kafka resources were isolated to
protect existing services from new ones
● Mitigated DDOS attacks via XDP/BPF
Gordon’s Summary

58. Cilium Project Status
• Cilium v0.12 release in October
• Docker, Kubernetes, and Mesos integration
• Looking for feedback and contributions

59. Take Action!
• Getting Started Using Docker: docs.cilium.io/
• Join our Slack community!
• Check out the project website for more details:
https://www.cilium.io/
Please ★
us on
GitHub

60. Thank You!
github.com/cilium/cilium
cilium.io
@ciliumproject