@ 2020/04/20 SDN x Cloud Native Meetup #27 隨著CNCF將Falco納入incubator project，eBPF這藏於Linux核心內的技術也開始受到矚目。 eBPF，一個從1992年就出現的技術，一路走來經過了甚麼樣的變化? 對於你我目前，或未來的工作又會有甚麼影響呢? 本次分享將會介紹eBPF的前世今生，帶各位了解何謂eBPF。並透過實際範例演示eBPF工具的特殊用法。

1. Introduction of eBPF
時下最夯的Linux Technology

2. 梁維恩
Jace Liang
SW / Infra. engineer at ITRI
Facebook: jace.liang
github: mJace

3. TOC Votes to Move Falco into CNCF Incubator
By Jessie January 8, 2020 in Blog
Today, the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC) voted to accept
Falco as an incubation-level hosted project.
Falco, which entered the CNCF Sandbox in October 2018, is an open source Kubernetes runtime security
project. It provides intrusion and abnormality detection for cloud native platforms such as Kubernetes,
Mesosphere, and Cloud Foundry.

4. BPF security capabilities
• Which processes are being
executed? By which processes?
• What network connections are
being made? By which processes?
• What permission denied errors
are happening on the system?
• Is this kernel/user function being
executed with these arguments?

5. Take away
• What’s eBPF
• Use eBPF based tools to debug
• New design idea
You don't need to know how to operate an X-ray machine,
but you do need to know that if you swallow a penny, an X-ray is an option!
www.bredangregg.com

6. What’s BPF?
• BPF全名為Berkeley Packet Filter, Introduced by Lawrence Berkeley
National Laboratory, 1992.
• 當時推出的目的是為了提高 BSD-based Kernel過濾封包的效率。
原理是將封包的過濾程式編譯後由Kernel中類似虛擬機的環境執
行。
• 和原先在Userspace過濾封包相比有更好的效能。
且透過編譯以及在核心內沙盒中執行的特性，能夠避免使用者把
Kernel搞壞掉。

7. Example of BPF – Tcpdump

8. Example of BPF – Tcpdump cont.
#檢查是否為IPV6，如果不是(jf)，則視為IPV4 (GOTO Line:006)
#檢查是否為TCP
#檢查dst port是否為7070(0x1b9e)，if so (jt) L014
#檢查是否為 ipv4封包
#檢查是否為 tcp封包
#檢查是否為 ip fragment packet
#找到tcp封包中 dest port 的所在位置
#檢查dst port是否為7070，若為真(jt) GOT L014
#Packet Match!
#Packet Mis-match!

9. How about eBPF (enhanced BPF)?
• 原先Kernel內bpf虛擬機的設計過時，不支援新硬體CPU架構
• eBPF相對bpf有更佳的硬體相容性，支援更大的register
• eBPF相對bpf有更快的編譯速度，在過濾網路封包時的效能也更好
• eBPF於2014年的版本後，便可直接從userspace操作
“Super powers have finally come to Linux“ – Brendan Gregg, Linux Conf. 2017

10. eBPF Architecture.

11. What can you do with eBPF?
• Filter traffic, at the lowest entry of linux network stack.
• Programs can be attached to tracepoints, kprobes, system calls, perf events,
etc.

12. Velocity 2017: Performance Analysis Superpowers with Linux eBPF - Brendan Gregg
https://www.youtube.com/watch?v=bj3qdEDbCD4

13. Use case of eBPF – Userspace tracing
https://github.com/iovisor/kubectl-trace

14. relationship between userspace threads
fnc
tid/pid/arg/ret
fnc
tid/pid/arg/ret
pkt
pkt
pkt
pkt
enqueue
tid/pid/arg/ret
dequeue
tid/pid/arg/ret
Get relationship by en/dequeue args and retval
https://github.com/mJace/ebpfKit/blob/master/Examples/cpp/README.md

16. eBPF related projects – XDP (express data path)
• Since Kernel v4.8
• Based on eBPF
• DDOS Protection
• Network security
• Network accelerate

17. eBPF related projects – sysdig
• Embed Security, Compliance and Performance Into Your DevOps Workflows

18. eBPF related projects – Falco
• Cloud-Native Runtime Security
Falco efficiently leverages Extended Berkeley Packet Filter (eBPF), a secure
mechanism, to capture system calls and gain deep visibility. By adding
Kubernetes application context and Kubernetes API audit events, teams can
understand who did what.

19. Other eBPF related implementations…
• Cilium – XDP based CNI
• Weavescope – ebpf based monitor tool
• Iptables – Bpfilter implementations to optimize ingress/outgress security
rules
• Calicio – Just release a alpha version that lavages ebpf
• Systemtap – Support eBPF now.

20. eBPF related projects – BCC
• BPF Compiler Collection (BCC)
BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful
tools and examples
https://github.com/iovisor/bcc

21. BCC tools example – tcpconlat (tcp latency)

22. BCC tools example – execsnoop ( trace syscall- exec)

23. bpftrace tool example – cpuwalk.

24. Demo 1 – containerized ebpf tool
• Bcc tools inside a container, and trace other container’s processes.
Target
container
ebpf
container
Host Machine
Kernel
ebpf
program
ebpf
map
https://github.com/mJace/ebpfKit/blob/master/Examples/bcc-demo/demo-01.md

25. Demo 2.
• Namespace-based tracing.
ebpf
container
Target Container
P3
P2
P1 How to trace all processes?????
Even process just created?
https://github.com/mJace/ebpfKit/blob/master/Examples/bcc-demo/demo-02.md

26. Software stack for ebpf related project
bpf,ebpf – main framework
XDP – Express data plane powered by ebpf
Bcc lib – library for higher app to communicate with bpf
go-bpf – golang lib for bpf
Bcc tools – userspace tool like tcptracer to trace all tcp status
bpftrace – high level userspace bpf based trace tool.
bpfebpf
Bcc lib
Kernel Space
User Space
Bcc tools
go-bpf
bpftrace tools
XDP

27. The future of eBPF
Kernel operations structures in BPF
what has been merged for 5.6 is not just a mechanism for hooking in TCP congestion-control algorithms……
this new infrastructure can be used to allow a BPF program to replace any "operations structure“ (in kernel)
https://lwn.net/Articles/811631/?fbclid=IwAR3otEAmjW4GS5i3hcWHzsy6hfmTIJwb_nUGHcT-
sS2aCOX1xcn9DuTfcwA
➢Update kernel without building kernel, even rebooting
➢Dynamic driver? Runtime configurable kernel driver, without re-bulding
➢Kernel layer cloud native application?

28. Q n’ A / Take away
• What’s eBPF
• 一種Linux內的技術，能讓人動態的觀察系統內的行為
• Use eBPF based tools to debug
• ebpf tool產生的overhead，遠低於傳統userspace monitor tool
• 可觀測幾乎所有系統內行為，從kernel到userspace
• New design idea
• eBPF打破以往kernel layer application可攜性極低的問題
You don't need to know how to operate an X-ray machine,
but you do need to know that if you swallow a penny, an X-ray is an option!
www.bredangregg.com

29. Reference.
• http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
• https://hackmd.io/@sysprog/SJTuuG9a7?type=view
• https://github.com/iovisor/bpftrace
• https://github.com/iovisor/bcc
• https://github.com/iovisor/kubectl-trace