This document discusses Linux tracing tools and the evolution from DTrace on BSD to eBPF on Linux. It begins with an overview of DTrace and its capabilities on BSD, then discusses the limitations of early Linux tracing tools. It introduces eBPF and the BCC compiler collection, which make it easier to write and use eBPF programs. Examples are given showing how BCC can be used to trace system calls, file opens, and command executions. The document argues that BCC and eBPF help address the problems of early Linux tracing by making the tools more approachable and powerful for production use.

2. Linux
tracing
superpowers
Ievgen Pyrogov @gmile Lviv 2017

3. Plan
1. BSD and
the DTrace era
2. Linux and
the BPF era

4. What is full-stack
development?

6. BSD and
the DTrace era

7. A little of history

9. I realized at a relatively
young age (~19)
that I love debugging.
Brian Cantril

11. The greatest satisfaction that I have had
is nailing a nasty bug —
it's an experience that (for me, anyway)
is so visceral as to be nearly primal.
Brian Cantril

14. 2003

16. Adam Leventhal
Mike Shapiro
Bryan Cantrill

17. DTrace

19. # dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
dtrace: description 'syscall::open*:entry ' matched 2 probes
CPU ID FUNCTION:NAME
0 14 open:entry gnome-netstatus- /dev/kstat
0 14 open:entry man /var/ld/ld.config
0 14 open:entry man /lib/libc.so.1
0 14 open:entry man /usr/share/man/man.cf
0 14 open:entry man /usr/share/man/windex
0 14 open:entry man /usr/share/man/man1/ls.1
0 14 open:entry man /usr/share/man/man1/ls.1
0 14 open:entry man /tmp/mpqea4RF
0 14 open:entry sh /var/ld/ld.config
0 14 open:entry sh /lib/libc.so.1
0 14 open:entry neqn /var/ld/ld.config
0 14 open:entry neqn /lib/libc.so.1
0 14 open:entry neqn /usr/share/lib/pub/eqnchar
0 14 open:entry tbl /var/ld/ld.config
0 14 open:entry tbl /lib/libc.so.1
0 14 open:entry tbl /usr/share/man/man1/ls.1
0 14 open:entry nroff /var/ld/ld.config
[...]

20. # dtrace -n 'syscall:::entry { @num[execname] = count(); }'
dtrace: description 'syscall:::entry ' matched 228 probes
^C
snmpd 1
utmpd 2
inetd 2
nscd 7
svc.startd 11
sendmail 31
poold 133
dtrace 1720

21. # dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'
dtrace: description 'syscall:::entry ' matched 228 probes
^C
fstat 1
setcontext 1
lwp_park 1
schedctl 1
mmap 1
sigaction 2
pset 2
lwp_sigmask 2
gtime 3
sysconfig 3
write 4
brk 6
pollsys 7
p_online 558
ioctl 579

22. $ dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'
dtrace: description 'syscall:::entry ' matched 228 probes
^C
1109 svc.startd 1
4588 svc.startd 2
7 svc.startd 2
3950 svc.startd 2
1626 nscd 2
870 svc.startd 2
82 nscd 6
5011 sendmail 10
6010 poold 74
8707 dtrace 1720

23. $ dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'
dtrace: description 'sysinfo:::readch ' matched 4 probes
^C
mozilla-bin 16
gnome-smproxy 64
metacity 64
dsdm 64
wnck-applet 64
xscreensaver 96
gnome-terminal 900
ttymon 5952
Xorg 17544

24. $ dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'
dtrace: description 'proc:::exec-success ' matched 1 probe
CPU ID FUNCTION:NAME
0 3297 exec_common:exec-success man ls
0 3297 exec_common:exec-success tbl /usr/share/man/man1/ls.1
0 3297 exec_common:exec-success neqn /usr/share/lib/pub/eqnchar -
0 3297 exec_common:exec-success nroff -u0 -Tlp -man -
0 3297 exec_common:exec-success col -x
0 3297 exec_common:exec-success sh -c more -s /tmp/mpzIaOZF
0 3297 exec_common:exec-success more -s /tmp/mpzIaOZF

28. dtrace -l
dtrace: system integrity protection is on, some features will not be available
dtrace: failed to initialize dtrace: DTrace requires additional privileges

44. Brendan Gregg

46. One the first guys to
use DTrace
outside of SUN

62. January 27, 2010

65. Resignations
Petitions and forks
Lawsuit
Resignations
Forks
Forks
Program closures

76. Linux and
the BPF era

77. Linux
tracing
tools

78. SystemTap
trace
strace

79. sysdig
ktap
LTTng

80. Limitations:

81. 1. Expensive to run

82. 2. Extremely hard to
use in production

83. 3. May be
unsafe to run

84. Result: no single
consolidated effort in
development

85. BPF

86. Berkeley
Packet
Filter

88. BPF – is a tiny VM
inside the kernel

89. Initial purpose – to
filter TCP packets!

92. How BPF looked like back then:
l0: ldh [12]
l1: jeq #0x800, l3, l2
l2: jeq #0x805, l3, l8
l3: ld [26]
l4: jeq #SRC, l4, l8
l5: ld len
l6: jlt 0x400, l7, l8
l7: ret #0xffff
l8: ret #0

93. 2013

95. Things start
to change
dramatically

97. Alexei Starovoitov

102. BPF is brutally
hard to write!

103. Far-far from
DTrace one-liners

104. Brendan Gregg:
“BPF is the only
language that
defeated me!”

105. We should not
write BPF programs
directly

106. Enter BCC!

107. BCC

109. BCC =
BPF Compiler
Collection

111. 1. Write BPF program
in restricted C

112. 2. Let Python or Lua
leverage the rest

113. 1. Write your BPF program in C... inline or in a separate file
2. Write a python script that loads and interacts with your
BPF program
3. Attach to kprobes, socket, etc.
4. Read/update maps
5. Configuration, complex calculation/correlations
6. Iterate on above and re-try...in seconds

114. from bpf import BPF
from subprocess import call
prog = """
int hello(void *ctx) {
bpf_trace_printk("Hello, World!n");
return 0;
};
“""
b = BPF(text=prog)
fn = b.load_func("hello", BPF.KPROBE)
BPF.attach_kprobe(fn, “sys_clone")
try:
call(["cat", "/sys/kernel/debug/tracing/trace_pipe"])
except KeyboardInterrupt:
pass

115. [root@localhost examples]# ./hello_world.py
python-20662 [001] d..1 1138.551706: : Hello, World!
tmux-1012 [002] d..1 1139.227627: : Hello, World!
tmux-1012 [002] d..1 1139.229636: : Hello, World!
byobu-20664 [006] d..1 1139.235396: : Hello, World!
byobu-20665 [007] d..1 1139.236660: : Hello, World!
byobu-20665 [007] d..1 1139.246109: : Hello, World!
^C

117. Brenden Blanco

121. Largest difference
between
DTrace and BPF is:
ease of use

122. Simpler tools
utilizing BPF are
already being built!

123. 1. BPF backend
for perf command

124. 2. ply

126. #!/usr/bin/env ply
kprobe:SyS_*
{
$syscalls[func].count()
}

127. 3. BPF backend
for SystemTap

129. #!/usr/bin/stap
/*
* opensnoop.stp Trace file open()s. Basic version of opensnoop.
*/
probe begin
{
printf("n%6s %6s %16s %sn", "UID", "PID", "COMM", "PATH");
}
probe syscall.open
{
printf("%6d %6d %16s %sn", uid(), pid(), execname(), filename);
}

132. Find / fix bcc issues

134. High-level language

135. Promotion

136. Promotion

137. Education

138. GUI integration

143. Examples

144. # ./execsnoop
PCOMM PID RET ARGS
bash 15887 0 /usr/bin/man ls
preconv 15894 0 /usr/bin/preconv -e UTF-8
man 15896 0 /usr/bin/tbl
man 15897 0 /usr/bin/nroff -mandoc -rLL=169n -rLT=169n -Tutf8
man 15898 0 /usr/bin/pager -s
nroff 15900 0 /usr/bin/locale charmap
nroff 15901 0 /usr/bin/groff -mtty-char -Tutf8 -mandoc -rLL=169n -rLT=169n
groff 15902 0 /usr/bin/troff -mtty-char -mandoc -rLL=169n -rLT=169n -Tutf8
groff 15903 0 /usr/bin/grotty

145. # ./opensnoop
PID COMM FD ERR PATH
17326 <...> 7 0 /sys/kernel/debug/tracing/trace_pipe
1576 snmpd 9 0 /proc/net/dev
1576 snmpd 11 0 /proc/net/if_inet6
1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/conf/lo/forwarding
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms
1576 snmpd 9 0 /proc/diskstats
1576 snmpd 9 0 /proc/stat
1576 snmpd 9 0 /proc/vmstat
1956 supervise 9 0 supervise/status.new
1956 supervise 9 0 supervise/status.new
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libtinfo.so.5
17358 run 3 0 /lib/x86_64-linux-gnu/libdl.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libc.so.6
17358 run -1 6 /dev/tty
17358 run 3 0 /proc/meminfo
17358 run 3 0 /etc/nsswitch.conf
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_compat.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libnsl.so.1
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_nis.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_files.so.2
17358 run 3 0 /etc/passwd
17358 run 3 0 ./run

146. # ./bashreadline
TIME PID COMMAND
05:28:25 21176 ls -l
05:28:28 21176 date
05:28:35 21176 echo hello world
05:28:43 21176 foo this command failed
05:28:45 21176 df -h
05:29:04 3059 echo another shell
05:29:13 21176 echo first shell again

147. # ./funccount 'vfs_*'
Tracing... Ctrl-C to end.
^C
FUNC COUNT
vfs_create 1
vfs_rename 1
vfs_fsync_range 2
vfs_lock_file 30
vfs_fstatat 152
vfs_fstat 154
vfs_write 166
vfs_getattr_nosec 262
vfs_getattr 262
vfs_open 264
vfs_read 470
Detaching...

148. # trace 'sys_execve "%s", arg1'
PID COMM FUNC -
4402 bash sys_execve /usr/bin/man
4411 man sys_execve /usr/local/bin/less
4411 man sys_execve /usr/bin/less
4410 man sys_execve /usr/local/bin/nroff
4410 man sys_execve /usr/bin/nroff
4409 man sys_execve /usr/local/bin/tbl
4409 man sys_execve /usr/bin/tbl
4408 man sys_execve /usr/local/bin/preconv
4408 man sys_execve /usr/bin/preconv
4415 nroff sys_execve /usr/bin/locale
4416 nroff sys_execve /usr/bin/groff
4418 groff sys_execve /usr/bin/grotty
4417 groff sys_execve /usr/bin/troff

149. # trace 'sys_read (arg3 > 20000) "read %d bytes", arg3'
PID COMM FUNC -
4490 dd sys_read read 1048576 bytes
4490 dd sys_read read 1048576 bytes
4490 dd sys_read read 1048576 bytes
4490 dd sys_read read 1048576 bytes

150. trace 'r:bash:readline "%s", retval'
PID COMM FUNC -
2740 bash readline echo hi!
2740 bash readline man ls

152. Bonus
track

159. Thanks!

160. Questions?