2. Who Am I?
• Co-inventor of DTrace
• Founder ZFS Storage Appliance team
• CTO at Delphix
• EIR at Sutter Hill Ventures
• Recent founder
3. What’s a System Call?
• User call into the kernel to do some work
• Typically implemented with a trap
• Mostly standardized in terms of function
• Fundamental to multi-user systems
9. Syscall Tracing For All
• Truss 1988
– “If your program doesn't work, put it in a truss.”
– Roger Faulker
– Used then-new structured /proc
– SunOS / UNIX
• Strace 1991
– “an alternative syscall tracer”
– Paul Kranenburg on comp.sources.sun
– SunOS then, famously, Linux in 1992
– Non-Linux code removed in 2012
10. Breakpoint Tracing
process
invoke syscall
execute syscall
proceed
scheduler
stop process
wake tracer
stop tracer
wake process
stop process
wake tracer
stop tracer
wake process
tracer
record arguments
resume process
record return value
signal process
11. Slow
# Slow the target command and print details for each syscall:
strace command
# Slow the target PID and print details for each syscall:
strace -p PID
# Slow the target PID and any newly created child process, pri
strace -fp PID
# Slow the target PID and record syscalls, printing a summary:
strace -cp PID
# Slow the target PID and trace open() syscalls only:
strace -eopen -p PID
Brendan Gregg
13. DTrace
• Dynamic tracing framework
– Solaris, macOS, FreeBSD, some Linux distros
• Safe for production by design
• Concise answers to arbitrary questions
• Systemic in scope
• Syscalls are a common place to start
19. Primacy of Syscalls
• Highly significant events
• 30 years later, still the right granularity
• Well-understood domain for tracers
• Ubiquitous
• De facto standard in Linux