Ftrace is the official tracer of the Linux kernel. It originated from the real-time patch (now known as PREEMPT_RT), as developing an operating system for real-time use requires deep insight and transparency of the happenings of the kernel. Not only was tracing useful for debugging, but it was critical for finding areas in the kernel that was causing unbounded latency. It's no wonder why the ftrace infrastructure has a lot of tooling for seeking out latency. Ftrace was introduced into mainline Linux in 2008, and several talks have been done on how to utilize its tracing features. But a lot has happened in the past few years that makes the tooling for finding latency much simpler. Other talks at P99 will discuss the new ftrace tracers "osnoise" and "timerlat", but this talk will focus more on the new flexible and dynamic aspects of ftrace that facilitates finding latency issues which are more specific to your needs. Some of this work may still be in a proof of concept stage, but this talk will give you the advantage of knowing what tools will be available to you in the coming year.
3. What is ftrace?
●
Official tracer of the Linux Kernel
– Introduced in 2008
●
Came from the PREEMPT_RT patch set
– Initially focused on finding causes of latency
●
Has grows significantly since 2008
– Added new ways to find latency
– Does much more than find latency
5. Introducing trace-cmd
Luckily today, we do not need to know all those files
●
trace-cmd is a front end interface to ftrace
●
It interfaces with the tracefs directory for you
https://www.trace-cmd.org
git clone git://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git
6. The Old Tracers
●
Wake up trace
– All tasks
– RT tasks
– Deadline Tasks
●
Preemption off tracers
– irqsoff tracer
– preemptoff tracer
– preemptirqsoff tracer
7. Wake up tracer
Task 1
Interrupt
Wake up Task 2 event
Task 2
Sched switch event
latency
16. Measuring latency from interrupts
●
You can easily trace the latency from interrupts
– For x86:
# trace-cmd record -p function_graph -l handle_irq_event
-l ‘*sysvec_*’ -e irq_handler_entry
-e 'irq_vectors:*entry*'
18. Problems with the latency tracers
●
No control over what tasks they trace
– They trace the highest priority process
– May not be the process you are interested in
●
Not flexible
– Has one use case
– General for the entire system
19. Tracing interrupt latency with function graph tracer
●
Does not have a “max latency”
– You see all the latency in a trace
– No way to record the max latency found
21. Introducing Synthetic Events
●
Can map two events into a single event
– sched_waking + sched_switch wakeup_latency
→
– irqs_disabled + irqs_enabled irqs_off_latency
→
– irq_enter_handler + irq_exit_handler irq_latency
→
22. Introducing Synthetic Events
●
Can map two events into a single event
– sched_waking + sched_switch wakeup_latency
→
– irqs_disabled + irqs_enabled irqs_off_latency
→
– irq_enter_handler + irq_exit_handler irq_latency
→
●
Have all the functionality as a normal event
23. Introducing Synthetic Events
●
Can map two events into a single event
– sched_waking + sched_switch wakeup_latency
→
– irqs_disabled + irqs_enabled irqs_off_latency
→
– irq_enter_handler + irq_exit_handler irq_latency
→
●
Have all the functionality as a normal event
– Can be filtered on
24. Introducing Synthetic Events
●
Can map two events into a single event
– sched_waking + sched_switch wakeup_latency
→
– irqs_disabled + irqs_enabled irqs_off_latency
→
– irq_enter_handler + irq_exit_handler irq_latency
→
●
Have all the functionality as a normal event
– Can be filtered on
– Can have triggers attached (like histograms)
27. Introducing libtracefs
A library to interact with the tracefs file system
All functions have man pages
The tracefs_sql man page has the sqlhist program in it
https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/
git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
cd libtracefs
make sqlhist # builds from the man page!
28. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
29. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
30. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
31. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
32. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
33. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
34. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
35. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
36. Synthetic events
# sqlhist -e -n wakeup_lat
‘SELECT start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) AS delta FROM ’
‘sched_waking AS start JOIN sched_switch AS end ON start.pid = end.next_pid'
‘WHERE end.next_pid = “cyclictest”’
37. cyclictest
A tool used by real-time developers to test wake up latency in the system
https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/
git clone git://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
cd rt-tests
make cyclictest