Share some tips for tracing linux kernel source dynamically

1. Viller Hsiao <villerhsiao@gmail.com>
Dec. 14, 2014
Small Talk about Dynamically
Trace Code

2. Rights to Copy
copyright © 2014 Viller Hsiao
Attribution – ShareAlike 3.0
You are free
● to copy, distribute, display, and perform the work
● to make derivative works
● to make commercial use of the work
Under the following conditions
Corrections, suggestions, contributions
and translations are welcome!
– Attribution. You must give the original author credit.
– Share Alike. If you alter, transform, or build upon this work, you may distribute the
resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of this work.
● Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above.
License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode
12/14/14 2/38

3. Who am I ?
Viller Hsiao
source url: http://goo.gl/n7NAfS
Embeded Linux / RTOS engineer
2007 ~ Present Realtek Corp.
2003 ~ 2007 BenQ/Qisda
12/14/14 3/38

4. An Embedded Programmer
gcc
Hardware
Spec
mmaakkeeffiillee menuconfig
uclibc
musl
armcc
newlib
ld objdump
kbuild
memory
management Linker
script
glibc
uboot
source url: http://ext.pimg.tw/aabb52043/1367377582-3526768649_n.png
12/14/14 4/38

5. Topic Today
How to trace C sources environment
We have cross reference tools.
ctags, cscope, LXR, source insight ... and so on
BUT
12/14/14 5/38

6. Difficulties of Tracing C/C++ Sources
● Preprocessor related
● Too much #if/#elif/#endif
● Too much level of macros
● Configurable included files
12/14/14 6/38

7. Difficulties of Tracing C/C++ Sources (Cont.)
● Runtime related
● Some debugging and error handling stuff
● Multi-task environment
● Hard to realize HW state change
Different task context
12/14/14 7/38

8. Topic Today
How to trace C sources environment
Dynamically
Trace code from runtime execution result
(Mainly in linux kernel source)
12/14/14 8/38

9. Tracing from real execution log
Provides clues to trace code
http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.png
http://goo.gl/4PnwFe
12/14/14 9/38

10. Agenda Today
● Build environment
● Config system
● Makefile / kbuild
● Toolchain tips
● Runtime trace
● Qemu log
● Linux ftrace
12/14/14 10/38

11. menuconfig
● Too much words in menuconfig
● Hard to map definition to code world (CONFIG_XXX)
12/14/14 11/38

12. menuconfig Tips – Search
● Press '/' to search specific configurations
12/14/14 12/38

13. menuconfig Tips – Search (Cont.)
● Search result
Path in menu
Kconfig path
Dependency
12/14/14 13/38

14. menuconfig Tips – Result output
● Linux/.config
#
#
# # Automatically Automatically generated generated file; file; DO DO NOT NOT EDIT.
EDIT.
# # Linux/Linux/arm arm 3.14.25 3.14.25 Kernel Kernel Configuration
Configuration
#
#
CONFIG_CONFIG_ARM=ARM=y
y
CONFIG_CONFIG_SYS_SYS_SUPPORTS_SUPPORTS_APM_APM_EMULATION=EMULATION=y
y
#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
12/14/14 14/38

15. Tips of Tracing makefile
● make V=1
● make --dry-run
12/14/14 15/38

16. Tips of Tracing makefile (Cont.)
● make --debug
12/14/14 16/38

17. Tips of Tracing Shell Script
● #!/bin/sh “-x”
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then
echo "get foo"
echo "get foo"
else
else
echo "no foo"
echo "no foo"
fi
fi
● Result
+ FOO=something_produced_from_command
+ '[' xsomething_produced_from_command '!=' x ']'
+ echo 'get foo'
get foo
+ FOO=something_produced_from_command
+ '[' xsomething_produced_from_command '!=' x ']'
+ echo 'get foo'
get foo
12/14/14 17/38

18. Tips of Tracing Macros/Definitions
● gcc options: “-save-temps” or “-E”
$ gcc -save-temps -o main main.c
$ ls
main main.c main.i main.o main.s Makefile
$ gcc -save-temps -o main main.c
$ ls
main main.c main.i main.o main.s Makefile
● Use astyle to beauty result
$$ aassttyylele --AA88 mmaainin..ii
12/14/14 18/38

19. Tips of Tracing Macros/Definitions (Cont.)
12/14/14 19/38

20. Tips of Structure Definitions
● Point to Real definition in the headers
# 5 "include/linux/linkage.h" 2
# 1 "include/linux/stringify.h" 1
# 6 "include/linux/linkage.h" 2
# 1 "include/linux/export.h" 1
# 26 "include/linux/export.h"
struct kernel_symbol
{
unsigned long value;
const char *name;
};
# 5 "include/linux/linkage.h" 2
# 1 "include/linux/stringify.h" 1
# 6 "include/linux/linkage.h" 2
# 1 "include/linux/export.h" 1
# 26 "include/linux/export.h"
struct kernel_symbol
{
unsigned long value;
const char *name;
};
12/14/14 20/38

21. Kbuild Tips
● Add compile flags for some object
CFLAGS_CFLAGS_ffoooo..oo == --OO00 --gg --ssaavvee--tteemmpp
● Remove compile flags for some object
CCFFLLAAGGSS__RREEMMOOVVEE__ffoooo..oo == --OOss
● All subdirectory
ssuubbddirir--ccccfflalaggss--yy == --ssaavvee--tteemmppss
12/14/14 21/38

22. Qemu Tips
● Qemu log (“-d op -D /path/to/qemu.log”)
Setup the search rule
12/14/14 22/38

23. Qemu Tips (Cont.)
● Qemu log + regular expression
Setup the search rule
Example: get first user space entry point
12/14/14 23/38

24. Qemu Tips (Cont.)
● Other tips
● Qemu log + disassembly (objdump)
● Qemu + gdb ( + qemu log)
– Breakpoint / stack trace
– Qemu options
● -s or -gdb tcp::2345: enable gdb port
● -S: freeze CPU at startup
12/14/14 24/38

25. Ftrace – Overview
● Kernel function tracer ( > 2.6.28 ?)
● Record execution log of kernel function
● Implemented by “-pg”
– Use dynamic ftrace to reduce overhead
– Add “-g -O0” to export more information
● Accessing API – debugfs
– /path/to/debug/tracing/
mount -t debugfs n mount -t debugfs noonnee //ssyyss//kkeerrnneel/l/ddeebbuugg
12/14/14 25/38

26. Ftrace – Configs
12/14/14 26/38

27. Ftrace – Steps
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
12/14/14 27/38

28. Ftrace – Available Tracer
[tracing] $ cat available_tracers
function
function_graph
nop
[tracing] $ cat available_tracers
function
function_graph
nop
12/14/14 28/38

29. ftrace
● function tracer
12/14/14 29/38

30. ftrace
● function tracer
12/14/14 30/38

31. ftrace
● function_graph tracer
12/14/14 31/38

32. ftrace
● function_graph tracer
Task
switch
12/14/14 32/38

33. ftrace
● Better front-end
● trace-cmd
● kernelshark
12/14/14 33/38

34. Ftrace for Code Tracing
● Ftrace provides clues of
● Per-module view
● Top-down view
● Procedure excution flow
● Tasks/Threads model relationship
12/14/14 34/38

35. Ftrace – Other tracer
● Trace event
● ftrace + tracepoint
● Probe event
● ftrace + kprobe
● Other tracer
● Stack tracer
● Irqoff latency tracer
● Syscall tracer
● And so forth … but not the topic today
12/14/14 35/38

36. Conclusion
● Trace code
● Cross reference tool helps
● Drill-down trace
– Start from interface functions
● Filesystem hooks: open/read/write/ioctl ...
● probe/exit
● ISR entry
● Architecture interface
– Global functions: EXPORT_SYMBOLS
● Execution log provide some hints, too.
– Log from build flow
– Qemu instruction trace log
– Kernel ftrace
12/14/14 36/38

37. Reference
[1] Building embedded linux demo case
i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux”
ii. Linaro wiki, “cross-NG”
[2] Linux-3.14 documentation,
i. “kbuild/makefile.txt”
ii. “trace/ftrace.txt”
[3] 池田宗廣，大岩尚宏，島本裕志，竹步晶雄，平松雅巳(Jan. 2014), “Linux kernel hacks”
[4] man of
i. astyle
ii. qemu
iii. Gnu make
12/14/14 37/38

38. THE END
12/14/14 Viller Hsiao