Slides from Linux Kernel Debugging Essentials workshop at base48 Hackerspace. Not really useful by themselves.

1. Linux Kernel Debugging Essentials
Lubomir Rintel
base48 Hackerspace, Brno
http://base48.cz/
February 19th , 2014
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
1 / 12

2. Plan
Understanding the way kernel can fail
Introduction to infrastructure
Obtaining & unserstanding the traces
Heplful in-kernel mechanisms
Building and trying the replacement code
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
2 / 12

3. Kernel failures
Warning, possibly non-fatal: OOPS
A fatal error: Panic
Usually accompained by a state dump
Result of a bug or a hardware failure
# echo c >/proc/sysrq-trigger
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
3 / 12

4. Kernel configuration
Vanilla kernels
Distribution kernels
Configuration options (build-time)
Runtime configuration (sysctls)
Boot time configuration (kernel parameters)
Documentation/kernel-parameters.txt
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
4 / 12

5. Lab setup
For details & non-Fedora setup, see the hackerspace wiki
# yum -y install kernel-debug{,-devel}
# debuginfo -y install kernel-debug
# yum -y install kexec-tools crash
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
5 / 12

6. Console output
Message buffer
Screen, UART, USB debugging dongle
Early console for boot problems
Slowed down output possible
Syslog via Netpoll API
# dmesg
rhgb, quiet
console={ttyS*,uart}
earlyprintk={vga,serial,dbgp}
kernel.printk_delay
kernel.printk_ratelimit
netconsole.ko
g_dbgp.ko
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
6 / 12

7. Reading OOPS messages
Processor status
Traceback
# objdump -lSd {vmlinux,module.ko}
# addr2line
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
7 / 12

8. Memory dumps
Crash dumps
OHCI1394 DMA, can be early too
Hypervisor specific dumps
We can extract console output from them!
crashkernel=auto
# systemctl start kdump.service
# firescope
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
8 / 12

9. Analyzing dumps
crash
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
9 / 12

10. Helpful kernel tools
Kernel hacking menu
Lockdep detector
Hung task detector
List corruption detector
Stack protector
Memory poison
CONFIG_DEBUG_*
kernel.sysrq
kernel.*panic*
kernel.prove_locking
kernel.hung_task_timeout_secs
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
10 / 12

11. Loading our code into kernel
Understand kernel coding
Pick a good book
Linux Kernel Development, Robert Love, Addison-Wesley:
Developer’s Library
Understanding the Linux Kernel, Corbet et al., O’Reilly (available
online)
Try out what do the debugging facilities do!
# make -C /lib/modules/$(uname -r)/build
SUBDIRS=$PWD modules
# insmod module.ko
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
11 / 12

12. Thank you for attending!
The source to the slides are embedded in the PDF file. Use pdftk or
Acrobat Reader’s ”attachments” to extract them.
lkundrak@v3.sk
Kernel Debugging
February 19th , 2014
12 / 12