Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Kernel debug log and console on openSUSE

128 Aufrufe

Veröffentlicht am

Describe the log buffer and consoles in Liunx kernel on openSUSE

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

Kernel debug log and console on openSUSE

  1. 1. Kernel debug log and console on openSUSE August, 2019, Taipei, COSCUP Joey Lee SUSE Labs Taipei
  2. 2. 2 Agenda • Kernel log • Consoles • Q&A
  3. 3. 3 Kernel log buffer and consoles
  4. 4. Kernel log
  5. 5. 5 Kernel log
  6. 6. 6 Kernel log buffer • SLE15 / openSUSE Leap 15.1 ‒ CONFIG_LOG_BUF_SHIFT=18 ‒ CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13 ‒ 2^18 = 256KB • log_buf_len=n[KMG] ‒ Sets the size of the printk ring buffer, ‒ in bytes. n must be a power of two and greater than the minimal size. The minimal size is defined by LOG_BUF_SHIFT kernel config parameter. There is also CONFIG_LOG_CPU_MAX_BUF_SHIFT config parameter that allows to increase the default size depending on the number of CPUs. See init/Kconfig for more details. [1]
  7. 7. 7 Kernel log buffer (cont.) • The kmsg is an ring buffer static char __log_buf[__LOG_BUF_LEN] __aligned(LOG_ALIGN); • Protected by spinlock DEFINE_RAW_SPINLOCK(logbuf_lock); • /dev/kmsg [11] = { "kmsg", 0644, &kmsg_fops, 0 }, • dmesg utility accesses /dev/kmsg
  8. 8. 8 dmesg • Print or control the kernel ring buffer • Default print all /dev/kmsg messages • Restrict message level ‒ dmesg -l err,warn ‒ dmesg -l info,debug • Set console level ‒ dmesg -n 8 • Wait for new messages ‒ dmesg -w • Clear ring buffer ‒ sudo dmesg -C
  9. 9. 9 loglevel • loglevel=0..7 • Console loglevel • All Kernel Messages with a loglevel smaller than the console loglevel will be printed to the console. suppress_message_printing(msg→level) static bool suppress_message_printing(int level) { return (level >= console_loglevel && !ignore_loglevel); }
  10. 10. 10 loglevel defined • The loglevels are defined as follows: 0 (KERN_EMERG) system is unusable 1 (KERN_ALERT) action must be taken immediately 2 (KERN_CRIT) critical conditions 3 (KERN_ERR) error conditions 4 (KERN_WARNING) warning conditions [quiet] 5 (KERN_NOTICE) normal but significant conditions 6 (KERN_INFO) informational 7 (KERN_DEBUG) debug-level messages [default] 10 (CONSOLE_LOGLEVEL_DEBUG)
  11. 11. 11 Kernel parameters for loglevel • ignore_loglevel ‒ Ignore loglevel setting - this will print /all/ kernel messages to the console. Useful for debugging. We also add it as printk module parameter, so users could change it dynamically, usually by /sys/module/printk/parameters/ignore_loglevel. [1] • debug ‒ [KNL] Enable kernel debugging (events log level). ‒ console_loglevel = CONSOLE_LOGLEVEL_DEBUG; // 10 • quiet ‒ [KNL] Disable most log messages ‒ console_loglevel = CONSOLE_LOGLEVEL_QUIET; // 4
  12. 12. 12 rsyslog • The rsyslog overwrites loglevel when booting • systemctl status rsyslog • /etc/rsyslog.conf # set log level 1 (same as in /etc/sysconfig/syslog). $klogConsoleLogLevel 1 • # cat /proc/sys/kernel/printk 1 4 1 7
  13. 13. 13 printk sysfs • cat /proc/sys/kernel/printk 1 4 1 7 • console_loglevel: messages with a higher priority than this will be printed to the console • default_message_loglevel: messages without an explicit priority will be printed with this priority • minimum_console_loglevel: minimum (highest) value to which console_loglevel can be set • default_console_loglevel: default value for console_loglevel
  14. 14. 14 Default loglevel of SLE/openSUSE • Kernel config: loglevel=7 ‒ CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7 • Grub2 config (grub.cfg): loglevel=4 ‒ Kernel parameter: quiet ‒ GRUB_CMDLINE_LINUX_DEFAULT="resume=/dev/sda1 splash=silent quiet showopts" • rsyslog config (/etc/rsyslog.conf): loglevel=1 ‒ $klogConsoleLogLevel 1 • Conclusion ‒ Before rsyslog start: console_loglevel = 4 ‒ After rsyslog start: console_loglevel = 1 ‒ Please add loglevel=9 or debug, and also “dmesg -n 8” for debugging
  15. 15. 15 Dynamic debug • ddebug_query ‒ [KNL,DYNAMIC_DEBUG] Enable debug messages at early boot time. See Documentation/dynamic-debug- howto.txt for details. [1] ‒ Dynamic debug is designed to allow you to dynamically enable/disable kernel code to obtain additional kernel information. Currently, if ``CONFIG_DYNAMIC_DEBUG`` is set, then all ``pr_debug()``/``dev_dbg()`` and ``print_hex_dump_debug()``/``print_hex_dump_bytes()`` calls can be dynamically enabled per-callsite. [2]
  16. 16. 16 Dynamic debug (cont.) • Kernel parameter: 'ddebug_query=file drivers/firmware/efi/efi-secret-key.c +p' 'ddebug_query=file kernel/module_signing.c +p; file crypto/asymmetric_keys/pkcs7_trust.c +p' • Userland command: # echo -n 'file drivers/base/firmware_class.c +p' > /sys/kernel/debug/dynamic_debug/control # cat /sys/kernel/debug/dynamic_debug/control
  17. 17. 17 pr_debug() / dev_dbg() #if defined(CONFIG_DYNAMIC_DEBUG) #include <linux/dynamic_debug.h> #define pr_debug(fmt, ...) dynamic_pr_debug(fmt, ##__VA_ARGS__) #elif defined(DEBUG) #define pr_debug(fmt, ...) printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__) #else #define pr_debug(fmt, ...) no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__) #endif
  18. 18. Consoles
  19. 19. 19 Consoles
  20. 20. 20 earlyprintk • Serial port: earlyprintk=ttyS0,115200 ‒ EFI framebuffer: earlyprintk=efi ‒ VGA: earlyprintk=vga ‒ USB2 debug port: earlyprintk=dbgp ‒ USB3 debug port: earlyprintk=xdbc • earlyprintk is useful when the kernel crashes before the normal console is initialized. It is not enabled by default because it has some cosmetic problems. ‒ It’s a OLD parameter in kernel • Only one of vga, efi, serial, or usb debug port can be used at a time. • early_printk() callers: e820, kaslr, mm(idt), perf
  21. 21. 21 earlycon • Output early console device and options. • CONFIG_SERIAL_EARLYCON ‒ v3.16 kernel ‒ Support for early consoles with the earlycon parameter. This enables the console before standard serial driver is probed. The console is enabled when early_param is processed. • earlycon ‒ [X86] ACPI SPCR (Serial Port Console Redirection Table) ‒ [ARM64] Device Tree • earlycon=uart[8250],io,<addr>[,options] =uart[8250],mmio,<addr>[,options] ... ‒ earlycon=uart,io,0x3f8,115200 • EARLYCON_DECLARE() or OF_EARLYCON_DECLARE() in kernel ‒ Set initial function to __earlycon_table section
  22. 22. 22 console • Output console device and options. • tty<n> ttyS<n>[,options] ttyUSB0[,options] ‒ Serial: console=ttyS0,115200n8 ‒ USB: console=ttyUSB0,115200n8 ‒ ARM: console=ttyAMA0,115200n8 • Maximum 8 consoles ‒ console_cmdline[MAX_CMDLINECONSOLES] ‒ #define MAX_CMDLINECONSOLES 8
  23. 23. 23 Reading messages from consoles • screen command ‒ # screen /dev/ttyS0 115200 ‒ # screen -h 10000 /dev/ttyUSB1 115200 • minicom ‒ # minicom -b 115200 -D /dev/ttyUSB0
  24. 24. 24 When can earlcon be enabled? No console before here
  25. 25. 25 The debug log in early booting stage • For booting ‒ In EFI stub, the efi_printk() log is only for expected error. ‒ In kernel, earlyprintk= or earlycon= only be enabled after parse_early_param(). ‒ “KASLR using” message should be exposed with earlyprintk= or earlycon= ‒ KASLR (Kernel Address Space Layout Randomization) • For problems that occur before the serial console and EFI framebuffer are initialized [4] ‒ Debugging a mysterious reset with an infinite loop ‒ Debugging a hang by triggering a reset
  26. 26. 26 DEBUG_LL and ARM • Kernel low-level debugging functions • CONFIG_DEBUG_LL ‒ ARM, Unicore32 ‒ Say Y here to include definitions of printascii, printch, printhex in the kernel. This is helpful if you are debugging code that executes before the console is initialized. ‒ Note that selecting this option will limit the kernel to a single UART definition, … this option should not be enabled for kernels that are intended to be portable. ‒ CONFIG_DEBUG_BCM2835 for Raspberry Pi 1 ‒ DEBUG_UART_PHYS = 0x20201000, DEBUG_UART_VIRT = 0xf0201000 ‒ CONFIG_DEBUG_BCM2836 for Raspberry Pi 2 ‒ DEBUG_UART_PHYS = 0x3f201000, DEBUG_UART_VIRT = 0xf0201000 • Should works with CONFIG_EARLY_PRINTK ‒ Enable by earlyprintk kernel parameter
  27. 27. 27 netconsole • This module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical. [3] • As a built-in, netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. While this doesn't allow capture of early kernel panics, it does capture most of the boot process. [3] • CONFIG_NETCONSOLE=y ‒ NIC driver must also be build-in • CONFIG_NETCONSOLE=m ‒ /etc/modules-load.d/netconsole.conf ‒ /etc/modprobe.d/netconsole.conf
  28. 28. 28 netconsole (cont.) • Sender: netconsole=[+][src-port]@[src-ip]/[<dev>], [tgt-port]@<tgt-ip>/[tgt-macaddr] netconsole=6665@,6666@ /ab:cd:23:b0:14:b6 netconsole=@,@ • Receiver: netcat netcat -k -l -u 6666
  29. 29. Q&A
  30. 30. 30 Reference • [1] Documentation/admin-guide/kernel- parameters.txt, Linux Kernel • [2] Documentation/admin-guide/dynamic-debug- howto.rst, Linux Kernel • [3] Documentation/networking/netconsole.txt, Linux Kernel • [4] Early x86 Linux boot debug tricks, Matt Fleming ‒ http://www.codeblueprint.co.uk/2015/04/15/early-x86-linux- boot-debug-tricks.html
  31. 31. Thank you. 31 Feedback to jlee@suse.com
  32. 32. Corporate Headquarters Maxfeldstrasse 5 90409 Nuremberg Germany +49 911 740 53 0 (Worldwide) www.suse.com Join us on: www.opensuse.org 33
  33. 33. Unpublished Work of SUSE. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.