kdump on rhel7

1. Kdump
What is kdump?
Kdump is a kexec based crash dumping mechansim for Linux. Kdump functionality is broken
mainly in two components, user space(kexec) and kernel space(kdump). Kernel space patches are
already part of main line kernel tree. User space component is nothing but a patch on top of existing
kexec tools. Kdump need two different kernel, one we can call standard(production) kernel and
another we can call crash(capture) kernel.
When the system kernel boots, it reserves a small section of memory for the crash kernel. This
ensures that ongoing Dirct Memory Access(DMA) from the standard kernel does not corrupt the
crash kernel. The kexec -p command loads the crash kernel into this reserved memory.
On x86 machines, the first 640KB of physical memory is needed to boot, regardless of where the
kernel loads. Therefore, kexec backs up this region just before rebooting into the crash kernel.
All of the necessary information about the system kernel's core image is encoded in the ELF format,
and stored in a reserved area of memory before a crash. The physical address of the start of the ELF
header is passed to crash kernel through the elfcorehdr=boot parameter. Optionally the size of the
ELF header can also be passed when using the elfcorehdr=size[KMG]@offset[KMG] syntax.
Setup and Installation
Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, and s390x architectures.
And Kdump is install in RHEL5, RHEL6, RHEL7 as default.
The kdump service uses the /etc/kdump.conf file for configuration. Releases prior to RHEL 6.3
added a special marker to this file in order to ensure kdump would not be enabled on Xen guests.
This marker must be removed to enable kdump on Xen HVM guests. Steps to remove the marker
are described below in "Modifying the kdump.conf file". The kdump service also needs to prepare
an initial ramdisk (dumprd) for the kexec'ed kernel to use on boot. As kexecing into a kernel using
paravirt drivers will not work, and is not supported, the dumprd cannot have the xen paravirt
modules in it. In place of those modules, the standard disk and network drivers that operate on
emulated devices presented from the Xen host must be present. How to prepare the dumprd is
described below in "Preparing the kdump initrd (dumprd)". Generally, users will want to use the
paravirt drivers and only switch to the emulated devices when a kexec is necessary. It is possible to
configure the guest to run with paravirt drivers without losing the option to use emulated devices.
How to configure the guest this way is described in "Booting with paravirt drivers, and without
unplug". Finally, the "Testing kdump" section describes how to test that kdump works.
Modifying the kdump.conf file
How to enable and test kdump in RHEL7.0 guest
1. Login RHEL7.0.
2. Download the kexec-tools user-space package from the following URL:

2. http://download.englab.nay.redhat.com/pub/rhel/rel-eng/latest-RHEL-
7/compose/Server/x86_64/os/Packages/kexec-tools-2.0.4-8.el7.x86_64.rpm (Make sure
the rpm file is the right version for your RHEL7 Version.)
3. Install the kexec-tools, if it's not already installed.
#rpm -Uvh/-ivh kexec-tools-*.rpm
4. Open /etc/kdump.conf in an editor.
5. Delete or comment out (using #) the line with the Kdump_not_supported_on_Xen_domU_guest
marker, if it's present.
Booting without paravirt drivers
6. Preparing the kdump initrd (dumprd)
#dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
7. Add the kernel command line parameter xen_emul_unplug=never crashkernel=128M to the
kernel's command line and boot. This boots using the emulated devices (and appropriate drivers)
and without paravirt drivers.
Note: Another way to disable paravirt drivers and with emulated devices instead is:
1). Change kernel command line parameter xen_emul_unplug=unnecessary
2). Black list the xen pv modules
#cat /etc/modprobe.d/blacklist.conf
[...]
# xen pv drivers
blacklist xen_blkfront
blacklist xen_netfront
8. Check whether the paravirt driver have been loaded in the guest,
# lsmod | grep xen
If loaded they need to be unloaded with the modprobe -r command:
# lsmod | grep xen
xen_netfront 18905 0
xen_blkfront 15687 3
# modprobe -r xen_netfront
# modprobe -r xen_blkfront
9. Start the kdump server in the guest, this will generate a dumprd with the drivers necessary for
the emulated devices

3. [guest]# service kdump start
Redirecting to /bin/systemctl start kdump.service
10. Check the kdump status
[guest]# service kdump status
Redirecting to /bin/systemctl status kdump.service
kdump.service - Crash recovery kernel arming
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled)
Active: active (exited) since Thu 2013-09-12 03:40:19 HKT; 1min 53s ago
Process: 609 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
Main PID: 609 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/kdump.service
Sep 12 03:40:19 systemd[1]: Starting Crash recovery...
Sep 12 03:40:19 kdumpctl[609]: kexec: loaded kdump ...
Sep 12 03:40:19 kdumpctl[609]: Starting kdump: [OK]
Sep 12 03:40:19 systemd[1]: Started Crash recovery ...
Sep 12 03:42:10 systemd[1]: Started Crash recovery …
11. Trigger a kerne panic in guest
[guest]# echo c > /proc/sysrq-trigger
[46185.558117] SysRq : Trigger a crash
[46185.559029] BUG: unable to handle kernel NULL pointer dereference at (null)
[46185.559029] IP: [<ffffffff813a7676>] sysrq_handle_crash+0x16/0x20
[46185.559029] PGD 3bc7a067 PUD 3b5ba067 PMD 0
[46185.559029] Oops: 0002 [#1] SMP
[46185.559029] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM
iptable_mangle bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables
sg crc32_pclmul xen_netfront crc32c_intel ghash_clmulni_intel microcode i2c_piix4 pcspkr
serio_raw i2c_core mperf nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sd_mod crc_t10dif
ata_generic pata_acpi xen_blkfront ata_piix libata floppy dm_mirror dm_region_hash dm_log
dm_mod
[46185.559029] CPU: 0 PID: 14330 Comm: bash Not tainted 3.10.0-15.el7.x86_64 #1
[46185.559029] Hardware name: Red Hat HVM domU, BIOS 3.1.2-371.el5 05/07/2013
[46185.559029] task: ffff88003bfe5c40 ti: ffff88003c450000 task.ti: ffff88003c450000
[46185.559029] RIP: 0010:[<ffffffff813a7676>] [<ffffffff813a7676>]
sysrq_handle_crash+0x16/0x20
[46185.559029] RSP: 0018:ffff88003c451e88 EFLAGS: 00010082
[46185.559029] RAX: 000000000000000f RBX: ffffffff819a6d40 RCX: ffff88003fc0fee0
[46185.559029] RDX: 0000000000000000 RSI: ffff88003fc0e368 RDI: 0000000000000063
[46185.559029] RBP: ffff88003c451e88 R08: ffffffff81c176bc R09: 000000000000021d
[46185.559029] R10: 000000000000021c R11: 0000000000000003 R12: 0000000000000063
[46185.559029] R13: 0000000000000246 R14: 0000000000000004 R15: 0000000000000000

4. [46185.559029] FS: 00007f5f667c3740(0000) GS:ffff88003fc00000(0000)
knlGS:0000000000000000
[46185.559029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[46185.559029] CR2: 0000000000000000 CR3: 000000001896a000 CR4: 00000000000006f0
[46185.559029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[46185.559029] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[46185.559029] Stack:
[46185.559029] ffff88003c451ec0 ffffffff813a7dd2 0000000000000002 00007f5f667c9000
[46185.559029] ffff88003c451f50 0000000000000002 0000000000000000 ffff88003c451ed8
[46185.559029] ffffffff813a82bf ffff88003ba59780 ffff88003c451ef8 ffffffff811feb4d
[46185.559029] Call Trace:
[46185.559029] [<ffffffff813a7dd2>] __handle_sysrq+0xa2/0x170
[46185.559029] [<ffffffff813a82bf>] write_sysrq_trigger+0x2f/0x40
[46185.559029] [<ffffffff811feb4d>] proc_reg_write+0x3d/0x80
[46185.559029] [<ffffffff8119cd3d>] vfs_write+0xbd/0x1e0
[46185.559029] [<ffffffff8119d709>] SyS_write+0x49/0xa0
[46185.559029] [<ffffffff81608fd9>] system_call_fastpath+0x16/0x1b
[46185.559029] Code: 45 39 65 34 75 e5 4c 89 ef e8 f7 f7 ff ff eb db 0f 1f 44 00 00 66 66 66 66 90
55 c7 05 e0 a9 57 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 66 66 66 66 90
55 31 c0 c7 05 3e
[46185.559029] RIP [<ffffffff813a7676>] sysrq_handle_crash+0x16/0x20
[46185.559029] RSP <ffff88003c451e88>
[46185.559029] CR2: 0000000000000000
kdump: dump target is /dev/mapper/rhel_dhcp--11--60-root
kdump: saving to /sysroot//var/crash/127.0.0.1-2013.09.13-16:30:30/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
Copying data : [100 %]
kdump: saving vmcore complete
Rebooting.
[ 5.048363] Restarting system.
12. Login the guest and check the vmcore file , you shuld first install the crash tool and kernel-debuginfo-*
packages.
For RHEL7.0 guest, you can get the kernel version via 'uname -r', and then get the currect version
of kernel-debuginfo-* & kernel-debuginfo-common-* packages from
https://brewweb.devel.redhat.com/

5. 13. Check the vmcore file, vmcore can be analyzed
Booting with paravirt drivers, and without unplug
14. Preparing the kdump initrd (dumprd)
#dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
15. Edit /etc/modprobe.d/blacklist.conf by adding the three lines shown below to blacklist the
drivers used for the emulated devices. This will ensure that even if the host presents the emulated
devices to the guest, the guest will use the paravirt drivers instead.
blacklist ata_piix
blacklist 8139too
blacklist 8139cp
16. Change xen_emul_unplug=never to xen_emul_unplug=unnecessary in kernel command line
and reboot
17. Ensure that the kdump service has started
#service kdump status
18. Trigger a kerne panic in guest
[guest]# echo c > /proc/sysrq-trigger
19. Login the guest and check the vmcore file, vmcore can be analyzed