grub2 troubleshooting OEL7 Boot Problem

1. ●
●
●
●
●
●
●
●
●
●
●
●
●
●
GRUB2 and Troubleshooting OL7 Boot Problems
Author: Bob W Wang (bob.w.wang@oracle.com)
Updated on Sep 2017
Agenda
1 GRUB2 New Features
2 OL7 Booting Process
3 GRUB2 Configuration Files
4 GRUB2 Environment Variables
5 Explaining menuentry
6 Demonstrating GRUB2 Related Commands
7 Fixing GRUB2 Issues
8 OL7 Troubleshooting Modes for Boot Problems
GRUB2 is the new Linux bootloader. GRUB2 stands
for GRand Unified Bootloader version 2. As GRUB was not maintained for some
time and lacked some critical features like GPT management needed to handle
disks bigger than 2.4TB it was decided to start a new version from scratch with
modularity in mind.
1 GRUB2 New Features
ability to boot on various file systems xfs ext4 ntfs hfs+ raid etc
gzip files decompression on the fly
management of all disk geometries
support for GPT GUID Partition Tables and MBR Master Boot Record
portability with different architectures BIOS EFI Coreboot etc
ability to load modules at execution time.
2 OL7 Booting Process

2. ●
3 GRUB2 Configuration Files
The GRUB2 configuration is spread over several files:
/boot/grub2/grub.cfg: this file contains the final GRUB2 configuration do

3. ●
●
●
●
●
●
●
not edit it directly!
/etc/grub2.cfg: this is a symbolic link to the /boot/grub2/grub.cfg file
/etc/default/grub: this file contains the list of the GRUB2 variables the
values of the environment variables can be edited
/etc/sysconfig/grub: this is a symbolic link to the /etc/default/grub file
/etc/grub.d: this directory contains all the individual files internally used
by GRUB2.
4 GRUB2 Environment Variables
GRUB_TIMEOUT: The time in seconds after the menu is displayed to boot
the default entry unless a key is pressed. The default is 5. Set to 0 to boot
immediately without displaying the menu or to -1 to wait indefinitely.
GRUB_DISTRIBUTOR: Set by distributors of GRUB and is used to generate
more informative menu entry titles. The example evaluates to "Oracle Linux
Server".

4. ●
●
●
●
●
GRUB_DEFAULT: The default menu entry to boot. A value of 0 boots the
first menuentry. A value of 1 boots the second menuentry. A value of saved
instructs GRUB2 to load the last successfully loaded operating system.
A GRUB_DEFAULT value of saved also allows you to use the grub2-set-
default and grub2-reboot commands to specify the default entry. These
two commands are described as follows:
grub2-set-default: Sets the default entry for all subsequent reboots
grub2-reboot: Sets the default entry for the next reboot only
For example with GRUB_DEFAULT=saved the following command sets the
default entry for all subsequent reboots to the second menuentry:
# grub2-set-default 1
GRUB_DISABLE_SUBMENU: By default the grub2-mkconfig command
generates a top-level menu entry for the kernel with highest version
number and puts all other found kernels or alternative menu entries for
recovery mode in a submenu. Setting GRUB_DISABLE_SUBMENU=true disables
this.
GRUB_TERMINAL_OUTPUT: The terminal output device. When specifying
multiple devices separate the valid terminal output names with spaces.
GRUB_CMDLINE_LINUX: Kernel boot parameters.
GRUB_DISABLE_RECOVERY: By default two menu entries are generated
for each Linux kernel: one default entry and one entry for recovery mode.
Setting GRUB_DISABLE_RECOVERY="true" disables this.
To better understand some of the environment variables here are the
standard display with
GRUB_DISABLE_RECOVERY="true" and GRUB_DISABLE_SUBMENU=true :

5. If GRUB_DISABLE_RECOVERY=“false” here is the new display:
Each kernel line gets an associated line in recovery mode.
If GRUB_DISABLE_RECOVERY is now set to “true” like in the initial standard
display and GRUB_DISABLE_SUBMENU is set to false here is the new display:
If the first entry is selected Oracle Linux Server the system boots. If

6. the second option is chosen the standard menu is shown with an additional
line at the bottom to go back to the first menu with the Esc key:
5 Explaining menuentry
The GRUB 2 configuration file /boot/grub2/grub.cfg contains menuentry
stanzas which represent an installed Linux kernel. Each stanza begins with the
menuentry keyword with options. Each menuentry is also a single boot menu
entry in the GRUB 2 menu. The associated block of code is enclosed in curly
brackets { }.

7. ●
○
●
○
●
○
○
The stanza includes a linux16 directive followed by the path to the kernel and
an initrd16 directive followed by the path to the initramfs image. The linux16
directive specifies the kernel version number to be booted as well as kernel
boot parameters. A separate /boot partition was created therefore the path to
the kernel as well as to the initramfs image are relative to /boot.
The initrd16 directive must point to the location of the initramfs file
corresponding to the same kernel version. In other words the kernel as given
on the linux16 /vmlinuz- line must match the version number of the initramfs
image given on the initrd16 /initramfs-[kernel_version].img line of each
stanza.
6 Demonstrating GRUB2 Related Commands
Add/remove entries
Use yum or rpm to add or remove installed kernels - Don t go toying
around manually on the exam - You will mess things up and cost
yourself time.
Show kernel that will load at boot: grub2-editenv list
Lists default kernel selection
Show all kernel boot options
grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
cat /etc/grub2.cfg | awk -F' '$1=="menuentry " {print i++ " : "
$2}'

8. ●
○
◆
○
◆
●
○
◆
◆
◆
○
◆
◆
◆
●
○
◆
◆
◆
◆
◆
Change Default Kernel: grub2-set-default
EX: grub2-set-default 'Oracle Linux Server 7.2, with Linux
3.10.0-514.10.2.el7.x86_64'
This will select that particular kernel - if it exists
EX: grub2-set-default 0
This will make the default kernel the first entry in the bootloader
arrayed numbering .
GRUB2 PW Protection
Edit /etc/grub.d/01_users and add the following lines:
set superusers='admin'
password admin pwd123
Supplement admin with whatever username you want and pwd123
with their password
Edit /etc/grub.d/10_linux to tell GRUB what entries to PW protect
By default each entry will have a --unrestricted directive
Replace --unrestricted with --users user without the quote and
specify which user by name you created. This will allow only those
users to access that menu option.
If you do not make user of the --unrestricted directive or a --user
directive the system will assume ONLY SUPER USER.
Encrypting PWs in GRUB2
grub2-mkpasswd-pbkdf2
it will prompt you for a password and it will output it in encrypted
form that is compatible with GRUB2.
Copy the hash and paste it in the template file where you
configured users 01_users file found in the /etc/grub.d/ folder.
The format is really messy but it looks something like:
set superusers='admin'
password_pbkdf2
admin grub.pbkdf2.sha512.10000.19074739ED80F115963D984BDCB3
5AA671C24325755377C3E9B014D862DA6ACC77BC110EED41822800A87FD
3700C037320E51E9326188D53247EC0722DDF15FC.C56EC0738911AD86C
EA55546139FEBC366A393DF9785A8F44D3E51BF09DB980BAFEF85281CBB

9. ◆
◆
◆
●
○
◆
◆
◆
◆
●
○
◆
●
○
◆
◆
◆
◆
◆
◆
●
○
○
○
○
◆
C56778D8B19DC94833EA8342F7D73E3A1AA30B205091F1015A85
Append that entry to the bottom and it will check for hashes.
If you are in terminal you may install gpm # yum install gpm to
enable mouse support in order to copy and paste the encrypted
password.
NOTE: IF YOU MESS THIS UP YOUR SYSTEM PROBABLY WONT
BOOT.
Resettings GRUB2
Sometimes you gotta restart shit
rm /etc/grub.d/* delete all grub.d scripts
rm /etc/sysconfig/grub remove all system configuration
yum reinstall grub2-tools reinstall the entire package
grub2-mkconfig -o /boot/grub2/grub.cfg reset the entire
configuration
Reinstalling GRUB2
grub2-install <device>
This will reinstall and restore any corrupted files in the /boot/grub2/
directory - If files are missing they will be recreated.
Finalizing changes:
IF YOU CHANGE ANYTHING IN GRUB YOU NEED TO RUN grub2-
mkconfig OR THE CHANGES WILL NOT STICK.
Two ways to do this:
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-mkconfig > /boot/grub2/grub.cfg # don t forget ">"
otherwise it will go to the standard output
USE ONLY IF ON TRADITIONAL BIOS
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
USE ONLY IF ON UEFI BIOS
Telling GRUB to boot into different targets
Move into the append/edit mode with the e key at boot menu
IF USING x86-64 -- append boot directive to the end of the linux16
line
IF USING UEFI -- append boot directive to the end of the linuxefi line
Possible Appends
systemd.unit=[systemctl.target]
So as you can see we can do a lot with GRUB2 - In many ways I think it's much
more convoluted than GRUB1.
7 Fixing GRUB2 Issues
If you install a second operating system or if your GRUB configuration file gets
corrupted due to human error there are ways you can get your system back on

10. ●
its feet and be able to boot again. In the initial screen press c to get a GRUB
command line remember that you can also press e to edit the default boot
options and use help to bring the available commands in the GRUB prompt:
Fix Grub Configuration Issues in Linux
We will focus on ls which will list the installed devices and filesystems and we
will examine what it finds. In the image below we can see that there are 4 hard
drives hd0 through hd3 .
Only hd0 seems to have been partitioned as evidenced
by msdos1 and msdos2 where 1 and 2 are the partition numbers and msdos is
the partitioning scheme .
Let s now examine the first partition on hd0 msdos1 to see if we can find
GRUB there. This approach will allow us to boot Linux and there use other high
level tools to repair the configuration file or reinstall GRUB altogether if it is
needed:
# ls (hd0,msdos1)/
As we can see in the highlighted area we found the grub2 directory in this
partition:

11. ●
●
●
Find Grub Configuration
Once we are sure that GRUB resides in hd0 msdos1 let s tell GRUB where to
find its configuration file and then instruct it to attempt to launch its menu:
set prefix=(hd0,msdos1)/grub2
set root=(hd0,msdos1)
insmod normal
normal
Find and Launch Grub Menu
Then in the GRUB menu choose an entry and press Enter to boot using it. Once
the system has booted you can issue the grub2-install /dev/sdX command
change sdX with the device you want to install GRUB on . The boot information
will then be updated and all related files be restored.
# grub2-install /dev/sdX
More complex scenarios
GRUB 2 boot problems can leave the system in several states. The user may
see one of the following displays on the monitor when a boot fails. The display
provides the first indication of what might be causing the failure to boot. Here
are the failure prompts and displays and the possible cause of each:

12. e.
f.
g.
h.
i.
j.
k.
grub> prompt: GRUB 2 loaded modules but was unable to find the grub.cfg
file.
grub rescue> prompt: GRUB 2 failed to find its grub folder or failed to
load the normal module.
grub>: - The grub prompt on a blank screen. GRUB 2 has found the boot
information but has been either unable to locate or unable to use an
existing GRUB 2 configuration file usually grub.cfg .
grub rescue>: - The rescue mode. GRUB 2 is unable to find the grub
folder or its contents are missing/corrupted. The grub folder contains the
GRUB 2 menu modules and stored environmental data.
GRUB - a single word at the top left of the screen with no prompt and no
cursor. GRUB has failed to find even the most basic information usually
contained in the MBR or boot sector.
Busybox or Initramfs: GRUB 2 began the boot process but there was a
problem passing control to the operating system. Possible causes include
an incorrect UUID or root= designation in the 'linux' line or a corrupted
kernel.
Frozen splash screen blinking cursor with no grub> or grub rescue
prompt. Possible video issues with the kernel. While these failures are not
of GRUB 2's making it may still be able to help. GRUB 2 allows pre-boot
editing of its menu and the user may restore functionality by adding and/or
removing kernel options in a menuentry before booting.
Each of the GRUB 2 failure modes can normally be corrected either from the
GRUB 2 terminal or by using an Oracle Linux Installation CD. Additionally there
are compatible 3rd party bootable "rescue" CD/USB options which may also
work. If using an Oracle Linux Installation CD it is recommended but not
always necessary to use the same version CD as the system you are trying to
repair. This ensures compatibility of any modules and configuration files that
may be loaded while attempting to repair the system.

13. Other more complex scenarios are documented along with their suggested
fixes in the "Ubuntu GRUB2 Troubleshooting guide: https://help.ubuntu.com/
community/Grub2/Troubleshooting". The concepts explained there are valid for
other distributions as well.
8 OL7 Troubleshooting Modes for Boot Problems
Booting into Rescue or Emergency Targets
To boot directly into rescue target add systemd.unit=rescue.target or just 1 to
the kernel command line. This target is useful if the problem occurs somewhere
after the basic system is brought up during the starting of "normal" services. If
this is the case you should be able to disable the bad service from here. If the
rescue target will not boot either the more minimal emergency target might.
To boot directly into emergency shell
add systemd.unit=emergency.target or emergency to the kernel command line.
Note that in the emergency shell you will have to remount the root filesystem
read-write by yourself before editing any files:
mount -o remount,rw /

14. Common issues that can be resolved in the emergency shell are bad lines in /
etc/fstab. After fixing /etc/fstab run systemctl daemon-reload to let systemd
refresh its view of it.
If not even the emergency target works you can boot directly into a shell
with init=/bin/sh. This may be necessary in case systemd itself or some
libraries it depends on are damaged by filesystem corruption. You may need to
reinstall working versions of the affected packages.
If init=/bin/sh does not work you must boot from another medium.
Early Debug Shell
You can enable shell access to be available very early in the startup process to
fall back on and diagnose systemd related boot up issues with various
systemctl commands. Before enable debug shell you may need to edit file /etc/
default/grub to remove "rhgb quiet" from kernel command line and then run
'grub2-mkconfig > /boot/grub2/grub.cfg' and then 'systemctl set-
default multi-user.target'
Enable it using:
systemctl enable debug-shell.service
or by specifying
systemd.debug-shell=1
on the kernel command line.
Once enabled the next time you boot you will be able to switch to tty9 using
CTRL+ALT+F9 and have a root shell there available from an early point in the
booting process. You can use the shell for checking the status of services
reading logs looking for stuck jobs with systemctl list-jobs etc.
Warning: Use this shell only for debugging! Do not forget to disable systemd-
debug-shell.service after you've finished debugging your boot problems.
Leaving the root shell always available would be a security risk.
Recovering a Lost Root Password

15. rd.break :~ "Ramdisk break"
mount -o remount,rw /sysroot
chroot /sysroot
passwd
touch /.autorelabel #(or you can disable selinux)
Then press Crtl-D, Ctrl-D to exit and reboot.
Additional Resources:
Thanks to the following internet sources which made this document more
elaborated.

16. –
–
–
–
●
–
–
●
–
–
http://docs.oracle.com/cd/E52668_01/E53499/html/section-b4s-kzd-
kn.html
https://opensourceblogging.wordpress.com/2017/02/15/rhel-7-booting-
process/
https://help.ubuntu.com/community/Grub2/Troubleshooting/
https://freedesktop.org/wiki/Software/systemd/Debugging/
Extract an initramfs Image on OL5 and OL6
How to Extract an initramfs Image and Edit/View it Doc ID 1671276.1
How to Rebuild the Initial Ramdisk Image in Oracle Linux 5 or Oracle Linux
6 Doc ID 1573625.1
Extract an initramfs Image on OL7 via skipcpio a built-in tool from dracut
/usr/lib/dracut/skipcpio initramfs-3.10.0-229.el7.x86_64.img | zcat |
cpio -ivd
http://thegeekdiary.com/centos-rhel-7-how-to-extract-initramfs-image-
and-editview-it/