Embedded Fest 2019. Руслан Биловол. Linux Boot: The Big Bang theory

2. 2 Who am I? • 10+ years background in Embedded Systems development • Texas Instruments, Marvell, Cisco • OpenSource contributor and enthusiast 0295 E4A8 BEA7 3111 24B8 E490 0A02 52FB 7C55 614B

3. 3 How Linux is booting? BIOS/UEFI/ ROM Bootloader Linux Kernel Userspace

5. 5 Questions • Why so many steps? • Why bootloader is needed? • How multiple CPU cores are started? • When Virtual Memory appears in the system? • How first process starts? • What Linux Kernel is doing after starting userspace?

7. 7 What is “Linux Boot”? • Boot is short for bootstrap • A self-starting process to init HW and prepare SW • Power on, wait - and universe is ready

8. 8 How Linux is booting? BIOS/UEFI/ ROM Bootloader Linux Kernel Userspace Initial HW startup Fetch and execute bootloader Low-level HW init Fetch and execute Linux Kernel System initialization Preparation for UserSpace execution Starting processes. System initialization from UserSpace

9. 9 The Big Bang theory •

11. 11 vim Chrome Office VLC Chrome Userspace processes

12. 12 vim Chrome Office VLC Chrome CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM

13. 13 vim Chrome Office VLC Chrome CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM

14. 14 vim Chrome Office VLC Chrome Timer CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM

15. 15 vim Chrome Office VLC Chrome Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM

16. 16 vim Chrome Office VLC Chrome Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM MMU

17. 17 vim Chrome Office VLC Chrome Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM MMU HDD GPU Keybd Mouse Network

18. 18 vim Chrome Office VLC Chrome Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM MMU Linux Kernel HDD GPU Keybd Mouse Network

19. 19 Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) RAM MMU HDD GPU Keybd Mouse Network

22. 22 I've got the power! • x86 - BIOS/UEFI – RAM detection – Hardware detection/Initialization – Boot sequence • BIOS: loads a 512 byte from the MBR and jumps into it • UEFI: loads UEFI application from a partition on a GPT boot device

23. 23 I've got the power! • ARM - ROM – Very minimal init (varies) – Load first-stage bootloader (SD, FLASH, UART)

24. 24 Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) RAM MMU HDD GPU Keybd Mouse Network

27. 27 Bootloader • Is a program that does: – Low-level hardware initialization – Fetch Linux Kernel image (disk, network, etc) and put it to the RAM – Prepare system for kernel boot (CPUs, caches, MMU, interrupts) – Jump into first kernel instruction BIOS/UEFI/ ROM Bootloader Linux Kernel Userspace

28. 28 Bootloaders • GRUB (GRand Unified Bootloader) – x86/x86-64 • LILO (Linux Loader) • U-Boot (“the Universal Boot Loader”) – ARM and other embedded systems

29. 29 Bootloader • ARM64 bootloader requirements (minimal) – Setup and initialise the RAM – Setup the device tree – Decompress the kernel image (if needed) – Call the kernel image

30. 30 ARM64 Kernel starting conditions • Following conditions must be met by bootloader: – Quiesce all DMA capable devices – All interrupts disabled (IRQ, FIQ, Debug, SError) – The MMU must be off – Same coherency domain on all CPUs – Primary (boot) CPU jumps directly to the kernel – Secondary CPUs a either disabled or spinning outside of Kernel

31. 31 Kernel starting conditions Power ON Bootloader Kernel CPU 0 (boot CPU) CPU 1 CPU 2 CPU 3 time

33. 33 Linux Kernel compression • Saves disk space • Faster to decompress in RAM than load full uncompressed image Kernel Image Type Decompressor vmlinux NA (raw ELF file) Image Not compressed Image.gz Bootloader zImage Kernel (self-decompressing)

34. 34 Linux Kernel compression (ARM example) vmlinux Image piggy_data piggy.S piggy.o head.o decompress.o misc.o ... vmlinux zImage Raw kernel executable (ELF) Stripped binary vmlinux gzip-ped Image included into piggy.S Combined with decompressor and bootstrapping part Combined kernel image (ELF) Stripped binary vmlinux

37. 37 Black hole in the Linux?

38. 38 Black hole in the Linux? • /dev/null !

40. 40 Linux kernel start vmlinux head.o main.o Architecture-dependent low-level initialization: - check CPU type and capabilities - enable MMU and virtual memory - call start_kernel() from main.o Initialization of all the kernel features (a lot of them!) Rest of the kernel binary jmp

41. 41 arch/*/kernel/head*[S,c] init/main.c: start_kernel() jmp init/main.c: rest_init() init/main.c: kernel_init() arch/*/kernel arch/*/mm cpu_idle (PID 0) init process (PID 1) Initialization Operation

42. 42 Linux init/main.c • start_kernel() – local_irq_disable() – boot_cpu_init() – setup_arch() /* architecture-specific initialization */ – mm_init() /* Kernel Memory Management */ – sched_init() /* Scheduler init (doesn’t run at this moment) */ – preempt_disable() – init_IRQ() /* IRQ subsystem initialization */ – tick_init() / init_timers() /* Timer sybsystem init */ – rand_initialize() – time_init() /* Platform timers init */ – local_irq_enable() /* IRQs enabled on local CPU */ – console_init()

43. 43 Linux init/main.c • rest_init() – pid = kernel_thread(kernel_init, NULL, CLONE_FS); – pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); – cpu_startup_entry() /* Call into cpu_idle */

44. 44 Linux init/main.c • kernel_init() – smp_init() – do_initcalls() /* drivers initialization */ – prepare_namespace() /* mount initrd if needed */ – free_initmem() – try to run init process: if (!try_to_run_init_process("/sbin/init") || !try_to_run_init_process("/etc/init") || !try_to_run_init_process("/bin/init") || !try_to_run_init_process("/bin/sh")) return 0;

45. 45 Linux initrd • Problem – need to mount root filesystem, but driver for it lives in root fs – chicken and egg issue • Solution – bootloader loads small filesystem with needed drivers to RAM – Kernel mounts it as root fs – initrd does all needed initialization – initrd switches root to actual root filesystem pivot_root bootloader kernel initrd /dev/sda2 Load drivers

46. 46 Linux initrd - typical boot sequence • bootloader loads the kernel and the initial RAM disk • kernel converts initrd into a “normal” RAM disk • root device (initrd) is mounted • /sbin/init is executed • init mounts the “real” root file system • init switches to the “real” root fs using pivot_root system call • init execs the /sbin/init on the new root filesystem – important: PID 1 is preserved • the initrd file system is removed

47. 47 init idletsk Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM MMU Linux Kernel HDD GPU Keybd Mouse Network

50. 50 Init process • The first process that is executed once the Linux kernel loads • Does system initialization on userspace level • Runs all other processes (bootstraps userspace) • Handles processes whose parent died (“zombie”) Kernel initialization time init (PID 0) User Space process 1 process 2 process 3 process n

51. 51 Typical Init system tasks • Desktop systems – find init configuration files/scripts – check and mount filesystems – run daemons (background tasks) – setup networking – run display manager • Embedded systems – depends on Embedded system purpose – often similar to desktop, but without graphics

52. 52 Init process • Popular init systems – SysVinit • survived from UNIX System V days – systemd • modern init system aimed to address the limitations of SysVinit – OpenRC • Gentoo users – busybox • simple, for small/embedded systems

53. 53 Run levels / Targets SysV Runlevel systemd Target Description 0 poweroff.target Shut down the system. 1, s, single rescue.target Single user mode. 2, 4 multi-user.target User-defined/Site-specific runlevels. By default, identical to 3. 3 multi-user.target Multi-user, non-graphical. Users can usually login via multiple consoles or via the network. 5 graphical.target Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login. 6 reboot.target Reboot emergency emergency.target Emergency shell

54. 54 vim Chrome Office VLC Chrome Timer Interrupt controller CPU0 CPU1 CPU2 CPU3 Caches (L1, L2, L3) Userspace processes RAM MMU Linux Kernel HDD GPU Keybd Mouse Network

57. 57 References https://kernel.org https://wiki.archlinux.org

Embedded Fest 2019. Руслан Биловол. Linux Boot: The Big Bang theory

EmbeddedFest

Embedded Fest 2019. Руслан Биловол. Linux Boot: The Big Bang theory