This document discusses optimizing Linux boot times on the Raspberry Pi. It begins with an overview of generic boot optimization concepts like identifying and measuring boot components, removing unnecessary functionality, and reordering initialization. It then presents a case study of optimizing boot for Raspbian on the Raspberry Pi through techniques like disabling unneeded services, assigning a static IP, using a minimal custom distro, and kernel optimizations like disabling initcalls and reducing the kernel size. The goal is to achieve an SSH login within 25 seconds instead of the original 30 seconds.

1. Andrea Righi - andrea@betterlinux.com
Ottimizzare i tempi di boot di Linux

2. Andrea Righi - andrea@betterlinux.com
Agenda
● Overview
● Case study: Raspberry Pi
● Kernel optimizations
● rootfs optimizations
● Q/A

3. Andrea Righi - andrea@betterlinux.com
Overview of boot phases
●
Firmware (boot-loader)
● Hardware probing
● Hardware initialization
● Kernel load and decompression
●
Kernel execution
● Core init (start_kernel)
● Driver init (initcalls)
●
User-space init
● /sbin/init
● RC scripts
●
Application start (first user impression)

4. Andrea Righi - andrea@betterlinux.com
Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization

5. Andrea Righi - andrea@betterlinux.com
Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization
Premature optimization is the root of all evil.
-Donald Knuth

6. Andrea Righi - andrea@betterlinux.com
Optimization
● Remove features
● The fastest code is the code you don't run!
● Defer loading of less important features
● Modularization (LKM)
● Parallelization
● async initialization (kernel/async.c)
● Reduce probing delays
● Filesystem tricks

7. Andrea Righi - andrea@betterlinux.com
Case study
● Raspberry Pi
● Boot time functionality
● Reponsive ssh login
● Tools and techniques
used to improve boot
time
● Focusing on cold-
boot optimizations

8. Andrea Righi - andrea@betterlinux.com
Boot steps
● RPi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

9. Andrea Righi - andrea@betterlinux.com
Boot steps
● Rpi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

10. Andrea Righi - andrea@betterlinux.com
Measure boot-time functionality
(kernel)
● CONFIG_PRINTK_TIME
● Configure it in “Kernel hacking” section
● Adds timing informations to kernel messages
(dmesg)
# dmesg
...
[ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs
[ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1
[ 0.022065] bcm_power: Broadcom power driver
[ 0.022080] bcm_power_open() -> 0
[ 0.022090] bcm_power_request(0, 8)
[ 0.522766] bcm_mailbox_read -> 00000080, 0
[ 0.522783] bcm_power_request -> 0
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

11. Andrea Righi - andrea@betterlinux.com
Kernel initcall tracer
● Introduced in Linux 2.6.28
● Add “initcall_debug” to the kernel boot options
● Allows to record the timings of each initcall in
dmesg
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

12. Andrea Righi - andrea@betterlinux.com
Kernel initcall tracer (example)
● dmesg > dmesg.log
● cat dmesg.log | perl
scripts/bootgraph.pl >
/boot/dmesg.svg

13. Andrea Righi - andrea@betterlinux.com
Raspbian
● http://www.raspbian.org
● General-purpose distro based on Debian
● Optimized for the Raspberry Pi
● compilation settings adjusted to produce hard-float
code
● 2013-05-25-wheezy-raspbian.img

14. Andrea Righi - andrea@betterlinux.com
Raspbian: boot time
# dmesg
...
[ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.664780] EXT4-fs (mmcblk0p2): recovery complete
[ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.700523] devtmpfs: mounted
[ 5.706028] Freeing init memory: 128K
[ 6.257687] init start
[ 7.337703] udevd[154]: starting version 175
[ 8.511002] Registered led device: led0
[ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 16.522927] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS
[ 30.436046] init done
- Shell after ~6.3sec
- SSH after ~30.5sec

15. Andrea Righi - andrea@betterlinux.com
sysv-rc-conf
● Run-level configuration for SysV like init scripts
● disable npt
● disable plymouth
● disable rsync
● disable x11-common and lightdm
● disable alsa-utils
● disable swap (dphys-swapfile)
● disable checkfs
● disable ntp

16. Andrea Righi - andrea@betterlinux.com
Static IP vs DHCP
● Assign a static IP to the board (save the time to
get an IP via DHCP)
● Disable CONFIG_IP_PNP in the kernel .config
● Used to mount rootfs via NFS

17. Andrea Righi - andrea@betterlinux.com
Raspbian: boot time after simple
optimizations
# dmesg
...
[ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.730131] EXT4-fs (mmcblk0p2): recovery complete
[ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.765958] devtmpfs: mounted
[ 5.771460] Freeing init memory: 128K
[ 6.310549] init start
[ 7.389321] udevd[154]: starting version 175
[ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229
[ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs
[ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229
[ 8.544067] Registered led device: led0
[ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs
[ 14.724169] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 15.202445] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 24.991030] init done
About 25 sec to login via ssh, instead of 30 sec (16% faster)

18. Andrea Righi - andrea@betterlinux.com
Build a minimal distro from scratch:
requirements
● build toolchain (gcc, glibc, binutils)
● git://github.com/raspberrypi/tools.git
● HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-
gnueabi)
● Linux (kernel)
● git://github.com/raspberrypi/linux.git
● busybox (rootfs)
● git://busybox.net/busybox.git
● dropbear (ssh server)
● https://matt.ucc.asn.au/dropbear/

19. Andrea Righi - andrea@betterlinux.com
Kernel initcalls (before optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs
[ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs
[ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs
[ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs
[ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs
[ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs
[ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs
[ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs
[ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs
[ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs
[ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs
[ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

20. Andrea Righi - andrea@betterlinux.com
Kernel optimizations
(initcalls)
● disable kprobes (CONFIG_KPROBES): ~95ms
● reduce the number of PTYs: ~90ms
● CONFIG_LEGACY_PTY_COUNT=256 =>
CONFIG_LEGACY_PTY_COUNT=1
● disable framebuffer: ~33ms
● remove loop device: ~5ms

21. Andrea Righi - andrea@betterlinux.com
Kernel initcalls (after optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs
[ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs
[ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs
[ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs
[ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs
[ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs
[ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs
[ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs
[ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs
[ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs
[ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs
[ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs
[ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs
[ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs
[ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs
[ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

22. Andrea Righi - andrea@betterlinux.com
Kernel optimizations
(other optimizations)
● preset loops_per_jiffy
● At each boot the kernel calibrates a delay loop (used later by the udelay()
function)
● 1 jiffy = time between 2 timer interrupts
● CONFIG_HZ=100 => 250ms!!!
● loop
● disable console output
● remove “console=xxx” and add “quiet” to the kernel boot parameters
● use LZO kernel decompression (CONFIG_KERNEL_LZO)
● LZO is a compression algorithm that is much faster than gzip, at the cost of a
slightly degrade compression ratio (+10%)
● reduce kernel size...
● A smaller kernel is faster to load, less code also means smaller working set
(good for caches)

23. Andrea Righi - andrea@betterlinux.com
Kernel image (before => after)

24. Andrea Righi - andrea@betterlinux.com
rootfs
● Generated using busybox and dropbear
● Hints about busybox (reduce fork()s):
● CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait
external binaries
● CONFIG_FEATURE_SH_NOFORK: call <applet>_main() directly without
spawning another task
● Build everything with -Os
● Use mklibs to strip system libraries (rootfs is mounted to /mnt)
● mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-
linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv
lib2 lib
● After all these stops total rootfs size is ~3.2MB

25. Andrea Righi - andrea@betterlinux.com
filesystem optimizations
● Split the filesystem into read-only portion and
read/write portion
● Read-only filesystem mounts faster
● Use Squashfs + tmpfs (or aufs

26. Andrea Righi - andrea@betterlinux.com
root filesystem: boot time

27. Andrea Righi - andrea@betterlinux.com
Kernel image (before => after)
1.5GB
3.2MB

28. Andrea Righi - andrea@betterlinux.com
Demo: custom kernel+rootfs boot time
...
[ 1.144025] Freeing init memory: 92K
[ 1.228550] init start
[ 1.249067] waiting eth0 to come up
[ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 1.346059] Indeed it is in host mode hprt0 = 00001101
[ 1.556942] hub 1-1:1.0: USB hub found
[ 1.557074] hub 1-1:1.0: 3 ports detected
[ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 1.940066] smsc95xx v1.0.4
[ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx
USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 2.086695] init done
[ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
- Shell after ~1.2 sec
- SSH after ~3.4 sec

29. Andrea Righi - andrea@betterlinux.com
Conclusion
● Why a faster boot?
● Consumer electronics products require very fast
boot times (digital camera, mobile phone, etc.)
● Fast recovery upon system failures (crashes)
● You can show to your friends your new awesome
board booting in a couple of seconds ;-)

30. Andrea Righi - andrea@betterlinux.com
References
● “LPC: Booting Linux in 5 Seconds” - Arjan van de Ven's talk:
http://lwn.net/Articles/299483/
● elinux wiki - Tim's fastboot tools:
http://elinux.org/Tims_Fastboot_Tools
● “Update on boot time reduction techniques” - Michael Opdenacker
● Linux: http:.//www.kernel.org
● Busybox: http://busybox.net
● Dropbear: https://matt.ucc.asn.au/dropbear/dropbear.html
● Raspberry Pi: http://www.raspberrypi.org
● http://it.emcelettronica.com/embedded-gnulinux-partendo-da-zero-
test-sulla-raspberry-pi

31. Andrea Righi - andrea@betterlinux.com
Q/A
● You're very welcome!
● Twitter
● @arighi
● #bem2013