This presentation is about the main tasks which Linux kernel platform engineers take care of. The talk includes real-life cases which help understand the role of respective specialists and might be helpful to those who consider such change in their careers.
The talk was delivered by Sam Protsenko (Software Engineer, Consultant, GlobalLogic) at GlobalLogic Embedded Career Day #2 on February 10, 2018.
More about GlobalLogic Embedded Career Day #2: https://www.globallogic.com/ua/events/globallogic-kyiv-embedded-career-day-2-materials
[OpenStack Day in Korea 2015] Track 1-6 - ę°ëźíęł ě¤ě ě´ęľŹěë, ě¸íëźě ě¤íěě¤ëĽź ěŹëŚŹë¤. ꡸ëě ëł´ě´...OpenStack Korea Community
2. 2
Confidential
Agenda
1. Role Overview
2. War Stories and Learnings
⢠Story #1: Board bring-up
⢠Story #2: Migration to a new kernel
⢠Story #3: Hardware fixing
4. 4
Confidential
Tasks
⢠Board bring-up
⢠Porting kernel (and bootloader) to a new board
- Writing the BSP
- Drivers development
⢠Migrating to a new kernel version
⢠Bug fixing
⢠Boot time optimizations
⢠Upstreaming
⢠Sometimes:
- Bootloader related work
- Low-level work in rootfs (Android, OpenEmbedded, etc.)
- Hardware debugging
5. 5
Confidential
Tasks
Make board minimally functional,
step by step, for all required
components. Make sure it works.
⢠Check power and reset lines
⢠Check clock signals
⢠Debug SoC issues (caches, etc.)
⢠Configure pin muxes
⢠Bring-up RAM (DDR), serial
console (UART), peripheral
(eMMC, USB, LCD, etc.)
Board bring-up
6. 6
Confidential
Tasks
Make kernel and bootloader fully
functional on a new board.
⢠Bootloader bring-up
⢠Device Tree file
⢠Configuration file
⢠SoC support
⢠Write missing peripheral drivers
Porting kernel to a new board
7. 7
Confidential
Tasks
Modify board related files to work in
a new kernel version.
⢠Sometimes company doesnât
want to upstream its kernel
⢠Migration work:
- board files -> device tree
- new API for kernel frameworks
- config options
- sometimes back-porting is needed
⢠Automated approach exists: see
âPrequelâ by Julia Lawall
Migrating to a new kernel version
8. 8
Confidential
Components and Tools
⢠Linux kernel
⢠Bootloader (U-Boot, UEFI/edk2)
⢠RootFS (Android, Debian, OpenEmbedded,
BusyBox)
⢠C language
⢠OS principles
⢠Microcontrollers basics (+schematics)
⢠Linux user experience
Components
⢠Software
- toolchain (ARM GCC): gcc, gdb, ld, as
- binutils (add2line, strings, objdump, nm)
- Git
- Linux CLI tools
- board specific and task specific tools
(fastboot, dfu-util, omapconf, minicom, adb)
⢠Hardware
- JTAG
- logical analyzer
- oscilloscope
- soldering iron
Tools
Knowledge
11. 11
Confidential
Story #1: NOR flash support and XIP boot
Task highlights:
⢠Provide NOR flash chip support:
- in kernel
- in U-Boot
⢠Provide XIP boot:
- for U-Boot
- for kernel
12. 12
Confidential
Overview: NOR flash
⢠NOR flash is XIP capable (eXecution In Place)
⢠XIP from NOR is often used for reducing the boot time
⢠Can be useful in automotive and mobile areas
⢠Itâs possible to run U-Boot and kernel from NOR flash
(Android or Linux rootfs is stored on eMMC)
⢠Concerns:
- drivers (GPMC, MTD, CFI, ELM)
- Device Tree definition
- timings
- U-Boot/kernel configuration for XIP
15. 15
Confidential
Overview: NOR and NAND flash comparison
Parameter NOR NAND
Capacity Small (64 KB - 512 MB) Huge (16 MB - 1 TB)
XIP (code execution) Yes No
Block erase time Very slow (1 sec) Fast (500 Îźsec)
Write time Slow (10 Îźsec / word) Fast (200 Îźsec / page)
Read time Fast (100 nsec / word) Fast (50 Îźsec / page)
Erase cycle range 10,000 to 100,000 1,000,000 with ECC
Access method Random Sequential
Price High (4x / MB) Very low (x / MB)
17. 17
Confidential
Sub-task 1: Timings
â GPMC controller must be programmed with correct timings
â Timings must be calculated from NOR chip characteristics
â There are a lot of them
â Usually board manufacturer should provide you with correct timings
â If NOR flash hangs - most likely timings are wrong
18. 18
Confidential
Sub-task 1: Timings
⢠cs-on
⢠cs-rd-off
⢠cs-wr-off
⢠oe-on
⢠oe-off
⢠we-on
⢠we-off
⢠rd-access
And much more else timings...
GPMC: Timings and parameters
22. 22
Confidential
Sub-task 2: Clocks and DPLLs
â GPMC clock must be enabled and be of correct frequency
â Check correct value (calculated for NOR timings)
â Check actual value (from kernel DebugFS)
â Fix clock if needed
â Clock derivation path can be seen from SoC TRM
27. 27
Confidential
Sub-task 2: Clocks and DPLLs
â Enable CONFIG_COMMON_CLK_DEBUG
â Mount DebugFS (in example below, itâs mounted to /d)
â Print clocks frequencies:
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck # cat clk_rate
532000000
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck/l3_iclk_div # cat clk_rate
532000000
But we see from TRM that L3_ICLK must be CORE_X2_CLK / 2
Check clock freq in DebugFS
29. 29
Confidential
Sub-task 2: Clocks and DPLLs
â Print clocks frequencies:
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck # cat clk_rate
532000000
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck/l3_iclk_div # cat clk_rate
266000000
So GPMC functional clock frequency now is 266 MHz, which is a desired frequency.
Check clock freq in DebugFS after fix
30. 30
Confidential
Sub-task 3: Kernel support
â NOR flash and eMMC were sharing the lines (found from schematic)
â So there was only 512 KiB of NOR flash accessible in kernel
â Means no kernel XIP
â But we can still:
â read/write NOR from kernel
â do U-Boot XIP
â NOR support wasnât ready in our kernel at the time (k3.8)
â But it was ready in mainline kernel (3.11)
31. 31
Confidential
Sub-task 3: Kernel support
â Patches often have dependencies
â Itâs hard to find all actual dependencies between branches
â Kernel maintainers have dedicated trees/branches for features like GPMC
â Use smart cherry-picking
Backporting patches
35. 35
Confidential
Sub-task 4: U-Boot XIP
â For XIP boot we donât need MLO (U-Boot SPL), only u-boot.bin
â Boot sequence:
â U-Boot is started by ROM code
â U-Boot copies all files needed to RAM (dtb, kernel, ramdisk)
â U-Boot switches to eMMC (e.g. via GPIO line)
â U-Boot starts kernel from RAM; now we are not in XIP boot anymore
â Problem: U-Boot hangs when running XIP from NOR
36. 36
Confidential
Sub-task 4: U-Boot XIP
â U-Boot is too big to figure out why itâs not working
â We canât even be sure XIP can work at all
â Letâs shrink U-Boot to minimal code and see if it works
â XIP âHello worldâ baremetal firmware was implemented:
â start assembler code (shrinked)
â linker script (minimal: text/rodata/data/bss, all in SRAM)
â BSS clear
â enable UART clock + dependency clocks
â pin muxes
â naiive UART driver
â print âHello worldâ to UART (serial console)
https://gitlab.com/joeskb7/dra7xx-hello
Approach the problem
37. 37
Confidential
Sub-task 4: U-Boot XIP
â Minimal app worked, but U-Boot wonât work
â Now we know NOR flash + XIP is functional and there is
some bug in U-Boot
â Use JTAG to investigate the issue
Investigation
39. 39
Confidential
Sub-task 4: U-Boot XIP
â git bisect-ing helped to find the actual bug
â We were given modified U-Boot with code like this
(in drivers/mmc/Makefile):
CFLAGS := $(subst -Os, -O0, $(CFLAGS)) -fPIC
Root cause (2)
41. 41
Confidential
Sub-task 4: U-Boot XIP
â So only MMC related files were built with -O0 -fPIC
â But other U-Boot files were build with -Os
â That led to NULL pointer dereference
â Removing that line fixed the problem
Fix
42. 42
Confidential
Learnings:
â Always check mainline kernel and mailing lists for existing patch
â Git is your friend:
â cherry-pick thoughtfully
â bisect often helps to find the regression
â If some task is repeated often -- develop some tool or script
â Share it if possible
â Isolating the problem helps to find the root cause
â JTAG is an ultimate tool for debugging
44. 44
Confidential
Story #2: Updating MAX732x driver
Task highlights:
⢠Update MAX732x driver to use Device Tree
⢠Provide bindings documentation
⢠Make it functional and bug-free
⢠Upstream changes
45. 45
Confidential
Overview: Chip description
⢠I2C I/O Expander
⢠Often used in automotive
⢠Have one-directional and
bidirectional ports
⢠Sends interrupt to SoC when input
event occurred
48. 48
Confidential
Sub-task 1: Device tree support
Platform data doesnât exist in case of OF provider.
- chip->irq_base = pdata->irq_base;
+ if (pdata)
+ chip->irq_base =
pdata->irq_base;
Check pdata
50. 50
Confidential
Sub-task 2: Interrupt controller
â GPIO framework is capable of handling IRQ controller code for you
â Remove IRQ domain code / manual IRQ calculation and use gpiochip_* API
config GPIO_MAX732X_IRQ
select GPIOLIB_IRQCHIP
#include <linux/gpio/driver.h>
gpiochip_add()
gpiochip_remove()
gpiochip_irqchip_add()
gpiochip_set_chained_irqchip()
GPIOLIB_IRQCHIP
51. 51
Confidential
Sub-task 3: DT definition and bindings
â We want to I2C address of MAX7325 to be 0x5d
â As per datasheet:
Pin Line
----------
AD2 V+
AD0 V+
Check I2C address on schematics
57. 57
Confidential
Learnings:
â If in doubt -- check schematics
â Check out existing code in kernel for examples
â Check if there is existing framework for that problem
â Take responsibility (technical debt)
â Ask your customer about upstreaming
60. 60
Confidential
Story #3: Flashing EEPROM
Task highlights:
⢠EEPROM chip wasnât flashed with Board ID
⢠It led to 2 issues:
- U-Boot was unable to find correct DTB file
- "board_rev" variable were missing in U-Boot environment
⢠Fix:
- Disable âRead onlyâ mode for EEPROM chip
- Figure out how to work with it
- Write correct Board ID value to EEPROM
- Set back âRead onlyâ mode