Adding support for you new shiny board in Xen on ARM is a simple task once you get a kernel running on bare metal. This session will cover the different steps to port Xen on ARM from the firmware to the shell prompt in DOM0. We will give you tips on the common pitfalls when you have your hypervisor, or your DOM0 kernel crashing. We will also provide suggestion on how to debug when the console is not working.

1. Porting Xen on ARM to a new SOC
Julien Grall <julien.grall@arm.com>
Xen Developper Summit 2016
© ARM 2016

2. Xen Architecture
2 © ARM 2016

3. Virtualization
ARMv7 and ARMv8
Provides virtualization for
Timer
Interrupt Controller
Page Table
3 © ARM 2016

4. Virtualization - 2
4 © ARM 2016

5. Xen on ARM
5 © ARM 2016

6. Dom0
First guest to start
Known as the hardware domain
Nearly all devices are assigned to DOM0
Serial, IOMMU, Timer and GIC are used by Xen
Some devices can be blacklisted by Xen
DOM0 kernel should discover devices via ACPI or Device Tree
6 © ARM 2016

7. Groundwork
7 © ARM 2016

8. Preparation before porting
Before starting to port Xen, some groundwork needs to be done:
Check the hardware support
Having the firmware/bootloader to boot the image at EL2
Having an OS supporting the targeted platform
8 © ARM 2016

9. Hardware
ARMv7 and ARMv8 processor with virtualization extension
General Interrupt Controller (GIC) v2 or later
9 © ARM 2016

10. Firmware and bootloader
The firmware or bootloader must drop into EL2 (hypervisor) before starting Xen.
Some vendors locked down the firmware/bootloader to drop into Non-secure EL1 (kernel mode).
Hypervisor Call instruction (HVC) must be enabled.
It can be done by setting SCR EL3.HCE (AArch64) or SCR.HCE (AArch32) to 1.
10 © ARM 2016

11. Firmware tables
Xen supports the below firmware tables out-of-box:
Device Tree
https://www.devicetree.org/
ACPI 6.0 and onwards
http://www.uefi.org/acpi/specs
UEFI only
Technical preview in Xen 4.7
DOM0 with ACPI support has been merged for Linux 4.8
11 © ARM 2016

12. DOM0 kernel
Before adding Xen in the equation, it is highly recommended to get the kernel booting natively
DOM0 support is upstreamed in Linux
Adding support to any other kernel is easy
See http://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd
12 © ARM 2016

13. DOM0 kernel - Linux
DOM0 support has been added in Linux 3.8
It is recommended to use the latest release when possible
Minimal list of options to enable:
CONFIG_XEN_DOM0=y
CONFIG_XEN=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
13 © ARM 2016

14. Porting Xen
14 © ARM 2016

15. A single binary to rule them all
A single Xen binary can
be loaded via different methods (e.g multiboot, UEFI).
boot on multiple hardware.
15 © ARM 2016

16. It is recommended to use the latest
version of Xen when porting to a new
SOC.
16 © ARM 2016

17. Early debugging with Xen
Xen provides early printk to debug crash before the UART driver is initialized.
Only available when CONFIG DEBUG=y
Xen will not be portable, intented only for development
UART selected on the build command line with
CONFIG EARL PRINTK=mach
CONFIG EARL PRINTK= INC>, BASE ADDRESS>, OTHER OPTIONS>
More details on
http://xenbits.xen.org/docs/unstable/misc/arm/early-printk.txt
17 © ARM 2016

18. Early debugging with Xen - 2
Major UARTs supported: pl011, 8250,...
8250: CONFIG EARL PRINTK=8250, BASE ADDRESS>, REG SHIFT>
REG SHIFT> is the left-shift to apply to register offsets within the uart (optional).
pl011: CONFIG EARL PRINTK=pl011, BASE ADDRESS>, BAUD RATE>
BAUD RATE is optional. We recommend to let the bootloader setting the baud rate.
18 © ARM 2016

19. Getting the firmware to load Xen and DOM0
The firmware needs to load in memory Xen, DOM0 kernel and potentially others modules (e.g
initramfs, XSM...). There are 3 methods to do it:
Multiboot
UEFI
GRUB via UEFI (work in progress)
19 © ARM 2016

20. Loading Xen and DOM0 using multiboot
Multiboot is a protocol based on Device Tree.
It is used to describe where the kernel, initramfs... reside in memory.
An example to generate multiboot nodes with U-Boot can be found on the wiki.
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#
Boot_Modules
20 © ARM 2016

21. Loading Xen and DOM0 via UEFI
On AArch64, Xen is built as an EFI application.
A configuration file is used to describe:
The command line
The binaries to load (device tree, kernel, initramfs...) in memory.
The configure file could be passed to the EFI application using the parameter -cfg=myxen.cfg
https://xenbits.xenproject.org/docs/unstable/misc/efi.html
21 © ARM 2016

22. Loading Xen and DOM0 via UEFI - Example
Configuration file example for UEFI:
[global]
default=model
[model]
options=console=dtuart conswitch=x dom0_max_vcpus=2 dtuart=serial0
kernel=vmlinuz console=hvc0 earlycon=pl011,0xf2a00000 root=/dev/ram1 rootwait
ramdisk=initrd.img
dtb=model.dtb
22 © ARM 2016

23. Troubleshooting - Xen is not entering in EL2
Xen will panic when it is not entered in EL2 with the following message:
For AArch32:
- Xen must be entered in Hyp mode -
- Please update the bootloader -
For AArch64:
- Xen must be entered in NS EL2 mode -
- Please update the bootloader -
23 © ARM 2016

24. What to do if Xen is not entered in EL2?
Even if the hardware supports virtualization extensions, the firmware/bootloader may be configured
to enter the kernel/hypervisor in EL1.
Find a version which dropped in EL2.
Find the source code and modify it to enter the hypervisor in EL2.
24 © ARM 2016

25. Platform specific code
In most of the case, platform specific code is not necessary.
Hooks in the core code is provided specific initialization is required.
Platform code resides in xen/arch/arm/platforms
25 © ARM 2016

26. Platform specific code - 2
List of callbacks available:
/* Platform initialization */
int (*init)(void);
int (*init_time)(void);
int (*smp_init)(void);
int (*cpu_up)(int cpu);
/* Specific mapping for dom0 */
int (*specific_mapping)(struct domain *d);
/* Platform reset */
void (*reset)(void);
/* Platform power-off */
void (*poweroff)(void);
/*
* Platform blacklist devices
* List of devices which must not pass-through to a guest
*/
const struct dt_device_match *blacklist_dev;
26 © ARM 2016

27. UART support
Xen has multiple UART drivers (pl011, 8250,...).
They can be found in xen/drivers/char.
The UART used by Xen will not be available for DOM0.
A virtual UART will shadow the real one.
Useful if the kernel use early printk
Very basic: Only write is supported
The UART configuration can be read from:
the parameter dtuart=cfg
stdout-path in the device tree
The SCPR table in ACPI
27 © ARM 2016

28. Debugging DOM0 kernel
Xen console
Switch from DOM0 console to Xen console via CTLR-a three times
Useful key
0 Dump Dom0 vCPUs
q Domains information
e Event channel information
R Reboot the machine
28 © ARM 2016

29. Using Xen debugging facilities in the kernel
Use of hvc 0xFFXX
Supported when Xen is compiled with debug=y
Requires to modify the kernel
0xFFEX
0xFFFD
0xFFFE
0xFFFF
Print the register rX/xX
Print the program counter
Print the character stored in r0/x0
Dump the state of the vCPU
29 © ARM 2016

30. SMP support
Xen is able to bring up secondary processors via different protocols:
Power State Coordination Interface (PSCI)
This is the recommended protocol to bring up CPU.
PSCI 0.1, 0.2 and 1.0 supported
It can be used for rebooting the platform (PSCI >= 0.2).
Spin table (AArch64 only)
Platform specific bringup (AArch32 only)
This protocol should be avoided in favor of PSCI.
It can be implemented with the callcack smp init and cpu up.
30 © ARM 2016

31. Upstreaming
Even if your platform does not require platform specific code, it is recommended to
Document the step to boot Xen on the wiki.
Testing new release of Xen.
31 © ARM 2016

32. Where to ask questions?
devel ML: xen-devel@lists.xenproject.org
#xenarm or #xendevel on freenode
32 © ARM 2016

33. Questions?
33 © ARM 2016

34. The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2016 ARM Limited
© ARM 2016