As the first ARM servers and microservers hit the market, Xen on ARM is becoming more mature, stable and reaching feature parity with x86. This talk will present the current status of the project, will describe the latest improvements, the gaps that still need to be filled and the roadmap going forward. ARMv8 silicon is now available for purchase: we can measure how well Xen on ARM 64-bit is performing on real hardware and compare the performance figures with other hypervisors. The presentation will show these results, it will measure the overhead introduced by Xen on ARM and will compare it with the overhead introduced by Xen and KVM on x86. The talk will explain the reasons behind performance shortfalls and present ideas on how to address them in the future. The performance results will be used to determine when it makes sense to use Xen on ARM and what are the best use cases for it.

1. Xen on ARM
How fast is it, really?
Stefano Stabellini
18 August 2014

2. Status

3. Xen Project 4.4 release: status
Features:
● 64-bit guest support in ARMv8
● stable hypercall ABI
● basic lifecycle operations
● memory ballooning
● scheduler configurations, vcpu pinning, cpu pools
HCL (not inclusive):
● ARMv8 FVP
● Arndale board
● Allwinner Sunxi (Cubieboard 2)
● TI OMAP5
● Applied Micro X-Gene
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

4. Linux v3.13+: status
● dom0 and domU
● 32-bit and 64-bit
● SMP
● swiotlb-xen
● PV drivers
○ disk
○ network
○ console
○ framebuffer
○ keyboard
○ mouse
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

5. Coming in Xen 4.5
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● SMMU support
● Device Assignment on ARM
● Xen UEFI host support
● GICv3 support
● support for up to 1TB of guest RAM
● super pages support
● support for sparse memory map
● support for 48-bit physical addresses
● QEMU PV backend on ARM: framebuffer,
mouse, keyboard, multi-console
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
● MiniOS on ARM
● PSCI v0.2

6. Coming in Xen 4.5
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● SMMU support
● Device Assignment on ARM
● Xen UEFI host support
● GICv3 support
● support for up to 1TB of guest RAM
● super pages support
● support for sparse memory map
● support for 48-bit physical addresses
● QEMU PV backend on ARM: framebuffer,
mouse, keyboard, multi-console
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
● MiniOS on ARM
● PSCI v0.2

7. xen-devel traffic
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

8. Maintainers & reviews
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

9. Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● super pages support
● support for up to 1TB of guest RAM
● support for sparse memory map
● support for 48-bit physical addresses
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

10. irq latency reduction: no maintenance interrupts
DomU
virq 109
Xen
irq 109
© 2013 Citrix | Confidential – Do Not Distribute
DomU
EOI
Xen
DomU
Xen
Maintenance
interrupt
GICH_LR
Write
GICH_LR
Clear
© 2014 Citrix

11. irq latency reduction: no maintenance interrupts
DomU
virq 109
Xen
irq 109
© 2013 Citrix | Confidential – Do Not Distribute
DomU
EOI
Xen
DomU
Xen
Maintenance
interrupt
GICH_LR
Write
GICH_LR
Clear
© 2014 Citrix

12. Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● super pages support
● support for up to 1TB of guest RAM
● support for sparse memory map
● support for 48-bit physical addresses
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

13. irq migration: physical follow virtual
© 2013 Citrix | Confidential – Do Not Distribute
vcpu0 vcpu1
virq 109
pcpu0 pcpu1
irq 109
© 2014 Citrix

14. irq migration: physical follow virtual
© 2013 Citrix | Confidential – Do Not Distribute
vcpu0 vcpu1
virq 109
pcpu0 pcpu1
irq 109
© 2014 Citrix

15. irq migration: physical follow virtual
© 2013 Citrix | Confidential – Do Not Distribute
vcpu0 vcpu1
virq 109
pcpu0 pcpu1
irq 109
© 2014 Citrix

16. Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● super pages support
● support for up to 1TB of guest RAM
● support for sparse memory map
● support for 48-bit physical addresses
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

17. multi-grants support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
Backend Frontend
Page
Page
Page
Grant
Grant

18. multi-grants support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
Backend Frontend
Page
Page
Page
Grant
Grant
mfn
==
mfn

19. multi-grants support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
Backend Frontend
Page
Page
Page
Grant
Grant
mfn
==
mfn
mfn_to_pfn
??

20. multi-grants support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
Backend Frontend
Page
Page
Page
Grant
Grant
Page
pfn == mfn
mfn
==
mfn
mfn_to_pfn
??

21. Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts
● support for interrupt priorities
● irq migration: physical follow virtual
● multi-grants support
● super pages support
● support for up to 1TB of guest RAM
● support for sparse memory map
● support for 48-bit physical addresses
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

22. Coming in Xen 4.5: hardware and interfaces
● SMMU support
● Xen UEFI host support
● PSCI v0.2
● GICv3 support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

23. Coming in Xen 4.5: features
● (Non-PCI) Device Assignment on ARM
● QEMU PV backend on ARM: framebuffer, mouse, keyboard, multi-console
● MiniOS on ARM
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

24. Looking ahead
● save/restore and live-migration
● PCI device assignment
● IOMMU emulation for remote processors
● GICv2m support
● SMMUv2 support
● ACPI support
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

25. Embedded and Automotive
● Embedded and Automotive Xen Project initiative
http://xenproject.org/about/events/viewevent/146-webinar-virtualization-in-the-automotive-industry-2.html
● Embedded and Automotive PV drivers project
http://xenproject.org/developers/teams/embedded-and-automotive.html
http://wiki.xenproject.org/wiki/Embedded_and_Automotive_PV_Drivers/Roadmap
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

26. Benchmarks

27. Execution environment
● Linux v3.15-rc8
dom0, domU and native kernel
● Xen v4.5-unstable+
● QEMU-KVM v2.0.0
● OpenSUSE 13.1
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
Applied Micro X-Gene Intel Xeon CPU X5650
ARMv8 64-bit 8 cores 2.4 Ghz x86_64 2 sockets x 6 cores 2.67 Ghz

28. Kernbench
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

29. PBZIP2
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

30. FIO rand-rw
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

31. SPECjbb2005
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

32. PGBench
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

33. PGBench
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

34. PGBench
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

35. PGBench
virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix
120

36. Benchmarks by

37. SpecINT 2006
Xen on ARM virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
2%
© 2014 Citrix

38. Conclusions

39. Conclusions
● many new features coming
● good performance results
● still lots of room for improvement
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

40. Conclusions
● need more reviewers
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

41. Questions?

42. More information
● http://www.xenproject.org
● Xen on ARM @wiki.xenproject.org http://goo.gl/FKNXe
● Xen on ARM whitepaper http://goo.gl/vWMn7D
● http://lists.xenproject.org/mailman/listinfo/xen-devel
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

43. © 2013 Citrix | Confidential – Do Not Distribute
Citrix
Work better. Live Better.
© 2014 Citrix

44. Slide parking lot

45. IPerf
© 2013 Citrix | Confidential – Do Not Distribute
Linerate
© 2014 Citrix

46. FIO
Xen on ARM virtualization overhead (percentage) - Lower is better
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix

47. Reading xen-devel
© 2013 Citrix | Confidential – Do Not Distribute
© 2014 Citrix