The Xen hypervisor is a great fit for embedded, automotive and avionics, thanks to its lean architecture, small code base (even smaller on ARM), flexibility and security. Low interrupt latency is an essential requirement for many embedded use cases. Unavoidably, virtualization increases interrupt latency, but different Xen configurations lead to very different results. This talk will explain how to achieve minimal interrupt latency. It will cover how scheduling interacts with interrupt delivery in Xen. It will introduce the new Xen "null" scheduler, designed for maximum determinism and minimum overhead on embedded platforms. Findings will be corroborated by showing interrupt latency numbers taken on Xen on ARM in various configurations, such as vCPU pinning, Credit 1 and 2, and null schedulers.

1. Xen Schedulers and Interrupt Latency
Dario Faggioli & Stefano Stabellini

2. The case for embedded
virtualization

4. Galois
SMACCMPPilot
Demo
Xen Summit 2014

5. Why Xen? Why an hypervisor?
• Efficiency and Consolidation
• Isolation and Partitioning
• Componentization
• Resilience
• Scaling
• Portability

6. Embedded != Cloud
Different requirements:
• short boot times
• small footprint
• small codebase (certifications)
• non-PCI device assignment
• driver domains
• co-processor virtualization
• low, deterministic irq latency
• real time schedulers

7. Embedded != Cloud
Different requirements:
• short boot times
• small footprint
• small codebase (certifications)
• non-PCI device assignment
• driver domains
• co-processor virtualization
• low, deterministic irq latency
• real time schedulers
Xen supports/enables

8. Xen Schedulers
CPU CPU CPU CPU
CPU CPU CPU CPU

9. Xen Schedulers
CPU CPU CPU CPU
CPU CPU CPU CPU
Real Time Scheduler
ARINC 653
Regular VM Scheduler
Credit
Dedicated
to 1 VCPU
Dedicated
to 1 VCPU

10. Automotive
Hardware
Xen
Dom0
Linux Control Domain
UI Domain
Automotive Grade Android
HW Drivers
GPU
Driver
PV Block & Net
frontends
PV Block & Net
Backends
Audio
Driver

11. GlobalLogic

12. EPAMEPAM

13. Xilinx Zynq MPSoC
Xen
Dom0
Linux
Baremetal
App
Toolstack FPGA Driver
Baremetal
App
FPGA Driver
Baremetal
App
FPGA Driver
Baremetal
App
FPGA Driver
FPGA
Dedicated CPU Dedicated CPU Dedicated CPU Dedicated CPU

14. Latency Impact of
Schedulers

15. pCPUs and vCPUs...
pCPU0
pCPU1
pCPU2
pCPU3
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu vcpu
vcpu
vcpu
vcpu
We want
to run!!
We are
blocked...
vcpu
vcpu
I’m running

16. Keeping vCPUs “organised”: runqueues
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpuvcpuvcpurunq
vcpu vcpurunq
runq
vcpu vcpu vcpurunq

17. Runqueues in Credit1
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpuvcpuvcpurunq0
vcpu vcpurunq1
runq2
vcpu vcpu vcpurunq3
1 runqueue x pCPU
vcpu
vcpu
vcpu

18. Runqueues in Credit2
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpuvcpurunq
runq
Runqueues are shared
vcpu
vcpu
vcpu

19. A vCPU Wake-Up in Credit1
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpuvcpuvcpurunq0
vcpu vcpurunq1
runq2
vcpu vcpu vcpurunq3
Case a:
1. vcpu goes in a runq1
2. where can vcpu run?
Hey, pCPU2 is idle!
3. put vcpu in runq2
4. pCPU2 picks up vcpu
from its runqueue
Case b:
1. vcpu goes in a runq3
2. hey, vcpu can prempt
what’s running on
pCPU3!
3. context switch
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
(1)
(2)
(3)
(4)
vcpu
vcpu
(1)
(2)
(3)

20. A vCPU Wake-Up in Credit2
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpurunqA
runqB
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
Case a:
1. vcpu goes in a runqA
2. load-balancer moves
vcpu to a less loaded
runq
3. pCPU2 picks up vcpu
from its runqueue
Case b:
1. vcpu goes in runqB
2. pCPU3 picks up vcpu

21. A vCPU Wake-Up in Credit20.000971102 irq_enter
0.000971102 irq_direct, vec fa, handler = apic_timer_interrupt
0.000971649 raise_softirq TIMER_SOFTIRQ
0.000971962 irq_exit, in_irq = 0
0.000974070 softirq_handler TIMER_SOFTIRQ
0.000976010 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=6 (softirq)
0.000976010 tasklet_enqueue fn=hvm_assert_evtchn_irq
0.000976510 raise_softirq TASKLET_SOFTIRQ on cpu 6
0.000978213 softirq_handler TASKLET_SOFTIRQ
0.000981562 tasklet_do_work fn=hvm_assert_evtchn_irq
0.000982017 vcpu_wake d1v1
0.000982437 runstate_change d1v1 blocked->runnable
0.000982987 csched2:update_load
0.000983230 csched2:update_rq_load rq# 0, load = 1, avgload = 34.830%
0.000983430 csched2:update_vcpu_load d1v1, vcpu_load = 11.824%
0.000983735 csched2:runq_insert d1v1, position 0
0.000984060 csched2:runq_tickle_new d1v1, processor = 6, credit = 5292567
0.000984490 csched2:runq_tickle cpu 6
0.000984842 raise_softirq SCHEDULE_SOFTIRQ on cpu 6
0.000985500 softirq_handler SCHEDULE_SOFTIRQ
0.000988941 csched2:schedule cpu 6, rq# 0, idle, SMT idle, tickled
0.000989344 csched2:runq_cand_check d1v1
0.000989611 csched2:runq_candidate d1v1, credit = 5292567
0.000990881 sched_switch prev idle, run for 344.6us
0.000991199 sched_switch next d1v1, was runnable for 5.862us, next slice 1000.0us
0.000991377 sched_switch prev idle next d1v1
0.000991697 runstate_change idle running->runnable
0.000991979 runstate_change d1v1 runnable->running
vcpu wakes-up;
goes in runq
Scheduler triggered
on CPU 6
CPU 6 schedules
vcpu runs
Interrupt arrives
Scheduling
Introduced latency:
9.9620 us
BEST CASE

22. _This_ _is_ _all_ _good_ ...
… Because, thanks to this, we can offer VMs/users:
• Overcommitting (i.e., having more vCPUs than pCPUs)
• Weighted fair share of pCPU time
• Hard and soft affinity
• Cache and NUMA awareness
• Caps and reservations on pCPU time
… but it _comes_ _at_ _a_ _price_

23. The ‘null’ Scheduler

24. A scheduler that does nothing
If we want features & flexibility, we must pay the price :-(
What if we don’t want (or need) them, e.g.:
• Static environments (some embedded usecases)
• Systems (or cpupools, where we know we’ll never have
overcommit)
• For testing/benchmarking (as reference)
Then you can use the ‘null’ scheduler

25. Runqueues in ‘null’. No, wait...
pCPU0
pCPU1
pCPU2
pCPU3
vcpu
vcpu
vcpu
vcpu
There are no runqs at all!
• vCPUs are statically assigned to pCPUs
• Only 1 vCPUs per pCPU
• Overcommit is possible (i.e.: the system won’t
explode), but use only if you really know what
you’re doing (i.e.: the VMs will likely explode!)

26. A vCPU Wake-Up in ‘null’
pCPU0
pCPU1
pCPU2
pCPU3
1. vcpu wakes up and run
:-)vcpu
vcpuvcpu
(1)

27. A vCPU Wake-Up in ‘null’
0.636884641 irq_enter
0.636884641 irq_direct, vec fa, handler = 0xffff82d080267ec4
0.636885492 raise_softirq TIMER_SOFTIRQ
0.636885922 irq_exit, in_irq = 0
0.636889583 softirq_handler TIMER_SOFTIRQ
0.636892021 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=5 (softirq)
0.636892021 tasklet_enqueue fn=hvm_assert_evtchn_irq
0.636892836 raise_softirq TASKLET_SOFTIRQ on cpu 5
0.636895074 softirq_handler TASKLET_SOFTIRQ
0.636895607 tasklet_do_work fn=hvm_assert_evtchn_irq
0.636896202 vcpu_wake d1v1
0.636896712 runstate_change d1v1 blocked->runnable
0.636897197 raise_softirq SCHEDULE_SOFTIRQ on cpu 5
0.636898470 softirq_handler SCHEDULE_SOFTIRQ
0.636899465 null:schedule cpu 5, vcpu d1v1
0.636899720 sched_switch prev idle, run for 999936.973us
0.636899970 sched_switch next d1v1, was runnable for 2.411us
0.636900155 sched_switch prev idle next d1v1
0.636900448 runstate_change idle running->runnable
0.636900738 runstate_change d1v1 runnable->running
vcpu wakes-up;
goes in runq
Scheduler triggered
on CPU 6
CPU 6 schedules
vcpu runs
Interrupt arrives
Scheduling
Introduced latency:
4.5360 us
(less than half of
Credit2)
NORMAL CASE

28. Benchmarks

29. Hardware and Software configuration
Hardware:
Xilinx Zynq MPSoC: 4 ARM A53 Cores
Physical Timer
Software:
Xen 4.9.0-rc7 (+ phys_timer forwarding patch)
Dom0: Linux 4.9, dom0_mem=1G, max_dom0_vcpus=2
1 vcpu TBM ctest

32. Fin