SR-IOV allows a physical network interface card to present multiple virtual interfaces to different virtual machines. The document discusses SR-IOV support in Xen, including the SR-IOV specification, Xen architecture with SR-IOV, and considerations around virtual function communication and device instance representation in the host. SR-IOV enables high performance network virtualization by allowing each guest direct access to hardware while avoiding hypervisor intervention at runtime.

1. SR-IOV support in Xen
Yaozu (Eddie) Dong (Eddie.Dong@intel.com)
Yunhong Jiang
Kun (Kevin) Tian

2. Agenda
• SR-IOV specification overview
• Xen/SR-IOV architecture
• Discussions
2

3. SR-IOV specification overview
Start with a single function device
VM1
IOVM VM0
• HW under the control of privileged SW
• Includes an SR-IOV Extended Capability
DD
DD DD
• Physical Function (PF)
Replicate the resources needed by a VM
PCIM
• MMIO for direct communication
• RID to tag DMA traffic
Physical PF VF VF
• Minimal configuration space
Function Shared
• Virtual Function (VF)
Resource
Virtual
Introduces PCI Manager (PCIM) Function
• Conceptual SW entity
DMA, PIO, and Interrupts
• Completes the configuration model Initialization and Configuration
• Translates VF into a full function
• Configures SR-IOV resources
3

4. Xen/SRIOV architecture
SR-IOV generic VF
CFGS emulation
Dom0 (Service VM) user Guest App
DM
CP
SR-IOV
SR-IOV
Normal PCI driver with hot
VF management
add/removal support
Network configuration
PF driver VF driver
Alt1: Inter-VM
communication
Dom0 kernel
Guest Kernel
CFGS access
Alt2: PIO/MMIO based virtual mailbox/doorbell
Xen
Alt3: HW mailbox/doorbell
HW
4

5. Control Panel
VF management: A generic tool with notification to PF for state
change
• VFnums setting / Show
• Enable/Disable/Migrate VFs
Network configuration
• Can Ethtool service for this?
• Or a totally new tool?
5

6. SR-IOV VF/PF Communication Channel
Inter-VM APIs VMM cooperative VF driver
• VF driver will depends on VMM, Guest OS, and even release
from VMM vendor
– Primitive hypercalls are not sufficient
– There is no Windows based Xen Inter-VM high level APIs in upstream
– Xen vendors may release their own abstract APIs with separately
Guest hardware VMM independent VF driver
• Real Hardware mailbox/doorbell – No SR-PCIM involvement
– Good Performance, but IHVs may not implement
• Virtual mailbox/doorbell – Need SR-PCIM support
– Need SR-IOV standard
VF/PF driver pair’s decision to use whatever mechanism,
but suggest using guest hardware
6

7. Virtual hardware for PF/VF communication
Dom0 (Service VM) user
VF->PF doorbell can use I/O
PF->VF doorbell uses virtual IRQ
DM
SRIOV
4: Emulation
5: Notification
3: Back to DM
6: Ack
Guest Kernel
PF driver
VF driver
Dom0 kernel
1: Guest write I/O
8: Done
7: Done
Xen
2: VM Exit to hypervisor
7

8. PCI Device Instance of VF in Host?
No create VF instance
• Need access path for Qemu to R/W VF CFGS
• Modifications are mostly in Qemu side
• Start from this for TTM
Created VF instance
• Pros: Easy for assignment
• Cons: Confuse to other pci modules, invasive change, need
community decision on how to change
• Long term solution
8

9. Summary
IOMMU based SR-IOV support is an ideal IO virtualization
• Each guest can get a portion of hardware
• VMM doesn’t need to intercept at runtime
• High throughput, Low CPU utilization, Perfect scalability
Early VMM support for SR-IOV is critical to IHVs to implement
PF/VF drivers
Dom0 Linux version also matters in terms of PF driver
development
9

10. Legal Information
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION
WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS
AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES
NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR
IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL
PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO
FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR
INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER
INTELLECTUAL PROPERTY RIGHT.
Intel may make changes to specifications, product descriptions, and
plans at any time, without notice.
All dates provided are subject to change without notice.
Intel is a trademark of Intel Corporation in the U.S. and other
countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2007, Intel Corporation. All rights are protected.
10

11. 11