Virtualization is becoming increasingly pervasive due to server consolidation and new uses like security and high availability. Virtualization reduces costs by improving server utilization. Future trends include live migration between servers, I/O virtualization using AMD's IOMMU to assign devices directly to virtual machines, and nested virtualization to run hypervisors inside guest virtual machines. AMD's new IOMMUv2 adds support for demand paging of devices using page faults, allowing more efficient use of memory.

1. Virtualization Futures
Tom Woller – Principal MTS
Xen Summit Spring 2010
AMD Corporation

2. Outline
Virtualization Trends
Virtualization technology today
Virtualization technology tomorrow
I/O virtualization with AMD IOMMU
AMD IOMMUv1 review
AMD IOMMUv2 definition and uses
Summary
2 | Xen Summit | April, 2010

3. Today
3 | Xen Summit | April, 2010

4. Trends that are driving virtualization
Reduce total cost of ownership (TCO)
Increased systems utilization (current servers have less
than 10% average utilization, less than 50% peak
utilization)
Reduce hardware (25% of the TCO)
Space, electricity, cooling (50% of the operating cost of a
data center)
Management simplification
Dynamic provisioning
Workload management/isolation
Virtual machine migration
Reconfiguration
Virtualization is a true scalable multi-core work load
4 | Xen Summit | April, 2010

5. CPU Virtualization Trends
The key trend is to eliminate the
overhead of virtualization
– Reduce overall world-switch times
– Reduce world-switch frequencies
Reduce world-switch times
– Tag TLB by ASID VM World-switch Times
– Better caching of VMCB state Fam0F Fam10 Future
1800
1600
Reduce world-switch frequencies 1400
– Nested paging (up to 70%) 1200
Cycles
1000
– Direct device assignment
800
– Implement more functions in the 600
guest OS through paravirtualization 400
200
0
Processor
5 | Xen Summit | April, 2010

6. The Future
6 | Xen Summit | April, 2010

7. Virtualization Futures
Virtualization is becoming increasingly
pervasive
Server consolidation is still the primary
reason
Client virtualization (Xen leading efforts)
Platform virtualization capabilities are
rounded out and exceed the capabilities
of mainframes
7 | Xen Summit | April, 2010

8. Virtualization Futures - Changing Landscape
Novel uses of virtualization
security, high-availability, manageability
Live Migration
key component towards enabling the
dynamic datacenter
I/O Virtualization is driving platform, adapter
and software stack changes
8 | Xen Summit | April, 2010

9. What Else Can You Expect?
A few more virtualization acceleration widgets
Virtualized interrupt controller (interrupts, IPIs)
Additional hardware RAS capabilities
Putting all your eggs in one basket
High-availability / Disaster recovery
VM failover
– SMP
Nested/recursive virtualization
Embedded hypervisors are driving this
Virtualization aware devices supporting I/O Virtualization
NICs, storage
9 | Xen Summit | April, 2010

10. AMD IOMMU (version 1)
H/W help for I/O
Virtualization is
already here…
IOMMU is to Devices
as
MMU is to CPUs
10 | Xen Summit | April, 2010

11. Virtualizing The Platform IOMMU Version 1
HT
ATC
DRAM Device
DRAM Tunnel
DRAM
ATS
PCIe optional
HT bridge
IOMMU
remote ATC
CPU
PCI Express™
CPU PCIe
CPU
switches
devices,
bridge
ATC
Device
Device
Device
ATS
HT PCIe
CPU bridge
CPU
IOMMU
CPU
IO Hub ATC = Address Translation Cache
DRAM (ATC a.k.a. IOTLB)
DRAM HT = HyperTransport™ link
DRAM
PCIe = PCI Express™ link
PCI, LPC, etc ATS = Address Translation Service
11 | Xen Summit | April, 2010

12. AMD IOMMU V1 - Uses
I/O Virtualization
Direct device assignment for efficient I/O
I/O interrupt steering avoids hv interaction
Legacy devices – avoid “bounce buffers”
PCI-SIG
– PCIe IOV – using SR-IOV
– PCIe ATS 1.0 - Address Translation Services
RAS
Device DMA containment
Denial-of-service protection -- interrupt flood or
MSI spoofing
12 | Xen Summit | April, 2010

13. AMD IOMMUv1 Limitations
• No ATS 1.1 PRI (Page Request Index) support
• No Hypervisor memory overcommit
• No Demand Paging, no Page Faults for devices
• Only 1 level of page table walking supported
• Page table supported are not AMD64 compatible
• Very difficult to share Page Tables with OS
• Must be managed by software (Hypervisor)
• Direct device assignment to guest applications in
virtualized systems requires 2nd level (gv to gp)
• No nested virtualization guest device
assignments
13 | Xen Summit | April, 2010

14. Introducing AMD IOMMU Version 2
• IOMMU version 1 compatibility
• ATS 1.1 PRI support (Page Request Index)
• Supports “Page Faults” for devices
• Allows Hypervisor memory overcommit for guests
(Demand paging)
• RDMA usage without pinning memory
• Nested Page Tables
• 2nd levels of page table walking supported
• L1: Guest virtual to Guest Physical (AMD64 compatible)
• L2: Guest Physical to System Physical (v1 compatibility)
• 100% AMD64 compatible level
• Allows direct device assignment in virtualized
systems to use guest virtual address
• Share OS PTs in assigning User Level I/O to devices
in native environments
14 | Xen Summit | April, 2010

15. IOMMUv1 (ATS 1.0) Caching Address Translations
Processor Peripheral
(ATC)
ATS request
TLB
lookup
ATS response
& 1 lvl
PT Use ATS
walk response
IOMMU
15 | Xen Summit | April, 2010

16. IOMMUv2 (ATS 1.1 PRI) Page Fault & Overcommit
Processor Peripheral
(ATC)
ATS request
TLB
lookup
ATS response
& 2 lvl
PT Evaluate ATS
walk PRI request response
• Swap in page
• Alloc new page PPR
queue IOMMU
• Reject request
SW
• Upgrade privs Cmd
• Copy-on-write queue PRI response
• Etc. Evaluate PRI
ATS request response
ATS response
16 | Xen Summit | April, 2010

17. IOMMUv2 Perf counters
& RAS info
Peripheral
page service
requests
(PRI)
Guest CR3
table
Interrupt AMD64 long
remapping page tables
tables (guest)
(host)
IOMMU
page tables
(host)
Command &
Event
buffers
IOMMUv1
base IOMMUv2
changes
17 | Xen Summit | April, 2010

18. Example: Smart NIC RDMA Use Case
Current RDMA
Overhead of managing pinned
buffers User
NIC
Level
Ininiband/
Pinned
VIA
Lack of demand-paging Memory
support
What do we want?
Eliminate need for Pinned
memory RDMA
+
User IOMMUv2
Smart NIC operates on Level
NIC
Infiniband
Unpinned
/VIA
unpinned region directly Memory
ATS 1.1 PRI
+
using ATS PRI and Page Page Faults
Faults
18 | Xen Summit | April, 2010

19. IOMMUv2 Direct Guest Mapping
User-level I/O
User-level I/O
x86 PTE, IOMMU nested paging Shared virtual
PRI+ATS address space
Advanced memory model
• Demand paging
• Swapping RDMA
User process
• Copy-on-write Buffers
Shared Virtual addresses among
smart devices
Direct access to devices at user- IOMMUv2
X86 nested
level reduces I/O overhead nested
paging
paging
System memory
19 | Xen Summit | April, 2010

20. Summary
• Virtualization is a critical part of the future
“dynamic“ data center and making considerable
inroads into the client space
• I/O Virtualization is an integral part of the next
generation server and client
• Next generation AMD IOMMU provides another
level of I/O Virtualization functionality
• Demand Paging for smart devices (NICs, GPGPU, …)
• Two levels of Page Table walking
• Guest User Level I/O direct access to devices
20 | Xen Summit | April, 2010

21. Enjoy the Summit! Any Questions?
Source: infosecurity.com
21 | Xen Summit | April, 2010

22. Trademark Attribution
AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the
United States and/or other jurisdictions. Other names used in this presentation are for identification
purposes only and may be trademarks of their respective owners.
©2010 Advanced Micro Devices, Inc. All rights reserved.
22 | Xen Summit | April, 2010