Slides from Linuxcon Japan 2011 https://video.linux.com/videos/virtual-networking-performance

1. Virtual Networking Performance
Stephen Hemminger
stephen.hemminger@vyatta.com
Linuxcon Japan
Virtualization Summit

3. Bi-directional Virtualization Loss Test
100%
75%
Throughput
50%
25%
0%
0 500 1000 1500
Packet Size

4. Houston, we have a problem!

5. Topics
What is virtual router?
Performance measurements
Analysis
Current work
How to help

6. Performance History
2005 2006 2007 2008 2009 2010 2011
1G line rate Multiqueue NIC 10G line rate
1M pps
Xen 3.0 KVM Hyper-V
drivers

8. Performance measurements
Forwarding
RFC 2544 minimum packet
Throughput
TCP bulk transfer
Latency
Request / Response

9. Router Benchmark
RFC2544
Router
Under
Test
Spirent

10. Virtualized Router Benchmark
Router
Under
Guest
Test
Hypervisor
Bridge Bridge
Spirent

11. 1 Gigabit bi-directional forwarding
100%
75%
Throughput
50%
25%
0%
0 200 400 600 800 1000 1200 1400 1600
Packet Size

12. Physical Networks
Rate limited by slowest
component
Adding more layers
does not slow down
throughput
Separate resources

13. Virtual Network overhead
Hyper- Bridge out
visor In
Bridge In Hyper-
visor
Router
Under
Test

14. Network Baseline
Intel i7 920
Intel 10G
LOM 1G network
Loopback – 12G bits/sec
Forwarding – 1M packets/sec per core
Your mileage may vary...

15. Emulated Network Interface
Fake PCI region
Guest Hypervisor
To
Network
Packet
Buffer

16. Legacy (emulated) NIC
Relative Performance vs Native
%
-25%
-50%
-75%
Transmit
-100% Receive
Hyper-V VmWare KVM

17. Virtual NIC
Guest Hypervisor
To
Network
Packet
Buffer

18. Virtual NIC performance
0%
-25%
-50%
-75%
-100% Transmit
Hyper-V VmWare Receive
KVM

19. VNIC characteristics
Hyper-V Vmware Xen KVM
vmxnet3 netfront virtio-net
MTU 1500 9000 65521 65535
Checksum Y Y Y
offload
Segmentation Y Y Y
offload
NAPI Y Y Y
LRO Y
VLAN Y
Multiqueue Y ? ?

20. Guest to Guest Throughput
TSO
Tx csum
No offload
0 1000 2000 3000 4000 5000
Megabits/sec

21. VM to VM performance
2000
Thorouhput (10^6 bits/sec)
0
100 1000 10000 100000
MTU (bytes)

22. TCP performance
2 Tier
NAT
Route
Bridge
Bare
0 2 4 6 8 10
Gigabytes / sec

23. Observations
Slow
Legacy emulated NIC
NAT
Firewall connection tracking
Fast
Virtual NIC
Hardware passthrough
Bare Bridge

24. Multithread benchmark
20,000 UP Guest
SMP Guest
Total Transactions/sec
15,000
10,000
5,000
0
Packet Steering (RPS)
Baseline Multi Queue NIC

25. Multiple Queue
HW or SW
Guest VM
Thread
Device
queue
Thread
Flow Device
VNIC
clas queue
sfier
Thread
Device
queue

26. TODO
More peformance measurement
MACVTAP, VEPA, …
Configuration
Pinning, IRQ assignment, …
Profiling

27. Future work
Virtio-net optimization Michael S. Tsirkin
Multiqueue Krishna Kumar
KVM ↔ QEMU
Zero copy Shirley Ma
Skb lifetime
Security
Batching Rusty Russell
Bufferbloat

28. How to help
Administrator →Tuning and Sizing
Tester →Measure and Test
Developer →Multiqueue and
Offload