2. Outline
ï Large Workload vMotion challenges
ï Enabling Multiple-NIC vMotion
ï Traffic Flow Considerations
ï QOS
ï NIOC
ï Class Based QOS on the 1000v
ï Real World Design Discussion
ï Quad 10Gb CNA / 1000v / FCoE on UCS
ï Questions
3. Challenges with Large Workloads
At Medtronic
ï 78% of servers are virtualized, the low hanging fruit is already gone.
ï Remaining physical servers are 64GB and larger â Exchange, Oracle,
SQL, SAP Middleware
ï Experienced vMotion failures with large workloads on ESX 4.1
ï Aging vMware Hosts (3+ Years)
Requirements for new a environment
ï Reduced physical footprint
ï Support for a few guests up to 256GB (Current requests are for
128GB)
ï High consolidation ratio â 100+ VMs per Host
ï Network cable consolidation and operationalize support
4. Large Workload vMotion
ï Two key features of ESX 5 provide better support for vMotion of
larger workloads than previous versions
ï Multiple-NIC vMotion provides more bandwidth to
Motion process
ï More bandwidth is always betterâŠthe faster the pre-copy phase
completes the less time the guest has to dirty the pagesâŠ
ï Reduced time to evacuate a host going into maintenance mode
ï Stun During Page-Send (SDPS)
ï SDPS can induce small delays in processor scheduling reducing
the rate that the guest is âdirtyingâ memory pages
ï Guest performance is only reduced if the guest is âdirtyingâ
memory pages faster than vMotion can pre-copy them
5. Multiple-NIC vMotion Performance
With QOS + FCoE, Without Jumbo Frames
vMotion Throughput on 10G CNA (Gb per
Second) on 1000v / UCS 6248 FI
14
12
10
8
6
4
2
0
1 10G CNA 2 10G CNA 4 10G CNA
6. Enabling Multiple-NIC vMotion
âą Follow best practices and use dedicated VMKernel interfaces for
mgmt, vmotion, storage, etcâŠ
âą Create a vMotion VMkernel interface for each physical NIC you would
like to use for vMotion traffic
âą For all practical purposes the vMotion VMKernel interfaces need to be
backed by the same VLAN and address within the same subnet.
âą All VMKernel interfaces enabled for vMotion will be used for both
single or multiple concurrent vMotions
âą Supports up to 16 interfaces with 1Gb NICS, or 4 interfaces with 10Gb
8. VMKernel to NIC Association - vDS
ï Create dvPortGroups before creating VMKernel adapters
ï Create one dvPortGroup for each physical NIC you want to
carry vMotion traffic
9. VMKernel to NIC Association â 1000v
ï VPC-HM with Mac-Pinning
ï Operates in a similar manner to vSwitch and vDS default
options, VMKernel interfaces are pinned to physical NIC
ï âchannel-group auto mode on mac-pinningâ is used to
enable in the ethernet (uplink) port-profile on the 1000v
ï âshow port-channel internal info allâ to learn pinning id
ï Apply the pinning-id command to pin a VMKernel interface
to a NIC
ï port profile type vethernet vMotionA
pinning id 1
ï port profile type vethernet vMotionB
pinning id 2
ï Verify = module vem # execute vemcmd show pinning
10. VMKernel to NIC Association â 1000v
ï VPC LACP
ï Traditional LACP based etherchannel (Active or Passive)
ï Upstream switch needs to support multi-chassis etherchannel
ï âchannel-group auto mode activeâ is used to enable in the
ethernet (uplink) port-profile on the 1000v
ï vMotion VMKernel traffic is distributed among the member
interfaces based on the selected load balancing algorithm
ï âport-channel load-balance ethernetâ to change algorithm.
ï If the default isnât distributing vMotion traffic evenly try
âsource-ip-vlanâ and use consecutive IP address for the
vMotion enabled VMKernel interfaces on a host.
ï Use increments of 2,4,8 ports for even distribution
11. Traffic Flow During a vMotion
ï vCenter steps through the list of vMotion VMKernel adapters on
each host in the order they were presented to vCenter and pairs
them off
ï Speed mismatch will be handled by bandwidth â multiple 1Gb
NICs can be paired to a single 10Gb
ï There isnât a way to reliably control which interfaces are paired
up, this could lead to vMotion traffic overwhelming switch
interconnects
ï A dedicated vMotion switch avoids switch interconnect issues
ï Multi-chassis etherchannel eliminates switch interconnect issues
ï If the NICs arenât dedicated to vMotion use QOS
14. Network IO Control
ï Only available on the dVS (requires enterprise+ )
ï Has built-in resource pools for classes of system traffic
such as vMotion, Management, iSCSI, NFS
ï Traffic shares assign a relative importance to traffic
that is used to create minimum bandwidth reservations
on a per dvUplink basis
ï Only applies to outbound traffic
ï Limits are used to cap traffic on a per dVS basis
15. Class Based WFQ on the 1000v
ï CBWFQ QOS provides minimum bandwidth reservations on a per-
physical port basis
ï Provides built in protocol matches to classify
n1kv, vMotion, management, and storage traffic
ï Only applies to outbound traffic
ï QOS on the 1000v is a three step process
1. Define traffic classes using the class-map command
2. Create a traffic policy with the policy-map command
3. Attach the traffic policy to an interface or port-profile with the
service-policy command
ï FOR CBWFQ QOS EXAMPLE PLEASE DOWNLOAD THE PRESENTATION
16. Real World Design Discussion
- Four 10Gb FcOE CNA on UCS
ï Design Goals â
ï Support large workloads with up to 256Gb of RAM
ï Operationalize support into existing frameworks
ï Support both FC and NFS storage to consolidate existing farms
ï UCS c460 with 1Tb of RAM, two P81E dual port FCoE VICs
ï UCS 6248 Fabric Interconnect with 2232 FEX
ï 40Gb VPC uplink from each fabric interconnect
ï Nexus 1010X / 1000v
ï 4 vNICs and 4 vHBAs presented to ESX
ï Four vMotion VMKernel interfaces per host
ï Currently running at a consolidation ratio of 157 â 1.
ï Replaced 96 ESX Hosts with 8âŠ
ï Succesful vMotion of an 8 way 256gb VM running SQLioSIM.
17.
18. Four 10Gb FCoE CNA on UCS
ï QOS Marking Policy âą Apply to vethernet port-profiles
policy-map type qos class-cos1 port-profile vMotionA
description vMotion
service-policy input class-cos1
class class-default
port-profile vMotionB
set cos 1
service-policy input class-cos1
policy-map type qos class-cos2
description NFS port policy NFS
class class-default service-policy input class-cos2
set cos 2 port-policy v174
policy-map type qos class-cos4 service-policy input class-cos4
description Gold-Data port-policy ESX-Management
class class-default service-policy input class-cos6
set cos 4
policy-map type qos class-cos6
description ESX-Management
class class-default
set cos 6
19. Four 10Gb FcOE CNA on UCS
MGMT
SAP
NFS
vMotion
âą vNIC QOS Policy must be set to âhost control fullâ to trust COS markings
âą 1000v has no visibility to vHBAs utilization of the link
âą Instead of queuing on the 1000v the UCS will Queue on the adapter and fabric
interconnect
âą The âPaloâ adapters are reduced to three queues when placed in host control
full,
âą The UCS fabric interconnect leverages the advanced QOS functions of the 5k
hardware such as virtual output queues to provide effective ingress queuing
20.
21. Key Takeaways
ï ESX 5 can vMotion larger guests than 4.1 with the
addition of SDPS, but more bandwidth reduces the
impact of vMotion on the guest
ï Consideration should be given to traffic flow when
implementing multiple-NIC vMotion, switch
interconnects can be easily overwhelmed
ï Dedicated vMotion adapters are best and should always
be used in 1G environments, but arenât always practical
in 10G environments
ï Without dedicated adapters QOS both on the virtual and
physical switch become important
22. Questions ?
ï Relevant Sessions Remaining :
ï INF-VSP1549 - Insight Into vMotion:
Architectures, Performance, Best Practices, and Futures
ï INF-NET2161 - VMware Networking 2012: Enabling the
Software Defined Network
ï INF-NET1590 - What's New in vSphere â Networking
ï INF-NET2207 - VMware vSphere Distributed Switchâ
Technical Deep Dive
ï Please complete your session surveys
heath@heathreynolds.com
24. Class Based WFQ on the 1000v
ï Step 1 â Classify the traffic
class-map type queuing match-any n1kv_control_packet-class
match protocol n1k_control
match protocol n1k_packet
match protocol n1k_mgmt
class-map type queuing match-all nfs-class
match protocol vmw_nfs
class-map type queuing match-all vmotion_class
match protocol vmw_vmotion
class-map type queuing match-all vmw_mgmt_class
match protocol vmw_mgmt
25. Class Based WFQ on the 1000v
ï Step 2 â Create a policy
policy-map type queuing uplink_queue_policy
class type queuing n1kv_control_packet_class
bandwidth percent 10
class type queuing nfs_class
bandwidth percent 25
class type queuing vmotion_class
bandwidth percent 20
class type queuing vmw_mgmt_class
bandwidth percent 10
ï Step 3 â Attach policy to an interface
port-profile uplink
service-policy type queuing output uplink_queue_policy