2. Agenda
1 Project Overview
2 NSX Conceptual & Logical Design
3 Deep Dive in to Network QoS – Design Decision Point
4 Q & A
CONFIDENTIAL 2
3. • Private Cloud – EMC FEHC-CA with custom enhancements
• vSphere VM as a Service
• Hyper-V VM as a Service
• Physical Server as a Service
• Backup as a Service
• Storage as a Service
• Environment / Facilities
• Two datacenters in metro distance (<5 ms)
• Remote Offices (Technical Rooms) in MPLS distance
• Products and Technologies
• CMP: vRealizeAutomation, vRealize Orchestrator, vRealize Business
• Infrastructure Virtualization: VMware vSphere, Hyper-V, NSX-v
• Servers: Cisco UCS
• Networking: Cisco Nexus
• Storage: EMC VIPR, EMC VPLEX, EMC VNX, VMware VSAN
• Backup: EMC Avamar, EMC Networker, EMC DataDomain
• Security: NSX + PaloAlto Networks
Project Overview
4. Overall Project High Level Concept
Datacenter A Datacenter B
vSphere Resource Pool - GOLD TIER
VMware vSphere Metro Cluster Stretched across two datacenters
Storage Stretched across two datacenters (VPLEX)
Technical Room
Resource Pool - TR TIER
(vSphere + VSAN)
Remote Location
Existing Core Network
Cloud
Consumers
Cloud
Administrators
vRealize
Automation
vRealize Business Std. + Adv.
IT Finance
vRealize Automation
vCenter Orchestrator
vRealize LogInsight
vRealize Operations Manager
vSphere Resource Pool - SILVER TIER
Cluster in single datacenter
Storage in single datacenter
(different storage tiers)
vSphere Resource Pool - SILVER TIER
Cluster in single datacenter
Storage in single datacenter
(different storage tiers)
Cloud Management Infrastructure Cluster
VMware vSphere Metro Cluster Stretched across two datacenters
Storage Stretched across two datacenters (VPLEX)
Cloud Management Software Stack
Cloud Management Platform, vSphere Management, NSX Management workloads
Hyper-V Resource Pool
Cluster in single datacenter
Storage in single datacenter
Hyper-V Resource Pool
Cluster in single datacenter
Storage in single datacenter
Physical Servers Resource Pool
Server in single datacenter
Storage in single datacenter
Physical Servers Resource Pool
Server in single datacenter
Storage in single datacenter
6. NSX-v Security Concept
vRA Business Group: HR
Logical Network
Micro Security Zone
Technical Service - SAP
[NSX Security Group of all VMs
having tag MSZ-SAP]
Micro Security Zone
Technical Service - A
[NSX Security Group of all VMs
having tag MSZ-A]
vRA Business Group: FINANCE
Logical Network
Micro Security Zone
Technical Service - B
[NSX Security Group of all VMs
having tag MSZ-B]
NSX Distributed
Logical Router
MSZ-SAP MSZ-SAP MSZ-SAP
MSZ-SAP MSZ-SAP
MSZ-A MSZ-A
MSZ-B MSZ-B MSZ-B
Default NSX Security Policy
NAME SOURCE DESTINATION SERVICE ACTION
Default Any Any Any Block
SECURITY TAGS
Security tags for technical services:
MSZ-<Technical-Service-from-CMDB>
For example: MSZ-SAP, MSZ-A, MSZ-B
Security tags for applications:
APP-<gkpke.APP-SEC-TAG[x]>
For example: APP-MSSQL, APP-IIS, APP-EXCHANGE
APP-MSSQL
APP-MSSQL NSX SECURITY GROUPS
We have NSX Security Group for each Technical Service.
This security group forms Micro Security Zone for particular
Technical Service.
For example: MSZ-SAP, MSZ-A, MSZ-B
All VMs tagged with the Security Group name will belong to
this security group.
NSX Security Policy for Micro Security Zones
NAME SOURCE DESTINATION SERVICE ACTION
Inside MSZ-A MSZ-A MSZ-A Any Allow
Inside MSZ-B MSZ-B MSZ-B Any Allow
Inside MSZ-SAP MSZ-SAP MSZ-SAP Any Allow
Other NSX Security Groups and Policies
Other NSX security groups and polices can be created based on
applications tags and other metadata available for NSX.
Physical or Hyper-V Server
belonging in to Micro
Security Zone
7. End to End Network QoS - Design Decision Point
• Requirements
• End to end network QoS is required to achieve guarantees for particular network traffics. These
traffics are
• FCoE Storage
• vSphere Management
• vSphere vMotion
• VM production
• VM guest OS agent based backup <== this is the most complex requirement in context of QoS
• Constraints
• CISCO Nexus 7k
• VMware NSX-v
• CISCO UCS servers B200 M4 with virtual interface card VIC1340 (2x10Gb ports - each port
connected to different fabric interconnect)
• Cloud Automation (vRA, vRO)
8. End to End Network QoS – Option 1 of 3
UCS Blade Server
B200 M4
NIC-A1 - 10Gb NIC port
vHBA0
FCoE
CoS 3 40%
Mark as CoS 3
vNIC0
Mgmt
VLAN 100
CoS 1 10%
Mark CoS 1
vNIC2
vMotion
VLAN 101
CoS 2 10%
Mark CoS 2
vNIC4
VM Traffic
VLAN 102
CoS 0 20%
Mark CoS 0
NIC-B1 - 10Gb NIC port
vHBA1
FCoE
CoS 3 40%
Mark as CoS 3
vNIC1
Mgmt
VLAN 100
CoS 1 10%
Mark CoS 1
vNIC3
vMotion
VLAN 101
CoS 2 10%
Mark CoS 2
vNIC5
VM Traffic
VLAN 102
CoS 0 20%
Mark CoS 0
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
vNIC7
Backup
VLAN 103
CoS 4 20%
Mark CoS 4
vNIC6
Backup
VLAN 103
CoS 4 20%
Mark CoS 4
VMwarevSphere-ESXi
vmkernel
Mgmt
(Native VLAN)
vmkernel
vMotion
(Native VLAN)
vmkernel
VTEP
VMware Distributed vSwitch
DVS portgroup (Native VLAN)
VTEP
DVS portgroup (native VLAN)
Backup
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production
VM vNIC
Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 50% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 30% (Backup)
vmnic0 vmnic2 vmnic4 vmnic6 vmnic1 vmnic3 vmnic5 vmnic7
VMware Standard vSwitch VMware Standard vSwitch VMware Distributed vSwitch
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 50% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 30% (Backup)
UCS QoS Policy 1 (vNIC):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None
9. End to End Network QoS – Option 2 of 3
UCS Blade Server
B200 M4
10Gb NIC port (NIC-A1)
vHBA0
FCoE
CoS 3 40%
Mark as CoS 3
10Gb NIC port (NIC-B1)
vHBA1
FCoE
CoS 3 40%
Mark as CoS 3
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
VMwarevSphere-ESXi
vmkernel
Mgmt
vmkernel
vMotion
vmkernel
VTEP
DVS portgroup
VLAN 102, Mark as CoS 0
VTEP
DVS portgroup
VLAN 103, Mark as COS 4
Backup
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production
VM vNIC
Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
vmnic0 vmnic1
VMware Distributed vSwitch (DVS)
DVS portgroup
VLAN 100, Mark as CoS 1
Mgmt
DVS portgroup
VLAN 101, Mark as CoS 2
vMotion
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
DVS per PortGroup Marking
CoS 0: System: VM Traffic
CoS 1: System: Mgmt
CoS 2: System: vMotion
CoS 4: User-def: Backup
vmnic2 vmnic3
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
UCS QoS Policy 1 (vNIC 0,1):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None
vNIC0
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
vNIC1
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
10. End to End Network QoS – Option 3 of 3
UCS Blade Server
B200 M4
10Gb NIC port (NIC-A1)
vHBA0
CoS 3 40%
FCoE
Mark as CoS 3
10Gb NIC port (NIC-B1)
vHBA1
CoS 3 40%
FCoE
Mark as CoS 3
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
VMwarevSphere-ESXi
vmkernel
Mgmt
vmkernel
vMotion
vmkernel
VTEP
DVS portgroup
VLAN 102
VTEP
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production & Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
vmnic0 vmnic1
VMware Distributed vSwitch (DVS)
DVS portgroup
VLAN 100, Mark as CoS 1
Mgmt
DVS portgroup
VLAN 101, Mark as CoS 2
vMotion
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
if DST IP = Backup Server mark as CoS 4 else CoS 0
DVS per PortGroup Marking
CoS 0: System: VM Traffic
CoS 1: System: Mgmt
CoS 2: System: vMotion
CoS 4: User-def: Backup
vmnic2 vmnic3
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
UCS QoS Policy 1 (vNIC 0,1):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None
vNIC0
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
vNIC1
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
11. End to End Network QoS – Final Decision
• Decision
• Option 3 – QoS (802.1p) marking in VDS and end-2-end bandwidth management in UCS
• Justification
• Decision is fully compliant with End to end network QoS requirement
• VXLAN protocol is designed to keep L2 CoS tags by copying inner Ethernet header into outer
Ethernet header => virtual overlay CoS tag is kept even in physical network underlay and it can be
leveraged in Cisco UCS bandwidth management (aka DCB ETS - Enhanced Transmission
Selection) to guarantee bandwidth for particular CoS traffics.
• Single vNIC in VM has positive impact on
• NSX Security Policies
• Simple In-guest OS routing (default gateway only) without need for additional static routes
• vRealize Automation Custom Integrations are simpler (single hostname, simpler integration
with IPAM, etc.)
• Impact
• DVS QoS Policy (conditional 802.1p marking) has to be configured manually for each DVS
portgroup used as NSX virtual wire (aka VXLAN) – can be automated by custom integration
(SOLUTION IMPROVEMENT)
• Detail Test Plan has to be prepared to validate correct QoS behavior (RISK MITIGATION)
12. Questions and Answers
Blog post with additional details:
http://blog.igics.com/2015/12/end-to-end-qos-solution-for-vmware.html
Twitter: @david_pasek
Blog: http://blog.igics.com