INF7827 vSphere DRS Deep Dive: Understanding the Best Practices, Advanced Concepts, and Future Direction of DRS

1. vSphere DRS Deep Dive: Understanding the Best Practices,
Advanced Concepts, and Future Direction of DRS
Naveen Nagaraj, VMware, Inc
Brian Graf, VMware, Inc
INF7827
#INF7827

2. • This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
CONFIDENTIAL 2

3. CONFIDENTIAL 3
Proven Best Practices
Tips and Tricks of how our customers are using DRS
Deep Dive with Advanced concepts
What they are, how they can work in your favor
Advanced Config Options
When you need to tweak DRS a little more than normal
Q&A
This time is for you to ask questions
Agenda
VMware Labs: Technical Preview
Let’s talk technical preview

4. Top DRS Questions from
Customers

5. Top Questions on DRS
• Does DRS take vMotion cost into account while Load Balancing?
• Is READY_TIME a metric that DRS looks into?
• After ESX upgrade using VUM, the cluster was slightly imbalanced and DRS did not achieve a
perfect balance – So what gives?
• How many affinity and anti-affinity rules is too much? What are the implications?
• I keep hearing VM-happiness, what does that mean and how is it related to application workload?
• Can I or should I change the DRS Aggressiveness level? Does it impact the # of vMotions?
• What is the typical deployment size of DRS cluster?
• Is it possible to get visibility on why (or why-not) DRS made a vMotion recommendations?

6. DRS by Numbers

7. vMotion
(100%)
HA
(99.6%)
DRS
(94.8%)

8. 81%Fully Automated
15%Partially Automated
4%Manual

9. 100%
Affinity/
Anti-affinity
rules
89%
48%
Maintenance
Mode
Resource
Pool

10. 34%16-32 nodes per cluster
30%8-16 nodes per cluster
36%4-8 nodes per cluster

11. Deep-Dive With Advanced
Concepts

12. Mission of DRS

13. So… What is VM Happiness? 
• A VM is Happy, when –
– All its demand for resources (CPU, memory, network, storage) are always satisfied.
– Its entitlements (as dictated by Admin settings) are met with no delay
– Minimum or No impact by other provisioning operations (power-on, clone, vMotion etc..)
– Not disturbed while it is BUSY!
– Has enough burst capacity for demand spikes!
– When all its constraints are respected 
• What does VM demand represent?
– VM’s demand is a direct correlation of Application needs!
CONFIDENTIAL 13

14. Why is VM Happiness Important?
• If a VM is Happy  :-
– Application SLA is guaranteed!
– It implies that hosts are not over loaded
– It means resource contention is effectively handled
CONFIDENTIAL 14

15. How does DRS ensure VM Happiness ?
• During Initial Placement :-
– It ensures there is minimum performance impact on already running VMs
– Takes ballooning, swapping, Co-ops time to do “what-if” analysis
– Does pre-requisite moves if needed
– Minimizes need for subsequent load balancing
• During Load Balancing :-
– It does Cost-Benefit Analysis to filter out expensive moves
– Accounts for vMotion cost in terms of CPU, memory and network BW utilization
– Keeps the # of vMotion to minimal
– Ensures enough head room is available on each host
– Picks VM diligently for migration (shared memory pages, balloon, swap, VM size etc..)
CONFIDENTIAL 15

16. Migration Threshold, Cluster Imbalance and VM Happiness
• Migration threshold
– Indicates how “aggressive” DRS should be to achieve “uniform” workload distribution.
– In other words, it indicates VI admin’s “tolerance” for # of vMotions.
– Dictates how strict or relaxed Cost-Benefit analysis should be (analogous to P/E ratio)
• Cluster Imbalance
– Cluster-level metric that indicates workload distribution (measured as a Standard Deviation)
– A slight imbalance is perfectly fine as long as VM is getting its entitlements!
– Good distribution provides head room for spikes and also helps availability (HA)
CONFIDENTIAL 16

17. De-Mystifying Resource Pool
• Resource Pool:
– Powerful abstraction to segregate resources in a cluster
– Set business requirements based on workload importance and characteristics
– Provides isolation between resource pools
– It is the fundamental building block for vCAN partners (Cloud Service Providers)
• Resource controls:
1. Reservation (MHz or MB)
• Minimum MHz or MB guaranteed
• By default, R = 0 <means, no dedicated resource>
2. Limit (MHz or MB)
• Maximum MHz or MB allowed
• By default, L = 0 <means unlimited>
3. Shares (No unit)
• Relative priority between siblings
• How to proportionally divvy resources when there is contention
CONFIDENTIAL 17

18. Resource Pool Example
Root RP
RP2
(Analytics)
RP1
(Production)
VM-P1 VM-P10 VM-A1 VM-A20. . . . . .
Total Cluster
Capacity = 100 GHz
R=80, S=400 R=0, S=100
Total Shares = 400+100 = 500
Contention for = 100 – 80 = 20GHz
RP1 quota = 400 x 20 = 16GHz
500
RP2 quota = 100 x 20 = 4GHz
500

19. Proven Best Practices

20. Best Practices - Tip #1 – Use “Latency Sensitivity” flag
• For latency sensitive VMs set “latency sensitivity” flag
• ESX CPU scheduler gives prioritized scheduling for this VM
• DRS ensures this VM is *not* disturbed during periodic load balancing
CONFIDENTIAL 20

21. Best Practices – Tip #2 – CPU Ready time?
• Check BIOS power management is set to “OS control” mode
• Ensure the ESX power management “Active Policy” is set to “Performance”
CONFIDENTIAL 21

22. Best Practices – Tip #3 – Full Storage Connectivity
• All the hosts have access to all the data stores
• Results in an efficient initial placement, load balancing and workload consolidation
• VM availability is improved significantly
CONFIDENTIAL 22

23. Advanced Options

24. Advanced Option #1 – vCPU to Core ratio
CONFIDENTIAL 24
= 4 vCPU
• 6 VMs provisioned ( 24 vCPU)
• vCPU: core = 24 : 8 = 3 : 1
• Suppose the specified vCPU-to-Core ratio is 3:1
 DRS admission control will prevent more VMs
on this host during Initial Placement
 DRS honors it during Load balancing
• Ratio violated only during mandatory
moves or during HA failover
• Best practice guideline:
• vCPUs should be similar in workload characteristics
• Demand spike variations are within reasonable range
= 8 cores
(2 sockets x 4 cores per socket)

25. Advanced Option #2 – Active vs Consumed Memory
CONFIDENTIAL 25
4GB
Configured VM Size
(8GB)
Consumed
Memory
(6GB)
Active
Memory
Idle
Memory
• DRS does initial placement and LB based on the active memory
• Why does DRS do it based on active memory – because it represents the actual demand or
working set of the VM.
• UI shows consumed memory per host/cluster confusing the customers (shame on us!! )
• Advanced option in UI to switch to consumed memory (if you choose to)

26. Tech Preview: DRS Labs
What we’re working on

27. Lots going on…
CONFIDENTIAL 27
• DRS Algorithm enhancements
• Resource utilization optimization
• UI Changes / Advanced Options
• Network-aware DRS
• Proactive High-Availability
• Predictive DRS
• NSX integration
• Network topology awareness and co-location
Tech Preview

28. DRS Algo Enhancements
• More aggressive
– Detects and corrects outlier situations
– Recommends/balances until no two hosts differ by a defined value
• maximum and minimum host entitlement
• Enhancements and additional Advanced Options
– TryBalanceVmsPerHost – workload distribution for availability – best effort
– CheckPairwiseImbalance – checks the difference between the max and min host entitlement
• And more!
CONFIDENTIAL 28
Tech Preview

29. Resource Utilization Optimization
• Throughput > 2.5x increase
• 70% resource reduction at scale
• VM Power-on Latency > 3x improvement
• DRS Cluster Compatibility check
• > 21x Improvement
• Less than 2% CPU utilization
• > 850 MB Reduction
CONFIDENTIAL 29
Tech Preview

30. Advanced Options in the UI
• Do not need to know the property name
• Easier to consume
• Commonly used options
CONFIDENTIAL 30
Tech Preview

31. Advanced Options in the UI
• Even distribution of virtual machines
– ‘TryBalanceVmsPerHost’
– Best effort attempt for purposes of availability
– Each host given a maxVMs limit (avg VMs per host)
• Only applied to the Load Balancing Algorithm (Initial Placement can violate this)
• Will try to balance VMs (count) but if there is an imbalance of resources, DRS will violate the VM balance
– Attempts to move small VMs to correct the maxVMs limit violations
– May introduce more vMotions
CONFIDENTIAL 31
Tech Preview

32. Advanced Options in the UI
Consumed Memory vs Active Memory
• ‘PercentIdleMBInMemDemand’
• Allow DRS to balance on Consumed Memory
• Specifically for environments are under-committed in memory
CONFIDENTIAL 32
Tech Preview

33. Advanced Options in the UI
CPU Over-commitment
• Used heavily by VDI
• Applies for certain application requirements (exchange and others may require specific ratio)
• MaxVcpusPerCore – Set max CPU Overcomittment per host for cluster
• MaxVcpusPerClusterPct – Set max CPU Overcommit for the cluster - Now in UI
CONFIDENTIAL 33
Tech Preview

34. Network-Aware DRS
CONFIDENTIAL 34
• Although CPU and Memory are important metrics, Network utilization can
also be important
• Network utilization is not a first-class citizen with CPU and Memory
– DRS still favors CPU and Memory
– Based on host pNIC saturation
– Used as a ‘check’
• Advanced option for Network Utilization %
– ‘NetworkAwareDrsSaturationThresholdPercent’
Tech Preview
UTILIZATION

35. Network-Aware DRS
• Current DRS logic
CONFIDENTIAL 35
Calculate Host
Load Standard
Deviation
Current Load
Deviation > Target
Load Deviation?
Done
No
Simulate vMotion
of each VM to a
different host and
calculate Current
Host Load
Deviation
Yes Cost/Benefit
Analysis
Is positive
result?
Move on to next
VM
No
Give priority rating
and add to
recommendation
list
Yes
Calculate Current
Host Load
Deviation from
recommendation
list
Balanced?
No
Publish DRS
Migration
Recommendation
List
Yes
Calculate Host
Load Standard
Deviation
Current Load
Deviation > Target
Load Deviation?
Done
No
Simulate vMotion
of each VM to a
different host and
calculate Current
Host Load
Deviation
Yes
Cost/Benefit
Analysis
Is positive
result?
Move on to next
VM
No
Give priority rating
and add to
recommendation
list
Calculate Current
Host Load
Deviation from
recommendation
list
Balanced?
No
Publish DRS
Migration
Recommendation
List
Yes
Is Destination Host
saturated?
Yes No
Yes
• Network-Aware DRS logic
Tech Preview

36. Predictive DRS
• Tight integration with vRealize Operations Manager (vROPs)
• Resource utilization trends are observed
• Predicted demand of workloads is incorporated in ‘initial placement’ and ‘load balancing’
• Current VM demands are honored before future demands are satisfied
CONFIDENTIAL 36
Tech Preview
vSphere
DRS
• Ingests forecasted
metrics
• Balances cluster
based on forecasted
utilization
vRealize
Operations
• Computes and forecasts
utilization based on metric
history.
• CPU
• Memory
• Dynamic Thresholds
created and data passed to
DRS

37. Predictive DRS
• Some workloads have predictable
resource utilization trends
• Having a high level of confidence
allows DRS to pro-actively prepare
for increased demand before
demand occurs
• Potentially faster balancing and
better performance from VMs
CONFIDENTIAL 37
Predicted spike: prepare
Proactive remediation complete
Observed
Predicted
Observed spike: react!
Remediation complete
time
resource
demand
Tech Preview

38. Proactive High Availability
• Most existing availability solutions are reactive in nature
• Increase the availability of VMs even more than current technology provides
• Proactive evacuation of VMs from degraded hosts based on hardware health metrics
• Tight integration, qualification and certification with hardware vendors
• Vendor plugin evaluates host health based on the hardware version and component
redundancy
– Moderately degraded
– Severely degraded
• VI admin can configure the DRS action for each health state event
– Host maintenance mode
– Host quarantine mode
• VI admin can filter the events
CONFIDENTIAL 38
Tech Preview

39. What would this look like?
39
vSphere
DRS
1. Servers running
in Datacenter
2. Hardware is monitored via OEM software
3. Health alerts/updates
pushed to vCenter
4. DRS and health state are invoked.
Workloads are moved according to
severity
Tech Preview

40. Degradation events generated in vCenter
CONFIDENTIAL 40
Provider Health Host Failure Condition
Remediation
Tech Preview

41. Customized Proactive HA automation settings
CONFIDENTIAL 41
Tech Preview

42. Server A Server B Server C Server D
VM
3
VM
4
VM
7
VM
8
VM
11
VM
12
VM
15
VM
16
VM
1
VM
2
VM
5
VM
6
VM
9
VM
10
VM
13
VM
14
VMs:
1, 6
Significant Traffic
VMs:
10, 11, 16
Heavy Traffic
VM Colocation DRS
What would you think of…

43. VM Colocation DRS
Server A Server B Server C Server D
VM
3
VM
4
VM
7
VM
8
VM
11
VM
15
VM
1
VM
5
VM
9
VM
10
VM
13
VM
14
VM
2
VM
6
VM
12
VM
16VMs:
1, 6
Significant Traffic
VMs:
10, 11, 16
Heavy Traffic
VMs:
1, 6
Significant Traffic
Now internal traffic
VMs:
10, 11, 16
Heavy Traffic
Now internal traffic
What would you think of…