This document discusses design considerations for building stretched clusters across multiple sites. Stretched clusters provide high availability across sites but introduce additional complexity. There are two main storage configurations - stretched SAN and distributed virtual storage. Distributed virtual storage using EMC VPLEX provides read/write access at both sites simultaneously but special behaviors like preferred sites must be considered. Key design challenges include controlling VM placement, dealing with single points of failure, and addressing network issues like horseshoe routing. The document recommends separate clusters at each site connected via vMotion instead of a single stretched cluster.

1. Elastic vSphere? Design Considerations for Building Stretched Clusters Scott Lowe, vSpecialist, EMC Corporation VCDX #39, VMware vExpert http://blog.scottlowe.org http://twitter.com/scott_lowe

2. Agenda Reasons for building stretched clusters Storage configurations for stretched clusters Design considerations for stretched clusters EMC VPLEX in stretched clusters Q&A

3. Reasons for Building Stretched Clusters Valid reasons: Provide high availability across sites Workload balancing across sites Invalid reasons: Because you can/because it’s cool (is that a valid business justification?) Enable vMotion over distance (stretched clusters are not a prerequisite) Use vMotion as a DR mechanism (vMotion not applicable when both ends aren’t up and available)

4. Storage Configurations for Stretched Clusters A review of storage configurations to support stretched cluster designs

5. Stretched SAN Configuration Literally just stretching the SAN fabric between locations Requires synchronous replication Limited in distance to ~100km in most cases Typically read/write in one location, read-only in second location Implementations with only a single storage controller at each location create a SPoF (single point of failure)

6. Stretched SAN Configuration X X Read/Write Read-Only

7. Distributed Virtual Storage Configuration Leverages new storage technologies to distribute storage across multiple sites Requires synchronous mirroring Limited in distance to ~100km in most cases Read/write storage in both locations, employs data locality algorithms Typically uses multiple controllers in a scale-out fashion Must address “split brain” scenarios

8. Distributed Virtual Storage Configuration X X Read/Write Read/Write

9. EMC VPLEX Overview EMC VPLEX falls into the distributed virtual storage category Keeps data synchronized between two locations but provides read/write storage simultaneously at both locations Uses scale-out architecture with multiple engines in a cluster and two clusters in a Metro-Plex Supports both EMC and non-EMC arrays behind the VPLEX

10. Preferred Site in VPLEX Metro VPLEX Metro provides read/write storage in two locations at thesame time (AccessAnywhere) In a failure scenario, VPLEX uses “detach rules” to prevent split brain A preferred site is defined on a per-distributed virtual volume basis Preferred site remains read/write;I/O suspended at non-preferred site Invoked only by entire cluster failure, entire site failure, or cluster partition Read/ write Read/ write I/O Suspended Distributed Virtual Volume X IP/FC links for Metro-Plex Preferred Site Non-Preferred Site

11. Design Considerations for Stretched Clusters A review of design considerations and design impacts when using stretched clusters

12. Stretched Cluster Considerations #1 Consideration:Without read/write storage at both sites, roughly half the VMs incur a storage performance penalty. With stretched SAN configurations: VMs running in one site are accessing storage in another site Creates additional latency for every I/O operation With distributed virtual storage configurations: Read/write storage provided, so this doesn’t apply

13. Stretched Cluster Considerations #2 Consideration:Prior to vSphere 4.1, you can’t control HA/DRS behavior. With stretched SAN configurations: Additional latency introduced when VM storage resides in other location Storage vMotion required to remove this latency With distributed virtual storage configurations: Need to keep cluster behaviors in mind Data is access locally due to data locality algorithms

14. Stretched Cluster Considerations #3 Consideration:With vSphere 4.1, you can use DRS host affinity rules to control HA/DRS behavior. With all storage configurations: Doesn’t address HA primary/secondary node selection With stretched SAN configurations: Beware of single-controller implementations Storage latency still present in the event of a controller failure With distributed virtual storage configurations: Plan for cluster failure/cluster partition behaviors

15. Stretched Cluster Considerations #4 Consideration:Host affinity rules don’t affect VMware HA admission control. With all storage configurations: Must configure admission control for 50% failure in order to guarantee resource availability Can’t configure “per site” admission control rules Impacts the reasons people would build stretched clusters, especially workload balancing

16. Stretched Cluster Considerations #5 Consideration:There is no supported way to control VMware HA primary /secondary node selection. With all storage configurations: Limits cluster size to 8 hosts (4 in each site) das.preferredprimaries is not a supported mechanism for controlling primary/secondary node selection Methods for increasing the number of primary nodes also not supported by VMware

17. Stretched Cluster Considerations #6 Consideration:Stretched HA/DRS clusters require Layer 2 adjacency (or equivalent) at the network layer. With all storage configurations: Complicates the network infrastructure Involves technologies like OTV, VPLS/Layer 2 VPNs With stretched SAN configurations: Can’t leverage vMotion at distance without storage latency With distributed virtual storage configurations: Data locality enables vMotion at distance without latency

18. Stretched Cluster Considerations #7 Consideration:The network lacks site awareness, so stretched clusters introduce new networking challenges. With all storage configurations: The movement of VMs from one site to another doesn’t update the network VM movement causes “horseshoe routing” (LISP, a future networking standard, helps address this) You’ll need to use multiple isolation addresses in your VMware HA configuration

19. Horseshoe Routing

20. Stretched Cluster Recommendations Use separate HA/DRS clusters in each datacenter Use separate distributed VMFS datastores for each clusters Use vMotion to move VMs as needed between clusters Keep preferred/non-preferred site behavior in mind! Try to keep related VMs together in a site Change detach rules to switch preferred site if necessary A VMware KB article is available discussing HA/DRS clusters with VPLEX

21. For More Information… VMware support with NetApp MetroCluster:http://kb.vmware.com/kb/1001783 Using VPLEX Metro with VMware HA:http://kb.vmware.com/kb/1026692 vMotion over Distance Support with VPLEX Metro:http://kb.vmware.com/kb/1021215 The Case For and Against Stretched ESX Clusters:http://virtualgeek.typepad.com/virtual_geek/2008/06/the-case-for-an.html

22. Q&A