2. Disclaimer
• Based on N+2 failure
• Focuses on large scale deployment of vCAC infrastructure
• Targets RTO of less than 1 min
• Components like ITBM, vCenter Orchestrator are not discussed
• These slides are aimed to be read rather presented. Therefore lot of texts is
seen
• This is first Draft of the PoV
3. vCAC Appliance
• A pre-configured virtual appliance
• Provides single portal for
– self-service provisioning
– Management of cloud services
– Authoring
– Administration
• Comes with embedded PostgreSQL database
• Recommended Hardware Configurations
Server Role CPU Memory (GB) Storage (GB)
vCAC Appliance 2 8 30
4. vCAC - High Availability & Load Balancing
• Scaling of vCAC web component is achieved using multiple instance of
vCAC appliance.
• The traffic entering vCAC appliance will be shared using load balancer
• Since Database is heart of vCAC appliance, PostgreSQL will be installed on
separate server
• To make database highly available, PostgreSQL DB will be installed on
multiple servers and DB can be clustered
• Recommended Hardware Configurations of PostgreSQL Server
Server Role vCPU RAM (GB) Storage (GB)
PostgreSQL 2 4 20
7. IaaS Website
• A Windows Instance, provides access to infrastructure capabilities in the
vCAC web console
• IaaS Website uses a MS SQL Server DB to maintain information about the
machines it manages and its own elements and policies.
• The machine that hosts first IaaS website node must also host the model
manager data component.
• Recommended Hardware Configurations (without Database)
Server Role CPU Memory (GB) Storage (GB)
IaaS Website 2 4 40
8. IaaS Website - High Availability & Load Balancing
• Scaling of IaaS website will be achieved using 3 instance of IaaS server.
• These 3 instance will be load balanced using 3rd party load balancer
• Since Database is heart of IaaS website, install MS SQL Server on separate
server
• Database will be made highly available (near zero downtime), By installing
MS SQL Server on two nodes and Failover cluster will be configured to
protect DB
• Recommended Hardware Configurations
Server Role vCPU RAM (GB) Storage (GB)
MS SQL Server 8 16 80
NB: Model Manager Data Component (MMDC) is needed to be installed only once and to put MMDC
behind load balancer you must disable loop backup check
10. Manager Service
• Co-ordinates communication between agents (including proxy agents), the
Database and SMTP
• To enable High availability, three VMs will host 3 instances of manager
services. Two manager service cannot be active at same time. On Node2
(passive node02) and Node3 (passive node03) manager service will be
stopped & disabled. Failover is manual process anywhere between 1-10
minutes are needed to bring service back online.
• Concurrent Data collection per agent by default is limited to two. Data
collection time will be observed for initial month and later either CPU or
agent limit will be increased.
• Data collection is CPU intensive activity. If High CPU utilization is observed
it will be compensated by increasing cores for the VM
11. Distributed Execution Manager
• DEM executes business logic of custom models by interacting with vCAC
DB and external databases and systems as required
• DEMs are used for the provisioning and management of virtual, cloud, and
physical machines on:
– Red Hat Enterprise Virtualization (RHEV) Manager
– Microsoft System Center Virtual Machine Manager (SCVMM)
– Amazon and VMware vCloud Director
– Dell, HP, and Cisco
• DEMs and agents enable vCAC to communicate with external systems for
provisioning and managing machines
12. Distributed Execution Manager Orchestrator
• Each DEM instance performs two roles
– Orchestrator
– Worker: Worker is responsible for execution of workflows
• DEM Orchestrator Monitors status of DEM worker.
• If DEM worker loses connectivity with model manager or stops, DEM
orchestrator puts workflows in the queue of another DEM worker’s to
execute.
• DEM Orchestrator manages scheduled workflow by creating new workflow
instances at the scheduled time.
• DEM Orchestrator also ensures only one instance of scheduled workflow is
running at any given time.
• Pre-processes workflow before executing, including checking if
preconditions for workflows are met.
• Creates workflow execution history.
13. DEM Orchestrator High Availability
Considerations
• Orchestrator DEMs support active-active (semi-active) high availability.
• When a DEM Orchestrator starts, it searches for another running DEM
Orchestrator. If none is found, it starts executing as the primary DEM
Orchestrator.
• If it does find another running DEM Orchestrator, it goes into a passive
mode and monitors the other primary DEM Orchestrator to detect an
outage.
• If it detects an outage, it takes over as the primary. When the previous
primary comes back online, it detects that another DEM Orchestrator has
taken over its role as primary and goes into a passive state.
• Hardware Recommendations DEM Orchestrator & Manager Service
High availability behavior is little bit similar to vSphere HA master behavior especially the failure detection
Server Role vCPU RAM (GB) Storage (GB)
Manager Service &
DEM Orchestrator
4 6 80
15. DEM Worker High Availability Considerations
• If a DEM Worker instance fails, the DEM Orchestrator detects the failure
and cancels any workflows being executed by the DEM Worker instance.
• When the DEM Worker instance comes back online, it detects that the DEM
Orchestrator has canceled the workflows of the instance and stops
executing them.
• To prevent workflows from being canceled prematurely, a DEM Worker
instance must be offline for several minutes before its workflows can be
cancelled.
16. Agents
• vCAC uses agents to integrate with external systems
• Types of Agents
– Hypervisor Proxy Agents for
• vSphere
• Citrix Xenservers
• Hyper-v Servers
• WMI Agents
– External Provisioning EPI Agents allows you to run VB script as extra
step during Guest OS provisioning
– WMI Agents
• The vCAC WMI agent collects data from windows VMs managed by
vCAC
• vCAC uses these agents to send command and collect data from ESX
server, XenServer and Hyper-V and Provisioned VMs.
17. Agent (cont.)
• Requires administrative access to hypervisors
• Communicates with manager service
• Is installed separately with its configuration file
• During installation you have option to install Only vSphere agent
• You can scale the solution by installing as many agents as you want for
additional throughput & concurrency
Server Role vCPU RAM (GB) Storage (GB)
Agents & DEM Worker 2 4 80
Hardware Recommendations DEM Worker & Agents
19. Design Decisions
• DD01 –DEM Orchestrator and Manager services will be installed on same host.
As these two components needs to have reliable network connectivity. Option to
host them on dedicate VM is ruled out to avoid spread of server estate and
potential increase management cost.
• DD02 – 3 VMs hosting both Manager Service & DEM Orchestrator will be
placed across three different hosts. This will allow protection against ESX node
failures. In worst case when more than one node fails, we still have at least one
DEM orchestrator running. 2 instances of manager service and DEM
Orchestrator will be idle under any given time.
• DD03: Virtual Machines to Hosts should rule be applied to keep each instance
of DEM Orchestrator & Manager and Worker & Agent VM on same host. This
will allow all three VMs to stick to individual host and in case of failure of more
than 2 ESX host, it will ensure at least 3 VMs are available at ESX host at any
given time.
• DD03 – DEM worker and Agent will be placed on same VM. Both these
components can be scaled up (DEM worker) and scale out (using multiple VMs).
Primary reason to keep both the components same VM is that these two
components can be scale out and scale up as required
• DD04 – DEM Orchestrator, DEM Worker, Agent, Manager services, vCAC
Appliance, Identity Appliance, ITBM, vCenter Orchestrator will be using single
VLAN for better network connectivity among the components
20. High Availability and Scalability
Component High Availability Downtime VM
Instances
vCAC Web Component vSphere HA + Load Balancer Zero 3
IaaS Web Component vSphere HA + Load Balancer Zero 3
Manager Service vSphere HA + Load Balancer 1-10 Min
3
DEM Orchestrator vSphere HA + In-built Application
Clustering
Zero
DEM Worker vSphere HA + In-built Application
Clustering
Zero
3
Agents vSphere HA + In-built Application
Clustering
Zero
Identity Appliance vSphere HA 1-10 Min 1
MS SQL Server vSphere HA + Failover Clustering Near Zero 2
PostgresSQL vSphere HA + Clustering Near Zero 2
vCloud Application Director
(vAppD)
vSphere HA 1-10 Min 1
ITBM vSphere HA 1-10 Min 1
21. Notes on High Availability and Scalability
• vSphere HA provides protection against ESX failover
• Load balancer provides protection against VM and application failover
• Since Manager service acts in Active-Passive configuration only, therefore
there is manual downtime between 1-10 mins involved
• Identity Appliance, vCloud AppD and ITBM are stand-alone application and
will be protected using vSphere HA
• MS SQL Server and PostgreSQL server will be protected using application
based clustering
• Near Zero time represents the time needed to failover Database from one
node to another
• Except for Manager services, all components of vCAC and IaaS can be
scale out or scaled up. Scaled out decision will be based on CPU utilization.
22. Additional Information
• Each DEM Worker instance can process 15 concurrent workflows. Excess
workflows are queued for execution.
• Increase in processing time will directly impact the number of concurrent
workflows that can be executed
• Review the number of pending workflows, or if workflows execution time, If
either of the thing is observed, more DEM Worker instances can be added
to pick up the workflows. CPU utilization should also be observed to make
decision either to add more DEM workers or add additional DEM machines
• Some workflows, particularly certain custom workflows, can be very CPU-
intensive. If the CPU load on the DEM Worker machines is high, consider
increasing the processing power of the DEM machine or adding more DEM
machines to your environment.
• You can use the Workflow History page to determine how long it takes to
execute a given workflow.
• The average workflow processing time (from when the DEM Orchestrator
starts preprocessing the workflow to when the workflow finishes executing)
increases with the number of concurrent workflows.
23. Additional Information (cont.)
• Do not deploy vCAC Appliance, vAPPD or ITBM over a WAN as
performance will be degraded.
• DEM Worker and Agent can be deployed in WAN
• Best performance VMware recommends installing DEMs and agents on
different machine
• VMware recommends using external vCenter Orchestrator for each tenant
to enforce multi tenancy
• When deploying vCloud Application Director you can use the vCloud
Automation Center Single Sign On (SSO) capability to manage users in one
place
24. vCAC Scalability –VMware Recommendation
Components Large Deployment
VM QTY
Medium Deployment
VM QTY
Identity Virtual Appliance 1 1
vPostgreSQL Appliance 1 1
vCAC Appliance (web component)* 2 2
IaaS Server (web component)* 2
2#
IaaS Manager Service* 2
IaaS DEM Server 2 2
IaaS Agent Server 2 2
MS SQL Server (2 node failover cluster) 2 2
Items Large Deployment Medium Deployment
Managed Machines 50,000 10,000
Catalog Items 2,500 2,500
Concurrent Deployments 100 50
• Must be placed behind the load balancer
• # Both roles (Manager service and Web components) are placed on same VMs
25. References
• Reference Architecture –vCloud Automation Center 6.0
• Hany Michael -Reference Architecture – vCloud Automation Center (vCAC)
6.0 Distributed Environment – Part 1 of 2