1. Apache CloudStack
Architecture
Alex Huang
Software Architect, Citrix Systems

2. Deployment Architecture

3. Components
• Hosts
• Servers onto which services will be provisioned VM
• Primary Storage Host
• VM disk storage Network
VM
• Cluster Host
• A grouping of hosts and their associated storage
Primary
• Pod Storage
• Collection of clusters in the same failure boundary
• Network Cluster
• Logical network associated with service offerings
Secondary
Cluster
• Secondary Storage Storage
• Template, snapshot and ISO storage
CloudStack Pod
• Zone
• Collection of pods, network offerings and
secondary storage CloudStack Pod
• Management Server Farm
• Management and provisioning tasks Zone

4. Two Types of Storage
Primary Storage
• Stores disk volumes for VMs in a cluster
• Configured at Cluster-level.
• Close to hosts for better performance L3 switch
• Cluster have at least one primary storage
• Requires high IOPs (can be expensive)
Pod 1 L2 switch
Secondary
Cluster 1 Storage
Host 1
Primary
Secondary Storage Host 2 Storage
• Stores all Templates, ISOs and Snapshots
• Configured at Zone-level
• Zone can have one or more secondary
storages
• High capacity, low cost commodity
storage

5. Deployment Architecture
Internet  Hypervisor is the basic unit
Management
Server Cluster of scale.
Zone 1  Cluster consists of one ore
more hosts of same
L3 hypervisor
Pod 1 Pod N
 All hosts in cluster have
L2 Secondary
access to shared (primary)
…. Storage
storage
Cluster N
 Pod is one or more clusters,
usually with L2 switches.
….
 Availability Zone has one or
Cluster 1 more pods, has access to
secondary storage.
Host 1
Primary
 One or more zones
Host 2 Storage
represent cloud

6. Management Server Cluster
 MS is stateless. MS can
be deployed as physical
server or VM
MySQL
User API Management  Single MS node can
Server manage up to 10K hosts.
Load
Balancer Replication Multiple nodes can be
Admin API Management deployed for scale or
Server redundancy
Replica
 RHEL 5.4+, Ubuntu
10.0.4, Fedora 16
Infrastructure
Resources

7. Managing Complexity

8. The Three C’s of Complexity
• Control
• Choice
• Compliance

9. Giving Control Brings Complexity
Org A Org B
• ACL
Users • Limits
Admin Admin
• Governance
End User Users Users
Admin Compute Network Storage
VM Net Local
Hyper-V Cisco ASA NFS
Ware Scaler Disk
Xen Oracle F5 iSCSI Swift
Server VM
Bare Jun. SRX FC HDFS
KVM
Metal

10. Guest Virtual Layer-2 Network
Guest 1 Virtual Network
10.1.1.0/24
Guest 1 Guest
VM 1 10.1.1.2
Public IP Guest 1 Gateway Guest 1 Guest
65.37.141.11 Virtual 10.1.1.1 VM 2 10.1.1.3
65.37.141.36 Router
Guest 1 Guest
VM 3 10.1.1.4
Internet
Guest 2 Guest
VM 1 10.1.1.2
Public IP Guest 2 Gateway Guest 2 Guest
65.37.141.24 Virtual 10.1.1.1 VM 2 10.1.1.3
65.37.141.80 Router
Guest 2 Guest
VM 3 10.1.1.4
Guest 2 Virtual Network
10.1.1.0/24

11. Multi-tier Network
Virtual Network Virtual Network
Virtual Network 10.1.2.0/24 10.1.3.0/24
10.1.1.0/24 VLAN 1001 VLAN 141
VLAN 100
App VM
10.1.2.31 1 10.1.3.21
Private IP Web VM
Public IP 10.1.1.1 1 10.1.2.21
Juniper 10.1.1.111
65.37.141. SRX App VM
111 Firewall 10.1.2.24 2 10.1.3.45
Web VM
10.1.1.3 2 10.1.2.18
Public IP Private IP
Netscaler 10.1.1.112
65.37.1 Load Web VM
41.112 Balancer 10.1.1.4 3 10.1.2.38 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4 10.1.2.39
DHCP,
Virtual DHCP, Virtual DNS Virtual
DHCP, DNS,
DNS Router User- Router User- Router
data data,
User-
Source
data
-NAT,
VPN Public IP
65.37.141.115

12. Unified Multi-tier Network
Internet
IPSec or SSL site-to-site VPN
Virtual Router Customer
Load Balancer Premises
Monitoring VLAN
Virtual Router Services
App
• IPAM 10.1.2.31
Web VM 1
• DNS 10.1.1.1
• LB [intra] VM 1
• S-2-S VPN App
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 VM 2
• NAT, PF
• FW [ingress & egress]
Web DB VM
• BGP 10.1.3.24
10.1.1.4 VM 3 1
Web
10.1.1.5 VM 4
Virtual Network Virtual Network Virtual Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24
VLAN 100 VLAN 1001 VLAN 141

13. Other Topologies
Dedicated VLAN with DHCP and DNS
No services [Static IPs]
User can request specific IP[s] for NIC
Guest Virtual Network Guest Virtual Network
10.1.1.0/24 10.1.1.0/24
VLAN 100 VLAN 100
Guest Guest
VM 1 10.1.1.1 VM 1
10.1.1.1
Gateway address
10.1.1.1
Guest Guest
10.1.1.3 VM 2 Gateway 10.1.1.3 VM 2
address
10.1.1.1
Guest Guest
Core switch 10.1.1.4 VM 3 VM 3
10.1.1.4
Guest
Core switch Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
Virtual
DHCP,
DNS
Router
User-data

14. Other Topologies
MPLS Use Case Shared VLAN with DHCP and DNS
Guest Virtual Network 10.1.1.0/24 Guest Virtual Network 10.1.1.0/24
VLAN 100 VLAN 100
MPLS VLAN 100 Guest Guest
VM 1 10.1.1.1 VM 1
10.1.1.100
Gateway address
10.1.1.1
Guest Guest
10.1.1.200 VM 2 Gateway 10.1.1.3 VM 2
address
10.1.1.1
Guest Guest
Core switch 10.1.1.101 VM 3 VM 3
10.1.1.4
Guest
Core switch Guest
10.1.1.11 VM 4 10.1.1.5 VM 4
5
CS CS
DHCP, Virtual DHCP, Virtual
Router Router
DNS DNS
User-data User-data

15. Layer 3 Networking (Amazon Style)
Web DB Web
VM VM VM
Web DB
Security Security
Group Group
Web Web DB
VM VM VM
… … …
Web Web
VM VM

16. Software Architecture

17. Cloud Other
UI CLI Clients
Portal
Management Server
REST API
OAM&P API End User API EC2 API Other APIs Pluggable Service API Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking Account Management
Connectors
Template
Access
Services API
DB
Plugin API
Deployment Planning
HA
Orchestration Engine
- Drives long running VM
Services API
Network Gurus
Usage operations
Calculations - Syncs between resources
managed and DB Network Elements
Additional - Generates events
Services
Hypervisor Gurus
Cluster Resource Job Alert & Event Database
Management Management Management Management Access
Message Bus
Event Bus Usage
Server
Resource API
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources

18. Orchestration Engine
• Understands how to orchestrate long running
processes (i.e. VM starts, Snapshot copies,
Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that
it needs

19. Plugins
• Various ways to add more capability to
CloudStack
• Implements clearly defined interfaces
• All operations must be idempotent
• All calls are at transaction boundaries
• Compiles only against the Plugin API module

20. Anatomy of a Plugin
• Can be two jars: server component to
be deployed on management server
and an optional ServerResource
Rest API
component to be deployed co-
- Optional. Required only if needs to expose located with the resource
configuration API to admin.
• Server component can implement
multiple Plugin APIs to add its feature
Plugin API
Implementation
• Can expose its own API through
Pluggable Service so administrators
Data Access Layer
can configure the plugin
• As an example, OVS plugin actually
implements both NetworkGuru and
ServerResource NetworkElement
- Optional. Required if Plugin needs to be co-
located with the resource
- Implements translation layer to talk to
resource
- Communicates with server component via
JSON

21. Plugin Interfaces Available
• NetworkGuru – Implements various network isolation and ip
address technologies
• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and
volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes

22. Separating Data and Control
Cloud  Management Servers
control all
Data Center 1
Data Center 3 resources, both virtual
Managem and physical
ent VR
VR Server
 SSVMs deployed to
CPVM SSVM SSVM CPVM transfer data between
Transfer of zones
Templates,
ISOs,
Snapshots
 CPVMs deployed to
Internet transfer VNC console
Data Center 2 traffic
VR SSVM  VR deployed for traffic
into public internet
CPVM
 Management Server is
never in the data path

23. Sequence Flow for VM Creation Kernel
End User Security User VM VirtualMac Network Storage Network Job
Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate IP
Allocate Volume
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status

24. Sequence Flow for VM Creation
Deploymen Server
User VM VirtualMac Network Storage Network Network Template t
Job Threads Services API Resources
Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner
Start VM
Start User VM
Start VM
Get a Deployment Plan (Host and StoragePool)
Prepare Nics
Reserve resources for Nic
Notify that Nic is about to be started in network
Agent Calls
Prepare Volumes
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result

25. Conclusion

26. Design Goals for CloudStack
• Design for complexity
– Clear interfaces
• Design for scalability
– Separate out data path and control paths
– Design to maximize the use of database connections
• Design against failure
– Provide clear boundaries (process and compilation)
– Utilize cloud administrator to give guidance

27. More Information
• http://cloudstack.org
• Apache mailing lists
– cloudstack-users-subscribe@apache.incubator.org
– cloudstack-dev-subscribe@apache.incubator.org
• Thank you
27