1. EC2 API Layer CloudStack
Access Control
Virtual Machine Manager
Console Proxy Manager
Async Job Manager
Network Manager
Template Manager
Snapshot Manager
Storage Manager
…
Agent Manager
XenServer KVM vSphere SRX F5 NetScaler OVM
Resource Resource Resource Resource Resource Resource Resource

2. Cloud Other
UI CLI Clients
Portal
Management Server
REST API
OAM&P API End User API EC2 API Other APIs Pluggable Service API Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking Account Management
Connectors
Template Services API
Access
Deployment Planning
Plugin API
HA
Kernel Job
- Drives long running VM
Queue
Services API
Network Configurations
Usage operations
Calculations - Syncs between resources
managed and DB Network Elements
Additional - Generates events
Services
Hypervisor Gurus
Cluster Resource Job Alert & Event Database
Management Management Management Management Access
DB
Event Bus
Message Bus
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources

3. Kernel Module
• Understands how to orchestrate long running
processes (i.e. VM starts, Snapshot copies,
Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that
it needs

4. Plugins
• Various ways to add more capability to
CloudStack
• Implements clearly defined interfaces
• All operations must be idempotent
• All calls are at transaction boundaries
• Compiles only against the Plugin API module

5. Anatomy of a Plugin
Rest API
- Optional. Required only if needs to expose configuration API to admin. ServerResource
- Optional. Required if
Plugin needs to be co-
located with the
resource
- Implements translation
layer to talk to resource
- Communicates with
Plugin API
Implmentation server component via
JSON
Data Access Layer

6. Anatomy of a Plugin
• Can be two jars: server component to be
deployed on management server and an optional
ServerResource component to be deployed co-
located with the resource
• Server component can implement multiple Plugin
APIs to affect its feature
• Can expose its own API through Pluggable Service
so administrators can configure the plugin
• As an example, OVS plugin actually implements
both NetworkGuru and NetworkElement

7. Plugin Interfaces Available
• NetworkGuru – Implements various network isolation technologies
and ip address technologies
• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and
volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes

8. Adding a Plugin to CloudStack
• Components are configured though
components.xml
• Supports DAO, Manager, and Adapter patterns
• Open to other component frameworks (OSGi a
possibility)

9. Components.xml Example
<components.xml>
<system-integrity-checker class="com.cloud.upgrade.DatabaseUpgradeChecker">
<checker name="ManagementServerNode" class="com.cloud.cluster.ManagementServerNode"/>
<checker name="EncryptionSecretKeyChecker"
class="com.cloud.utils.crypt.EncryptionSecretKeyChecker"/>
<checker name="DatabaseIntegrityChecker" class="com.cloud.upgrade.DatabaseIntegrityChecker"/>
<checker name="DatabaseUpgradeChecker" class="com.cloud.upgrade.PremiumDatabaseUpgradeChecker"/>
</system-integrity-checker>
<interceptor library="com.cloud.configuration.DefaultInterceptorLibrary"/>
<management-server class="com.cloud.server.ManagementServerExtImpl"
library="com.cloud.configuration.PremiumComponentLibrary">
<adapters key="com.cloud.storage.allocator.StoragePoolAllocator">
<adapter name="LocalStorage" class="com.cloud.storage.allocator.LocalStoragePoolAllocator"/>
<adapter name="Storage" class="com.cloud.storage.allocator.FirstFitStoragePoolAllocator"/>
</adapters>
<pluggableservice name="VirtualRouterElementService"
key="com.cloud.network.element.VirtualRouterElementService"
class="com.cloud.network.element.VirtualRouterElement"/>
</management-server>
</components.xml>

10. Sequence Flow for deploy VM Kernel
End User Security User VM VirtualMac Network Storage Network Job
Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate IP
Allocate Volume
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status

11. Sequence Flow for deploy VM
Deploymen Server
User VM VirtualMac Network Storage Network Network Template t
Job Threads Services API Resources
Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner
Start VM
Start User VM
Start VM
Get a Deployment Plan (Host and StoragePool)
Prepare Nics
Reserve resources for Nic
Notify that Nic is about to be started in network
Agent Calls
Prepare Volumes
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result

12. ServerResource
• Translation layer between CloudStack
commands and resource API
• May be Co-located with resource
• Have no access to DB
• API defined in JSON messages

13. DAO
• SQL generation done mostly in GenericDaoBase
• Uses JPA annotations
• Very little code to write for each individual DAO
• Database Access Layer for Kernel
• No support for more complicated features such
as fetch strategy
• Welcome to use other types of ORM in other
modules but like to hear about preferred library.
(Hibernate is out due to licensing issues)

14. Example DAO
// ExampleVO.java // ExampleDao.java
@Entity public interface ExampleDao
@Table(name=“example”) extends GenericDao<ExampleVO, Long> {
public class ExampleVO { }
@Id
@GeneratedValue(strategy= // ExampleDaoImpl.java
GenerationType.IDENTITY) @Local(value=ExampleDao.class)
@Column(name=“id”) public class ExampleDaoImpl
long id; extends GenericDaoBase<ExampleVO, Long>
implements ExampleDao {
@Column(name=“name”)
String name; protected ExampleDaoImpl() {
}
@Column(name=“value”) }
String value;
}

15. Triggering High Availability
VM HA are triggered via the following methods:
• VM Sync detects out of band VM death
• Resource Management detects that a resource is
unreachable and its state can not be determined.
• VM start/stop has been sent to the resource but
resource does not return
• Details of how high availability is done is at
http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/CloudStack_High_Availability_-
_Developer's_Guide

16. High Availability Future
• Moving toward using the native HA capability
of the hypervisor.
• Looking to do more in the DRS area to
coordinate recovery of wide spread outage.

17. VM Sync
• Currently a sync of VM state, not entire VM
• VM Sync happens between management server and hypervisor resources
• Peer-to-peer sync
• Hypervisor DB is considered to be the DB of truth
• Two steps:
– Full Sync
– Intermittent delta sync
• Establishes full sync when first connecting to the hypervisor resource
• After full sync, hypervisor resource keeps track of the last sync results and
only report out of band changes on delta sync
• Utilizes the most abundant resources in data center: CPU and memory
• Conserve the most scarce resource: DB connections
• Virtually no DB connections utilized during delta sync unless there are out
of band changes.

18. Storage
Zone-Level Layer 3 Switch Private Network
Pod 1 Pod 2 Pod N
Pod-Level Layer-2 Switch
…
Scale-Out NFS
Computing Server
1 Primary Storage
Cluster 2
Computing Server
2 Primary Storage
Computing Server
3
Cluster 1
Primary Storage
Computing Server
4

19. Storage
• CloudStack supports two types of storage
– Primary Storage: block device to the VM
– WORM Storage: Secondary or Object Store for
templates, ISO, and snapshot archiving
• Primary storage is high on IOPs (expensive)
• Secondary storage is high on capacity (cheap)
• CloudStack manages the storage between the
two to achieve maximum benefit and
resiliency

20. Disk Offering
• Disk Offering is how disks are offered to the
end user
• Disk Offering has storage tags which can be
used to implementing storage tiering
• Service Offering actually contains a disk
offering for the root disk

21. Snapshots
• Snapshots are used as backups
• Taken on the primary storage and moved to
secondary storage
• Full snapshots on VmWare and KVM. Need
help.
• Incremental snapshots on XenServer

22. XenServer Snapshot
Base Copy
Full1 Full2
D1 Template
S11 S21
D2 S1
S12 S22
VM S2
S13 S23

23. Storage Future
• Secondary Storage multi-homed in Pod
• Object Store for between zones
• Direct Access to Storage for enterprise
• Storage Plugins