Enhanced introduction to CloudStack for the Geneva Java User Group. Includes Apache processes, DevCloud use cases, opportunities for Java developers and 4.0 release testing procedure.

1. Apache CloudStack
Sebastien Goasguen
GenevaJUG, Oct 30th
Geneve, Suisse

2. Info
• Apache incubator project
• http://www.cloudstack.org
• http://incubator.apache.org/cloudstack/
• #cloudstack on irc.freenode.net
• @cloudstack on Twitter
• http://www.slideshare.net/cloudstack
• http://cloudstack.org/discuss/mailing-lists.html
Welcoming contributions and feedback, Join the fun !

3. A bit of History
• Original company VMOPs (2008)
– Founded by Sheng Liang former lead dev on JVM
• Open source (GPLv3) as CloudStack
• Acquired by Citrix (July 2011)
• Relicensed under ASL v2 April 3, 2012
• Accepted as Apache Incubating Project April
16, 2012
• First Apache (ACS 4.0) release expected Sept
26th

4. Apache Software Foundation

5. Apache Process
• 100% community driven
• New ideas, decisions only taken on mailing lists.
Votes taken by community
• Project led by Project Management Committee
(PMC):
– http://www.apache.org/dev/pmc.html
– http://incubator.apache.org/guides/committer.html
• Non committers get invited as committers:
– http://community.apache.org/newcommitter.html

6. Apache Processes

7. First Release almost out
• Apache CloudStack 4.0
• Check out the testing procedure:
– https://cwiki.apache.org/CLOUDSTACK/cloudstack
-40-test-procedure.html
• Or watch my amazing screencast:
– http://vimeo.com/52150218

8. Contributions from outside Citrix
Examples:
Sungard: Announced that
6 developers were joining
the Apache project
Schuberg Philis: Big
contribution in
building/packaging and
Nicira support
Go Daddy: Early proto of
Maven building
Caringo: Support for own
object store

9. CloudStack and Citrix
• CloudPlatform powered
by CloudStack
• Feature parity, moving
to CloudStack being
upstream
• Enterprise support
• Enterprise training
• Main contributor to
Apache CloudStack, up
to now

10. Why make it Open Source ?
• Large partner ecosystem needed a way to integrate
quickly
• Users drive adoption in early markets – providing
shortest path to adoption is open source
• Open source communities are driving integrations
and standards
• Faster time to market, short feedback loop from
user community
• Enterprise ready and wanted an open source
solution for customers to develop on.

11. What Does it do ?
• Open source Infrastructure as a Service (IaaS)
platform.
• Data Center orchestrator
• Hypervisor agnostic (with addition of bare
metal provisioning)
• Support complex enterprise networking (e.g
Firewall, load balancer, VPN, VPC…)
• Multi-tenant

12. Other OSS IaaS Players

13. A Very Flexible IaaS Platform
Compute Hypervisor
Storage Block & Object
Primary Storage Secondary Storage
Network Network & Network Services
http://www.slideshare.net/cloudstack/cloudstack-architecture

14. Architecture / Language
• Java application
• Tomcat6, Axis2, Maven build + ant
– Ant going away in 4.1
• Moving towards a plugin architecture
– Cocoon ? Spring ?
• Collaboration Conference, Nov 30th -Dec 2nd
– http://collab12.cloudstack.org/

15. Build and Run in 4.1
• git clone https://git-wip-
us.apache.org/repos/asf/incubat
or-cloudstack.git
• mvn clean
• mvn install
• mvn –P developer –pl developer
–Ddeploydb
• mvn –pl :cloud-client-ui
jetty:run

16. Cloud Interactions OVM Cluster Primary
Storage
Ap i
Xen vcenter
vcenter
Monitoring Primary
CS API Clu I
ste AP vSphere Cluster
Storage
r M n ter
End
End
gm vCe
t
User UI
User UI
Primary
XS Cluster Storage
Admin
Admin Clustered
Clustered
CloudStack XAPI
UI
UI CloudStack
Domain
Domain
CS Admin & CloudStack
CloudStack
CloudStack
CloudStack
End-user API Primary
Admin
Admin
UI
UI
Management
Management JSON KVM Cluster Storage
Server
Server NetConf
Juniper SRX
Cloud user Nitro API
{API client (Fog/etc)} VNC
JSON
ec2 API JSON Netscaler
Cloud user Console
Console
Console
{ec2 API client } ProxyConsole
VM
Proxy VM
Proxy VM
Proxy VM NFS
MySQL
MySQL Server
Sec. Storage
Sec. Storage NFS NFS
{Proxied} SSH Sec. VM
Storage
Sec. Storage
VM
Ajax
Ajax HTTPS VM
VM
Console
Console Router VM
Router VM HTTP (Template Download)
Router VM
Router VM HTTP (Template Copy)
Router VM
Router VM
Cloud user HTTP (Swift)
http://www.slideshare.net/cloudstack/cloudstack-architecture

17. Terminology
Zone: Availability zone,
aka Regions. Could be
worldwide. Different data
centers
Pods: Racks or aisles in a
data center
Clusters: Group of
machines with a common
type of Hypervisor
Host: A Single server
Primary Storage: Shared
storage across a cluster
Secondary Storage:
Shared storage in a single
Zone

18. Storage
• Primary Storage:
– Anything that can be mounted on the node of a cluster.
– Cluster LVM…iSCSI…
– Holds disk images of running VMs
– Support for CEPH with KVM hypervisors
• Secondary Storage:
– Available across the zone
– Holds snapshots and templates (image repo)
– Can use Openstack swift or any object store (Gluster FS…)
– New support for Caringo
• Can use NFS for both to start
• Storage Abstraction refactoring underway

19. Networking
• Extremely flexible to:
– Provide isolation with VLANs
– Provide isolation at L3 with shared L2 (scalability)
– Support hardware devices that exposes API
– Deployed on existing networking infrastructure
– Support new networking paradigm (SDN)
• Support for Nicira Virtual P
• Extensive use of Open VSwitch

20. Physical Network
Operations
Users
Admin and
Cloud API
CloudStack
Mgmt Server
Cluster Router
MySQL
Load Balancer
Availability Zone
L3 Core Switch
Access
Layer
Switches
…
Secondary
Servers
Storage
… … … …
Pod 1 Pod 2 Pod 3 Pod N
Slide from Chiradeep Vittal, http://www.slideshare.net/cloudstack/cloudstack-networking

21. Layer-2 Guest Virtual Network
CS Virtual Router provides Network Services External Devices provide Network Services
Network Hardware exposing API can be controlled
Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8
VLAN 100 VLAN 100
Public Public
Network/Inter Network/Inter
net Guest net Guest
Guest
Guest Private IP
VM 11 Public IP 10.1.1.1 VM 11
10.1.1.1 VM 10.1.1.111 VM
Gateway 65.37.141.11 Juniper
Juniper
Public IP 1 SRX
SRX
address
65.37.141.11 CS Firewall
Firewall
CS 10.1.1.1 Guest
Guest Guest
Guest
Virtual
Virtual
10.1.1.3 VM 22
VM 10.1.1.3 VM 22
VM
Router
Router
Public IP Private IP
DHCP, DNS 65.37.141. NetScaler 10.1.1.112
NetScaler
NAT Guest 112 Load Guest
Guest Load Guest
Load Balancing 10.1.1.4 VM 33 Blancer VM 33
VM Blancer 10.1.1.4 VM
VPN
Guest
Guest Guest
Guest
10.1.1.5 VM 44
VM 10.1.1.5 VM 44
VM
CS
CS
DHCP, Virtual
Virtual
Router
Router
DNS
Slide from Chiradeep Vittal, http://www.slideshare.net/cloudstack/cloudstack-networking

22. L3 isolation
• To go beyond the limitation of VLANs (4096
vlans per switch)
• Move to L3 isolation
– Shared L2
– Manage one firewall per VM
• Potential use of GRE tunnels to create single
L2 overlay
• GRE in tech preview for site to site VPN.

23. L3 isolation with distributed firewalls
Tenant 10.1.0.2
Public Public IP address
1 VM 1
Internet 65.37.141.11
65.37.141.24
65.37.141.36 10.1.0.1
Pod 1 L2 Tenant 10.1.0.3
65.37.141.80 Switch 2 VM 1
Tenant 10.1.0.4
1 VM 2
L3 Core
Pod 2 L2
Switch
10.1.8.1
…
Tenant 10.1.16.12
10.1.16.1 2 VM 2
Load Pod 3 L2
Balancer Switch
Tenant
2 VM 3 10.1.16.21
… Tenant
1 VM 3
10.1.16.47
Tenant
10.1.16.85
1 VM 4
Slide from Chiradeep Vittal

24. A customizable GUI
AJAX + API

25. A very extensive API

26. API
• Not really REST
• A set of methods available over http(s)
• Unauthenticated on integration port
• Authenticated on 8080 using Access and
Secret Key
• Python/Ruby clients available
• Internal Marvin client
• CLI under development
• Other clouds client support the API

27. Making API calls
Can be authenticated or not.
HTTP call.
User Keys can be generated via the GUI
Base url: http://<manager-host>:8080/client/api?….
def make_request(requests, secretKey):
request = zip(requests.keys(), requests.values())
request.sort(key=lambda x: str.lower(x[0]))
requestUrl = "&".join(["=".join([r[0], urllib.quote_plus(str(r[1]))]) for r in request])
hashStr = "&".join(["=".join([str.lower(r[0]),
str.lower(urllib.quote_plus(str(r[1]))).replace("+", "%20")]) for r in request])
sig = urllib.quote_plus(base64.encodestring(hmac.new(secretKey, hashStr,
hashlib.sha1).digest()).strip())
print "Signature: %s"%sig
requestUrl += "&signature=%s"%sig
print requestUrl
if __name__ == '__main__':
requests = {
"apiKey": "BRZ5j4E8O4di2MZWnQsYBLThCrTGO-LGeZaMjsnvelkHuY5P8FdTnluNZTDQhCUy-
wqeJzk8EAc_NbcZxTF_FA",
"response" : "json",
"command" : "listZones"
}
secretKey = "bFlx2llt3OmM4AiHzfwV1ZbuJ5tsv6hAx6IeM32CkM-obCA77BRwBr3_yQ0bO1-kdZyfD3-lY6khsXCx18n3Mw"
make_request(requests, secretKey)

28. EC2 /S3 compatibility
• Significant development work happening to make
Cloudstack highly compatible with EC2 /S3 API.
• http://wiki.cloudstack.org/display/RelOps/EC2+API
+support+in+CloudStack
• http://www.slideshare.net/sebastiengoasguen/clou
dstack-ec2-configuration
• Euca tools, boto etc…should work with cloudstack.

29. Enabling EC2 and S3
• Via the GUI
• Via API call on integration API port 8096
http://localhost:8096/client/api?
command=updateConfiguration&name=enable.s3.api&value=true
http://localhost:8096/client/api?
command=updateConfiguration&name=enable.ec2.api&value=true

30. Highly Scalable
• See:
– http://www.slideshare.net/cloudstack/scalability-
12819428
– From Alex Huang. 10k “resources” managed per
Mgt server. 30k resources with 30k VM in
simulation.
• Management server can be setup in a multi-
node configuration with a load-balancer and
replicated MySQL.

31. SandBox: DevCloud
• A Virtual box appliance packaged to provide a
working CloudStack environment.
• Aimed at developers but has other use cases:
– Xen PV hosts gives nested virtualization
– Local EC2/S3 Cloud on your laptop
– Networking experiments ?
• http://wiki.cloudstack.org/display/comm/Dev
Cloud

32. DevCloud: self-contained
CloudStack runs in the appliance

33. Testing “4.0” code in DevCloud
• Deploy new CloudStack code in self-contained
DevCloud:
• mvn –P deps
• ant rdeploy
• ant rdeploydb
– Wipes database of mgt server, you will need to
reconfigure the “data center”
• ant rdebug

34. DevCloud: as Host
Run CloudStack on local machine
Use DevCloud to setup hosts

35. DevCloud to test 4.1 branch
• mvn -P developer clean
• mvn -P developer install
• mvn -P developer -pl developer –Ddeploydb
• mvn -P developer -pl tools/devcloud –
Ddeploydb
• mvn -pl :cloud-client-ui jetty:run
Configure infrastructure:
• mvn -P developer,deploysvr -pl
tools/devcloud -Ddeploysvr

36. Testing Framework –for the PyUG 
[environment]
• Marvin is a Python dns=10.147.28.6
mshost=10.147.39.69
based framework to run mysql.host=10.147.39.69
tests against a
[cloudstack]
private.gateway=10.147.40.1
CloudStack install private.pod.startip=10.147.41.121
private.pod.endip=10.147.41.160
• Could be used as a private.netmask=255.255.254.0
public.gateway=10.147.40.1
simulator of a public.vlan.startip=10.147.41.162
datacenter
public.vlan.endip=10.147.41.200
public.netmask=255.255.254.0
• Used to configure an
hypervisor=XenServer
host=10.147.40.10
infrastructure on a mgt host.password=password
#storage pools
server primary.pool=nfs://10.147.28.7:/expo
rt/home/automation/sadhu/primary
secondary.pool=nfs://10.147.28.6:/ex
port/home/automation/sadhu/secondary

37. Opportunities for Java developers
• Improve Maven build
• Move to Spring framework
• True REST API
• Plugin framework with better
componentization – Cocoon ?
• Junit tests
• Support for Hadoop as storage backend
• …

38. Info
• Apache incubator project
• http://www.cloudstack.org
• #cloudstack on irc.freenode.net
• @cloudstack on Twitter
• http://www.slideshare.net/cloudstack
• http://cloudstack.org/discuss/mailing-lists.html
Welcoming contributions and feedback, Join the fun !