How OpenNebula is being used at BIT.nl.

1.
OpenNebula @BIT
Stefan Kooman (UNIX engineer)

2.
(A) BIT (MORE)
● BIT is a business to business internet service
provider specialized in colocation and managed
hosting
● BIT delivers to quality aware customers the
backbone of their IT and internet infrastructure
● Reliability is the focus of BIT’s services
(redundancy is keyword (n+1))

3.
● No costs upfront for startups (no need to buy iron, support contracts, hardware
management, etc.)
● Short time­to­deliver
● Flexible (grow / shrink storage, VM capacity, # VM's)
● Infrastructure as a Service (IaaS)
Why choose “BIT” instead of $PUBLIC_CLOUD?
● Customers want to know were their data gets stored (compliancy)
● Alternative for shared hosting (customers with special requirements)
● Hybrid Solutions: bare metal & BIT VM's possible
● ISO 27001
Why we do have a “cloud” @ BIT

4.
Project Requirements
● Remote console / control (stop, start, reboot)
● Customer should be able to change VM
capacity (resource pool aka Virtual DataCenter)
● VM speed should be on par with bare metal
● API (provisioning, billing, integration)

5.
Why did we choose OpenNebula?
● Simple but powerful / flexible
● Works out of the box
● Easy to maintain / upgrade
● (API) Interface(s)
● OSS
● Great community / development organization (C12G)

6.
Customer Cloud setup @BIT
● Ubuntu with KVM
● Openvswitch (ready for SDN)
● Cisco Switches
● NetApp MetroCluster Storage (NFS)
● Fujitsu servers

7.
Customer Cloud setup @BIT
● Oned: Active – standby “OpenNebula” (manual fail­
over) located in seperate DataCenters
(mysql master­master repl., Pacemaker/Corosync)
● Apache with Phusion Passenger (Ruby app server)
● Load­balanced “front­end” (Sunstone, OCCI)
● Cfengine2 configuration Management

8.
(Particular) Nice OpenNebula
Features
● Hooks
● Multiple Datastores functionality
● Requirements on VM provisioning (filtering)
● Virtual Data Centers (VDC) in ONE 4.6

9.
What do we use hooks for?
● XML­RPC call to BIT Administration &
Configuration Engine (BACE)
● Live migrate VM's (on hypervisors) as well as in
BACE (near­realtime)

10.
Multiple Datastores
● NetApp Qtree (NFS)
→ Provide seperation (“partitioning”)
→ Billing (IOPS / Disk Space)
→ Tiering possible (SAS, SSD/SATA)
● Future?
→ Distributed Object Storage (CEPH, Gluster)

11.
Multiple datastores (continued)
Datastore seperation (security, performance)
● Production datastores @filer1, volume1
→ Customer A
→ Customer B
● Test / Development @filer2, volume2
→ Test VM's

12.
Provisioning Requirements
(filtering)
● DATASTORES
● CLUSTERS
● HOSTS
→ Custom Attributes (awesome o/)
Examples:
SCHED_REQUIREMENTS="WINDOWSLICENSED="TRUE""
SCHED_REQUIREMENTS="DATACENTER=”BIT­1”"
SCHED_DS_REQUIREMENTS="NAME=system_ds_1_kvm_cluster"

13.
Virtual Data Center
● Virtual Resource abstraction
● Pool of resources, flexibility for end­users
● Possibilities for Resellers

14.
Integration with BIT: provisioning
Command Line Tool
● Vmadm
For now funtcionality is limited:
→ stop, start, reboot, list
Future:
→ full fledged provisioning:
create, destroy, clone, resize capacity, etc.

15.
Integration with BIT: provisioning
Command Line Tool (interfaces with OCCI)
● Vmadm
For now only:
→ stop, start, reboot, list
Future:
→ full fledged provisioning
create, destroy, clone, resize capacity, etc.

16.
Integration with BIT: Portal
Customer Portal Interface to securely manage all services
(DNS, MAIL, VM's, MONITORING, BILLING, etc.)
● For now only:
→ stop, start, reboot
→ Out of band management: Console access (KVM)
Future:
→ full fledged provisioning
create, destroy, clone, resize capacity, etc.

17.
Integration with BIT:
Developer perspective
So far, all good news!!!
● Good documentation (and lot's of it)
● Hooks (again :))

18.
Virtualization @BIT (currently)
Currently
● Ad hoc managed KVM hypervisors (virt­
manager, virsh)
● VM creation process relatively labour intensive
● Live migration of VM's takes a long time (but
could be scripted / automated)
● No Integration with BACE

19.
Virtualization @BIT (Future)
OpenNebula Cloud dedicated for BIT services
Benefits:
● Integration with BACE
● New features available to do cool stuff
→ OneFlow
→ CloudFederation (BIT­3, Partner DC)
→ Internal AppMarket (ready to go BIT modified vms to play /
test /develop)
→ CCS / R&D self service (no need to ask the UNIX engineers)

20.
Virtualization @BIT (Future)
OpenNebula Cloud dedicated for BIT services
Benefits:
● Integration with BACE
● New features available to do cool stuff
→ OneFlow
→ CloudFederation (BIT­3, Partner DC)
→ Internal AppMarket (ready to go BIT modified vms to play /
test /develop)
→ CCS / R&D self service (no need to ask the UNIX engineers)

21.
Challenges
● Network Integration (SDN) (ip­allocations, vlan
provisioning, routing, firewalling, load­
balancing, etc.)
● DNS automation (PTR's, Forwards)
→ Probably going to solve with hooks
● Snapshot integration (NetApp)

22.
BIT and OpenNebula
What you might expect from us
● More feature requests ;)
● More bug reports (hopefully not ;))
● Support on mailinglist
● Translate Sunstone into Dutch (to be completed in two
days from now)
● Zerocopy driver functionality (NetApp)
→ faster provisioning, less host IO