The document discusses automating the deployment of a VMware vCloud Hybrid Service (vCHS) using Puppet. It describes using Puppet to deploy and configure the various vCHS components through their respective APIs, including vCenter, vShield, and vCloud Director. It addresses challenges in automating these components and proposes treating everything as a native Puppet resource, with a "transport" resource to handle connecting to the different component APIs over SSH or REST.

1. © 2012VMware Inc. All rights reserved
Confidential
VMware vCloud Hybrid Service and Puppet
Nan Liu, Sr. Systems Engineer - VMware Inc

2. 2 Confidential
Overview
§ VMware vCloud Hybrid Service
§ Automation with Puppet
§ Lesson’s learned the hard way

3. 3 Confidential
vCloud Hybrid Service
VMware vCloud
Hybrid Service
Your Data Center Software-Defined
Data Center
VMware vSphere &
vCloud Suite
Existing & New Apps
Seamless Networking
Common management
One Support call

4. 4 Confidential
vCloud Hybrid Service (vCHS)
§ Customer:

5. 5 Confidential
vCloud Hybrid Services
§ Engineer:
-------------- 1 ---------------
Init VPC
- Engine
- API Call
------------- 3 --------------
Install ESXi
Engine Broker Plugin
- Razor
Inputs:
- Razor IP
- Blade
- Hostname
- Network Information
Tasks:
- Create Policy
- Boot Blade
------------- 3 --------------
Prep TLM
Engine Broker Plugin
- Puppet vCenter
Inputs:
- TLM VCSA IP
- TLM VCSA Creds
- TLM DVS
- Portgroup Name
- Portgroup VLAN
- Portgroup Settings
- Ports: 128
- Load Balancing: IP Hash
- Reset at Disconnect: Enabled
Tasks:
- Create PGs
- d#p#v#-dmz-pg-##
- d#p#v#-esx-pg-##
- d#p#v#-pvt-pg-##
----------- 4 ------------
Deploy VPC VCSA
Engine Broker Plugin
- OVFTool
Inputs:
- OVA
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Name
- Network
- VM Hardware Settings
- Memory: 16384
Tasks:
- Deploy VPC VCSA
- d#p#v#-mgmt-vc0
- Update VM Hardware
----------- 4 ------------
Deploy VPC VSE Pub Pair
Engine Broker Plugin
- Puppet vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- Name
- Hostname
- Enable HA
- Declare Dead Time: 6
- Interface 1
- Heartbeat IP Addresses
- TLM Datacenter
- TLM Cluster
- TLM Shared Datastore
- Size: Compact
- Interface Settings
- Interface 0
- Name
- Type
- Portgroup
- Subnet Settings
- Interface 1
- Name
- Type
- Portgroup
- Subnet Settings
Tasks:
- Create VPC Pub Edge
- d#p#v#-mgmt-vse-pub
----------- 4 ------------
Deploy VPC VSE Priv Pair
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- Name
- Hostname
- Enable HA
- Declare Dead Time: 6
- Interface 2
- Heartbeat IP Addresses
- TLM Datacenter
- TLM Cluster
- TLM Shared Datastore
- Size: Compact
- Interface Settings
- Interface 0
- Name: InterVPC
- Type
- Portgroup
- Subnet Settings
- Interface 1
- Name: DMZ
- Type
- Portgroup
- Subnet Settings
- Interface 2
- Name: PVT
- Type
- Portgroup
- Subnet Settings
- Interface 3
- Name: ESX
- Type
- Portgroup
- Subnet Settings
Tasks:
- Create VPC Priv Edge
- d#p#v#-mgmt-vse-priv
----------- 5 ------------
Configure VPC VSE Pub Pair
Engine Broker Plugin
- Puppet vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- DNS Settings
- Firewall Settings
- IPsets
- Application Groups
- Applications
- Firewall Rules
- Load Balancer Settings
- Enable
- Pools
- VIPs
- Default Route
- Syslog
Tasks:
- Configure VPC VSE Pub Pair
----------- 5 ------------
Configure VPC VSE Priv Pair
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings:
- Syslog
- DNS
- Firewall Settings
- IPsets
- Application Groups
- Applications
- Firewall Rules
- Load Balancer Settings
- Enable
- Pools
- VIPs
- Default Route
Tasks:
- Configure VPC VSE Priv Pair
----------- 7 ------------
Init VPC VCSA
Engine Broker Plugin
- Puppet VMware-VCSA
Inputs:
- Credentials
- NTP Settings
- Syslog Settings
- DB Settings
- Sizing: Large
Tasks:
- Init VPC VCSA
----------- 4 ------------
Deploy VPC VSM
Engine Broker Plugin
- OVFTool
Inputs:
- OVA
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Name
- Network
Tasks:
- Deploy VPC VSM
- d#p#v#-mgmt-vsm0
----------- 9 ------------
Configure VPC VSM
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- VPC VSM IP
- VPC VSM Creds
- VPC VCSA IP
- VPC VCSA Creds
- NTP
- Syslog
Tasks:
- Configure VPC VSM
----------- 4 ------------
Deploy VPC vCloud
Engine Broker Plugin
- ??
Inputs:
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Names
- VM Settings
- Network
Tasks:
- Deploy VPC vCloud Cells
- Deploy VPC vCloud NFS
- Deploy VPC vCloud DB
-------------- 2 -----------------
Reserve VPC
Engine Broker Plugin:
- Rez
Inputs:
- VPC #
----------- 8 ------------
Configure VPC vCenter
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP Address
- VPC VCSA Credentials
- Licenses
- vCenter
- ESXi
- vCloud Net & Sec
- Retention Policies
- Task: Enabled
- Event: Enabled
- Datacenter Name
- Cluster Settings
- Name
- DRS Settings
- EVC Settings?
- DVS Configuration
- Name: d#p#v#
- Settings
- Uplinks: 4
- MTU: 9000
- Enable NIOC
- Portgroup settings
- Name
- VLAN
- Settings
-Tasks:
- Configure VPC VCSA
- Configure DVS
----------- 7 ------------
Init VPC VSM
Engine Broker Plugin
- ??
Inputs:
- VPC VSM Network Settings
Tasks:
- Init VPC VSM
----------- 6 ------------
VPC VCSA Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC VCSA IP
Tasks:
- VPC VCSA Port Check
----------- 6 ------------
VPC VSM Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC VSM IP
Tasks:
- VPC VSM Port Check
----------- 6 ------------
Verify VPC ESXi
Engine Broker Plugin
- Puppet Util?
Inputs:
- VPC ESXi IP
- VPC ESXii Creds
Tasks:
- Verify VPC ESXi
----------- 9 ------------
Add ESXi to VPC Datacenter
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC ESXi IP
- VPC ESXii Creds
- VPC VCSA IP
- VPC VCSA Creds
- VPC Datacenter Name
Tasks:
- Add ESXi to VPC Datacenter
----------- 10 ------------
Configure VPC ESXi
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- DNS Settings
- NTP Settings
- Syslog Settings
- VMK Configuration
- vMotion
- Storage
- FT
- ???
Tasks:
- Configure ESXi
- Add to DVS
- Add VMK Networking
----------- 11 ------------
Add Storage to VPC ESXi
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- iSCSI Settings
- LUN Information
Tasks:
- Configure iSCSI
- Add LUNs
------------- 3 ---------------
Configure VNX VPC Storage Group
Engine Broker Plugin:
- Storage Controller
Inputs:
- VPC #
- LUN IDs
- ESXi iSCSI information
Tasks:
- Create VPC Storage Group
- Configure VNX Host registrations
- Add LUNs to VPC Storage Group
----------- 6 ------------
VPC vCloud Cell Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud Cell Port Check
----------- 6 ------------
VPC vCloud Cell Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud Cell Port Check
----------- 6 ------------
VPC vCloud NFS Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud NFS Port Check
----------- 6 ------------
VPC vCloud DB Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud DB Port Check
----------- 7 ------------
Configure VPC vCloud NFS
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud NFS IP
- VPC vCloud NFS Creds
- NFS Export Settings
- ???
Tasks:
- Config VPC vCloud NFS
----------- 7 ------------
Configure VPC vCloud DB
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud DB IP
- VPC vCloud DB Creds
- vCloud Database Config
- ???
Tasks:
- Configure VPC vCloud DB
----------- 8 ------------
Configure VPC vCloud Cell
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud Cell IP
- VPC vCloud Cell Creds
- VPC vCloud NFS Config
- VPC vCloud Installation
Responses.properties
- VPC vCloud Cell Cert
- NTP Settings
Tasks:
- Install and Configure NTP
- Install VPC vCloud Cert
- Configure vCD
- Configure vCD Transfer Service
----------- 9 ------------
Configure VPC vCloud Cell
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud Cell IP
- VPC vCloud Cell Creds
- VPC vCloud Installation
Responses.properties
- VPC vCloud NFS Config
- VPC vCloud Cell Cert
- NTP Settings
Tasks:
- Install and Configure NTP
- Install VPC vCloud Cert
- Configure vCD
- Configure vCD Transfer Service
----------- 12 ------------
Create VPC VM Storage Profile
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- Storage Profile Name
Tasks:
- Create VPC VM Storage Profile
- Tag VPC Datastores
----------- 13 ------------
Add VPC vCenter to vCloud
Engine Broker Plugin
- Puppet VMware-vCloud
Inputs:
- VPC vCloud IP
- VPC vCloud Creds
- VPC VCSA IP
- VPC VCSA Creds
- ???
Tasks:
- Add VPC vCenter to VPC vCloud
----------- 10 ------------
Configure VXLAN
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- VPC VSM IP
- VPC VSM Creds
- Multicast Information
- VPC DVS Information
- VPC Cluster
- ???
Tasks:
- Configure VXLAN
----------- 11 ------------
Reconfigure VXLAN VMK Ports
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- VXLAN VMK Network Settings
Tasks:
- Configure VXLAN VMK Ports
----------- 14 ------------
Configure VPC vCloud
Engine Broker Plugin
- Puppet VMware-vCloud
Inputs:
- VPC vCloud IP
- VPC vCloud Creds
- Provider VDC Name
- External Org Network Name
- ???
Tasks:
- Configure VPC vCloud
----------- 12 ------------
Configure VPC Cluster HA
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- VPC ESXi Hosts
- VPC Cluster
- HA Settings
- Failure: percentage
Tasks:
- Add VPC ESXi hosts to Cluster
- Configure VPC Cluster HA Settings
----------- 5 ------------
Prep VPC vCloud Cell
Engine Broker Plugin
- Puppet vCenter
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC vCloud Cell Name
- Static Routes
Tasks:
- PowerOn VM
- Set Hostname
- Set Static Routes
- Configure Networking
----------- 5 ------------
Prep VPC VCSA
Engine Broker Plugin
- ??
Inputs:
- Credentials
- Network Settings
Tasks:
- Configure Networking
----------- 5 ------------
Prep VPC vCloud Cell
Engine Broker Plugin
- Puppet vCenter
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC vCloud Cell Name
- Static Routes
Tasks:
- PowerOn VM
- Set Hostname
- Set Static Routes
- Configure Networking
----------- 5 ------------
Prep VPC VSM
Engine Broker Plugin
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC VSM name
- Network Settings
Tasks:
- PowerOn VM
- Configure Networking

6. 6 Confidential
vCloud Hybrid Service
§ Start your engine:

7. 7 Confidential
Click Fail
Add VPC Hosts to VCSA
§ Continuing from the vCenter view of the vSphere Web Client
§ In the main content pane, click Related Objects > Clusters
§ Click on VPC on the list
§ Click the second plus icon to add a host
• Host name: Enter first host FQDN
• Location: Confirm the location is set.
• Click NEXT
§ …
§ Repeat Manually?

8. 8 Confidential
Problem
Challenges:
§ Reduce deployment time
§ Reduce complexity
§ Scale … Fast
Solution:
1. Automate
2. See first rule

9. 9 Confidential
vCloud Director

10. 10 Confidential
Infrastructure Services
Server & OS Services (Puppet)
§ NTP
§ Syslog
§ RabbitMQ
§ …

11. 11 Confidential
Automation Challenges
§ Service APIs:
• vCenter: vSphere API (soap)
• vCNS: vShield API (REST)
• vCD: vCD API (REST)
§ vCenter/vCNS servers are appliances

12. 12 Confidential
Why not ‘puppet device’?
§ Limited to one device at a time:
Puppet Management
Server
VMware vCenter
VMware vShield
Datacenter
ESX
ESX
ESX
Datacenter
ESX
ESX
vSphere API
vShield API
Edge

13. 13 Confidential
Solution: Take the road less traveled
§ Everything is a Native Resource
§ Transport to the rescue
• vCenter/vCNS Appliance: SSH
• vCenter API: RbVmomi
• vShield API: rest-client + subset of savon project
• vCD API: rest-client

14. 14 Confidential
Transport Resource
§ Credentials
§ Connectivity Options
§ Multiple connections

15. 15 Confidential
Transport Example

16. 16 Confidential
Transport
Puppet Management
Server
VMware vCenter
SSH
Puppet Management
Server
VMware vCenter
vSphere API
Datacenter
Folders
Datacenter
ESX

17. 17 Confidential
vCenter Appliance Resource

18. 18 Confidential
vCenter Resource

19. 19 Confidential
vCD Resources

20. 20 Confidential
Transport
§ Persistent shared connection
§ Connection cleanup after catalog apply
§ Open to supporting additional transport
• VMware-RabbitMQ (REST)

21. 21 Confidential
Modules
Puppet Enterprise Users (PE 2.7):
§ http://forge.puppetlabs.com/vmware
Puppet Developers (Developing Puppet 3):
§ http://github.com/vmware/vmware-vmware_lib
§ http://github.com/vmware/vmware-vcsa
§ http://github.com/vmware/vmware-vcenter
§ http://github.com/vmware/vmware-vsphere

22. 22 Confidential
Lessons Learned
§ Working with APIs
§ Puppet 2.7.x -> Puppet 3.x
§ Puppet Wat?

23. 23 Confidential
Working with APIs
§ Functionalities not always in API*
§ Dealing with API versions
§ Metaprogramming
* Even if you are @wlam :)

24. 24 Confidential
Dealing with outliers
§ Accept work around
ssh.exec( "
esxcfg-vmknic -i #{opts[:new_mgmt_ip]}
-n #{opts[:new_mgmt_mask]}
-p 'Management Network' &&
esxcfg-route -a default #{opts[:new_mgmt_gw]} &&
esxcfg-vswitch -p 'Management Network'
-v #{opts[:new_mgmt_vlan]} vSwitch0
" )
§ Last resort, file a ticket
t = ServiceNow::Request.new(
:subject => ”Please click yes"
)
ServiceNow.create(t)

25. 25 Confidential
Testing API versions
§ puppet apply --libdir=/dev/null
§ bundler exec + Gemfile
source 'file:///opt/repo/’
source 'https://rubygems.org’
gem 'facter’
gem 'puppet', '3.2.2’
gem 'gyoku', '1.0.0z2’
gem 'hashdiff’
gem 'net-ssh’
gem 'nokogiri’
gem 'nori', '1.1.4’
gem 'rbvmomi', '1.6.0.z1’
gem 'rest-client’
gem 'pry'

26. 26 Confidential
Metaprogramming
Good
§ Reduces boiler plate code
§ No more API ‘transfer’ bugs
§ API reference = Resource reference
Bad
§ Difficult to debug
§ Fixing API issues results in surprises
§ Not all APIs are designed to be idempotent
• action => { :create, :modify }

27. 27 Confidential
Puppet 3 Upgrade
§ What the scope?
§ Where’s my HOME?
§ Ruby 1.9.3

28. 28 Confidential
What the scope?
§ Fully qualify ::class::var
§ Fix your template @var, scope.lookup(‘::class::var’)

29. 29 Confidential
Where’s my HOME?
§ Exec specify HOME.
§ Providers Fix:
if respond_to? :has_command
has_command(:brew, "/usr/local/bin/brew") do
environment({ 'HOME' => ENV['HOME'] })
end
else
commands :brew => "/usr/local/bin/brew”
end

30. 30 Confidential
Puppet Wat
Boolean:
§ adrientthebo/boolean
§ vmware_lib property
§ Symbool in Hash
What the undef?
§ Careful about behavior
ENC Data
§ ENC integer .to_s

31. 31 Confidential
Where we are Today
§ Deploy VPC: #
§ 95% reduction in deployment time
§ Configuration Management = Version
§ Metrics:
• 47 Modules
• 70 Custom Resources
• 1400 Resources

32. 32 Confidential
Work at VMware
Challenges:
§ Software Defined Datacenter
• Data driven configuration management
§ Software Defined Networking
• vCNS, Nicira VSX
§ Scale + Speed
• 10X, 100X, 1000X ?
• Faster !!! Now !!!

33. 33 Confidential
Thanks!
§ Nicholas Weaver
§ Randy Brown
§ Shawn Holland
§ Floyd Arguello
§ David Scherer
§ Ryan Zenker
§ Justin Guidroz
§ Dan Pittman
§ Branan Purvine-Riley
§ Zach Leslie
§ vCHS R&D Team

34. 34 Confidential
Q & A