These are the slides from the presentation given at the OpenStack Summit in Hong Kong in Fall 2013
PayPal has adopted a hypervisor agnostic stance within our Openstack Grizzly cloud. This presentation will cover the details surrounding our grizzly implementation and integration of both KVM and ESX hypervisors under one management umbrella. Grizzly deployment details configuration details for ESX integration Reasons for execution of this strategy benefits and pitfalls of this plan This will be an audience modified presentation of one that I am giving at VMWorld 2013 in San Francisco in August 2013.
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Marriage of Openstack with KVM and ESX at PayPal OpenStack Summit Hong Kong Fall 2013
1. MARRIAGE OF OPENSTACK
WITH KVM AND ESX AT
PAYPAL
MULTI-VENDOR AGILITY
Open Stack Summit – Hong Kong - 2013
2. ABOUT PAYPAL
PayPal offers flexible and innovative payment solutions for consumers
and merchants of all sizes.
• 137,000,000 Users.
• $300,000 Payments processed by PayPal each minute.
• 193 markets / 26 currencies.
• PayPal is the World‟s Most Widely Used Digital Wallet.
2
5. PAYPAL INTERNAL CLOUD
2012/2013 Shift toward an internal cloud model
• Shift from Enterprise design model to cloud-based design
• Elastically scale and self-heal infrastructure to accommodate
unpredictable usage patterns of customers and internet commerce
• Separate rapidly iterating customer experiences from core
services
• reduce overall cost per transaction within the environment
5
6. CLOUD IS THE GREAT ENABLER
ENABLE THE DEVELOPER
ENABLE THE BUSINESS
One-Click
Developer
Self Service
Global
Compute &
Data
Fulfillment
Payment Delivery
SelfOrganizing &
Optimizing
Infrastructure
System Intelligence
Driven Operation
Code
6
Deploy
Enjoy
7. PAYPAL CLOUD PLATFORM –
GUIDING PRINCIPLES
• Technology
− Adopt Open Source Solutions where ever possible
− No Vendor Lock-in
− Industry Best Practices
− Leverage Industry/ebay Inc Investments
• Functionality
− Self-Service tool for application life cycle management.
− Robust Automation & Orchestration
− Seamless On-Demand Capacity Fulfillment
7
8. OPENSTACK
PayPal deploying Openstack in order to help transform our global
infrastructure into an agile and open cloud platform.
Agility - time to market for customer facing services
Agility - speed to service developer requests for VM resources
Agility – utilize the engineering culture of PayPal to
deliver specialized cloud services where needed
8
10. CLOUD BEFORE INTEGRATION
WEB
F
Z
F
Z
F
Z
KVM
Local Disk
“Stateless & Disposable”
F
Z
MID
Cloud Management Zone
VCenter Management
F
Z
F
Z
F
Z
KVM
Local Disk
F
Z
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE
FZ = Logical Fault Zones
13. COMPARING
But isn‟t Openstack a direct replacement for ESX? Why would
you keep them both?
ESX/Vsphere != Openstack
NOVA != vSphere || vCenter || ESXi
NOVA =~ vCD, vCAC
KVM =~ ESX
To connect to any hypervisor, the Openstack cloud „proxies‟
connections to any supported hypervisor via Nova. That
abstracts the „Cloud‟ from the hypervisor
13
14. BRINGING ESX „INTO‟ THE CLOUD
• Equivalent functionality on KVM and ESX
• Full birth to death lifecycle management of virtual machines
− Build new, power on, power off, console, rebuild, delete
• Auto-configuration of host resources following t-shirt sizes
standards
− CPU, RAM, NIC, IP, OS Version
• IP Address Management
• Build from “Snapshot”/”Template”
• Deploy resources following appropriate fault zone model
• Must work from within single Horizon/Asgard interface
14
15. HYPERVISOR REQUIREMENTS
• ESX 5.1
− 5.0 works but too many back-ports (for us) / tweaks
• Single security zone per hypervisor
− No sharing of confidential & non-confidential on same hardware (PCI)
• Openstack management network communication
− This is NOT necessarily the VKERNEL network
15
16. STORAGE REQUIREMENTS
• “Shared storage” required
− Data Store Cluster
− Single Data Store support
• DRS Enabled with auto-placement
• Data Stores must be created in advance
− No Cinder support
16
17. OPENSTACK GRIZZLY
⁃ O
penS ack C
t
ommand Line T
ools (nova-client, swif t-client, et c.)
⁃ C
loud M anagement T
ools (Right scale, E rat ius, et c.)
nst
⁃ G t ools (C
UI
yberduck, iPhone client, et c.)
Int er net
O
penS ack
t
O
bject API
O
penS ack
t
C
omput e API
O
penS ack
t
Image API
O
penStack
Identity
AP
I
O
penStack
D
ashboard
H (S)
TTP
Amazon
Web Ser vices
E 2 API
C
VNC VMRC
/
/ Spice
O
penS ack
t
Block S orage API
t
Hor izon
O
penS ack
t
Net wor k API
O
penStack
O
bject AP
I
O
penStack
Image AP
I
O
penS ack
t
O
bject API
swif t-proxy
O
penStack C
ompute
AP /
I
Admin AP
I
O
penStack
Identity
AP
I
O
penS ack
t
O
penS ack
t
Block S orage API Block S orage API
t
t
nova-api
O
penS ack
t
Image API
glance-api
(O E 2, Met adat a, Admin)
S, C
nova-comput e
nova-cert/
objectstore
glance-regist r y
cont ainer
object
cinder-api
nova-console
nova-*proxy
O
penS ack
t
Image
API
memcached
account
O
penS ack
t
Net wor k API
O
penS ack
t
Net wor k API
H
TTP(S)
cinder-volume
quant um
agent (s)
nova
dat abase
object
st ore
O
penStack
Identity
AP
I
O
penS ack O
t
bject St ore
Queue
net wor k
provider
quant um
plugin(s)
quant um
dat abase
Queue
volume provider
Queue
hyper visor
cont ainer
D
B
cinder-backup
libvirt, XenAPI, et c.
glance
dat abase
account
D
B
quant um-ser ver
cinder
dat abase
nova-conduct or
nova-consoleauth
cinder-scheduler
http://www.solinea.com
O
penStack
Identity
AP
I
O
penS ack
t
Image Ser vice
nova-scheduler
O
penS ack C
t
omput e
O
penS ack
t
Block S orage
t
O
penS ack
t
Net wor k Ser vice
O
penStack
Identity
AP
I
O
penStack
Identity AP
I
keyst one
(ser vice & admin APIs)
O
penStack
Identity
Service
t oken backend
cat alog
backend
policy
backend
O
penStack O
bject AP
I
O
penStack
Identity
AP
I
O
penStack
Identity
AP
I
ident it y
backend
19. CONFIG OF NOVA
Nova is the project name for OpenStack Compute, a cloud computing fabric controller,
the main part of an IaaS system. Individuals and organizations can use Nova to host
and manage their own cloud computing systems.
#compute_driver = libvirt.LibvirtDriver
compute_driver = vmwareapi.VMwareVCDriver
Can be multiple
vmwareapi_host_ip=192.168.20.50
clusters now!
vmwareapi_host_username=root
vmwareapi_host_password=vmware
vmwareapi_cluster_name=openstack_test
vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl
Vcenter 5.1 Appliance
19
Confidential and Proprietary
20. GLANCE AND IMAGES
Rules for Glances images for VMWare
• Saved in VMDK Format
• Imported as VMDK Format
• Thick Provisioned VMDK Required
• No split VMDK allowed (must be merged)
• In a multi-hypervisor cloud, all images are separate
„per hypervisor‟ (no launching KVM VM‟s on ESX)
glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"
vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk
20
Confidential and Proprietary
21. BUILDING AND INSTALLING OS
• Kickstart
• Build a small root disk
• Use kickstart to image machine
• Post-install with puppet to customize machine and
add additional mount points depending on
application requirements
• Image Deploy
• Currently does not support „config-drive‟
• Need Guest Tools to „duplicate‟ functionality
21
Confidential and Proprietary
22. WHAT ABOUT THE NETWORK
•
22
Quantum requires NVP 3.2
• Cannot talk directly to VSphere API to allocate VDS
Port to NIC
• Uses vAPP – integration bridge or native in 5.5
• Configured as separate transport zone within Nicira
Confidential and Proprietary
23. WHAT‟S LEFT
• Component “at-scale” testing
• Currently manage “tens” at a time, need to
move to “hundreds” or “thousands”
• Most fixes in Havanna, every bug-fix needs to be
reviewed and possible back-ported to Grizzly
• Multiple Data Store enumeration on a cluster
• Full Certification on VCE VBLOCK with Vision
Intelligent Operations, auto-upgrades, and full
Openstack support of all components
23
Confidential and Proprietary
24. READING MATERIALS
•
•
•
•
•
•
http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple
http://www.slideshare.net/kenhui65/getting-started-with-openstack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui
http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc
http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc
http://www.ebay.com - Buy It Now
http://www.paypal.com - and then Pay for it Here!
Paypal is a Payments companyBetter for merchantsEasier for youCan be considered your digital walletWe’re not just on the internet, but we are also in store in places like Home Depot and GNCScale is importantShopping habits importantPaypal scales to meet needs worldwide and can be deployed in-country depending on local laws.
PayPal moved from 0% virtualized in production to 90% virtualized in one year.We took advantage of the converged infrastructure to shorten ramp up time and utilized resources from the vendor supplierAPI used for monitoring and deployment scriptsThere is a predictible scaling pattern for the front-tier on paypal.com and measuring the resultant output of the VM’s allowed us to scale as appropriate to meet demandVBLOCK measured at 99.999% availability which gave us confidence in running paypal apps on it
Clouds are a work in progress
Shift from Enterprise design model to cloud-based designElastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerceSeparate rapidly iterating customer experiences from core servicesreduce overall cost per transaction within the environment
Open Source to enable innovation fasterDefine Standards and APIsIt enables vendors to exactly know what large scale operators need to manage their cloudIndustry best practices prevents “reinventing the wheel”
Two Entry Points for InfrastructurePayPal Product DevelopersCloud Operators to manage CloudCentrally Orchestrated using HeatLocal StorageHP 4X600 GB(MirrorCisco 4948 & Arista 7050Nicira NVPF5 10.2.2 LB
New Hardware PlatformAll new Application PlatformGive infrastructure to developersFault Zones – Built to support maintenance activities and disposable compute for stateless load balanced poolsFront and business tiers separate infrastructure to retain separation for PCI
New Hardware PlatformAll new Application PlatformGive infrastructure to developers
Although many have mistakenly made direct comparisons between OpenStack Nova and vSphere, that is actually quite inaccurate since Nova actually sits at a layer above the hypervisor layer. OpenStack in general and Nova in paticular, is most analogous to vCloud Director (vCD) and vCloud Automation Center (vCAC), and not ESXi or even vCenter. In fact, it is very important to remember that Nova itself does NOT come with a hypervisor but manages multiple hypervisors, such as KVM or ESXi. Nova orchestrate these hypervisors via APIs and drivers. The list of supported hypervisors include KVM, vSphere, Xen, and others; a detailed list of what is supported can be found on the OpenStack Hypervisor Support Matrix.