More Related Content
Similar to OpenStack Architecture (20)
OpenStack Architecture
- 1. PAGE 1Copyright © 2014 Mirantis, Inc. All rights reservedCopyright © 2014 Mirantis, Inc. All rights reserved Release H2.0.3
OpenStack
Architecture
Polina Petriuk
Sr. Technical Trainer
Mirantis Inc.
- 2. PAGE 2Copyright © 2014 Mirantis, Inc. All rights reserved
OpenStack Architecture
Provision a VM Request Flow
- 3. PAGE 3Copyright © 2014 Mirantis, Inc. All rights reserved
VM Provisioning
• Is the most common and complex process in
OpenStack
• Involves interaction of most of OpenStack
components
- 4. PAGE 4Copyright © 2014 Mirantis, Inc. All rights reserved
Initial State
Assumes Project is created,
provisioning quota is available, user
has an access to Horizon/CLI
Cloud Operator, DevOp, etc.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
- 5. PAGE 5Copyright © 2014 Mirantis, Inc. All rights reserved
Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify parameters of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Neutron)
• Optional (SSH Keys, Persistent volumes, comments, etc.)
• Select "Create" button
- 6. PAGE 6Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Dashboard
(Horizon)
Horizon provides a
baseline user interface
for managing OpenStack services.
- 7. PAGE 7Copyright © 2014 Mirantis, Inc. All rights reserved
Horizon
• Is “stateless” — doesn’t require a database
• Delegates error handling to the back-end
• Doesn’t support all the API functions
• Can use memcached or database to store
sessions
• Gets updated via API polling
- 8. PAGE 8Copyright © 2014 Mirantis, Inc. All rights reserved
Step 1: Request VM
Provisioning via UI/CLI
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Cloud Operator, DevOp, etc.
User logs in to UI
Specifies VM params: name,
flavor, keys, etc. and hits
"Create" button
- 9. PAGE 9Copyright © 2014 Mirantis, Inc. All rights reserved
Step 1: Request Provisioning
– Under the Hood
• Form parameters are converted to POST data
• "Create" request initiates HTTP POST request
to back-end
• To Keystone if auth token is not cached – step 2
- 10. PAGE 10Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Identity
Service (Keystone)
Keystone provides
Identity, Token, Catalog and
Policy services
for use specifically by projects
in the OpenStack family.
- 11. PAGE 11Copyright © 2014 Mirantis, Inc. All rights reserved
Keystone:
Identity Management
• User
• Credentials
• Token
• Associated with a user, an arbitrary bit of text that is used to access
resources
• Group of users
• Project
• Synonym to tenant
• Role
• Assigned to sers or groups for projects
• Domain
• Higher level of hierarchy – users and projects belong to domains
- 12. PAGE 12Copyright © 2014 Mirantis, Inc. All rights reserved
Keystone:
Service Catalog
• Service
• An OpenStack service, such as Compute (Nova), Object Storage
(Swift), or Image Service (Glance).
• Endpoint
• A network-accessible address, usually described by URL, from where
you access an OpenStack service
• Rule
• A set of requirements for performing an action over the endpoint.
- 13. PAGE 13Copyright © 2014 Mirantis, Inc. All rights reserved
Keystone Architecture
OpenStack
Services
Catalog
Backend
Token
Backend
Policy
Backend
Assignments
Backend
Identity
Backend
Credentials
Backend
Rule management
interface and rule-
based authorization
Contains
temporary
tokens
Contains
endpoint registry
Contains users
and groups
Contains
credentials, e.g.
EC2 tokens
Contains domains,
projects, roles and
role assignments
Deploys with its own DB but
can also be substituted with
LDAP or other EASKeystone API
- 14. PAGE 14Copyright © 2014 Mirantis, Inc. All rights reserved
Keystone:Role Based Access
Control (RBAC)
1. User gets Token from Keystone.
2. Token includes the list of user Projects and Roles in
them.
3. User calls the Service specifying the Token.
4. Service interprets the Roles:
• Service consults its policy.json file.
• Policy.json specifies the list of available rules.
• "admin_required": [["role:admin"], ["is_admin:1"]],
• “owner" : [["project_id:%(project_id)s"]],
• "admin_or_owner": [["rule:admin_required"], ["rule:owner"]],
• Policy.json specifies which rules are enforced for operations and
resources.
• "volume:create": [["rule:admin_or_owner"]],
- 15. PAGE 15Copyright © 2014 Mirantis, Inc. All rights reserved
Step 2: Validate Auth Data
Horizon sends HTTP request to
Keystone. Auth info is specified
in HTTP headers.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
- 16. PAGE 16Copyright © 2014 Mirantis, Inc. All rights reserved
Step 2: Validate Auth Data - Success
Keystone sends temporary token
back to Horizon via HTTP.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
- 17. PAGE 17Copyright © 2014 Mirantis, Inc. All rights reserved
Step 3: Send API Request to Nova API
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Horizon sends POST request to
Nova API (signed with given token).
- 18. PAGE 18Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Compute API
(Nova API)
Nova API is a
RESTful API web service
which is used to interact with Nova.
- 19. PAGE 19Copyright © 2014 Mirantis, Inc. All rights reserved
Nova API
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains:
• EC2-compatible—starting to be deprecated
• Compute API—all innovation happens here
• Is the only "allowed" way to interact with Nova
• Is “stateless”
- 20. PAGE 20Copyright © 2014 Mirantis, Inc. All rights reserved
Step 4: Validate API Token
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova API sends HTTP request to
validate API token to Keystone.
- 22. PAGE 22Copyright © 2014 Mirantis, Inc. All rights reserved
Keystone /w PKI - Token
Validation
• User gets one-time-password on creation
• User uses it to establish a key-pair
• Public key is signed and stored on Keystone
• From this point user uses client certificate to login
• Nova API performs offline check of the validity of token using CA&Cert it has from Keystone
Keystone
Key
generate_cms_token
(meta, keystone_key)user/pass/tenant
signed_cms_token
signed_cms_token
Nova
CA&Cert from
Keystone
verify(signed_cms_token,
Cafile,certfile)
- 23. PAGE 23Copyright © 2014 Mirantis, Inc. All rights reserved
Step 4: Validate API Token - Sucess
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Keystone validates API token and
sends HTTP response with token
acceptance/rejection info.
- 24. PAGE 24Copyright © 2014 Mirantis, Inc. All rights reserved
Step 5:
Process API Request
• Validate request parameters:
• Typographical errors are verified on code level
• Cloud-related parameters are validated via database
requests
• If the request cannot be processed:
• Throw an exception
• If the request can be processed:
• Save initial state to the database
- 25. PAGE 25Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Database
Nova Database stores current
state of all objects in compute
cluster.
- 26. PAGE 26Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Database
• In theory, can be any relational database
• Most of the deployments are done with MySQL or
PostgreSQL
• Nova API talks to database via SQLAlchemy
(python ORM (Object Relational Mapper))
• Database HA should be done via external tools:
• Galera
• Multi-Master replication Model for MySQL (MMM)
- 27. PAGE 27Copyright © 2014 Mirantis, Inc. All rights reserved
Step 5: Process API Request
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova API parses request to
python object model and
validates it by fetching data from
Nova DB. If request is valid, it
saves initial db entry about VM to
the database.
- 28. PAGE 28Copyright © 2014 Mirantis, Inc. All rights reserved
Step 6:
Process API Request
• Validate request parameters
• Typographical errors are verified on code level.
• Cloud-related parameters are validated via database
requests.
• If request cannot be processed:
• Throw an exception.
• If request can be processed:
• Save initial state to the database.
• Send message with next actions to Message Queue
(MQ)
- 29. PAGE 29Copyright © 2014 Mirantis, Inc. All rights reserved
Message Queue
Message Queue is a unified way for
collaboration between sub-
components.
- 30. PAGE 30Copyright © 2014 Mirantis, Inc. All rights reserved
Messaging Process Example
• OpenStack uses two modes:
• rpc.cast—don't wait for result (fire and forget)
• rpc.call—wait for result (when there is something to return)
Ex. Nova API Ex. Nova Scheduler
- 31. PAGE 31Copyright © 2014 Mirantis, Inc. All rights reserved
OpenStack Messaging Notes
• Uses multiple queues within single RabbitMQ
instance.
• Used by services to build machine state
• Each compute node has a queue
• Message traffic is not intensive.
• Doesn't send broadcast messages.
• For example: monitoring uses API polling
• HA should be configured separately.
• For example: mirrored queues not handled by OpenStack
- 32. PAGE 32Copyright © 2014 Mirantis, Inc. All rights reserved
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 6: Publish Provisioning Request
Nova API makes rpc.cast to
Scheduler. It publishes a
short message to scheduler
queue with VM info.
Request has been validated, but no
action has been taken yet, i.e. which
host, IP address, etc.
- 33. PAGE 33Copyright © 2014 Mirantis, Inc. All rights reserved
Step 7: Pick up Provisioning Request
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler picks up the
message from MQ.
- 34. PAGE 34Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler
Nova Scheduler is a daemon, which
determines, on which compute
host the request should run.
• Only provisioning time component—not like VMware’s Distributed Resource
Scheduler (DRS)
• Typically co-located with the Cloud Controller
- 35. PAGE 35Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Available
Schedulers
Scheduler Description Behavior
Chance Picks a host that is up Random
Filter
Picks the best-suited
host which satisfies
selected filter(s)
Custom Prioritisation
- 36. PAGE 36Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filtering
Affinity, Anti-affinity,
etc.
Eliminate
inapplicable hosts
- 37. PAGE 37Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• All Hosts
• AllHostsFilter
• All the available hosts, even non-active
• ComputeFilter
• All hosts that are operational and enabled
• Based on Host statically configured properties
• SimpleCIDRAffinityFilter
• Host within the specified IP range
• --hint build_near_host_ip=<1st IP in range> --hint cidr=<subnet
size>
• AvailabilityZoneFilter
• Hosts that are in the desired Availability Zone
• --availability-zone <availability-zone>
- 38. PAGE 38Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• Based on already running individual VMs
• SameHostFilter
• Same host as another instance in a set of instances
• --hint same_host=<VM ID>
• DifferentHostFilter
• Different host from a set of instances
• --hint different_host=<VM ID>
• Collocate/Distribute group of VMs
• GroupAffinityFilter
• Each instance in a group is on the same host
• --hint group=<group name>
• GroupAntiAffinityFilter
• Each instance in a group is on a different host
• --hint group=<group name>
- 39. PAGE 39Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• Based on flavor used (“nova-manage instance_type
list”)
• ComputeCapabilitiesFilter
• Hosts with properties that match flavor extra specs (“nova
hypervisor-show <host name>”)
• AggregateInstanceExtraSpecsFilter
• Hosts from host aggregates with properties that match flavor extra
specs ( “nova aggregate-details <aggregate ID>”)
• TrustedFilter
• Hosts with integrity obtained from the Attestation service (Intel TXT)
that matches flavor extra specs (“nova hypervisor-show <host
name>”)
• Based on Project (Tenant)
• AggregateMultiTenancyIsolation
• Hosts from host aggregates with filter_tenant_id property value
containing the Project (tenant) ID
- 40. PAGE 40Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• Based on Host resources left
• CoreFilter
• Hosts with sufficient CPU cores left, uses global
cpu_allocation_ratio setting to overcommit (16.0 by default)
• RamFilter
• Hosts with sufficient RAM left, uses global ram_allocation_ratio
setting to overcommit (1.5 by default)
• DiskFilter
• Hosts with sufficient disk space left for root and ephemeral storage,
uses global disk_allocation_ratio setting to overcommit (1.0 by
default)
• AggregateCoreFilter
• Same as CoreFilter but uses cpu_allocation_ratio set for Host
Aggregate
• AggregateRAMFilter
• Same as RAMFilter but uses ram_allocation_ratio set for Host
Aggregate
- 41. PAGE 41Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• Based on Host load
• IoOpsFilter
• Hosts with less than max_io_ops_per_host concurrent I/O operations
(8 by default)
• NumInstancesFilter
• Hosts with less than max_instances_per_host instances (50 by
default)
• Based on image used
• ImagePropertiesFilter
• Hosts that can support the specified image properties: architecture,
hypervisor type and virtual machine mode (hypervisor ABI)
• IsolatedHostsFilter
• Specific set of images and a specific set of hosts (from nova.conf)
- 42. PAGE 42Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Filters
• Scheduling history
• RetryFilter
• Hosts that haven’t been attempted for scheduling purposes
• Custom
• JsonFilter
• Hosts that pass a custom filter constructed by passing a scheduler
hint in JSON format
• Supported operators: =, <, >, in, <=, >=, not, or, and
• Supported variables: $free_ram_mb, $free_disk_mb,
$total_usable_ram_mb, $vcpus_total, $vcpus_used
• --hint query='[">=","$free_ram_mb",1024]‘
• Inherit from nova.scheduler.driver.Scheduler
• Override schedule_prep_resize and schedule_run_instance
methods
- 43. PAGE 43Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Weighting
RAM, CPU, etc.
Integer values
- 44. PAGE 44Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Scheduler: Weights
• Weigher - host property of integer type
• Every weigher can have multiplier for it
• weight = sum(weigher_i * multiplier_i)
• The only currently available RamWeigher:
• Spread VMs across hosts evenly
• ram_weight_multiplier=1.0
• Stack VMs on the hosts (fill-up)
• ram_weight_multiplier=-1.0
- 45. PAGE 45Copyright © 2014 Mirantis, Inc. All rights reserved
Step 8a: Schedule Provisioning
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler fetches
information about the
whole cluster from
database, filters, selects
compute node and
updates DB with its ID
- 46. PAGE 46Copyright © 2014 Mirantis, Inc. All rights reserved
Step 8b: Provision Scheduled
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler publishes
message to the compute
queue (based on host ID) to
trigger VM provisioning
- 47. PAGE 47Copyright © 2014 Mirantis, Inc. All rights reserved
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 9a: Start VM Provisioning
Nova Compute gets
message from MQ
- 48. PAGE 48Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
Nova Compute is a worker
daemon, which primarily creates
and terminates VMs via
Hypervisor API.
- 49. PAGE 49Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute Drivers
Nova
Compute
XCP
VM
VM
VMWare
Power
VM
VM
VM
libvirt
KVM
VM
VM
Xen
VM
VM
Qemu
VM
VM
LXC
Cont
ainer
Cont
ainer
Allows multiple hypervisor types per
cloud. Libvirt / KVM is most
commonly used in deployment
Maintained
by Microsoft
Maintained
by IBM
Bare
Metal
Docker
Contain
er
Contain
er
XenAPI
Naitive support
comes in IcehousePXETilera
Somewhat
experimental
Maintained
by Citrix
HyperV
VM
VM
ESXi
VM
VM
vSphere
VM
VM
- 50. PAGE 50Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute Notes
• Functionality provided by drivers is not 100%
similar.
• Exact "run_instance" flow depends on the
driver implementation.
• Most features are developed and tested on
KVM.
- 51. PAGE 51Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15
- 52. PAGE 52Copyright © 2014 Mirantis, Inc. All rights reserved
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 9b: Start VM Provisioning
Nova Compute makes rpc.call to
Nova Conductor for information on
VM from DB
- 53. PAGE 53Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Conductor
The Nova Conductor service is key
to completing no-db-compute
- 54. PAGE 54Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability: spawn multiple worker threads operating in parallel (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 55. PAGE 55Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15
- 56. PAGE 56Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Networking
(Neutron)
network-as-a-service
- 57. PAGE 57Copyright © 2014 Mirantis, Inc. All rights reserved
Neutron Notes
• Provides a flexible API (POST / GET) for service
providers or their tenants to manage OpenStack
network topologies.
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of
network connectivity from its implementation.
• In Icehouse, one can still choose to stay with nova-
network (Essex approach) or to go with Neutron.
- 58. PAGE 58Copyright © 2014 Mirantis, Inc. All rights reserved
Step 10: Configure Network
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute makes a call to
Neutron API to provision
network for the instance
- 59. PAGE 59Copyright © 2014 Mirantis, Inc. All rights reserved
Step 10: Configure Network (Continued)
Neutron configures IP, gateway,
DNS name, L2 connectivity, etc.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
- 60. PAGE 60Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15
- 61. PAGE 61Copyright © 2014 Mirantis, Inc. All rights reserved
Open Stack Storage Concepts
• Ephemeral storage:
• Persists until VM is terminated
• Accessible from within VM as local file system
• Used to run operating system and/or scratch space
• Managed by Nova
• Block storage:
• Persists until specifically deleted by user
• Accessible from within VM as a block device (e.g. /dev/vdc)
• Used to add additional persistent storage to VM and/or run operating system
• Managed by Cinder
• Object storage:
• Persists until specifically deleted by user
• Accessible from anywhere
• Used to add store files, including VM images
• Managed by Swift
- 62. PAGE 62Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Block Storage
(Cinder)
block storage as-a-service
- 63. PAGE 63Copyright © 2014 Mirantis, Inc. All rights reserved
Cinder Resources
• Volume:
• Is a persistent R/W block storage device
• Can be attached to VMs as secondary storage
• Can be root store to boot VMs
• Can be attached only to one instance at a time
• Keeps its state independent of an instance
• Snapshot:
• Is a read-only point in time copy of a volume
• Can then be used to create a new volume
• Backup:
• An archived copy of a volume
- 64. PAGE 64Copyright © 2014 Mirantis, Inc. All rights reserved
Cinder
Cinder Architecture
Cinder DB
Queue
Cinder Volume
Backend
Storage
Devices
Scheduler
Cinder API
Cinder CLI NovaHorizon
HTTP
AMQP
SQL
3rd-party
Cinder Backup
Object Storage
- 65. PAGE 65Copyright © 2014 Mirantis, Inc. All rights reserved
Cinder Volume Driver
• iSCSI:
• Dell EqualLogic
• EMC VMAX/VNX
• Hitach HDS
• HP 3PAR (StoreServ)
• HP / Lefthand SAN (StoreVirtual)
• Huawei T/Dorado/HVS
• IBM Storwize family/SVC/XIV
• LVM (Reference Implementation)
• Nexenta
• NetApp
• SolidFire
• VMware VMDK
• Windows Server 2012
• Zadara
• GlusterFS NFS (volumes as sparse files)
• IBM General Parallel File System (GPFS) (volumes as sparse
files):
• GPFS NSD
• ATA over Ethernet (AoE):
• Coraid
• Fibre Channel:
• NetApp
• HP 3PAR (StoreServ)
• Huawei T/Dorad/HVS
• IBM Storwize family/SVC/XIV
• VMware VMDK
• NFS (volumes as sparse files):
• NFS
• Nexenta
• NetApp
• VMware VMDK
• Zadara
• XenAPI Storage Manager
• RADOS Block Devices (RBD):
• Ceph
• Shared SAS:
• VMware VMDK
• Scale Out File System (SOFS) (volumes as sparse files):
• Scality
• VirtIO (Local raw storage) (volumes as sparse files)
- 66. PAGE 66Copyright © 2014 Mirantis, Inc. All rights reserved
Cinder Backup Drivers
• Swift
• Ceph
• IBM Tivoli Storage Manager (TSM)
- 67. PAGE 67Copyright © 2014 Mirantis, Inc. All rights reserved
Cinder Notes
• Functionality provided by drivers is not 100%
similar.
• Exact volume provisioning and attachment flow
depends on driver implementation and
Hypervisor used.
• Most features are developed and tested with
LVM over ISCSI and KVM.
- 68. PAGE 68Copyright © 2014 Mirantis, Inc. All rights reserved
Step 11: Request Volume
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
It is assumed a volume is
already created. Nova
Compute contacts Cinder to
get volume data. Can also
attach volumes after VM is
built.
- 69. PAGE 69Copyright © 2014 Mirantis, Inc. All rights reserved
Step 11: Request volume (Continued)
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute sets up the
host mount if needed &
instructs the Hypervisor to
use vol. as a new block
device
- 70. PAGE 70Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15
- 71. PAGE 71Copyright © 2014 Mirantis, Inc. All rights reserved
The OpenStack Image Service
(Glance)
Glance provides services for
discovering, registering, and
retrieving virtual machine images.
- 72. PAGE 72Copyright © 2014 Mirantis, Inc. All rights reserved
Definition of Image
• Copy of the entire contents of a storage device,
for example:
• hard drive
• DVD/CD
• other storage media
• Includes all the partition information, boot
sectors, the file allocation table, operating
system installation and application software.
- 73. PAGE 73Copyright © 2014 Mirantis, Inc. All rights reserved
Custom Image Creation Steps
• Get Operating System installation ISO
• Create VM (qemu-img create)
• Install Operating System
• Start VM with ISO connected as CD-ROM
• Connect to VM via VNC console or SPICE
• Install Operating System as you normally do
• Install and configure cloud-init (Linux) or cloudbase-init (Windows)
• Install virtIO drivers (Windows only)
• Prepare and cleanup the image
• Remove the MAC address details, etc, with virt-sysprep (Linux) or
sysprep.exe (Windows)
• Install any additional applications (OpenSSH, Apache, MySQL, etc.)
• Convert image to a different format if desired (qemu-img convert)
- 74. PAGE 74Copyright © 2014 Mirantis, Inc. All rights reserved
Glance
• Provides Images-as-a-Service.
• Can use multiple back-ends for image storage.
• Can store the same image in multiple locations.
• Supports multiple image formats.
- 75. PAGE 75Copyright © 2014 Mirantis, Inc. All rights reserved
Glance API
Glance CLI NovaHorizon
Glance
Glance Architecture
HTTP
AMQP
SQL
3rd-party
Glance Registry
Glance DB
Store Adapter
Swift
Cinder
File System
Amazon S3
HTTP
GridFS
Sheepdog
Ceph (RBD)
- 76. PAGE 76Copyright © 2014 Mirantis, Inc. All rights reserved
Glance Notes
• Functionality provided by backends is not 100%
similar.
• Exact Glance capabilities available depend on
the backend and Hypervisor used.
• Most of the features are developed and tested
with the file system or Swift.
- 77. PAGE 77Copyright © 2014 Mirantis, Inc. All rights reserved
Glance Disk Formats
Disk Format Description
raw an unstructured (unrestricted) disk image format
vhd VHD disk format, a common disk format used by virtual machine monitors from VMWare, Xen,
Microsoft, VirtualBox, and others
vmdk Another common disk format supported by many common virtual machine monitors
vdi disk format supported by VirtualBox virtual machine monitor and the QEMU emulator
iso archive format for the data contents of an optical disc (e.g. CDROM)
qcow2 disk format supported by the QEMU emulator that can expand dynamically and supports Copy
on Write
aki indicates what is stored in Glance is an Amazon kernel image
ari indicates what is stored in Glance is an Amazon ramdisk image
ami indicates what is stored in Glance is an Amazon machine image
- 78. PAGE 78Copyright © 2014 Mirantis, Inc. All rights reserved
Step 12: Request VM Image from Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute requests VM
image from Glance via Image ID
- 79. PAGE 79Copyright © 2014 Mirantis, Inc. All rights reserved
Step 13: Get Image URI from Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
If image with given image ID
can be found - return URI –
HTTP Get URI
- 80. PAGE 80Copyright © 2014 Mirantis, Inc. All rights reserved
Step 14: Direct Image File Copy
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute can download
image using URI, given by
Glance, directly from Swif
- 81. PAGE 81Copyright © 2014 Mirantis, Inc. All rights reserved
Step 14 alternative: Image Copy through Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
To leaverage Glance Server caching
mechanism and additional access restriction,
the Image copy can go through Glance
- 82. PAGE 82Copyright © 2014 Mirantis, Inc. All rights reserved
Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15
- 83. PAGE 83Copyright © 2014 Mirantis, Inc. All rights reserved
Step 15: Start VM Rendering via Hypervisor
In case of KVM / libvirtd this is
a single XML VM config file
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute creates a
command to Hypervisor and
delegates VM rendering to
Hypervisor.
- 84. PAGE 84Copyright © 2014 Mirantis, Inc. All rights reserved
Step 16: VM is UP
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute sends a
message to Nova
Conductor to update
DB with VM state
- 85. PAGE 85Copyright © 2014 Mirantis, Inc. All rights reserved
Step 17: User is Happy
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Horizon polls Nova API
for VM status and power
state, which is taken
from Database.
- 86. PAGE 86Copyright © 2014 Mirantis, Inc. All rights reserved
OpenStack Architecture
Recap
• Users log into Horizon and initiate a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Neutron configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift
• VM is rendered by Hypervisor