Nova architecture, instance deployment flow & capacity planning for compute.

1. TABLE OF
CONTENTS
Deep Drive into Nova
By
Udayendu Kar
Sr Technical Architect
Avaya India Pvt. Ltd.
PuneLet'sOpenStack2017Meetup

2. TABLE OF
CONTENTS
Agenda:
 Nova – OpenStack Compute Service
 Nova Architecture
 Instance provision request flow in OpenStack
 Nova command lines
 Nova log locations
 Computation for Compute
 Questions & Answers

3. TABLE OF
CONTENTS
Nova – OpenStack Compute Service
This diagram shows the number of OpenStack Nova drivers for Hypervisors, which
allow you to choose which Hypervisor(s) to use for your Nova Deployment.

4. TABLE OF
CONTENTS
Quality groups are defined as follows:
o Group C: These drivers have minimal testing and may or may not work at any given time.
Test coverage may include unit tests that gate commits. There is no public functional
testing.
o Group B: Test coverage includes unit tests that gate commits. Functional testing is
provided by an external system (such as our CI loop) that do not gate commits, but
advises patch authors and reviewers of results in OpenStack Gerrit.
o Group A: Test coverage includes unit tests and functional testing that both gate commits.
Because of it KVM based compute is the widely used in industry till date and mostly with
RHEL, CentOS, Ubuntu based operating system.

5. TABLE OF
CONTENTS
Nova Architecture:
Keystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue

6. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Auth request placedInstanceprovisionrequestflowinOpenStack:

7. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Auth request successfully
validated

8. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
REST API call to nova api to
launch the instance

9. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova API send auth token to keystone for validation
and make DB entry for new instance

10. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
1. Keystone authenticate successfully and response is back to nova api
2. API request
details
2. API request
processed and
stored in DB
with all initial
instance
details

11. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
1. It publishes a short message
to scheduler queue with
instance info
2. Scheduler
picks message
from message
queue

12. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
1. Scheduler publishes
the message to compute
queue to provision the
instance
2. Nova compute get
message from MQ

13. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova compute make call to nova
DB
Nova compute make call to nova
conductor for information on vm from
DB

14. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
1. Nova compute makes a call
network for the instance
1. Nova compute makes a call
to neutron api to provision
network for the instance
2. Neutron configures IP,
etc
2. Neutron configures IP,
GW, DNS, L2 connectivity
etc

15. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova compute placed a
request to create a volume

16. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova compute instruct the
hypervisor to use the new
volume as a block volume

17. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
1. Nova compute place a
request to get a image
from glance
2. Nova compute can download
the image using URI from swift

18. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova compute place a
request to launch the vm
on hypervisor

19. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Nova compute send a message
to nova conductor to update
the DB with vm’s current state

20. TABLE OF
CONTENTSKeystone
Keystone API
Keystone DB
Horizon OR CLIHorizon OR CLI
Glance
Glance DB
Object Store
Swift
Glance API
Proxy Server
Glance Registry
Nova
Nova DB
Nova SchedulerNova Scheduler
Nova ConductorNova Conductor
Nova APINova API Queue
Compute Node
Nova Compute
Hypervisor NetworkNetwork
VM VM
Cinder Neutron
Block Storage Network Node Ceilometer
Queue
Cinder DB
Cinder Volume
scheduler
Cinder API
Cinder Backup
Queue
scheduler
Agent/Plugin
Neutron API
Neutron DB
Storage DHCP / Metadata
Agent
Router / GW
Tunnel/External
Bridge
Ceilometer DB
Ceilometer API
AgentCollector
Queue
Horizon polls nova api for vm
status and power on state
which is taken from nova DB

21. TABLE OF
CONTENTS
Nova command lines:
root@node-6:~# nova list | egrep -i "ID|7aacf85c-e266-4ade-a184-d5388de830f5"
| ID | Name | Status | Task State | Power State | Networks |
| 7aacf85c-e266-4ade-a184-d5388de830f5| sansm1| ACTIVE |- | Running | admin_internal_net=192.168.111.156 |
root@node-6:~# ps -aux | grep -i "7aacf85c-e266-4ade-a184-d5388de830f5"
libvirt+ 13047 99.9 0.3 2675984 130764 ? Sl May15 16305:17 qemu-system-x86_64 -enable-kvm -name instance-000001a3 -S -
machine pc-i440fx-vivid,accel=kvm,usb=off -cpu
Penryn,+dca,+pdcm,+xtpr,+tm2,+est,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 4 -realtime mlock=off -smp
3,sockets=3,cores=1,threads=1 -uuid 7aacf85c-e266-4ade-a184-d5388de830f5 -smbios type=1,manufacturer=OpenStack
Foundation,product=OpenStack Nova,version=13.0.0,serial=3875ad95-0e48-4459-9ed0-8a17c9910099,uuid=7aacf85c-e266-4ade-a184-
d5388de830f5,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-
000001a3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-
pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:compute/7aacf85c-e266-4ade-a184-
d5388de830f5_disk:id=compute:key=AQBHULdXtGMaKhAAwRpChroKtaXEsWua812okQ==:auth_supported=cephx;none:mon_host=192.
168.1.4:6789;192.168.1.6:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback -device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=rbd:compute/7aacf85c-e266-4ade-a184-
d5388de830f5_disk.config:id=compute:key=AQBHULdXtGMaKhAAwRpChroKtaXEsWua812okQ==:auth_supported=cephx;none:mon_hos
t=192.168.1.4:6789;192.168.1.6:6789,if=none,id=drive-virtio-disk25,format=raw,cache=writeback -device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk25,id=virtio-disk25 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=28 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:4d:74:4e,bus=pci.0,addr=0x3 -chardev
file,id=charserial0,path=/var/lib/nova/instances/7aacf85c-e266-4ade-a184-d5388de830f5/console.log -device isa-
serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-
tablet,id=input0 -vnc 0.0.0.0:2 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-
pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
root@node-6:~# virsh list --all | grep -i instance-000001a3
4 instance-000001a3 running

22. TABLE OF
CONTENTSLog file Service name (RPM Based) Service name (DEB Based)
nova-api.log openstack-nova-api nova-api
nova-cert.log [1] openstack-nova-cert nova-cert
nova-compute.log openstack-nova-compute nova-compute
nova-conductor.log openstack-nova-conductor nova-conductor
nova-consoleauth.log openstack-nova-consoleauth nova-consoleauth
nova-network.log [2] openstack-nova-network nova-network
nova-manage.log nova-manage nova-manage
nova-scheduler.log openstack-nova-scheduler nova-scheduler
Log files used by Compute services:
Footnotes:
[1] The X509 certificate service (openstack-nova-cert/nova-cert) is only required by
the EC2 API to the Compute service.
[2] The nova network service (openstack-nova-network/ nova-network) only runs in
deployments that are not configured to use the Networking service (neutron).

23. TABLE OF
CONTENTS
Facts to Consider:
NOTE: Overcommitting increases the amount of instances you are able to
run, but reduces instance performance.
CPU and memory overcommit settings are represented as a ratio. OpenStack uses the
following ratios by default:
o Default CPU overcommit ratio – 1:16
o Default memory overcommit ratio - 1.5:1
These default settings have the following implications:
o The default CPU overcommit ratio of 16 means that up to 16 virtual cores can
be assigned to a node for each physical core. For example, if a physical node
has 12 cores, the scheduler sees 192 available virtual cores.
o The default memory overcommit ratio of 1.5 means that instances can be
assigned to a physical node if the total instance memory usage is less than
1.5 times the amount of physical memory available. For example, if a physical
node has 48 GB of RAM, the scheduler allocates instances to that node until
the sum of the RAM associated with the instances reaches 72 GB (such as nine
instances, in the case where each instance has 8 GB of RAM).
Use the cpu_allocation_ratio and ram_allocation_ratio directives
in /etc/nova/nova.conf to change these default settings.

24. TABLE OF
CONTENTS
ComputationforCompute:
CPU Requirements:
Most modern CPUs support hardware virtualization (AMD-V technology provided by AMD, VT-x technology provided by
Intel). Ensure that the CPU you choose and the hypervisor both support hardware virtualization. The most common CPU
architecture for OpenStack cloud is x86-64.
To identify the number of servers, number of sockets and cores per socket you need to have the requirements for your
workload. Usually you need a number of VMs, a number of VCPU per VM and average expected performance of each
VCPU. For example, you can calculate the total number of CPU cores you need:
Cores = VMs / HT / CO
Where:
Cores – the total number of CPU cores you need
VMs – the total number of VMs you need to run
HT – the Hyper-Threading coefficient
CO – the CPU oversubscription coefficient (1 for no oversubscription)
HT equals 1.3 if the CPU supports Hyper-Threading, and equals 1 if it does not.
The number of CPU sockets: Sockets = Cores / Cores_Per_Socket
Where:
Sockets – the number of CPU sockets you need
Cores – the total number of CPU cores you need
Cores_Per_Socket – the number of cores per socket
And finally, the number of servers you need: Servers = Sockets / Sockets_Per_Server
Where:
Sockets_Per_Server - is the number of sockets per server.

25. TABLE OF
CONTENTS
Example:
Let say you need to run 100 VMs:
VMs = 100
You will not use Hyper-Threading
HT = 1
We will use CPU oversubscription equals to 2, which means that in average we want to use a single CPU core (for example,
2.4GHz) for 2 VCPUs.
CO = 2
The total number of CPU cores you need:
Cores = VMs / HT / CO = 100 / 1 / 2 = 50
You prefer to use 8 core CPU:
Cores_Per_Socket = 8
The number of CPU sockets you need (rounded to the next integer number):
Sockets = Cores/Cores_Per_Socket = 50 / 8 = 7
You prefer to use 2 CPU sockets per server:
Sockets_Per_Server = 2
The number of servers you need (rounded to the next integer number):
Servers = Sockets / Sockets_Per_Server = 7 / 2 = 4
Therefore, to run 100 VMs with 2 virtual CPUs each, without Hyper-Threading and CPU oversubscription, you will need 4
servers for the compute nodes. Each server will contain 2 CPU sockets and 8 2.4GHz CPU cores per socket.

26. TABLE OF
CONTENTS
Memory Requirements:
You can use the following formulas to calculate memory required for the compute node:
VMs_Per_Server = VMs / Servers
RAM = RAM_Per_VM * VMs_Per_Server / MO + OS_RAM
Where:
VMs – the total number of VMs you need to run
Servers – the number of servers you need
VMs_Per_Server – the calculated number of VMs per server
RAM_Per_VM – RAM per VM
MO – the RAM oversubscription coefficient (1 for no oversubscription)
OS_RAM – RAM required for the operating system and the hypervisor
Note, that the hypervisor can also use system memory for disk caches. The amount of RAM required for the
disk caches for all of the VMs is hard to estimate: it depends on the number of VMs on the host, the number
of disks for each VM and type of cache for each disk.

27. TABLE OF
CONTENTS
Example:
Let say you need to run 100 VMs:
VMs = 100
Each VM requires 4GB RAM:
RAM_Per_VM = 4
And let say you will use 4 servers for this setup:
Servers = 4
The number of VMs per server:
VMs_Per_Server = 100 / 4 = 25
No RAM oversubscription:
MO = 1
You will use 16GB RAM for the operating system
OS_RAM = 16
RAM required for the compute node:
RAM = 4 * 25 / 1 + 16 = 116
Therefore, to run 100 VMs with 4GB RAM each, without RAM oversubscription and disk caches you will need
116GB RAM on each compute node.