This document summarizes an OpenStack deployment with high availability (HA) capabilities. The deployment uses Pacemaker for HA across controller, backend, storage, network, and compute nodes. It also describes Masakari, an OpenStack project that provides VM-level HA by automatically recovering virtual machines from failures of compute nodes, hypervisors, or individual virtual machines.

1. Copyright©2015 NTT corp. All Rights Reserved.
Matcha: fully HA configured
OpenStack
Masahito Muroi, NTT
IRC: masahito
muroi.mashaito@lab.ntt.co.jp

2. 2Copyright©2015 NTT corp. All Rights Reserved.
• Bitter Japanese Green Tea
• Used for Japanese Tea ceremony
• Many kind of snacks have Matcha flavor
• Try it in Tokyo!
What’s Matcha
Photo by (c)Tomo.Yun, http://www.yunphoto.net

3. 3Copyright©2015 NTT corp. All Rights Reserved.
•Fully HA configured OpenStack
•Having VM-HA feature
•Also having full scalability
What’s Matcha
Controller Node(2)
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy
Network Node(4)
Compute Node(4)
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
Backend Node(3)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(nAct-1Sby)
• neutron-linuxbridge-agent
• neutron-dhcp-agent
• neutron-l3-agent
pacemaker(nAct)
Storage Node(2)
glance-api
glance-registry
Apache(API Filter)
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
pacemaker(3Act)
•VIP(MQ, PXC)

4. 4Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•Access through VIP controlled by
Pacemaker
Controller Node HA
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy

5. 5Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•RabbitMQ using mirrored queue
•MySQL with Percona-cluster backend
•Access through VIP controlled by
Pacemaker
Backend Node HA
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)

6. 6Copyright©2015 NTT corp. All Rights Reserved.
•Active-Stanby
•Controlled by Pacemaker
Storage and Network Node HA
Storage Node(2)
glance-api
glance-registry
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
Network Node(4)
pacemaker(nAct-1Sby)
•neutron-linuxbridge-agent
•neutron-dhcp-agent
•neutron-l3-agent
glance-api
glance-registry
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
pacemaker(nAct-1Sby)
•neutron-linuxbridge-agent
•neutron-dhcp-agent
•neutron-l3-agent

7. 7Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•Masakari [1] for VM-HA
•can rescue VM down
•can rescue nova-compute down
•can rescue Hypervisor down
Compute Node HA
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)
[1]: https://github.com/ntt-sic/masakari
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)

8. 8Copyright©2015 NTT corp. All Rights Reserved.
• Service Model: Pets vs Cattle
• Cloud-native application handles High Availability
at its own layer. (Cattle service model)
• Sometimes, customers and/or applications still
prefer “Pets” service model.
• Masakari
• Provides VMHA for “Pets” service model
• Automatic VM instance recovery in case of VM or
hypervisor failure to minimize the downtime
• Requires Nova and Shared file system such as NFS
• Needs No modification of OpenStack components
• Uses Pacemaker to detect a host failure
• Is now at https://github.com/ntt-sic/masakari
• Author: Kentaro TANAKA
<tanaka.kentaro@lab.ntt.co.jp>
• Is mostly written in Python
• Is open source under Apache license
Masakari: VMHA for OpenStack Compute

9. 9Copyright©2015 NTT corp. All Rights Reserved.
Masakari: Architecture and Data Flow
OpenStack API
ComputeNodesControllerNodes
&BackendNodes