Liveperson have being using the openstack as it's main PaaS and IaaS infra for a long time. Upgrading is part of the deal. In this presentation we shared our experience with upgrading and what's important to pay attention to.

1. Openstack Upgrade
It can be done!
June 2017, Tel Aviv
Koby Holzer
Liran Cohen

2. About us
Liran Cohen Koby Holzer
Director of Platform ENG
~5 years @ Liveperson
Cloud ENG group leader
~5 years @ Liverperson

3. Agenda
● About Liveperson
● Openstack @ Liveperson
● Planning - Openstack upgrade.
● Testing - Openstack upgrade.
● The Upgrade...

4. About LivePerson

5. Liveperson in one sentence:
We created a
Digital Contact Center
to allow
Transitioning from Voice to
Messaging

6. Creating Meaningful Connections
Between Brands and Customers
Founded in 1995,
public since 2000
(NASDAQ: LPSN)
More than 18,000
customers worldwide
More than 1000
employees
25B site visits and +300M live
engagements Yearly
7 DC’s around the globe Headquartered in NYC with
offices in Americas, EMEA
and APAC

7. 18,000 customers around the globe, including:
Customers
8 of the Top 10 Fortune
500 companies
10 of the Top 15
Commercial Banks
(Fortune 500)
Top 4 of Top 5 Telecommunication
Companies (Fortune 500)
3 of the Top 5 Internet
Service Companies (Fortune
500)

8. 2011 2012 2013 2014
● Started early 2012
● 100% of our services
Physical
Openstack
2015 2017
Openstack @ LivePerson
● 70K virtual cores.
● ~1K physical servers.
● 7 Data Centers.
● 7K instances.
80%

9. Major Release Upgrades @ LivePerson...

10. Openstack @ LivePerson
Nova Neutron Glance Horizon Keystone

11. Operating System
CM - Deployment
Network - Provider networks (VLAN)
Hypervisor.
Storage - non persistent using hypervisor Local disks.
Openstack @ LivePerson
Ubuntu 14.04 LTS (Trusty Tahr)

12. Role: Controller
nova-api
nova-cert
nova-consoleauth
nova-novncproxy
nova-objectstore
nova-scheduler
glance-registry
glance-api
keystone neutron-server
neutron-dhcp
neutron-openvswitch-agent
openvswitch-switch
MySQL
neutron-l3
neutron-metadata
rabbitMQ
Images Network ComputeIdentity
Services - Controllers

13. Services - Compute
Role: Compute
nova-compute
neutron-openvswitch-agent
libvirt-bin
openvswitch-switch

14. Web Tier
App tier
Data Tier
Tech Stack @ Liveperson
OS tier
Supporting tools

15. Planning
Testing
Upgrading...

16. Planning

17. Openstack Support model On Ubuntu ● LTS support 5 years
● Non LTS support 18 months
Mitaka - OS upgrade - 16.04
Queens - OS upgrade - 18.04

18. Ubuntu Kernel Support

19. Release Notes
Review the release notes to learn about new,
updated, and deprecated features.

20. Impact ?
Users ?Apps ? Automations ?
Network Storage Compute

21. Choose your upgrade path
Possible ways to upgrade:
● Big Bang (in-place) Upgrades.
● Parallel Cloud - Side by Side clusters.
● Controller Side by Side.
● Rolling upgrade (Upgrade Levels).*

22. The upgrade plan...
● Create an upgrade procedure.
● Automate - As Much As Possible.
● Rollback is part of the plan.

23. Testing
LAB

24. production like:
● Hardware.
● Network configuration.
● OS Version , kernel.
● repos - Minor versions.
● Tools - CM , Upgrade Automations.
Testing Environment

25. ● Apply your upgrade plan.
● Assess and Modify.
● Rollback.
Testing Environment

26. Upgrading

27. The path we chose
Side By Side Controller Upgrade.
2 steps:
● New controller upgrade & DB sync
● Compute upgrades
Tests after every step

28. Old
Controller
Nova
Neutron
Glance
Keystone
Nova
Neutron
Glance
Keystone
● Take a snapshot of the cloud.
● Shut down OpenStack services gracefully.
● Sync the data , images and DB.
Old(Active) Controller
New
Controller

29. ● Import DB and upgrade schema.
● Start OpenStack services.
● Do basic API checks and Review Logs.
New Controller

30. ● Upgrade control group.
● Run tests on the new controller.
● Monitor (Api , AMQP , DB etc…)
● Upgrade the next group.
Compute

31. Control
Group
Compute Servers
Group 1 Group 2 Control Group
Old
Controller
New
Controller

32. Compute Servers
Group N Group 1 Control Group
Group
1
Old
Controller
New
Controller

33. Compute Servers
Group N Group 1 Control Group
Group
N
Old
Controller
New
Controller

34. ● During the Upgrade
● After the Upgrade
Must have for every upgrade

35. Koby HolzerLiran Cohen