CERN, the European Laboratory for Particle Physics, provides the infrastructure and resources to thousands of scientists all around the world to uncover the mysteries of the Universe. In the quest to build a private Cloud Infrastructure to support its users, CERN started early evaluating the OpenStack project, building several prototypes and engaging with the community. Finally, in 2013 CERN released its production Cloud Infrastructure using OpenStack. Since then we moved from a few hundred cores to a multi-cell deployment spread between different regions. After 7 years deploying and managing OpenStack in production at a large scale, we now look back and discuss the challenges of building a massive scale infrastructure from 0 to +300K cores. In this talk we will dive into the history, architecture, tools and technical decisions behind the CERN Cloud Infrastructure over the years.

2. 2
10 years of OpenStack at
CERN
(From 0 to 300k cores)
Virtual Open Infrastructure Summit
Belmiro Moreira
belmiro.moreira@cern.ch @belmiromoreira

4. 2009 - 2011
4
Virtualization and Server Consolidation

5. CVI - CERN Virtual Infrastructure
5

6. LxCloud - Virtualize Batch Infrastructure
6

7. 2010 - 2011
7
OpenStack – Early Days

8. OpenStack at CERN - Early days
8
https://indico.cern.ch/event/118726/attachments/60920/87520/OpenStack.pdf

9. OpenStack at CERN - Early days
● Snapshot of one of the first development versions of OpenStack Horizon
9

10. 2011 - 2013
10
Agile Infrastructure and Cloud Prototype

11. Agile Infrastructure Project
11

12. Prototyping CERN OpenStack Cloud
12

13. Prototyping CERN OpenStack Cloud
13
ESSEX
(April 2012)
FOLSOM
(September 2012)
GRIZZLY
(April 2013)
“Guppy”
June 2012
“Ibex”
March 2013
“Hamster”
October 2012

14. Prototyping CERN OpenStack Cloud
14

15. Engaging with the Community
15

16. 16

17. 2013 - 2020
17
From 0 to +300k cores

18. 18

19. CERN OpenStack Cloud - 2013
19

20. CERN OpenStack Cloud - 2013
20

21. CERN OpenStack Cloud - 2013
21

22. CERN OpenStack Cloud - 2013
22

23. CERN OpenStack Cloud - Growth
23

24. CERN OpenStack Cloud - Growth
● OpenStack projects available in the CERN Cloud over releases
24
Grizzly Havana Icehouse Juno Kilo Liberty Mitaka Newton Ocata Pike Queens Stein
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic
Mistral
Manila
Qinling ?
Watcher ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic
Mistral
Manila
Qinling ?
Watcher ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic *
Mistral *
Manila *
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic ?
Mistral ?
Manila *
Trove ?
Murano ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic ?
Mistral ?
Manila ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic ?
Mistral ?
Manila ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum *
Barbican *
Neutron *
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat *
Rally *
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Nova
Glance
Horizon
Keystone
Ceilometer *
Nova
Glance
Horizon
Keystone
Ceilometer *
* - Pilot service
? - Trial service
Rocky Train Ussuri
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic
Mistral
Manila
Qinling ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic
Mistral
Manila
Qinling ?
Nova
Glance
Horizon
Keystone
Ceilometer
Cinder
Heat
Rally
EC2API
Magnum
Barbican
Neutron
Ironic
Mistral
Manila
Qinling ?

25. 25

26. Milestone Highlights
26

27. Nova - Cells
● Allows Nova to scale to thousands of compute nodes
● Biggest Nova Cells deployment
● Moved from 2 cells to +80 cells
● Upgrade from CellsV1 to CellsV2 in 2018
27

28. Ceilometer - The Rise & Fall
● OpenStack Ceilometer deployed
● Removed after run it for 3 years. Not scalable and difficult to retrieve data
28

29. Storage - Cinder, Manila, S3
● OpenStack Cinder with Ceph backend (2014)
○ Several volume types available
● OpenStack Manila (Fileshare service). Backed by CephFS (2017)
● S3 available (end 2018)
29

30. Container Orchestration - Magnum
● OpenStack Magnum service available since 2016
● Extremely popular service, +500 clusters
30

31. Networking - Nova-network to Neutron
31

32. Baremetal Provisioning - Ironic
● In production since 2018
● All new hardware is enrolled using Ironic. +5000 nodes managed by Ironic
● Existing hardware will be enrolled into Ironic during 2020
32

33. Operations - Rundeck and Mistral
● OpenStack Mistral
● RunDeck
33

34. Operations
34

35. Operations
● Experience growing/managing the Infrastructure during the last 7 years
● Several upgrades during this journey
○ OpenStack release cycle is every 6 months!
○ SLC6 to CC7 upgrade
○ CC7 upgrades
○ CC7 to C8 upgrade?
● Supported for few years KVM and HyperV in the same infrastructure
○ Migrated CVI VMs to OpenStack HyperV and then to OpenStack KVM
● Security updates required reboot of all cloud
● Most user management operations are automated
○ project creation; quotas; ...
35

36. https://techblog.web.cern.ch/techblog/post/region-split/
Splitting the Infrastructure into Regions
36

37. ● Public Clouds
○ Based on different pricing/SLA considering resource availability
○ Reserved instances vs spot-market
● Private Clouds
○ Quotas are hard limits. Leads to a reduction in resource utilization
○ Preemptible instances
■ Projects that exhausted their quota can continue to create instances
● Opportunistic workloads
● Low SLA
● Preemptible Instances Workflow in OpenStack Nova
○ The creation of a non preemptible VM fails because there aren’t available resources
○ Instances that fail with “Nova Valid Host”, go to “PENDING” state instead of “ERROR”
○ The Reaper service is notified and it tries to free the requested resources
■ Rebuild the instance
■ Or change instance state to “ERROR”
Preemptible Instances
37
https://techblog.web.cern.ch/techblog/post/preemptible-instances/

38. Work with the Community
38

39. 39

40. 2020 - ...
40
What’s next?

41. Challenges
● Leveraging Container Orchestration to deploy OpenStack control plane
● Re-enroll existing physical resources into OpenStack Ironic
● Introduction of GPU resources
● Move all resources from nova-network to Neutron
● Exploring how to provide ML platforms and Functions as a Service to our
users.
41

42. 42

43. Summary
● During the last 10 years, resource management and deployment model
changed completely
○ From Virtualization and Server consolidation to a Cloud Infrastructure
○ From Baremetal to VMs, to managed Baremetal to Containers
● Continue to adapt the Infrastructure to the new technologies and
requirements
○ Iterative approach to introduce new services, new functionality
○ Continue to explore new approaches to deploy/manage a large infrastructure
■ Control Plane managed by kubernetes
■ New regions
■ Preemptible instances
43

44. Hall of Fame
Stefano Zilli
Wataru Takase
Thomas Hartland
Mihai Patrascoiu
Belmiro Moreira
Mateusz Kowalski
Thomas Oulevey
Arne Wiebalck
Jan van Eldik
Jose Castro
Spyridon Trigazis
Daniel Abad
Luis Pigueiras
Vitor Araujo
Luis Fernandez Alvarez
Daniel Fernandez Rodriguez
Gary McGilvary
Marek Denis
Andrea Giardini
Bruno Bompastor
44
Joe Harrison
Thodoris Tsioutias
Clenimar Filemon
Markus Sommer
Vipin Rathi
Ran Du
Cris Cordeiro
Luca Tartarini
Dinika Saxena
Shweta Oak
Jakub
Pavel
Antonio Marino
Marcos Fermin Lobo
Davide Michelino
Parin Pocheba
Nitin Aggarwal
Sean Crosby
Ignacio Dominguez
Martinez-Casanueva
Monika Talach
Mathieu Velten
Bertrand Noel
Konstantinos Samaras-Tsakiris
Surya Seetharaman
Robert Vasek
Ricardo Brito da Rocha
Costin Gament
Domenico Giordano
Iago Santos Pardo
Victor Araujo
Chirag Arora
Cas van der Laan
Zygimantas Matonis
Patrycja Gorniak
Elizaveta Svitanko
Venkata Ravicharan Nudurupati
Fedor Kitashov
Juan Dupuis
Serena Ziviani
Diogo Guerra
Evangelia Santorinaiou
Henni Mohamed
Roberto Soares
Theodoros Tsioutsias
Dheeraj Gupta
Vineet Menon
Lalit Dagre
Pranav Gaur

45. @belmiromoreira