Slides used in JINR/CERN "GRID and Advanced Information Systems" school of computing. Right belonging to author (Breogan Costa), CERN and JINR. You can freely use mentioning authorship. Logo of CERN cannot be used without explicit CERN permission.

2. The Real World of Virtual
Datacenters:
The enabling technology for Cloud Computing
X. Breogan Costa

3. TOC
● Motivation
● Introduction to virtualization and Cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options
● Supporting material (after the slide 60, for free!)
3/60

5. Use-case I (quite trivial): old game
● You want to run an old software, let's say you
absolutely love an old game made for
ZX Spectrum CPU:
Z80 8-bit
HD64180/Z180 architecture
5/60
●
But you cannot just buy a ZX Spectrum today_

6. Use-case I (quite trivial): old game
6/60

7. Use-case II: you have old servers
● 2003 Sun Fire (4800/4810)
● CPU(s): UltraSPARC III...
– Architecture: SPARC V9
7/60

8. Use-case II: you have old servers
● (1998) Compaq ProLiant (1600r)
● CPU: Pentium II Xeon Drake (1998)
– Architecture: x86
8/60

9. Use-case II: old software running
● And your organization depends on old software
made for those architectures
● Sometimes old software not portable
(proprietary or no resources to do that)
● For example...
(See Use-Case I)
9/60

10. Problems?

11. ● 2014 HP ProLiant (DL380 G8)
● CPU: 2013 Intel Xeon
(E5-2600 v2),
– Architecture: EMT64 (x86-64)
– Unsupported by old OSs
● http://www8.hp.com/us/en/products/proliant-servers/product-detail.html?oid=5177953
● http://ark.intel.com/products/series/75291/Intel-Xeon-Processor-E5-2600-v2-Product-Family#@All
Solution: a new server!
11/60

12. Problems?
● Installation time?

13. One Possible Solution:
● Fast deployment
● Move (even running) VMs to new servers, no downtime
● You should be able to emulate previous architectures (if they
are implemented)
13/60 Let's do it!

14. Intro
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options
But we need to know more

15. Is this new?
● First implementation: 1960's, at IBM Cambridge Scientific
Center:
– Virtualization development → starts with CP-40
Is this a mature
technology?

16. How this continued?
● IBM worked almost alone until the 1980's
– VM technology in 360, 370 and 390 series
● 1980's: workstation vendors get interested in
virtualization
● 1985:
– V86-mode (8086)
16/60
[Wikipedia]

17. 1998: release of the first true virtualization
of the full Intel processor architecture
What we can use today?
New (big) players in the game (2000-2013)
17/60

18. Hypervisors tech: elements
● Hypervisor
(= Virtual Machine Monitor -VMM)
● Host Machine
● Virtual Machines
What's
inside?

19. Hypervisors tech: elements
● Management console
interface

20. HW Emulation
– Memory address translation
– Byte ordering: little endian (Intel) vs. big endian (PowerPC, Sun,
Internet)
– Totally different
architecture
↓
Instruction
emulation
↓
Instruction set
translation

21. Hardware emulation
● Host-system interface
– VM running in hosted mode → certain host
resources are exposed to the VM (FS's, printers,
clipboard, etc)
● Virtual device subsystem
– Virtual devices to real host devices mapping
21/60

22. Summarizing: Why Virtualization?
HW independence
● Generic HW architecture
● + OS compatibility
● Generic drivers for most OS's

23. Summarizing: Why Virtualization?
Scalability
Performance
Improved by
modern HW
Ecological
benefits
23/60
Availability
Portability
Server sprawl
Centralizedmanagement

24. Why Virtualization? Example
● The Dynamic Datacenter (according to Microsoft)
1) Physical Layer
● Bare-metal HW and base SW
2) Virtual Layer
● Hypervisor and VMs
3) Application Layer
● Virtual servers, server consolidation
4) Model Layer
● Service/application components running in more than one server
● App/s requirements → App/s architecture → Deployment model
5) Management
● Datacenter management, VMs management
24/60

25. Why Virtualization? Extra benefits
● Hardware-assisted virtualization:
– CPU
● privileged instructions (generation 1 in x86): Intel VT-x, AMD-V
● Memory Management Unit (generation 2 in x86): Intel EPT, AMD RVI (RVI →
+42% performance according a VMware research paper)
– Chipset: I/O (AMD-Vi and VT-d), Networking (VT-c), PCI-E (IOV), ...
● Previous States restoration
– Snapshots: just for sort term: they must not be used as backups
● ...
¬¬!
25/60

26. Extra: Why Virtualization?
Cloud Computing!
26/60

28. Cloud Computing Main Service
Definitions
● IaaS
– Infrastructure as a Service
● PaaS
– Platform as a Service
● SaaS
– Software as a Service
● NaaS
– Network as a Service
● XaaS
– Everything as a Service
28/60

29. [Wikipedia]

30. ● HET (no)
Virtualization, pre-requisite?
Image by

31. But not all is good
● Security
– Cracker gain access to:
● Management tools
● Host management
– Virtual Networking

32. Virtualizing the
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options (Access and Safety System)

33. We did...
● Planification of what and how to virtualize
servers in the access and safety datacenters
– Nothing to do with the (great) CERN general virtual
platform
● Prototypes in testing facilities
– LHC0
– PS0
● Production environments ...
You can read our Paper for ICALEPCS 2013 Conference
33/60

34. In 2013

35. What our vClusters run...
● SCADA Systems
– Siemens WinCC, ARC PcVue
● Access Software: Gegelec Evolynx
● Video Servers
● Biometric servers: LG IRIS
● Distributed monitoring servers:
– Zabbix servers, Zabbix agents and Zabbix proxies
● Security auditing tools
35/60

36. What our vClusters run...
● Servers OS's:
– SLC (Scientific CERN Linux)
● CERN + Fermilab, based on RedHat Linux.
– SuSE Linux
● mainly as virtual appliances giving some service to the
virtual cluster management, as backups system
– Debian GNU/Linux: for security auditing tools
– Windows Servers (several versions)
– (sometimes) Vyatta OS (a GNU/Linux implementing
a virtual router)
36/60

37. Requirements & classifications
37/60
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options

38. Requirements
● Virtual CPU architecture
– At least, Intel VT-x, AMD-V
– vmx or svm in /proc/cpuinfo (egrep '(vmx|svm)' --color=always /proc/cpuinfo)
– CPU-Z in Windows
– Enabled on BIOS
● + generic/compatible hardware* (servers use to be)
38/60
Yes, you can do it at home!
(at your own risk
;)

39. Classification: Virtualization
● Partial
– some but not the entire target environment is
simulated. Historical milestone
● Examples: first-generation time-sharing system CTSS
(IBM M44/44X experimental paging system, 1960's)
● Full:
– complete HS (HW System) emulation
● Examples: VMware ESXi/Workstation/Player, Virtualbox,
Parallels Desktop
39/60

40. Classification: Virtualization
● Paravirtualization
– Not necessarily simulate hardware,
– offers a special API that can only be used by
modifying the "guest" OS.
● Examples: Win4Lin 9x, Sun's Logical Domains...
● Operating System-level virtualization
– OS's Kernel allows multiple isolated user-space
instances
● Examples: Parallels Virtuozzo Containers, openVZ...
40/60

41. Classification: Hypervisors
● Bare metal (“native” or “Type 1”)
– VMware ESX/ESXi, KVM, Xen, Microsoft Hyper-V
Server (Windows Server 2012 +)
● Hosted (“Type 2”)
– VMware Workstation/Player, VirtualBox, Microsoft
Windows Server Hyper-V Service (Windows Server
2008 R2 +)
41/60

42. What we should put in our virtual
Datacenter?
42/60
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure for virtualized datacenterse
● Common features, considerations
● Some advanced options

43. Virtual Infrastructure of a virtualized
datacenter
● Hosts & Hypervisors *
● Storage
● Virtual Network
● Virtual Machines
● Management platform
– Management Server
– Database
– Client platform
43/60

44. Important: Virtual Networking
● Defined at Datacenter level
44/60
● Defined at Datacenter level
– Every VM → different virtual MAC
[Cisco Web]

45. Common features, considerations
45/60
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options

46. High Availability & redundancy
● Downtime reduction
– NAS / Backups (/ Snapshots -not recommended for Backup)
– Restoration in different host
● Optional no-downtime using redundancy
– Execution in parallel
● Master VM
● Slave VM
46/60

47. Integrity
● Internal RAID disks
● NAS systems
– In vSphere they must be added as datastore
● Backup complete systems
● NAS servers support
– For backups
– For OS installation
47/60

48. Disaster recovery
● There are several backup tools to prevent this
situation
● Usage of NAS servers
● Programmed backups
– Commonly used snapshots as a base
● Backup keeping policy
● Image sharing
48/60

49. Basic Security
● General risks (according Gartner researches)
– Information security isn't initially involved in the
virtualization projects (40% in 2009)
– Compromise of Virtual Layer (VMM) → could
compromise of all hosted workloads (VMs)...
– … adequate controls on administrative access to
the Hypervisor/VMM layer and to administrative
tools are lacking
49/60

50. Basic Security
● Recommendations:
– Be careful with host system interface (shared
resources)
– VM isolation
– Don't use generic and shared administration accounts
(for traceability), even delete generic admin accounts
– Restrict root access at Hypevisor level
– Use the right permissions in user roles definition
– Be careful with roles' permissions hierarchy **
50/60

51. Migrations & conversions
● Tools:
– “P2V” tools
– “V2V” tools
● Also:
– VM cloning (excepting MAC address)
– Importing:
● OVF / other virtualization provider formats
● Cloned images (Acronis, Norton Ghost, etc)
– Exporting:
● OVF format, etc
51/60

52. Some advanced options
Table of Contents
● Motivation
● Introduction to virtualization and cloud
● My experience with virtualization at CERN
● Requirements & classifications
● Infrastructure
● Common features, considerations
● Some advanced options

53. Advanced options
● Hardware pass-through
– USB
● USB port assignation
– Real pass-through (PCI-*, etc) →
● VMware VMDirectPath I/O
● KVM
● Xen
● NOT implemented in Hyper-V
(at this moment)
53/60
If we have special
requirements...
Siemens CP1613
(Industrial Ethernet)

54. Advanced configurations
● Embedded architectures
– KVM in system-on-chip architectures:
● ARM Virtual Express (Cortex-A15 + Expansions FPGA)
● Virtualization on mobile devices
– Single-core/Multi-core devices
● Cortex-A15 was the first
– Android
– Devices
● Cellphones / smartphones
● Tablets
● Netbooks
● M2M devices
54/60

55. Main virtualization platforms
55/60
Which virtualization
provider select?

56. ● VMware vSphere Infrastructure
– ESXi hypervisor [free*] + vCenter [proprietary + license]
● KVM hypervisor [GPL/LGPL packages
or RedHat RHEV complete suite** + license]
KVM or Xen + Management tools
(RHEV and XenServer include management tools)
● Xen hypervisor [GPL packages
or Citrix XenServer ** + license]
● Microsoft Hyper-V Service or Hyper-V Server
[proprietary + license]
Xen and KVM are
Linux kernel
customizations
Hyper-V Service runs over Windows
and Hyper-V server uses a Windows based kernel
ESXi uses a VMware microkernel and depends on a Linux kernel

57. Takeaway
● With virtualization you can emulate different
architectures
● With virtualization you can run different OSs in
the same server, even made for different
platforms
● Virtualization increases availability
● Virtualization increases scalability
57/60

58. Takeaway
● Virtualization reduces power consumption:
good for environment and to save many money
● Virtualization enables IaaS (Infrastructure as a
Servicere), part of Cloud Computing stack
● There are several alternatives and they offer
different possibilites
● NEVER, absolutely never forget about security
58/60

59. 59/60

60. 60/60
Questions?

61. The Real World of Virtual
Datacenters:
The enabling technology for Cloud Computing
X. Breogán Costa
Yesss, you can do it
at home!
(at your own risk
;)

62. TOC
● An extra of Why virtualization (Microsoft things)
● An extra of disaster recovery
– Just an advice: try to prevent it ;)
● An extra of basic security
● An extra of virtualization platforms
● An extra of... (well, we haven't spoke about this, just
introduce it) Let's speak about cloud platforms
2/28

63. Extra: Why Virtualization?
The Dynamic Datacenter (according to Microsoft)

64. Can your computer be a host
machine?
● Hardware virtualization
– Virtual CPU architecture
● At least, Intel VT-x, AMD-V
● vmx or svm in /proc/cpuinfo (egrep '(vmx|svm)' --color=always /proc/cpuinfo)
● CPU-Z in Windows
● Enabled on BIOS
– + generic/compatible hardware* (servers use to be)
4/28

65. Disaster recovery
● There are several backup tools to prevent this
situation
● Usage of NAS servers
● Programmed backups
– Commonly used snapshots as a base
● Backup keeping policy
● Image sharing
5/28

66. Basic Security
● General risks (according Gartner researches)
– Information security isn't initially involved in the
virtualization projects (40% in 2009)
– Compromise of Virtual Layer (VMM) → could
compromise of all hosted workloads (VMs)...
– … adequate controls on administrative access to
the Hypervisor/VMM layer and to administrative
tools are lacking
6/28

67. Basic Security
● General risks (according Gartner researches)
– Workloads of different trust levels are consolidated onto
a single physical server without sufficient separation
– vNetworks/vSwitchs: lack of visibility and controls on
internal virtual networks created for VM-to-VM
communications blinds existing security policy
enforcement mechanisms...
– … there is a potential loss of separation of duties for
network and security controls
Source article: http://bit.ly/aHzzRB
7/28

68. Basic Security
● Recommendations:
– Be careful with host system interface (shared
resources)
– VM isolation
– Don't use generic and shared administration accounts
(for traceability), even delete generic admin accounts
– Restrict root access at Hypevisor level
– Use the right permissions in user roles definition
– Be careful with roles' permissions hierarchy **
8/28

69. Basic Security
** About user roles
– Roles → templates
– Role permissions have sense at a certain level
– An user have different views depending on his roles
– One user could have different roles at different
datacenter levels
● Combine roles is normal and a good praxis
● Roles combination avoid problems with permissions
hierarchy
9/28

70. 10/28
Security:vSphereexample

71. 11/28
Security:vSphereexample

72. Virtualization platforms
12/28

73. Datacenter Virtualization market in
2012
Note that thanks to RHEV (KVM
based) expansion with Cloud
Computing platforms (i.e: OpenStack)
integration and support, the market
could be different today
13/28

74. VMware vSphere Infrastructure
● Bare-metal hypervisor
– VMware ESXi (before v. 4.0: “ESX”)
– Own microkernel: VMware vmkernel
– It uses (and depends on) a Linux kernel (service console, the 1st vm)
● Management server:
– VMware vCenter Server
– Database (SQL Server / Oracle)
● Management Client
– VMware vCenter Client app
● Extra Tools (HA, DRS, Operations Management, ...)
– Some available in vSphere Server by default
14/28

75. VMware vSphere Infrastructure
● Bare-metal hypervisor
– VMware ESXi (before v. 4.0: “ESX”)
– Own microkernel: VMware vmkernel,
– It uses (and depends on) a Linux kernel (service console, the 1st vm)
● Management server:
– VMware vCenter Server
– Database (SQL Server / Oracle)
● Management Client
– VMware vCenter Client app
● Extra Tools (HA, DRS, Operations Management, ...)
– Some available in vSphere Server by default
15/28

76. 16/28
Vmware vSphere Infrastructure

77. VMware ESXi hypervisor
17/28

78. VMware ESXi hypervisor
18/28

79. KVM hypervisor (GPL/LGPL)
19/28

80. Xen hypervisor (GPL)
20/28

81. Xen hypervisor (GPL)
21/28

82. Xen hypervisor (GPL)
● Runs in a more privileged CPU state than any
other SW on the machine
● Memory management and CPU scheduling of
all “domains” (VMs)
● Uses dom0 (the only VM which by default has
DA to the HW.
● From Dom0 the Hypervisor can be managed
and domU's could be launched.
22/28

83. Xen hypervisor (GPL)
● Dom0 is typically a modified version of Linux,
NetBSD or Solaris
● Proprietary version of Citrix and also Citrix
management tools for Citrix XenServer
23/28

84. KVM/Xen datacenter/virtual cluster
management tools
● RHEV (Red Hat Enterprise Virtualization)
● oVirt [Red Hat Inc.]
– RHEV is based in oVirt + another tools
● ConVirt [Convirture]
● OpenQRM (IaaS Cloud)
● ...
24/28

85. Microsoft Hyper-V Service & Server
● Hyper-V Windows Server Service
– Released as a Windows Server 2008 R2 service
● Hyper-V Server
– Released as an independent bare-metal server
based on Windows Server 2012 kernel
● Several features not supported as real pass-
through
25/28

86. 26/28
Related cloud computing platforms

87. Related Cloud Computing Platforms
IaaS Project started by Citrix & Cloud.com
Now Apache SW Foundation
Works with KVM, Xen and vSphere
Supports AWS API
Works with KVM, Xen but also with VMware vSphere, Hyper-V
Supports AWS API
Project started by Rackspace Hosting and NASA
Works with KVM, Xen and vSphere
Open source (Eucalyptus Systems Inc) SW to build AWS
Works with vSphere
It seems vCloud Director is not as successful as vSphere
27/28