Logical Domains (LDoms) is a virtualization technology from Sun Microsystems that partitions a physical SPARC server into independent virtual machines called logical domains. Each domain functions as a separate server with its own operating system and resources. Key features of LDoms include the ability to dynamically add or remove CPUs and memory from domains while they are running. Future roadmap items include live migration of domains between physical servers and dynamic configuration of virtual I/O. LDoms improves server utilization by allowing multiple independent systems to run concurrently on a single physical platform.

1. USE IMPROVE EVANGELIZE
Logical Domains (LDoms)
Liam Merwick
LDoms Developer
Sun Microsystems – SPARC Platform S/W Group

2. USE IMPROVE EVANGELIZE
Contents
• Background
• Features
• Architecture
• Components
• Future Roadmap
• Configuration Examples
• Q&A
Page: 2

3. USE IMPROVE EVANGELIZE
Background
Page: 3

4. USE IMPROVE EVANGELIZE
Consolidation
• Conventional Unix server utilisation is 7 to 15%
• Data center compression
> Goal: Increase utilisation of equipment
10% Utilisation
20%
Time
10%
Page: 4

5. USE IMPROVE EVANGELIZE
Solutions from Sun
Hard Partitions Virtual Machines OS Virtualisation Resource Mgmt.
App
Server Database Identity
Server
File
Server
Web
Server
Mail
Server
Calendar Database Web
Server Server
SunRay Database
Server
App
Server App
OS
Server
Multiple OSes Single OS
Trend to flexibility Trend to isolation
Dynamic System Domains SPARC Logical Domains Solaris Containers Solaris Resource Manager
Solaris xVM (Zones + SRM) (SRM)
VMware ESX Solaris Containers
for Linux Applications
Microsoft Virtual Server
Solaris Trusted Extensions
Page: 5

6. USE IMPROVE EVANGELIZE
Virtualisation for SPARC platforms
• New “sun4v” architecture Operating
System
Solaris X Solaris X
update (genunix)
(genunix)
sun4u code Solaris X (sun4v)
sun4v
US-Z CPU
interface
code
SPARC hypervisor
CPU “Z” SPARC CPU
Platform
Page: 6

7. USE IMPROVE EVANGELIZE
Virtual Machine for SPARC
• Thin software layer between OS and
platform hardware sun4v virtual machine
User User User
App App App
• Hypervisor + sun4v interface
• Virtualises machine HW and isolates OS from Solaris
register-level
• Delivered with platform not OS OpenBoot
• Not itself an OS
Hypervisor
SPARC hardware
stable
interface
“sun4v”
Page: 7

8. USE IMPROVE EVANGELIZE
Logical Domains Technology
• Virtualisation and partitioning of machine resources
> Each domain is a full virtual machine, with a dynamically reconfigurable sub-
set of machine resources, and its own independent OS
> Protection & isolation via SPARC hardware and LDoms Hypervisor firmware
LDom A LDom B LDom C LDom D
OS
Environment
of choice
Linux FreeBSD
LDoms
Hypervisor
CPU CPU CPU CPU CPU CPU CPU CPU
Platform Memory Memory Memory Memory
Hardware
I/O I/O
Page: 8

9. USE IMPROVE EVANGELIZE
Features
Page: 9

10. USE IMPROVE EVANGELIZE
Fundamentals
• Each virtual machine should appear as an entirely
independent machine
> own kernel, patches, tuning parameters
> own user accounts, administrators
> own disks
> own network interfaces, MAC & IP addresses
> Start, stop and reboot independently of each other
• Paravirtualised OS
Page: 10

11. USE IMPROVE EVANGELIZE
Features
• Add and remove cpus while OS is running
• Protection and isolation via SPARC hardware and
LDoms Hypervisor firmware
• Service domains for I/O
> direct I/O access
Page: 11

12. USE IMPROVE EVANGELIZE
Features – UltraSPARC-T1 specific
• Up to 32 LDoms per box
> CPU threads individually assignable to different LDoms
• Memory assignable to with an 8KB granularity
• Two PCI busses independently assignable to the
same or different LDoms.
• Guest Domains can be configured, started and
stopped independently
> Without requiring a power-cycle of machine
Page: 12

13. USE IMPROVE EVANGELIZE
Features – UltraSPARC-T2 specific
• 64 threads => up to 64 domains
> more domains per box than with UltraSPARC-T1
• On-chip NIU assignable to any domain.
Page: 13

14. USE IMPROVE EVANGELIZE
Architecture
Page: 14

15. USE IMPROVE EVANGELIZE
Architecture
• Logical Domains overview
• Hypervisor
• LDoms Manager
• Virtualised I/O
• Dynamic Reconfiguration (DR)
Page: 15

16. USE IMPROVE EVANGELIZE
Hypervisor Support
• Hypervisor software is responsible for maintaining
separation between domains
> Using extensions built into a sun4v CPU
• Also provides Logical Domain Channels (LDCs) so
that domains can communicate with each other
> Mechanism by which domains can be virtually networked
with each other, or provide services to each other
Page: 16

17. USE IMPROVE EVANGELIZE
LDoms Manager
• One Manager per host HV
> Controls Hypervisor and all its LDoms
• Exposes control interface via CLI
• Maps Logical Domains to physical resources
> Constraint engine
> Heuristic binding of LDoms to resources
> Assists with performance optimisation
> Assists in event of failures / blacklisting
Page: 17

18. USE IMPROVE EVANGELIZE
Direct I/O Logical Domain
App App
• Traditional model App
Logical Domain
owns PCI root
> Existing drivers and devices and tree
continue to work Device Driver
/pci@B/qlc@6
Nexus Driver
/pci@B
Privileged
Hyper Hypervisor Virtual Nexus I/F
Privileged
I/O MMU
Hardware I/O
Bridge PCI
Root
PCI-Express
Page: 18

19. USE IMPROVE EVANGELIZE
Virtualised I/O
Logical Service
Domain A Domain
App Device Driver
App /pci@B/qlc@6
App
App
Virtual Device Nexus Driver
Service /pci@B
Virtual Device
Privileged Driver
Virtual Nexus I/F
Hyper Hypervisor Domain Channel
Privileged
I/O MMU
Hardware I/O
PCI
R Bridge
oot
PCI B
Page: 19

20. USE IMPROVE EVANGELIZE
Virtual (Block) Disk device
Logical Domain Logical Logical Service
1 Domain 2 Domain 3 Domain
App App App vDisk
App App App App App App Server
App App App vDisk
Server
vDisk vDisk vDisk
client client client Device-
Driver driver driver Driver
Hyper- Virtual SAN 2
visor Virtual SAN 1
I/O Bridge
FC-AL
I/F
Page: 20

21. USE IMPROVE EVANGELIZE
Virtual Ethernet device
Logical Logical Logical Service
Domain 1 Domain 2 Domain 3 Domain
App App App V-Ether
App App App App App App
Switch
App App App V-Ether
Switch
V-Ether V-Ether V-Ether
Driver Driver Driver Device-
Driver
Hyper- Virtual LAN 2: 63.24/16
visor Virtual LAN 1: 192.168.0/24
I/O Bridge
Gb
Ether I/F
Page: 21

22. USE IMPROVE EVANGELIZE
Dynamic Reconfiguration (DR)
• Ability to dynamically grow or shrink compute
capacity of an LDom on demand
• No need to re-boot Solaris
• Simply add / remove:
> CPUs
> Memory (future)
> I/O (future)
• Improve utilisation by balancing resources between
LDoms
Page: 22

23. USE IMPROVE EVANGELIZE
LDoms Manager
Logical Logical Logical Service
Domain 1 Domain 2 Domain 3 Domain
App App App LDoms
App App App App App Manager
App App
OS DR OS DR OS DR
Stack Stack Stack
Hyper- HV DR
Stack
visor
Page: 23

24. USE IMPROVE EVANGELIZE
Example: vCPU reconfiguration
Logical Logical Logical
Domain 1 Domain 1 Domain 2
App App App
App App App
V V V V V
Hyper- Hyper-
visor P P visor P P P
Example command line operations:
% ldm remove-vcpu 1 Logical_Domain_1
% ldm add-vcpu 1 Logical_Domain_2
Page: 24

25. USE IMPROVE EVANGELIZE
Components
Page: 25

26. USE IMPROVE EVANGELIZE
Components – Hardware & Software
• Hardware
> UltraSPARC-T1 (Niagara) family of CMT processors
> T1000 Netra CP3060 T6300
> T2000 Netra T2000
> UltraSPARC-T2 (Niagara2) family of CMT processors
> T5120 T5220 T6320
• Operating System
> OpenSolaris (build 70+ recommended)
> Solaris 10 11/06 (U3) or later
> Linux 2.6.23
Page: 26

27. USE IMPROVE EVANGELIZE
Components – Hardware & Software
• Firmware
> 6.4.6 or later
> 6.5.x and 7.0.x now available
• LDoms Manager Software
> 1.0.1 (Solaris package - available as Free download)
> 1.0 (For use with 6.4.x firmware)
• OS can and will be released/updated independently
of Firmware/LDoms Manager
Page: 27

28. USE IMPROVE EVANGELIZE
Future Roadmap
Page: 28

29. USE IMPROVE EVANGELIZE
Latest Release: LDoms 1.0.1
• Rebooting of Control, Service and IO domains
> Without bringing down guest domains
• Support for new platforms
• Updates to Firmware and LDoms Manager package
• Upgrade to Solaris 10 8/07 (U4) recommended
> Independent of LDoms 1.0.1
Page: 29

30. USE IMPROVE EVANGELIZE
Key Features on Roadmap
• Warm Migration
> Suspend and resume
• Live Migration
• Hybrid I/O
• Dynamic Reconfiguration
> Virtual and Physical I/O
> Memory
• Booting/Installing from DVD/ISO
• Using ZFS/SVM volumes
Page: 30

31. USE IMPROVE EVANGELIZE
Configuration
Examples
Page: 31

32. USE IMPROVE EVANGELIZE
Configuring a machine to run LDoms
• Make resources available (remove from control domain)
primary#> ldm set-mau 2 primary
primary#> ldm set-vcpu 8 primary
primary#> ldm set-mem 8g primary
• Configure services
primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary#> ldm add-vds primary-vds0 primary
primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
Page: 32

33. USE IMPROVE EVANGELIZE
Adding another domain (I)
• Add disk device
primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0
• Adding resources
primary#> ldm create domain1
primary#> ldm add-mem 2g domain1
primary#> ldm add-vcpu 4 domain1
primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1
primary#> ldm add-vnet vnet1 primary-vsw0 domain1
• Show domain constraints
primary#> ldm list -l domain1
Page: 33

34. USE IMPROVE EVANGELIZE
Adding another domain (II)
• Booting the newly created domain
primary#> ldm bind domain1
primary#> ldm start domain1
• Connect to the console
primary#> telnet localhost 5000
Page: 34

35. USE IMPROVE EVANGELIZE
LDoms Community
• Newly Created community
• Website
> http://www.opensolaris.org/os/communites/ldoms
• Alias
> ldoms-discuss@opensolaris.org
Page: 35

36. USE IMPROVE EVANGELIZE
More information
• BigAdmin Portal
> http://www.sun.com/bigadmin/hubs/ldoms/
• Sun Blueprint
> Beginners Guide to LDoms: Understanding and Deploying
Logical Domains
> http://www.sun.com/blueprints/0207/820-0832.pdf
• Training Course
> Solaris 10 Logical Domains Administration (SA-345-S10)
> http://uk.sun.com/training/catalog/courses/SA-345-S10.xml
Page: 36

37. USE IMPROVE EVANGELIZE
Liam Merwick
Sun Microsystems
Liam.Merwick@sun.com
http://blogs.sun.com/merwick