Tailoring NAS Proxies for Virtual Machines

Caitlin.Bestler@nexenta.com
Alex.Aizman@nexenta.com
Sponsored by:

&

• Traditional tradeoff:
Local vs. Central storage:
– Central storage migrates very simply.
• Just complete last work from old location
– Then start at the new location.
• Extracts non-local latency tax on every access.
– Purely local storage makes migration a challenge.
• Must migrate VM image and local storage.
• And synchronize them.

Xen Summit Aug 2, 2011 –
Caitlin.Bestler/Alex.Aizman@nexenta.com 2

• SSD vs HDD:
–SSDs provide most IOPS per $.
• SSDs need to be on host, or at
least near the VM, or network
latencies minimize SSD speed.
–HDDs provide most TBs per $.
• HDDs provide capacity, not
processing speeds.
Caitlin.Bestler/Alex.Aizman@nexenta.com

• A very common solution currently features Host

DomU
a Block Backend that co-ordinates a local App

block device (typically SSD) with remote VFS
File System

block device (still NAS or SAN). Block
Driver

• Problem: Device Domain

– Any cache solution only stores a portion Block Backend SSD Proxy

of a VM‟s storage locally. Remote Block Local Block
Driver
Access

– With NAS Proxy the “portion stored
locally” can be understood as files and File System

Block
NAS Client
directories. Network
Driver
Network

• A block-oriented solution never sees Driver Driver

more than a collection of blocks. Local Block
Device

• Many solutions are enhanced or External
NAS
External
SAN
made possible with File-orientation. Server(s) Server(s)

Caitlin.Bestler/Alex.Aizman@nexenta.com 4

Host
• DomU can support both NAS DomU App

access and Virtual Disk Access VFS

– Most VMs will prefer directly-attached boot File NAS Client
System
drive. Block Network
Driver Driver

• Backend can map virtual drive to Device Domain
Block Network
NAS. Backend Backend

• Meanwhile NAS Client can access
NAS Proxy

Network Block

NAS Proxy via local internal Driver Driver

network. Local Block
Device

• NAS Proxy can utilize local SSD External
NAS
Server(s)
drive to optimize either.
5 Xen Summit Aug 2, 2011 –

• NAS Benefits
– Files enable understanding of context:
– Deduplication of files is far easier than deduplication of block.
– Hierarchical Storage Management works on files, not blocks.
• Traditional NAS Proxy Caveats
– Traditional NAS is unaware of VM-specific semantics.
– Excessive syncing – Historical assumptions are that files are
shared by default.
– Proxy overhead for non-cached files – Historically, the only easy
configuration was to proxy all NAS access.
• VM Storage Proxy provides full benefits of NAS without
limitations of traditional NAS Proxies.


• Distributed NAS protocols such as NFSv4 and
DFS/CIFS have features that make NAS proxies far
more efficient.
• They allow efficient caching via exclusive delegation
and oplocking (“opportunistic locking”).
– Proxy can temporarily claim a file.
• They have advanced session features to protect
against data loss from connection loss.
– File reverts to central NAS server if Proxy dies.
– System recovers from network partitions and temporary
outages of distributed servers and/or proxies as long as
they all have persistent redundant storage.
8 Caitlin.Bestler/Alex.Aizman@nexenta.com

• Client can simply mount from NAS
server.
VM Client
• NAS server can refer select
directories to a NAS “Server” that
is actually the NAS Proxy. NAS Proxy /
Server
– NAS Proxy/Server must act as Referral

a NAS Server for NAS Replication
replication procedures. Cached
Files
• Client does not need to identify Directory access

itself.
– NAS server can infer identity
from IP address. All
NAS Server
Directories
• Advantage that Proxy/Server is
enabled/disabled for each directory
from the central NAS Server.


• NAS Proxy can act as a NAS
VM Client
client and claim an exclusive
delegation on a file.
• It can then treat its cached copy Local copy can
be treated as
THE copy.
as the copy.
Cached NAS Proxy /
• It must replicate it back to the Files Server
central NAS server before
Delegation
ending the delegation. Granted
Claim Exclusive Delegation
• Keep-alive messages track the
NAS client with the delegation. All
NAS Server
Directories
• Protocol rules already cover
loss of contact/failure
scenarios.

• Generic NAS Proxy does not understand that it is the sole
writer for the VM Private Directory.
• NAS Proxy management is not integrated with VM
Management.
– The NAS system must detect that the VM using its VM Private
Directory has migrated.
• Proxies only optimize storage for selective files – but all
traffic goes through them.
– If there are directories/files that are not worth caching the clients
would ideally just go directly to the real server.
– NAS Referrals can be applied selectively, but Management Plane
controls are awkward for dynamically controlling referrals.
• Optimized cache sharing across multiple clients is
challenging without understanding of VM life-cycle.

• Allow VM‟s files to be stored and archived as
files.
• Allow storage associated with commodity servers
to be deployed effectively for VMs in combination
with NAS storage.
• Don‟t replicate to local storage those files that
probably won‟t be accessed anyway.
• Allow very thin provisioning of local storage.
– SSDs are designed to provide IOPs cost effectively,
not storage.
– If IOPS are not needed, avoid cost of redundant
storage.

• The VM NAS Proxy is a specialized NAS Proxy that is optimized to
support VMs and the Hypervisor Platform supporting them:
• A VM Private Directory is maintained for each VM:
– A per VM directory in the NAS global name space that
holds files that are “local” for this VM.
– A “platform” sub-directory hold virtual disk images and
other files used by the Hypervisor and/or Dom0.
– A “mount” sub-directory holds NAS mount points that
are available only to this VM.
• These are implicitly named. The client does not know the
path to its VM Private Directory because it does not know
which VM it is.
• A “proxy” sub-directory holds files used by the VM NAS Proxy
for this VM, such as the NAS Response Cache.

14

VM NAS Proxy as a VM. VM NAS Proxy for a Rack
Virtualization Host Rack (or other enclosure)
Virtualization Host

Hypervisor Platform VM NAS Proxy
VM13478 VM
Hypervisor Platform VM13478
VM
/foo VM NAS Proxy
/foo
/bar
/bar

NAS VMS

/vmexports /vmexports
/vm13478 NAS VMS /vm13478
/platform /platform
/mounts /mounts
NAS NAS
/foo /foo
` /bar ` /bar
/vm13483 /vm13483
/platform Virtualization Management
Virtualization Management /platform
/mounts /mounts
/foo /foo
/bar /bar


• Virtualization Management
– Unchanged, but new information is available to it.
• NAS VMS (Virtualization Management System)
– Extends Virtualization Management to assign VM NAS Proxies.
• Hypervisor Platform
– The Hypervisor and supporting domains.
• NAS system with NFSv4/v4.1 and/or DFS/CIFS.
– Unchanged, but optional enhancements are defined.
• VMs
– Unchanged, but optional enhancements for NAS Client are
defined.
• VM NAS Proxy
– A specialized NAS Proxy optimized for support of VMs
16

• A VM NAS Proxy can control what is
stored locally through several techniques:
– Which sub-directories of the VM‟s private
directory are referred to it.
– Whether it provides local storage for specific
files within those directories.
– Which blocks of a file that is stored locally are
actually stored locally, vs. being accessed
over the network.
– pNFS-aware clients can be enabled to access
network block servers directly.
17

• VM will be migrated from Current Host Virtualization Host
Current Home

to a new Host.
10.1.3.115Hypervisor Platform
VM

• VM has public IP address, which 192.168.1.115
10.1.1.1
192.168.1.1

Virtualization Management will migrate. VM NAS Proxy
10.1.1.2

• VM has private IP Address which it will
192.168.1.2

use in new network without realizing that
it is on a new network. Planned Home

• New VM NAS Proxy will have same Virtualization Host

private IP address on the new network. 10.1.3.115
Hypervisor Platform
10.1.2.1
192.168.1.1 NAS
192.168.1.115

• New VM NAS Proxy does not respond
10.1.2.2
192.168.1.2

to the VM‟s private IP Address until it is
ready. VM Storage Proxy


• By assigning VM NAS Proxies to an Internal Network
migration of a VM can be made transparent to the VM.
– All Internal Networks can look alike.
– Just as almost all home networks are 192.168.1.*
• NAS client does not need to know about the migration.
– NAS clients will see a dropped connection to a NAS server.
– NAS clients will assume the NAS server is simply rebooting.
– Recovery will be complete and transparent to client layers.
– No change to the IP address of the VM NAS Proxy the Client
uses.
• As far as the client is concerned it just reconnected to
the same server
– It even has the same response cache.
– The response cache is stored within the VM Private Directory.


• VM will be migrated from Current
Current Home

Virtualization Host

Host to a new Host.
– 10.1.3.115 is migrated.
VM
Hypervisor Platform

10.1.3.115
10.1.1.1

• VM is still instructed to use the old
VM NAS Proxy (10.1.1.2). 10.1.1.2 VM NAS Proxy

– It will continue to use it until explicitly
redirected to the new VM NAS Proxy ac
ce
ss

d
ue
(10.1.2.2). Co
nt
in

Planned Home

– This is very network inefficient. Virtualization Host

– Current RTT measurements and
Hypervisor Platform
10.1.3.115
congestion window will be wildly 10.1.2.1
NAS

incorrect. 10.1.2.2

– VM NAS Proxy must accept clients
from anywhere.
VMNAS Proxy


• Distributed NAS features for Referral and
Delegation have nothing to do with each other.
– Referral/redirection is a Management Plane
feature.
• It is used to refer/redirect clients to other NAS
servers.
– Delegation/oplocking is a Control Plane feature.
• It is used by NAS clients.
• A VM NAS Proxy integrates them in
collaboration with NAS servers.


• VM NAS Proxies use Delegation/Oplocking to
exclusively claim specific sub-directories
within a VM Private Directory.
– This is an enhancement to a NAS server.
• Supporting NAS servers will then
automatically direct referrals for the claimed
sub-directories to that VM NAS Proxy for the
duration of the delegation/oplock.
– Reverting is automatic, preferable to having to
reverse a management plane edit.

• Only the correct clients are referred,
• Other clients are refused access.
• Allowed access types:
– Hypervisor Platform access a „platform‟ sub-
directory for virtual disk images and snapshots.
– VM itself accesses NAS mounts in a „mount‟ sub-
directory.
– VM NAS Proxy keeps its own data in a „proxy‟
sub-directory.
• This is where the NAS Response Cache is kept.


2

3 1

VM NAS Proxy

NAS
NAS
Server
Server
Locally
Cached
Files

• 1 – VM NAS Proxy takes exclusive delegation on a directory.
• 2 – Client referencing delegated directory is referred to VM NAS
Proxy
• 3 – Client accesses directory on VM NAS Proxy

• Progression of Solutions for Controlling File
Locations:
– Without NFSv4 Referrals/DFS Redirections:
• Load balancing requires editing each client‟s configuration.
• Or directing all traffic through a proxy.
– Using just standard Referrals/Redirections:
• Central control of which directories are referred
• But only via Management Plane edits of exports.
– Integrating Delegation to allow Automatic Referrals:
• Dynamic control over which directories are referred is
exercised by the VM NAS Proxy and/or VM NAS
Management.
• Dynamic referral automatic ends when delegation ends.


• One of the features of the VM NAS Proxy is that it can
dynamically control which of a VM‟s private directories
are referred to it.
– But standard granularity for referral is an entire directory.
• Partial Referrals is a new technique to publish alternate
referrals for subsets of a directory:
– Each partial referral identified by a regular expression.
– The alternate server is then specified for that regular expression,
which can be “revert to default” to ignore the main referral.
– This information is published as named attributes for the
directory.
• If a NAS client does not understand them then it will simply
go to the default server for the directory
• The default server then acts as a proxy for the actual server.


2B

1
2A

VM NAS Proxy

NAS
NAS
Server
Server
Locally
Cached
Files

• 1 – VM NAS Proxy advertises portions of directory that Client should access directly
on the home NAS server.
• 2a – Client continues to access non-excluded portions of directory on the VM NAS
Proxy, which caches them locally.
• 2b – But now accesses files covered by partial referral on the home NAS Server

1
2 3

VM NAS Proxy

NAS
NAS
Server
Server
Locally
Cached
Files

• 1 – VM NAS Proxy advertises portions of directory that Client should access
directly on the home NAS server.
• 2 – Client does not understand or ignores the partial referral, so all access for the
directory goes through the VM NAS Proxy
• 3 – When files are not stored locally access is proxied via the NAS Server.
31

• When the VM‟s NAS Client
indicates that it understands
pNFS, the VM NAS Proxy Locally Cached Blocks

can supply pNFS layouts Logical Physical
Block # Block #
that mix references to its Logical Physical
Proxy pNFS Layout

own storage and NAS Block # Block # Logical
Block #
Local
Volume ID
Physical
Block #
storage. Logical Physical
•
Volume ID
Referencing pNFS blocks Block # Block #
Logical Physical
directly is safe, because the Block #
Volume ID
Block #
VM NAS Proxy is the only pNFS Layout Logical Local Physical
client allowed to update the Logical Physical
Block # Volume ID Block #
Volume ID Logical Physical
file while it is claimed. Block # Block #
Block #
Volume ID
Block #

• pNFS-aware clients can Logical
Block #
Volume ID
Physical
Block #
Logical
Block #
Volume ID
Physical
Block #
access unmodified and Logical
Volume ID
Physical Logical Physical
Block # Block # Volume ID
uncached blocks without Logical Physical
Block # Block #

overhead of going through Block #
Volume ID
Block #
the proxy.

33

• Named attributes can identify files that are
to be stored only in the local cache.
– The file is never updated on the NAS server.
• Regular expression can easily identify
common temporary files.
• These files would be stored only on local
SDD
– never consuming any network bandwidth.

34

• VM NAS Proxy is only
on the path for a VM pNFS Block
Server
accessing storage Direct access to un-cached blocks

when it wants to be. VM

– Control is fully dynamic.
Default Access, may be referred to VM Storage Proxy
Access to non-referred directories
Access to partially un-referred files

• Access to Non- NAS
Server

optimized Access to referred directories, files

directories/files can
VM NAS PRoxy
bypass the Proxy.
• VM NAS Proxy is only
identified as referral
target for the correct
clients.

Tailoring NAS Proxies for Virtual Machines

Tailoring NAS Proxies for Virtual Machines

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (6)

Ähnlich wie Tailoring NAS Proxies for Virtual Machines

Ähnlich wie Tailoring NAS Proxies for Virtual Machines (20)

Mehr von The Linux Foundation

Mehr von The Linux Foundation (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Tailoring NAS Proxies for Virtual Machines