VMworld 2012 Featured speakers: - Anton Gostev, Veeam Software, @Gostev - Doug Hazelman, Veeam Software, @VMDoug Learn more about Veeam Backup & Replication - http://www.veeam.com/vmware-esx-backup.html Veeam® Backup & Replication™ provides powerful, easy to use, and affordable data protection for virtualized applications and data on VMware vSphere. It unifies backup and replication in a single solution, and its patented vPower® technology leverages virtualization to reinvent data protection. There are no agents to manage, no need to babysit backup jobs, and you can verify the recoverability of every backup, every time. Veeam Backup & Replication supports vSphere 5.1.

1. SPO3292
Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug
#vmworldsponsor

2. Quick Overview of v6 Architecture
 Backup servers
Backup proxy servers
Backup repositories
 ―Automated everything‖
 Intelligent load balancing
 Centralized management via Enterprise Manager

4. Scaling your backups
3 simple rules

5. Scaling your backups
1. Keep the management server happy
● Disable default proxy (requires 6.1 or later)
● Allocate enough RAM for job manager processes
● Keep concurrent (running) jobs under 100 per management server
2. Backup proxy servers: the more, the better?
● No! Too much load on storage and network
● Use max concurrent tasks limit on proxies or repositories
● Watch for ―job timed out waiting for resources‖ messages
3. Backup repository considerations
● Be careful with the reversed incremental backup mode!
● Limit concurrent jobs on backup repositories to a reasonable amount
● Use ingest rate throttling for cross-SAN backups

6. A word on backup repositories
 Don’t underestimate the importance of performance!
● By far, the most commonly reported bottleneck
 What makes the best backup repositories?
● Windows or Linux server (can be same as backup proxy server)
● Local storage, DAS or SAN mounted for physical server
● pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI
for virtual server
 Sub-optimal backup storage
● NAS or network share
● VMDK on VMFS (size and recoverability considerations)
 RAID level
● If you can afford it, use RAID10 (again, performance)

7. Processing Modes
All you need to know

8. Direct SAN Access: The good
 Fastest processing mode
 Least impact on production
● Backup processing is fully offloaded to dedicated backup proxy servers
● Backup traffic is isolated to the storage network (aka LAN-free)
 Does not impact consolidation ratio, so cheapest too

9. Direct SAN Access: The bad
 Supports block storage only
● FC (fibre channel): physical backup proxy server only!
● iSCSI: physical and virtual backup proxy servers both supported
 Physical backup proxy server requirement for FC SAN
● Might not go along well with your virtualization project
● Consider repurposing older servers
 Might be hard for beginners to setup
● See Veeam Forums FAQ for step-by-step guide
 Manual datastore mapping might be required
● For certain SANs, B&R may not able to detect proxy connectivity

10. Direct SAN Access: The ugly
 What’s the worst that could happen?
 Windows re-signaturing your VMFS LUNs!
● vSphere will no longer recognize datastores
● Don’t panic, VMware Support should be able to fix
 Three easy ways to get into trouble
1. Windows Explorer automounting new volumes (but not with Veeam)
2. Clicking Disk Management snap-in popup without reading
3. Giving Local Administrator rights to random people

11. Direct SAN Access: The safe way
 Present VMFS LUNs to backup proxy server as read-only
● Most SANs support it these days—chase your vendor if yours does not
 Disable automount on your backup proxy servers
● Do it the right way: use SANPolicy Windows setting!
● Veeam backup proxy server setup does this automatically for you
 Disable Disk Management snap-in with Group Policy
User Configuration > Administrative Templates
Window Components > Microsoft Management Console >
Restricted/Permitted snap-ins > Disk Management
 Keep Local Administrator rights on backup proxy servers
to yourself
● Cannot really do this for default proxy due to FLR requirement
● Another reason to use dedicated backup proxy server!

12. Direct SAN Access: Tips & tricks
 Got a fast SAN? Get a modern backup proxy server!
● Multi-core CPU (compression) and fast RAM (inline deduplication)
 Update firmware and drivers across the board
 Disabling MPIO may increase performance
 iSCSI SAN? Tweak TCP/IP on backup proxy
netsh interface tcp set global autotuning level = disable
 Increase read-ahead buffer
● Default is 4MB (optimal setting for most SANs)
● To change, create the new value in bytes:
VddkPreReadBufferSize (DWORD)

13. Hot Add: The good
 Easy to setup—very little planning involved
● Any Windows VM can be made a Hot Add backup proxy
 Fast data transfers with any storage
● Direct storage access (albeit through ESXi storage stack)
 Supports all types of storage (including NFS)
● Shared storage: at least 1 backup proxy server per vSphere cluster
● Local storage or DAS: at least 1 backup proxy server per host
 Use your existing Windows VMs (save on licensing)
● Data processing engine process runs with lower priority (6.1)
● Further CPU usage reduction in 6.5
 Allows for 100% virtual deployment

14. Hot Add: The bad
 Not as mature as other modes
 Affects your consolidation ratio
● Backup proxy servers take host resources
● Ultimately means more ESXi hosts, and more VMware licenses
 Hot Add process itself is slooow
● Can take up to 1–2 minutes to complete for each VM—adds up quickly!
 Hot Add as a vSphere feature has a number of limitations
● Good news—many are being removed as VDDK matures
● See FAQ on Veeam forums FAQ for the complete list

15. Hot Add: The ugly
 Snapshot removal problems due to locks
● Veeam B&R: multiple hooks in place to work around
 CBT must be disabled on backup proxy VM
● Prevents stun on Hot Add due to CBT initialization
 NFS-specific issue
● Extended VM stun on hot remove in some scenarios

16. Hot Add: Tips & tricks
 Add extra virtual SCSI controller to backup proxy server
● A single SCSI controller can have a maximum of 16 disks attached
● Concurrent jobs on the same backup proxy server can result in more!
 Keep vSphere and Veeam up to date
● Single block size in VMFS5 removes the most common hot add issue
● Latest Veeam Backup & Replication will have latest and greatest
VDDK version
 Try increasing read-ahead buffer
● Seems to really help with certain NFS storage
 Avoid cloning backup proxy VM
● For example, to provision additional backup proxies

17. Network (NBD): The good
 Easy to setup—in fact, no setup is required
● Any existing server (physical or virtual) would do
 Supports all types of storage, including NFS
● Server placement does not matter (unlike with Hot Add)
 Very quick to initialize data transfer
 Can be quite fast—with 10Gb Ethernet

18. Network (NBD): The ugly
 Painfully slow performance on 1Gb Ethernet
● Average speed reported is 10-20 MB/s
 Leverages ESXi management interface

19. Network (NBD): Tips & tricks
1 Gb Ethernet
 Use for sites with low change rate
● Works faster than other processing modes in such conditions
 Keep at least one Hot Add backup proxy server around
● Full VM and virtual disk restores take forever over NBD
 Keep in mind intelligent load balancing algorithms
● Network backup proxy servers have lowest priority!
 Upgrade to 6.1 or later
● Improved network proxy location awareness

20. One last thing
 This hack significantly reduces supportability!
 Cut up to 5 minutes of processing time per VM by
disabling VDDK logging
 Apply in stable environments only!
● Create the new value and set to 1:
DisableVDDKNetworkOutput (DWORD)

21. Deduplicating Storage
Yes, you can afford it!

22. Deduplicating storage: The good
 What gives? Global dedupe!
● Deduplication across backup files from different jobs
● Perfect for long-term backup archival
 Top hardware appliances among Veeam users
● EMC DataDomain
● ExaGrid
● HP StoreOnce
 Top software appliances among Veeam users
● ZFS-based appliances
● StarWind
 Windows Server 2012 dedupe is awesome

23. Deduplicating storage: The bad
 Hardware appliances are expensive
● Although they do provide excellent dedupe ratio
 Software appliances are resource hogs
● Both performance and dedupe ratio are sub-par, too
 Windows Server 2012 dedupe is awesome
● Included free of charge—start using it today!
● Provides very decent dedupe ratio

24. Deduplicating storage: The ugly
 Random access performance is lacking
● A problem for all solutions featuring inline deduplication
● Typically insufficient out-of-the-box for large-scale vPower usage
 Exception: post-process deduplication
 ExaGrid
● Raw disk landing zone (full-speed vPower from recent backups)
● Veeam-specific logic further optimizes performance
 Windows Server 2012
● Backups ―land‖ on raw storage at full speed
● Only old backup files are deduplicated—great for vPower
● Decent speed even off already deduped backups

25. Deduplicating storage: Tips & tricks
 Already own storage with inline deduplication?
● Inline data ―rehydration‖ process is what makes vPower slow
 Reduce the block size in Advanced job settings
● WAN (256KB) and LAN (512KB)
● Reduced block size might impact backup performance
 Use Linux-based backup repository
● Large client cache, or even caching client file system (FS-cache),
can significantly improve vPower performance

26. Deduplicating storage: Tips & tricks (continued)
 Keep Veeam dedupe on
 Use incremental backup mode
● Choice of synthetic or active fulls depending on actual storage
 For best dedupe ratio on device side…
● Disable compression (significantly increases amount of data transferred
from backup proxy server to backup repository over network)
 For best backup performance and smallest window
● Keep compression at default
 If you like to avoid extremes…
● Set compression to Low (dedupe-friendly)

27. Deduplicating storage: Tips & tricks (continued)
Got more than one deduplicating storage device?
 Use internal replication to sync backups offsite!
● Extremely traffic-efficient approach
● Many customers use and report great success!
 Keep the backups imported for easy DR
● To automate repository refresh in DR site, use:
Get-VBRBackupRepository -Name "DR_Repository" |
Sync-VBRBackupRepository

28. WAN Accelerators
Your WAN on steroids

29. WAN accelerators: The good
 Two types of WAN accelerators
● Caching WAN accelerators provide significant bandwidth savings with
Veeam replication, but are typically quite expensive
● Transport layer WAN optimizers are unlikely to offer significant
bandwidth savings with Veeam replication, and are usually cheap
 Both improve reliability of TCP
● Long distance wireless or satellite links
● IPsec rekey operations on a VPN tunnel
● WAN links with high jitter, packet loss or occasional drops
 Both allow long-running jobs to finish more consistently
● For example, initial replication over network
 Both improve WAN utilization for most workloads

30. WAN accelerators: What’s hot?
 Top caching WAN accelerators among Veeam users
● Cisco WAAS
● Riverbed
● SilverPeak
 Top WAN optimizers among Veeam users
● Hyper-IP

31. WAN optimizers: Tips & tricks
 Veeam Backup & Replication leaves little room for
bandwidth reduction by WAN optimizers
 Built-in WAN optimizations in v6:
● Multiple TCP streams to maximize throughput
● Network traffic compression
 Consider using on unreliable networks, but don’t
expect them to add you extra bandwidth
● Update to Backup & Replication 6.1 Patch 1 before evaluating
 Veeam Backup & Replication not using all available
bandwidth?
● Increase the amount of TCP streams (default is 5)
DownloadStreamsNumber (DWORD)

32. Caching WAN accelerators: Tips & tricks
 Multiple TCP streams can cause issues!
● Disable multiple streams in B&R traffic throttling settings
 Disable network traffic compression in Veeam
● Low (dedupe-friendly) compression level might be a better option
 If required, have network admins configure bypass on
Veeam backup proxy servers to avoid polluting caches

33. Veeam Backup & Replication 6.5
We never stand still

34. What’s coming in 6.5
 Veeam Explorer for Microsoft Exchange
 Veeam Explorer for SAN Snapshots
 VMware vSphere 5.1 support
 Windows Server 2012 support

35. Veeam Explorer for Microsoft Exchange
 Visibility into Exchange VM backups
● Immediate: No need to provision storage, restore the VM or
restore the mailbox store
● Agentless
● Requires no special backups or metadata collection—even works with
existing Veeam backups (and SAN snapshots)
 Free!
● Included in all versions of Veeam Backup & Replication 6.5,
including Free Edition
● Eliminates need for expensive standalone tools licensed per-mailbox
 Currently available in ―exclusive beta‖
Just restored a 145 GB #MSExchange Public
Folder database in 2 min. using @veeam. Then
restored a single item all under 10 min. SWEET!
The new #veeam explorer for exchange
looks veeamy. That's right, I just made up a
new word (it means awesomesauce)
Really excited with the new @veeam Explorer
beta for #Microsoft Exchange VM backups - we
have lots of interested customers ready for this!
Even the *beta* of @veeam Exchange
Explorer works a treat. Saved literally, hours of
work.. and saved my bacon. #recommend

36. Veeam Explorer for Microsoft Exchange (continued)
 Capabilities
● Browse: familiar Explorer-type interface
● Search: familiar Outlook-like Find, including Advanced Find
● Export: export to PST file, MSG file or attachment
 Uses cases
● E-discovery
● Item-level restore: export and send to affected user
● Mailbox archive
 Supports Exchange Server 2010

37. Veeam Explorer for SAN Snapshots
 Veeam restores from SAN snapshots
 Supports tiered data protection strategy
 Perform all restores through familiar, easy-to-use
Veeam interface
 Supports HP StoreVirtual VSA and HP LeftHand
SAN snapshots + Veeam restore =
Best RPOs and RTOs for operational recovery

38. Veeam Explorer for SAN Snapshots (continued)
 Fast: recover entire VM or individual items in < 2 minutes
● Fully automated: clone & promote snapshot, present to vSphere, clean up
● Restores directly from VM files on the SAN snapshot: no staging or
intermediate restores required
 Flexible
● Specific VM
● Individual guest files: Windows, Linux, et al
● Individual Microsoft Exchange items
 Free
● Worry-free: automated process eliminates human errors and protects
integrity of SAN snapshots and production LUNS
● Agent-free: no agents to deploy on hosts or VMs
● Literally free: included in all editions of Veeam Backup & Replication 6.5,
including Free Edition

39. Questions? Comments?

40. Thanks for attending!

41. FILL OUT
A SURVEY
AT WWW.VMWORLD.COM/MOBILE
COMPLETE THE SURVEY
WITHIN ONE HOUR AFTER
EACH SESSION AND YOU WILL
BE ENTERED INTO A DRAW
FOR A GIFT FROM THE
VMWARE COMPANY STORE

42. SPO3292
Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug
#vmworldsponsor