1. Citrix Support Secrets
Webinar Series
Implementing and Troubleshooting PVS
Gareth O’Brien, Escalation Engineer, Citrix Support
February 21, 2013

2. Introduction
• Gareth O’Brien, escalation engineer
• Intermediate to advanced PVS knowledge expected
2 © 2012 Citrix | Confidential – Do Not Distribute

3. Agenda
• Planning
ᵒDatabase, storage, network, farm layout
• Installing
• Configuring
ᵒFiles needed for booting, boot process
• Troubleshooting
ᵒStream Service
ᵒConsole, MAPI and IPC
3 © 2012 Citrix | Confidential – Do Not Distribute

4. Implementing PVS

5. Planning
• Planning the database configuration
• Hardware requirements
ᵒClient & Server considerations
• Network
• Farm layout considerations
5 © 2012 Citrix | Confidential – Do Not Distribute

6. Database
• Size of farm determines database server choice
• Fault tolerance and failover
ᵒMirroring
• dbOffline – if all else fails
• SQL native client needed for mirroring
• Permissions
ᵒDbcreator & securityadmin for Configuration Wizard
ᵒDb_datareader, db_datawriter & execute permissions on stored procedures
6 © 2012 Citrix | Confidential – Do Not Distribute

7. Hardware Requirements
• Supported Operating systems
ᵒWindows Server 2003 R2, 2008, 2008 R2
• Minimum spec of the PVS server
• Known hardware issues
ᵒhttp://support.citrix.com/article/CTX131611
• NIC Teaming
ᵒMake sure hardware is not on above list
ᵒIf streaming doesn’t work with team, try break the team
Processor: 3GHz
RAM: 4GB
NIC: 1GB per 250 targets
7 © 2012 Citrix | Confidential – Do Not Distribute

8. Network
• Configure UDP & TCP Ports
ᵒServer to server ports must be the same on all servers
ᵒTarget device communication ports
• 6910-6930
ᵒSOAP Server
• 54321 & 54322
ᵒTFTP & TSB
• 69 & 6969
• Firewall must be configured to allow all these ports
ᵒIf in doubt, allow ANY/ANY to test
• PortFast must be enabled
8 © 2012 Citrix | Confidential – Do Not Distribute

9. Plan the farm Provisioning
Services
Farm
License MSSQL Shared
Server Server Storage
Database
Provisioning Servers Provisioning Servers
Site 1 Site 2
Device Device Device Device Device
Collection Collection Collection Collection Collection
9 © 2012 Citrix | Confidential – Do Not Distribute

10. Installing PVS

11. Installing PVS
• Server installation
ᵒIf installing a new version of .NET, reboot when prompted
ᵒSpecify the mirror failover if applicable
• Console installation
ᵒNeeds PowerShell 2.0
ᵒIf upgrading, make sure you remove the old version
• Target installation
ᵒMake sure you set the target to boot from network first
ᵒIf imaging fails
• Start with a vanilla install & try to image. Remove any teams
• XenConvert & BNImage are other options
11 © 2012 Citrix | Confidential – Do Not Distribute

12. Configuring PVS

13. Configuring PVS
• PXE Boot
• Two Stage Boot (TSB)
• Boot Process
• Stores
• IPC Key
13 © 2012 Citrix | Confidential – Do Not Distribute

14. PXE Boot
• Must configure DHCP
ᵒ Option 66: IP of PVS Server
ᵒ Option 67: Name of bootstrap (ardbp32.bin)
• Uses TFTP Protocol
• TFTP listens on one NIC only
ᵒ Configure using c:program Filescitrixprovisioning servicestftpcpl.cpl
• TFTP Downloads bootstrap, ARDBP32.BIN
• Other TFTP servers can be used
14 © 2012 Citrix | Confidential – Do Not Distribute

15. Two Stage Boot (TSB)
• Used in situations where DHCP or PXE cannot be used
• Uses ISO burned from PVS Server
• Uses a boot strap called tsbbdm.bin
• Listens on NIC configured for streaming in configuration wizard
ᵒ Uses port 6969
• Uses Two Stage Boot service on PVS server
15 © 2012 Citrix | Confidential – Do Not Distribute

16. Boot Process
• Target device either uses DHCP to obtain an IP or loads static IP (for TSB)
• Once it has an IP, it will contact the 1st logon server specified in the bootstrap
• The logon server will then create a context for the device, determine which
vDisk the target is assigned and the least loaded server
ᵒ This calculation is based on both RIMS information and the load information in the database
ᵒ RIMS uses notifier and inventory service to update load information
ᵒ If subnet affinity is enabled, this will alter the calculation
• Once a server has been selected, a streaming IP and port will be provided
ᵒ IP information will be in HEX, not decimal
16 © 2012 Citrix | Confidential – Do Not Distribute

17. Boot Process (cont)
• Excerpt from log of a target logging on
17 © 2012 Citrix | Confidential – Do Not Distribute

18. Boot Process (cont)
• As streaming begins, the boot control device (BCD) is downloaded
ᵒThis phase is referred to as single I/O
• Once Single I/O has finished, we enter the admin phase
ᵒBNIStack handshake
ᵒVdisk configuration (name, mode, write cache type) is requested and provided
ᵒTarget device requests personality string
• HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesBNIStackParameters
- DWORD EnableGetComputerName set to 1
ᵒAD information requested
• Multi I/O phase entered
18 © 2012 Citrix | Confidential – Do Not Distribute

19. Stores
• Configure vDisk stores
ᵒCentralised vs Distributed stores
• Centralised
ᵒWorks well in HA environment 
ᵒLow storage overhead 
ᵒHigh cost 
ᵒPotential SAN/Storage issues 
• Distributed
ᵒLow implementation cost 
ᵒHigh storage overhead 
ᵒConfiguration 
• IPC is the method for communication between servers for stores
19 © 2012 Citrix | Confidential – Do Not Distribute

20. IPC Key
• Defines which NIC to use for IPC communication in a multi NIC environment
ᵒHKEY_LOCAL_MACHINESoftwareCitrixProvisioningServicesIPC
• Reg_sz called IPv4Address with the IP of the NIC for IPC
• Without it, stores, replication, load balancing etc won’t work
• Affects stream service
• Manager key for MAPI works the same way
ᵒHKEY_LOCAL_MACHINESOFTWARECitrixProvisioningServicesManager
• RegSZ called GeneralInetAddr with the IP of the NIC and port
- Eg 10.1.1.2:6909
20 © 2012 Citrix | Confidential – Do Not Distribute

21. Troubleshooting PVS

22. A more structured method of troubleshooting PVS

23. Troubleshooting PVS
• Stream service troubleshooting
ᵒThe basics
ᵒTracing
ᵒDumps
ᵒLog Analysis
ᵒCommon Issues
• SOAP/Console
ᵒUnable to validate
ᵒConsole timeouts
23 © 2012 Citrix | Confidential – Do Not Distribute

24. Stream Service Basics
• The devil is in the detail.
ᵒNetwork Topology
ᵒHardware of target, server & database
ᵒStorage type & location
ᵒWrite cache type
ᵒDatabase location, version, HA
ᵒMSInfo32
• Bluescreens
ᵒCommon Image
ᵒ7B
• What does it mean
• Common causes
24 © 2012 Citrix | Confidential – Do Not Distribute

25. Stream Service Tracing
• Network traces
ᵒWireshark trace on the server
ᵒTcpdump on the target
• Make sure you always use –s 0 to not limit capture size
• PVS Logs
ᵒSet to trace level
ᵒC:programdataCitrixProvisioning ServicesLogs
ᵒTarget side logging also
• All logs and traces should be captured simultaneously, while the issue is
happening.
25 © 2012 Citrix | Confidential – Do Not Distribute

26. Stream Service Tracing (cont)
26 © 2012 Citrix | Confidential – Do Not Distribute

27. Stream Service Tracing (cont)
27 © 2012 Citrix | Confidential – Do Not Distribute

28. Stream Service Tracing (cont)
28 © 2012 Citrix | Confidential – Do Not Distribute

29. Crash Dumps
• If possible, take it in private mode
• In 2008, it’s possible to redirect to a non system drive
ᵒhttp://support.citrix.com/article/CTX123642
ᵒNot possible on 2003
• Configure citrix symbols
• Live debugging can be done, but difficult without symbols
ᵒUse live debugger to help set it up http://support.citrix.com/article/CTX131945
ᵒCheck for BNIstack being loaded
• !devnode 0 1
ᵒOn 5.6 check for bnns
• !ndiskd.miniports
29 © 2012 Citrix | Confidential – Do Not Distribute

30. Common issues
• Imaging
ᵒLet the wizard do the work for you
• Always revert to a clean install as a first step
• If the wizard doesn’t work, pre-create the disk and use xenconvert
• BNImage can work well if all else fails
• Computer Account
ᵒIs AD Management configured on the vDisk? Is Active directory correctly configured?
• GPO disabling password update, and password age
• Reset the password within the PVS console
• Check all relevant Microsoft hotfixes are installed – Google!
• BNDevice 1015 in the event viewer – unable to negotiate a new password
• Trace logs will show you the requests
30 © 2012 Citrix | Confidential – Do Not Distribute

31. Performance
• Follow the best practice guide
ᵒhttp://support.citrix.com/article/CTX127549
ᵒhttp://support.citrix.com/article/CTX117374
• Intermediate buffering
ᵒGenerally enabled if free space on target device disk is greater than vDisk, otherwise
disabled
ᵒToggling the state is always worth trying to see the performance impact
• Check interrupt safe mode is not enabled
• Check the network with a network monitor
31 © 2012 Citrix | Confidential – Do Not Distribute

32. SOAP & Console
• Console timeout
ᵒThe console timeout can be set in the registry
• Use DWORD ConnectionTimeout in
HKEY_LOCAL_MACHINESOFTWARECitrixProvisioningServices. Default value is 120s
• Can’t communicate with other servers
ᵒCheck IPC and manager keys
• HKEY_LOCAL_MACHINESOFTWARECitrixProvisioningServicesIPCIPv4Address
• HKEY_LOCAL_MACHINESOFTWARECitrixProvisioningServicesManagerGeneralInetAddr
• Mentioned above
• Load Balancing issues
ᵒCheck the StreamProcess & Inventory logs to make sure there are no issues around
RIMS
32 © 2012 Citrix | Confidential – Do Not Distribute

33. Questions?

34. About Educate | Guide | Support | Succeed
Citrix Services
Citrix Services make sure How we can help
you succeed with your
virtualization programs. Citrix Education – The fastest, most efficient way to
get your team the virtualization skills they need. Online,
on-site or in class.
citrix.com/training
Citrix Consulting – Intensive engagements for
complex, critical or just plain massive projects.
citrix.com/consulting
Citrix Support – Always-on support services that
leverage everything we know about best-practice
deployment and maintenance.
citrix.com/support
© 2012 Citrix | Confidential – Do Not Distribute

35. The desktop virtualization project space with best practice built in.
Check it out
© 2012 Citrix | Confidential – Do Not Distribute

36. Secrets of the Citrix Support Ninjas
• 40 insider troubleshooting tips
• Covering XenDesktop, XenServer, XenApp and NetScaler
• Citrix Support top engineers
• FREE eBook
• Citrix Auto Support
• Now available!
© 2012 Citrix | Confidential – Do Not Distribute

37. Premier Support Calculator
Check it out
© 2012 Citrix | Confidential – Do Not Distribute

38. Work better. Live better.