Citrix Provisioning Services (PVS) functionality targets network client machines that boot and connect to provisioning services servers under the guidelines specified in the PXE 2.1 specification. This in-depth technical session will provide a step-by-step detailed overview of the target boot process used by provisioning services versions 5.6.x and 6.x.
In this session you will learn about:
• Components and architecture of the PVS boot process
• Troubleshooting PVS boot issues
• Load balancing and failover technology
1. Citrix Support Secrets
Webinar Series
Troubleshooting Provisioning Services Target
Boot Processes
Konstantin Cvetanov– PVS Escalation Engineer
December 5th 2013
Good Afternoon and welcome to SupportWebinar 2013.My name is Konstantin Cvetanov and I am an Escalation Engineer with the Provisioning Services Escalation Team from our offices Alpharetta, Georgia. The purpose of this presentation is to provide a step-by-step detailed overview of the Provisioning Services (PVS) Boot Process used by Citrix Provisioning Services.
In today’s session we will discuss the PXE specification and how it relates to the PVS boot process. Next we will discuss the PVS boot process in depth and go over some common troubleshooting steps and scenarios. I will provide you with a list of resources that I used in creating this presentation in the event that you want to do further research. And lastly I will open the floor to any questions you may have.
The PXE protocol operates as follows. (CLICK) The client initiates the protocol by broadcasting a DHCPDISCOVER containing an extension that identifies the request as coming from a client that implements the PXE protocol. (CLICK) A DHCP server or a proxy DHCP server implementing this extended protocol sends the client a list of appropriate Boot Servers. The client also receives the name of an executable file on the chosen Boot Server. (CLICK) The client uses TFTP to download the executable from the boot server. (CLICK)(CLICK) Finally, the client initiates execution of the downloaded image and then contacts a server on the network to load its operating system (CLICK)
Now lets talk about what happens during a normal PC Boot Process.1. The machine starts up and 2. the BIOS starts the POST process. 3. The BIOS then identifies a boot device 4. which allows the operating system to load.
Specifically the way the boot device gets identified is by the PXE (CLICK) Process. Once we have the information on where to download the Network Boot Program or NBP we contact the TFTP server and download the bootstrap which then allows us to login to PVS
The PVS boot process is broken down into 5 steps.IP AcquisitionBootstrap DownloadPVS Logon ProcessSingle Read ModeBNISTACK / MIO Read ModeThis process is very similar to the PXE boot process, and in fact in order to be able to use a target device with PVS it myst be PXE 2.1 or higher compatible
Specific to Provisioning Services a PVS PXE client will identify itself with DHCP option 60 signifying that it is a PXE clientThis is used in conjunction with the PXE service running on the PVS server The PXE service, which acts like a DHCP relay will respond to DHCP Discover packets that have option 60 in them and provide the client with the location and filename of the network boot programDHCP Options 66 & 67 can be used to specify the location of the network boot programThe PVS PXE Service acts as a DHCP relay that responds to DHCP Discover packets that have option 60 specified
Specific to Provisioning Services a PVS PXE client will identify itself with DHCP option 60 signifying that it is a PXE clientThis is used in conjunction with the PXE service running on the PVS server The PXE service, which acts like a DHCP relay will respond to DHCP Discover packets that have option 60 in them and provide the client with the location and filename of the network boot programDHCP Options 66 & 67 can be used to specify the location of the network boot programThe PVS PXE Service acts as a DHCP relay that responds to DHCP Discover packets that have option 60 specified
In the event that you would like to assign an IP address statically you can use the Citrix BDM or Boot device manager to create either a boot disk or boot iso that will allow you specify the IP address per target. You will be required to create one iso per machine in order to prevent any issues with duplicate IPs
Network tracing is the best method of troubleshooting DHCP/IP Acquisition issuesSTP (Spanning tree protocol issues)Using spanning tree protocol causes the port to stay in a blocking state for up to 30 seconds while checking for bridge loopsRapid Spanning Tree ProtocolReduced the delay from switching from a blocking to forwarding state from 30 seconds to under 6 seconds
Once the target device obtains an IP address, it goes through the phase of downloading the bootstrap file.Using the address configured in DHCP option 66, the PVS target device contacts the TFTP server (or boot server) and requests the boot file that is configured in DHCP option 67. This bootstrap file is called ARDBP32.bin. Using BDM a PVS target will download TSBBDM.bin over port 6969Dell devices using an OEM version of BDM built into the BIOS and download TSBOROM.binOnce the target device obtains an IP address it uses TFTP to download a bootstrap filePXE Boot (NIC)ARDBP32.binTFTP – UDP port 69BDM Boot (ISO)TSBBDM.binTFTP – UDP port 6969DELL OEM (BIOS)TSBOROM.binTFTP – UDP port 6969
Right-click the server that acts as the TFTP server and click Configure Bootstrap.Click Add and type the IP address of the Provisioning Server that will act as a login server.Type the port number that is configured for the Stream Service.Choose to use DHCP or BOOTP supplied netmask and gateway settings or type this information in the appropriate fields and click OK.Click Move Up or Move Down to place the Provisioning Servers in the order in which target devices should contact for login processing.Click the Options tab and select Restore network connection as the network recovery method.(Optional) Make any additional configuration changes.
Common bootstrap download issuesConfiguration issuesTFTP info not provided with DHCP DORAPacket loss
First thing we are going to take a look at is the DHCP packets [CLICK]After DHCP packets complete we see the target device ARPs for the TFTP server[CLICK]Lastly we will see the TFTP transfer Start [CLICK]
If the target does not get a IP address, this portion of the network trace can help you ID the root cause of the issue. Router in network path No Ip-helper address. [CLICK] PXE Service and DHCP options configured.Target gets IP but times out to TFTP TFTP service not on. Issue with network connectivity.
Login Port RequestPresent MAC addressLoad BalancingTransfer to I/O
Here you will see the end of the TFTP transfer [CLICK] You will know the transfer has completed correctly by verifying that the last TFTP packet sent by the server is tagged (last)Once the bootstrap downloads you will see a series of packets going through the login process. When the login process is completed the target will send a ARP request for the IP of the Streaming Server.
All PVS servers are capable of acting as both a login server and an I/O server. A PVS login server normally attempts to load balance devices between all servers that have access to a given vDisk when the device initially logs in. The login server only bypasses load balancing if the server override property is set for the vDisk locator. The load balancing algorithm provides simple connection count balancing. (i.e. the login server attempts to place the same number of devices on each server that has access to a given vDisk.)
Stream Service must be on.Target not being in the DBThis can be caused by bad bootstrap IP information.Firewall blocking the login port 6910.
The bootstrap directs any block level requests for data as to the PVS server as single read requests The single read mode procedure will continue until the Microsoft Windows Operating System starts loading drivers and BNISTACK is successfully loaded
After the login the target will continue to process traffic using the UNDI driver. (Universal Network Driver Interface and will send one to one packets. [CLICK]During this time the target will display the windows logo. Low level drivers are being loaded here. Failures at this level include:
If the server does not respond to a request within 5 seconds the target device will request the same data again. This will happen a maximum of 3 time before the target will attempt to reconnect and login to the PVS server againThis is generally the longest portion of the boot process due to the amount of data that has to be transferredResponse delays here contribute the most to long boot timesLarge Send OffloadNetwork instabilityConflicts with UNDI DriverStarting with Single Read Mode network traces need to be analyzed by Citrix
MIO (Multiple Input/Output) mode begins when BNISTACK is loaded by the operating systemBNISTACK is loaded into memory and takes over for the bootstrap, it manages the communication between the target device and the PVS serverAt this point the following information is exchanged:Image Mode Active Directory Password Management Option Write Cache Type and Size Client Name Licensing
BNISTACK uses multiple threads to communicate with the PVS serverAdmin ThreadUsed for passing additional configuration settings after initialization and for the heartbeat mechanismRead/Write ThreadsTwo Read threadsOne Write threadOnly used when write cache exists on the PVS serverMIO uses a single request -> multiple response structureSingle read request -> Multiple read repliesMultiple write request -> Single write reply
Once UNDI hands off to BNIStack (PVS Target Device’s Main Filter Driver) the target will begin to request larger amounts of data. This activity will continue up to the windows login screen.
Filter Driver conflictsFirewall/Antivirus Services kickoffAntivirus Updates being loadedDomain Profile creation
At Citrix Services - we’re Citrix consultants, teachers and support engineers and we’re all about one thing: making sure you succeed.With our help, you’ll deploy high-performance, robust virtualization and networking projects, faster – with dramatically lower risk and higher return.The best Citrix architects and administrators are the ones who never stop learning – and Citrix Education is here to help you learn those skills.Citrix Consulting gives you direct access to our most experienced virtualization and networking experts.When it’s complex; when it’s mission-critical; when it’s big; That’s when Citrix consultants can really help.On your virtualization journey, you’ll want always-on support from people who really care about your success.There’s no better insurance for your Citrix investment than with Citrix Support.
Secrets of the Citrix Support Ninjas is a FREE eBook available next week.The eBook contains 40 insider troubleshooting tips for administrators.So the purpose of the eBook is to help administrators like you keep your Citrix deployments on track.We’ve collected some of their best tips and tricks for running robust Citrix environments and packaged them up into a free eBook.In it, you’ll discover some of the little-known tricks that our own support people use every day to tune, tweak, troubleshoot and test Citrix solutions. You may know a few of these tips. But you probably don’t know them all.And – you never know – you might discover just one that will change your life as an administrator.Let me give you a sneak peak now.