Part of the Citrix Support Secrets Webinar Series
This troubleshooting tools webinar will cover how to quickly define and troubleshoot problems in your Citrix XenApp and Citrix XenDesktop environment. We will look at various tools offered by Citrix to help identify, analyze and resolve the most common problems observed by Citrix Technical Support engineers.
What you will learn:
- How to troubleshoot and isolate connectivity/policies/session sharing issues
- How to use Citrix XDPing tool to troubleshoot
- How to use Citrix Quick Launch tool to troubleshoot
- How to use Scout for Citrix XenApp and Citrix XenDesktop
This webinar took place on Nov 29th, see recording here.
https://www1.gotomeeting.com/register/284459393
Troubleshooting Tools – How to isolate and resolve issues in your XenApp & XenDesktop environment more rapidly.
1. November 29, 2012
Tech Support Secrets
Webinar Series
Troubleshooting Tools – How to isolate and
resolve issues in your XenApp & XenDesktop
environment more rapidly.
29. About
Citrix Services
Citrix Services make sure
you succeed with your
virtualization programs.
How we can help
Citrix Education – The fastest, most efficient way to
get your team the virtualization skills they need. Online,
on-site or in class.
citrix.com/training
Citrix Consulting – Intensive engagements for
complex, critical or just plain massive projects.
citrix.com/consulting
Citrix Support – Always-on support services that
leverage everything we know about best-practice
deployment and maintenance.
citrix.com/support
Educate | Guide | Support | Succeed
30. • Available for XenDesktop, XenApp, NetScaler and XenServer
• Secure upload of data
• Auto-analysis capabilities
• Plugins
• Agile development – updates every 3 weeks
• Notification via email
taas.citrix.com/AutoSupport
Citrix Auto Support
31. • Access: Unlimited Citrix Worldwide Technical Support
– 24x7x365 or business hours
• Coverage: Hardware returns & software updates
• Products Covered: NetScaler, Access Gateway,
Branch Repeater
• citrix.com/support
Hardware and Software Tech Support
• Access: Unlimited Citrix Worldwide Technical Support
– 24x7x365
• Coverage: Version updates & Tech Support
• Products Covered: VDI-in-a-Box, CloudPlatform,
CloudPortal Business Manager, AppDNA, XenClient
• citrix.com/support
32. Support Programs
• Access: Unlimited Citrix Worldwide Technical Support
– 24x7x365
• Coverage: Major/minor software updates
• Products Covered: All software products eligible for
Subscription Advantage
• citrix.com/premiersupport
• Proactive: Knowledge of environment helps avoid issues
and drives best practices
• Responsive: Single point of contact to accelerate issue
resolution
• Relationship: Trusted advisor and customer advocate in
Citrix
• citrix.com/support
Hinweis der Redaktion
How to troubleshoot and isolate issues using Citrix tools introduction
In this presentation we will be going through some of the common issue encountered by Citrix Partner and customers. We will look at ways to quickly isolate the problems and try to diagnose the root cause with the use of Citrix Troubleshooting tools.
XenApp connectivity issues
We will look at some common XenApp connectivity issues with some real world scenarios and the tool we can use to troubleshoot them.
Citrix Policies
We will examine the Policy architecture and processing order . Looking at the Citrix policy modelling wizard and GP Results can be use to trouble shoot,
Session Sharing issues
Session Sharing is common enough problem which can be difficult to isolate where the problems lies, we will look at a troubleshoot methodology to tackle the problem and CDF Control to establish the root cause.
XenDesktop registration issues
Desktop registration comprise of large chunk of Citrix Technical Support XenDesktop, we will look at XDPing to quickly troubleshoot and isolate these problems.
Citrix Scout
Run from a single XenDesktop controller (DDC) or XenApp server, Citrix Scout captures key data points and CDF traces for selected computers followed by secure and reliable upload of the data package to Citrix Technical Support
TAAS 2.0 Tool as a Service
Online auto-analysis capabilities to help you collect environment information, analyze that information and receive tailored recommendations based on your Citrix environment and configuration
To be begin we will look at some common XenApp connectivity issues with real world scenarios and how to use troubleshooting tools to resolve the issue.
Application launch failures:
“An error has occurred while connecting to the requested resource”
ICA connectivity 1494/2598
DNS resolution
Application launches stuck at splash screen
Application launch failures:
Generic Error “An error has occurred while connecting to the requested resource” – this error message does not give much feedback or where to start. The error was designed this way so limit the information that a would be attacker may exploit.
To quickly eliminate if the issue is related to the Data Collector (Controller), XML Broker, Receiver, Web Interface , Storefont, AGEE or NetScaler
We can use quick launch tool to test XML enumeration on the Server by checking the XML port, can we enumerate the applications via XML on the server?
Using user credentials we can test, enumeration of apps and launching per users. ICA files can be created saved for testing purposes .
We can use the tool to test connectivity to that application and XenApp server to ensure we have access. Ensuring the ICA listener is working correctly.
Citrix Quick Launch
http://support.citrix.com/article/CTX122536
ICA connectivity 1494/2598
An example of a real world situation, user reported that their Citrix sessions disconnected and they could not reconnect.
After trouble shooting it was noticed that the XTE service (Session Reliability) had crashed and hadn't restarted.
In this situation the ICA traffic should have fallen over back over to 1494 nevertheless it failed to do so.
There was a firewall between users and the XenApp servers. A security network admin disabled port 1494 on the firewall because there was no activity for some time.
This is how we used Citrix Quick Launch tool to figure the problem:
We used Citrix Quick Launch tool to test connectivity through firewalls 2598 and then 1494 by toggling the Enable Session Reliability tick box.
It appeared that 1494 was blocked for ICA connections through the firewall.
ICA / HDX TCP 1494 Access to applications and virtual desktops
Session Reliability TCP 2598 Access to applications and virtual desktops
Other Features that the Quick Launch Tool can be used for
Display size, color Depth, and Encryption
Enables the desktop Viewer, Windows Key combination enable compression
Control mapping of end point devices, such as CDM (read only), Audio Mapping and Printers.
Speed Screen – Keyboard and Mouse Timers
Speed Screen – Latency reduction features; local text echo, queued mouse movements and clicks
Speed Screen – Bitmap Caching, size and location
An example of a real world situation, When a user goes to launch an app the get the error message “An error occurred while making the requested connection”. And the connection fails
So how did we solve the problem?
We used Citrix Quick Launch tool to test DNS address resolution works successfully in the environment. Working correctly….
We then check the ICA file which is generated during the failure.
We notice that in the failure ICA file address is set to 192.168.1.25. This address is not routable from the client.
The customer was using PVS with two nics for diff VLANS, with one for PVS traffic and another NIC to handle the network traffic.
The ica file contained the ip address of the pvs nic causing the application to fail.
The resolution was to DNS address resolution in XenApp policies
http://support.citrix.com/article/CTX131554
An example of a real world situation, after publishing a new application in the XenApp farm, users report that the application remains stuck at the splash screen for several minutes before successfully loading.
Using the Citrix Seamless Settings Configuration Tool the administrator is quickly able to establish if this is a seamless application problem. By combining all the Global Flags together we disable the seamless components. If the application then works we know that one the flags. We can then use the configuration tool narrow down to the offending flag.
sehook20.dll. This dll is responsible for Active Accessibility Hooking the seamless engine.
Microsoft Access 2000 Database Hangs at a Gray Splash Screen
http://support.citrix.com/article/CTX128939
Farm Seamless Settings Configuration Helper Tool
http://support.citrix.com/article/CTX115146
The figure illustrates the conceptual architecture behind the Citrix policy system.
When Citrix policies are created or edited within the GPMC and the Group Policy Object Editor, the configuration is stored in the following location: \\domain\SYSVOL\domain\Policies\guid\machine or user\Citrix\GroupPolicy\Policies.gpf. When Citrix policies are created or edited within the AppCenter, the IMA-based policy settings are stored as metadata in the data store database and are propagated to servers as GPF/X files stored in their local SYSVOL directory. In both instances, the settings are written to each server registry.
Each time group policies are evaluated on the XenApp server, the GPF/X files are retrieved from the SYSVOL and farm data store. The client-side extension evaluates the filters and merges the results into a single Resultant Set of Policy within the HKLM\Software\Policy\Citrix registry key. Various software components read the registry values and enforce the settings.
During the XenApp and AppCenter installations, Citrix client-side extensions are installed, which allow Citrix policy integration within the Microsoft Group Policy engine. These extensions add a Citrix Policies node within the existing Computer and User nodes in the Group Policy Object Editor. The Citrix Policies node allows you to create Citrix policies as either User or Computer policies within the GPO.
Group Policy Integration Benefits
Group policy integration allows organizations to take advantage of the Group Policy management features for their XenApp environment. For example, the GPMC allows you to:
Back up and restore policies.
Migrate policies from one domain to another.
View the resultant set of policies for a server, user, or session.
Perform modeling by retrieving policy reports for any user connection scenario.
Create Active Directory delegated administration for Citrix settings and policies.
The gpo and ima based policies that apply to a user or computer do not all have the same precedence. If there is no conflicting settings configured within the polices the settings are merged into the RSOP for the computer or user. However settings in the policies that are later can override earlier applied settings .polices are processed and applied in the following order
There are some exceptions to the rule for example .The highest encryption setting in our environment, including the operating system and the most restrictive shadowing setting, always overrides other settings and policies.
Policy Modeling and GPResults
You can use the Citrix Group Policy Modeling wizard to simulate a user connection in order to test the policy settings ultimately applied to a user session after processing. With the Citrix Group Policy Modeling wizard, you can specify conditions for a connection scenario such as domain controller, users, Citrix policy filters, and simulated environment settings such as slow network connection. The wizard connects to the domain controller, reads settings from the SYSVOL, and produces a report that lists the Citrix policies that likely would take effect in the scenario.
Policy modeling also can be performed using Microsoft Group Policy Modeling. However, this tool will not reflect Citrix policy filters and, instead, assumes that all settings within a policy will be applied. Therefore, using the Citrix Group Policy Modeling wizard is recommended.
To start the wizard from the Group Policy Management Console, right-click the Citrix Group Policy Modeling node and select Citrix Group Policy Modeling wizard. To start the wizard from the AppCenter, right-click the Citrix Policies node and select Run the modeling wizard.
When running the wizard while logged on to the server as a domain user in an Active Directory domain, the wizard calculates the Resultant Set of Policy (Active Directory policies) by including settings from Active Directory GPOs. When running the wizard from the AppCenter, the modeling calculation includes the IMA-based GPO residing on the server (AD and IMA policies). However, when running the wizard from the AppCenter while logged on to the server as a local user, the wizard calculates the Resultant Set of Policy model using only the farm GPO.
Group Policy Results
The Group Policy Results tool helps to evaluate the current state of GPOs in the environment and generates a report that describes how these objects, including Citrix policies, are currently being applied to a particular user and server. The Group Policy Results tool connects to the XenApp server and reads the applied Computer and User policy settings within the registry. HKEY_LOCAL_MACHINE\SOFTWARE\Policies
As a result, the tool can be useful for troubleshooting policy settings that were already applied to the user session.
Group Policy Results requires the user to have logged on to the server at least once.
Read World Scenario:
The Session Idle Limit (Server idle timer interval) has been set via Citrix IMA Policies and appears to be successfully applied to the XenApp Server. However the session idle timeout does not apply at the given interval. In this scenario it was noticed that the session idle time interval was set to 6 minutes but did not apply until an hour later.
Customer confirmed that running the policy modeling wizard ensured the server was eligible for policy
We checked applying policies then GPO – Same Issue
Running a GP results also show that the policy applied successfully.
Blocking the Inheritance on the XenApp Farm OU the issue disappeared
Adding it back again, the issue happened again.
Further investigation into group policies results applied to the server revealed that a Remote Desktop Services idle session limit had been applied.
As the policy was applied to Remote Desktop Services it affects both ICA/RDP listening protocols. Its recommended that you use Citrix IMA or Group policies to control the server idle timer interval.
XenApp Session Idle Timeout Does Not Apply at the Set Interval
http://support.citrix.com/article/CTX134756
When viewing user sessions in the AppCenter, if a user is consuming multiple sessions as evidenced by two or more session numbers on distinct servers, session sharing may not be functioning properly.
By its very nature, session sharing ensures efficiency and minimizes resources on the server. For this reason, the AppCenter should show a single session number assigned to multiple applications. By default, when a user accesses multiple published applications, these will be shared within a single session so long as those applications are available from the same server.
However, there are some specific reasons why session sharing does not function this way:
Published applications have different settings, such as display, audio, client drive mapping or encryption
All applications should have the same settings in order to ensure that subsequent application launches share the existing session.
Session sharing has been specifically disabled within the registry, such as may be required for non-Citrix WAN optimization technologies.
If users access multiple applications in succession.
Web Interface by default includes a two-second delay as part of the MultiLaunchTimeout parameter, which specifies the time for which resource icons are inactive following the initial click by the user to start the resource. This behavior is designed so that the first application can start launching and subsequent applications can partake of session sharing. This behavior is controlled within Web Interface; however, depending on the network connection and environmental conditions, the default setting may need to be adjusted.
Session Sharing
We can use CDF Control to trouble the issue.
Separate Zone Data Collector (Zone Controller) (This is to limit the amount of data capture in the trace)
Try to narrow down the behaviour as best possible (try to find a pattern apps, servers or users)
Use CDF Control to capture trace on ZDC/Controller (trace session sharing modules and logon disconnects)
Take note of key user info: user name, client name, server and session id
Analyse Trace for root cause
The connection details are recorded for each session the user makes on the server in the below key
HKLM\SOFTWARE\Citrix\Ica\Session\[n]\Connection
AppName(s)
Inside here we can see the session sharing applications
App State introduced for XenApp 6.5
N/A
ACTIVE
LINGERING
PRE-LAUNCHED
APPLICATION NOT RUNNING
Color Depth
User Name
Domain Name
Encyrption
Session Guid
Session Sharing Key
CDF Control
CDFControl is an event tracing controller/consumer, geared towards capturing Citrix Diagnostic Facility (CDF) trace messages that are output from the various Citrix tracing providers. Now in its second version, CDFControl offers new capabilities such as real-time trace viewing and performance data collection.
Dynamic TMF file download feature:
TMF files are automatically downloaded as needed and temporarily stored in the user’s temp directory. These files are deleted automatically after use.)
http://support.citrix.com/article/CTX111961
An example of real world scenario, a customer reports that they are seeing multiple sessions for the some user across the farm. They have confirmed that all application settings are the same so session sharing should work.
Modules to trace:
IMA_Runtime_DynamicStore
IMA_Sals_Comapp
IMA_Sals_Comsrv
IMA_Sals_LMS
IMA_Sals_MfApp
IMA_Sals_MfServer
IMA_Subsystems_ComApp
IMA_Subsystems_ComSrv
IMA_Subsystems_LMS
IMA_Subsystems_MfApp
IMA_Subsystems_MfServer
IMA_Subsystems_User
IMA_Subsystems_UserWin
IMA_System_Support
MF_DLL_Ctxgina
MF_DLL_Ctxnotif
MF_DLL_Cutildll
MF_DLL_Wsxica
MF_Library_System
MF_Service_PnSvc
MF_Session_CTXRDPCTLS
MF_Session_ICACTLS
MF_Session_ICASTExe
MF_Session_TWI
SharingSettings the session sharing settings that are to be compared with the user settings
Check the trace for the SharingSettings, compare this between application launches
1st App Launch
CDF_INFO - Exiting. SharingSettings: ColorDepth=24bpp, EncryptionLevel=basic, AudioQuality=<NULL>, DomainName=nickdublab, UserName=userone, FarmName=FarmXenApp65, SpecialFolderRedirection=Off, VirtualCOMPortEmulation=Off, COMPortMapping=<NULL>, ClientPrinterPortMapping=On, ClientPrinterSpooling=On, TWIDisableSessionSharing=<NULL>, EnableSessionSharing=<NULL>, DisplaySize=<NULL>
CDF_ENTRY - Entered. App Name = IE 9. Client Name = WI_q43FV9LZtntZ36D38. User Name = userone. Domain Name = nickdublab.
CDF_INFO - SessionSharingKey: -LZv2VKBd3hT2HHdxrOBldW
2nd App Launch
CDF_INFO - Exiting. SharingSettings: ColorDepth=24bpp, EncryptionLevel=basic, AudioQuality=<NULL>, DomainName=nickdublab, UserName=userone, FarmName=FarmXenApp65, SpecialFolderRedirection=Off, VirtualCOMPortEmulation=Off, COMPortMapping=<NULL>, ClientPrinterPortMapping=On, ClientPrinterSpooling=On, TWIDisableSessionSharing=<NULL>, EnableSessionSharing=<NULL>, DisplaySize=<NULL>
CDF_ENTRY - Entered. App Name = WritePad. Client Name = WI_q43FV9LZtntZ36D38. User Name = userone. Domain Name = nickdublab.
We can see that the session sharing fails because the session does not have session sharing enabled? Someone disabled this seamless flag as part troubling another problem.
CDF_INFO - Session {B0738710-6681-44F7-B68D-80238B54A5EF}_0 does not have Session sharing enabled. Returning false.
CDF_INFO Session {B0738710-6681-44F7-B68D-80238B54A5EF}_0 is NOT sharable with this app. Moving on.
CDF_ENTRY - Exiting. Sharable Sessions = 0. result = 0x0.
CDF_INFO - No sharable sessions found. Exiting early.
The XDPing tool is a command-line based application which automates the process of checking for the causes of common configuration issues in a XenDesktop environment.
The tool can be used to verify configuration settings on both the XenDesktop Broker and VDA machines, both from the console and remotely.
Depending on how the tool is run, and from where, the following checks and information can be displayed:
Information and status of Network Interfaces and Network settings. (Console Only)
Performs DNS lookup and reverse lookup on the IP address of the device.
Information on Time synchronization and time check for Kerberos Authentication (Console Only)
User information for login User (Console Only)
Including User details, Authentication type used, Group Membership.
Machine information (Console Only)
Environment information (Computer Name, operating system version, Domain)
Domain membership verification (Membership = Verified, SID:S-X-X-XX-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXXX-XXXXX [OK])
Information on XenDesktop Services (Windows Communication Foundation Endpoints) installed and confirms if each installed service is responsive. (Console Only)
Displays information on the Windows Firewall installed on the VDA and checks if the important ports are configured correctly.
Queries the local event log to check for known events that are related to XenDekstop.
Provides client bandwidth and response time information from the VDA to the client.
XDPing Tool
http://support.citrix.com/article/CTX123278
Run from a single XenDesktop controller (DDC) or XenApp server,
Citrix Scout captures key data points and CDF traces for selected computers
followed by secure and reliable upload of the data package to Citrix Technical Support.
Citrix Scout
http://support.citrix.com/article/CTX130147
We can use Scout to collect data from multiple XenDesktop Controllers, Virtual Desktop Agents and XenApp Servers
10 is the default but this can be changed through advanced menu option.
Once we have collected this data we can upload to CTX Support securely with your case number or save it locally.
When the upload has completed the Technical Support engineer will get an email notification that the files are ready.
The Technical Support engineer can then open the package using their scout to read Site or Farm Details.
We can collect the data Farm Info, Policies, Remote Desktop Services Settings
Per machine we can collect Citrix info about binaries, hardware, software, registry and event logs
Scout create a single place where you can compare machine settings.
This content can also be opened later, you find this useful to pinpoint changes made in your environment.
XenDesktop Collector captures key data points for Citrix XenDesktop Controller (DDC) servers and virtual desktop agents. The Tools As a Service (TaaS) platform uses this information to reduce the time to resolution for customer reported issues.
Access http://support.citrix.com and enter XenDesktop Collector in the Knowledge Center search field to access the latest version of XenDesktop Collector.
At Citrix Services - we’re Citrix consultants, teachers and support engineers and we’re all about one thing: making sure you succeed.
With our help, you’ll deploy high-performance, robust virtualization and networking projects, faster – with dramatically lower risk and higher return.
The best Citrix architects and administrators are the ones who never stop learning – and Citrix Education is here to help you learn those skills.
Citrix Consulting gives you direct access to our most experienced virtualization and networking experts.
When it’s complex; when it’s mission-critical; when it’s big; That’s when Citrix consultants can really help.
On your virtualization journey, you’ll want always-on support from people who really care about your success.
There’s no better insurance for your Citrix investment than with Citrix Support.
Citrix Auto Support is a fairly new initiative from Citrix. We relaunched the product at Synergy Barcelona last month under this new name. You maybe familiar by its old name TaaS.
Citrix Auto Support is focused on making the support of your Citrix environment as easy as possible. Citrix has developed tools and online auto-analysis capabilities to help you collect environment information, analyze that information, and receive tailored recommendations based on your Citrix environment and configuration. The tools are focused on a single mission (data collection), and their impact to your environment is minimal from a disk space, pre-requisites, and performance impact during the data collection process.
Secure upload of Data
Auto-analysis capabilities
What is a TaaS plug-in?
A plug-in is a module developed to identify known issues, best practices, and performance optimizations that automatically run for the uploaded product data.
These are growing all the time across our products
Agile Development – Updates to TaaS site and collection tools every 3 weeks
Notification via email when analysis is complete,
If you have a support case opened, you can associate the update with your case number so the Citrix support engineer is notified and can start working on solving the issue much faster.
Auto Support currently supports XenDesktop, XenServer, NetScaler and most recently XenApp.
Go to taas.citrix.com/autosupport and login today. Best of all, its FREE to use.
Now lets look at some of the options available from Citrix Support.
Citrix Hardware Support
Also known as appliance maintenance, provides technical support from our experts, which can help you diagnose and resolve issues encountered with your appliance.
Some of the key benefits are listed on the slide, most notably the unlimited incidents you can open.
This offering is available to our hardware products – NetScaler, Branch Repeater and Access Gateway.
Also as part of this offering you are entitled to software updates and bug fixes as well as RMA replacement.
Its important to note that with access to the latest software upgrades and replacement of malfunctioning appliances help to minimize downtime at your organization.
Citrix Software Support
Also known as software maintenance, combines 24x7x365 unlimited support with product version upgrades.
Products covered are listed on the slide.
With both of these offerings you can Add- On: Technical Relationship Management (TRM). I’ll cover TRM in more detail on the next slide.
I’ll be sending an email to all registered attendees of this webinar with further details around our Services offerings and oppurtunites to learn more.
Premier Support
Citrix Premier Support provides 24x7x365 unlimited-incidents worldwide support for Citrix software products covered by Subscription Advantage.
Products covered are: XenApp, XenDesktop, XenServer, NetScaler VPX, Branch Repeater VPX, Access Gateway VPX and Universal Licenses, EdgeSight,
Provisioning Server and Application Streaming for Desktops
This offering is available to all GEOs with our reps providing support in 9 languages
Premier Support customers are also invited along to Citrix TechEdge event at Synergy. This is a one-day training event on the latest troubleshooting techniques, methodologies, and diagnostic tools from Citrix technical support escalation engineers. TechEdge is held twice annually, once each during Citrix Synergy US and Citrix Synergy Europe
TRM
With any of the previous support offerings we discussed, you can add on a TRM – Technical Relationship Manager.
This will offer your business a designated, highly trained single point of contact.
The TRM has extensive experience delivering Citrix solutions and is equipped to troubleshoot and resolve complex issues quickly.
Our goal is to minimize downtime exposure in your business through proactive advice and by managing support requests through a designated contact
Mostly importantly the TRM is intimately familiar with your Citrix infrastructure, like a virtual member of your team.
As mentioned already I’ll be sending an email to all registered attendees of this webinar with further details around our Services offerings and oppurtunites to learn more.