This session will cover common deployment methods for StoreFront using NetScaler Gateway as well as review troubleshooting techniques to isolate deployment issues.
What you will learn
- Configuration steps for deploying StoreFront server with NetScaler Gateway
- Design considerations when preparing for deployment
- Tools for troubleshooting it isolate issues
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Citrix TechEdge 2014 - How to Troubleshoot Deployments of StoreFront and NetScaler Gateway
1. How To Troubleshoot Deployments of
StoreFront and NetScaler Gateway
Citrix Synergy, May 2014
Juan Zevallos, Escalation Engineer
Tweet about this session with hashtag #SYN401 and #citrixsynergy
Thank you for joining this session on How To Troubleshoot Deployments of StoreFront and NetScaler Gateway.
In this session, we are going to cover how to try and avoid issues altogether
How to find the real issue by understanding the communication flow
Then once you find the issue, what tools can you use to troubleshoot
Now that we went over the flow from authentication to establishing the ICA session, let’s go over what is needed to accomplish this
Let’s quickly go over the StoreFront integration steps and what you will need
Step 1: Enable Single Sign-On Authentication on StoreFront
This setting will allow StoreFront to evaluate the incoming HTTP request and perform the Authentication Callback if it determines that the user is coming from a Gateway
Step 2: In the StoreFront management console you will need to add a Gateway instance to associate with the StoreFront Store. Let’s go over each field
Display name: This can be whatever you’d like, just keep in mind that end users WILL see this display name if they open their Receiver options to select a Gateway. If you have multiple Gateway in different geographical locations, you can name each Gateway accordingly and have the user select which Gateway to access based on their current location, or you may have a disaster recovery environment that you want to include. This piece of information is included in the Discovery file that the Receiver client downloads to add the account.
NetScaler Gateway URL: This is the FQDN that end users will be accessing from the external network, end users should be typing this exact FQDN into their browser address bar. Receivers on mobile devices or windows and mac devices will automatically use this FQDN after it downloads this information from the StoreFront Store via the Store’s Discovery file.
We’ll cover the Subnet IP later in the presentation
The logon type should match the authentication method configured on the Gateway. So if you have LDAP and RSA authentication, change this field accordingly. This information gets entered into the Discovery file also
Callback URL: Whatever FQDN is entered in here, you should be able to open Internet Explorer on the StoreFront server and browse to this FQDN without certificate warnings and successfully load the logon page. If not, then Single Sign-On from the Gateway will most likely not work.
The last thing to configure is the Secure Ticket Authority (STA). This is the ticketing service used to securely launch an ICA session through the Gateway
Step 2: In the StoreFront management console you will need to add a Gateway instance to associate with the StoreFront Store. Let’s go over each field
Display name: This can be whatever you’d like, just keep in mind that end users WILL see this display name if they open their Receiver options to select a Gateway. If you have multiple Gateway in different geographical locations, you can name each Gateway accordingly and have the user select which Gateway to access based on their current location, or you may have a disaster recovery environment that you want to include. This piece of information is included in the Discovery file that the Receiver client downloads to add the account.
NetScaler Gateway URL: This is the FQDN that end users will be accessing from the external network, end users should be typing this exact FQDN into their browser address bar. Receivers on mobile devices or windows and mac devices will automatically use this FQDN after it downloads this information from the StoreFront Store via the Store’s Discovery file.
We’ll cover the Subnet IP later in the presentation
The logon type should match the authentication method configured on the Gateway. So if you have LDAP and RSA authentication, change this field accordingly. This information gets entered into the Discovery file also
Callback URL: Whatever FQDN is entered in here, you should be able to open Internet Explorer on the StoreFront server and browse to this FQDN without certificate warnings and successfully load the logon page. If not, then Single Sign-On from the Gateway will most likely not work.
The last thing to configure is the Secure Ticket Authority (STA). This is the ticketing service used to securely launch an ICA session through the Gateway
Once the Gateway is created, you’ll be able to bind it to the Store
The first thing you’ll have to do is select ‘No VPN tunnel’. The Full VPN tunnel is not necessary, unless you have XenMobile App Controller publishing Internal Web Links that require a full VPN tunnel from the client to the Gateway, this requires different configuration (See CTX139319)
Then you’ll select the Gateways to bind with the Store – you do have the option to bind multiple Gateways to a single Store
And then select the Default appliance, in cases where you have multiple Gateways
A file that can be used to automatically configure the Store Account into Receiver for any platform – Win/Mac, mobile, linux
End users can access the Discovery file by logging into the Receiver for Web site and clicking on Activate on the top right corner of the web site
Administrators can access the file by Exporting the Provisioning File from the StoreFront Management console, and then distribute the file to the end users
The Discovery file is meant to be opened by Receiver to add the account, but it can also be opened using a text editor since it just contains XML content
The top of the content contains the Store information, including the SRID, Name of the Store, and the BaseURL
Next, we have the information about the Gateway that is bound to the Store, including the Display Name, Authentication type, and the External FQDN that Receiver would use when connecting remotely
So how does Receiver determine whether it should use StoreFront’s BaseURL or the Gateway URL? It relies on the beacons
It’s going to first try to access this Internal beacon by sending an HTTP request to it, if the request comes back successful, then Receiver will connect to the StoreFront FQDN
If the HTTP request is NOT successful, then Receiver is going to check the External beacons and then eventually fallback to the Gateway FQDN
These beacon values can actually be changed from the StoreFront console, in case you want granular control of how and when users access StoreFront or the Gateway
The StoreFront’s BaseURL is the FQDN, configured during the initial setup of StoreFront, used by end users for internal access. This FQDN should be added to your DNS server and needs to resolve to the StoreFront’s server IP address or, if you have multiple StoreFront servers, the load balancer’s virtual IP address. The BaseURL can be found in the StoreFront’s management console and can be changed at any time.
Let’s go through the Gateway configuration steps
To start the wizard, change the Deployment Type to NetScaler Gateway on the NetScaler console logon page
Then click on Create New NetScaler Gateway on the top right of the page
The first step in the wizard is to create the Gateway virtual server by giving it a name, IP address, and port number.
There’s also an option create a virtual server that will redirect users who didn’t type HTTPS in there web browser’s address bar
The next step in the Wizard is to bind the certificate
You can select one already installed on the NetScaler
Or, you can upload one right then and there
The next step in the Wizard is to configure the authentication settings, the primary authentication is typically LDAP and once again you can choose an existing LDAP profile or configure a new one
You also have the option to setup a Secondary authentication
Keep in mind that the StoreFront FQDN and the Use HTTPS options should be based on the StoreFront BaseURL
A common mistake made is forgetting to specify the STA port
No matter how much we prepare, we can still run into problems. Now we are going over some typical issues that we run into in Support and how we troubleshoot these issues using various tools.
To troubleshoot an issue, or to narrow it down to something more specific, we first have to understand how all the pieces work together
“It’s easier to play the game, if you know the rules”
The user will establish an SSL connection to the Gateway virtual server and get prompted to enter their credentials
NetScaler will verify the credentials with Active Directory
Once authenticated, the user will be redirected to StoreFront
StoreFront will realize that the user authenticated at the Gateway and will retrieve those credentials
Once those credentials are received, the user’s resources will be enumerated
When the user clicks on a desktop to launch, StoreFront sends the ICA file to the user
The ICA file contains the necessary information to launch the Desktop through the Gateway (STA ID and Gateway FQDN)
The end user’s Receiver will establish a connection back to the Gateway on the NetScaler
The STA ticket StoreFront originally created for the ICA file will be retrieved by the NetScaler
NetScaler will then establish a connection to the server hosting the Desktop or App
One of the first things the user has to do is successfully authenticate at the Gateway before they have access to anything in the internal network
When authentication fails, there’s not much information presented to the client, other than their credentials were rejected.
One of the best tools to use for authentication issues is Aaad.debug – this is the output of the authentication pipe on the NetScaler that will display authentication and authorization processes that are happening
To start this output – start an SSH session into the NetScaler and go into the shell
Once in the shell, go the /tmp directory and then run cat aaad.debug to begin displaying the information
Here is an example of a failed attempt captured with aaad.debug
There’s usually a lot more lines of information but I just cut out the key ones
The first line listed here is that the authentication process for user juanz is starting
The first thing it does is an LDAP check, which means its trying to access the domain controller with the Service Account configured in the LDAP profile
In this case, it failed with Invalid credentials – this would be considered an LDAP error that would end the entire authentication process
Finally, the kernel is instructed to REJECT the user trying to log in
More examples/information about this tool can be found in CTX114999, including invalid usernames or password and group extraction failures
Internal Server Error 29 is a common error that we see in support.
It’s usually either 1 of 3 things that cause this error
DNS – NS can’t resolve the BaseURL
Network communication from the NS SNIP to the IP of the StoreFront server or load balancer VIP
StoreFront services or IIS is not accepting connections
Once Authentication is successful – the user will be issued their respective policy, depending if they’re coming from a web browser or citrix receiver
The wizard creates and bind 2 session policies to the Gateway virtual server.
One policy is for Receiver – with the Expression that looks for CitrixReceiver in the HTTP Header User-Agent OR the Referer HTTP header does not exist in the HTTP request
The other policy is for the Web Browser which has a general ns_true expression. The thought here is that if the HTTP request does not meet the requirements for the Receiver policy, then the request MUST be coming from a Web Browser.
On the right hand side, a Session profile is associated, that’s where the FQDN, sson domain, and ICA Proxy settings are configured
To make sure you’re hitting the right policy, you can use the nsconmsg tool from the NetScaler’s CLI
Verify that you’re hitting the right policy with the nsconmsg command in a SSH session.
This tool shows which authentication policy you’re hitting also – so the first policy the user gets is the LDAP policy. So you can use this tool to verify which authentication policy the end user is hitting when the user firsts accesses the logon page
If authentication is successful, then the session policy will need to be applied right after.
If the policy that is bound to the Gateway virtual server, created with the wizard, is not being hit, then you’ll need to verify the policy priorities on the NetScaler.
Policies will be applied in 4 levels – to the User, which is the highest priority, then Group, Virtual Server, and Global level which is the lowest priority.
However, no matter at what level the policy is bound, the policy with the highest priority will always take precedence. Keep in mind, the lower the number, the higher the priority.
All of that can be tested and confirmed by just changing the Web Interface Address in the Session Profile for Web browsers
You can change the FQDN to an IP address, to see if DNS is causing the issue
You can also bypass a load balancer this way by entering the IP address of the StoreFront server itself, to verify if there’s an issue with the load balancer
Try changing protocols from HTTPS to HTTP to narrow it down to a possible port communication issue or an SSL communication issue
When StoreFront determines that the end user is coming from a Gateway, StoreFront will attempt to access the callback URL to grab the user credentials
So assuming the communication is working from the SNIP to the StoreFront server, the end user may be presented with a double authentication issue
So this indicates the single sign-on is NOT being engaged by the StoreFront server
Or Remote Access is NOT enabled for the Store NetScaler is directing you to
What if both those options are checked? Why doesn’t the StoreFront server start the authentication callback process instead of asking for credentials from the end user?
Single Sign-On is invoked by the NetScaler Gateway URL setting in the StoreFront config
This value must match exactly what the end user types into their Web browser
How does StoreFront know what the user is typing into their Web Browser address bar?
NetScaler includes this information in the HTTP Header XCitrixVia
StoreFront analyzes every HTTP request that comes in and if it finds this Header value matches a Gateway FQDN, then single sign-on will be invoked
I was able to see this information using StoreFront’s verbose logging, CTX139592 provides instructions on how to gather them
The NetScaler knows to inject the hostname that the user typed into their web browser address bar into an HTTP header call X-Citrix-Via
This value must match the Gateway URL configured on StoreFront
If you see this error during the SSON process, it’s most likely the StoreFront’s Callback process that’s failing
To make sure, check Event Viewer and also test StoreFront internally, to make sure StoreFront is functioning properly outside of the NetScaler integration
StoreFront may not be able to resolve the FQDN, or there’s a typo in the configuration. Check DNS or modify the HOSTS file on the StoreFront server
There could be a network issue as well, including some kind of Proxy interfering with the communication
The quickest way to eliminate these two issues is to open up Internet Explorer and try to browse to that FQDN – if you can successfully reach the logon page of the Gateway, then all should be well
So even though Internet Explorer successfully connected to the Gateway logon page, StoreFront can definitely run into an SSL Trust issue if the Certificate chain is not properly linked on the Gateway
Using http://www.digicert.com/help/ in this example
It verifies the FQDN being used and most importantly, the Certificate Chain
This example shows a properly configured certficate chain, indicated by the blue links
You can verify the chained certificates by opening up the Certificate itself and looking at the Intermediate certs under the Certification Path tab
Also, taking care of this now will help avoid issues with Mobile devices launching ICA sessions through the Gateway
A common problem we see in support is when there are multiple Gateways being load balanced, fronting a single StoreFront server group.
When an end user authenticates to one of the NetScalers and gets routed to a StoreFront server, the StoreFront server needs to be able to communicate back to the NS where the user authenticated from
In this scenario, you will have to configure a gateway instance for each NetScaler respectively, even though they have the same Gateway FQDN.
Each Gateway will have its own Callback URL that resolves to the Gateway virtual server on different NetScalers
Each NetScaler will have a Gateway virtual server with a different IP address – just enter the virtual server IP address in the Subnet IP address field.
StoreFront will decide which callback URL to use based on that Subnet IP address value, by comparing it to the IP address that comes in the HTTP request header X-Citrix-Via-VIP
This value, along with other HTTP header values, can be seen with DebugView on the StoreFront server
Now, when the user goes through NetScaler 1 – the NetScaler automatically adds the virtual server IP address into this X-Citrix-Via-VIP header for StoreFront to analyze.
StoreFront will know which Callback URL to use based on this
In the DebugView, you can verify the credentials that StoreFront grabs from the Callback service
It will show the username, single sign-on domain, and whether a password was supplied – passwords aren’t supplied with Smart Card authentication
StoreFront will send verify the credentials again and then send the request to the Farm XML brokers to enumerate your applications and desktop(s)
At this point, we have completed the single sign-on process and we are ready to launch
And now – we get an error
When launching an application, StoreFront sends an ICA file to the client that contains the STA ticket information and the Gateway FQDN
Receiver first establishes a connection to the NetScaler, and then the NetScaler first needs to retrieve the STA ticket that StoreFront created for the ICA file
DebugView needs to request a ticket from the STA server
The critical information here is the IP address of the XA server or VDA desktop that will be hosting this session
The STA server then responds with its STA ID and the Ticket number
This information gets added in the ICA file that gets sent back to the client
Here’s a snippet of 2 key values in the ICA file
The Address = the first value is the number 40 – which tells the Gateway that we want to use Session Reliability and instructs the Gateway to communicate to the back end server over port 2598
If Session Reliability was disabled, it would show 10, which would force the Gateway to use port 1494
The second value is the STA server ID, this is how the Gateway knows which STA server to reach out to in cases where there are multiple STA servers
Then there’s the STA ticket ID that’s being held on the STA server which has the session information that StoreFront provided
You can verify that the STA server is reachable and the ID that it is returning back to the NS
While the app is launching, you can watch the NetScaler grab the STA ticket by running a tcp dump command
Here’s an example of the request you would see the NS make to the STA server during app launch
In blue, you’ll see highlighted the Ticket ID that was found in the ICA file generated by StoreFront
Here’s the response from the STA server
It was able to find the STA ticket and retrieve the server details that is going to be hosting this application or desktop
Once the NetScaler has the information from the STA ticket, it’s time for it to establish the connection to that server
At this point of the process, if it fails, it’s usually a communication issue or DNS issue
A DNS issue can occur if you have DNS Translation Policy enabled on the farm, which will return the FQDN of the servers to the NetScaler, instead of the IP address as we have been seeing in these samples.
One of the quickest test you can do is test connectivity from the NetScaler to the back end servers over 1494 and 2598
All you have to do is create a service, specify the IP and port and check the state
For XenDesktop VDA – this is a little trickier – VDAs do not actively listen on the ICA ports until they’re about to begin a session
One trick is to open a VDA session on the internal network, and create the services to the IP address of the internally launched VDA desktop
This whole time, I was showing screenshots from a web browser
That is because it is MUCH easier to troubleshoot with the web browser, Receiver does a very good job of masking the real errors
Once the Web Browser is working, you know that the configuration on StoreFront is accurate, so, in most cases, you can eliminate that out of the troubleshooting equation
So you can focus on the NetScaler and the client
The StoreFront Store is inaccessible (internally)
Misconfigured StoreFront BaseURL in Session Profile for Receiver
Internal Beacon is reachable externally
Customizations on the Gateway logon page
iOS Receiver does not support SHA256 SSL Certificates
Android does not support SAN SSL Certificates
Enable Windows Receiver logging – CTX134101