Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Advanced Tools and Techniques for 
Troubleshooting NetScaler Appliances 
Andrew Redman | Lead Escalation Engineer 
Citrix ...
2 © 2014 Citrix. Confidential. 
NetScaler System Overview 
Troubleshooting Tools & Techniques 
Case Studies 
Resources 
Co...
NetScaler System Overview
Key NetScaler Processes 
Process Description 
ns_master/NSPPE 
4 © 2014 Citrix. Confidential. 
nsvpnd 
nsaaad 
nsconf 
nsa...
NetScaler File System 
/var (hard drive) Logs - /var/log & /var/nslog 
5 © 2014 Citrix. Confidential. 
Install - /var/nsin...
NetScaler File System (cont.) 
/flash (cont.) User Monitors - /flash/nsconfig/monitors 
6 © 2014 Citrix. Confidential. 
Cu...
Troubleshooting Tools & Techniques
NetScaler Tech Support Bundle 
> show techsupport 
Critical 
System Data 
In-Depth 
Performance 
Monitoring 
Stats 
Detail...
The NetScaler Tech Support Bundle 
9 © 2014 Citrix. Confidential.
Citrix Insight Services 
10 © 2014 Citrix. Confidential.
11 © 2014 Citrix. Confidential. 
Data Collection Analyze Data 
‘Single Mission … Data Collection’ 
FAQ: http://support.cit...
https://taas.citrix.com 
12 © 2014 Citrix. Confidential.
Technical Issues Flagged 
13 © 2014 Citrix. Confidential.
Investigate Issues Further 
The ‘BELL’ 
14 © 2014 Citrix. Confidential.
Crash File(s) Location 
15 © 2014 Citrix. Confidential. 
nscollect-542.gz
Intuitive Navigation 
16 © 2014 Citrix. Confidential. 
Select a different 
newnslog file to view
Detailed Graphs 
17 © 2014 Citrix. Confidential. 
Mouse over graph 
to see more detail 
Informative 
reference legend 
Dow...
IPMI – Intelligent Platform Management Interface 
Change NetScaler 
18 © 2014 Citrix. Confidential. 
Default LOM IP Addres...
Common CLI Show Commands 
Common show commands for system information: 
show node, show info, show license 
Common show co...
Common CLI Stat Commands 
Common stat commands for system information: 
stat ns, stat cpu, stat interface 
Common stat com...
Leveraging ‘nsconmsg’ 
Nsconmsg common use cases: 
View events 
View console messages 
View statistics 
Debug system count...
Example ‘nsconmsg’ Usage 
# cd /var/nslog 
# nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg 
22 © 2014 Citri...
Displaying debug performance information 
NetScaler current V20 time Performance is Sun Mar Data 
23 18:33:43 2014 
NetSca...
# nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more 
# nsconmsg -K newnslog -s ConMon=1 -d oldc...
Log File Analysis 
# cd /var/log 
# zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE) 
ns.log:Mar 20 16:45:0...
NetScaler + Wireshark = ‘thumbs up’ 
http://www.wireshark.org 
26 © 2014 Citrix. Confidential.
RED HOT Wireshark Tip 
27 © 2014 Citrix. Confidential. 
Custom 
Columns 
Custom Menu 
Options 
Pre-build 
Custom Filters 
...
Troubleshooting Techniques & Case Studies
Top Tips 
29 © 2014 Citrix. Confidential. 
Use Citrix Insight Services 
Pay attention to the issues! 
Note the highlighted...
Case #1 - High Availability Synchronization
Same type of 
appliance 
31 © 2014 Citrix. Confidential. 
HA Pre-requisites 
Same firmware 
version 
Recommend 
same nsroo...
Insight Services Flags The Issues 
32 © 2014 Citrix. Confidential.
The HA Pair Struggled To Synchronize 
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted) ...
The ‘newnslog’ Time-Frame 
# nsconmsg -K newnslog -d setime 
Displaying start and end time information 
NetScaler V20 Perf...
Case #2 - XA/XD Slow Performance
Smartphones XenDesktop 
Tablets XenApp 
36 © 2014 Citrix. Confidential. 
Insight 
Services 
Critical Insight 
Gleaned 
Pre...
Insight Services Again Flags The Issues 
37 © 2014 Citrix. Confidential. 
http://support.citrix.com/article/CTX136926
Performance Was Extremely Latent 
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved 
4263 0 23 nic_tot_bdg_m...
Networking Issues Again? 
# nsconmsg -K newnslog -d statswt0 | grep nic_err 
4274 0 1995 nic_err_rl_pkt_drops interface(1/...
40 © 2014 Citrix. Confidential. 
The Moral of the Story 
Leverage Citrix Insight Services 
Leverage 
Insight Services 
Pay...
Resources
Helpful Resources 
Comprehensive NetScaler Counters 
Wireshark Developer Editions 
Customizing Wireshark Tutorial 
Citrix ...
Conclusion
What We’ve Actually Covered 
An Overview of the NetScaler System to give you a high-level understanding of the core system...
Fuel your talent with continuous learning. 
93% of Citrix Education students became more effective in their role after att...
Simplify your journey, let us guide you. 
Accelerate your implementation and minimize risk by taking advantage of Citrix 
...
47 © 2014 Citrix. Confidential. 
WORK BETTER. LIVE BETTER.
Nächste SlideShare
Wird geladen in …5
×

Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

17.577 Aufrufe

Veröffentlicht am

This session will cover advanced techniques in troubleshooting the Citrix NetScaler Appliance using tools such as Citrix TaaS, IPMI, nsconmsg, wireshark and log analysis. We will review usages of these tools along with case studies showing how to best troubleshoot common issues seen in operating Citrix NetScaler Appliances.

What you will learn
- Various tools available to troubleshoot issues and how to use them to isolate NetScaler Issues
- Common deployment problems and how to isolate the causes

Veröffentlicht in: Technologie
  • Your wife will never find out! .. Just send me a message and ask to F.U.C.K. ▲▲▲ http://t.cn/AiuW9zn5
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • The #1 Woodworking Resource With Over 16,000 Plans, Download 50 FREE Plans... ▲▲▲ http://ishbv.com/tedsplans/pdf
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Discover a WEIRD trick I use to make over $3500 per month taking paid surveys online. read more... ◆◆◆ https://tinyurl.com/make2793amonth
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Have you ever heard of taking paid surveys on the internet before? We have one right now that pays $50, and takes less than 10 minutes! If you want to take it, here is your personal link ●●● http://ishbv.com/surveys6/pdf
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

  1. 1. Advanced Tools and Techniques for Troubleshooting NetScaler Appliances Andrew Redman | Lead Escalation Engineer Citrix Support Webinar August 26 & 27, 2014
  2. 2. 2 © 2014 Citrix. Confidential. NetScaler System Overview Troubleshooting Tools & Techniques Case Studies Resources Conclusion Q&A Agenda
  3. 3. NetScaler System Overview
  4. 4. Key NetScaler Processes Process Description ns_master/NSPPE 4 © 2014 Citrix. Confidential. nsvpnd nsaaad nsconf nsauthd nslog.sh nssync nsreadfile nslcd nsfsyncd nsnetsvc nsconmsg nscollect Runs Citrix NetScaler OS SSL VPN File Transfer RBA and SSL VPN external authorization Writes the ns.conf file CLI authentication Controls logging for the newnslog HA synchronization Used to read SSL certificate files Runs the front panel LCD Synchronizes bookmarks and SSL certificates Used by the GUI for configuration changes Controls writing of the newnslog Statistics gathering for historical purposes
  5. 5. NetScaler File System /var (hard drive) Logs - /var/log & /var/nslog 5 © 2014 Citrix. Confidential. Install - /var/nsinstall Trace - /var/nstrace Core Dumps - /var/crash & /var/core /flash (flash drive) Config - /flash/nsconfig SSL Certificates - /flash/nsconfig/ssl
  6. 6. NetScaler File System (cont.) /flash (cont.) User Monitors - /flash/nsconfig/monitors 6 © 2014 Citrix. Confidential. Custom Options - /flash/nsconfig / (ram drive) OS - (operating system)
  7. 7. Troubleshooting Tools & Techniques
  8. 8. NetScaler Tech Support Bundle > show techsupport Critical System Data In-Depth Performance Monitoring Stats Detailed Log Files USER Command Logging /var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz
  9. 9. The NetScaler Tech Support Bundle 9 © 2014 Citrix. Confidential.
  10. 10. Citrix Insight Services 10 © 2014 Citrix. Confidential.
  11. 11. 11 © 2014 Citrix. Confidential. Data Collection Analyze Data ‘Single Mission … Data Collection’ FAQ: http://support.citrix.com/article/CTX131233 Tailored Recommendations
  12. 12. https://taas.citrix.com 12 © 2014 Citrix. Confidential.
  13. 13. Technical Issues Flagged 13 © 2014 Citrix. Confidential.
  14. 14. Investigate Issues Further The ‘BELL’ 14 © 2014 Citrix. Confidential.
  15. 15. Crash File(s) Location 15 © 2014 Citrix. Confidential. nscollect-542.gz
  16. 16. Intuitive Navigation 16 © 2014 Citrix. Confidential. Select a different newnslog file to view
  17. 17. Detailed Graphs 17 © 2014 Citrix. Confidential. Mouse over graph to see more detail Informative reference legend Download the data as an excel sheet
  18. 18. IPMI – Intelligent Platform Management Interface Change NetScaler 18 © 2014 Citrix. Confidential. Default LOM IP Address: http://192.168.1.3 IP Address Obtain Health Monitoring Detail Harvest Serial Number Determine MAC Address
  19. 19. Common CLI Show Commands Common show commands for system information: show node, show info, show license Common show commands for vserver and service: show lb vserver, show cs vserver, show service, show persistencesession show connectiontable Other common show commands: show route, show ip 19 © 2014 Citrix. Confidential.
  20. 20. Common CLI Stat Commands Common stat commands for system information: stat ns, stat cpu, stat interface Common stat commands for vserver and service: stat lb vserver, stat cs vserver, stat service Other common stat commands: stat dns, stat ssl, stat http 20 © 2014 Citrix. Confidential.
  21. 21. Leveraging ‘nsconmsg’ Nsconmsg common use cases: View events View console messages View statistics Debug system counters Debug load balancing issues Debug CPU/Memory utilization 21 © 2014 Citrix. Confidential. Make absolutely sure that you use a capital -K and NOT a lower-case -k
  22. 22. Example ‘nsconmsg’ Usage # cd /var/nslog # nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg 22 © 2014 Citrix. Confidential. newnslog current log file fqdn-ssl-vip name of vserver ConLb=1 LB stats
  23. 23. Displaying debug performance information NetScaler current V20 time Performance is Sun Mar Data 23 18:33:43 2014 NetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43 current time is Sun Mar 23 18:33:43 2014 ------------------------------------------------------- NATSession : Free(6553)A(6553)InUse(0) NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0]) NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0]) Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0 SSF: Conn (Srvr 0 Clnt 0) U:0 Mon: Probes: 434562009, Failed: 15 CM: VIP(Conn 10.54.169.75:(Srvr 0 Clnt 0) Sessions 443:UP:PCB LEASTCONNS): 0 NATPCB 0 Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Z(SIP[S(10.54.148.201:0], C[0], SSL[0] Server[80:0] UP) SIPDIP[Hits(0] 7317, DIP[0] SO[0/sec, 0]) P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 Mon: Probes: 434562009, Failed: 15 VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0 S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) Load(0) LConn_Idx: (C:0; V:0,I:1) ------------------------------------------------------- CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014 23 © 2014 Citrix. Confidential. kbits) RspTime(0.00 ms) CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
  24. 24. # nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more # nsconmsg -K newnslog -s ConMon=1 -d oldconmsg # nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg # nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg ConDebug - Debugging ConLb - Load Balancing ConMon - Monitoring Probes ConMEM - Memory Management ConCSW - Content Switching ConSSL - SSL Offload ConCMP - Compression ConIC - Integrated Caching 24 © 2014 Citrix. Confidential.
  25. 25. Log File Analysis # cd /var/log # zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE) ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success" ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success" 25 © 2014 Citrix. Confidential. interface down vServer down panic signaled
  26. 26. NetScaler + Wireshark = ‘thumbs up’ http://www.wireshark.org 26 © 2014 Citrix. Confidential.
  27. 27. RED HOT Wireshark Tip 27 © 2014 Citrix. Confidential. Custom Columns Custom Menu Options Pre-build Custom Filters Much Faster Analysis Get the red hot details on how to empower your default Wireshark configuration in the Reference Section at the end of this presentation.
  28. 28. Troubleshooting Techniques & Case Studies
  29. 29. Top Tips 29 © 2014 Citrix. Confidential. Use Citrix Insight Services Pay attention to the issues! Note the highlighted counter(s) Use nsconmsg to see even more detail Correlate time-frames in other log files Achieve root cause analysis faster!
  30. 30. Case #1 - High Availability Synchronization
  31. 31. Same type of appliance 31 © 2014 Citrix. Confidential. HA Pre-requisites Same firmware version Recommend same nsroot password Same RPC Node password Open requisite TCP ports Primary NetScaler Secondary NetScaler
  32. 32. Insight Services Flags The Issues 32 © 2014 Citrix. Confidential.
  33. 33. The HA Pair Struggled To Synchronize # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted) 57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1) 57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2) 9861 0 65 nic_err_bdg_muted interface(0/1) 9862 0 65 nic_err_bdg_muted interface(0/2) 33 © 2014 Citrix. Confidential. 71,837,018 MAC Moves 65 Interface Mutes
  34. 34. The ‘newnslog’ Time-Frame # nsconmsg -K newnslog -d setime Displaying start and end time information NetScaler V20 Performance Data NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41 start time Fri Feb 28 21:49:58 2014 total duration 00.00:03:30 end time Fri Feb 28 21:53:28 2014 total duration 00.00:03:30 data size 1,718,949 bytes 34 © 2014 Citrix. Confidential.
  35. 35. Case #2 - XA/XD Slow Performance
  36. 36. Smartphones XenDesktop Tablets XenApp 36 © 2014 Citrix. Confidential. Insight Services Critical Insight Gleaned Preventative Approach Don’t Underestimate XA/XD Slow Performance
  37. 37. Insight Services Again Flags The Issues 37 © 2014 Citrix. Confidential. http://support.citrix.com/article/CTX136926
  38. 38. Performance Was Extremely Latent # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved 4263 0 23 nic_tot_bdg_mac_moved interface(0/1) 4264 0 51 nic_tot_bdg_mac_moved interface(1/1) 4265 0 28 nic_tot_bdg_mac_moved interface(1/2) 38 © 2014 Citrix. Confidential. 23, 51 & 28 MAC Moves
  39. 39. Networking Issues Again? # nsconmsg -K newnslog -d statswt0 | grep nic_err 4274 0 1995 nic_err_rl_pkt_drops interface(1/1) 4275 0 40736 nic_err_rl_pkt_drops interface(1/2) 4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1) 4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2) 4678 0 42731 allnic_err_rl_rate_pkt_drops 39 © 2014 Citrix. Confidential. System Limits Exceeded Rate-limited Packets!
  40. 40. 40 © 2014 Citrix. Confidential. The Moral of the Story Leverage Citrix Insight Services Leverage Insight Services Pay Attention Gain Quick Insight Dig Into ‘nsconmsg’ On Target for Success!
  41. 41. Resources
  42. 42. Helpful Resources Comprehensive NetScaler Counters Wireshark Developer Editions Customizing Wireshark Tutorial Citrix Insight Services Forum NSTRACE Options How To Manage VLAN’s, Interfaces and Subnets 42 © 2014 Citrix. Confidential.
  43. 43. Conclusion
  44. 44. What We’ve Actually Covered An Overview of the NetScaler System to give you a high-level understanding of the core system. I shared with you some excellent Troubleshooting Tools that are available at your disposal. I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues. I then highlighted two different Case Studies leveraging the tools & techniques that I shared with you in the presentation. In addition I provided you with a few Resources for your future reference and edification. 44 © 2014 Citrix. Confidential.
  45. 45. Fuel your talent with continuous learning. 93% of Citrix Education students became more effective in their role after attending a course. TVID: CFB-61B-A26 Citrix Education offers the following technical training for Networking professionals: CNS-205: Citrix Netscaler 10 Essentials and Networking CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam CNS-301: Citrix NetScaler 10 Advanced Implementation 45 © 2014 Citrix. Confidential. Visit (bit.ly/05Webinar) to save 10% off through September 30* *Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.
  46. 46. Simplify your journey, let us guide you. Accelerate your implementation and minimize risk by taking advantage of Citrix Consulting. You’ll get the expertise of certified Citrix Consulting Architects to successfully deploy Citrix solutions in any phase of your project. 93% of Citrix Education students became more effective in their role after attending a course. TVID: CFB-61B-A26 Visit bit.ly/CTXConsulting to learn more about our proven methodology. 46 © 2014 Citrix. Confidential.
  47. 47. 47 © 2014 Citrix. Confidential. WORK BETTER. LIVE BETTER.

×