Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Citrix TechEdge 2014 - Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

7.575 Aufrufe

Veröffentlicht am

This session will cover advanced techniques in troubleshooting the Citrix NetScaler Appliance using tools such as Citrix TaaS, IPMI, nsconmsg, wireshark and log analysis. We will review usages of these tools along with case studies showing how to best troubleshoot common issues seen in operating Citrix NetScaler Appliances.

What you will learn
- Various tools available to troubleshoot issues and how to use them to isolate NetScaler Issues
- Common deployment problems and how to isolate the causes

Veröffentlicht in: Technologie
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Citrix TechEdge 2014 - Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

  1. 1. SYN402: Advanced Tools and Techniques for Troubleshooting NetScaler Appliances Andrew Redman | Lead Escalation Engineer May 8, 2014
  2. 2. © 2014 Citrix. Confidential.2 Tweet about this session with hashtag #SYN402 and #citrixsynergy
  3. 3. © 2014 Citrix. Confidential.3 NetScaler System Overview Troubleshooting Tools & Techniques Case Studies Resources Conclusion Q&A Agenda
  4. 4. NetScaler System Overview
  5. 5. © 2014 Citrix. Confidential.5 Key NetScaler Processes ns_master/NSPPE nsvpnd nsaaad nsconf nsauthd nslog.sh nssync nsreadfile nslcd nsfsyncd nsnetsvc nsconmsg nscollect Runs Citrix NetScaler OS SSL VPN File Transfer RBA and SSL VPN external authorization Writes the ns.conf file CLI authentication Controls logging for the newnslog HA synchronization Used to read SSL certificate files Runs the front panel LCD Synchronizes bookmarks and SSL certificates Used by the GUI for configuration changes Controls writing of the newnslog Statistics gathering for historical purposes Process Description
  6. 6. © 2014 Citrix. Confidential.6 NetScaler File System /var (hard drive) Logs - /var/log & /var/nslog Install - /var/nsinstall Trace - /var/nstrace Core Dumps - /var/crash & /var/core /flash (flash drive) Config - /flash/nsconfig SSL Certificates - /flash/nsconfig/ssl
  7. 7. © 2014 Citrix. Confidential.7 NetScaler File System (cont.) /flash (cont.) User Monitors - /flash/nsconfig/monitors Custom Options - /flash/nsconfig / (ram drive) OS - (operating system)
  8. 8. Troubleshooting Tools & Techniques
  9. 9. > show techsupport Critical System Data In-Depth Performance Monitoring Stats Detailed Log Files USER Command Logging /var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz NetScaler Tech Support Bundle
  10. 10. © 2014 Citrix. Confidential.10 The NetScaler Tech Support Bundle
  11. 11. © 2014 Citrix. Confidential.11 Citrix Predictive Support
  12. 12. © 2014 Citrix. Confidential.12 ‘Single Mission … Data Collection’ FAQ: http://support.citrix.com/article/CTX131233 Data Collection Analyze Data Tailored Recommendations
  13. 13. © 2014 Citrix. Confidential.13 https://taas.citrix.com
  14. 14. © 2014 Citrix. Confidential.14 Technical Issues Flagged
  15. 15. © 2014 Citrix. Confidential.15 Investigate Issues Further The ‘BELL’
  16. 16. © 2014 Citrix. Confidential.16 Crash File(s) Location nscollect-542.gz
  17. 17. © 2014 Citrix. Confidential.17 Intuitive Navigation Select a different newnslog file to view
  18. 18. © 2014 Citrix. Confidential.18 Detailed Graphs Mouse over graph to see more detail Informative reference legend Download the data as an excel sheet
  19. 19. © 2014 Citrix. Confidential.19 IPMI – Intelligent Platform Management Interface Default LOM IP Address: http://192.168.1.3 Change NetScaler IP Address Obtain Health Monitoring Detail Harvest Serial Number Determine MAC Address
  20. 20. © 2014 Citrix. Confidential.20 Common CLI Show Commands Common show commands for system information: show node, show info, show license Common show commands for vserver and service: show lb vserver, show cs vserver, show service, show persistencesession show connectiontable Other common show commands: show route, show ip
  21. 21. © 2014 Citrix. Confidential.21 Common CLI Stat Commands Common stat commands for system information: stat ns, stat cpu, stat interface Common stat commands for vserver and service: stat lb vserver, stat cs vserver, stat service Other common stat commands: stat dns, stat ssl, stat http
  22. 22. © 2014 Citrix. Confidential.22 Leveraging ‘nsconmsg’ Nsconmsg common use cases: View events View console messages View statistics Debug system counters Debug load balancing issues Debug CPU/Memory utilization Make absolutely sure that you use a capital -K and NOT a lower-case -k
  23. 23. © 2014 Citrix. Confidential.23 Example ‘nsconmsg’ Usage # cd /var/nslog # nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg current log file newnslog name of vserver fqdn-ssl-vip LB stats ConLb=1
  24. 24. © 2014 Citrix. Confidential.24 Displaying debug performance information NetScaler V20 Performance Data NetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43 current time is Sun Mar 23 18:33:43 2014 ------------------------------------------------------- NATSession : Free(6553)A(6553)InUse(0) NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0]) NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0]) Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0 SSF: Conn (Srvr 0 Clnt 0) U:0 CM: Conn (Srvr 0 Clnt 0) Sessions PCB 0 NATPCB 0 Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0]) Mon: Probes: 434562009, Failed: 15 VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0 S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) Load(0) LConn_Idx: (C:0; V:0,I:1) ------------------------------------------------------- CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014 current time is Sun Mar 23 18:33:43 2014 Mon: Probes: 434562009, Failed: 15 VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF)S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
  25. 25. © 2014 Citrix. Confidential.25 # nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more # nsconmsg -K newnslog -s ConMon=1 -d oldconmsg # nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg # nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg ConDebug - Debugging ConLb - Load Balancing ConMon - Monitoring Probes ConMEM - Memory Management ConCSW - Content Switching ConSSL - SSL Offload ConCMP - Compression ConIC - Integrated Caching
  26. 26. © 2014 Citrix. Confidential.26 Log File Analysis # cd /var/log ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success" ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success" # zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE) interface down vServer down panic signaled
  27. 27. © 2014 Citrix. Confidential.27 NetScaler + Wireshark = ‘thumbs up’ http://www.wireshark.org
  28. 28. © 2014 Citrix. Confidential.28 RED HOT Wireshark Tip Custom Columns Custom Menu Options Pre-build Custom Filters Much Faster Analysis Get the red hot details on how to empower your default Wireshark configuration in the Reference Section at the end of this presentation.
  29. 29. Troubleshooting Techniques & Case Studies
  30. 30. © 2014 Citrix. Confidential.30 Top Tips Use Citrix Predictive Support Pay attention to the issues! Note the highlighted counter(s) Use nsconmsg to see even more detail Correlate time-frames in other log files Achieve root cause analysis faster!
  31. 31. Case #1 - High Availability Synchronization
  32. 32. © 2014 Citrix. Confidential.32 Primary NetScaler Secondary NetScaler Same type of appliance Same firmware version Same nsroot password Same RPC Node password Open requisite TCP ports HA Pre-requisites
  33. 33. © 2014 Citrix. Confidential.33 Predictive Support Flags The Issues
  34. 34. © 2014 Citrix. Confidential.34 The HA Pair Struggled To Synchronize # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted) 57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1) 57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2) 9861 0 65 nic_err_bdg_muted interface(0/1) 9862 0 65 nic_err_bdg_muted interface(0/2) 71,837,018 MAC Moves 65 Interface Mutes
  35. 35. © 2014 Citrix. Confidential.35 The ‘newnslog’ Time-Frame # nsconmsg -K newnslog -d setime Displaying start and end time information NetScaler V20 Performance Data NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41 start time Fri Feb 28 21:49:58 2014 end time Fri Feb 28 21:53:28 2014 total duration 00.00:03:30 data size 1,718,949 bytes total duration 00.00:03:30
  36. 36. Case #2 - XA/XD Slow Performance
  37. 37. © 2014 Citrix. Confidential.37 XenAppTablets Smartphones XenDesktop Predictive Support Critical Insight Gleaned Preventative Approach Don’t Underestimate XA/XD Slow Performance
  38. 38. © 2014 Citrix. Confidential.38 Predictive Support Flags The Issues http://support.citrix.com/article/CTX136926
  39. 39. © 2014 Citrix. Confidential.39 Performance Was Extremely Latent # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved 4263 0 23 nic_tot_bdg_mac_moved interface(0/1) 4264 0 51 nic_tot_bdg_mac_moved interface(1/1) 4265 0 28 nic_tot_bdg_mac_moved interface(1/2) 23, 51 & 28 MAC Moves
  40. 40. © 2014 Citrix. Confidential.40 Networking Issues Again? # nsconmsg -K newnslog -d statswt0 | grep nic_err 4274 0 1995 nic_err_rl_pkt_drops interface(1/1) 4275 0 40736 nic_err_rl_pkt_drops interface(1/2) 4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1) 4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2) 4678 0 42731 allnic_err_rl_rate_pkt_drops System Limits Exceeded Rate-limited Packets!
  41. 41. © 2014 Citrix. Confidential.41 The Moral of the Story Leverage Citrix Predictive Support Leverage Pred. Support Pay Attention Gain Quick Insight Dig Into ‘nsconmsg’ On Target for Success!
  42. 42. Resources
  43. 43. © 2014 Citrix. Confidential.43 Helpful Resources Comprehensive NetScaler Counters Wireshark Developer Editions Customizing Wireshark Tutorial Citrix Predictive Support Forum NSTRACE Options How To Manage VLAN’s, Interfaces and Subnets
  44. 44. Conclusion
  45. 45. © 2014 Citrix. Confidential.45 What We’ve Actually Covered An Overview of the NetScaler System to give you a high-level understanding of the core system. I shared with you some excellent Troubleshooting Tools that are available at your disposal. I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues. I then highlighted two different Case Studies leveraging the tools & techniques that I shared with you in the presentation. In addition I provided you with a few Resources for your future reference and edification.
  46. 46. Q & A
  47. 47. © 2014 Citrix. Confidential.47 Before you leave… Conference surveys are available online at www.citrixsynergy.com starting Thursday, May 8 at 9:00 a.m. Provide your valued feedback by 6:00 p.m. today to be entered to win one of many prizes! Download presentations starting Monday, May 19 from the My Event Planning tool
  48. 48. © 2014 Citrix. Confidential.48 WORK BETTER. LIVE BETTER.

×