In-depth Troubleshooting on NetScaler using Command Line Tools

Andrew Sandford
Senior Readiness Specialist, Worldwide Support Readiness EMEA
Citrix Support Secrets Webinar
Series
In-depth Troubleshooting on NetScaler using Command
Line Tools
27 March 2014

netscaler.descr: NetScaler Virtual Appliance 3G
netscaler.num_pe_running: 1
netscaler.version: NetScaler NS10.1: Build
124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.model: 3000
netscaler.vmpe_max_cpus: 2
netscaler.nCore: 1

50185 root 1 44 -52 814M 815M CPU1 1 20.2H 100.00% NSPPE-00

LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS
State: UP
Client Idle Timeout: 180 sec
Down state flush: ENABLED
Disable Primary Vserver On Down : DISABLED
Appflow logging: ENABLED
Port Rewrite : DISABLED
No. of Bound Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
Mode: IP
Persistence: NONE
Vserver IP and Port insertion: OFFWarning: Feature(s) not enabled [LB]

1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1

1) Monitor Name: mon-http-ecv
State: UP Weight: 1 Passive: 0
Probes: 14887 Failed [Total: 124 Current: 0]
Last response: Success - Pattern found in response.
Response Time: 10.220 millisec
State: UP

Show commands
System show node
show info
show license
Vserver/Service show lb vserver
show cs vserver
show service
show persistencesession
show connectiontable
IP related show route
show ip
show dns addrec -type proxy
Diagnostic show techsupport

2) Node ID: 1
IP: 192.168.1.45
Node State: STAYSECONDARY
Master State: Secondary
Fail-Safe Mode: OFF
INC State: DISABLED
Sync State: ENABLED
Propagation: ENABLED
Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4
1/3 1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2
1/1
SSL Card Status: UP
Local node information:
Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
>
Node State: NOT UP
Master Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3
1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Node State: STAYSECONDARY

NetScaler Processes
Process Description Process Description
nsppe NetScaler Packet Engine nsfsyncd
Sync bookmarks and SSL
certificates
nsaaad
RBA and SSL VPN External
Auth nsnetsvc
Used by the GUI for config
changes
nsconf Writes the ns.conf file nsumond Runs the scriptable monitors
nslog.sh Controls Logging for newnslog nsconmsg Controls writing of newnslog
nssync HA sync nscollect
Statistics gathering for
historical reporting
nsreadfile Used to read SSL Cert Files
imi/ripd/
ospfd/bgpd
Routing processes
nscrlrefresh SSL CRL list update

System
Entities
Protocols

> stat interface 1/1
Interface [1/1]:
Interface StateUP
Link uptime 00:40:21
Link downtime00:00:00
Throughput Statistics
Rate (/s) Total
Bytes received42393 54497294
Bytes transmitted 2584 20222135
Packets received 629 710246
Packets transmitted 47 69066
Packet Statistics
Rate (/s) Total
Multicast packets 18 41219
NetScaler packets 85 98954
LACP Statistics
Rate (/s) Total
LACPDUs received 0 0
LACPDUs transmitted 0 0
Error Statistics
Rate (/s) Total
Error packets received (hw) 0 0
Error packets transmitted (hw) 0 0
Inbound packets discarded (hw) 0 0
Outbound packets discarded (hw)0 0
Packets dropped in Rx (sw) 539 599904
Packets dropped in Tx (sw) 0 0
NIC hangs -- 0
Status stalls -- 0
Transmit stalls -- 0
Receive stalls -- 0
Error-disables -- 0
Duplex mismatches -- 0
Link re-initializations -- 0
MAC moves registered 0 0
Times NIC became muted -- 0

Non-authoritative entries 0
Authoritative entries 98
Error Statistics
Nonexistent domain 102359
Response class unsupported 0
Invalid query format 0
Stray answers 0
Incorrect RD length 0
Requests refused 0
Response type unsupported 0
Query class unsupported 0
Invalid response format 0
No answer responses 102334
Multi queries disabled 0
Other errors 0

Vserver hits 0 0
Requests 0 0
Responses 0 0
Request bytes 0 65
Response bytes 0 188
Total Packets rcvd 0 5
Total Packets sent 0 4
Current client connections -- 0
Current Client Est connections -- 0
Current server connections -- 0
Requests in surge queue -- 0
Requests in vserver's surgeQ -- 0
Requests in service's surgeQs -- 0
Spill Over Threshold -- 0

NetScaler File System

View events
View console messages
View statistics
Debug system counters
Debug load balancing issues
Debug CPU/Memory utilization
Use
cases

nsconmsg Common Syntax
Make sure to use –K, NOT -k

Live CPU related stats
Archived events
HA Failover cause
LB stats
CS related counters
Real-time policy hits
SSL related counters
Compression related counters

1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013
1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013
1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)'
DOWN Wed Aug 14 12:27:25 2013
1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)'
DOWN Wed Aug 14 12:27:25 2013
1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-
default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure - Probe timed out. Wed Aug
14 12:27:25 2013
1911 0 PPE-0 MonServiceBinding_192.168.204.50:53_(ping-
default)(SVC_CTXANG_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug
14 12:27:25 2013

VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec)
Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024
Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE)
override(0) newlyUP(0)
Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0
OnVserver: 0 OnServices: 0)
slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex:
0 Borrowed: 0 TotActiveConn: 0] Available: 0-

ipConflictAddr = 192.168.47.1
last
message repeated 169 times

Troubleshooting Techniques
Common NetScaler issues/resolutions
HDD Issues Logging fails
/var missing
Flash Issues Config fails to save
Config saves partially
Sync fails
Device fails to boot
Memory starvation Dropped sessions
CPU starvation All services failing
All VIPS down
Degraded performance

nstrace & nstcpdump
nstrace
Common syntax:
Nstrace.sh –sz 0
Nstrace.sh –sz 0 –filter “SOURCEIP =
10.198.4.10” –link enabled
Filter qualifiers and operators:
SOURCEIP, SOURCEPORT, DESTIP,
DESTPORT, SVCNAME, VSVRNAME, STATE
==, eq, !=, neq, >, gt, <, lt, >=, ge, <=, le,
BETWEEN
Compound filters using || and &&
nstcpdump
Common syntax:
Nstcpdump.sh –X tcp port 80
Nstcpdump.sh –w testcapture.cap –X src
host 10.198.4.10 tcp port 80
Filter qualifiers and operators:
tcpdump standard

Packet Tracing
Differences between nstrace.sh & nstcpdump.sh
Nstcpdump.sh Nstrace.sh
Useful if traces are to be viewed on standard output
nstcpdump.sh –w <filename> option helps
writing output to file
Useful for offline collection
nstrace.sh –sz 0
Saves traces in /var/nstrace in cap format
can be used with expressions so that you get to see
filtered traffic
nstcpdump.sh host <IP>
nstcpdump.sh port 21
useful for collection of traces in separate log files based
on NICs
nstrace.sh –tcpdump1 –nic 1

-h - prints this message - exclusive option
-nf - number of files to be generated in cycle (def. 24)
-time - seconds per file (def. 3600) (could be an expression)
-sz - size of the captured data (bytes from 60 to 1514)
-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format
-nic - use separate trace files for each interface (only works if -tcpdump
option is set)
-name - name of the trace file
-filter - Filter expression for nstrace. The maximum length of filter
expression is 255 and it can of following format: <expression> [<relop>
<expression>]
-link - Log filtered connection's peer's (linked connection's) traffic. Works
only with -filter option

Packet Tracing
nstrace.sh nstcpdump.sh
Proprietary capture format TCPdump PCAP capture format
Native format captures more information Useful for live capture from CLI
Files are stored in /var/nstrace Option to write to a file
Needs custom dissector in Wireshark (1.6+) Most TCPdump options supported
Works in standard Wireshark

Trace analysis
nstrace.sh
By Default if nstrace.sh is executed from shell prompt
Trace files are stored in NetScaler proprietary (.cap) format
The trace capture runs for 1 hr. (3600 sec) if not interrupted
Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB and RX packets
are captured)
The size of the captured data for each packet is 164

Trace Analysis
nstrace syntax examples
Command Purpose
# nstrace.sh Stores the traces in default (proprietary) format
# nstrace.sh -tcpdump 1 Begins to save the traces in the TCPDUMP format for a default 3600 seconds
# nstrace.sh -tcpdump 1 -nic 1 Logs the traces (in TCPDUMP format) into separate log files based on the NIC IDs
# nstrace.sh –nf <value> No of files to be generated in cycle by default is 24
# nstrace.sh –time <value> Seconds per file by default 3600 seconds
# nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire packet length
# nstrace.sh –m Capturing mode: sum of the values (def. 6):
1-Transmitted packets (TX)
2 - Packets buffered for transmission (TXB)
4 - Received packets (RX)
# nstrace.sh -stop Can be used to disable tracing (when 'nstrace.sh' is run in the background

Trace Analysis
nstcpdump.sh example syntax
Command Purpose
# nstcpdump.sh <type> Possible types are host, net and port .If there is no type qualifier host
assumed
# nstcpdump.sh <dir> Qualifiers specify a particular transfer direction to and/or
from id. Possible directions are src, dst, src or dst and
src and dst. If there is no dir qualifier, src or dst is
assumed. src foo', `dst net 128.3', `src or dst port ftp-
data'
# nstcpdump.sh <proto> ether, fddi, ip, arp, rarp, decnet, tcp and udp.
# nstcpdump.sh –c <value> Exit after receiving ‘value’ number of packets
# nstcpdump.sh –F –I –r Not supported and not to give them as options to the script
# nstcpdump.sh –w <file
name>
Write the raw packets to file rather than parsing and printing them out.
Read by typing tcpdump –r test

CTX109304 Data Collection Procedure to Troubleshoot NetScaler Related Issues
http://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for-
the-first-time
http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Ty
pe,Technotes&refinement=Product+Family,NetScaler
CTX114999 How to Troubleshoot Authentication with Aaad.debug
https://taas.citrix.com/

Maximize your knowledge.
Continue your journey with Citrix Education.
Recommended next step for hands-on technical training:
CNS-205 Citrix NetScaler 10 Essentials and Networking
Identify the capabilities and functionality of the NetScaler
Explain basic NetScaler network architecture
Obtain, install, and manage NetScaler licenses
Explain how SSL is used to secure the NetScaler
Implement NetScaler TriScale Technology, including Clustering
Visit bit.ly/NSCOURSE to save 10% now through April 30.*
*Not valid with any other promotions, packages or discounts. Applies only to new purchases. Regional limitations may apply.

About
Citrix Services
Citrix Services make sure
you succeed with your
virtualization programs.
How we can help
Citrix Education – The fastest, most efficient way to
get your team the virtualization skills they need.
Online,
on-site or in class.
citrix.com/training
Citrix Consulting – Intensive engagements for
complex, critical or just plain massive projects.
citrix.com/consulting
Citrix Support – Always-on support services that
leverage everything we know about best-practice
deployment and maintenance.
citrix.com/support
Educate | Guide | Support | Succeed

• 40 insider troubleshooting tips
• Covering XenDesktop, XenServer, XenApp and NetScaler
• Citrix Support top engineers
• FREE eBook
• Citrix Auto Support
• Now available!
Secrets of the Citrix Support Ninjas

Premier Support Calculator
Check it out

Work better. Live better. Use NetScaler

In-depth Troubleshooting on NetScaler using Command Line Tools

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie In-depth Troubleshooting on NetScaler using Command Line Tools

Ähnlich wie In-depth Troubleshooting on NetScaler using Command Line Tools (20)

Mehr von David McGeough

Mehr von David McGeough (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

In-depth Troubleshooting on NetScaler using Command Line Tools

Hinweis der Redaktion