Weitere ähnliche Inhalte Ähnlich wie Troubleshooting Firewalls (2012 San Diego) (20) Mehr von Cisco Security (20) Kürzlich hochgeladen (20) Troubleshooting Firewalls (2012 San Diego)2. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Troubleshooting Firewalls
Session BRKSEC-3020
3. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
3
5. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Understanding the Packet Flow
To effectively troubleshoot a problem, one must first understand the
packet path through the network
Attempt to isolate the problem down to a single device
Then perform a systematic walk of the packet path through the device to
determine where the problem could be
For problems relating to the Cisco ASA/FWSM, always
‒ Determine the flow: SRC IP, DST IP, SRC port, DST port,
and protocol
‒ Determine the interfaces through which the flow passes
Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and
Egress) and the Rules Tied to Both
5
6. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Example Flow
Flow
SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP
DST IP: 198.133.219.25 DST Port: 80
Interfaces
Source: Inside Destination: Outside
With the Flow
Defined, Examination
of Configuration
Issues Boils Down
to Just the Two
Interfaces: Inside
and Outside
Packet Flow
Accounting
Eng
Client: 10.1.1.9
Servers
Outside
Server: 198.133.219.25
6
7. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: Ingress Interface
Packet arrives on ingress interface
Input counters incremented
Software input queue is an indicator of load
No buffers and overruns indicates packet drops (usually traffic bursts)
ASA# show interface GigabitEthernet0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel10
MAC address 4055.3980.0e67, MTU 1500
IP address unassigned
410179 packets input, 42620992 bytes, 0 no buffer
Received 138749 broadcasts, 0 runts, 0 giants
…
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
7
8. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: Locate Connection
Check first for existing connection
If connection exists, flow is matched: bypass ACL check
If no existing connection
‒ TCP non-SYN packet, drop and log
‒ TCP SYN or UDP packet, pass to ACL checks
Established Connection:
ASA-5540# show conn
TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No existing connection, and non-SYN Packet:
ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags
PSH ACK on interface inside
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
8
9. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: NAT Un-Translate
Incoming packet is checked against NAT rules
Starting version 8.3+, packet is un-translated first, before ACL check
‒ In version 8.2, incoming packet was subjected to ACL check prior to un-translation
More on this in the NAT section of this presentation…
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
9
10. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: ACL Check
First packet in flow is processed through interface ACLs
ACLs are first match
First packet in flow matches ACE, incrementing hit count by one
Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside
access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog when Packet Is Denied by ACL:
ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-
group "inside"
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
10
11. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: Inspections/Sec Checks
Inspections are applied to ensure protocol compliance
(Optional) customized AIC inspections
NAT-embedded IPs in payload
Additional security checks are applied to the packet
(Optional) packets passed to Content Security and Control
(CSC) module
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to
209.165.202.130 on interface inside
ASA-4-405104: H225 message received from outside_address/outside_port to
inside_address/inside_port before SETUP
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
11
12. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: NAT IP Header
Translate the IP address in the IP header
Translate the port if performing PAT
Update checksums
(Optional) Following the above, pass packet to IPS (AIP) module
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
12
13. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: Egress Interface
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)
Egress interface is determined first by translation rules NOT the routing table
If translation rules do not specify egress interface (e.g., outbound initial packet) the
results of a global route lookup are used to determine egress interface
Example:
DMZ
Inside Outside
172.16.0.0/16
172.16.12.0/24
172.16.12.4
Inbound Packets to
192.168.12.4 Get Routed to
Inside Based on Order of
Statics
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net
nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net
13
14. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: L3 Route Lookup
Once on egress interface, an interface route lookup
is performed
Only routes pointing out the egress interface are eligible
Remember: NAT rule can forward the packet to the egress interface, even though the
routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
%ASA-6-110003: Routing failed to locate next hop for TCP from
inside:192.168.103.220/59138 to dmz:172.18.124.76/23
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
14
15. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: L2 Address Lookup
Once a Layer 3 route has been found, and next hop identified, Layer 2
resolution is performed
Layer 2 rewrite of MAC header
If Layer 2 resolution fails—no syslog
show arp will not display an entry for the L3 next hop
debug arp will indicate if we are not receiving an ARP reply
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
arp-req: generating request for 10.1.2.33 at interface outside
arp-req: request for 10.1.2.33 still pending
15
16. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Processing: Transmit Packet
Packet is transmitted on wire
Interface counters will increment on interface
ASA# show interface GigabitEthernet 0/2
Interface GigabitEthernet0/2 "", is up, line protocol is up
Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel11
MAC address 4055.3980.0e68, MTU 1500
IP address unassigned
…
101315 packets output, 13086040 bytes, 0 underruns
0 pause/resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 rate limit drops
0 switch egress policy drops
0 input reset drops, 0 output reset drops
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
16
17. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
17
18. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Cisco ASA — Understanding the Architecture
ASA processes all packets in software (via the central CPU)
All packets are processed first in… usually also first out
ASA platforms have software imposed connection limits
Multi-CPU / Multi-Core systems hash packets in the same flow to the
same CPU/core.
10 Gig interfaces hash flow to same RX ring.
Architecture optimized for multi-flow traffic patterns
ASASM packet processing is also done in software, unlike FWSM
18
19. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Cisco ASA — Understanding the Architecture
Global Shared Memory Pool
‒ Global Shared Pool of Memory used by ASA processes
Show free memory with CLI, ASDM, or ‗show memory‘ and ‗show memory detail‘ commands
ASA# show memory
Free memory: 250170904 bytes (47%)
Used memory: 286700008 bytes (53%)
------------- ------------------
Total memory: 536870912 bytes (100%)
ASA#
• If available memory trends down over time, call Cisco TAC
• For 64-bit software (8.4+) use CISCO-ENHANCED-MEMPOOL-MIB.my
for accurate counters
Use ASDM to graph free memory available
%ASA-3-211001: Memory allocation Error
19
20. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Cisco ASA — Understanding the Architecture
Memory Blocks
‒ Fixed-size blocks of memory allocated at startup, used for packet processing,
VPN, etc
ASA# show blocks
SIZE MAX LOW CNT
0 400 397 400
4 100 99 99
80 403 379 401
256 1200 1190 1195
1550 6511 803 903
2048 1200 1197 1200
2560 264 264 264
4096 100 100 100
8192 100 100 100
16384 102 102 102
65536 16 16 16
ASA#
1550 and 2048 byte blocks are used for processing ethernet frames
Use ASDM to graph free blocks available
Current number of free
blocks available
%ASA-3-321007: System is low on free memory blocks of size
1550 (10 CNT out of 7196 MAX)
20
21. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Maximum ACL Limits
No hard-coded limit on the number of elements (ACEs) in an
ACL. Bound only by Memory.
Each ACE uses a minimum of 212 bytes of RAM
However, maximum performance may decrease (typically 10-
15%) as you reach or exceed the Max Recommended ACEs.
5505 5510 5512-X 5515-X 5520 5525-X 5540 5545-X 5550 5555-X 5580
5585
10/20/40/6
0
ASA
SM
Max
Recommended
ACEs
25k 80k 100k 100k 200k 200k 500k 300k 700k 500k 750k
500k / 750k
1 / 2 million
2 million
Tested ACEs 80k 300k 700k 700k
1
million+
500k / 750k
1 / 2 million
2 million
Max Observed
(from customers)
2.74 million
15.959
million
3.5 million
(SSP-40)
Note: Issue show access-list | include elements to see how many ACEs you have
21
22. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Warning - ACE Explosion
Object-groups:
• Sources (10 addresses)
• Destinations (21 addresses)
• Ports (33 ports)
• Result: 10x21x33 = 6,930 rules
Nested object-groups:
• Assume you add a SRC object-group to the above, which contains 25 additional sources
• Result: (10+25)x21x33 = 24,255 rules (ACEs)
• New command to reduce ACL memory impact for large ACLs. Available starting in 8.3(1)
ASA-5585(config)# object-group-search access-control
Single line
ACL
explodes to
22
23. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Global ACLs
Global ACLs introduced in version 8.3
Best used for new installations, or config migration from other vendors
Interface Independent Policies
ASA
Only
ASA(config)# access-group <access_list> global
Global access-list
Interface Specific access-list
Default (implicit) deny ip any any
Policy Ordering
23
24. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Object-NAT (Auto-NAT) (version 8.3+)
Object NAT is the simplest form of NAT, and is defined within an object
object network obj-WebServer
host 10.3.19.50
nat (inside,outside) static 198.51.100.50
object network Servers
subnet 10.0.54.0 255.255.255.0
nat (inside,outside) static 203.0.113.0
object network InternalUsers
subnet 192.168.2.0 255.255.255.0
nat (inside,outside) dynamic interface
Host NAT
Network NAT
Dynamic PAT (interface overload)
24
25. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Manual-NAT (Twice-NAT) (version 8.3+)
Manual NAT can specify the source and the destination translation
object network 10.10.10.0-net
subnet 10.10.10.0 255.255.255.0
!
object network 192.168.1.0-net
subnet 192.168.1.0 255.255.255.0
nat (inside,outside) source static 10.10.10.0-net 10.10.10.0-net
destination static 192.168.1.0-net 192.168.1.0-net
Network Objects
Manual NAT Config
25
26. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
NAT Order of Operation version 8.3+
The ASA configuration is built into the NAT table
The NAT Table is based on First Match (top to bottom)
The show nat command will display the NAT table in order
Manual NAT Policies
(Section 1)
Auto NAT Policies
(Section 2)
Manual NAT [after auto] Policies
(Section 3)
Static NAT
Dynamic NAT
NAT Table
Longest Prefix
Shortest Prefix
Longest Prefix
Shortest Prefix
First Match
(in config)
First Match
(in config)
For your
reference
26
27. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Network Address Translation in ASA 8.3+
no-proxy-arp 8.4.(2)+/8.5(1)
‒ For static NAT, disables proxy ARP for incoming packets to the mapped IP addresses.
route-lookup 8.4.(2)+/8.5(1)
‒ For identity NAT in routed mode, determines the egress interface using a route lookup instead of using the
interface specified in the NAT command. If you do not specify interfaces in the NAT command, a route
lookup is used by default.
pat-pool <obj> 8.4.(2)+/8.5(1)
‒ Enables a PAT pool of addresses; all addresses in the object are used as PAT addresses and not dynamic 1-
to-1
round-robin 8.4.(2)+/8.5(1)
‒ Enables round-robin address allocation for a PAT pool. By default, all ports for a PAT address will be
allocated before the next PAT address is used. Round Robin will pick one address from each PAT address
extended 8.4.(3)
‒ Enables extended PAT for a PAT pool. Extended PAT uses 65535 ports per service, as opposed to per IP
address, by including the destination address and port in the translation information.
Optional NAT Parameters in Manual NAT
27
For your
reference
28. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Real-IP (version 8.3+)
Finally, a reminder that with 8.3+ Real-IPs are used in ACLs
object network obj-WebServer
host 10.3.19.50
nat (inside,outside) static 198.51.100.50
!
access-list allowIn permit tcp any host 10.3.19.50 eq 80
!
access-group allowIn in interface outside
ACL contains REAL
(local) IP of server
28
29. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—Understanding the Architecture
FWSM Processes most packets in hardware, with some packets needing
to be processed in software—via the Control Point (CP)
Packets processed in hardware have zero impact on CPU
Similarly, if the CPU is pegged at 100%, this has zero impact on packets
processed in hardware
Note that FWSM packet processing is different from ASA
29
30. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
C6K Backplane Interface
Session Manager
NP 3
Control Point (CP)
Central CPU
Fast Path
NP 1
FWSM
Fast Path
NP 2
FWSM Architectural Overview
Software
Hardware
Control Point
ACL Compilation,
Fixups, Syslog, AAA, IPv6 in
Software
Session Manager
Session Establishment and
Teardown, AAA Cache, ACLs
Fast Path
Flow Identification
Security Checks and
NAT in hardware
30
31. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—Hardware Limits
FWSM has several hardware limits that should be considered in your
network design
Limits are hard set, but vary based on single or multimode
Some limits include:
See
Appendix
2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode)
3.2 / 4.0 /4.1
Configurable
ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X
AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X
Global Statements 1K (1K) 4K (4K) 4K (4K)
Static NAT Statements 2K (2K) 2K (2K) 2K (2K)
Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X
NAT Translations 256K (256K) 256K (256K) 256K (256K)
Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990)
Route Table Entries 32K (32K) 32K (32K) 32K (32K)
Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X
Filter Statements 3942 (606) 2764 (425) 3747 (576) X
Increase over 2.3 Increase over 3.2
*Complete List in FWSM Docs, Appendix A (Specifications)
31
32. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show np blocks (FWSM Only)
The show np blocks command is used to see if the FWSM is over
subscribed
FWSM# show np blocks
MAX FREE THRESH_0 THRESH_1 THRESH_2
NP1 (ingress) 32768 32768 0 0 550
(egress) 521206 521206 0 0 0
NP2 (ingress) 32768 32768 0 0 92
(egress) 521206 521206 0 0 0
NP3 (ingress) 32768 32768 13 460417 4427509
(egress) 521206 521206 0 0 0
Warning
Data packets
dropped
Data and Control
packets dropped
32
33. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
33
34. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Failover Basics
Active/Standby vs. Primary/Secondary
Stateful failover (optional)
A failover only occurs when either firewall
determines the standby firewall is healthier
than the active firewall
Both firewalls swap MAC
and IP addresses when a failover occurs
Level 1 syslogs will give
reason of failover
LAN Link
Stateful
Internet
Corp
Secondary
(Standby)
Primary
(Active)
34
35. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
ASA# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover Redundant5 (up)
Unit Poll frequency 200 milliseconds, holdtime 1 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 2 of 250 maximum
Version: Ours 8.2(2), Mate 8.2(1)
Last Failover at: 10:37:11 UTC May 14 2010
This host: Primary - Active
Active time: 1366024 (sec)
slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
Interface outside (10.8.20.241): Normal
Interface inside (10.89.8.29): Normal
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys)
Interface outside (10.8.20.242): Normal
Interface inside (10.89.8.30): Normal
Stateful Failover Logical Update Statistics
Link : stateful Redundant6 (up)
Stateful Obj xmit xerr rcv rerr
General 424525 0 424688 0
sys cmd 423182 0 423182 0
Verifying Failover Configuration
Interface Monitoring
35
36. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
What Triggers a Failover?
Power loss/reload (this includes crashes) on the
Active firewall
SSM interface/module failure
The Standby becoming healthier than the Active firewall
In LAN based Failover, what happens if the LAN interface
communication is severed?
36
37. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
What Triggers a Failover? (Con’t)
Two consecutive hello messages missed on any monitored interface
forces the interface into testing mode
Both units first verify the link status on the interface
Next, both units execute the following tests
Network activity test
ARP test
Broadcast ping test
The first test passed causes the interface on that unit to be marked
healthy; only if all tests fail will the interface be marked failed
37
38. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
How Well do you Understand Failover?
What Happens When…
You disable failover? (By issuing no failover)
You RMA/Replace the Primary unit?
You don‘t define Standby IP addresses on interfaces?
38
39. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
What to Do After a Failover
Always check the syslogs to determine root cause
‒ Example: switch port failed on inside interface of
active firewall
Syslogs from Primary (Active) ASA
Syslogs from Secondary (Standby) ASA
ASA-4-411002: Line protocol on Interface inside, changed state to down
ASA-1-105007: (Primary) Link status ‗Down‘ on interface 1
ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier
ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active
39
40. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
40
41. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Troubleshooting Tools
Syslogs
Debug commands
Show commands
Packet capture
Packet tracer
TCP Ping
41
42. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Uses of Syslogs
Primary mechanism to record traffic to and through
the firewall
The best troubleshooting tool available
Local
Buffer
Monitor (ssh, telnet)
Archival Purposes Debugging Purposes
Syslog/FTP Server
SNMP Server
Console
Trap .
Syslog
ASDM
flash
42
43. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
ASA Syslog Level vs. Number of Messages
Log
Level
Description
Number of Messages (SUM)
Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1 Ver. 8.2 Ver. 8.3 Ver. 8.4
0 Emergencies 0 0 0 0 0 0 0
1 Alerts 62 (62) 77 (77) 78 (78) 87 (87) 87 (87) 95 (95) 109 (109)
2 Critical 29 (91) 35 (112) 49 (127) 50 (137) 56 (143) 57 (152) 63 (172)
3 Errors 274 (365) 334 (446) 361 (488) 363 (500) 384 (527) 408 (560) 448 (620)
4 Warnings 179 (544) 267 (713) 280 (768) 281 (781) 315 (842) 324 (884) 357 (997)
5 Notifications 161 (705) 206 (919) 216 (984) 218 (999) 237 (1079) 246 (1130) 265 (1242)
6 Informational 234 (939) 302 (1221) 335 (1319) 337 (1336) 368 (1447) 377 (1507) 395 (1637)
7 Debugging
217
(1156)
258 (1479) 266 (1585) 267 (1603) 269 (1716) 269 (1776) 276 (1913)
More messages
43
44. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
What Are Modifiable Syslog Levels?
Modifiable syslog levels
Allows one to move any syslog message to
any level
Problem
You want to record what exec commands are
being executed on the firewall; syslog ID 111009
records this information, but by default it is at
level seven (debug)
%ASA-7-111009: User ‗johndoe‘
executed cmd: show run
The problem is we don‘t want to log all 1775
other syslogs that are generated at debug level
[no] logging message <syslog_id> level <level>
Levels
0—Emergency
1—Alert
2—Critical
3—Errors
4—Warnings
5—Notifications
6—Informational
7—Debugging
44
45. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
How to Create Modifiable Syslog Levels
Lower syslog message 111009 to level 3 (error)
Now our syslog looks as follows
To restore the default syslog level
Tip: Use show logging message all to see the default level for any message
ASA(config)# logging message 111009 level 3
ASA-3-111009: User ‗johndoe‘ executed cmd: show run
ASA(config)# no logging message 111009 level 3
If you were only interested in logging one syslog message, how
could you do it?
45
46. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Logging – Common Issues
logging flash-bufferwrap – should only be used when logging to buffer
at Level 1
logging history – should only be used when you really have an SNMP
server that you want to receive all syslogs
logging console – should only be enabled while actively
troubleshooting on the Console
logging standby – should only be used if you want to receive double
the syslogs
logging permit-hostdown – should always be used with TCP
syslogging
46
47. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem
2. Debugs can negatively impact the CPU of the box, and also the
performance of it; use with caution
3. Debugs are not conditional*
4. Know how much traffic, of the specified type, is passing through the
firewall before enabling the respective debug
47
48. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Show Output Filters
Use output filters to filter the output of show command to only the
information you want to see
To use them, at the end of show <Command>, use the pipe character ―|‖
followed by
‒begin Start displaying the output beginning at the first match of the
RegEx, and continue to display the remaining output
‒include Display any line that matches the RegEx
‒exclude Display any line that does not match the RegEx
‒grep Same as include
‒grep –v Same as exclude
See
Appendix
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
48
49. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show cpu usage
Under normal conditions the CPU should stay below 50% (baseline as per
network); if the CPU reaches 100% the firewall will start dropping packets
FWSM CPU is used for limited traffic processing;
during ACL compilation CPU is expected to be near 100% until ACL is
compiled
The show cpu usage command displays the CPU over time as a
running average
ASA# show cpu usage
CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
49
50. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show processes cpu-usage
The show processes cpu-usage command displays the amount of
CPU used on a per-process basis for the last 5sec, 1min, and 5min
ASA# show process cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
0x08dc4f6c 0xc81abd38 14.4% 8.2% 8.0% SNMP Notify Thread
0x087798cc 0xc81b0658 6.8% 5.0% 4.9% esw_stats
0x081daca1 0xc81bcf70 1.3% 1.1% 1.0% Dispatch Unit
0x08e7b225 0xc81a28f0 1.2% 0.1% 0.0% ssh
0x08ebd76c 0xc81b5db0 0.6% 0.3% 0.3% Logger
0x087b4c65 0xc81aaaf0 0.1% 0.1% 0.1% MFIB
0x086a677e 0xc81ab928 0.1% 0.1% 0.1% ARP Thread
ASA#
*First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1)
50
51. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Show traffic
The show traffic command displays the traffic received and
transmitted out each interface of the firewall since last ‗clear traffic ‘
ASA# show traffic
outside:
received (in 124.650 secs):
295468 packets 167218253 bytes
2370 pkts/sec 1341502 bytes/sec
transmitted (in 124.650 secs):
260901 packets 120467981 bytes
2093 pkts/sec 966449 bytes/sec
inside:
received (in 124.650 secs):
261478 packets 120145678 bytes
2097 pkts/sec 963864 bytes/sec
transmitted (in 124.650 secs):
294649 packets 167380042 bytes
2363 pkts/sec 1342800 bytes/sec
51
52. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show xlate and show xlate debug
The show xlate command displays information about the translations
through the firewall
You can limit the output to just the local or global IP
ASA-5585# show xlate
5014 in use, 5772 most used
TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756
flags ri idle 0:00:00 timeout 0:00:30
TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464
flags ri idle 0:00:00 timeout 0:00:30
ASA-5585# show nat pool
TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1
TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0
TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 2321
ASA-5585#
Added in version 8.3
52
53. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show nat detail
The show nat command displays information about the nat table of the
firewall
The detail keyword will display object definitions
ASA-5585# show nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj
vpn-obj
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16
Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24
Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static webserver-obj 14.36.103.83
translate_hits = 0, untranslate_hits = 3232
Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32
2 (inside) to (outside) source dynamic science-obj interface
translate_hits = 37723, untranslate_hits = 0
Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16
ASA-5585/admin#
53
54. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Show Conn and Show Conn Detail
54
ASA# show conn
2 in use, 64511 most used
TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO
UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –
ASA# show conn detail
2 in use, 64511 most used
Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,
E - outside back connection, F - outside FIN, f - inside FIN,
G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
k - Skinny media, M - SMTP data, m - SIP media, n - GUP
O - outbound data, P - inside back connection, q - SQL*Net data,
R - outside acknowledged FIN,
R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,
X - inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,
flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127
UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,
flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
detail Adds Uptime
and Timeout in 7.2(4),
8.0(4)
Idle Time,
Bytes Transferred
Connection
Flags
Real Interface
Names Added in
7.2(4), 8.0(4)
55. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
3
ACK
5
Data
1
SYN
4
Data
2
SYN+ACK
Example—Connection Build Up
Firewall receives an initial SYN packet from the inside; the SYN is permitted
by the access-list, a translation (xlate) is built up, and the connection is also
created with the flags saA
The outside device responds to the SYN packet with a SYN+ACK;
the connection flags are updated to reflect this, and now show A
The inside device responds to the SYN+ACK with an ACK and this
completes the TCP three-way handshake, and the connection is now
considered up (U flag)
The outside device sends the first data packet; the connection is updated and
an I is added to the flags to indicate the firewall received Inbound data on
that connection
Finally, the inside device has sent a data packet and the connection is
updated to include the O flag
U
saA
A
UIO
UI
Connection Flags
Client Server
Outside
Inside
55
56. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
1
FIN
2
FIN+ACK
UfFR
Uf
3
ACK
UfFR
r
Example—Connection Teardown
Firewall receives a FIN packet from the inside; as the FIN passes through
the firewall, it updates the connection flags by adding an f to indicate that the
FIN was received on the Inside interface
The outside device immediately responds to the FIN packet with a FIN+ACK;
the connection flags are updated to reflect this, and now
show UfFR
The inside device responds to the FIN+ACK with a final ACK and the
firewall tears down the connection; thus, there are no more connection flags,
because the connection no longer exists
Connection Flags
Client Server
Outside
Inside
56
57. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
For your
reference
Outbound Connection Inbound Connection
Connection Flags—Quick
Reference
57
58. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Connection Termination Reasons
If a TCP connection is built through the firewall, it will
always have a teardown reason
The TCP teardown syslog is logged at level six
If you are having problems with connections abnormally
closing, temporally increase your logging level (or move
the syslog down), and check the teardown reason
ASA-6-302014: Teardown TCP connection number for
intf_name:real_IP/real_port to intf_name:real_IP/real_port
duration time bytes number [reason] [(user)]
What does the Reset-O Termination reason mean in the
Teardown TCP connection syslog?
58
59. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
For your
reference
TCP Connection Termination Reasons—Quick Reference
Reason Description
Conn-Timeout
Connection Ended Because It Was Idle Longer Than the
Configured Idle Timeout
Deny Terminate Flow Was Terminated by Application Inspection
Failover Primary Closed
The Standby Unit in a Failover Pair Deleted a Connection
Because of a Message Received from the Active Unit
FIN Timeout
Force Termination After Ten Minutes Awaiting the Last ACK or
After Half-Closed Timeout
Flow Closed by Inspection Flow Was Terminated by Inspection Feature
Flow Terminated by IPS Flow Was Terminated by IPS
Flow Reset by IPS Flow Was Reset by IPS
Flow Terminated by
TCP Intercept
Flow Was Terminated by TCP Intercept
Invalid SYN SYN Packet Not Valid
Idle Timeout
Connection Timed Out Because It Was Idle Longer than the
Timeout Value
IPS Fail-Close Flow Was Terminated Due to IPS Card Down
SYN Control Back Channel Initiation from Wrong Side
59
60. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
For your
reference
TCP Connection Termination Reasons—Quick Reference
(Cont.)
Reason Description
SYN Timeout
Force Termination After Two Minutes Awaiting
Three-Way Handshake Completion
TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission
TCP Fins Normal Close Down Sequence
TCP Invalid SYN Invalid TCP SYN Packet
TCP Reset-I TCP Reset Was Sent From the Inside Host
TCP Reset-O TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment
TCP Unexpected Window
Size Variation
Connection Terminated Due to a Variation in the
TCP Window Size
Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down
Unauth Deny Connection Denied by URL Filtering Server
Unknown Catch-All Error
Xlate Clear User Executed the ‗Clear Xlate‘ Command
60
61. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show local-host command
A local-host entry is created for any IP tracked through the firewall
It groups the xlates, connections, and AAA information
Very useful for seeing the connections terminating on servers
ASA# show local-host detail connection tcp 50
Interface dmz: 0 active, 0 maximum active, 0 denied
Interface inside: 1 active, 1 maximum active, 0 denied
local host: <192.168.103.220>,
TCP flow count/limit = 798/unlimited
TCP embryonic count to host = 0
TCP intercept watermark = unlimited
UDP flow count/limit = 0/unlimited
Conn:
TCP outside:172.18.124.76/80 inside:192.168.103.220/34078,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
TCP outside:172.18.124.76/80 inside:192.168.103.220/34077,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
(output truncated)
61
62. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show service-policy
The show service-policy command is used to quickly see what
inspection policies are applied and the packets matching them
ASA-5585/admin# show service-policy
Global policy:
Service-policy: global_policy
Class-map: inspection_default
Inspect: ftp, packet 0, lock fail 0, drop 0, reset-drop 0
…
Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0
Inspect: http, packet 1215927, lock fail 0, drop 0, reset-drop 0
Inspect: icmp, packet 57, lock fail 0, drop 0, reset-drop 0
ASA-5585/admin# …
62
63. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show service-policy flow
Use to determine what policies a given flow will match in the Modular
Policy Framework (MPF)
ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521
Global policy:
Service-policy: global_policy
Interface outside:
Service-policy: outside
Class-map: oracle-dcd
Match: access-list oracle-traffic
Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet
Action:
Input flow: set connection timeout dcd
63
64. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show asp drop
Packets dropped in the Accelerated Security Path (ASP) will increment a
counter
FWSM – applies only to traffic sent to the control-point
Frame drop counters are per packet, flow drops are per flow
Some counters have corresponding syslogs
ASA# show asp drop
Frame drop:
Invalid encapsulation (invalid-encap) 10897
Invalid tcp length (invalid-tcp-hdr-length) 9382
Invalid udp length (invalid-udp-length) 10
No valid adjacency (no-adjacency) 5594
No route to host (no-route) 1009
Reverse-path verify failed (rpf-violated) 15
Flow is denied by access rule (acl-drop) 25247101
First TCP packet not SYN (tcp-not-syn) 36888
Bad TCP flags (bad-tcp-flags) 67148
TCP option list invalid (tcp-bad-option-list) 731
TCP MSS was too large (tcp-mss-exceeded) 10942
Bad TCP Checksum (bad-tcp-cksum) 893
*Drop Counters Are Documented in the CMD Ref, Under show asp drop
64
65. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Capture
Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later
ASA 7.2(3) and 8.0(3) added a real-time option
ASDM 6.0 adds a capture wizard
Capture sniffs packets on an interface that match an ACL, or match line
Key steps
‒ Use the ‗match‘ keyword to specify what traffic to capture (implicitly bi-directional)
‒ Define the capture and bind it to an access-list and interface
‒ View the capture on the firewall, or copy it off in .pcap format
ASA# capture outcap interface outside match ip any host 10.1.2.3
ASA# capture incap interface inside match ip any host 10.1.2.3
ASA#
ASA# show cap outcap
2 packets captured
1: 16:04:28.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request
2: 16:04:29.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request
65
66. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Capture (Cont.)
Traffic can be captured both before and after it passes
through the firewall; one capture on the inside interface,
one capture on the outside interface
Capture buffer saved in RAM (default size 512 KB)
Default is to stop capturing when buffer is full
Default packet length is 1518 bytes
Copy captures off via TFTP or HTTPS
Outside
Inside
Capture In Capture Out
See
Appendix
Inside Capture Outside Capture
66
67. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Where Packets Are Captured in Packet Flow
Packets are captured at the first and last points
they can be in the flow
Ingress packets are captured before any packet
processing has been done on them
Egress packets are captured after all processing
(including L2 source MAC rewrite)
Ingress Packets
Captured
Egress Packets
Captured
RX
Pkt
Ingress
Interface
Existing
Conn
NAT
Untranslate
ACL
Permit
Inspections
Sec
Checks
CSC
Modul
e
NAT IP
Header
Egress
Interface
L3
Route
TX
Pkt
IPS
Module
L2
Addr
No
Yes
No
DROP
Yes
No
DROP
No
DROP
No
DROP
No
DROP
Yes Yes
Yes
67
68. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Capturing Packets Dropped by the ASP
Capture all packets dropped by the ASP
Capture on a specific drop reason
Applies to both ASA and FWSM
ASA# capture drop type asp-drop ?
acl-drop Flow is denied by configured rule
all All packet drop reasons
bad-crypto Bad crypto return in packet
bad-ipsec-natt Bad IPSEC NATT packet
bad-ipsec-prot IPSEC not AH or ESP
bad-ipsec-udp Bad IPSEC UDP packet
bad-tcp-cksum Bad TCP checksum
bad-tcp-flags Bad TCP flags
ASA
Only
ASA# capture drops type asp-drop all
ASA# capture drops type asp-drop tcp-not-syn
68
69. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Tracer - Overview
Introduced in ASA version 7.2
A packet tagged with the trace option is injected into the interface, and
processed in the data-plane
Each action taken on the packet is recorded in the packet itself
When the packet reaches the egress interface,
or is dropped, it is punted to the control-plane
The control-plane reads and displays the actions
taken on the packet, along with the associated lines
in the configuration
ASA
Only
69
70. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Link Back
to Edit Rule
Matching
Config
Define
Packet
Action
Final Result
Packet Tracer: ASDM (Located off Tools Menu)
70
71. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Tracer: Example Output
ASA# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
NAT divert to egress interface dmz
Untranslate 172.18.254.139/3389 to 192.168.103.221/3389
…….
71
72. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Tracer: Example Output
Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_in in interface outside
access-list outside_in extended permit tcp any any eq 3389
Additional Information:
……
Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface jajohnstWin7-vm service rdp-outside rdp-
outside
Additional Information:
Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234
……
Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16538274, packet dispatched to next module
72
73. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Packet Tracer: Tracing Captured Packet
Create a capture using the trace option
Find the packet in the capture you want traced
Then select that packet to be traced
ASA# show capture inside trace packet-number 4 .
ASA# capture inside access-list web interface inside trace
ASA# show capture inside
68 packets captured
1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S
2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack
3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack
4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack
5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack
...
Important!
73
74. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Ping
New
www server
(209.165.200.225)
New troubleshooting tool added in ASA ver 8.4.1
Why is it needed??? Consider the following…
10.1.1.7
74
75. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Ping
Previously – limited reachability tools:
Ping and Traceroute
Access to client machine?
www server
(209.165.200.225)
ICMP Echo Request
ICMP Echo Reply
ICMP Echo Request
ICMP Echo Reply
ICMP Echo Reply
ICMP Echo Request
Attempts to validate the path
…but with ICMP
What about
NAT and/or PAT?
10.1.1.7
75
76. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Ping
Sources TCP SYN packet with Client’s IP and
injects it into Client’s interface of the ASA
Internal hosts are PATed
to 198.51.100.2
www server
(209.165.200.225)
10.1.1.7
inside outside
Packet with SRC
of 10.1.1.7 injected
on Inside interface
Packet PATed to
198.51.100.2
on Egress
ASA Datapath
Validated
(NAT, ACLs, etc)
TCP SYN sent
to server
TCP SYN+ACK
sent from server
76
77. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Ping – The Big Picture
Validates 2 of the 3 legs of the connection from
client to server
www server
(209.165.200.225)
10.1.1.7
inside outside
TCP path from client side of ASA
to Server through the cloud
-Validated-
2nd Leg
1st Leg 3rd Leg
77
78. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Ping - Example
www server
(209.165.200.225)
10.1.1.7
inside outside
asa# ping tcp
Interface: inside
Target IP address: 209.165.200.225
Target IP port: 80
Specify source? [n]: y
Source IP address: 10.1.1.7
Source IP port: [0]
Repeat count: [5]
Timeout in seconds: [2]
Type escape sequence to abort.
Sending 5 TCP SYN requests to 209.165.200.225 port 80
from 10.1.1.7 starting port 3465, timeout is 5 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Specify Client‘s
source Interface
Specify Client‘s
real IP Address
78
79. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
79
81. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Smart Call Home
Objective – Send the output of a command directly to your e-mail.
This is easily accomplished with SCH. Use the command:
call-home send <―cmd‖> email <email_addr>
Example:
call-home send ―show run‖ email userid@cisco.com
This will send a plain-text e-mail with the output of the command to the e-
mail address specified, with the command in the subject line.
‒ Example:
Subject: CLI ‗show run‘ output
Email CMD Output to You
ASA
Only
81
82. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Smart Call Home
Objective – Memory appears to be depleting over time on your ASA.
Use SCH to collect the detailed memory output hourly, for further
investigation.
This is easily accomplished with SCH. Setting a ‖snapshot‖ alert-
group to e-mail commands at a specified interval
Snapshot will contain the following command:
show conn count
show memory detail
Collecting Memory Diagnostics over Time
82
83. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Smart Call Home
Example Config
service call-home
call-home
alert-group-config snapshot
add-command ―show conn count‖
add-command "show memory detail―
contact-email-addr user@cisco.com
sender from user@cisco.com
sender reply-to user@cisco.com
mail-server smtp-server.cisco.com priority 1
profile SENDCMD
active
destination address email user@cisco.com
destination preferred-msg-format long-text
destination transport-method email
subscribe-to-alert-group snapshot periodic hourly
83
85. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Clients
Problem
Most external clients are not able to load company‘s
web page
10.1.1.50
ASA-5510
HTTP Requests to 192.168.1.50
Web Server
NATed to 10.1.1.50
Internet
Case Study: Intermittent Access to Web Server
85
86. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Traffic Spike
Case Study: Intermittent Access to Web Server
86
87. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
show perfmon indicates high number of embryonic
connections
ASA-5510# show perfmon
PERFMON STATS: Current Average
Xlates 0/s 0/s
Connections 2059/s 299/s
TCP Conns 2059/s 299/s
UDP Conns 0/s 0/s
URL Access 0/s 0/s
URL Server Req 0/s 0/s
TCP Fixup 0/s 0/s
TCP Intercept Established Conns 0/s 0/s
TCP Intercept Attempts 0/s 0/s
TCP Embryonic Conns Timeout 1092/s 4/s
HTTP Fixup 0/s 0/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
VALID CONNS RATE in TCP INTERCEPT: Current Average
N/A 95.00%
Case Study: Intermittent Access to Web Server
87
88. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Issue show conn to see ‗who‘ is creating the connections
ASA-5510# show conn
54764 in use, 54764 most used
TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB
TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB
TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB
TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB
TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB
TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB
TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB
TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB
Random Sources Embryonic Conns
Case Study: Intermittent Access to Web Server
88
89. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Intermittent Access to Web Server
Connection Count
Jumps
Traffic
Permitted
SYN Flood
Detected
89
90. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www
!
class-map protect
description Protect web server from attacks
match access-list 140
!
policy-map interface_policy
class protect
set connection embryonic-conn-max 100
!
service-policy interface_policy interface outside
Case Study: Intermittent Access to Web Server
90
91. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
TCP Intercept Applied
Few Clients Represent 50+
% of Traffic
Case Study: Intermittent Access to Web Server
91
92. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Apply per-client-max option to limit the number of
connections any single client can establish
Case Study: Intermittent Access to Web Server
access-list 140 extended permit tcp any host 192.168.1.50 eq www
!
class-map protect
description Protect web server from attacks
match access-list 140
!
policy-map interface_policy
class protect
set connection embryonic-conn-max 100 per-client-max 25
!
service-policy interface_policy interface outside
92
93. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Intermittent Access to Web Server
TCP Intercept
per-client-max
93
94. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Attacks Still
Occurring
Attacks Being
Mitigated
Case Study: Intermittent Access to Web Server
94
95. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
95
96. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Online Resources
Support Communities - Supportforums.cisco.com
TAC Security Show Podcast
Online learning modules (VoD Training)
Security RSS Feeds
96
97. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Supportforums.cisco.com
Public wiki – anyone can author articles
Combines supportwiki and Netpro forums
Sections for: ASA, FWSM and PIX
Hundreds of Sample Configs
Troubleshooting Docs
FAQs
http://supportforums.cisco.com/
97
98. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Great way to obtain valuable
troubleshooting insights.
Conversational shows, which
focus on providing in-depth
information on a given feature.
New episodes posted Monthly
IPSec VPNs
Cisco Identity Services
Engine
ASA Services
Module
Interesting
TAC Cases
Useful ASA and
IPS Commands
Using Certificates on
ASA and IOS
TCP Connections
through ASA and FWSM
HTTP filtering
on the ASA
ASA AnyConnect VPN
Cisco Identity Services
Engine Multiple Context Mode
Monitoring Firewall
Performance
Using Packet Captures
on ASA and FWSM
98
99. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Online Learning Modules – VoD
Training
Great way to learn about new features in the ASA
From www.cisco.com select: Products and Services
Security
Network Security (expand)
Cisco ASA 5500 Series
Training resources
Online learning modules
Search cisco.com for ASA Online Learning Modules
Direct link
‒ http://www.cisco.com/en/US/partner/products/ps6120/tsd_
products_support_online_learning_modules_list.html
99
100. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Security Hot Issues – RSS Feeds
Subscribe with an RSS reader
Receive weekly updates on the
Hot Issues customers are facing
Separate feeds for: ASA, FWSM,
ASDM
https://supportforums.cisco.com/docs/DOC-5727
100
101. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Agenda
Packet Flow
Understanding the Architecture
Failover
Troubleshooting
Case Studies
Online Resources
Best Practices
101
102. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Cisco ASA/FWSM Best Practices
Enable ip verify reverse-path on all interfaces
Set embryonic and maximum connection counts for traffic
destined to your servers and destined to the internet
Configure logging to syslog server
Move messages you want to see to lower levels, instead
of raising logging levels and capturing messages you don‘t
want to see
Enable authentication for management access (console/
SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL
as the fallback
102
103. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Cisco ASA/FWSM Best Practices
Baseline CPU load, connection counts, xlate counts, and
traffic (per interface)
Monitor stats using MRTG or other snmp graphing tools
Keep config archives and show tech outputs (use smart
call home)
Run the latest maintenance release in your train
Upgrade major feature trains only when you need new
features, or after train has matured
103
104. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Complete Your Online
Session Evaluation
Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
Receive 20 Passport points for each
session evaluation you complete.
Complete your session evaluation
online now (open a browser through
our wireless network to access our
portal) or visit one of the Internet
stations throughout the Convention
Center.
Don‘t forget to activate your
Cisco Live Virtual account for access to
all session material, communities, and
on-demand and live activities throughout
the year. Activate your account at the
Cisco booth in the World of Solutions or visit
www.ciscolive.com.
104
105. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Final Thoughts
Get hands-on experience with the Walk-in Labs located in World of
Solutions, booth 1042
Come see demos of many key solutions and products in the main Cisco
booth 2924
Visit www.ciscoLive365.com after the event for updated PDFs, on-
demand session videos, networking, and more!
Follow Cisco Live! using social media:
‒ Facebook: https://www.facebook.com/ciscoliveus
‒ Twitter: https://twitter.com/#!/CiscoLive
‒ LinkedIn Group: http://linkd.in/CiscoLI
105
106. © 2012 Cisco and/or its affiliates. All rights reserved.
Presentation_ID Cisco Public
108. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Appendix
Lucky You
This appendix contains extra information which you may
find useful, but I just didn‘t have enough time to cover in
the lecture – or which was covered in previous years.
Enjoy… :-)
108
109. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Appendix
ASA 8.3 Memory Requirements
SNMP OIDs to Monitor
Example: Show Output Filters
Code Base History
Case studies
Poor Voice Quality
Out-of-order packet buffering
TCP MSS issue
Out of memory
High CPU
Capture Example
FWSM Additional Architecture Slides
Failover Extras
Packet Capture Example
Online Tools
ASDM
Information to include when opening a TAC case
109
110. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Redirecting Debugs to Syslog
Problem
‒ Log only debug output to syslog
Solution
‒ Create a logging list with only syslog ID 711001
‒
Enable debug output to syslogs
‒
‒ Log on the logging list
ASA(config)# logging trap Networkers .
ASA(config)# logging list Networkers message 711001 .
ASA(config)# logging debug-trace
INFO: 'logging debug-trace' is enabled. All debug messages are
currently being redirected to syslog:711001 and will not
appear in any monitor session
110
111. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
ASA 8.3 Memory Requirements
ASA Models 5505 – 5540 Require Memory
Upgrades before upgrading to ASA version 8.3
New ASAs ship with the upgraded RAM installed
* For the 5505, only the Security Plus or Unlimited licenses require the memory upgrade
ASA Model Original
Default RAM
Required RAM
for version 8.3
Upgrade Kit Part
Number
5505 * 256 MB 512 MB ASA5505-MEM-512=
5510 256 MB 1024 MB ASA5510-MEM-1GB=
5520 512 MB 2048 MB ASA5520-MEM-2GB=
5540 1024 MB 2048 MB ASA5540-MEM-2GB=
111
112. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
ASA Software Trains
7.2
8.0
8.1
8.2
8.3
8.3.1
8.0.5
8.0.4
8.0.3
8.0.2
8.2.2
8.2.1
8.1.2
8.1.1
7.2.1 7.2.2 7.2.3 7.2.4
EOL
7.2.5
8.3.2
8.2.3
Bug Fixes
Waterfall Down
8.4
8.4.1 8.4.2
8.2.4 8.2.5
8.5
8.5.1
ASA-5580 only
ASA-SM only
8.6
8.6.1
EOL
8.4.4
8.4.3
112
113. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
High Availability – Zero Downtime Upgrades
Copy new image over and reboot
Wait for failover to finish syncing,
and to ―normalize‖ – approx 2 min
Verify config; conns replicated
Issue ―failover active‖
Copy new image over and reboot
Wait for failover to finish syncing,
and to ―normalize‖ – approx 2 min
Verify config; conns replicated
Upgrade Complete
Issue ―failover active‖
Secondary
Primary
Stb
Act
Stb
Act
State
State
Act Stb
Start
113
114. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
SNMP OIDs
CPU usage
• 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 (5 sec)
• 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 (1 min)
• 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 (5 min)
Connections
• 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6 (Current total)
• 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.7 (Max total)
Traffic
• 1.3.6.1.2.1.2.2.1.{10|16}.n (in/out octets)
• Use SNMPwalk to verify the interfaces!
For your
reference
114
115. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Example: Show Output Filters
Examples
Display the interface stats starting with the ‗inside‘ interface
‒show interface | begin inside
Display the access-list entries that contain address 10.1.1.5
‒show access-list | grep 10.1.1.5
Display the config, except for the access-lists
‒show run | exclude access-list
Display only access-list entries that have non-zero hitcounts
‒show access-list | grep –v hitcnt=0
Display a count of the number of connections each host has
‒show local-host | include host|count/limit
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Note: You must Include a Space on Either Side of the Pipe for the Command to Be Accepted;
Also, Trailing Spaces Are Counted
115
116. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Feature Releases
Cisco PIX/ASA/FWSM Code Base
History
6.0(1) 6.1(1) 6.2(1) 6.3(1)
PIX
1.1(3)
FWSM
1.1(1)
Feature Releases
1.1(2)
Maintenance
Releases
2.2(1)
Port Features
Bug Fixes
2.3(1)
2.3(2)
Time
7.0(1)
3.1(1)
7.1(1) 7.2(1)
3.2(1)
PIX/ASA
In Sync
4.0(1)
3.1(2) 3.1(6)
GD
3.1(10)
SafeHarbor
3.2(2) 3.2(4)
SafeHarbor
4.1(1)
8.0(2) 8.3(1)
3.2(17)
3.1(17)
4.0(2) 4.0(4) 4.0(11)
SafeHarbor
116
117. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Debug ICMP Trace
Valuable tool used to troubleshoot connectivity issues
Provides interface and translation information to quickly
determine flow
Echo-replies must be explicitly permitted through ACL, or ICMP inspection
must be enabled
http://www.cisco.com
Internet
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80
ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22
ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80
ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
Example debug icmp trace output
117
119. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Problem
Poor outbound voice quality at SOHO sites
Case Study: Poor Voice Quality
100 Mbps 100 Mbps
Cable
Modem
2 Mbps
WAN
ASA-5505
Outbound RTP Stream
119
120. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Poor Voice Quality
Solution: Traffic Shaping
What is traffic shaping, and why is it needed here?
Why won‘t policing work?
Why won‘t priority queuing alone work?
100 Mbps 100 Mbps
Cable
Modem 2 Mbps
WAN
ASA-5505
Shape to
2 Mbps
120
121. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Poor Voice Quality – Configuration
Example (Traffic Shaping)
class-map voice-traffic
match dscp af13 ef
!
policy-map qos_class_policy
class voice-traffic
priority
!
policy-map qos_outside_policy
class class-default
shape average 2000000
service-policy qos_class_policy
!
service-policy qos_outside_policy interface outside
Solution
Prioritize voice traffic and shape all traffic down to 2 Mbps on the
outside interface.
To view statistics on the operation of the shaper, use the
command
show service-policy shape
121
122. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Poor Voice Quality
Things to Keep in Mind:
Shaping can only be applied to the class class-default
Shaping only works in the outbound direction on
an interface
The shaping value is in bits per second, and must be a multiple
of 8000
The shaping policy is applied to all sub-interfaces on
a physical interface
Not supported on the ASA-5580 platform
Not supported in Transparent or Multi-context mode
122
124. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out-of-Order Packets
Inspections require ordered packets
Packets sent to the SSM (AIP and CSC) require ordered
packets
Cisco ASA/PIX will buffer up to three packets by default
Buffering can be increased on ASA by using the queue-
limit option under the tcp-map
124
125. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out-of-Order Packets
Some networks have high numbers of out-of-order
packets; often caused by asymmetric traffic flows
If the out-of-order packet buffer isn‘t large enough,
traffic is dropped and packets must be retransmitted
Outside
Inside
Client Server
Packet 10
10.16.9.2
192.168.1.30
Packet 12
Packet 13
Packet 14
Packet 15
Buffer
Dropped by Firewall
Packet 11
Dropped on Network
Problem
125
126. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out-of-Order Packet Buffering Example
How to detect?
ASA# show asp drop
Frame drop:
...
TCP packet SEQ past window 46331
TCP packet buffer full 90943
...
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0
!
tcp-map OOO-Buffer
queue-limit 6
!
class-map tcp-options
match access-list OOB-nets
!
policy-map global_policy
class tcp-options
set connection advanced-options OOO-Buffer
!
service-policy global_policy global
How to fix?
126
127. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out-of-Order Packet
Buffering Example
How to verify?
ASA# show service-policy
Global policy:
Service-policy: global_policy
Class-map: inspection_default
...
Class-map: tcp-options
Set connection policy:
Set connection advanced-options: OOB-Buffer
Retransmission drops: 0 TCP checksum drops : 0
Exceeded MSS drops : 0 SYN with data drops: 0
Out-of-order packets: 2340 No buffer drops : 0
127
129. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: TCP MSS
MSS is the Maximum Segment Size—or the maximum
amount of data that can be sent in a single packet
The MSS is set in the SYN packets
The device that receives the MSS advertisement cannot
send more data in a single packet to the peer than
specified by the MSS
129
130. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: TCP MSS
Some servers have broken TCP stacks and ignore
the MSS advertised by the Client
The firewall will drop packets that exceed the advertised
MSS
Outside
Inside
Client Server
SYN MSS=1380
SYN+ACK MSS=1400
DATA=1390
10.16.9.2
192.168.1.30
Problem
130
131. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: TCP MSS Example
How to detect?
ASA# show asp drop
Frame drop:
TCP MSS was too large 943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to
inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
access-list MSS-hosts permit tcp any host 10.16.9.2
!
tcp-map mss-map
exceed-mss allow
!
class-map mss
match access-list MSS-hosts
!
policy-map global_policy
class mss
set connection advanced-options mss-map
!
service-policy global_policy global
How to fix?
131
132. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: TCP MSS Example
How to verify?
ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518
ASA# show capture mss-capture
0 packets captured
0 packets shown
How else could you verify?
132
134. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out of Memory
Users are unable to access the Internet
No new connections are working
All old (long lived) connections continue to work
Step 1: Check the Syslogs
%PIX-3-211001: Memory allocation Error
%PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available
Hardware: PIX-515E, 64 MB RAM
pixfirewall# show memory
Free memory: 714696 bytes
Used memory: 66394168 bytes
------------- ----------------
Total memory: 67108864 bytes
Problem
134
135. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out of Memory
Step 3: What Eats Up Memory (RAM) on the Cisco PIX?
Cisco PIX image (run from RAM)
Configuration
IPSec database
Xlates (translations)
Connections
What Can Eat Up
64 MB on a Cisco
PIX-515E?
Step 4: Let’s Check the Translations
pixfirewall# show xlate
251 in use, 258 most used
PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716)
PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276)
Global 209.165.201.25 Local 10.1.1.102
PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783)
PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197)
PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
Varied Source IPs
A Small Global Pool Is
Used, Overloading to a
PAT Address
135
136. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall# show conn
147456 in use, 147456 most used
TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags
OIU
TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU
TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA
TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags
OIU
TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags
OIU
TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA
TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags
OIU
TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU
TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags
OIU
TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags
OIU
TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags
OIU
TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU
TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags
OIU
TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA
. . .
Case Study: Out of Memory
Step 5: Check the Connections
136
137. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall# show traffic
outside:
received (in 25.000 secs):
1475 packets 469050 bytes
59 pkts/sec 18762 bytes/sec
transmitted (in 25.000 secs):
167619 packets 9654480 bytes
6704 pkts/sec 386179 bytes/sec
inside:
received (in 25.000 secs):
180224 packets 10410480 bytes
7208 pkts/sec 416419 bytes/sec
transmitted (in 25.000 secs):
1050 packets 118650 bytes
42 pkts/sec 4746 bytes/sec
Case Study: Out of Memory
Vast majority of traffic is coming in the inside interface and going
out the outside interface
Outside
Inside
Traffic
Flow
Take a Look at the Traffic Load
137
138. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall# show conn count
147456 in use, 147456 most used
pixfirewall# show xlate count
251 in use, 258 most used
Case Study: Out of Memory
Step 6: Review What We Know and Take Action
Conn Count Is Very High, but xlate Count Is Low
Many connections per xlate
Probably one, or a few hosts, are generating the vast majority
of connections
Most likely due to a virus on the host(s)
138
139. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall# show local-host | include host|count/limit
local host: <10.1.1.131>,
TCP connection count/limit = 0/unlimited
UDP connection count/limit = 0/unlimited
local host: <10.1.1.51>,
TCP connection count/limit = 2/unlimited
UDP connection count/limit = 0/unlimited
local host: <10.1.1.236>,
TCP connection count/limit = 0/unlimited
UDP connection count/limit = 0/unlimited
. . .
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
UDP connection count/limit = 0/unlimited
Case Study: Out of Memory
Step 7: Find the Host(s) Generating All the Connections
Only Show Lines That Have
the Word host or
count/limit in Them
Host 10.1.1.99 is eating up all the connections, and they are
TCP-based connections
139
140. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall# show local-host 10.1.1.99
Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
TCP embryonic count = 146606
UDP connection count/limit = 0/unlimited
Xlate(s):
Global 209.165.201.21 Local 10.1.1.99
Conn(s):
TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA
TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA
TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saA
TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saA
TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saA
TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA
TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA
. . .
Case Study: Out of Memory
Step 8: Now that We Found the Host, Let‘s Look at the
Connections It Is Generating
Note: All Connections Are
Embryonic
Connections to Random
Destinations on TCP/135–
MS Blaster
140
141. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out of Memory
Cisco PIX provides two methods to limit the number
of connections per host
‒ TCP intercept
‒ Max connections
TCP intercept won‘t help because the source address is
valid
Limiting the maximum number of connections each
internal host can have is the only option
Question: Which One Can Be Used Here?
141
142. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out of Memory
Step 9: Limit Infected Host(s) Impact on Network
Configure the MAX TCP connections for NATed hosts to be 50
Note: the local-host must be cleared before the new connection
limits are applied
pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
pixfirewall(config)# clear local-host 10.1.1.99
pixfirewall(config)# show local-host 10.1.1.99
Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 50/50
TCP embryonic count = 50
TCP intercept watermark = unlimited
UDP connection count/limit = 0/unlimited
. . .
The Infected Host
Is Limited to 50
TCP Connections
142
143. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Out of Memory
Things look much better now
Question: How could we configure the Cisco PIX so the
connection limit was only applied to the one host (10.1.1.99)
which was infected with the virus?
pixfirewall# show conn count
126 in use, 147456 most used
pixfirewall# show memory
Free memory: 47716152 bytes
Used memory: 19392712 bytes
------------- ----------------
Total memory: 67108864 bytes
nat (inside) 1 10.1.1.99 255.255.255.255 50 0
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Take One Last Look at the Memory and Connection Counts After
Applying the TCP Connection Limit
143
144. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Classifier in Multimode
When the firewall receives a packet, it must classify
it to determine where to send the packet (which context)
Packets are classified based on the following
‒ Unique ingress interface/VLAN
‒ Packet‘s destination IP matches a global IP
FWSM has a single MAC address for all interfaces
ASA has single MAC for shared interfaces (physical interfaces have
unique MACs)
‒ ASA Ver 7.2 introduces mac-address auto option to change this
144
145. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Classifier in Multimode on FWSM
Inbound traffic is classified to context CTX3, based on the global IP in the
NAT translation
VLAN
3—10.14.3.x
Inside
10.1.2.2
Inside
10.1.1.2
Inside
10.1.3.2
Inbound Packet
Outside
VLAN 4
VLAN 5
VLAN 6
FWSM
CTX1
CTX2
CTX3
MSFC
.1
.2
.3
DST IP SRC IP
10.14.3.89 192.168.5.4
static (inside, outside) 10.14.3.89 10.1.3.2
Shared interface
Example
145
146. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Multi-Context - Common Issues on FWSM
Overlapping statics (globals) across contexts
Missing statics (globals), and unable to classify packets (shared inside
interface) – check Admin context log
Forgetting to ‗monitor-interface‘ for Failover
Forgetting to assign unique IP for each Transparent mode context
Transparent mode, multi-BVI, one routing table
%FWSM-6-106025: Failed to determine security context for packet:
vlan3 tcp src 192.168.5.4/1025 dest 198.51.100.50/80
146
147. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—ACL Rule Limits
ACL rules are about the only hardware limit users encounter
In multimode, ACL resources are divided in 13 equal partitions (12 active,
one backup)
If you have less than 12 contexts, wasted reserved space
Multi-Context
Single Context
Tree 0 : Active
100,567 ACEs
Backup Tree:
100,567
(mirror of active tree)
Tree 0 : active = 14,801 ACEs
Tree 1 : active = 14,801 ACEs
Tree 2 : active = 14,801 ACEs
Tree 3 : active = 14,801 ACEs
Tree 4 : active = 14,801 ACEs
Tree 5 : active = 14,801 ACEs
Tree 6 : active = 14,801 ACEs
Tree 7 : active = 14,801 ACEs
Tree 8 : active = 14,801 ACEs
Tree 9 : active = 14,801 ACEs
Tree 10 : active = 14,801 ACEs
Tree 11 : active = 14,801 ACEs
Tree 12 : backup
177612
combined total
ACEs
147
148. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM(config)# context Accounting
FWSM(config-context)# allocate-acl-partition 0
FWSM(config-context)# show np 3 acl tree
--------------------------------------------
ACL Tree Instance <-> Context Name (ID) Map
--------------------------------------------
Tree Instance 0 Context (001) admin
Tree Instance 1 Context (002) core
Tree Instance 2 Context (003) Engineering
Tree Instance 0 Context (004) Accounting
--------------------------------------------
FWSM and ACLs (Multimode)
Use the command resource acl-partition <num-of-
partitions> to reduce the number of active partitions created;
default is 12
Use the command allocate-acl-partition <num> to assign a
context to a specific ACL tree
Both Use Tree 0
148
150. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
High CPU Usage on the Cisco PIX
A quick overview of the show processes command
pixfirewall(config)# show processes
PC SP STATE Runtime SBASE Stack Process
Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer
Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC
Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace
Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger
Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll
Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean
The Name of
the Process
Number of msec
This Process Has Been on
the CPU
Problem: Cisco PIX CPU Running Very High
For more Information on the Output of the show processes Command, See
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
150
151. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
High CPU Usage on the Cisco PIX
Step 1: Determine What Process Is Eating the CPU
Take the difference in output of two show processes
over a period of time
The following output was a diff of the processes taken one
minute apart
Process_Name Runtime (msec)
Logger 25940
pix/intf3 18410
557poll 9250
i82543_timer 4180
i82542_timer 2230
In One Minute, These
Processes Account for 44
Seconds of CPU Time ~ 73%
The Interface Polling
Processes Always Run, and
Are not Counted in the CPU
Usage
151
152. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall(config)# show log
Syslog logging: enabled
Standby logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: level alerts, 0 messages logged
Trap logging: level warnings, 5919412 messages logged
Logging to lab 172.18.173.123
History logging: disabled
. . .
pixfirewall(config)# show log
Syslog logging: enabled
Buffer logging: level alerts, 0 messages logged
Trap logging: level warnings, 6172472 messages logged
Logging to lab 172.18.173.123
High CPU Usage on the Cisco PIX
Step 2: Focus on the Processes with High CPU Time
Logging is taking up much of the CPU; let‘s review what we have
configured to log
This Is Cumulative Since
the Cisco PIX Was Last
Rebooted
Notice the Change Over a
Few Minutes
152
153. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall(config)# show log
Buffer logging: level warnings, 31527 messages logged
Trap logging: level warnings, 6453127 messages logged
Logging to lab 172.18.173.123
. . .
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
High CPU Usage on the Cisco PIX
Enable buffered logging to same level as syslog server,
and examine the buffered messages
Syslog Server Is Controlled by a Different Group
153
154. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
pixfirewall(config)# show run | grep audit
ip audit name IDS info action alarm
ip audit interface lab IDS
High CPU Usage on the Cisco PIX
Syslog service was down on the syslog server
ICMP unreachable was generated by syslog server for each syslog
message the Cisco PIX sent it
Cisco PIX‘s IDS configuration also logged every ICMP unreachable
message, creating the exponentially increasing problem
Syslog
Server
Outside
Syslog Message
ICMP Unreachable
IDS Syslog Message
Lab
Examine IDS Configuration
154
155. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
ip audit signature 2001 disable
or
no logging message 400011
High CPU Usage on the Cisco PIX
Bring back up syslog service on server
Take server offline
Configure Cisco PIX to not log IDS ICMP unreachable
messages
pixfirewall# show run | grep signature
ip audit signature 2001 disable
pixfirewall# show cpu usage
CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes:
99%
155
156. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
High CPU Usage on the Cisco PIX
Examine the DIFF of two show processes taken over a
one minute interval
Find the process taking up the highest amount of CPU
(excluding the polling processes)
Take actions to lower that process‘s CPU time
Reexamine the CPU output, and repeat as necessary
Summary
156
157. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Show process cpu-hog
The show processes cpu-hog command displays
a list of processes, and the function stack (Traceback) which executed, and lead to a
process running on the CPU longer than the minimum platform threshold
ASA# show processes cpu-hog
Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10
LASTHOG At: 14:18:47 EDT May 29 2009
PC: 8b9ac8c (suspend)
Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971
8ba02b4 8062413
CPU hog threshold (msec): 10.240
Last cleared: None
A corresponding syslog message is also generated
Note: The Traceback syslog below does not signify a crash
May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec,
Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA77ED
0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413
157
158. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM
Additional architecture information
158
159. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM Syslog Level vs. Number of Messages
Log
Level
Description
Number of Messages (SUM)
Ver. 2.3 Ver. 3.1 Ver. 3.2 Ver. 4.0 Ver. 4.1
0 Emergencies 0 0 0 0 0
1 Alerts 58 (58) 67 (67) 67 (67) 67 (67) 67 (67)
2 Critical 21 (79) 29 (96) 29 (96) 29 (96) 29 (96)
3 Errors 94 (173) 305 (401) 306 (402) 318 (414) 318 (414)
4 Warnings 131 (304) 194 (595) 196 (598) 199 (613) 199 (613)
5 Notifications 26 (330) 167 (762) 169 (767) 178 (791) 178 (791)
6 Informational 116 (446) 245 (1007) 248 (1015) 255 (1046) 259 (1050)
7 Debugging 23 (469) 225 (1232) 225 (1240) 226 (1272) 231 (1281)
159
160. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM and ACLs
ACLs on the FWSM are compiled on the control point and
pushed down into hardware (NP 3)
During compile time, CPU should stay at ~ 99%
‒ ACL compile uses all free CPU cycles
‒ Allows compile to complete in shortest time possible
Once compile is complete, rules are attempted to be pushed
into hardware
‒ Successful download
Access Rules Download Complete: Memory Utilization:
49%
‒ Failed download (exceeded HW memory)
ERROR: Unable to add, access-list config limit reached
160
161. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM and ACLs (Multimode)
Use show np 3 acl stats to see the current ACL
resource utilization in that context
FWSM/admin(config)# show np 3 acl stats
----------------------------
ACL Tree Statistics
----------------------------
Rule count : 9584
Bit nodes (PSCB's): 8760
Leaf nodes : 8761
Total nodes : 17521 (max 24260)
Leaf chains : 6912
Total stored rules: 15673
Max rules in leaf : 3
Node depth : 32
----------------------------
Total Number
of ACEs
This Is the
Hardware Limit
Note: One ACE Does not Equal One Node
161
162. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM and ACLs (Multimode)
Use show np 3 acl tree to see which ACL tree a
context is mapped to
FWSM# show np 3 acl tree
--------------------------------------------
ACL Tree Instance <-> Context Name (ID) Map
--------------------------------------------
Tree Instance 0 Context (001) admin
Tree Instance 1 Context (002) core
Tree Instance 2 Context (003)
Engineering
Tree Instance 3 Context (004) Accounting
--------------------------------------------
ACL Tree Number
Context Name
162
163. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—ACL Rule Limits
FWSM 2.3 introduced
‒ resource acl-partition—set the number of ACL partitions
allocate-acl-partition—assigns a context to a specific partition
FWSM 3.2 introduced
‒ resource-rule—allows further customization of a partition
FWSM 4.0 introduced
‒ resource partition—customize the size of individual partitions
access-list optimization enable—merges and/or deletes
redundant and conflicting ACEs without affecting the policy
163
164. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM(config)# context Accounting
FWSM(config-context)# allocate-acl-partition 0
FWSM(config-context)# show np 3 acl tree
--------------------------------------------
ACL Tree Instance <-> Context Name (ID) Map
--------------------------------------------
Tree Instance 0 Context (001) admin
Tree Instance 1 Context (002) core
Tree Instance 2 Context (003) Engineering
Tree Instance 0 Context (004) Accounting
--------------------------------------------
FWSM and ACLs (Multimode)
Use the command resource acl-partition <num-of-
partitions> to reduce the number of active partitions created;
default is 12
Use the command allocate-acl-partition <num> to assign a
context to a specific ACL tree
Both Use Tree
0
164
165. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—Resource Rule
FWSM 3.2 introduced
‒resource-rule—allows further customization of a partition
show resource-rule—displays information about the current rule
allocation
resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283
FWSM# show resource rule
Default Configured Absolute
CLS Rule Limit Limit Max
-----------+---------+----------+---------
Policy NAT 1843 1843 10000
ACL 74188 74188 74188
Filter 2764 2764 5528
Fixup 4147 4147 10000
Est Ctl 460 460 460
Est Data 460 460 460
AAA 6451 6451 10000
Console 1843 1843 3686
-----------+---------+----------+---------
Total 92156 92156
Partition Limit - Configured Limit = Available to allocate
92156 - 92156 = 0
165
166. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—Resource Partition
FWSM 4.0 introduced
‒ resource partition—allows customization of the
size of individual partitions (multi-context mode)
FWSM(config)# resource partition 10
FWSM(config-partition)# size 1000
WARNING: The rule max has been reset based on partition size 1000.
The <size> command leads to re-partitioning of ACL Memory.
It will not take effect until you save the configuration and reboot.
FWSM# show resource rule partition 10
Default Configured Absolute
CLS Rule Limit Limit Max
-----------+---------+----------+---------
Policy NAT 384 384 833
ACL 14801 14801 14801
Filter 576 576 1152
Fixup 1537 1537 3074
Est Ctl 96 96 96
Est Data 96 96 96
AAA 1345 1345 2690
Console 384 384 768
-----------+---------+----------+---------
Total 19219 19219
Partition Limit - Configured Limit = Available to
allocate
19219 - 19219 = 0
FWSM# show resource rule partition 10
Default Configured Absolute
CLS Rule Limit Limit Max
-----------+---------+----------+---------
Policy NAT 20 20 43
ACL 770 770 770
Filter 30 30 60
Fixup 80 80 160
Est Ctl 5 5 5
Est Data 5 5 5
AAA 70 70 140
Console 20 20 40
-----------+---------+----------+---------
Total 1000 1000
Partition Limit - Configured Limit = Available to allocate
1000 - 1000 = 0
Before After
166
167. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM and Control Point
The traffic that makes it to the
control point is traffic that requires
Layer 7 fixup (embedded NAT,
or cmd inspection)
‒FTP
‒VoIP (SIP/SKINNY/H.323/RTSP)
‒DNS
‒XDMCP, etc.
Traffic sourced from, or destined
to, the FWSM also goes through
the control point
‒Syslogs
‒AAA (RADIUS/TACACS+)
‒URL filtering (WebSense/N2H2)
‒Management traffic
(telnet/SSH/HTTPS/SNMP)
‒Failover communications
‒Routing protocols (OSPF/ RIP)
‒etc.
Control Point (CP) Central CPU
C6K Backplane Interface
Session Manager
NP 3
Fast Path
NP 1
Fast Path
NP 2
FWSM
167
168. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM and Network Processors
The session manager—NP 3
‒Processes first packet in a flow
‒ACL checks
‒Translation creation
‒Embryonic/established connection counts
‒TCP/UDP checksums
‒Sequence number randomization
‒TCP intercept
‒etc.
The fast path—NP 1 and 2
‒Performs per packet session lookup
‒Maintains connection table
‒Performs NAT/PAT
‒TCP checks
‒Fragmentation reassembly
‒etc.
Control Point (CP) Central CPU
C6K Backplane Interface
Fast Path
NP 1
Fast Path
NP 2
FWSM
Session Manager
NP 3
Control Point (CP) Central CPU
C6K Backplane Interface
Fast Path
NP 1
Fast Path
NP 2
FWSM
Session Manager
NP 3
168
169. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
FWSM—Enabling the Completion Unit
Due to the FWSM‘s NP architecture, there exists a
possibility that packets arriving with a low inter-packet gap
might be re-ordered by the firewall
This issue might be encountered when performing TCP throughput
testing, or passing high speed TCP flows through the FWSM
Examples: CIFS, FTP, AFP, backups
FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt
np completion-unit to ensure the firewall maintains the packet
order (by enabling a hardware knob on the NPs called the completion
unit)
In multiple mode enter this command in the admin context
configuration; It will then be enabled for all contexts on the firewall
4 3 2 1 4 2 3 1
169
171. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Problem – Find Services which are permitted through the
firewall, yet the servers no longer exist
Get a fast Linux/Solaris machine with a decent amount of memory
Learn to use the following commands:
• cat
• grep, egrep, fgrep
• cut
• awk (basic)
• sort
• uniq
• Perl (advanced manipulation)
Pipe the commands to construct the necessary outputs!
Case Study: Advanced Syslog Analysis
171
172. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Interesting syslogs appear as follows:
May 24 2010 23:19:53: %ASA-6-302014: Teardown TCP
connection 1019934 for outside:203.0.113.126/6243 to
inside:10.100.19.190/21 duration 0:00:30 bytes 0 SYN
Timeout
Syslog ID
Reason
Destination
Case Study: Advanced Syslog
Analysis
172
173. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Case Study: Advanced Syslog Analysis
grep – used to find the syslogs we want
awk – used to print the destination column (IP/port)
uniq – used to print only unique entries, with a count
sort – used to display ordered list, highest count first
syslogserver-sun% grep 302014 syslog.txt | grep "SYN Timeout" | awk '{print $13}' |
uniq -c | sort -r -n
673 inside:10.100.19.190/21
451 dmz:192.168.5.13/80
392 dmz:192.168.5.11/443
358 inside:10.0.0.67/1521
119 inside:10.0.1.142/80
Results:
173
174. © 2012 Cisco and/or its affiliates. All rights reserved.
BRKSEC-3020 Cisco Public
Failover
What to Do After a Failover
Additional Failover Commands
174