Multilayer Campus Architectures and Design Principles

Multilayer Campus Architectures and Design Principles
– BRKCRS-2031
Mark Montañez
Principal Engineer, CCIE #8798
Enteprise Networking Group
Cisco Plus Canada 2012

Title of Slide Would Go Here

• Bullet Level 1
– Bullet Leve 2
• Bullet Level 3
– Bullet Level 4

#CiscoPlusCA

Enterprise-Class Availability
Resilient Campus Communication Fabric
Campus Systems Approach to High Availability
Ultimate Goal……………..100%
• Network-level redundancy
Next-Generation Apps
• System-level resiliency Video Conf., Unified Messaging,
Global Outsourcing,

• Enhanced management E-Business, Wireless Ubiquity

• Human ear notices the difference in Mission Critical Apps.
voice within 150–200 msec Databases, Order-Entry,
CRM, ERP
10 consecutive G711 packet loss
• Video loss is even more noticeable
Desktop Apps
• 200-msec end-to-end campus E-mail, File and Print

convergence
APPLICATIONS DRIVE REQUIREMENTS FOR
HIGH AVAILABILITY NETWORKING

Next-Generation Campus Design
Unified Communications Evolution
• VoIP is now a mainstream technology
• Ongoing evolution to the full spectrum of Unified Communications
• High-definition executive communication application requires stringent Service-Level
Agreement (SLA)
– Reliable service—high availability infrastructure
– Application service management—QoS

Agenda Data Center Services
Block
• Multilayer Campus
Design Principles
• Foundation Services
• Campus Design Best
Practices
• IP Telephony Si Si

Considerations
• QoS Considerations Si Si Si Si

• Security
Considerations Si Si Si Si

• Putting It All Together Distribution Blocks

• Summary

High-Availability Campus Design
Structure, Modularity, and Hierarchy
Access

Si Si Si Si Si Si

Distribution

Core Si Si

Distribution Si Si
Si Si
Si
Si

Access WAN Data Center Internet

Hierarchical Campus Network
Structure, Modularity and Hierarchy

Not This!!
Si

Si

Si

Si Si Si

Si
Si Si

Si Si Si

Server Farm

WAN Internet PSTN

Hierarchical Network Design
Without a Rock Solid Foundation the Rest Doesn’t Matter

 Offers hierarchy—each layer has specific
Access
role
 Modular topology—building blocks
 Easy to grow, understand, and Si Si

Distribution troubleshoot
 Creates small fault domains— clear
demarcations and isolation
Core
 Promotes load balancing and redundancy
Si
Si

 Promotes deterministic traffic patterns
Distribution  Incorporates balance of both Layer 2 and Si Si

Layer 3 technology, leveraging the
strength of both
Access  Utilizes Layer 3 routing for load balancing, Building Block
fast convergence, scalability, and control

Access Layer
Feature Rich Environment
• It’s not just about connectivity
• Layer 2/Layer 3 feature rich environment; convergence,
HA, security, QoS, IP multicast, etc. Core
• Intelligent network services: QoS, Si Si

trust boundary, broadcast suppression, IGMP snooping
• Intelligent network services: PVST+,
Rapid PVST+, EIGRP, OSPF, DTP, PAgP/LACP, UDLD,
FlexLink, etc. Distribution
Si Si
• Cisco Catalyst® integrated security features IBNS
(802.1x), (CISF): port security, DHCP snooping, DAI,
IPSG, etc.
• Automatic phone discovery, conditional trust boundary,
Access
power over Ethernet, auxiliary VLAN, etc.
• Spanning tree toolkit: PortFast, UplinkFast,
BackboneFast, LoopGuard, BPDU Guard, BPDU See BRK-CRS 3037—Integrating Intelligent Access
Filter, RootGuard, etc.

Distribution Layer
Policy, Convergence, QoS, and High Availability
• Availability, load balancing,
QoS and provisioning are the important Core
considerations at this layer Si Si

• Aggregates wiring closets
(access layer) and uplinks to core
Distribution
• Protects core from high density peering Si Si

and problems in access layer
• Route summarization, fast convergence,
redundant path load sharing Access
• HSRP or GLBP to provide first hop
redundancy

Core Layer
Scalability, High Availability, and Fast Convergence
• Backbone for the network—
connects network building blocks Core
• Performance and stability vs. Si Si

complexity— less is more in the
core
• Aggregation point for distribution Si Si
Distribution

layer
• Separate core layer helps in
scalability during future growth Access
• Keep the design technology-
independent

Do I Need a Core Layer?
It's Really a Question of Scale, Complexity, and Convergence
• No Core
• Fully-meshed distribution layers
• Physical cabling
requirement
• Routing complexity
Second Building
Block–4 New
Links

4th Building 3rd Building Block
Block 8 New Links
12 New Links 12 Links Total
24 Links Total
5 IGP Neighbors
8 IGP Neighbors

Do I Need a Core Layer?
It’s Really a Question of Scale, Complexity, and Convergence
• Dedicated Core Switches
• Easier to add a module
• Fewer links in the core 2nd Building Block
• Easier bandwidth upgrade 8 New Links
• Routing protocol
peering reduced
• Equal cost Layer 3 links
for best convergence

4th Building Block
3rd Building Block
4 New Links
4 New Links
16 Links Total
12 Links Total
3 IGP Neighbors
3 IGP Neighbors

Design Alternatives Come Within a
Building (or Distribution) Block
Layer 2 Access Routed Access Virtual Switching
System

Access

Si Si Si Si

Distribution

Core Si Si

Distribution Si Si
Si Si
Si
Si

Access WAN Data Center Internet

Layer 3 Distribution Interconnection
Layer 2 Access—No VLANs Span Access Layer

• Tune CEF load balancing Si Si
Core
• Match CatOS/IOS EtherChannel settings
and tune load balancing
• Summarize routes towards core Layer 3

• Limit redundant IGP peering Si Si Distribution
• STP Root and HSRP primary tuning or Point-to-
Point
GLBP to load balance on uplinks Link
• Set trunk mode on/no-negotiate
• Disable EtherChannel unless needed
• Set port host on access layer ports:
– Disable trunking VLAN 20 Data VLAN 40 Data Access
Disable EtherChannel 10.1.20.0/24 10.1.40.0/24
VLAN 120 Voice VLAN 140 Voice
Enable PortFast 10.1.120.0/24 10.1.140.0/24
• RootGuard or BPDU-Guard
• Use security features

Layer 2 Access—Some VLANs Span Access Layer
• Tune CEF load balancing
• Match CatOS/IOS EtherChannel settings and tune load Si Si
Core
balancing
• Summarize routes towards core
Layer 2
• Limit redundant IGP peering
• STP Root and HSRP primary or GLBP and STP port Si Si Distribution
Trunk
cost tuning to load balance on uplinks
• RootGuard on downlinks
• LoopGuard on uplinks
VLAN 20 Data VLAN 40 Data Access
• Set port host on access 10.1.20.0/24 10.1.40.0/24
Layer ports: VLAN 120 Voice VLAN 140 Voice
10.1.120.0/24 10.1.140.0/24
– Disable trunking VLAN 250 WLAN
Disable EtherChannel 10.1.250.0/24
Enable PortFast

Routed Access and Virtual Switching System
Evolutions of and Improvements to Existing Designs

Si Si Si Si
Core

Layer 3 VSS & vPC
P-to-P Link New Concept Distribution
Si Si

VLAN 20 Data
10.1.20.0/24
VLAN 40 Data
VLAN 20 Data VLAN 40 Data 10.1.40.0/24
VLAN 120 Voice Access
10.1.20.0/24 10.1.40.0/24 10.1.120.0/24
VLAN 140 Voice
VLAN 120 Voice VLAN 140 Voice 10.1.140.0/24
VLAN 250 WLAN
10.1.120.0/24 10.1.140.0/24 10.1.250.0/24

See BRK-CRS3035—Advanced Enterprise Campus Design: VSS
See BRK-CRS3036—Advanced Enterprise Campus Design: Routed Access

Agenda Data Center Services
Block

• Multilayer Campus Design
Principles
• Campus Design Best Practices
• IP Telephony Considerations
• QoS Considerations
Si Si

• Security Considerations
• Putting It All Together Si Si Si Si

• Summary
Si Si Si Si

Distribution Blocks

Foundation Services
• Layer 1 physical things
• Layer 2 redundancy—
spanning tree
• Layer 3 routing protocols
• Trunking protocols—(ISL/.1q)
• Unidirectional link detection
• Load balancing
HSRP
– EtherChannel link aggregation
– CEF equal cost load balancing
Spanning
• First hop redundancy protocols Routing Tree
– VRRP, HSRP, and GLBP

Best Practices— Layer 1 Physical Things
• Use point-to-point
interconnections—no L2
aggregation points between
nodes Si Si Si Si Si Si

• Use fiber for best convergence
(debounce timer) Layer 3 Equal Layer 3 Equal

• Tune carrier delay timer Cost Links
Si Si
Cost Links

• Use configuration on the
physical interface not VLAN/SVI Si Si Si Si

Si Si

when possible

WAN Data Center Internet

Redundancy and Protocol Interaction
Link Neighbour Failure Detection Hellos
• Indirect link failures are harder to detect Si

• With no direct HW notification of link loss Si

or topology change convergence times
are dependent on SW notification Layer 2 Si

• Indirect failure events in a bridged
environment are detected by spanning
tree hellos BPDUs
• In certain topologies the need for TCN Si

updates or dummy multicast flooding
(uplink fast) is necessary for convergence Si

• You should not be using hubs in a high- Layer 2 Si

availability design

Link Redundancy and Failure Detection
• Direct point-to-point fiber provides for fast failure detection Cisco IOS® Throttling:
Carrier Delay Timer
3
• IEEE 802.3z and 802.3ae link negotiation define the use of
remote fault indicator and link fault signaling mechanisms
• Bit D13 in the Fast Link Pulse (FLP) can be set to indicate a 2 Linecard
Throttling:
physical fault to the remote side Debounce Timer

• Do not disable auto-negotiation on GigE and 10GigE
interfaces
• The default debounce timer on GigE and 10GigE fiber 1
linecards is 10 msec
• The minimum debounce for copper is
300 msec 1

• Carrier-delay
Si Si
– 3560, 3750, and 4500—0 msec Remote IEEE
– 6500—leave it set at default Fault Detection
Mechanism

Layer 2 and 3—Why Use Routed Interfaces
• Configuring L3 routed interfaces provides for faster convergence than
an L2 switch port with an associated L3 SVI

L3 L2
Si Si Si Si

1. Link Down 1. Link Down
2. Interface Down 2. Interface Down
3. Routing Update 3. Autostate
4. SVI Down
~ 8 msec loss ~ 150–200 msec loss 5. Routing Update

21:38:37.042 UTC: %LINEPROTO-5-UPDOWN: Line 21:32:47.813 UTC: %LINEPROTO-5-UPDOWN: Line
protocol on Interface GigabitEthernet3/1, changed protocol on Interface GigabitEthernet2/1, changed state to
state to down down
21:38:37.050 UTC: %LINK-3-UPDOWN: Interface 21:32:47.821 UTC: %LINK-3-UPDOWN: Interface
GigabitEthernet3/1, changed state to down GigabitEthernet2/1, changed state to down
21:38:37.050 UTC: IP-EIGRP(Default-IP-Routing- 21:32:48.069 UTC: %LINK-3-UPDOWN: Interface Vlan301,
Table:100): Callback: route_adjust GigabitEthernet3/1 changed state to down
21:32:48.069 UTC: IP-EIGRP(Default-IP-Routing-
Table:100): Callback: route, adjust Vlan301

Best Practices— Spanning Tree Configuration
Same VLAN
• Only span VLAN across multiple access
Same VLAN Same VLAN

layer switches when you have to! Layer 2 Loops
• Use rapid PVST+ for best convergence
• More common in the data center Si Si Si Si Si Si

• Required to protect against user side
loops Layer 3 Equal Layer 3 Equal

• Required to protect against operational Cost Links
Si Si
Cost Links

accidents (misconfiguration or hardware
failure) Si Si Si Si

• Take advantage of the spanning tree Si Si

toolkit


Multilayer Network Design
Layer 2 Access with Layer 3 Distribution

Si Si Si Si

Vlan 10 Vlan 20 Vlan 30 Vlan 30 Vlan 30 Vlan 30

• Each access switch has • At least some VLANs span
unique VLANs multiple access switches
• No Layer 2 loops • Layer 2 loops
• Layer 3 link between distribution • Layer 2 and 3 running over
• No blocked links link between distribution
• Blocked links

Optimizing L2 Convergence
PVST+, Rapid PVST+ or MST
• Rapid-PVST+ greatly improves the restoration times for any VLAN that requires a
topology convergence due to link UP
• Rapid-PVST+ also greatly improves convergence time over backbone
fast for any indirect link failures
35
• PVST+ (802.1d)

Time to Restore Data Flows (sec)
– Traditional spanning tree 30
implementation Upstream
25
Downstream
• Rapid PVST+ (802.1w)
20
– Scales to large size
(~10,000 logical ports) 15
– Easy to implement, proven, scales 10
• MST (802.1s) 5
– Permits very large scale STP
implementations 0
PVST+ Rapid PVST+
(~30,000 logical ports)
– Not as flexible as rapid PVST+

Layer 2 Hardening
Spanning Tree Should Behave the Way You Expect
• Place the root where you LoopGuard
want it
– Root primary/secondary macro STP Root

• The root bridge should
Si Si
stay where you put it
– RootGuard RootGuard
– LoopGuard LoopGuard
– UplinkFast
– UDLD
• Only end-station traffic
should be seen on
an edge port
BPDU Guard or
– BPDU Guard RootGuard
– RootGuard PortFast
Port Security
– PortFast
– Port-security

Best Practices - Layer 3 Routing Protocols
• Typically deployed in distribution
to core, and core-to-core interconnections
• Used to quickly reroute Si Si Si Si Si Si

around failed node/links while providing load
balancing over redundant paths
• Build triangles not squares for deterministic
Layer 3 Equal Layer 3 Equal
convergence Cost Links Cost Links
Si

• Only peer on links that you intend to use as transit
Si

• Insure redundant L3 paths to avoid black holes
• Summarize distribution to core to limit EIGRP Si Si Si Si

Si Si

query diameter or OSPF LSA propagation
• Tune CEF L3/L4 load balancing hash to achieve
maximum
utilization of equal cost paths (CEF polarization)

Best Practice—Build Triangles not Squares
Deterministic vs. Non-Deterministic
Triangles: Link/Box Failure Does not Squares: Link/Box Failure
Require Routing Protocol Requires Routing Protocol
Convergence Convergence

Si Si
Si Si

Si Si
Si Si

Model A Model B

• Layer 3 redundant equal cost links support fast convergence
• Hardware based—fast recovery to remaining path
• Convergence is extremely fast (dual equal-cost paths: no need for OSPF or
EIGRP to recalculate a new path)

Best Practice - Passive Interfaces for IGP
Limit OSPF and EIGRP Peering Through the Access Layer
• Limit unnecessary peering using
passive interface: Distribution Si Si

Routing
– Four VLANs per wiring closet
Updates
– 12 adjacencies total
– Memory and CPU requirements
increase
with no real benefit
Access
– Creates overhead for IGP
OSPF Example: EIGRP Example:

Router(config)#routerospf 1 Router(config)#routereigrp 1
Router(config-router)#passive- Router(config-router)#passive-
interfaceVlan 99 interfaceVlan 99

Router(config)#routerospf 1 Router(config)#routereigrp 1
Router(config-router)#passive- Router(config-router)#passive-
interface default interface default
Router(config-router)#no passive- Router(config-router)#no passive-
interface Vlan 99 interface Vlan 99

Why You Want to Summarize at the Distribution
Limit EIGRP Queries and OSPF LSA Propagation
• It is important to force
summarization at the No Summaries
distribution towards the core Queries Go Beyond the Core
Rest of Network
• For return path traffic an Core
OSPF or EIGRP re-route
is required
• By limiting the number of peers Si Si

an EIGRP router must query or
the number of LSAs an OSPF
peer must process we can Distribution
optimize this reroute
• EIGRP example:
interface Port-channel1 Si Si
description to Core#1
ip address 10.122.0.34
Access
255.255.255.252
ip hello-interval eigrp 100
1
ip hold-time eigrp 100 3
ip summary-address eigrp 100 10.1.1.0/24 10.1.2.0/24
10.1.0.0 255.255.0.0 5

Why You Want to Summarize at the Distribution
Reduce the Complexity of IGP Convergence
Summaries
• It is important to force summarization at the Stop Queries at the Core
distribution towards the core Rest of Network
Core
• For return path traffic an OSPF or EIGRP re-route is
required
• By limiting the number of peers an EIGRP router Si
Si
must query or the number of LSAs an OSPF |
peer must process we can optimize his reroute
Summary:
• For EIGRP if we summarize at the distribution we 10.1.0.0/16
Distribution
stop queries at the core boxes for an access layer
flap
• For OSPF when we summarize at the distribution Si Si

(area border or L1/L2 border) the flooding of LSAs
is limited to the distribution switches; SPF now deals Access
with one LSA not three

10.1.1.0/24 10.1.2.0/24

Best Practice— Summarize at the Distribution
Gotcha—Distribution-to-Distribution Link Required

• Best practice—summarize at the distribution layer
to limit EIGRP queries or OSPF LSA propagation Core
• Gotcha: Si Si

– Upstream: HSRP on left
distribution takes over when Summary:
link fails 10.1.0.0/16
– Return path: old router still
advertises summary to core Distribution
– Return traffic is dropped on Si Si
right distribution switch
• Summarizing requires a link between the
distribution switches
• Alternative design: use the access layer for transit Access

10.1.1.0/24 10.1.2.0/24

Provide Alternate Paths
• What happens if fails?
• No route to the core anymore? Si Si Core

• Allow the traffic to go through the access? Single Path
to Core
– Do you want to use your access
switches as transit nodes?
– How do you design for scalability if the Si Si
Distribution
access used for transit traffic?
• Install a redundant link to the core
• Best practice: install redundant link to core
Access
and utilize L3 link between distribution layer

A B

Equal-Cost Multipath
Optimizing CEF Load-Sharing
• Depending on the traffic flow patterns and IP
Addressing in use one algorithm may provide Si Si

better load-sharing results than another
• Be careful not to introduce polarization in a 30% of 70% of
multi-tier design by changing the default to the Flows Flows
Si
same thing in all tiers/layers of the network

Catalyst 4500 Load-Sharing Options
Original Src IP + Dst IP Load-Sharing Si Si

Universal* Src IP + Dst IP + Unique ID Simple
Include
Src IP + Dst IP + (Src or Dst Port) + Unique ID
Port

Catalyst 6500 PFC3** Load-Sharing Options Load-Sharing
Default* Src IP + Dst IP + Unique ID Full Simple Si Si

Full Src IP + Dst IP + Src Port + Dst Port
Full Exclude Port Src IP + Dst IP + (Src or Dst Port)
Simple Src IP + Dst IP Load-Sharing
Full Simple Src IP + Dst IP + Src Port + Dst Port
Simple
Si

* = Default Load-Sharing Mode
** = PFC3 in Sup720 and Sup32 Supervisors

CEF Load Balancing
Avoid Underutilizing Redundant Layer 3 Paths
Redundant Paths Ignored

• CEF polarization: without some tuning CEF
will select the same path left/left or
right/right
Distribution
Si Si
• Imbalance/overload could occur
Default L3 Hash
• Redundant paths are ignored/ underutilized
L R • The default CEF hash input is L3
Core
Si Si
• We can change the default
Default L3 Hash
to use L3 + L4 information as input to the
hash derivation
L
Distribution
Default L3 Hash Si
R Si

CEF Load Balancing
Avoid Underutilizing Redundant Layer 3 Paths
All Paths Used

• The default will for Sup720/32 and latest
hardware (unique ID added to default).
However, depending on IP addressing, and
Distribution
Si Si
flows imbalance could occur
L3/L4 Hash
• Alternating L3/L4 hash and L3 hash will
L R L R give us the best load balancing results
Core
Si Si
• Use simple in the core and full simple in
Default L3 Hash
the distribution to add L4 information to the
algorithm at the distribution and maintain
L differentiation tier-to-tier
Distribution
L3/L4 Hash Si
R Si

Best Practices—Trunk Configuration
• Typically deployed on interconnection
between access and distribution layers
• Use VTP transparent mode to decrease 802.1q
potential for operational error
• Hard set trunk mode to on and
Si Si
Trunks
Si Si Si Si

encapsulation negotiate off for optimal
convergence
• Change the native VLAN to something Cost Links Cost Links
unused to avoid VLAN hopping Si Si

• Manually prune all VLANS except those
needed Si Si Si Si

• Disable on host ports:
Si Si

– CatOS: set port host
– Cisco IOS: switchport host

VTP Virtual Trunk Protocol
• Centralized VLAN management Set
VLAN 50 Pass
• VTP server switch propagates VLAN Trunk Through
Update
database to VTP client switches
A Server F
Transparent
• Runs only on trunks
Ok, I
• Four modes: Trunk Trunk Just
Learned
– Server: updates clients
Ok, I VLAN
and servers Just 50!
– Client: receive updates— Learned
cannot make changes VLAN
50!
Client
Trunk
Client B
– Transparent: let updates
Drop
pass through
VTP
– Off: ignores VTP updates Updates

Off C

DTP Dynamic Trunk Protocol
• Automatic formation of trunked switch-to- Si Si

switch interconnection On/On
– On: always be a trunk Trunk
– Desirable: ask if the other side can/will
Si Si
– Auto: if the other sides asks I will Auto/Desirable
– Off: don’t become a trunk
• Negotiation of 802.1Q or ISL
Trunk
encapsulation Si Si

– ISL: try to use ISL trunk encapsulation Off/Off
– 802.1q: try to use 802.1q encapsulation NO Trunk
– Negotiate: negotiate ISL or 802.1q encapsulation
Si
with peer Si

– Non-negotiate: always use encapsulation that is
hard set Off/On, Auto, Desirable
NO Trunk

Optimizing Convergence: Trunk Tuning
Trunk Auto/Desirable Takes Some Time

• DTP negotiation tuning improves link up convergence time
– IOS(config-if)# switchport mode trunk
– IOS(config-if)# switchport nonegotiate

2.5
Time to Converge in Seconds

2

1.5 Si
Two Seconds
1 of Delay/Loss
Tuned Away
0.5
Voice Data
0
Trunking Desirable Trunking Nonegotiate

Trunking/VTP/DTP—Quick Summary
• VTP transparent should be used; there is a trade off between
administrative overhead and the temptation to span existing VLANS
across multiple access layer switches
• Emerging technologies that do VLAN assignment by name (IBNS,
NAC, etc.) require a unique VLAN database per access layer switch
if the rule: A VLAN = A Subnet = AN access layer switch is going to
be followed
• One can consider a configuration that uses DTP ON/ON and NO
NEGOTIATE; there is a trade off between performance/HA impact
and maintenance and operations implications
• An ON/ON and NO NEGOTIATE configuration is faster from a link
up (restoration) perspective than a desirable/desirable alternative.
However, in this configuration DTP is not actively monitoring the
state of the trunk and a misconfigured trunk is not easily identified
• It’s really a balance between fast convergence and your ability to
manage configuration and change control …

Best Practices—UDLD Configuration
• Typically deployed on any fiber
optic interconnection
• Use UDLD aggressive mode for
Si Si Si Si Si Si

most aggressive protection Fiber
Layer 3 Equal

• Turn on in global configuration to Cost Links Interconnection
Layer 3 Equal
Si Si
Cost Links

avoid operational error/misses s
• Config example
Si Si Si Si

Si Si

– Cisco IOS:
udld aggressive WAN Data Center Internet

Unidirectional Link Detection
Protecting Against One-Way Communication

• Highly-available networks require UDLD to protect against
Si
one-way communication or partially failed links and the effect
that they could have on protocols like STP and RSTP
• Primarily used on fiber optic links where patch panel errors
could cause link up/up with mismatched transmit/receive pairs Are You
• Each switch port configured for UDLD will send UDLD protocol ‘Echoing’
My Hellos?
packets (at L2) containing the port’s own device/port ID, and
the neighbor’s device/port IDs seen
by UDLD on that port
• Neighboring ports should see their own device/port ID (echo)
in the packets received from the other side
• If the port does not see its own device/port ID in the incoming Si
UDLD packets for a specific duration of time, the link is
considered unidirectional and is shutdown

UDLD Aggressive and UDLD Normal

Si Si

• Timers are the same—15-second hellos by default
• Aggressive Mode—after aging on a previously bi-directional link—tries eight
times (once per second) to reestablish connection then err-disables port
• UDLD—Normal Mode—only err-disable the end where UDLD detected other
end just sees the link go down
• UDLD—Aggressive—err-disable both ends of the connection due to err-
disable when aging and re-establishment of UDLD communication fails

Best Practices— EtherChannel Configuration
• Typically deployed in distribution to core,
and core to core interconnections
• Used to provide link redundancy—while
reducing peering complexity Si Si Si Si Si Si

• Tune L3/L4 load balancing hash to
achieve maximum utilization of channel
members Cost Links Cost Links
Si Si

• Deploy in powers of two (two, four, or
eight)
• Match CatOS and Cisco IOS PAgP
Si Si Si Si

Si Si

settings
• 802.3ad LACP for interop if you need it
• Disable unless needed WAN Data Center Internet

– Cisco IOS: switchport host

Understanding EtherChannel
Link Negotiation Options—PAgP and LACP
Port Aggregation Protocol Link Aggregation Protocol

Si Si
Si Si
On/On On/On
Channel Channel

Si Si
On/Off Si
On/Off
Si

No Channel No Channel

Si Si
Si
Auto/Desirable Si
Active/Passive
Channel Channel

Si Si
Si Si

Off/On, Auto, Desirable Passive/Passive
No Channel No Channel
On: always be a channel/bundle member On: always be a channel/bundle member
Desirable: ask if the other side can/will Active: ask if the other side can/will
Auto: if the other side asks I will Passive: if the other side asks I will
Off: don’t become a member of a Off: don’t become a member of a
channel/bundle channel/bundle

EtherChannels or Equal Cost Multipath
10/100/1000 How Do You Aggregate It?
10 GE and
Core Si Si
10-GE Channels
Typical 4:1
Data Over-
Subscription
Distribution
Si Si

Typical 20:1
Data Over-
Subscription Access

Reduce Complexity/Peer Relationships

• More links = more routing peer relationships and
associated overhead
• EtherChannels allow you to reduce peers by
Si Si Si Si Si Si
creating single logical interface to peer over
• On single link failure in a bundle
– OSPF running on a Cisco IOS-based switch will reduce
link cost and reroute traffic
Layer 3 Equal Layer 3 Equal – OSPF running on a hybrid switch will not change link
Cost Links Cost Links
Si Si cost and may overload remaining links
– EIGRP may not change link cost and may overload
remaining links
Si Si Si Si

Si Si


Why 10-Gigabit Interfaces
• More links = more routing peer relationships
and associated overhead
• EtherChannels allow you to reduce peers by
Si Si Si Si Si Si
creating single logical interface to peer over
• However, a single link failure is not taken into
consideration by routing protocols. Overload
possible
Cost Links
Si Si
Cost Links
• Single 10-gigabit links address both
problems. Increased bandwidth without
increasing complexity or compromising
Si Si Si Si
routing protocols ability to select best path
Si Si


EtherChannels—Quick Summary
• For Layer 2 EtherChannels: Desirable/Desirable is the recommended configuration
so that PAgP is running across all members of the bundle insuring that an individual
link failure will not result in an STP failure
• For Layer 3 EtherChannels: one can consider a configuration that uses ON/ON.
There is a trade-off between performance/HA impact and maintenance and
operations implications
• An ON/ON configuration is faster from a link-up (restoration) perspective than a
Desirable/Desirable alternative. However, in this configuration PAgP is not actively
monitoring the state of the bundle members and a misconfigured bundle is not easily
identified
• Routing protocols may not have visibility into the state of an individual member of a
bundle. LACP and the minimum links option can be used to bring the entire bundle
down when the capacity is diminished.
– OSPF has visibility to member loss (best practices pending investigation). EIGRP does not…
• When used to increase bandwidth—no individual flow can go faster than the speed
of an individual member of the link
• Best used to eliminate single points of failure (i.e., link or port) dependencies from a
topology

Best Practices—First Hop Redundancy
• Used to provide a resilient default
gateway/ first hop address to end-
stations 1st Hop
• HSRP, VRRP, and GLBP alternatives
Si
Redundanc
Si Si Si Si Si

• VRRP, HSRP, and GLBP provide y
millisecond timers and excellent Layer 3 Equal
Cost Links
Layer 3 Equal
Cost Links
convergence performance Si Si

• VRRP if you need multivendor
interoperability Si Si

Si Si
Si Si

• GLBP facilitates uplink load balancing
• Preempt timers need to be tuned to WAN Data Center Internet
avoid black-holed traffic

First Hop Redundancy with VRRP
IETF Standard RFC 2338 (April 1998) R1—Master, Forwarding Traffic; R2,—Backup
VRRP ACTIVE VRRP BACKUP
IP: 10.0.0.254 IP: 10.0.0.253
MAC: 0000.0c12.3456 MAC: 0000.0C78.9abc
• A group of routers function as one vIP: 10.0.0.10 vIP:
vMAC: 0000.5e00.0101 vMAC:
virtual router by sharing one virtual IP
R2
address and one virtual MAC address R1

• One (master) router performs packet Si Si

forwarding for local hosts Distribution-A Distribution-B
VRRP Active VRRP Backup
• The rest of the routers act as back up Access-a

in case the master router fails
• Backup routers stay idle as far as
packet forwarding from the client side
IP: 10.0.0.1 IP: 10.0.0.2 IP: 10.0.0.3
is concerned MAC: aaaa.aaaa.aa01 MAC: aaaa.aaaa.aa02 MAC: aaaa.aaaa.aa03
GW: 10.0.0.10 GW: 10.0.0.10 GW: 10.0.0.10
ARP: 0000.5e00.0101 ARP: 0000.5e00.0101 ARP: 0000.5e00.0101

First Hop Redundancy with HSRP
R1—Active, Forwarding Traffic;
RFC 2281 (March 1998) R2—Hot Standby, Idle
HSRP ACTIVE HSRP STANDBY
IP: 10.0.0.254 IP: 10.0.0.253
MAC: 0000.0c12.3456 MAC: 0000.0C78.9abc
• A group of routers function as one virtual vIP: 10.0.0.10
vMAC: 0000.0c07.ac00
vIP:
vMAC:
router by sharing one virtual IP address
R1 R2
and one virtual MAC address
• One (active) router performs packet Si Si

forwarding for local hosts Distribution-A Distribution-B
• The rest of the routers provide hot HSRP Active HSRP Backup
Access-a
standbyin case the active router fails
• Standby routers stay idle as far as
packet forwarding from the client side is
concerned
IP: 10.0.0.1 IP: 10.0.0.2 IP: 10.0.0.3
MAC: aaaa.aaaa.aa01 MAC: aaaa.aaaa.aa02 MAC: aaaa.aaaa.aa03
GW: 10.0.0.10 GW: 10.0.0.10 GW: 10.0.0.10
ARP: 0000.0c07.ac00 ARP: 0000.0c07.ac00 ARP: 0000.0c07.ac00

Why You Want HSRP Preemption
• Spanning tree root and HSRP
primary aligned
Si Si Core
• When spanning tree root is re-
introduced, traffic will take a two-
hop path to HSRP active
Spanning
Tree HSRP
Root Active
HSRP
Si
Spanning Distribution
HSRP Preempt
Active Si Tree
Root

• HSRP preemption will allow HSRP
to follow spanning tree topology Access

Without Preempt Delay HSRP Can Go Active Before Box Completely Ready
to Forward Traffic: L1 (Boards), L2 (STP), L3 (IGP Convergence)
standby 1 preempt delay minimum 180

First Hop Redundancy with GLBP
Cisco Designed, Load Sharing, Patent Pending
R1- AVG; R1, R2 Both Forward Traffic
• All the benefits of HSRP plus load GLBP AVG/AVF, SVF GLBP AVF, SVF
balancing of default gateway  utilizes IP: 10.0.0.254 IP: 10.0.0.253
MAC: 0000.0C78.9abc
MAC: 0000.0c12.3456
all available bandwidth vIP: 10.0.0.10 vIP: 10.0.0.10
vMAC: 0007.b400.0101 vMAC: 0007.b400.0102
• A group of routers function as one virtual
R1
router by sharing one virtual IP address
but using multiple virtual MAC addresses Si Si

for traffic forwarding Distribution-A Distribution-B
GLPB AVF,
• Allows traffic from a single common GLBP AVG/
SVF
AVF, SVF Access-a
subnet to go through multiple redundant
gateways using a single virtual IP
address

IP: 10.0.0.1 IP: 10.0.0.2 IP: 10.0.0.3
MAC: aaaa.aaaa.aa01 MAC: aaaa.aaaa.aa02 MAC: aaaa.aaaa.aa03
GW: 10.0.0.10 GW: 10.0.0.10 GW: 10.0.0.10
ARP: 0007.B400.0101 ARP: 0007.B400.0102 ARP: 0007.B400.0101

First Hop Redundancy with Load Balancing
Cisco Gateway Load Balancing Protocol (GLBP)
• Each member of a GLBP redundancy group owns a unique virtual MAC address for a common
IP address/ default gateway
• When end-stations ARP for the common IP address/default gateway they are given a load-
balanced virtual MAC address
• Host A and host B send traffic to different GLBP peers but have the same default gateway

GLBP 1 ip 10.88.1.10 GLBP 1 ip 10.88.1.10
vMAC 0000.0000.0001 vIP vMAC 0000.0000.0002
R1 R2
.1 10.88.1.10 .2
ARP
Reply

10.88.1.0/24
.4 .5
ARPs for 10.88.1.10
Gets MAC 0000.0000.0001
A B ARPs for 10.88.1.10
Gets MAC 0000.0000.0002

Optimizing Convergence: VRRP, HSRP, GLBP
Mean, Max, and Min—Are There Differences?

• VRRP not tested with sub-second timers and all flows go through a
common VRRP peer; mean, max, and min are equal
Si Si

• HSRP has sub-second timers; however all flows go through same HSRP
peer so there is no difference between mean, max, and min
• GLBP has sub-second timers and distributes the load amongst the GLBP
peers; so 50% of the clients are not affected by an uplink failure
Distribution to Access Link Failure
Access to Server Farm
50% of Flows
Have ZERO GLBP Is
Loss W/ 50% Better
GLBP

If You Span VLANS, Tuning Required
By Default, Half the Traffic Will Take a Two-Hop L2 Path
• Both distribution switches act as default gateway
• Blocked uplink caused traffic to take less than optimal path

Core Core
Layer 3 Distribution-A Distribution-B
GLBP Virtual GLBP Virtual
MAC 1 MAC 2
Distribution
Layer 2/3
Si Si

Access F:
Layer 2 Forwarding
B: Blocking
Access-a Access-b

VLAN 2 VLAN 2

Daisy Chaining Access Layer Switches
Avoid Potential Black Holes
Return Path Traffic Has a 50/50 Chance of Being ‘Black Holed’

Core
Layer 3 Si Si
50% Chance That
Traffic Will Go Down
Path with No
Connectivity
Distribution Layer 3 Link
Distribution-A Distribution-B
Layer 2/3 Si Si

Access
Layer 2
Access-a Access-n Access-c

VLAN 2 VLAN 2 VLAN 2

Daisy Chaining Access Layer Switches
New Technology Addresses Old Problems

• Stackwise/Stackwise-Plus technology eliminates the concern
– Loopback links not required
– No longer forced to have L2 link in distribution
• If you use modular (chassis-based) switches, these problems
are not a concern

Forwardin HSRP
g Si Active

Layer 3

Forwarding Si
HSRP
Standby
3750-E

What Happens if You Don’t Link the Distributions?
STP Secondary
• STPs slow convergence can cause considerable Core
Root and HSRP
Standby
periods of traffic loss
• STP could cause non-deterministic traffic STP Root and
flows/link load engineering HSRP Active
Hellos
Si Si
• STP convergence will cause Layer 3 onvergence
• STP and Layer 3 timers are independent
F 2 B
• Unexpected Layer 3 convergence and 2
convergence could occur Access-a Access-b
• Even if you do link the distribution switches
VLAN 2 VLAN 2
dependence on STP and link state/connectivity
Traffic Traffic
can cause HSRP irregularities and unexpected Dropped Until Dropped Until
state transitions Transition to
Forwarding;
MaxAge
Expires Then
As much as 50 Listening and
Seconds Learning

What if You Don’t?
Black Holes and Multiple Transitions …

STP
 Aggressive HSRP
Core Secondary
STP Root and
Core Root and timers limit black
Layer 3 HSRP Active HSRP hole #1
Standby
 Backbone fast limits
Distribution HSRP Active (30 seconds)
time
Layer 2/3 Hellos (Temporarily)
to event #2
Si Si
 Even with rapid
PVST+ at least
one second
before event #2
Access F: MaxAge
Forwarding
Layer 2 Seconds Before
B: Blocking Failure Is
Access-a Access-b Detected…
Then Listening
and Learning
VLAN 2 VLAN 2

• Blocking link on access-b will take 50 seconds to move to forwarding  traffic black hole until HSRP goes
active on standby HSRP peer
• After MaxAge expires (or backbone fast or Rapid PVST+) converges HSRP preempt causes another transition
• Access-b used as transit for access-a’s traffic

What If You Don’t?
Return Path Traffic Black Holed …
 802.1d: up to
STP 50 seconds
Core Secondary
Core
Layer 3 STP Root and Root and  PVST+: backbone
HSRP Active HSRP fast 30 seconds
Standby

Distribution  Rapid PVST+:
Layer 2/3 Hellos address by the
Si Si protocol (one
second)

Access F:
Layer 2 Forwarding
Access-a B: Blocking Access-b

VLAN 2 VLAN 2

• Blocking link on access-b will take 50 seconds to move to forwarding return traffic black hole until then

Asymmetric Routing (Unicast Flooding)
• Affects redundant topologies
with shared L2 access
• One path upstream and two Asymmetric
Equal Cost
paths downstream Return Path

• CAM table entry ages out on CAM Timer Has Upstream Packet
standby HSRP Aged Out on
Standby HSRP
Si Si
Unicast to
Active HSRP
• Without a CAM entry packet Downstream
Packet
is flooded to all ports in the Flooded
VLAN

VLAN 2 VLAN 2 VLAN 2 VLAN 2

Best Practices Prevent Unicast Flooding
• Assign one unique data and voice
VLAN to each access switch
Asymmetric
• Traffic is now only flooded down Equal Cost
one trunk Return Path

• Access switch unicasts correctly;
Upstream Packet
no flooding to all ports Downstream Si Si
Unicast to
Packet
Active HSRP
• If you have to: Flooded on
Single Port
– Tune ARP and CAM aging timers;
CAM timer exceeds ARP timer
– Bias routing metrics to remove
equal cost routes
VLAN 3 VLAN 4 VLAN 5 VLAN 2

Building a Converged Campus Network
Infrastructure Integration, QoS, and Availability
 Access layer
Auto phone detection Inline power Access
QoS: scheduling, trust boundary and
classification
Fast convergence Si Si Si Si Si Si

Distribution
 Distribution layer
High availability, redundancy,
fast convergence
Layer 3 Layer 3
Policy enforcement Core Equal Equal
Si Si
QoS: scheduling, trust boundary and Cost Cost
classification Links Links
 Core Distribution Si Si Si Si

High availability, redundancy, fast convergence Si Si

QoS: scheduling, trust boundary

Access

Infrastructure Integration
Extending the Network Edge

Switch Detects IP Phone and Applies Power

CDP Transaction Between Phone and Switch

IP Phone Placed in Proper VLAN

DHCP Request and Call Manager Registration

Phone Contains a Three-Port Switch that Is Configured in Conjunction with the Access Switch
and CallManager
1. Power negotiation
2. VLAN configuration
3. 802.1x interoperation
4. QoS configuration
5. DHCP and CallManager registration

Enhanced Power Negotiation
802.3af Plus Bidirectional CDP (Cisco 7970)

PSE—Power Source PD Plugged in
Equipment Switch Detects IEEE PD
Cisco 6500,4500,
3750, 3560 PD Is Classified

Power Is Applied

Phone Transmits a CDP Power Negotiation
Packet Listing Its Power Mode
PD—Powered
Switch Sends a CDP Response Device Cisco 7970
with a Power Request

Based on Capabilities Exchanged
Final Power Allocation Is Determined

• Using bidirectional CDP exchange exact power requirements are negotiated
after initial power-on

Infrastructure Integration: Next Steps
VLAN, QoS, and 802.1x Configuration

Phone VLAN = 110 PC
(VVID)
VLAN =
10
802.1Q Encapsulation
with 802.1p
(PVID)
Native VLAN (PVID) No
Configuration Changes Needed
Layer 2 CoS on PC

• During initial CDP exchange phone is configured with a Voice VLAN ID (VVID)
• Phone also supplied with QoS configuration via CDP TLV fields
• LLDP/LLDP-MED is available …
• Additionally switch port currently bypasses 802.1x authentication for VVID if
detects Cisco phone

Best Practices—Quality of Service
• Must be deployed end-to-end to
be effective; all layers play End-to-End QoS
different but equal roles
• Ensure that mission-critical Si Si Si Si Si Si

applications are not impacted by
link or transmit queue Layer 3 Equal Layer 3 Equal
congestion Cost Links
Si Si
Cost Links

• Aggregation and rate transition
points must enforce QoS policies Si Si Si Si

• Multiple queues with Si Si

configurable admission criteria
and scheduling are required

Transmit Queue Congestion
10/100m Que 128k
Uplink
ued WAN
Router

100 Meg in 128 Kb/S out—Packets Serialize in Faster than They Serialize Out
Packets Queued as They Wait to Serialize out Slower Link

1 Gig Link Que 100 Meg
Link
ued
Distribution Switch Access Switch

1 Gig In 100 Meg out—Packets Serialize in Faster than They Serialize Out
Packets Queued as They Wait to Serialize out Slower Link

Auto QoS VoIP - Making It Easy …
Configures QoS for VoIP on Campus Switches

Access-Switch(config-if)#auto qos voip ?
cisco-phone Trust the QoS marking of Cisco IP Phone
cisco-softphone Trust the QoS marking of Cisco IP SoftPhone
trust Trust the DSCP/CoS marking

Access-Switch(config-if)#auto qos voip cisco-phone
Access-Switch(config-if)#exit

!
interface FastEthernet1/0/21
srr-queue bandwidth share 10 10 60 20
srr-queue bandwidth shape 10 0 0 0
mls qos trust device cisco-phone
mls qos trust cos
auto qos voip cisco-phone
end

Best Practices—Campus Security
• New stuff that we will cover!
– Catalyst integrated security feature set!
– Dynamic port security, DHCP snooping,
Dynamic ARP inspection, IP source guard End-to-End Security
• Things you already know—we won’t cover…
– Use SSH to access devices instead of Telnet Si Si Si Si Si Si

– Enable AAA and roles-based access control (RADIUS/TACACS+)
for the CLI on all devices
– Enable SYSLOG to a server. Collect and
archive logs
– When using SNMP use SNMPv3
– Disable unused services: Si Si

– No service tcp-small-servers
No service udp-small-servers
– Use FTP or SFTP (SSH FTP) to move
Si
images and configurations around—avoid Si Si Si

TFTP when possible Si Si

– Install VTY access-lists to limit which
addresses can access management and
CLI services
– Enable control plane protocol authentication where it is available
WAN Internet
(EIGRP, OSPF, BGP, HSRP, VTP, etc.)
– Apply basic protections offered by
implementing RFC2827 filtering on external edge inbound interfaces
For More Details, See BRKSEC-2002 Session, Understanding and Preventing Layer 2 Attacks

Securing Layer 2 from Surveillance Attacks
Cutting Off MAC-Based Attacks

00:0e:00:aa:aa:aa Only Three MAC
00:0e:00:bb:bb:bb Addresses
Allowed on the
250,000 Port: Shutdown
Bogus MACs
per Second

Solution:
Problem: Port Security Limits MAC Flooding
Script Kiddie Hacking Tools Enable Attack and Locks Down Port and Sends
Attackers Flood Switch CAM Tables with an SNMP Trap
Bogus Macs; Turning the VLAN into a Hub switchport port-security
and Eliminating Privacy switchport port-security maximum 10
switchport port-security violation restrict
Switch CAM Table Limit Is Finite Number of switchport port-security aging time 2
Mac Addresses switchport port-security aging type inactivity

DHCP Snooping
Protection Against Rogue/Malicious DHCP Server

1
DHCP
1000s of Server
DHCP
Requests to 2
Overrun the
DHCP Server

• DHCP requests (discover) and responses (offer) tracked
• Rate-limit requests on trusted interfaces; limits DoS attacks on DHCP server
• Deny responses (offers) on non trusted interfaces; stop malicious or errant DHCP
server

Securing Layer 2 from Surveillance Attacks
Protection Against ARP Poisoning
Gateway = 10.1.1.1
• Dynamic ARP inspection protects
Si
MAC=A
against ARP poisoning (ettercap, dsnif,
arpspoof)
• Uses the DHCP snooping binding table Gratuitous ARP
10.1.1.50=MAC_B
• Tracks MAC to IP from DHCP
transactions Gratuitous ARP
10.1.1.1=MAC_B
• Rate-limits ARP requests from client
ports; stop port scanning
• Drop bogus gratuitous ARPs; stop ARP
poisoning/MIM attacks Attacker = 10.1.1.25 Victim = 10.1.1.50
MAC=B MAC=C

IP Source Guard
Protection Against Spoofed IP Addresses
Gateway = 10.1.1.1
Si
MAC=A
• IP source guard protects against
spoofed IP addresses
• Uses the DHCP snooping binding table
• Tracks IP address to port associations
• Dynamically programs port ACL to drop
Hey, I’m 10.1.1.50 !
traffic not originating from IP address
assigned via DHCP

Attacker = 10.1.1.25 Victim = 10.1.1.50

Catalyst Integrated Security Features
Summary Cisco IOS ip dhcp snooping
ip dhcp snooping vlan 2-10
IP Source Guard ip arp inspection vlan 2-10
!
Dynamic ARP Inspection
interface fa3/1
DHCP Snooping switchport port-security
switchport port-security max 3
Port Security switchport port-security violation
restrict
switchport port-security aging time 2
• Port security prevents MAC flooding attacks switchport port-security aging type
• DHCP snooping prevents client attack on the switch inactivity
and server ip arp inspection limit rate 100
• Dynamic ARP Inspection adds security to ARP using ip dhcp snooping limit rate 100
DHCP snooping table ip verify source vlandhcp-snooping
• IP source guard adds security to IP source address !
using DHCP snooping table Interface gigabit1/1
ip dhcp snooping trust
ip arp inspection trust

Hierarchical Campus

Access

Si Si Si Si Si Si

Distribution

Core
Si Si

Si Si Si Si Distribution
Si Si

Access

Layer 2 Access—No VLANs Span Access Layer

• Tune CEF load balancing Si Si
Core
• Match CatOS/IOS EtherChannel settings and
tune load balancing
Layer 3
Distribution
• Limit redundant IGP peering Si
Point-to-
Si

Point
• STP Root and HSRP primary tuning or GLBP Link
to load balance on uplinks
Access
• Set port host on access layer ports: VLAN 20 Data
10.1.20.0/24
VLAN 40 Data
10.1.40.0/24
– Disable trunking VLAN 120 Voice VLAN 140 Voice
Enable PortFast

Layer 2 Access—Some VLANs Span Access Layer
• Tune CEF load balancing
Si Si
• Match CatOS/IOS EtherChannel settings and tune load Core
balancing
Layer 2
• Limit redundant IGP peering
• STP Root and HSRP primary or GLBP and STP port cost tuning Si Si Distribution
to load balance on uplinks Trunk
• RootGuard on downlinks
• LoopGuard on uplinks
• Set port host on access VLAN 20 Data VLAN 40 Data Access
Layer ports: 10.1.20.0/24 10.1.40.0/24
– Disable trunking VLAN 120 Voice VLAN 140 Voice
VLAN 250 WLAN
Enable PortFast 10.1.250.0/24

SmartPorts—Predefined Configurations
Access-Switch#show parser macro brief
default global : cisco-global
default interface: cisco-desktop
default interface: cisco-phone
default interface: cisco-switch Si Si
default interface: cisco-router
default interface: cisco-wireless

Access-Switch(config-if)#$ macro apply cisco-phone
$access_vlan 100 $voice_vlan 10

Access-Switch#show run int fa1/0/19
Si Si
!
interface FastEthernet1/0/19
switchport access vlan 100
switchport mode access
switchport voice vlan 10
switchport port-security maximum 2
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
switchport port-security aging type inactivity
srr-queue bandwidth share 10 10 60 20
srr-queue bandwidth shape 10 0 0 0
mls qos trust device cisco-phone
mls qos trust cos
macro description cisco-phone
auto qosvoipcisco-phone
spanning-tree portfast
spanning-tree bpduguard enable
end

Agenda
Data Center Services
Block
• Multilayer Campus
Design Principles
• Campus Design
Best Practices
• IP Telephony Considerations
• QoS Considerations
Si Si

• Security Considerations
• Putting It All Together
• Summary
Si Si Si Si

Si Si Si Si

Distribution Blocks

Multilayer Campus Architectures and Design Principles

Multilayer Campus Architectures and Design Principles

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Multilayer Campus Architectures and Design Principles

Ähnlich wie Multilayer Campus Architectures and Design Principles (20)

Mehr von Cisco Canada

Mehr von Cisco Canada (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Multilayer Campus Architectures and Design Principles