Highly available and tunable control planes are difficult to build and manage. Is there an alternate way to build a control plane for cloud scale fabrics that will reduce operational expense (coming as close to zero touch provisioning as possible), while allowing the network to be tuned in near real time based on telemetry and application requirements? LinkedIn is currently working on such a control plane, starting from the concept of layering different control plane functionality. This talk will provide an overview of the functional division, consider some tools which can be used to meet each, and the consider the resulting operational profile.
1. LinkedIn’s Approach to Programmable Data Center
Shawn Zandi
Principal Architect
Infrastructure Engineering
2. LinkedIn Infrastructure
• Infrastructure architecture based on application’s behavior & requirements
• Pre-planned static topology
• Single operator
• Single tenant with many applications
• As oppose to multi-tenant with different (or unknown) needs
• 34% infrastructure growth on annual basis and close to half a billion users
3. Edge Network to Eyeballs
Backbone Network
Bare Metal
Operating System
Container
Application
End to End Network Design
Data Center Network
End to end control enables us to tackle problems at different parts of the stack
From application code, os, network or client software to solve by architecture…
Bare Metal
Operating System
Container
Application
4. Traffic Demands
• High intra and inter-DC bandwidth demand due to organic growth
• Every single byte of member activity, creates thousands bytes of east-west traffic inside
data center:
• Application Call Graph
• Metrics, Analytics and Tracking via Kafka
• Hadoop and Offline Jobs
• Machine Learning
• Data Replications
• Search and Indexing
• Ads, recruiting solutions, etc.
5. Scaling Out Data Centers Network - Hardware
• White-box Switches (ODM)
• Vendor Switches Based (OEM)
• Based on Merchant Silicon
• Big Chassis Switches
• Designed around robustness (NSR, ISSU, etc.)
• Feature-rich but mostly irrelevant to LinkedIn needs
Project Falco
6. Data center were designed by redundant chassis at core
controlling and forwarding east-west and north-south traffic
8. Why No Chassis?
• Robust-yet-Fragile
• Complex due to NSR, ISSU, feature-sets, etc.
• Larger fault domain, Fail-over/Fail-back
• Indeterministic boot up process and long upgrade procedures
• Moved complexity from big boxes to our advantage, where we can manage and control!
• Better control and visibility to internals by removing black-box abstraction!
• Same Switch SKU on ToR, Leaf and Spine (Entire DC)
• Single chipset uniform IO design (same bandwidth, latency and buffering)
• True 5-Staged Clos Topology! with deterministic latency
• Dedicated control plane, OAM and CPU for each ASIC
9. W X Y Z
W X Y Z
W X Y Z
Distributed Control Plane Complexity
Pod 1
2 32…1
Pod 11
322 352…321
Pod 21
642 672…641
Pod 31
962 992…961
W X Y Z
2171217021692168213121302129212820912090208920882051205020492048
2339233823372336 2368 2369 2370 2371 2400 2401 2402 24032307230623052304
10. “Fabric wide visibility and telemetry”
The wider the fabric, flow tracking and fault isolation becomes more difficult
Problem 1
11. “Fabric wide traffic distribution and packet scheduling!”
Forwarding is different than routing, and out of scope for routing protocols.
Problem 2
We need a robust and scalable control protocol designed for a data center fabric
12. Control Plane :: Routing
• Routing protocols provide destination-based reachability information
• Routing protocols are not traffic aware.
• Best path selection is elementary.
• Network graph is built based on series of ECMP groups,
“Routing protocols are more about the destination than the journey”
14. ECMP is not really equal!
• Elephants and mice issue
• ECMP Hashing is not bandwidth aware. Devices use an algorithm to
distribute traffic amongst links regardless of load.
• Traffic is routed using shortest path, not all the available paths,
hence not maximizing all the available capacity. Some links may
suffer while the other may be underutilized.
• Flows stick to a certain path, as hashing is performed per flow. An
established socket cannot be moved to a different path easily!
15. “We need a robust and scalable fabric-wide forwarding policy”
Problem #4
16. Lack of Centralized Policy and Control
• The more parallel links you add, forwarding decision becomes more
random.
• Devices were configured and maintained individually
• Routing/Forwarding policy management tasks are performed
individually and hop by hop.
• Know when/where to centralize or distribute to scale out!
17. “End to End Path Selection & Control”
No application, protocol or packet can dictate a path
Centralized flow based routing does not scale!
Problem #5
18. “Using the same familiar, robust and well-known solutions brings along the same
restrictions when they were originally designed”
Problem #6
20. IP Routing History
• IP routing is defined hop-by-hop
• BGP is “the” IDR designed to work between different autonomous
system, to provide policy and control between different routing
domains to select a best path.
• True: BGP can scale and is extensible. BGP has many policy knobs.
• A datacenter fabric operated under a single administrative domain
instead of series of individual routers with different policies and
decision process.
21. Forwarding traffic based on demands & patterns:
• Application
• Latency
• Loss
• Bandwidth (Throughput)
Programmable Data Center
A data center fabric that distributes traffic amongst all available links efficiently and
effectively while maintaining lowest latency and providing the most possible bandwidth to
different applications based on different needs and priorities.
25. Distributed control plane for topology discovery and reachability information
+
Use a controller software for forwarding policy and optimizations
Approach #3
26. Scale: No state or flow information required to be stored on every box
Network can choose and move flows dynamically
Application can choose and move flows dynamically
Works with existing data plane (merchant silicon support)
Supports ECMP with fallback to IP routing
Automatic Local Repair / LFA
27. Hardware
Routing
Policy
Applications
Link Selection and Scheduling
Topology Discovery and Network Graph
Control
Telemetry/Visibility, Machine Learning, Prediction Engine, Self Healing, etc.
Forwarding
Merchant Silicon
Rethinking The Network Stack
29. Network ElementNetwork ElementNetwork ElementNetwork Element
Management
Plane
SNMP, Syslog, etc.
System &
Environmental
Data
Packet & Flow
Data
Network Operating System
Kafka Agent
Monitoring and Management System
Kafka Broker
Machine Learning & Data Processing
Alert
Processor
Log Retention
Data Store
Event
Correlation
Kafka Pub/Sub Pipeline
Record, Process and Replay Network State
30. Open19
OpenFabric
ASIC ODM
RIB / Forwarding Abstraction Layer
FALCO
Apps
Linux OS
Hardware
Physical Layer
Hardware Abstraction Layer
Metrics & Analytics
Machine Learning
Self Healing
etc. (API to Infrastructure)
Policy & Control
Operating System
Base Networking
LinkedIn Infrastructure Strategy
31. • Unified Architecture
• We used a single SKU (hardware and software) for all switches while procuring
hardware from multiple ODM channels (multi-homing)
• One Software: Base Networking on Merchant Silicon with minimum req. features
• No Overlay - For the infrastructure, the application is stateless
• No Middle-box (Firewall, Load-balancer, etc.) Moved to application
• Network is only a set of intermediate boxes running linux
Simplified Infrastructure to Own
32. • To control and own your architecture:
• End to end stack (app, operating system, network and architecture.)
• Ultimate sophistication: Simplicity
• In house support as far as possible
• Move complexity to your comfort zone!
Stay in Control
33. • SDN is nor a protocol or a tool or product off the shelf
• SDN is the whole network stack and architecture that enables
applications to meet and interact with infrastructure to:
SDN for LinkedIn
• Discover and Learn
• Provision
• Manage
• Control
• Monitor
34. Project Altair: The Evolution of LinkedIn’s Data Center Network
Project Falco: Decoupling Switching Hardware and Software
Open19: A New Vision for the Data Center
Hinweis der Redaktion
This is the luxury that public cloud doesn’t offer: solving scale or performance issues, many different possible ways as we own the end to end stack from the application serving in the DC to the app on clients phone!
Scale Example: one possible approach is to throw more machines at it
We believe if application performance and user experience is our primary mission, owning end to end infrastructure is not optional. That is why we do not have any plan to move to any infrastructure outside LinkedIn’s control from top to bottom.
The infrastructure growth more than anything was on data center networking as traffic demands.
and we chose to build our network on top of merchant silicon. a very common strategy for mega scaled data center.
Let’s look at the history of data center network:
DCN were built by same building blocks that constructs campus or enterprise networks, chassis architecture and hierarchical access, distribution and core model of deployment
Vendors: same set of software features set, protocols and architecture was being used for campus, enterprise and data center networks.
Making a big switch out of pizza box switches that can scale horizontally. In this model each plane can have up to 32 switches, with 4 planes we can have 128 switches in the fabric.
That is 4,096 x100 Gig ports. You cannot buy a chassis switch with 4096 ports in the market.
No control over what line cards boot first, and black holing traffic until software is fully loaded and control plane in sync
Chassis switches simplify management and control plane of multiple line cards (modules) by managing multiple chipsets under the same roof, as the code abstracts the complexity inside the chassis from the rest of the network however it translates into code complexity where we did not have control over.
Our strategy is to simplify what we don’t have control over, and move complexity to where we can manage and control. We can manage multiple pizza-box switches via distributed control plane, software automation and zero touch provisioning…
In our case, we used the same switch code and chipset as the building block for the entire data center where operation staff no longer need to focus on hardware variables and can shift their attention to the software and automation pieces.
As we grow there will be more switches and links to manage, the more the feel and need to enhance programmability into the network. This wide ecmp structure creates complexity
chassis switches provide cell switching, dbd, virtual output queue (head of the line blocking) across backplane, since there’s no big enough switch to serve the whole data center, this task should be performed across the fabric!
non of the routing protocols were designed (originally) for a fully mesh, ecmp network, extensions to support multi path and features such as mesh group added later, but it does not really solve this use case,
routing protocols says: here is the destination prefix, and here is the next hop, good luck with that
Folded CLOS is not perfect, once that network utilizes more bandwidth we will see some links carry more traffic than the others because ECMP algorithm or equal cost multi path (sharing bandwidth across multiple links) is based on a hashing mechanism.
Once that a path (a series of links) instead of series of nodes is determined, apply and enforce this in an end to end .
Distributed architectures are usually faster to compute, but depending on the case, there are examples that centralize compute can be faster than overall distributed form, due to eliminating duplicated tasks and event propagation…
BGP in DC not the complete answer. It works for what is expected for BGP, not necessarily for what expected in a data center. BGP does a perfect job for the intent behind its design, the intent was to have a good enough, path-vector protocol to scale to internet size with policy control…
Although protocols will change over time as they gain new functionality, the foundation and fundamental thought process behind them when there were designed does not or cannot be changed.
While we are looking for fabric based forwarding, higher-level intent based forwarding, operation is
bgp brings an illusion of control in the dc, ask those operator if they really control whats happening there?
ForCES and OpenFlow, providing flow based routing
We already know that this does not scale!
Examples, Segment Routing, MPLS, or Cisco ACI by utilizing VXLAN
OpenFabric is open and extensible software code that extends to run on different silicons and operating systems.
Open19 provides passive backplane for data without a need of wiring and power rails to fit servers, storage and switches in standard 19 inch cabinets.
Simplified architecture is desirable to simplify ownership
We believe it is important as a content provider company to control our own destiny,
own to customize based on your needs, and only your use cases
Priority #1 is to stick to and own our architecture, the simple architecture that is driven by our application. in order to control this we need to own several pieces of this puzzle.