PLNOG16: Coping with Growing Demands – Developing the Network to New Bandwidths, Arne Heitman

1. Arne Heitmann | Sr. System Engineer | EMEA
PLNOG | Warsaw | February 29th 2016
Coping with Growing Demands
Developing the Network to New Bandwidths

2. © 2015 Mellanox Technologies 2
Entering The Era of 25GbE, 50GbE And 100GbE
Copper (Passive, Active) Optical Cables (VCSEL) Silicon Photonics
100GbE Adapter
(10 / 25 / 40 / 50 / 56 / 100GbE)
Multi Host Solution
32 100GbE Ports, 64 25/50GbE Ports
(10 / 25 / 40 / 50 / 56 / 100GbE)
Throughput of 6.4Tb/s

3. © 2015 Mellanox Technologies 3
Demand - More Virtual Machines Per Server
Interconnect Bandwidth Determines VM Density
10GbE
adapter card
Mellanox
40GbE
adapter card
20 VMs
+
Vs.
+
60 VMs

4. © 2015 Mellanox Technologies 4
Demand - The World of Bandwidth is changing
 International bandwidth growth (projected
2012-2019)
 Global IP traffic by type in petabytes/month
Source: Ars Technica, 2012Source: TeleGeography/ITU

5. © 2015 Mellanox Technologies 5
Description Hres Vres
Colour
depth
(bits)
Pixels
RGB
FPS
RAW BW
(MB/sec)
RAW BW
(Gbits/sec)
8Gb FC
lanes
16Gb FC
lanes
No. 10Gb
lanes
No. 40Gb
lanes
No. 56Gb
lanes
No.
100Gb
lanes
Storage
GB/sec
Strg 90
min
Movie
(TB)
HD Video - Low FPS 1920 1080 16 3 30 373.25 2.99 1 1 1 1 1 1 0.37 2.02
HD Video (US) 1920 1080 16 3 50 622.08 4.98 1 1 1 1 1 1 0.62 3.36
HD Video (EMEA) 1920 1080 16 3 60 746.50 5.97 1 1 1 1 1 1 0.75 4.03
2K Video (US) 2048 1080 16 3 50 663.55 5.31 1 1 1 1 1 1 0.66 3.58
2K Video (EMEA) 2048 1080 16 3 60 796.26 6.37 2 1 1 1 1 1 0.80 4.30
4K UHD (Std FPS) 3840 2160 16 3 30 1492.99 11.94 2 1 2 1 1 1 1.49 8.06
4K UHD (3D FPS) 3840 2160 16 3 60 2985.98 23.89 4 2 4 1 1 1 2.99 16.12
4K Cinema (Std FPS) 4096 2160 16 3 30 1592.52 12.74 3 2 2 1 1 1 1.59 8.60
4K-Full Cinema (Std FPS) 4096 3112 16 3 30 2294.42 18.36 4 2 3 1 1 1 2.29 12.39
4K Cinema (3D FPS) 4096 2160 16 3 60 3185.05 25.48 5 3 4 1 1 1 3.19 17.20
5K Cinema (Std FPS) 5120 2700 16 3 30 2488.32 19.91 4 2 3 1 1 1 2.49 13.44
5K Cinema (3D FPS) 5120 2700 16 3 60 4976.64 39.81 7 4 6 2 1 1 4.98 26.87
8K UHD (Std FPS) 7680 4320 16 3 30 5971.97 47.78 8 4 7 2 1 1 5.97 32.25
8K UHD (3D FPS) 7680 4320 16 3 60 11943.94 95.55 16 8 14 3 2 2 11.94 64.50
Super Hi-Vision 7680 4320 16 3 120 23887.87 191.10 32 16 28 6 4 3 23.89 128.99
Demand:
Media/Entertainment – Acceleration already happening
 Data rates and storage are exploding, due to high pixel counts and frame rates
10GbE Ethernet is not
going to provide the
necessary BW going
forward

6. © 2015 Mellanox Technologies 6
Moving to 25GbE, 50GbE And 100GbE
Compute
Nodes
150% Higher
Bandwidth
Storage
Nodes
25% Higher
Bandwidth
Network
150% Higher
Bandwidth 100GbE
25GbE 50GbE
Same Connectors
Similar Infrastructure
Better Cost / Power
Compute
Nodes
Storage
Nodes
40GbE
Network
40GbE
10GbE

7. © 2015 Mellanox Technologies 7
Silicon - SerDes
 Silicon is connected to board
 SerDes
• Serializer / Deserializer
• May work at
• ~10Gb/s
• ~14Gb/s
• ~25Gb/s
• Addresses a certain number of ports
 Can be bundled:
• i.e. 4x10G for a 40Gb/s link
• i.e. 4x14G for a 56Gb/s link
• i.e. 4x25G for a 100G link

8. © 2015 Mellanox Technologies 8
Silicon – Port Architecture Example
100 GigE
50 GigE
50 GigE
40 GigE
40 GigE
10 GigE
10 GigE
40 GigE
OR
OR
OR
OR
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
20Gig
20Gig
20Gig
20Gig
10Gig
10Gig
10Gig
10Gig
10Gig
10Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
25Gig
n x 100Gig
Port Basic UnitPort Options ASIC

9. © 2015 Mellanox Technologies 9
Silicon - Many Port Configuration Options for n*25GbE
SFP28
SFP+
25GbE (1x25Gb/s)
10GbE (1x10Gb/s)
SFP+10GbE (1x10Gb/s)
SFP+10GbE (1x10Gb/s)
SFP+10GbE
(1x10Gb/s)
SFP2825GbE (1x25Gb/s)
SFP2825GbE (1x25Gb/s)
SFP2825GbE
(1x25Gb/s)
QSFP40GbE
(4x10Gb/s)
QSFP28
50GbE (2x25Gb/s)
QSFP28
50GbE
(2x25Gb/s)
QSFP28100GbE
(4x25Gb/s)
135 watts

10. © 2015 Mellanox Technologies 10
Transceiver – some Numbers
 What comes next?
 50G over a single Lane
• IEEE 802.3 50 Gb/s Ethernet Over a Single Lane and Next Generation 100 Gb/s and 200 Gb/s Ethernet Study Group
 IEEE P802.3by 25 Gb/s Ethernet Task Force
• Standard to come
 IEEE P802.3bs 400 GbE Task Force
• Adopted timeline says Standard for 2018??

11. © 2015 Mellanox Technologies 11
Transcerver - The Evolution between 10Gb, 25Gb, 40Gb, 50Gb &
100Gb
IEEE 802.3bm

12. © 2015 Mellanox Technologies 12
Transceiver - Pluggable Module Standards
 CFP
The CFP MSA defines hot-pluggable optical transceiver form factors to enable 40 Gbit/s and 100 Gbit/s applications.
CFP modules use the 10-lane CAUI-10 electrical interface.
 CFP2
CFP2 modules use the 10-lane CAUI-10 electrical interface or the 4-lane
CAUI-4 electrical interface.
 CFP4
CFP4 modules use the 4-lane CAUI-4 electrical interface.
 QSFP28
QSFP28 modules use the 4-lane CAUI-4 electrical interface.
 CPAK
Cisco has the CPAK optical module that uses the four lane CEI-28G-VSR electrical interface.
 CXP
There are also CXP and HD module standards. CXP modules use the CAUI-10 electrical interface.

13. © 2015 Mellanox Technologies 13
Cables - Optical Connector types for Parallel and single fiber
infrastructures
MPO Optical Connectors
4 Transmit 4 Receive
12-fiber Optical Connector
(4-unused fibers in middle)
Duplex LC 2-fiber Optical Connector
Typically for Single-Mode
Can be used for
Multi-mode (SR4)
or
Single-mode (PSM4)
Also called MTP or MPO/MTP
Single-mode (LR4)

14. © 2015 Mellanox Technologies 14
Cables - Solutions for data center applications
14
Data Center Fabrics
Link Length (m)
10 100 500150 300 1000 2000
10
25
50
3 51
20
DataRateperLane(Gbs)
10000500020 30 50 752
Single mode fiber
OM4OM3
Copper Multi-mode fiber
Silicon Photonics
Direct Attach Copper
• Zero power
• Demo’s 8m at 100G
• Best fit 3m
VCSELsDACs
Active Optical Cables
• VCELs or SiP
• Reaches to 200m
• Best fit for 5-20m
VCSEL Transceivers
• Reaches to 100m
• Best fit for MMF
SiP Transceivers
• Reaches to 2km
• Best fit for SMF
• Parallel or WDM

15. © 2015 Mellanox Technologies 15
Cables/Transceivers - 100GbE Products
100G SR4
Ethernet-Only Transceiver
100G Copper DAC
InfiniBand & Ethernet
100G AOC
InfiniBand & Ethernet
For lowest-cost optical 100G
switch-to-switch links .
For Breakouts to 25G / 50G servers +
storage with breakout fibers.
For low-cost, 100G links up to 100m.
Lowest-cost, 100G-to-Quad-
25/50G Breakout cables.
For Linking Servers + Storage to
ToR Switch & NICs.

16. © 2015 Mellanox Technologies 16
Possible Scenarios

17. © 2015 Mellanox Technologies 17
Multi-Mode Optics
3m-100m
Where Interconnects are Being Used in Data Center
DAC
Server/ToR-to-ToR
SR4
For structured cabling
Short Reaches
PSM4
DAC
WDM4
8-Fiber
MPOAOC: 3-50m
AOC
ToR-Leaf/Spine
“DAC In the Rack”
3m
Quad 25G SFP
breakout
Dual 50G
Breakout
25G SFP
Quad 25G SFP
breakout
Dual 50G
Breakout
25G SFP
For Structured Cabling
Long Reaches
2-Fiber
LC
Single-Mode Optics
Up to 2Km
Optical
Patch
Panel

18. © 2015 Mellanox Technologies 18
Small/Medium Cloud Deployment 10GbE endpoints
(48 + 48)x
10/25GbE
active HA
(48 + 48)x
10/25GbE
active HA
15x Racks
Full L2 solution
1440x 10GbE ports
WAN
access
2x 10/40GbE
uplink to
router
2x SN2700
Spines
To Spine
mLAG
2x SN2410 ToRs
400G
960G
Per Rack
 Pure L2 Network; full HA and no-SPoF
 Phase-1: Start with as small as 1 Rack and
2x ToR
 Phase-2: Add 2x spines (32x100G) and
build up-to 15x Racks in pure L2 domain
 ToR to spine uplinks with 50GbE to ensure
(2+2)x link bundle; mitigate cable failure
 (4+4)x 100/40GbE or (8+8)x 50GbE ports
available per rack for high performance/”fat”
storage nodes
 48+48x 10GbE for compute/hyper-
converged infrastructure
50GbE
100GbE

19. © 2015 Mellanox Technologies 19
Small/Medium Cloud Deployment 25GbE endpoints
(48 + 48)x
10/25GbE
active HA
(48 + 48)x
10/25GbE
active HA
7x Racks
Full L2 solution
672x 25GbE ports
WAN
access
2x 10/40GbE
uplink to
router
2x SN2700
Spines
To Spine
mLAG
2x SN2410 ToRs
800G
2400G
Per Rack
 Pure L2 Network; full HA and no-SPoF
 Ideal for small/medium private cloud
 Phase-1: Start with as small as 1 Rack and
2x ToR
 Phase-2: Add 2x spines (32x100G) and
build up-to 7x Racks in pure L2 domain
 mLAG on ToRs and spines for full active-
active HA
 (2+2)x 100GbE or (4+4)x 50GbE ports
available per rack for high performance/”fat”
storage nodes
 48+48x 25GbE for compute/hyper-
converged infrastructure
100GbE

20. © 2015 Mellanox Technologies 20
1024-port 1:1 100GbE, 2048-port 1:1 50GbE
32x Spines
64x Leafs
16x
100GbE
16x
100GbE
16x
100GbE
1024x 100GbE 1:1 non-
blocking network
All leaf-spine
links are 50GbE
32x 50GbE
uplinks
Same Concept can be used for 2048 port 50GbE 1:1
Can be used as Spine for 3-level networks

21. Thank You

22. © 2015 Mellanox Technologies 22
References
 www.ieee802.org/3/ad_hoc/bwa/BWA_Report.pdf
 http://www.ieee802.org/3/50G/public/adhoc/
 http://www.ethernetalliance.org/wp-content/uploads/2013/04/Ethernet-Alliance-Technology-
Roadmap-FINAL.pdf
 https://en.wikipedia.org/wiki/Terabit_Ethernet
 http://www.ieee802.org/3/bs/
 http://www.open-ethernet.com/
 http://25gethernet.org/
 http://www.mellanox.com/ethernet/