Next Wave of Data Centers & Interconnects

Next Wave of Data Centers &
Interconnects
10GBase-T, DAC,
Avago
40G/100G transceivers,
AOCs, Embedded Modules

Brad Smith
Sr. VP LightCounting
Brad@LightCounting.com

LIGHTCOUNTING 9/18/2012 1
Market Research on High-Speed Interconnects
Datacom, Telecom, CATV, FTTX, Consumer markets © Copyright 2011 LightCounting LLC All material proprietary and confidential.

Agenda
• What is going on in the data centers that is
driving the interconnect space
• 10GBase-T, Direct Attach Copper
• 40G & 100G transceivers
• Active optical cables
• Embedded Optical modules

LIGHTCOUNTING 2
Market Research on High-Speed Interconnects 9/18/2012

About LightCounting
High speed interconnect Research
LH telecom & FTTx -to- Data centers
Ethernet, Fibre Channel, InfiniBand
Vertical areas: 40/100G, AOCs, EOMs
Transceiver Research Databases

http://www.lightcounting.com/reports.cfm
LIGHTCOUNTING 3

Problem with Optical Interconnects
All Predictions are 35%-90% Traffic Growth
IEEE BWA, Cisco VNI, LightCounting Model

LIGHTCOUNTING 4

Virtually all the Internet traffic – goes through an optical
Virtually all the Internet traffic – goes through an optical
transceiver somewhere in the telecom and datacom network!
transceiver somewhere in the telecom and datacom network!

Supplied by transceiver companies…
About one quarter away for bankruptcy

LIGHTCOUNTING 5

Virtually all the Internet traffic – goes through an
optical transceiver somewhere in the telecom and
datacom network!

Supplied by transceiver companies…
About one quarter away from
bankruptcy!

LIGHTCOUNTING 6

Problem with Optical Interconnects
What’s Wrong with this Picture?
IEEE BWA, Cisco VNI, LightCounting Model Optical Stocks

Traffic going through roof
Profits going through the floor

LIGHTCOUNTING 7

Big Changes in Data Centers
Changes in Data Centers Drives Interconnects

LIGHTCOUNTING 8

M&As Restructure the Industry
~24-months
Systems
Cabling
Transceivers
Semiconductors


Cabling Companies Buy AOC Startups + BUs
• TE bought Zarlink BU
• FCI Bought MergeOptics
• Molex bought Luxtera BU
• Samtec bought AlpenIO

Result:
• $ xx Billion, international connector, cabling, companies
• Offering transceiver and AOCs, EOM, VCSELs
– New set of competitors for transceiver companies
– Full line of copper & optical products
– Power cables, every protocol, every MSA, connector, etc

LIGHTCOUNTING 10

Cisco Introduced Blade Servers
Competes against former server partners
• HP bought 3Com/3Par
• IBM bought Blade Network Technologies
• Dell bought Force10
• Oracle bought Sun Microsystems

• Mellanox bought Voltaire (InfiniBand)
• Intel bought QLogic and Fulcrum (InfiniBand) & Crays HPC
Intereconnect BU

Now integrated Server and Switching companies

LIGHTCOUNTING 11

Cisco Buys Transceiver Companies
CoreOptics & LightWire
• CoreOptics: 100G Long haul in-house
• LightWire: Silicon photonics in-house
– 4x25G (100G) -- WDM
– High speed modulation (40G) & integrated electronics
– Data center (500 meters) & Metro (10Km)

Result:
• Servers, switches, and interconnects – all in-house

LIGHTCOUNTING 12

Software Defined Networking
• SDN Virtualizes the networking hardware
• VMWare bought Nicira
• Citrix bought Nutanix
• Oracle bought Xsigo
• Cisco Systems’ investment in Insieme
• Bottom line:
– Big technology companies are buying their way into
the virtualized storage and virtualized networking
markets.
– Virtualizations will likely increase all the DC traffic
LIGHTCOUNTING 13

Same thing that happened with PCs in 1980s with
Taiwanese clone PCs and big name brands

……is happening to servers and switches!

This will change everything!

LIGHTCOUNTING 14

Enter the White Box Clones
Servers
• Big Internet related companies (Google, Facebook, Amazon, etc.) buying
servers directly from ODMs
• Making designs available for anyone to buy
• Server business already at 22% GM & heading lower
• NET: White box commodity servers!

Switches
• White box ToR and EoR switches
• Buy switch chips from Broadcom, Marvel, Mellanox, …Intel
• Run OpenFlow (open source network switching software)
• No longer special switching software play
• No longer a Microsoft-Intel instruction set lock
• NET: White box commodity switches!

LIGHTCOUNTING 15

Reactions
Reactions likely: “Rack as a Sku”
• M&As: Buy transceiver companies Cisco/Lightwire
• Advanced technology: Use technology as a weapon against
low cost manufacturing white boxes
• More highly integrated Blade server-switch boxes
• Rack-as-a-Sku”
• Non-Intel Alternatives: ARM cores, Power7, SPARC
microprocessors competing with Intel
• HPC technologies hit high-end data centers
• Software Defined Networking Switch
Integrated Blade Server/Switches
blade
Mez board
All these changes are going to rewrite Copper KR
the Server/Switch and Interconnect
landscape going forward Server
blade

LIGHTCOUNTING 16

Server-Switch Architectures
1RU Servers with Top-of-rack Switches Blade Servers with Integrated Switches

2 Sockets 4 Sockets

No single solution!

LIGHTCOUNTING 17

Server & Switch Industry Transitions
Custom (e.g Google) Multiple Blade Server Bays per Rack
Traditional Rack Servers & White Box Clones
& Top-of-rack switches DAC, QSFR SR & AOCs
DAC, QSFR SR & AOCs SMF 2Km nR4 transceivers
From Major System Suppliers
SMF 10Km LR4 transceivers
Cisco, HP, IBM, etc.

Focus on advanced
architectures & long term

Internally connected with EOMs
and AOCs

SMF 2Km nR4 transceivers

“Rack as a Sku”

Focus on low prices & 2015+
3 yr life cycles

Today 2013-2014

Brad Smith Brad@LightCounting.com
LIGHTCOUNTING 18

State of the Industry
Mid 2012

Waiting for New Servers
Waiting for 10GBASE-T
Waiting for Europe’s economy to
stop crashing

Preparing for the Exa Flood of Data

• Data centers are largely very cost sensitive;
• Cost governs all decision making (over technology)
– Decision algorithm (Cost of new product) / Server cost
– Buy 100G LR4 transceiver OR 8 servers?

• Average and large data center needs 4-5G, not 10G yet
• Getting ready for 10G and 40G QSFP

• Most popular optical transceiver = 1G SFP+ & 10G SFP+

LIGHTCOUNTING 20

Servers & Switches
• Intel’s next gen microprocessor “Romley” server architecture – delayed ~6-9
mo’s
– 4 cores  10 cores (14 cores AMD) -- PCI Express Gen 2  Gen 3 with twice I/O
– QuickPath to bypass bus bottlenecks -- 32-nm very energy efficient

• Top-of-rack & End-of-row switches are 10G SFP+ + 40G QSFP ready but ….

• Entire industry was “Waiting for new Romley servers”
• And 10GBASE-T low-power 28-nm parts

LIGHTCOUNTING 21

Intel’s Tick-Tock Model
Sets the Clock for Data Center Upgrades
Tick= Architecture upgrade ; Tock= Silicon Shrink

Data center spending delayed by 8 months

LIGHTCOUNTING 22

Microprocessors
2-4 Cores 10-16 Cores 40-50 Cores

2 MPUs/Server 4 MPUs/Server ? MPUs/Server
128GBytes DRAM 768G Bytes DRAM 1T+ Bytes DRAM
PCIe Gen2 (8 lanes @5G/lane PCIe Gen3 (8/16 lanes @8G/lane PCIe Gen3 (8G/lane)
ARM, INTEL, AMD

?
Data center
in a server

LIGHTCOUNTING 23

I/O for Interconnects
Virtualization software
• Sends jobs to non busy MPUs
– Increases utilization from 20% to 90%

PCIe Bus
20% 90%

• I/O increases: Data Direct I/0 - transfer of data directly to from the
network interface and cache, bypassing main memory

PCI Express Bottom line: ~200-%300 % faster
I/O bandwidth
• PCI Express Gen 2 5G 8b/10b and 8 lanes
• PCI Express Gen 3 8G 128/132 and 8-16 lanes

LIGHTCOUNTING 24

Servers per Rack
60-80 Severs / Rack
48 - Severs / Rack

LIGHTCOUNTING 25

Traditional Data center
Computer & Switch History Museum

LIGHTCOUNTING 26

Modern Data center

keyitec.com

bestwebhostinggeek.com

LIGHTCOUNTING 27

Mega Data Center

.primecaretech.co
LIGHTCOUNTING 28

SuperComputer Site (HPC)
18M Watts of power

LIGHTCOUNTING 29

Apple South Carolina Data Center

LIGHTCOUNTING 30

Interconnect Technologies

LIGHTCOUNTING 31

“Waiting for 10GBASE-T
• Will be Reeaaaaly big!
• “When it gets small ….….. it will be big” 130-nm - This will be the year! NIC
90-nm - This will be the year !
• Lan On Motherboard (LOM) is key to market 65-nm - This will be the year !
- PHY chip needs 28-nm (now at 40-nm) 40-nm - This will be the year!
28-nm – This will be the year!
- Late 2013/2014
- 10GBASE-T likely the last LOM implementation

• VCs spent $850M on 10+ startups LOM

• Vendor community still shrinking
– Broadcom, Marvel are the 2 main players left
– Also Applied Micro, Aquantia in game
– PLX/Teranetics
• Sold Phy division to Aquantia
– Aquantia: Last remaining VC funded startup
Main advantage: No huge markup like with Optical transceivers
Will be big with LOM 2014
LIGHTCOUNTING 32

IEEE Update
40GBASE-T
• New CFI at the IEEE (July meeting) – everything is indeterminate at this point

• No vendor is anywhere close to payback 10GBASE-T investments so real
support is likely to be slow – 2017-2018 event
• Designed for 2016 when Romley+ moves the server to 40G uplinks
• 85-90% use will be Server uplinks to top-of-rack switch
• 4 twisted pairs
• Encoding(?): 10G using differential signaling across pairs
• Scheme: 4x10G configuration
• Reach(?): 50 m (Not likely 100m anymore)
• Connector(?): Likely new RJ-4x; Likely not field terminated
• Cable(?): Cat8? Cat7a
– In any case – it’s a NEW cable! Meaning the “existing installed cable is not usable”
• Biggest advantage is = no optical price mark ups by suppliers
• Not likely to be supported by any startup so its Broadcom/Marvel game
LIGHTCOUNTING

Direct Attach copper
• Has become very popular <5-7meters
• Filling the 10GBASE-T void
• Low-cost, low-power, Plug & Play
• Cheaper than optical SFP+
• Active & Passive DAC
• Outselling 10GBase-T 4:1

• Numerous problems surfacing
– Electrically linking different vendors equipment together directly – different
than linking optically
– “Limiting” and “non-limiting” ports – no warning labels

• Net: “DAC in the Rack” 
• ---until 10GBASE-T LOM shows up  10G = 7 meters
14G = 3meters
28G = ????

LIGHTCOUNTING 34

100G SR & LR4
• So called 2 Km solution really 4-6 dB issue
• IEEE is still working the technical issues on
Reach 400m-1,200m
• 25G VCSELS maybe reach 25 meters!
– 125 meters with FEC chips added (more cost,
more power)
• 50G maybe 12 meters without FEC

LIGHTCOUNTING 35

Why 100G Needed in Data Centers
100G Driving Forces
• 48 – 80 Servers/ rack
• 10Gbps server uplinks

• 480G -800G problem!

• Will demand 100G uplinks

• Timing: Romley Q2/2012 : Volume 2H/2012

• ~ 2H/ 2013 need for 100G transceivers climb rapidly
• Then it pops to 40G server uplinks in 2015/16!
LIGHTCOUNTING 36

100G

LIGHTCOUNTING 37

Mega Data Center
400-600metes
Not 2Km!
4-6dB link budget
To this rack

Link this rack

.primecaretech.co
LIGHTCOUNTING 38

Active Optical Cables

LIGHTCOUNTING 39

What is an Active Optical Cable?
• Parallel optic transceiver in a standard MSA shell but… integrates
the fibers directly into the MSA
• No optical connectors
• Changes the game!
Finisar

Molex

LIGHTCOUNTING 40

Active Optical Cables
• Very popular in High Performance computers (HPCs) TOP25
• Technology is proven out and is rippling down to Top 25-100 & High-
end data centers
• Used to connect top-of-rack switches to end-of-row switches (4x10G
QSFP)
• Large router companies are starting to use 12x10G CXP
• Advantages:
– Plug & Play; green light;
– No optical connector cleaning, familiar copper connector;
– low power (700mW per end)
• InfiniBand today but moving to Ethernet, SAS, PCI Express
LIGHTCOUNTING 41

1 Core AOC Technology
Avago
Creates Multiple Products
• Multiple types
– 1x10G SFP+ - 4x5G CX-4
–
Photonera
4x10G QSFP - 4x14G QSFP AlpenIO

– 12x10G CXP - 12x12.5G CXP
– 8 x 8G iPass
• Breakout AOCs: 4-SFP/QSFP and 3-QSFP/CXP
FCI

• Embedded Optical Module (EOM) replacing SNAP12
– For inside chassis applications (no MSAs yet)
• Parallel Optical Transceivers
• AOC/Transceiver: AOC with connectorized fibers
• Multiple applications: HPC to consumer
GigaLight
LIGHTCOUNTING 42

CES AOCs

LED lights
To
USB and HDMI cables
To
Supercomputers AOCs

LIGHTCOUNTING 43

Consumer Embedded AOCs 4x12.5G

LIGHTCOUNTING 44

Embedded AOCs 4x12.5G

Parallel
AOC

LIGHTCOUNTING 45

Embedded Optical Modules
The End of Pluggable Optics?

LIGHTCOUNTING 46

“Avago MicroPOD-Like Thingies”
• Inside chassis applications
• Mid-board optics
• Start out at 10G but the big market will be at 25G
• Start out in large routers; later move to servers
Finisar

Avago

NEC

LIGHTCOUNTING 47

Three Generations of EOMs

LIGHTCOUNTING 48

cal mess! 

LIGHTCOUNTING 49

Its all about achieving high
density interconnects

LIGHTCOUNTING 50

IBM HPCs & Avago MicroPODs

LIGHTCOUNTING 51

IBM

LIGHTCOUNTING 52

IBM

LIGHTCOUNTING 53

Massively complex mechanical assembly!

LIGHTCOUNTING 54

8-10 Companies Showing Prototypes
2-3 With Announced products

LIGHTCOUNTING 55

Where Are EOMs Used

Demo board from
US Conec and Avago

LIGHTCOUNTING 56

Sony AOC/EOM
CXN2006, CXN2007 and CXN2300 optical interconnection modules

• Designed for 4K x2K video for HDTVs and digital cameras
• 4x10G based on VCSELs with a lens-less internal design
• 10mm x 16mm

http://www.sony.net/Products/SC-
P/cx_news/vol68/pdf/featuring_vol68_interconnection.pdf

LIGHTCOUNTING 57

Fujitsu

LIGHTCOUNTING 58

SNAP12 vs MiniPOD
Clearly EOMs have an advantage over SNAP12s

LIGHTCOUNTING 59

NEC 12x25G EOM at OFC2011

LIGHTCOUNTING 60

Where Are EOMs Used?

LIGHTCOUNTING 61

Where do you put an EOM?

LIGHTCOUNTING 62

Avago EOMs

LIGHTCOUNTING 63

Offers extremely high density Optical interconnects to board edge

125mW/ch @ 10G
12 channels simplex =
1.45W
8.8 W

Duplex link would use 12
EOMs = 17W

LIGHTCOUNTING 64

Optically Enabled Hybrid Chip Packages

• FTTChip
– Mez cards
– Chip/Optical

Reflex/Fulcrum

LIGHTCOUNTING 65

The End of Pluggable Optics?
• Answer=NO! But it is one more opportunity.
– No standard electrical or optical connectors?
– No interoperability requirements?
– No standard configurations 4x, 12 channel?
– Every supplier has unique products (no second source)
• The industry will be able to achieve enough density in the data
center with pluggable optics for at least next 4-5 years
– 8 CFP/2s running 100G = 800G
– 36 QSFP28 running 100G = 3,600G

1. Significant concerns about mixing 70C optics in 100C ASICs
2. Biggest limiter is repairs on failure Biggest inhibitor
– Pluggable = changing fan belt on car of market
– EOM = changing the piston adoption
3. Rigorous vendor qualification and selection
3. Few major players
LIGHTCOUNTING 66

Conclusion
• Data center are in tremendous change
– Impact or clone Servers/switches
– M&As re-writing the landscape
• Low cost AOCs and Embedded Optical
Modules
• Transition to 25G signaling for SR and LR4
• 25G is likely to “thin the heard” in transceivers

LIGHTCOUNTING 67

Thank You

Our Website:
www.lightcounting.com Brad@LightCounting.com

Headquarters Location:

858 West Park Street
Eugene, OR 97401
Tel: 408.962.4851
Email: info@lightCounting.com

LIGHTCOUNTING 68

Next Wave of Data Centers & Interconnects

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (14)

Ähnlich wie Next Wave of Data Centers & Interconnects

Ähnlich wie Next Wave of Data Centers & Interconnects (20)

Next Wave of Data Centers & Interconnects