Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Physical layer aggregation
Similar to Physical layer aggregation (20)
More from Raafat younis
More from Raafat younis (14)
Recently uploaded
Recently uploaded (20)
Physical layer aggregation
- 1. 1 IEEE 802.3 HSSG Physical Layer Aggregation update Howard Frazier Shimon Muller Drew Perkins November, 2006 Dallas, TX
- 2. 2 IEEE 802.3 HSSG Outline • Application needs • Resources • 802.3ad Link Aggregation • 802.3ah PME Aggregation • Aggregation at the Physical Layer (APL) • Configuration examples • Application with various PHYs • Summary
- 3. 3 IEEE 802.3 HSSG Application needs Backplane Scalable speed Resiliency Many ports in small space Trade-off cost of signaling rate vs. number of channels Copper (data center) Scalable speed Resiliency Relatively inexpensive medium Cable characteristics limit signaling rate Short Haul Fiber Scalable speed Resiliency Relatively inexpensive medium Cable characteristics limit signaling rate Long Haul Fiber Scalable speed Resiliency Relatively expensive medium Economics limits signaling rate
- 4. 4 IEEE 802.3 HSSG Resources Backplane 4 x 2.5 G common today Variable lane widths common XAUI, IB, PCI-e 10GBASE-KR nearing completion Signaling rates > 10 G will be challenging Copper (data center) 10GBASE-T, 10GBASE-CX4 With time, cost and power consumption will decline Signaling rates > 10 G will be challenging Short Haul Fiber 10GBASE-SR, 10GBASE-LX4, 10GBASE-LRM Multimode fiber at 850 & 1300 nm Signaling rates > 10 G will be challenging Long Haul Fiber 10GBASE-LR, 10GBASE-ER, 10GBASE-LW, 10GBASE-EW Single mode fiber at 1300 & 1550 nm Signaling rates > 10 G will be expensive
- 5. 5 IEEE 802.3 HSSG Resources Backplane 25 G serial may require multilevel signaling, higher cost channels, more signal processing higher power, cost and complexity n x 25 G will not be economically feasible any time soon Copper (data center) Quad (octal?) 10GBASE-T PHYs drive growth of multi-port switches Packaging in multiples of four fits well with aggregation in multiples of four Short Haul Fiber Parallel ribbon fiber VCSEL arrays demonstrated Multi-port (quad?) 10GBASE-LRM Long Haul Fiber Photonic Integrated Circuits 10 x 10 G demonstrated Declining cost of WDM
- 6. 6 IEEE 802.3 HSSG 802.3ad Link Aggregation • Specified in Clause 43 • LAG is performed above the MAC • LAG assumes all links are: – full duplex – point to point – same data rate • Provides graceful recovery from link failures • Traffic is distributed packet by packet • All packets associated with a given “conversation” are transmitted on the same link to prevent mis-ordering
- 7. 7 IEEE 802.3 HSSG 802.3ad Link Aggregation
- 8. 8 IEEE 802.3 HSSG 802.3ad Link Aggregation MAC Client PHY MAC PHY MAC PHY MAC PHY MAC distributor collector MAC Client PHY MAC PHY MAC PHY MAC PHY MAC distributor collector
- 9. 9 IEEE 802.3 HSSG 802.3ad Link Aggregation 43.2.4 Frame Distributor … This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 43.2.3, the algorithm shall not cause a) Mis-ordering of frames that are part of any given conversation, or b) Duplication of frames. The above requirement to maintain frame ordering is met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to re-order frames. …
- 10. 10 IEEE 802.3 HSSG 802.3ad Link Aggregation • Does not change packet format – No added headers or sequence numbers – Type/Length interpretation unchanged • Does not require added buffers – No fragmentation or reassembly • Does not re-order or mis-order packets • Does not add significant latency • Does not increase the bandwidth for a single conversation • Achieves high utilization only when carrying multiple simultaneous conversations • Is not transparent to some 802.1 sub-layers
- 11. 11 IEEE 802.3 HSSG 802.3ad Link Aggregation • Can link aggregation be “fixed”? • Not so easy to accomplish – Inspect headers deep into packet or – Add sequence number to packet • Change the packet format, start a food fight with dot1 and – Add LARGE buffers to receiver – Add LONG delay • Still will not achieve linear scaling
- 12. 12 IEEE 802.3 HSSG 802.3ad Link Aggregation • Is a very good thing – It does what it was intended to do – It is relatively easy to implement and use • Does not always provide a linear multiple of the data rate of a single link – N aggregated links usually do not provide N times the bandwidth • Incurs a linear multiple of the cost of a single link – N aggregated links cost N times as much as a single link, because everything must be replicated • Appears to the user as N individual links, which must be individually managed
- 13. 13 IEEE 802.3 HSSG 802.3ah PME Aggregation • Specified in Clause 61 • Supports multi-pair bonding for 10PASS-TS and 2BASE-TL • Performed in the Physical Layer • Aggregates up to 32 Physical Medium Entities (PMEs) • Ensures low packet latency and preserves packet sequence • Scalable and resilient to PME failure
- 14. 14 IEEE 802.3 HSSG 802.3ah PME Aggregation MII γ
- 15. 15 IEEE 802.3 HSSG 802.3ah PME Aggregation MAC Client MAC PME PME Aggregation PMEPMEPME MAC Client MAC PME PME Aggregation PMEPMEPME
- 16. 16 IEEE 802.3 HSSG 802.3ah PME Aggregation
- 17. 17 IEEE 802.3 HSSG 802.3ah PME Aggregation • Fragments can vary in size from 64 to 512 octets • 16 or 32 bit Fragment Check Sequence (FCS) provided by the underlying TC sublayer • Data rate can vary by 4:1 between fastest and slowest PME in an aggregation • Differential delay restricted to 15,000 bits • The combination of fragment size, differential delay, and speed ratio limits the size of the reassembly buffer to 214 bits (2 kB) per PME
- 18. 18 IEEE 802.3 HSSG 802.3ah PME Aggregation • Fragmentation and reassembly supports much higher link utilization compared to 802.3ad • Traffic in a single conversation is not limited to traversing a single link • Provides a nearly linear increase in bandwidth for a (less than) linear increase in cost
- 19. 19 IEEE 802.3 HSSG Aggregation at the Physical Layer (APL) • The PME aggregation concept can be used with existing 10GBASE PHYs • Will need a fragment CRC (or equivalent) to protect against fragment loss or corruption – must protect fragmentation header at the very least • Fragmentation header can be changed if desired • Preamble should be propagated, not stripped – various standards-based and proprietary uses • maxFragmentSize, minFragmentSize, maxDifferentialDelay and maxSpeedRatio can be changed as needed to balance efficiency vs. buffer size and latency
- 20. 20 IEEE 802.3 HSSG APL fragment format IPG Preamble MAC headers & data IPG Fragment 1Frag Hdr Frag Hdr Frag Hdr Frag Hdr Frag Hdr Frag Hdr PCS 1 PCS 2 PCS n Seq # S E 14 1 1 Preamble CRC-8 FCS SFD Fragment n+1CRC-8 Fragment 2CRC-8 Fragment nCRC-8 Fragment n+2CRC-8 Fragment n+nCRC-8
- 21. 21 IEEE 802.3 HSSG Fragment format considerations • Shorter fragment header + longer fragment length = lower overhead • Longer fragment sequence # = greater skew tolerance – Estimate of required skew tolerance = 10 us Eff (%)*Frag size 95.564 9132 8416 738 * assuming 14 bit sequence # • 16 or 32 byte fragment size seems reasonable • 14 bit sequence number seems reasonable • Fragments are delimited just like packets on any given interface
- 22. 22 IEEE 802.3 HSSG Fixed vs. variable fragment size Efficiency as a function of packet length for fixed size fragments 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 64 192 320 448 576 704 832 960 1088 1216 1344 1472 8 16 32 64 min=65.6% avg=72.2% max=72.7% min=68.4% avg=82.9% max=84.2% min=61.9% avg=88.6% max=91.4% min=48.5% avg=89.8% max=95.5%
- 23. 23 IEEE 802.3 HSSG Fixed vs. variable fragment size Efficiency as a function of packet length for variable size (64, 32, 16, 8) fragments 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 64 192 320 448 576 704 832 960 1088 1216 1344 1472 min=83.3% avg=94.2% max=95.5% CAIDA avg=94.1% (Cooperative Association for Internet Data Analysis packet size probability distribution courtesy of Glen Kramer)
- 24. 24 IEEE 802.3 HSSG Fragment size summary • Decimation into variable size fragments yields the highest efficiency for all packet sizes • The bulk of a packet is always decimated into the largest size fragments • The tail of a packet is decimated into the optimal combination of fragments • Will need to indicate fragment size in the fragment header
- 25. 25 IEEE 802.3 HSSG APL location MAC Client MAC PCS APL PCSPCSPCS PMAPMAPMAPMA PMDPMDPMDPMD MAC Client MAC PCS APL PCSPCSPCS PMAPMAPMAPMA PMDPMDPMDPMD Existing 10GBASE PHY new
- 26. 26 IEEE 802.3 HSSG APL summary • Packet fragmentation, distribution, collection and reassembly similar to 802.3ah PME aggregation • Accommodates reasonable differential delay • Assumes equal speed links • Assumes point to point, full duplex links • Resilient and scalable
- 27. 27 IEEE 802.3 HSSG APL summary • Ensures ordered delivery • Detects lost or corrupted fragments • Minimal added latency • Fits well with multi-port (quad/octal) PHYs • Line code independent thus providing compatibility with all existing PHYs • Uses existing compatibility interfaces (e.g. XGMII, XAUI)
- 28. 28 IEEE 802.3 HSSG APL control protocol • Will need a protocol to: – detect aggregation link partners – configure aggregations – detect failures – reconfigure • Link Aggregation Control Protocol (LACP) uses Slow Protocol frames for these functions – Existence proof, see 43.4 • Clause 61 PME Aggregation Discovery relies on remote PHY register access – Probably do not want to assume this
- 29. 29 IEEE 802.3 HSSG Configuration Examples
- 30. 30 IEEE 802.3 HSSG Default configuration MAC 10GBASE-R PHY APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY X-point (example implementation for illustrative purposes)
- 31. 31 IEEE 802.3 HSSG 40Gb/s Agg link connected MAC X-point 10GBASE-R PHY APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY PLA (example implementation for illustrative purposes)
- 32. 32 IEEE 802.3 HSSG 20Gb/s Agg link added MAC X-point 10GBASE-R PHY APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY APL 10GBASE-R PHY 10GBASE-R PHY (example implementation for illustrative purposes)
- 33. 33 IEEE 802.3 HSSG Port reconfigured MAC X-point 10GBASE-R PHY APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL MAC APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY APL 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY 10GBASE-R PHY APL 10GBASE-R PHY 10GBASE-R PHY (example implementation for illustrative purposes)
- 34. 34 IEEE 802.3 HSSG Application with various PHYs
- 35. 35 IEEE 802.3 HSSG ……. ……. Multi-wavelength PHY MAC RECONCILIATION APL PMD PMA PCS PMD PMA PCS PMD PMA PCS Multiwavelength PHY Optical Mux / DeMUX MEDIUM MDI …….
- 36. 36 IEEE 802.3 HSSG ……. Multi-wavelength PHY, XGMII MAC RECONCILIATION APL PMD PMA PCS PMD PMA PCS PMD PMA PCS Multiwavelength PHY Optical Mux / DeMUX MEDIUM MDI ……. ……. ……. XGMII
- 37. 37 IEEE 802.3 HSSG ……. Multi-wavelength PHY, XAUI MAC RECONCILIATION APL PMD PMA PCS PMD PMA PCS PMD PMA PCS Multiwavelength PHY Optical Mux / DeMUX MEDIUM MDI ……. XGXSXGXS XGXS XGXSXGXS XGXS ……. ……. ……. ……. XAUI
- 38. 38 IEEE 802.3 HSSG Multi-wavelength PHY, nXAUI MAC RECONCILIATION APL nXGXS nXGXS nXAUI …….PMD PMA PCS PMD PMA PCS PMD PMA PCS Multiwavelength PHY Optical Mux / DeMUX MEDIUM MDI …….
- 39. 39 IEEE 802.3 HSSG Multi-channel PHY MAC RECONCILIATION PMD PMA PCS MEDIUM PMD MDI PMA PCS PMD PMA PCS Multiconductor Cu Cable Ribbon Fiber Multichannel PHY ……. ……. ……. APL
- 40. 40 IEEE 802.3 HSSG ……. Multiple 10GBASE-T PHYs MAC RECONCILIATION APL Auto-Negotiation PMA PCS Auto-Negotiation PMA PCS Auto-Negotiation PMA PCS 10GBASE-T PHY MEDIUM MDI ……. ……. ……. XGMII MEDIUM MDI MEDIUM MDI …….
- 41. 41 IEEE 802.3 HSSG ……. Multiple 10GBASE-CX4 PHYs MAC RECONCILIATION APL PMD PMA 10GBASE-X PCS PMD PMA 10GBASE-X PCS PMD PMA 10GBASE-X PCS 10GBASE-CX4 PHY MEDIUM MDI ……. ……. ……. XGMII MEDIUM MDI MEDIUM MDI …….
- 42. 42 IEEE 802.3 HSSG Summary • There is a broad set of applications for aggregated links – Backplanes, data centers, short and long haul optical • Existing 10GBASE Physical layers can be aggregated • 802.3ad LAG does not fully address the need • 802.3ah PME aggregation is optimized for DSL- based links
- 43. 43 IEEE 802.3 HSSG Summary • Concepts from 802.3ad LAG and 802.3ah PME aggregation can be re-used to define Aggregation at the Physical Layer (APL) • APL can provide a scalable, resilient interface using existing 10G PHYs • APL can also be used with newly defined PHYs that provide higher serial rates