1. Sun SPARC Enterprise
T5440
Rack Servers
Train-the-Trainer Course
Presenter
Sun Microsystems, Inc.
Sun Confidential
CDA Required
1

2. NDA Notification
This presentation contains information related to projects that are currently in planning
and/or development stages. This information represents the current intentions of Sun
Microsystems, Inc. However, all aspects of these projects, including, but not limited to
funding, availability, shipping dates, configurations, capacities, performance, and all
other characteristics are subject to change and/or cancellation without notice.
This material is confidential to Sun Microsystems and should be disclosed to non-
employees only under the terms of an executed confidential-disclosure agreement.
Distribution of this material to Sun Field Representatives should be done through the
GSO CDA process. Under no circumstances should these roadmaps be given directly to
a Sun Field Representative.
The information contained herein is considered privileged, confidential and protected
from disclosure. If you are not the intended recipient or their employee/agent
responsible for its delivery, you are requested to refrain from any duplication or
distribution of this message or its contents, in respect of the rights of the designated
recipient and sender and to delete it from your computer.
Sun Internal and Approved Partners Only 2

3. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC T2 Plus
• Sun SPARC Enterprise T5440 server
> Overview of Features
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 3

4. Industry's Most Highly Threaded Servers
Maximum Threading =
Higher Throughput, Greater Energy & Space Efficiency
T5440
Up to 256 Threads
T5140/T5240
Up to 128 Threads
T5120/T5220
Up to 64 Threads
SB6320
US T2 blade
Sun Internal and Approved Partners Only 4

5. UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHz
TrueSystem On a Chip • Up to 64 threads per CPU
• Up to 16 FB-DIMMs, 4 memory
controllers
FB DIMM FB DIMM FB DIMM FB DIMM
> Up to 64GB memory (4GB DIMMs)
> 2.5x memory BW of UltraSPARC T1
FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point units /
core, 1 per core
MCU MCU MCU MCU • Dual 10Gbit Ethernet and PCI-E
integrated onto chip
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
• 4MB L2 (8 banks) 16 way associative
Full Cross Bar
• Enhanced MAU/Security co-processor
C0 C1
FPU FPU
C2
FPU
C3
FPU
C4
FPU
C5
FPU
C6
FPU
C7
FPU
per core
MAU MAU MAU MAU MAU MAU MAU MAU > DES, 3DES, AES, RC4, SHA1,
DMA Sub-System SHA256, MD5, RSA to 2048 key,
ECC,CRC32
NIU Sys I/F
• Advanced Power saving features
(E-net+) Buffer Switch Core PCI-E
• 65nm process technology
Power ~95W
<1.5W/thread x8 @2.5GHz
2x 10GE Ethernet 4GBytes/s bi-directional
Sun Internal and Approved Partners Only 5

6. UltraSPARC T2 Processor
SPC SPARCv9 CMT Core
L2 Data L2 Data
Bank 0 Bank 4 CCX Core-Cache Crossbar
L2B0 SPARC SPARC SPARC SPARC L2B4 L2$ Level 2 Cache Bank
L2 Data
Core 0 Core 1 Core 5 Core 4
L2 Data CLC Coherence & Link Controller
Bank 1 Bank 5
MCU Memory Control Unit
L2B1 L2B5
SIU System Interface Unit
L2 L2 L2 L2
MCU0 MCU2 NCU Non-cacheable Unit
NCU
TAG0 TAG1 TAG5 TAG4 FSR
MAC Media Access Controller
EFU
MCU1 MCU3
DMU Data Management Unit
SIO
SII
L2B3 CCX L2B7
FSR CCU PEU PCI Express Unit
L2 Data L2 Data
Bank 3 Bank 7
L2 L2 L2 L2
L2B2 L2B6
TAG2 TAG3 TAG7 TAG6
L2 Data L2 Data
Bank 2 Bank 6
SPARC SPARC SPARC SPARC
DMU
Core 2 Core 3 Core 7 Core 6 RDP TDS
PEU
RTX
PSR ESR FSR MAC
For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4
Sun Internal and Approved Partners Only 6

7. UltraSPARC T2 Plus: • Per Socket
Multi-Socket US T2 > Up to 8 cores @1.2GHz or
Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total
1.4GHz
4 x Coherency links give Snoop bandwidth of 51.2GB/s > Up to 64 Threads per socket
4x
Coherency > 4MB L2$ 8 banks x 16 Way SA
Links
> Up to 8 FPU's, 1 per core
FB-DIMM FB-DIMM FB-DIMM FB-DIMM
> Up to 8 Crypto cores, 1 per core
FB-DIMM FB-DIMM FB-DIMM FB-DIMM > DES, 3DES, AES, RC4, SHA1, SHA256,
MD5, RSA to 2048 key, ECC,CRC32
MCU MCU > 2 memory Controllers, 16 DIMMS
CU CU CU CU > 25Read+13WriteGB/s Memory BW
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ > One x8 PCIe interface
Full Cross Bar • Per system
CO C1 C2 C3 C4 C5 C6 C7 > 2 sockets glue-less
FPU
MAU
FPU
MAU
FPU
MAU
FPU
MAU
FPU
MAU
FPU
MAU
FPU
MAU
FPU
MAU
> 4 sockets using UST2 Plus XBR
> 16 or 32 PCIe lanes on 2 or 4
System I/F PCI-Express Socket systems
• 65nm process technology
SSI x8 @2.5GHz
Bus Power ~105W 4GBytes/s
<1.5W/thread bi-directional
Sun Internal and Approved Partners Only 7

8. UltraSPARC T2 Plus Processor
SPC SPARCv9 CMT Core
CCX Core-Cache Crossbar
L2$ Level 2 Cache Bank
CLC Coherence & Link Controller
MCU Memory Control Unit
SIU System Interface Unit
NCU Non-cacheable Unit
NCX Non-cacheable Crossbar
DMU Data Management Unit
PEU PCI Express Unit
Sun Internal and Approved Partners Only 8

9. CMT Processor Comparison
UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Technology 90nm 65nm 65nm
# of Cores 8 8 8
# of Threads 32 64 64
# of Sockets 1 1 1–4
Core Frequency (Ghz) 1 – 1.4 1.2 – 1.4 1.2 – 1.4
Execution pipelines per core 1 2 2
Instructions per cycle per pipe 1 1 1
FPUs per chip 1 8 8
I$ size (per core) 16KB 16KB 16KB
D$ size (per core) 8KB 8KB 8KB
L2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-Way
Memory Type DDR2 FBDIMM-1 FBDIMM-1
Memory Bandwidth (GB/s) 21.3 42READ + 21WRITE 25READ + 13WRITE
Coherence Bandwdth (GB/s) - - 4 x (6.4READ +6.4WRITE)
IO Bus JBUS x8 PCIe x8 PCIe
IO Bandwidth (GB/s) 2.5 2+2 2+2
On-chip Networking - 2 x 10GE -
Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5
Power (Watts, Watts/thread) 72, 2.25 95, 1.5 105, 1.6 *
Sun Internal and Approved Partners Only 9

10. UltraSPARC T2 Plus 2-Socket System
Coherency Links
Dual Channel Dual Channel Dual Channel Dual Channel
FBDIMM FBDIMM FBDIMM FBDIMM
Memory Controller Memory Controller Memory Controller Memory Controller
Coherence Coherence Coherence Coherence Coherence Coherence Coherence Coherence
Unit Unit Unit Unit Unit Unit Unit Unit
UltraSPARC T2 Plus Cores, Crossbar, L2$ UltraSPARC T2 Plus Cores, Crossbar, L2$
(8 cores, 64 threads, 4MB L2$) (8 cores, 64 threads, 4MB L2$)
PCI-Express NCU, DMU NCX PCI-Express NCU, DMU NCX
System IO (Network, Disk, etc.)
Sun Internal and Approved Partners Only 10

11. UltraSPARC T2 Plus 4-Socket System
6.4GB/s 6.4GB/s
Dual Channel
FBDIMM
Dual Channel
US T2 Plus US T2 Plus
FBDIMM
PCI-Express PCI-Express
8 Cores 6.4GB/s 6.4GB/s 8 Cores
6.4GB/s 6.4GB/s
64 Threads Coherence 64 Threads
Dual Channel
4MB L2$ 4MB L2$
Coherence(4)
Hubs
FBDIMM
Dual Channel
FBDIMM
Coherence
Coherence
UltraSPARC
Hubs (4)
Hubs (4)
HubsPlus
T2 (4)
6.4GB/s XBR 6.4GB/s
44 ports/hub
ports/hub
6.4GB/s 4 ports/hub 6.4GB/s
Dual Channel
inin each dir
each dir
FBDIMM
Dual Channel
in each dir
FBDIMM
PCI-Express US T2 Plus
VF 4 ports/hub US T2 Plus
PCI-Express
6.4GB/s per port
6.4GB/s 6.4GB/s
8 Cores
8 Cores in each dir 6.4GB/s 8 Cores
6.4GB/s
64 Threads
8 Threads/Core 64 Threads
Dual Channel
FBDIMM
4MB L2$ 4MB L2$
Dual Channel
4MB L2$ FBDIMM
6.4GB/s 6.4GB/s
Sun Internal and Approved Partners Only 11

12. US T2 Plus XBR Coherency Hub Chips...
• 4 XBR chips in a system • XBR routes non-cacheable PIOs to the
• Each operates in its own appropriate IO device
Coherency Plane(CP) > Forwarded to the Zambezi that owns the
> A region of addressable space, unique address space for processing
to that plane • DMA reads and writes are done
> Each XBR only tracks addresses in its
own space through cachable memory space just
> Serializes accesses to coherent loads, like processor initiated memory
stores and writebacks (TSO model) accesses
> 40 bit Physical Address used
• CP links are based on an FB-DIMM
> VF Snoop links to pass transactions
physical interface, but a Sun protocol
• Cacheable transactions > 6.4GBps/Direction/Port
– Most Memory transactions > 51.2GB/s bandwidth total
• Non-Cacheable transactions > CP links are tolerant to a single lane failure
– IO, Non-cachable memory, (14 lanes per link)
Interrupts etc. Coherency Plane > CRC and replay features in the protocol
Mem Interleave by
Designator allow no-stop recovery
Coherency Plane
Cachable/
Non-Cachable
Coherency plane
Designator
(512B/1GB) > Memory structures are Parity protected
39 31 30 40 Bit Physical Address 10 9 8 7 0
Sun Internal and Approved Partners Only 12

13. UltraSPARC T2/Plus CPU Core Block Diagram
• IFU – Instruction Fetch Unit
> 16 KB I$, 32B lines, 8-way SA
TLU IFU > 64-entry fully-associative ITLB
• EXU0/1 – Integer Execution Units
> 4 threads share each unit
> Executes one integer
instruction/cycle
EXU0 EXU1
• LSU – Load/Store Unit
> 8KB D$, 16B lines, 4-way SA
MMU/ > 128-entry fully-associative DTLB
HWTW • FGU – Floating/Graphics Unit
• SPU – Stream Processing Unit
> Cryptographic acceleration
SPU FGU LSU
• TLU – Trap Logic Unit
> Updates machine state, handles
exceptions and interrupts
• MMU – Memory Management Unit
> Hardware tablewalk (HWTW)
Gasket
– TLB miss handled by hardware
> 8KB, 64KB, 4MB, 256MB pages
xbar/L2
Sun Internal and Approved Partners Only 13

14. Floating Point Unit
• Each Core has its own Floating Point Unit. This removes
completely the 2% FP restriction of T1.
• Only a 6 cycle penalty to the FPU
> T1 required upwards of 28 cycles
• Fully-pipelined (except divide/sqrt)
Data
> Divide/sqrt in parallel with add or multiply operations of other
threads
• Full VIS 2.0 implementation in hardware, no emulation
• FPU performs integer multiply, divide, population count
• Multiplier incorporates enhancements for offloading modular
arithmetic operations to the Crypto Unit
●
● Sun Internal and Approved Partners Only 14

15. UltraSPARC T2 Plus Crypto Performance
Absolute maximums, measured by engineering using micro-benchmarks
Application throughputs will be lower than these numbers
Iterations/ Block Cycles/
Function Comment Block (bytes) Block GB/S
DES 2 rounds/cycle 16 rounds 8 8/9+ 12.0
3DES 2 rounds/cycle 48 rounds 8 25/24 4.0
RC4 3 overlapped cycles 1 1 37683 12.0
Data into 44 words
with UltraSPARC T2 Plus
AES-128K 4 words - - 32 -
AES-192K 6 words into 52 words - - 37 -
New Ciphers
AES-256K 8 words into 60 words - - 41 -
AES-128 2 column over two cycles 10 rounds 16 30/30 6.36
AES-192 2 column over two cycles 12 rounds 16 36/36 5.28
AES-256 2 column over two cycles 14 rounds 16 42/42 4.56
SHA-1 2 cycles per step 80 steps 64 160/160 4.80
SHA-256 2 cycles per step 64 steps 64 128/128 6.00
MD5 2 cycles per step 64 steps 64 128/128 6.00
CRC32c 32 bits per cycle 1 4 1 48.00
Wire Speed for UltraSPARC T2 Plus is around 5 GB/S
16,000 new SSL sessions/sec using 1024 Asymmetric Key Exchange
Export restrictions apply to some of the ciphers, requires installation of the Crypto Enabling
patch to use. Purchase a CD which contains the patch.
Affects IPsec related ciphers, SSL typeSun Internal and Approved Partners Only
acceleration is unaffected and does not need enabling. 15

16. FB-DIMM Memory Interface
Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Require Pins 1000 440 208
Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socket
Up to 64x DIMM slots Up to 32x DIMM slots/socket
Memory Capacity up to 16x DIMM slots
T5x20 uses 16 DIMM slots T5x40 uses 8/16 DIMM slots
Data
Memory Controllers 4 4 2
Memory Latency 96ns 126ns 126ns
DIMM speeds 400MHz 667MHz 667MHz
DIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM
●
● Sun Internal and Approved Partners Only 16

17. 10/100/1000/10GbE Network Interfaces
• Uses the Neptune Chip on the Motherboard which provides
> Two RJ-45 ports dedicated for 10/100/1000
> Two RJ-45 ports capable of 10/100/1000
– Fitting a XAUI adapter disables the one of these ports per XAUI
• Same 10 GbE XAUI cards as T5x20 server
• 16 Rx and 24 Tx DMA channels for virtualization
– Integrated NIU in T2 has 16 Tx channels
Data
– Raw UDP Tx performance on Neptune > T2 NIU
– T2 processor network typically outperforms Neptune on most network
benchmarks
• Packet classification
> Rx data can be HW classified at layer 2 ,3 and 4
– Put into the separate Rx DMA buffers by classification rules
• Same 'nxge' device driver as T2 NIU.
• Performance estimates, testing/tuning ongoing
– 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle
●
– 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports)
● Sun Internal and Approved Partners Only 17

18. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 18

19. SPARC Enterprise Compute Density
• 4RU chassis
T5440 • 4-sockets – UltraSPARC T2 Plus
Quad Socket US T2 Plus > Up to 32 cores @ 1.2GHz or 1.4Ghz
> Up to 256 threads
• Up to 64 FBDIMM slots
> 512 GB of memory (using 8GB DIMMS)
Reliable
• 4 Hot-Plug 2.5” SAS disks (Raid 0,1)
• DVD-RW
• 4 x Redundant(2+2), hot-swappable PSUs
●
Up to 256 threads / 512GB memory • 4 x Redundant, hot-swappable Fans
• 2N PSU Redundancy, 1900 watts typical
in a robust, redundant chassis consumption
●
Taking CMT to the Mid-Range Expandable
• 4 x Processor+Memory card pairs
●
LDOMs consolidation dream box • 8 x8 PCIe expansion slots (2 x XAUI)
• 4 Gigabit Ethernet (or up to 2 x 10GbE and
2 x1GbE)
• 4 x USB 2.0
Sun Internal and Approved Partners Only 19

20. T5440 Standard Configurations (PTO)
Part Number Description 4RU
2 x 1.2GHz, 8 Core CPU, 32GB (16 X 2GB), 2x 146GB SAS 2.5”disks, 4x
SEVPCJF1Z
1120W PSU, DVD-RW, 2 x Memory Expansion Boards
4 x 1.2GHz, 8 Core CPU, 64GB (32 X 2GB), 2x 146GB SAS 2.5”disks, 4x
SEVPGLF1Z
1120W PSU, DVD-RW, 4 x Memory Expansion Boards
4 x 1.4GHz, 8 Core CPU, 128GB (32 X 4GB), 2x 146GB SAS 2.5”disks,
SEVPHSF1Z
4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards
●
All configs feature...
●
4 x 1GbE interface,
●
1 x DVD-RW slimline drive
●
8 x PCIe 8 lane slots (2 shared XAUI slots), 2 x16 mechanical
●
4 x USB 2.0 slots (2 front/2 back)
●
1 x DB9 Serial port
● 4 x Redundant, Hot-plug PSUs
●
4 x Redundant, hot-plug fans
●
2 x 146GB 2.5” SAS drives (can accommodate up to 4 drives)
●
RAID 0,1 from onboard LSI 1068E controller
●
1 Year On-Site Warranty
● Rack mount kit with cable management arm
●
4 x Power Supply cables (country specific)
Sun Internal and Approved Partners Only 20

21. Sun Fire T5240/T5440 Differences
Feature SPARC Enterprise T5240 SPARC Enterprise T5440
Form Factor 2U, 28” deep 4U, 24.5” deep
2x UltraSPARC T2 Plus Up to 4 x UltraSPARC T2 Plus
CPU
1.2/1.4 GHz 64-128 threads 1.2/1.4 GHz 128-256 threads
FB-DIMM 128GB MAX FB-DIMM, 512GB MAX
Memory 32x slots 64x Slots
4x GbE or 4x GbE or
Network 3 x GbE + 1 x 10GbE or 3 x GbE + 1 x 10GbE or
2 x GbE + 2 x 10GbE 2 x GbE + 2 x 10GbE
Internal Storage Up to 16 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap
Removable Media 1x DVD-RW 1x DVD-RW
Serial 1x RS-232, 4x USB 1x RS-232, 4x USB
PCI Express slots 6x (low profile) 8x (low profile)
2x 1100W AC/2 x1200Watt DC 4 x 1120W AC
Power Supply
Redundant/Hot-Swap Redundant/Hot-Swap
Fans 5x Redundant/Hot-Swap 4 x Redundant Hot-Swap
Sun Internal and Approved Partners Only 21

22. Front Panel
Service Required LED
Power/OK LED
Locator LED Power Button
/Button
Front Fan Fault LED System Label
PSU Fault LED
System Overheat Fault LED
Serial Number label
DISK 2
DISK 3
DVD-RW
DISK 0
DISK 1
Dual USB 2.0 ports
Sun Internal and Approved Partners Only 22

23. Rear Panel
PCI Card Securing
Latches (internal)
PSU
Status
LED's
PSU 3
PSU 2
PSU 1
PCI 0
PCI 1
PCI 2
PCI 3
PCI 4
PCI 5
PCI 6
PCI 7
PCI 0
PCI 2
PCI 3
PCI 5
PCI 6
PCI 7
PSU 0
PCI 0 = PCIe x8, mechanical x8 Quad Gigabit
PCI 1 = PCIe x8, mechanical x16 SP Ethernet Ports
Serial
PCI 2 = PCIe x8, mechanical x8 Port SP NET 2 NET 3
Network
PCI 3 = PCIe x8, mechanical x8 POSIX DB-9 Chassis Port
Serial Port Status LED's Dual
PCI 4 = PCIe x8, mechanical x8, or XAUI 0 Card and Locator Button USB 2.0 NET 0 NET 1
Ports
PCI 5 = PCIe x8, mechanical x8, or XAUI 1 Card
XAUI 1 XAUI 0
PCI 6 = PCIe x8, mechanical x16
PCI 7 = PCIe x8, mechanical x8
Sun Internal and Approved Partners Only 23

24. Sun Fire T5440 Block Diagram Mem CPU CPU Mem Mem CPU CPU Mem CPU Module =
Module Module Module Module Module Module Module Module - 1 x VF CPU
0 0 2 2 1 1 3 3 - 4 x FB-DIMM Slots
Mem Module =
- 12 x FB-DIMM Slots
Plus
Plus
Plus
Plus
T2
T2
T2
T2
x4
VRM
VRM
VRM
VRM
VRM
VRM
VRM
VRM
SAS
LSI1068e
x8 SAS
4x FB Links SSI SSI 4x FB Links 4x FB Links SSI SSI 4x FB Links
PCIe x8 PCIe x8
PCIe x8 CL0 CL0 PCIe x8
US T2 XBR LPC
CL0 0 CL0
x8 CL1 CL1 x8 x8
PLX3 PLX2 x8 US T2 XBR LPC PLX1 PLX0
PEX8548 PEX8548 CL1 CL1 PEX8548 PEX8548
1
x8 x1 x8
CL2 CL2
x8 x8 US T2 XBR
Zambezi x8 x8
x8 CL2 2 LPC CL2 x8 x8
PCIe-PCI
x8 CL3 CL3 XAUI 1
XAUI 0 US T2 XBR LPC Neptune
CL3 3 CL3
USB
host
x8
QUAD GbE
USB Hub USB-IDE 4x SSI PHY
LPC FPGA ILOM BCM5466R
PCI Card
BACK FRONT BACK 0 1 2 3
panel panel panel
NET SER DB-9
0 1 2 3 4 USB 2.0 USB 2.0 DVD
Sun Internal and Approved Partners Only MGT MGT Quad 10/100/1000 5 6 7
24

25. Internal View with Cards installed(T5440) System Configuration
4 Fans are required
Card (SCC)EEPROM
MEM 3 PCIe 7 x8(m)/x8(e)
PCIe 6 x16(m)/x8(e)
MB/FT0
MB/FT2
MB/FT2 CPU 3 XAUI/PCIe 5
x8(m)/x8(e)
CPU 1
MEM 1 Service Processor
Daughter Card
MEM 2
XAUI/PCIe 4
x8(m)/x8(e)
CPU 2
MB/FT3
MB/FT1
MB/FT3
PCIe 3 x8(m)/x8(e)
PCIe 2 x8(m)/x8(e)
CPU 0 PCIe 1 x16(m)/x8(e)
PCIe 0 x8(m)/x8(e)
MEM 0
4 x SAS
Disk Chassis
Power
Distribution Power
Disk Board
Chassis Supplies
Power BUS Bars
RED = +12Volts
BLUE = Ground
Chassis Intrusion Switch
Extension to top cover from MBD
Sun Internal and Approved Partners Only 25

26. Internal View, cards removed(T5440) System Configuration
US T2 Plus XBR Coherency Plane Chips Card (SCC)EEPROM
PCIe 7 x8(m)/x8(e)
MEM 3
PCIe 6 x16(m)/x8(e)
MB/FT0
MB/FT2
MB/FT2 CPU 3 XAUI 1/PCIe 5
x8(m)/x8(e)
CPU 1
MEM 1 Service Processor
Daughter Card
MEM 2
XAUI 0/PCIe 4
x8(m)/x8(e)
CPU 2
MB/FT3
MB/FT1
MB/FT3
PCIe 3 x8(m)/x8(e)
PCIe 2 x8(m)/x8(e)
CPU 0 PCIe 1 x16(m)/x8(e)
PCIe 0 x8(m)/x8(e)
MEM 0
Power
Distribution Power
Disk Board
Chassis Supplies
Chassis SeperatorFence
Sun Internal and Approved Partners Only 26

27. Internal View, Motherboard only(T5440) Neptune Quad
DC-DC Converter Modules GbE/10GbE NIC
MEM 3
CPU 3
MB/FT0
MB/FT2
MB/FT2
Quad GbE RJ-45
Ports
CPU 1
NET MGT port
Dual USB 2.0 Port
MEM 1 System Status LEDs
SER MGT port
MEM 2 DB-9 Serial Port
CPU 2 Service Processor
Card Connector
MB/FT3
MB/FT1
CPU 0
MEM 0
PLX PCIe Switch
Chips
'Southbridge'
Peripheral Chip
Fan connectors, with LSI1068E Disk Controller
speed control logic
Sun Internal and Approved Partners Only 27

28. Motherboard and CPU/MEM Card layout
MEM 3
CPU 3
CPU 1
MEM 1
MEM 2
CPU 2
CPU 0
MEM 0
Motherboard
CPU/Mem Board Memory Board
Sun Internal and Approved Partners Only 28

29. Memory Configuration Rules
• 1 or 4 CPU cards in a system
> Each CPU card has an associated
Memory Card Card/Chassis
> Unpopulated Cards must be filled Top
with spacers
> Unpopulated DIMMs must be filled UltraSPARC T2
Plus
with spacers VRM
CPU
• All DIMMS on a CPU & Memory VRM
card pair must be the same size BR1/CH0/DO
BR1/CH1/DO BANK 0
> 2GB, 4GB or 8GB only (no BR0/CH0/DO
VRM Install ALL DIMMs
1GB) BR0/CH1/DO
> May support Different DIMM
sizes on different pairs... BR1/CH1/D2
BR1/CH1/D3 BANK 3
• 3 Configs... BR1/CH0/D2
BR1/CH0/D3
Install ALL DIMMs
> Fill Bank 0 BR0/CH1/D2
BANK 2
> Fill Banks 0,1 VRM BR0/CH1/D3
Install ALL DIMMs
BR0/CH0/D2
> Fill Banks 0,1,2,3 BR0/CH0/D3
BR0/CH1/D1
VRM BR0/CH0/D1 BANK 1
• All standard configs provide
Install ALL DIMMs
BR1/CH1/D1
BR1/CH0/D1
exactly 8 free DIMM slots
simultaneously Sun Internal and Approved Partners Only 29

30. Network Ports and Options
XAUI 0
XAUI 1
Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1
Driver nxge0 nxge1 nxge2 nxge3 nxge0* nxge1*
PCI Device PCI@500/0/c/0,0 PCI@500/0/c/0,1 PCI@500/0/c/0,2 PCI@500/0/c/0,3 PCI@500/0/c/0,0 PCI@500/0/c/0,1
NOTES
All Ethernet ports provided by a single Neptune Chip on the motherboard NET 2 NET 3
NET 0 and NET 1 are disabled by XAUI 1 and XAUI0,if installed, to take advantage of 10GbE networking
* If a XAUI card is installed, the matching RJ-45 port is disabled (NET 0 and NET 1)
XAUI cards take one of three current Fibre 10GbE transceivers
Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM) NET 0 NET 1
Copper versions of the XAUI card are expected Post-RR
XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive
with PCI cards in the same location
Sun Internal and Approved Partners Only 30

31. 10GbE XAUI Option Cards
• Interface to the Neptune chip to upgrade two of its ports to 1GbE
• Cards each consume a PCI-E slot if installed
• Attach to dedicated XAUI sockets on special risers
• 10Gbe XAUI Card
– SESY7XA1Z
> Requires Fibre Tranceivers
– 10GbE Short Reach(SR)
– SESY7XT1Z
– 10GbE Long Reach (LR)
– SESY7XT2Z
– 10GbE Long Reach multimode (LRM)
– part No. TBD
• Copper connector XAUI versions to follow post-RR
Sun Internal and Approved Partners Only 31

32. PCIe Slots
Built In
• No Riser Cards are used, slots are directly on the motherboard
• Low Profile PCI-Express cards only
• Each Slot is capable of 25Watts power
• Total power capacity limited to 140Watts
> Systems design point was to accommodate 8 x 17.5Watt Atlas(Quad GbE using
Neptune) card
• Each Slot has x8 PCIe electrical capability and at least x8 mechanical connectors
• Slots 1 and 6 have x16 mechanical connectors
• XAUI cards in slots 4 and 5 replace PCIe cards in those slots
Sun Internal and Approved Partners Only 32

33. PCIe External IO Expansion Unit
aka Expansion Chassis, I/O Box
• Up to 2 x PCI Expansion Chassis are supported
> P/N : SENX8BE1Z
> 4RU, 19” rack compatible , 28” deep
> 1 Host card supported per US T2 Plus
> Each chassis is managed by host SP
• Only support PCIe card IO boats ( PCI-X ?)
> Each 'boat' requires an Link Card in the host
(ensign) and a link Card in the boat (captain)
with special card between them
> IO 'boat' supports 6 x PCIe x8 full-length, hot-
pluggable PCIe cards
• Chassis default configs contain
> 2 x Hot-plug redundant 600Watt PSU/FAN
FRU's
> 1 x PCIe 'boat', 1 link card pair(host-chassis)
and cable
> 6 x PCIe card carriers for x8 PCIe Cards
> Customers can add a further PCIe 'boat'
– P/N : SENX8TE1Z and more
Sun Internal and Approved Partners Only 33

34. Disk Drives
• Disk drives are all 2.5” form factor
• Original Galaxy NEMO bracket (not Marlin Bracket)
• Support SAS only (Can mix SAS/SATA after qual)
• SAS Drives
– 73GB @ 10K RPM
– 146GB @ 10K RPM
– 73GB @ 15K RPM
– SSD (post-RR)
• Same disks as T2x00 systems
Disk LED's
Ready to Remove
Fault
Status
Sun Internal and Approved Partners Only 34

35. Disk Controller Options
• T5440 has embedded LSI 1068E on the motherboard
– JBOD up to 4 disks
– Controls internal disks only
– Unused 4 ports are not used
– RAID 0 (striping) and RAID 1(mirroring)
– Hot-plug supported
– Hot-swap in non-RAID configs is done with 'cfgadm'
• SGXPCIESAS-R-INT-Z NOT supported
> Not available for T5440 due to chassis design and only 4 disk capacity
• SGXPCIESAS-R-EXT-Z Supported
– For external disk arrays only (no internal ports)
– 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available)
– Adaptec Card and Firmware with Intel SunRiseLake chipset
– Battery Backed Write Cache (BBWC) of 256MB, at least 72 hours capacity, much
longer when new
Sun Internal and Approved Partners Only 35

36. Fans and Cooling
• Front to Back cooling
• Two rows of fans provide redundancy
> PSU fans provide sufficient redundancy
for Disk cooling Back Front
• Requires 4 Fans
Disks
PSU's
– Main System Cooling
MEM0
– MB/FT0
MB/FT2
MB/FT3
MB/FT3
CPU0
– MB/FT1 CPU2
Motherboard MEM2
– MB/FT2 MEM1
MB/FT1
MB/FT0
MB/FT1
– MB/FT3 CPU1
CPU3
– PSU integrated Fan Cooling flow MEM3
– Cools disk drives and PSU's
– PSUFAN0
– PSUFAN1
– PSUFAN2
– PSUFAN3
Sun Internal and Approved Partners Only 36

37. Fans modules and boards
Fan Fault
Amber LED
Lit if fan faulty
• 4 Fans Required
> 60 Seconds grace if FAN removed for service
• Fans are speed controlled by the Service Processor
• Speeds are independantly monitored by SP
• Speeds tuned to minimise vibrations
> Built in vibration sensor
• Speeds chosen to minimise harmonic resonance
with disk drives and other fans.
• Each fan has an Amber FAULT LED
– if lit the fan module is faulty
Sun Internal and Approved Partners Only 37

38. Power Supplies
• PSU's are high efficiency
• Output 12V (and 3.3v standby)
PSU Fault
• Most DC-DC conversion done in MBD LED
• Integrated Fans
• 4 PSU's
> 2 + 2 Redundancy (Grid Redundancy)
> Rated at 1120Watts each.
> BEWARE...
– Same form factor as SunFire X4600 PSU's ... DO NOT MIX
Sun Internal and Approved Partners Only 38

39. Rack Mounting Options
• Two Rack mounting options are available..
• Standard Rack Mount (4 post)
> P/N : SEVY9RK1Z
> Same as Sun Fire X4600 server
> Standard configs contain this kit
• Tool-less Rack Kit (4 post)
> P/N : SEVY9RK2Z
> Optional kit, purchased seperately
> Snaps into Sun and most 3rd party Racks
> Requires no tools to fit or remove
> Will replace tool-required version once available
Sun Internal and Approved Partners Only 39

40. Solaris for Sun SPARC Enterprise T5x40
• T5x40 servers will require Solaris10 5/08 (update 5) with
patches
> A Patch CD is supplied in the box
– Likely to be out of date, so customers should check SunSolve...
• This incorporates several specific UltraSPARC T2/T2 Plus
optimizations and for CMT servers in general
• Also includes some specific T5x40 optimisations
> UltraSPARC T2 Plus MPO optimisations
> nxge driver tweaks for 10GbE devices
> MMU DMA relaxed scheduling
> LDOMs pre-install upgrade to version 1.0.3 (US T2 Plus support)
Sun Internal and Approved Partners Only 40

41. Solaris FMA
• Now aware of CPU Core offlining
• PCIe Fabric diagnostics improved
• Crypto Unit offlining
• Neptune awareness (though XAUI's not diagnosed)
• POST memory CE tolerance improved
• FRU identity info improved in fault reports (part/ser #)
• Diagnostic Engines (DE)
– Existing DE have been extended for T5x40
– New Eversholt DE used for HostBridge, PCIe diagnostics
• Limitations...
– Can't isolate Uncorrectable Errors to a single DIMM
– PCIe riser cards not in topology
– XAUI card and XFP not in the topology
– SouthBridge chip is not diagnosed at this time (DVD, USB,...)
– IO Expansion box is not diagnosed via FMA
Sun Internal and Approved Partners Only 41

42. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> System Architecture and Tours
> Extra Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 3.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 42

43. ILOM Features at a Glance
• Hardware Control, Inventory, Monitoring
• Host Configuration and Administration
• SP Configuration and Administration
• Logging, Auditing and Alerting
• Authentication :
– LDAP, RADIUS, Active Directory, Local Users
• Services :
– SSH, HTTP(S), NTP, Syslog, SMTP
• Out-of-Band
– Via Serial port, or dedicated Ethernet port
• In-Band
– Via Host OS, firmware upload, Explorer, Syslog
Sun Internal and Approved Partners Only 43

44. ILOM – Industry Standard Manageability
• ILOM enables Sun systems to be easily integrated into third
party management infrastructures
• Key functionality
> IPMI 2.0
> SNMP V1, V2c, V3
> SSH 2.0
> LDAP, RADIUS & MS Active Directory authentication
> Browser UI, SMASH DMTF CLP + ALOM style CLI
• NOTE : No RKVMS functionality like Galaxy ILOM
> No Java plugin required
• Browsers tested include, these base versions an later
releases
– Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox
1.0 and 1.5 on W2K server
Sun Internal and Approved Partners Only 44

45. ILOM – Industry Standard Manageability
• Identical to ILOM 2.0 for T5x20
• Default user is 'root' with password 'changeme'
> Use for ILOM WEB GUI access and SSH access and IPMI
• The ALOM 'admin' user is not defined by default, create this
user with an ALOM CLI
• Lost 'root' password and ILOM config is reset using jumpers
> T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2)
> T5440 - SP Daughter card Jumper J1127 (short pins 1-2)
• Note... 'sunservice' user shares the password of the 'root'
user ONLY FOR SERVICE ACCESS*
• Passwords must be 8 to 16 chars (used to be max 8)
• Max of 10 users allowed (used to be 16)
Sun Internal and Approved Partners Only 45

46. ILOM Common Tasks
• Setting up an ALOM 'admin' user from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Warning: password is set to factory default.
->create /SP/users/admin role=Administrator cli_mode=alom
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/admin2
-> exit
SUNSP00144F4641BD login: admin
Password: ********
Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
sc>
Sun Internal and Approved Partners Only 46

47. ILOM Common Tasks
• Setting up ILOM IP configuration from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Warning: password is set to factory default.
-> set /SP/network state=enabled
Set ’state’ to ’enabled’
-> set /SP/network pendingipaddress=10.8.31.163
Set ’pendingipaddress’ to ’10.8.31.163’
-> set /SP/network pendingipdiscovery=static
Set ’pendingipdiscovery’ to ’static’
-> set /SP/network pendingipnetmask=255.255.252.0
Set ’pendingipnetmask’ to ’255.255.252.0’ Static IP
-> set /SP/network pendingipgateway=10.8.31.248 configuration
Set ’pendingipgateway’ to ’10.8.31.248’
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’
->
-> set/SP/pendingipdiscovery=dhcp
Set ’pendingipdiscovery’ to ’dhcp’
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’
-> Dynamic IP
(DHCP)
configuration
Sun Internal and Approved Partners Only 47

48. ILOM IPMI functionality
• IPMI is a widely used management protocol for x86 servers
• ... But its new for SPARC boxes...
• Download the required tools from
> The Solaris Supplemental CD (an older binary package)
> Latest is available at http://ipmitool.sourceforge.net
– Binary packages (usually older versions) are available for Solaris x86 and SPARC, as
well as Linux, Windows, Mac...etc
> There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC
now too)
– http://www.sun.com/blueprints/0107/820-1011.pdf
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr
/MB/V_+1V0_VDD | 1.01 Volts | ok
/MB/V_+3V3_STBY | 3.36 Volts | ok Get the Sensor
/MB/V_+3V3_MAIN | 3.36 Volts | ok Data
/MB/V_+5V0_VCC | 4.97 Volts | ok Repository
/MB/V_+12V0_MAIN | 12.16 Volts | ok
..
..
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set
/SYS/LOCATE on
/SYS/LOCATE | ON Set the System
-bash-ventus2$ Locator LED
flashing
Sun Internal and Approved Partners Only 48

49. ILOM Common Tasks
• Logging into the ILOM Web GUI
• Point Web Browser at the IP address of
the ILOM
> Will automatically redirect to a
secure HTTPS port
> Best to disable any Web Proxies for
best performance
> Make sure port 80, 443 are open on
any firewall
• Log in as any defined user, default
Administration user is 'root'
Sun Internal and Approved Partners Only 49

50. ILOM/ALOM Common Tasks
• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address
configurations for ILOM
• Configure additional ILOM/ALOM users
• Access Sensor Readings
• Control System configurations
• Carry out firmware updates.
• etc....
Sun Internal and Approved Partners Only 50

51. ALOM 2.0 vs. 1.4 CLI Differences
• Default CLI appearance is ILOM
– User is 'root'
– Must create an 'admin' Administrator user with the ALOM CLI
• Flashupdate in new ALOM expects a TFTP server not an FTP server
• ALOM IP address changes now active only after running ...
– 'setsc netsc_commit true'
– Previously changes were active only after SC reset
– DHCP is enabled by default
• View firmware versions with
– 'showhost version' sc> showhost version
Sun System Firmware 7.0.0_build_13 2007/08/10 23:44
• Only Admin and Operator roles
Host flash versions:
Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54
OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25
are defined
Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD***
2007/08/10 13:51
Sun Internal and Approved Partners Only 51

52. ALOM backward Compatibilty CLI
• Uses ALOM-CMT 1.2 for a base feature set
• Includes ALOM-CMT 1.4 features...
> break -d
> Host Watchdog Timer
> Bootmode LDOMs awareness
> Forward SP events to Hot/Solaris syslog via inband channel
Sun Internal and Approved Partners Only 52

53. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> Comparison to SunFire T1000/T2000/T5x20
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 53

54. Logical Domains 1.0.3 Features
• UltraSPARC T2 Plus Platform Support
> 128 domains supported at RR
– 256 domains work on T5440, may be qualified later, requires Solaris fixes
• Feature Set very similar to 1.0.2
• I/O domain can reboot while the other domains keep running
> Dependent domains would wait for I/O transactions to complete
• LDoms SNMP MIB
> Modeled after industry standard DMTF CIM (Common Information Model)
> Running on control domain to enable remote SNMP monitoring
• Domain minimization support
> Minimal/reduced Solaris installation
> Base minimal packages plus optional ones for guest, control, I/O or service
domains
Sun Internal and Approved Partners Only 54

55. Logical Domains 1.0.3 Requirements
• Solaris 10 05/08 (update 5) and required patches
• Systems firmware version
– T5x40 Firmware 7.1.5b = SP FW 2.0.4.20+
– T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+
– T2000/T1000/T6300/Pelton Firmware 6.5.x
• SUNWldm.v LDOMs Manager Package
– Version 1.0.3
Sun Internal and Approved Partners Only 55

56. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> Comparison to SunFire T1000/T2000/T5x20
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 56

57. Option Parts List for T5440(ATO)
Part Number Option Description Available
SEVY1CB1Z T2 Plus CPU module, 8 core, 1.2GHz, 8GB(4x2GB) with memory board RR
SEVY1CC1Z T2 Plus CPU module, 8 core, 1.2GHz, 16GB(4x4GB) with memory board RR
SEVY1CD1Z T2 Plus CPU module, 8 core, 1.2GHz, 32GB(4x8GB) with memory board
SEVY1DD1Z T2 Plus CPU module, 8 core, 1.4GHz, 32GB(4x8GB) with memory board
SEVY1DB1Z T2 Plus CPU module, 8 core, 1.4GHz, 8GB(4x2GB) with memory board Subject to 1.4GHz availability
SEVY1DC1Z T2 Plus CPU module, 8 core, 1.4GHz, 16GB(4x4GB) with memory board Subject to 1.4GHz availability
SESY2A1Z 2GB Memory Expansion kit(2x1GB)
SESY2B2Z 4GB Memory Expansion kit(2x2GB) Non-Extended ECC version, no mixing
SESY2D1Z 16GB Memory Expansion kit (2 X8 GB)
SESY2C1Z 8GB Memory Expansion kit(2x4GB)
SESVY3A11Z 73GB 10Krpm, 2.5” SAS Drive
SESVY3C11Z 146GB 10Krpm, 2.5” SAS Drive
SESVY3A21Z 73GB 15Krpm, 2.5” SAS Drive
SEVY9PS51Z 1120W AC Power Supply
SEVY9DV1Z DVD, 8x DVD-RW Slimline module
SEVY9RK1Z Slide Rail Kit, Screw Mounting
SEVY9RK2Z Slide Rail Kit, Toolless Mounting
Sun Internal and Approved Partners Only 57

58. Option Parts List for T5440 (cont..)
Part Number Option Description Avialable Max number
SG-XPCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port RR 8
SG-XPCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port RR 8
SG-XPCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port RR 8
SG-XPCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port RR 8
SG-XPCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 8
SG-XPCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 8
SGXCPCIESAS-R-EXT-Z SAS, 8 Port External RAID Controller RR 2
X7281A-2 Northstar Dual port GigE MMF Low profile PCI-E RR 8
X7280A-2 Northstar Dual port GigE UTP Low profile PCI-E RR 8
X1236A-Z Infiniband HBA, Dual 4x ports, PCI-E RR 8
X4447A-Z Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR 8
X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 4
X5558A Tranceiver 10GE SR XFP (XAUI/Atlas) RR -
X5560A-Z Tranceiver 10GE LR XFP (XAUI/Atlas) RR -
X3000A-Z XVR-300, PCI-E x8 Graphics cards (up to 1) RR 4
SESY7XA1Z 10GbE XAUI card RR 2
TBD 10GbE XAUI Copper 10g Base-T TBD 2
X6000A Sun Crypto Accelerator 6000 RR 2
SENY8BE1Z External IO Expansion Unit – PCI-E base Config RR 2
SENY8TE1Z PCI-Express “boat” for External IO Expansion Unit (6 Hot Plug carriers) RR 4
SELY8LK1Z Copper link kit for IO Expansion unit “boat” RR 4
Sun Internal and Approved Partners Only 58

59. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> Comparison to SunFire T1000/T2000/T5x20
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 59

60. Tuning Out of the Box
• Using Solaris 10 05/08 (update 5) with Patches
> MPO optimisations used
> DMA relaxed Ordering implemented
> Default pages sizes for
– Initdata Segments is 64kB pages
– Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages
– ISM and DISM use 256MB pages
– New variables introduced to control these sizes if changes are needed
– max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize,
max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize
> L2 Cache hashing implemented
– helps randomise accesses to L2 caches and ease thrashing
> Stack biasing implemented
– D-Cache is only 4 Way associative... 8 threads
– Running multiple copies of a binary can lead process stacks aligning in the cache
and to hot lines in the cache
– Stack Bias slews the stack variable for processes.
Sun Internal and Approved Partners Only 60

61. Tuning Out of the Box (continued..)
> Processor Groups introduced
– Helps associate CPUs with some shared resources (int/FP pipelines,
Crypto units etc)
– Enables improved multi-level CMT load balancing and affinity policies
> 'bcopy'/'uiomove' Optimizations
– Key kernel operations
– new implementation about 2x faster than previous
• Optimisations will require patching of S10 05/08
> These will be post RR patches as they become available
> Most fixes are already committed in update 5
> Check patch levels for customers with performance issues
> Also check Firmware versions as this can improve performance
too...
Sun Internal and Approved Partners Only 61

62. Tuning Recommendations
• Less involved than with T2000
• S10 5/08 is more 'automatic'
– OOB optimised page size selection, thread scheduling, block copy, IP instances for
zones, page coloring etc...
• If tuning is necessary remember...
> Improved pipeline results in Single threaded performance around 20% better
> Floating Point is no longer an issue
> Start with T2000 tuning options
> Use updated trapstat, cpustat,
• LDOMs may be an easier way to scale poorly threaded applications
• Solaris Crypto framework is still being tuned
> Great for big packets, working on improving small packets
• A Blueprint is being prepared (Denis Sheahan)
Sun Internal and Approved Partners Only 62

63. Tuning Recommendations (cont..)
• Refer to UltraSPARC T1 applications tuning best practices...
> http://www.sun.com/servers/coolthreads/tnb/applications.jsp
• CoolTools are being updated for UltraSPARC T2 and T2 Plus
> http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware
> Recompiling without code changes should benefit T1/T2/T2 Plus apps
• LDOMs configs can benefit significantly from a transition to T5x20 platform
> Multiple IO/Service domains with higher bandwidth PCI busses
> Make sure appropriate LDOMs tuning and sizing is carried out
• Wider Crypto capability, so ensure code is using PKCS#11 framework
> Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers),
n2rng(random number generator) drivers are used appropriately
Sun Internal and Approved Partners Only 63

64. PCIe Performance
• Easily two to four times the performance of T5x20 (two or
four PCIe Busses)
• All PCI card slots are x8 PCI electrically
> Single level hierarchy
> Cut through switching
> PCI-E relaxed ordering for higher performance
• Performance per bus...
> Marketing performance = 4000MB/sec
> Theoretical Rate with overheads = 3790MB/sec
> Max. DMA Achievable(bidirectional) = 3096MB/sec
– with 128byte packets
– And 512 byte read size
Sun Internal and Approved Partners Only 64

65. Agenda
• Introduction
• Comparision of the UltraSPARC T2 to UltraSPARC VF
• Sun SPARC Enterprise T5x40 servers
> Overview of Features
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x40
• ILOM 2.0
• LDOMs for T5x40
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 65

66. Tools and Information
• Cool Tools
– http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• SUNWIN Tokens
> Sun SPARC Enterprise T5440
– Architecture Whitepaper : 526122
– JTF : 526118
> T5440 RAS Whitepaper : TBA
• Video Tours
> Check the Sun.com product pages
• Power Calculators
> Check the Sun.com product pages
Sun Internal and Approved Partners Only 66

67. Further Information
• Presto
> T5440
> http://sunwebcollab.east.sun.com/gm/document-1.26.680780/Batoka_PRESTO.html
• Installed Business Base (IBB) Promotions
> http://sunwebcollab.east.sun.com/gm/document-1.26.680782/Batoka_IBB_Promo.html
• Start Up Essentials program
> http://sunwebcollab.east.sun.com/gm/document-1.26.680794/Maramba_Batoka_SU_April1.html
Sun Internal and Approved Partners Only 67

68. Reference Information
• T5440 Engineering TOI
– http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/
• ILOM/ALOM User Guides
– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf
– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf
• Solaris FMA
– http://fma.eng
– http://lightside.sfbay/n2-fma/docs.php
•
Sun Internal and Approved Partners Only 68

69. Thank You
Sun Microsystems
69

70. T5x40 PCI Topology
Sun Internal and Approved Partners Only 70

71. 10GbE Networking Substantiation
10GbE price for 2 x SR-fiber PCI-E adaptors = $6,740. From HP DL380G5 pricing on HP Store, www.hp.com, 12/17/07. Cost for 2 x 10GbE
and SR transceivers for Maramba and Batoka are $1,800. From sun.com, 12/17/07
Sun Internal and Approved Partners Only 71