1. Error Protected Data Bus Inversion Using Standard DRAM Components Maurizio Skerlj Qimonda AG Server Memory Systems Engineering D-81739 Munich, Germany [email_address] Paolo Ienne Ecole Polytechnique Fédérale de Lausanne (EPFL) School of Computer and Communication Sciences CH-1015 Lausanne, Switzerland [email_address]

2. Data Center Energy Demand Higher performance at lower power consumption but also higher power consumption per real-estate area (ESHRA) Increased power supply and cooling capacity Energy demand for IT doubled in 2000-2005 (EPA) Does this look sustainable? Barrel price in USD

3. High-end Server Power Breakdown (fully populated) Processors and associated fans account for ~45% of energy consumption. Architectural changes help lowering the energy consumption. Main memory and associated fans account for ~45% of energy consumption. Higher density and higher bandwidth increase the energy consumption.

4. Energy Cost of Data Center 100.000 ft 2 data center size Building cost: 20 USD/Watt of peak power 250 W/ft 2 energy density USD 500 million in building cost Electricity cost: 0.80 USD/Watt per Year 20 MWatt yearly power consumption Electricity annual bill of USD 16 million The incentive for green data centers over a 10 year lifetime 1% less energy means USD 6.6 million savings

7. Bus-invert Technique for Lower I/O Energy Consumption [Fletcher, US Pat. 4667337, 1987] [Stan and Burleson, IEEE Tran on VLSI, 1995] Communication Channel RX V dd V dd R TT R TX 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 0 0 1 0 1 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 1 1 0 1 0 1 0 1 1 1 1 0 0 1 1 0 1 1 1 1 0 1 1 1 1 Original data encoded on a 4-bit bus 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 0 0 1 1 0 1 1 1 1 0 1 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 1 0 1 1 0 1 0 1 0 1 1 1 0 1 1 0 0 0 1 1 1 1 1 0 1 0 1 1 1 0 0 1 1 1 1 0 Low-power encoded 4-bit data on a 5-bit bus Bus-invert encoding reduces peak I/O power since all ’0’ pattern is avoided Bus-invert reduces average I/O power since no more than n/2 bits can be ’0’

9. Bus-invert Usage in Commercial Memory Systems The bus-invert technique is today successfully used in some applications in order to reduce the energy consumption Client ECC BI BI Memory Error free channel Architecture Processor ECC BI BI Memory Inter-chip CPU On-chip Cache [Mulla and Tu, US Pat. Appl. 289435, 2005] GPU BI BI Memory Short point-to-point GPU GDRAM Graphic Systems [Ihm et al., ISSCC 2007]

11. Extension of the Architecture Lead to Reliability Issues Cache ECC BI BI SDRAM Long multi-stub bus Cache ECC BI SDRAM Long multi-stub bus DIMMs DIMMs CPU CPU BER > 0 Non-standard BER > 0 Soft and Hard Failures System reliability and usage of standard parts is paramount in servers. Architectures which do not fulfil those requirements are not viable.

12. If it uses ECC, it’s reliable. Really? System reliability is as weak as the weakest of its components.

15. Exploiting Code Linearity to Combine Error Protection and Bus-invert Coding x c x ECC Calculation x 1  x 2 c 1  c 2 x 1  x 2 ECC Calculation Systematic codes are simple ... ... and linear Inverting a codeword means x ECC Calculation ECC Calculation + c’ A fixed value can be hard-coded

16. Proposed Encoding Scheme All the bits in the message enjoy the ECC protection.

17. How to Squeeze More Bits in the Standard Frame? 64bit 64bit 8bit 64bit 64bit 8bit Data SECDED check bits 72-bit bus 8bit 8bit Standard frame (burst length of 4) 67 bits: 64-bit data + 3 inversion bits 5 bits: SECDED check bits (on 2 columns) Modified frame with bus inversion (burst length of 4) 8-bit ECC 2-bit parity 22bit b 0 22bit b 1 20bit b 2 22bit b 0 22bit b 1 20bit b 2 8-bit ECC 2-bit parity 22bit b 0 22bit b 1 20bit b 2 22bit b 0 22bit b 1 20bit b 2 Error detection and correction code efficiency increases with the block size.

19. Stochastical Model States of the Model: S - exactly one bit is wrong due to a soft error; H - exactly one bit is wrong due to a hard failure; 0 - N b bits are correct; 0’ - (N b -1) bits are correct; W – one bit wrong due to a communication error during a write operation (communication errors during reads can be restored with retransmission) Assumptions: All failure mechanisms are Poissonian processes.  H ,  S , and  b are rates of occurrence of respectively hard, soft and communication fails. T 0 is the time period with witch the word is purged from soft errors. T b is the bit-unit interval. [Noorlag et al., IEEE JSSC., 1980]

20. Our Solution Saves Energy with no Reliability Issues or Cost Increase

25. Thank You