SlideShare ist ein Scribd-Unternehmen logo
1 von 71
Downloaden Sie, um offline zu lesen
Kajaani	
  Datacenter	
  Case-­‐Study	
  
Peter	
  Jenkins	
  
System	
  Specialist	
  &	
  Project	
  Manager	
  
CSC	
  –	
  IT	
  Centre	
  for	
  Science	
  	
  
CSC Modular Datacenter
Contents
! About CSC
–  High Performance Computing
! Drivers
–  Why a new datacenter?
–  Why Kajaani?
! The project
! What we built
! Construction and commissioning
! Lessons learned
I’m from
there
Who am I?
! Computer Science background
! 6 years at Sun Microsystems
! HPC consultant in Financial Services
! 3 years in Finland working for CSC
! I’m not a datacenter specialist!
– It helps!
My role at CSC
IT
•  Rapid change
•  New hardware
challenging datacenters
Facilities
•  Traditionally very stable
•  Disruptive innovations in
last 5 years
•  Traditionally separate departments must now
work together
•  I try to bridge the gap
About	
  CSC	
  
•  100%	
  owned	
  by	
  ministry	
  of	
  educaGon	
  
– Public	
  sector,	
  Non	
  profit	
  
•  Major	
  service	
  areas:	
  
– High	
  Performance	
  CompuGng	
  (HPC)	
  
– Storage	
  and	
  archive	
  
– FUNET	
  –	
  Academic	
  network	
  in	
  Finland	
  
– Managed	
  services	
  and	
  IaaS	
  cloud	
  
– Server	
  hosGng	
  and	
  co-­‐lo	
  
CSC and High Performance Computing
 	
  
CSC	
  CompuGng	
  Capacity	
  1989–2011	
  
CSC Modular Datacenter
338	
  
227	
  
146	
  
141	
  
124	
  
95	
  
42	
  
37	
  
35	
  
34	
  
167	
  
Users	
  of	
  CSC	
  resources	
  by	
  discipline	
  2011	
  
(total	
  1386	
  users)	
  	
  
Biosciences	
  
Physics	
  
Nanoscience	
  
Language	
  research	
  
Chemistry	
  
Grid	
  usage	
  
ComputaGonal	
  fluid	
  dynamics	
  
Structural	
  analysis	
  
ComputaGonal	
  drug	
  design	
  
Earth	
  sciences	
  
Other	
  
WHY A NEW DATACENTER?
0	
  
1	
  
2	
  
3	
  
4	
  
5	
  
6	
  
7	
  
8	
  
9	
  
10	
  
2005	
   2006	
   2007	
   2008	
   2009	
   2010	
   2011	
  
Energy	
  GWh	
   CSCs	
  Datacenter	
  Energy	
  Use	
  2005	
  -­‐	
  2011	
  
DC	
  2	
  Infra	
   DC	
  2	
  IT	
  
DC	
  1	
  Infra	
   DC	
  1	
  IT	
  
 -­‐	
  €	
  	
  
	
  10	
  €	
  	
  
	
  20	
  €	
  	
  
	
  30	
  €	
  	
  
	
  40	
  €	
  	
  
	
  50	
  €	
  	
  
	
  60	
  €	
  	
  
2000	
   2001	
   2002	
   2003	
   2004	
   2005	
   2006	
   2007	
   2008	
   2009	
   2010	
   2011	
  
Elspot	
  prices	
  FI	
  in	
  EUR/MWh	
  
New	
  site	
  drivers	
  
•  Capacity	
  limiGng	
  growth	
  
–  Super	
  computers	
  
–  Managed	
  hosGng	
  
•  Costs	
  increasing	
  
–  Tax	
  
–  Energy	
  
–  Carbon	
  neutral	
  
•  New	
  site	
  focused	
  on	
  
efficiency	
  and	
  capacity	
  
0	
  
1	
  
2	
  
3	
  
4	
  
5	
  
6	
  
7	
  
8	
  
9	
  
10	
  
2005	
   2006	
   2007	
   2008	
   2009	
   2010	
   2011	
  
Energy	
  GWh	
  
CSCs	
  Datacenter	
  Energy	
  Use	
  2005	
  -­‐	
  2011	
  
DC	
  2	
  Infra	
   DC	
  2	
  IT	
  
DC	
  1	
  Infra	
   DC	
  1	
  IT	
  
2011 Energy cost per MW/hour
Espoo Kajaani
VAT
Tax
Transfer
Electricity
WHY KAJAANI?
10/10/2012 18
Where is Kajaani?
CSC Modular Datacenter
CSC Modular Datacenter
CSC Modular Datacenter
national
grid
fibre
fibre
CSC Modular Datacenter
CSC Modular Datacenter
CSC Modular Datacenter
CSC Modular Datacenter
CSC Modular Datacenter
Electricity	
  reliability	
  
29	
  
	
  
	
  
Finnish	
  naGonal	
  grid	
   Renforsin	
  Ranta	
  substaGon	
  
Fingrid connections
to Kajaani
Power capabilities
! Within perimeter fence:
–  National grid connection access to 340 MW
–  110 kV / 10 kV main transformer capacity
!  Current capacity 240 MW
–  Biopower on site
! Green power options
–  3 hydro power plants within 3 km
!  feeding directly to site.
! Diverse power supply = reliable power
CSC Modular Datacenter
Related	
  datacenter	
  sites	
  
Facebook:	
  Luleå	
  
120	
  MW	
  Free	
  air	
  cooling	
  
CSC:	
  Kajaani	
  
1.4	
  MW	
  Free	
  air	
  cooling	
  
0.9	
  MW	
  Water	
  cooling	
  
PUE	
  design	
  
Google:	
  Hamina	
  
???	
  MW	
  Sea	
  water	
  
cooling	
  
CSC:	
  Espoo	
  
1.6	
  MW	
  ConvenGonal	
  
PUE	
  1.4	
  &	
  1.8	
  
Government support
! Google Hamina = wakeup call
–  Unused assets ideal for fast growing industry
–  Jobs, skills, international competitiveness
! Government acted
–  Regional development money
–  Extra money to CSC to build a new site
! Site selection: long story short
–  Initial concept study 2010
–  Several former paper mills considered
–  Kajaani was successful in bidding
THE PROJECT
Approach
! Design goal: multi-MW facility PUE <1.2
! Leverage features of site
–  Matched to business requirements
–  Avoid redundancy and backup
! Only 100kW UPS from day one
–  < 5% of load
–  core network, management, automation
–  Emphasis on monitoring and rapid recovery
! No generators day one
! Option to add 100% UPS and generators
Approach continues
! Free cooling year round
! Use modular to right-size and scale quickly
! Green
–  CSC buys certificates of carbon neutral energy
–  100% Finnish hydro power
Out
side	
  
Timeline	
  
Paper	
  mill	
  
2010	
   2011	
   2012	
  
Site	
  selecGon	
  
Planning	
  
Warehouse	
  
2013	
  
1st	
  
ITT	
  
Datacenter	
  
Supercomputer	
  
October	
  
2nd	
  
ITT	
  
Analysis	
  
Delivery	
  
Conven&onal	
  
build	
  
Modular	
  
build	
  
WHAT DID WE BUILD?
CSC’s new Kajaani datacenter
!   Renovated paper warehouse
!   12 000 m2 of space
!   ~ 1.7 x Old Trafford football ground
Warehouse	
  has	
  two	
  DC	
  tenants	
  
Hub	
  
datacenter	
  
SGI	
  Ice	
  Cube	
  R80	
  
TWO	
  DATACENTERS	
  IN	
  ONE	
  
3D WALKTHROUGH VIDEO
Specification
! 2.4 MW combined hybrid capacity
! 1.4 MW modular free air cooled datacenter
–  Upgradable in 700kW factory built modules
–  Order to acceptance in 5 months
–  35kW per extra tall racks – 12kW common in
industry
–  PUE forecast < 1.08 (pPUEL2,YC)
! 1MW HPC datacenter
–  Optimised for Cray super & T-Platforms
prototype
–  90% Water cooling
Hub	
  building	
  and	
  DC	
  
Hub
datacenter
Hub building
! Redundant network rooms
! 100kW UPS – upgradable to MW’s
! 10KV switchgear
! Fire suppression
! Storage rooms – warehouse is cold
! Hub Datacenter
Hub	
  Datacenter	
  
Chillers	
  
2	
  x	
  450	
  kW	
  
External	
  dry	
  
coolers	
  
CSC Modular Datacenter
Hub DC facts
! Due November
! 900kW water cooling
–  + 100kW air from hub
! Purpose built for water
cooled HPC
IT	
  summary	
  
•  Cray	
  “cascade”	
  supercomputer	
  
–  10M€	
  -­‐	
  five	
  year	
  contract	
  
–  Fastest	
  computer	
  in	
  Finland	
  due	
  mid-­‐November	
  
–  Phase	
  one	
  385	
  kW	
  2012	
  of	
  Intel	
  processors	
  
–  Very	
  high	
  density,	
  large	
  racks	
  
•  T-­‐Plamorms	
  prototype	
  
–  Very	
  high	
  density	
  hot-­‐water	
  cooled	
  racks	
  
–  Intel	
  processors,	
  Intel	
  and	
  NVIDIA	
  accelerators	
  
–  TheoreGcal	
  400	
  TFlops	
  performance	
  
SGI	
  Ice	
  Cube	
  R80	
  
CSC Modular Datacenter
SGI	
  Ice	
  Cube	
  R80	
  
SGI Ice Cube R80
! One head unit and two expansion modules
! More modules can be added
! Fully automated free cooling system
–  Dozens of cooling fans, louvers and sensors
! Extremely energy efficient – pPUE 1.08
! Set point allowed to vary (10-27C for us)
! Adiabatic cooling on warm days
! Exhaust heat used to warm incoming air
Factory	
  visit	
  August	
  2012	
  
CSC Modular Datacenter
CSC Modular Datacenter
CSC Modular Datacenter
! “Super Cluster”
–  4.5M€ five year contract
–  1,152 Intel CPUs
–  190 TFlop/s
–  30kW 47U racks
! HPC storage
–  3PB of fast parallel storage
–  Supports Cray and HP systems
IT summary
CONSTRUCTION
Site	
  in	
  January	
  2012	
  
MDC TIME LAPSE VIDEO
Head	
  load	
  test	
  
•  700kW	
  of	
  load	
  banks	
  
– 20	
  racks	
  
– 120	
  x	
  6kW	
  load	
  banks	
  
– 2	
  days	
  to	
  rack	
  mount	
  
– Half	
  MDC	
  capacity	
  
•  pPUE	
  1.05	
  
•  Very	
  useful	
  
•  Silenced	
  the	
  criGcs	
  
Commissioning
!   For MDC it took longer than
construction!
!   MDC internal workings and
drawings are sensitive
–  Test plan process challenging
–  Training required for new free
cooled system type
!   Where possible plan in
advance
!   Allow plenty of time
LESSONS LEARNED
Lessons learned
! Simplify your procurement!
! Share every detail of your site with bidders
! A modular datacenter is still a datacenter
–  Even if design fixed is expect many decisions
! Pay special attention to integration
–  fire, security, power, electricity, etc.
–  Any small issue can cause major delay
! Get expert advice
–  don't assume!
Reference or define every ’standard’
Don't assume
! Racks
–  Define depth, height, RU, posts
–  Strongly consider alternatives like
opencompute
! Door heights
! Make a list of required standards
–  Fire suppression – insurance, building permit
! Get a datacenter contract
–  Modular buildings contracts != IT contracts
–  If possible multiple phases of acceptance
Things we got right
! Make the selection based on 10 year TCO
–  IT TCO was 5 years per phase
–  highlights energy efficiency without defining 'green’
–  could also use an artificially high energy price
! Require that IT vendors list
–  PDU efficiency use 80Plus
–  Temperature and humidity ranges
–  ASHRAE TC 9.9 2011
!  All our suppliers support A2 range (5 to 35C)
Datacenter operations team
QUESTIONS?
Thanks
More information
! Contact me
–  peter.jenkins@csc.fi
! Blog:
–  http://www.csc.fi/blogs/gridthings

Weitere ähnliche Inhalte

Ähnlich wie CSC Modular Datacenter

Expectations for optical network from the viewpoint of system software research
Expectations for optical network from the viewpoint of system software researchExpectations for optical network from the viewpoint of system software research
Expectations for optical network from the viewpoint of system software researchRyousei Takano
 
Building European competitiveness: Smart specialization for European datacenters
Building European competitiveness: Smart specialization for European datacentersBuilding European competitiveness: Smart specialization for European datacenters
Building European competitiveness: Smart specialization for European datacentersCSC - IT Center for Science
 
COHEAT @ BEIS May 2017
COHEAT @ BEIS May 2017COHEAT @ BEIS May 2017
COHEAT @ BEIS May 2017Marko Cosic
 
Hyper efficient data centres – key ingredient intelligence networkshop44
Hyper efficient data centres – key ingredient intelligence   networkshop44Hyper efficient data centres – key ingredient intelligence   networkshop44
Hyper efficient data centres – key ingredient intelligence networkshop44Jisc
 
Aged Data Center Infrastructure.pptx
Aged Data Center Infrastructure.pptxAged Data Center Infrastructure.pptx
Aged Data Center Infrastructure.pptxSchneider Electric
 
Dock It Customer Intro 14 Aug 09
Dock It Customer Intro 14 Aug 09Dock It Customer Intro 14 Aug 09
Dock It Customer Intro 14 Aug 09DockIT
 
The next generation - Dave Mullender
The next generation - Dave MullenderThe next generation - Dave Mullender
The next generation - Dave MullenderTalkTalk Business
 
How A Next Generation Data Centre Can Make All The Difference
How A Next Generation Data Centre Can Make All The DifferenceHow A Next Generation Data Centre Can Make All The Difference
How A Next Generation Data Centre Can Make All The Differencerlawlor
 
The Next Generation
The Next GenerationThe Next Generation
The Next GenerationPartner Talk
 
Data Center Market and Technology Trends Power Electronics presentation held ...
Data Center Market and Technology Trends Power Electronics presentation held ...Data Center Market and Technology Trends Power Electronics presentation held ...
Data Center Market and Technology Trends Power Electronics presentation held ...Yole Developpement
 
Optalysis: Disruptive Optical Processing Technology for HPC
Optalysis: Disruptive Optical Processing Technology for HPCOptalysis: Disruptive Optical Processing Technology for HPC
Optalysis: Disruptive Optical Processing Technology for HPCinside-BigData.com
 
Tech 2 Tech - My network refresh
Tech 2 Tech - My network refreshTech 2 Tech - My network refresh
Tech 2 Tech - My network refreshJisc
 
Datacenter 2014: Schneider Electric A/S - Jørgen Bjerg
Datacenter 2014: Schneider Electric A/S - Jørgen BjergDatacenter 2014: Schneider Electric A/S - Jørgen Bjerg
Datacenter 2014: Schneider Electric A/S - Jørgen BjergMediehuset Ingeniøren Live
 
Practical IEC 61850 for Substation Automation for Engineers & Technicians
Practical IEC 61850 for Substation Automation for Engineers & TechniciansPractical IEC 61850 for Substation Automation for Engineers & Technicians
Practical IEC 61850 for Substation Automation for Engineers & TechniciansLiving Online
 
MIG 5th Data Centre Summit 2016 PTS Presentation v1
MIG 5th Data Centre Summit 2016 PTS Presentation v1MIG 5th Data Centre Summit 2016 PTS Presentation v1
MIG 5th Data Centre Summit 2016 PTS Presentation v1blewington
 
ClusterVision & Intel: Top500 class Computing at the University of Paderborn
ClusterVision & Intel: Top500 class Computing at the University of PaderbornClusterVision & Intel: Top500 class Computing at the University of Paderborn
ClusterVision & Intel: Top500 class Computing at the University of PaderbornIntel IT Center
 
Green cloud computing
Green cloud computingGreen cloud computing
Green cloud computingMathews Job
 

Ähnlich wie CSC Modular Datacenter (20)

Expectations for optical network from the viewpoint of system software research
Expectations for optical network from the viewpoint of system software researchExpectations for optical network from the viewpoint of system software research
Expectations for optical network from the viewpoint of system software research
 
Building European competitiveness: Smart specialization for European datacenters
Building European competitiveness: Smart specialization for European datacentersBuilding European competitiveness: Smart specialization for European datacenters
Building European competitiveness: Smart specialization for European datacenters
 
COHEAT @ BEIS May 2017
COHEAT @ BEIS May 2017COHEAT @ BEIS May 2017
COHEAT @ BEIS May 2017
 
Green ICT
Green ICTGreen ICT
Green ICT
 
Hyper efficient data centres – key ingredient intelligence networkshop44
Hyper efficient data centres – key ingredient intelligence   networkshop44Hyper efficient data centres – key ingredient intelligence   networkshop44
Hyper efficient data centres – key ingredient intelligence networkshop44
 
Aged Data Center Infrastructure.pptx
Aged Data Center Infrastructure.pptxAged Data Center Infrastructure.pptx
Aged Data Center Infrastructure.pptx
 
1.1 integrating renewables srm
1.1 integrating renewables   srm1.1 integrating renewables   srm
1.1 integrating renewables srm
 
Dock It Customer Intro 14 Aug 09
Dock It Customer Intro 14 Aug 09Dock It Customer Intro 14 Aug 09
Dock It Customer Intro 14 Aug 09
 
The next generation - Dave Mullender
The next generation - Dave MullenderThe next generation - Dave Mullender
The next generation - Dave Mullender
 
How A Next Generation Data Centre Can Make All The Difference
How A Next Generation Data Centre Can Make All The DifferenceHow A Next Generation Data Centre Can Make All The Difference
How A Next Generation Data Centre Can Make All The Difference
 
The Next Generation
The Next GenerationThe Next Generation
The Next Generation
 
Data Center Market and Technology Trends Power Electronics presentation held ...
Data Center Market and Technology Trends Power Electronics presentation held ...Data Center Market and Technology Trends Power Electronics presentation held ...
Data Center Market and Technology Trends Power Electronics presentation held ...
 
Optalysis: Disruptive Optical Processing Technology for HPC
Optalysis: Disruptive Optical Processing Technology for HPCOptalysis: Disruptive Optical Processing Technology for HPC
Optalysis: Disruptive Optical Processing Technology for HPC
 
Tech 2 Tech - My network refresh
Tech 2 Tech - My network refreshTech 2 Tech - My network refresh
Tech 2 Tech - My network refresh
 
Datacenter 2014: Schneider Electric A/S - Jørgen Bjerg
Datacenter 2014: Schneider Electric A/S - Jørgen BjergDatacenter 2014: Schneider Electric A/S - Jørgen Bjerg
Datacenter 2014: Schneider Electric A/S - Jørgen Bjerg
 
Practical IEC 61850 for Substation Automation for Engineers & Technicians
Practical IEC 61850 for Substation Automation for Engineers & TechniciansPractical IEC 61850 for Substation Automation for Engineers & Technicians
Practical IEC 61850 for Substation Automation for Engineers & Technicians
 
Venture Capital World Summit 2015 Agenda Dr Brian Antao
Venture Capital World Summit 2015 Agenda Dr Brian AntaoVenture Capital World Summit 2015 Agenda Dr Brian Antao
Venture Capital World Summit 2015 Agenda Dr Brian Antao
 
MIG 5th Data Centre Summit 2016 PTS Presentation v1
MIG 5th Data Centre Summit 2016 PTS Presentation v1MIG 5th Data Centre Summit 2016 PTS Presentation v1
MIG 5th Data Centre Summit 2016 PTS Presentation v1
 
ClusterVision & Intel: Top500 class Computing at the University of Paderborn
ClusterVision & Intel: Top500 class Computing at the University of PaderbornClusterVision & Intel: Top500 class Computing at the University of Paderborn
ClusterVision & Intel: Top500 class Computing at the University of Paderborn
 
Green cloud computing
Green cloud computingGreen cloud computing
Green cloud computing
 

Kürzlich hochgeladen

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 

Kürzlich hochgeladen (20)

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 

CSC Modular Datacenter

  • 1. Kajaani  Datacenter  Case-­‐Study   Peter  Jenkins   System  Specialist  &  Project  Manager   CSC  –  IT  Centre  for  Science    
  • 3. Contents ! About CSC –  High Performance Computing ! Drivers –  Why a new datacenter? –  Why Kajaani? ! The project ! What we built ! Construction and commissioning ! Lessons learned
  • 5. Who am I? ! Computer Science background ! 6 years at Sun Microsystems ! HPC consultant in Financial Services ! 3 years in Finland working for CSC ! I’m not a datacenter specialist! – It helps!
  • 6. My role at CSC IT •  Rapid change •  New hardware challenging datacenters Facilities •  Traditionally very stable •  Disruptive innovations in last 5 years •  Traditionally separate departments must now work together •  I try to bridge the gap
  • 7. About  CSC   •  100%  owned  by  ministry  of  educaGon   – Public  sector,  Non  profit   •  Major  service  areas:   – High  Performance  CompuGng  (HPC)   – Storage  and  archive   – FUNET  –  Academic  network  in  Finland   – Managed  services  and  IaaS  cloud   – Server  hosGng  and  co-­‐lo  
  • 8. CSC and High Performance Computing
  • 10. CSC  CompuGng  Capacity  1989–2011  
  • 12. 338   227   146   141   124   95   42   37   35   34   167   Users  of  CSC  resources  by  discipline  2011   (total  1386  users)     Biosciences   Physics   Nanoscience   Language  research   Chemistry   Grid  usage   ComputaGonal  fluid  dynamics   Structural  analysis   ComputaGonal  drug  design   Earth  sciences   Other  
  • 13. WHY A NEW DATACENTER?
  • 14. 0   1   2   3   4   5   6   7   8   9   10   2005   2006   2007   2008   2009   2010   2011   Energy  GWh   CSCs  Datacenter  Energy  Use  2005  -­‐  2011   DC  2  Infra   DC  2  IT   DC  1  Infra   DC  1  IT  
  • 15.  -­‐  €      10  €      20  €      30  €      40  €      50  €      60  €     2000   2001   2002   2003   2004   2005   2006   2007   2008   2009   2010   2011   Elspot  prices  FI  in  EUR/MWh  
  • 16. New  site  drivers   •  Capacity  limiGng  growth   –  Super  computers   –  Managed  hosGng   •  Costs  increasing   –  Tax   –  Energy   –  Carbon  neutral   •  New  site  focused  on   efficiency  and  capacity   0   1   2   3   4   5   6   7   8   9   10   2005   2006   2007   2008   2009   2010   2011   Energy  GWh   CSCs  Datacenter  Energy  Use  2005  -­‐  2011   DC  2  Infra   DC  2  IT   DC  1  Infra   DC  1  IT  
  • 17. 2011 Energy cost per MW/hour Espoo Kajaani VAT Tax Transfer Electricity
  • 29. Electricity  reliability   29       Finnish  naGonal  grid   Renforsin  Ranta  substaGon  
  • 31. Power capabilities ! Within perimeter fence: –  National grid connection access to 340 MW –  110 kV / 10 kV main transformer capacity !  Current capacity 240 MW –  Biopower on site ! Green power options –  3 hydro power plants within 3 km !  feeding directly to site. ! Diverse power supply = reliable power
  • 33. Related  datacenter  sites   Facebook:  Luleå   120  MW  Free  air  cooling   CSC:  Kajaani   1.4  MW  Free  air  cooling   0.9  MW  Water  cooling   PUE  design   Google:  Hamina   ???  MW  Sea  water   cooling   CSC:  Espoo   1.6  MW  ConvenGonal   PUE  1.4  &  1.8  
  • 34. Government support ! Google Hamina = wakeup call –  Unused assets ideal for fast growing industry –  Jobs, skills, international competitiveness ! Government acted –  Regional development money –  Extra money to CSC to build a new site ! Site selection: long story short –  Initial concept study 2010 –  Several former paper mills considered –  Kajaani was successful in bidding
  • 36. Approach ! Design goal: multi-MW facility PUE <1.2 ! Leverage features of site –  Matched to business requirements –  Avoid redundancy and backup ! Only 100kW UPS from day one –  < 5% of load –  core network, management, automation –  Emphasis on monitoring and rapid recovery ! No generators day one ! Option to add 100% UPS and generators
  • 37. Approach continues ! Free cooling year round ! Use modular to right-size and scale quickly ! Green –  CSC buys certificates of carbon neutral energy –  100% Finnish hydro power
  • 38. Out side   Timeline   Paper  mill   2010   2011   2012   Site  selecGon   Planning   Warehouse   2013   1st   ITT   Datacenter   Supercomputer   October   2nd   ITT   Analysis   Delivery   Conven&onal   build   Modular   build  
  • 39. WHAT DID WE BUILD?
  • 40. CSC’s new Kajaani datacenter !   Renovated paper warehouse !   12 000 m2 of space !   ~ 1.7 x Old Trafford football ground
  • 41. Warehouse  has  two  DC  tenants  
  • 42. Hub   datacenter   SGI  Ice  Cube  R80   TWO  DATACENTERS  IN  ONE  
  • 44. Specification ! 2.4 MW combined hybrid capacity ! 1.4 MW modular free air cooled datacenter –  Upgradable in 700kW factory built modules –  Order to acceptance in 5 months –  35kW per extra tall racks – 12kW common in industry –  PUE forecast < 1.08 (pPUEL2,YC) ! 1MW HPC datacenter –  Optimised for Cray super & T-Platforms prototype –  90% Water cooling
  • 46. Hub datacenter Hub building ! Redundant network rooms ! 100kW UPS – upgradable to MW’s ! 10KV switchgear ! Fire suppression ! Storage rooms – warehouse is cold ! Hub Datacenter
  • 47. Hub  Datacenter   Chillers   2  x  450  kW   External  dry   coolers  
  • 49. Hub DC facts ! Due November ! 900kW water cooling –  + 100kW air from hub ! Purpose built for water cooled HPC
  • 50. IT  summary   •  Cray  “cascade”  supercomputer   –  10M€  -­‐  five  year  contract   –  Fastest  computer  in  Finland  due  mid-­‐November   –  Phase  one  385  kW  2012  of  Intel  processors   –  Very  high  density,  large  racks   •  T-­‐Plamorms  prototype   –  Very  high  density  hot-­‐water  cooled  racks   –  Intel  processors,  Intel  and  NVIDIA  accelerators   –  TheoreGcal  400  TFlops  performance  
  • 51. SGI  Ice  Cube  R80  
  • 53. SGI  Ice  Cube  R80  
  • 54. SGI Ice Cube R80 ! One head unit and two expansion modules ! More modules can be added ! Fully automated free cooling system –  Dozens of cooling fans, louvers and sensors ! Extremely energy efficient – pPUE 1.08 ! Set point allowed to vary (10-27C for us) ! Adiabatic cooling on warm days ! Exhaust heat used to warm incoming air
  • 59. ! “Super Cluster” –  4.5M€ five year contract –  1,152 Intel CPUs –  190 TFlop/s –  30kW 47U racks ! HPC storage –  3PB of fast parallel storage –  Supports Cray and HP systems IT summary
  • 61. Site  in  January  2012  
  • 62. MDC TIME LAPSE VIDEO
  • 63. Head  load  test   •  700kW  of  load  banks   – 20  racks   – 120  x  6kW  load  banks   – 2  days  to  rack  mount   – Half  MDC  capacity   •  pPUE  1.05   •  Very  useful   •  Silenced  the  criGcs  
  • 64. Commissioning !   For MDC it took longer than construction! !   MDC internal workings and drawings are sensitive –  Test plan process challenging –  Training required for new free cooled system type !   Where possible plan in advance !   Allow plenty of time
  • 66. Lessons learned ! Simplify your procurement! ! Share every detail of your site with bidders ! A modular datacenter is still a datacenter –  Even if design fixed is expect many decisions ! Pay special attention to integration –  fire, security, power, electricity, etc. –  Any small issue can cause major delay ! Get expert advice –  don't assume!
  • 67. Reference or define every ’standard’ Don't assume ! Racks –  Define depth, height, RU, posts –  Strongly consider alternatives like opencompute ! Door heights ! Make a list of required standards –  Fire suppression – insurance, building permit ! Get a datacenter contract –  Modular buildings contracts != IT contracts –  If possible multiple phases of acceptance
  • 68. Things we got right ! Make the selection based on 10 year TCO –  IT TCO was 5 years per phase –  highlights energy efficiency without defining 'green’ –  could also use an artificially high energy price ! Require that IT vendors list –  PDU efficiency use 80Plus –  Temperature and humidity ranges –  ASHRAE TC 9.9 2011 !  All our suppliers support A2 range (5 to 35C)
  • 71. More information ! Contact me –  peter.jenkins@csc.fi ! Blog: –  http://www.csc.fi/blogs/gridthings