This document discusses progress towards developing kilometer-scale Earth system simulations. It outlines a goal of achieving globally uniform 1 km weather and climate modeling by defining fundamental algorithmic building blocks ("Weather and Climate Dwarfs") and exploring hardware adaptation. It also reviews requirements for precision and verification as resolution increases. Achieving 1 km simulations would represent a major advance but require orders of magnitude increases in computational resources.
From Weather Dwarfs to Kilometre-Scale Earth System Simulations
1. From Weather Dwarfs to Kilometre-Scale
Earth System Simulations
Nils P. Wedi, Peter Bauer, ECMWF colleagues, ESCAPE project partners
European Centre for Medium-Range Weather Forecasts (ECMWF)
PASC2018, Basel
2. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Outline
• Numerical weather prediction & climate, a brief (HPC) history
• Ensemble-based assimilation and forecasts of Weather & Climate
• Earth-System complexity and the notion to resolve rather than parametrise
• An intermediate goal: globally uniform weather & climate modelling at 1 km horizontal resolution
– Defining and encapsulating the fundamental algorithmic building blocks ("Weather and Climate Dwarfs")
– Pioneering algorithm development with hardware adaptation using DSL toolchains
– Reviewing the need for precision
– A HPCW benchmark and cross-disciplinary Verification, Validation, and Uncertainty Quantification (VVUQ)
2
3. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 3
ECMWF’s progress in degrees of freedom
(levels x grid columns x prognostic variables)
(Schulthess et al, 2018)
4. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Sustained HPC performance
4
5. October 29, 2014
I/O - Impact of NVRAM on Data Access
Byte Addressable Hypercubes
• Longitude (3600)
• Latitude (1800)
• Atmospheric levels, Physical parameters (~200)
• Time steps (~100)
• Probabilistic pertubations (50)
@ double precision
• 9km 48 TiB
• 5km 192 TiB
• 1.25km 1.82 PiB
Not included: historical observations, multiple models, etc...
5EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Clients want to do different analytics
across multiple axis
Tiago Quintino
ECMWF archives ~150TB / Day
Growing exponentially …
6. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Ensemble of assimilations and forecasts
6
8. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 8
Copernicus climate change monitoring service C3S
9. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 9
Observation impact on reducing the short-range forecast error
Future:
Mobile phones,
cars, homes, ….
11. Extreme events: Hurricane IRMA
11
NASA GOES-16
10 day forecast
6 day forecast
High impact disruptive events: Use of high-resolution ensembles
to estimate uncertainty in large-scale (steering) flow.
12. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Hurricane IRMA 18km vs 5km ensemble
12
ECMWF Strategy 2025
a 5km ensemble …
13. October 29, 2014
Global KE - Spectra ~500hPa
~9km ~2.5km ~1.25km
Resolve rather than parametrize much of the
crucial vertical transport of momentum and heat
14. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
TCo1279 9km
NARVAL region
15. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
TCo7999 1.25km
NARVAL region
16. 16
48h forecast ~9km
48h forecast ~1km
http://gigapan.com/gigapans/206287
Take the “Turing test” of climate & weather modelling (T. Palmer)
17. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
(Daniel Klocke, DWD)
19. 26%
24%
30%
9%
9%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD WAVEMODEL OCEANMODEL
coupled TCo1279 L137 (~9km operational) run
Where do we spent the time ?
Where do we spend the time ?
Single electrical group:
~47minutes wallclock time
(single electrical group==384 nodes)
1408 MPI tasks x 18 threads
306 FC/day
20. October 29, 2014
Ocean resolution requirements
Hallberg 2013
From the presentation of H. Hewitt, at ECMWF (2016)
21. 21EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
AWI FESOM2 team
Animation: Nikolay Koldunov (AWI)
Arctiv Ocean
circulation at ~1km
4 months/day, time step 2 min, 1700 Broadwell
Cores on Cray CS400
22. October 29, 2014 22EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
AWI FESOM2 team
Animation: Nikolay Koldunov (AWI)
Arctic sea-ice evolution at
~1km
23. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Cut-off where power is 50% of k-5/3
IFS KE spectra compared to altimeter observations
Resolved 1 km scales need at least x6 more resolution and x10000s
in compute => intermediate goal 1 km grid
[Courtesy S. Abdalla]
25. October 29, 2014
(hpc-
escape.eu)
Weather & Climate Dwarfs
Extract model dwarfs…
… explore alternative
numerical algorithms
…
… hardware
adaptation …
… reassemble
model and
benchmark
32. October 29, 2014
0
20
40
60
80
100
120
799 1279 2047 3999 7999
dgemm FLT
Number of floating point operations for direct or inverse spectral
transforms of a single field, scaled by N2log3N
(Wedi et al, 2013)
Fast Legendre Transform
Reduces number of
floating point operations
35. 16%
45%
34%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD
Example: TCo7999 L62 (~1.25km)
Where do we spent the time ?
The cost profile of a 1.25km IFS atmosphere simulation on Piz Daint (CPU only)
4880 MPI tasks x 12 threads
69 FC/day ~ 0.19 SYPD
single precision / FLT
75% comms;
25% compute
~85.21 MWh / SY
Based on the Piz Daint, Swiss
Cray XC50 Haswell, Aries
interconnect, ~5000 nodes total
36. October 29, 2014 October 29, 2014
Simulating performance and scalability of
MPI communications
36
Zheng and Marginaud (GMD, 2018)
Communication time as a function
of halo size, topology & routing algorithm,
compared to spectral transpositions
MPI collectives at least across a subset
of nodes are required in NWP & climate!
37. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Communication is bad – small time steps are worse
Same time to solution!
Energy efficiency?
Spectral element model
Time step = 4s
Spectral transform model
Time step = 240s
Spectral element model
Time step = 4s
Data movement x100 (x1000)
more expensive than
computations in time (energy)!
[Shalf et al. 2011]
40. 52
96
154
197
70
133
239
380
295
536
10
21
170
323
0
100
200
300
400
500
600
120 240 480 960 1920
FORECASTDAYS/DAY
XC-40 NODES (X36 == CORES)
IFS single precision performance – Atmosphere only (no I/O)
TCo3999 (2.5km) TCo1999 (5km) TCo1279 (9km) TCo7999 (1km) TCo1279_DP (9km)
Singe vs double precision
10 days in 1 hour
137 levels
62 levels
(Vana, Dueben et al 2017)
41. A scale-selective approach: Track of Hurricane Irma
• Spectral models allow to treat different scales at different
precision.
• We can reduce precision when calculating the small scales.
• This is intuitive due to the high inherent uncertainty in small scale
dynamics (parametrisation, viscosity, data-assimilation,...).
Chantry, Düben and Palmer in preparation
42. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
IFS 1km: strong scaling on PizDaint
42
Goal ~1 year / Day
Many thanks to
Thomas Schulthess &
Maria Grazia Giuffreda !
48. October 29, 2014
ECMWF’s mission strategy
• In collaboration with its member and associated states
– Address the challenges in sustainable computing and data handling
– Move towards Earth-System complexity with rigorous quantification of forecast uncertainty
• High-resolution global ensemble of assimilation and forecasts coupling the ocean, sea-ice, land and
atmosphere
– Supporting Copernicus Services
• Climate monitoring services for the atmosphere, hydrological and carbon cycle
• European Reanalysis (currently producing ERA-5)
• Atmospheric composition monitoring
• Emergency alert system (Floods, Fires, …)
48EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
51. 18%
60%
17%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD
Example: TCo7999 L62 (~1.25km)
Where do we spent the time ?
The cost profile of a 1.25km (non-hydrostatic) IFS atmosphere simulation Piz Daint
4880 MPI tasks x 12 threads
32 FC/day ~ 0.088 SYPD
single precision / FLT
63% comms;
37% compute
Based on the Piz Daint, Swiss
Cray XC50 Haswell, Aries
interconnect, ~5000 nodes total
~191.74 MWh / SY
52. The IFS model grid
52
N24 reduced Gaussian grid N24 octahedral Gaussian grid
A further ~20% reduction in gridpoints
=> ~50% less points compared to full grid
(Wedi et al, 2014, 2015; Malardel et al, 2015; Deconinck et al, 2017)
Integrated Forecasting System (IFS)