C-IFS: how are developments integrated ?

1. Copernicus
Atmosphere
Monitoring
Service
CAMS General Assembly, Athens, 14-16 June 2016
Anna Agusti Panareda, Samuel Remy, Vincent Huijnen,
J.J-Morcrette, Olaf Stein, Joaquim Arteta, Simon
Chabrillat, Johannes Flemming & Angela Benedetti,
Antje Inness, Sebastien Massart, Richard Engelen as well
as all contributors to IFS and C-IFS
C-IFS: How are
developments integrated

2. 2
IFS : Integrated Forecasting System of ECMWF
A very good NWP forecast and data assimilation model

3. 3
10th anniversary of IFS
1997
IFS : Integrated Forecasting System of ECMWF
A complex model system for forecast and assimilation

4. 4
Adding composition to IFS : Composition -IFS
• In GEMS project:
• Coupled system IFS-MOZART for chemistry
• GHG and aerosol on-line (integrated) in the IFS
• MACC I-III: chemistry on-line in IFS
• Chemistry - IFS (2009)
• Renamed to Composition –IFS: all composition aspects
• Composition – IFS : global production system in CAMS
at ECMWF

5. 5
Integration of chemistry & aerosol modules in IFS
Dynamics & Physics
Chemistry
ctm
Dynamics & Physics
Transport &
Chemistry
oasis4
oasis4
oasis4
IFS IFS CTM
Feedback Flow
Coupled System
Feedback: slow
Flexibility: high
Integrated System
Feedback: fast
Flexibility: low
Coupled System
IFS- MOZART3 / TM5
C-IFS
On-line Integration
Flemming et al. 2009
Flexible
but
very un-
efficient
Fast,
consistent
but
higher
coding effort

6. 6
Composition – IFS : multiple schemes
Composition –
IFS
MOZART
chemistry
Cariolle
Strat. O3
CO2 & CH4
GLOMAP aerosol
MOCAGE
chemistry
CAMS
Procurement
Open IFS
Interface
BASCOE
stratospheric
chemistry
TM5 (CB05)
chemistry
MACC (LMDz)
aerosol
BMS
Strat. O3
MACC III heritage

7. 7
Benefits for CAMS using C-IFS
• IFS is the best NWP model on the planet
• IFS is a very efficient global model
• Operational IFS resolution is currently 9 km globally
• CAMS o-suite resolution is 40 km globally
• IFS data assimilation (4D-VAR, ENS) used for
composition
• Using 4D-Var algorithm (Ensemmble DA)
• Infra structure to process assimilated observations

8. 8
Benefits of high resolution model
Mid-tropospheric CH4 [ppb] at 450 hPa
Low resolution FC (80 km, L60) High resolution FC (16 km, L137)
Anna Agusti-Panareda

9. 9
Challenges to use C- IFS for CAMS
• Adaptation of data assimilation system to specifics of
composition field and observations
• IFS advection does not formally conserve mass
• Global mass fixers implemented
• Link CAMS development with ongoing IFS development
• 2-3 new cycles each year
• Reproducibility of older cycles
• IFS coding standards

10. 10
Towards better integration between C-IFS Components
• Between Chemistry, Aerosols and GHG modules
• Secondary aerosol formation based on chemistry
• Photolysis and surface chemistry modulation by
aerosol
• Unified modelling of methane in Chemistry and GHG
• Code harmonisation
• Composition on NWP (and back !!)
• Aerosol in radiation
• Ozone in radiation
• Land surface and fluxes (emissions and deposition)

11. 11
CAMS ozone fields in IFS radiation scheme I

12. 12
CAMS ozone fields in IFS radiation scheme II
New CAMS
Ozone climatology
used in next IFS
cycle

13. 13
How are C-IFS developments by CAMS partners
integrated …

14. 14
IFS - coding rules
http://intra.ecmwf.int/publications/cms/get/ifs/4663
REAL(KIND=JPRB),INTENT(IN) :: PRR(KLON,NREAC)
REAL(KIND=JPRB),INTENT(IN) :: PRJ(KLON,NPHOTO)
DO JL=KIDIA,KFDIA
ZP1=PRJ(JL,jbno3)*PY(JL,ino3)
……
ENDDO
not a coding rule but advised for efficiency

15. 15
Code efficiency
Use Profiling to find bottle necks
Profiling information for program='/fws2/lb/work/rd/disr/g99u/2014120100/gfc/tmp.g99u_fc_fcgroup1.model.1.32453/ifsMASTER', proc#3:
No. of instrumented routines called : 1254
Instrumentation started : 20160502 143312
Instrumentation ended : 20160503 035310
Instrumentation overhead: 35.69%
Memory usage : 1449 MBytes (heap), 1452 MBytes (rss), 0 MBytes (stack), 0 (paging)
Total CPU-time is 98250.51 sec on proc#3, 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6) (32 procs, 2 threads)
Thread#1: 55730.71 sec (56.72%), 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6)
Thread#2: 42519.80 sec (43.28%), 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6)
# % Time Cumul Self Total # of calls MIPS MFlops Div-% Routine@<thread-id>
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec)
1 13.13 12900.110 12900.110 12900.290 563 0 0 0.0 >MPL-TRGTOL_COMMS ( 803)@1
2 8.53 21278.660 8378.550 16104.600 56683200 0 0 0.0 *UKCA_DCOFF_PAR_AV_K@1
3 8.41 29536.700 8258.040 15986.450 55751976 0 0 0.0 *UKCA_VGRAV_AV_K@2
4 8.37 37761.020 8224.320 16033.160 56683200 0 0 0.0 UKCA_VGRAV_AV_K@1
5 8.33 45946.390 8185.370 15861.140 55751976 0 0 0.0 UKCA_DCOFF_AR_AV_K@2
6 4.73 50594.410 4648.020 37276.250 4048800 0 0 0.0 *UKCA_DDEPAER_INCL_SEDI@1
7 4.58 55095.040 4500.630 36813.330 3982284 0 0 0.0 UKCA_DDEPAER_INCL_SEDI@2
S.Remy, C-IFS GLOMAP profiling
Usage of resources per routine call

16. 16
CAMS
partner
CAMS
ECMWF
IFS team
RD ECMWF
CAMS FD
ECMWF
RD research dep.
FD forecast dep.
e-suite
o-suite
CAMS
VAL
How are C-IFS
developments
integrated ?
Up to 1 year from
development to
o-suite
implementation

17. 17
Thank you!
ευχαριστώ
Tower of Winds
A meteorological monument nearby
with a CAMS theme:
Skiron (NW) distributes the ashes

18. 18
How are C-IFS developments integrated …
1. Contributing partner (or ECMWF):
• Testing (Test A) of individual model development
• Delivery to ECMWF/CAMS
2. CAMS-ECMWF Section:
• Integrate development in CAMS branch
• Testing (Test B) of all integrated model improvements
• Submit to ECMWF RD IFS section for ECMWF cycle upgrade
3. ECMWF RD IFS group
• Merge new cycle from all ECMWF contributions
4. Forecast Department Copernicus section:
• Run experimental CAMS suite (e-suite) and tested by VAL
5. Forecast Department Copernicus section:
• Run operational CAMS suite (o-suite)
• Each of the steps can take 1-3 month so that it takes up to a year month from model
update to implementation in o-suite
• Time line of ECMWF cycle upgrades will be announced to CAMS partners well in
advance

19. 19
Computational Cost C-IFS
16 km40 km80 km