The EGI Federation celebrates 15 years of distributed computing in 2019. Many milestones were achieved to bring distributed computing from a vision to a real-life international production platform that today enables data-intensive processing at an unprecedented scale, supporting some of the greatest groundbreaking scientific discoveries of the XXI century.

1. www.egi.eu
@EGI_eInfra
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme
Past, present and future of
advanced computing for
data-driven science
Technical Director, EGI Foundation
EGI Conference 2019
Tiziana Ferrari

2. @EGI_eInfrawww.egi.eu 09/05/2019 2
… From design to full scale operations of
the more advanced research computing
infrastructure in the world …

3. @EGI_eInfrawww.egi.eu 09/05/2019 3
1999: Networking as the foundation of
distributed computing
TEN-155 physical topology, April 1999 (Credits: R. Sabatino & J. M. de Arce,
TERENA/NORDUnet Conference 1999)
• In 1999 TEN-155 is the European
network to support co-operative
research
 Predecessors: TEN-34 and EuropaNet
 TEN-155 supersedes these in terms of
capacity offered but also in terms of the
Managed Bandwidth Service (MBS)
offered to guarante bandwidth, for
specific research projects in connected
countries
• GÉANT was born on 1 November 2000

4. @EGI_eInfrawww.egi.eu 09/05/2019 4
• The next generation High Energy
Physics experiments at CERN
become the driver of the
establishment of a Grid
Infrastructure in Europe
2000: A new computational model for
international research collaborations – the Grid
J. Phys. G: Nucl. Part. Phys. 32 (2006) N1–N20

5. @EGI_eInfrawww.egi.eu 09/05/2019 5
The model
Credits: L. Robertson, CHEP 2000
… National funding agencies start mobilizing funding and national
operational structures. Figure: the UK computing Grid for particle physics

6. @EGI_eInfrawww.egi.eu 09/05/2019 6
2001: From Implementation to Operations
The DataGrid EC project proposal is conceived under the
lead of F. Gagliardi/CERN

7. @EGI_eInfrawww.egi.eu 09/05/2019 7
2003: Innovation in federated authentication
and authorization with VOMS
The first VOMS testbed
(EU DataTAG Project)
2005: The first IdP
federation is constituted
with IGTF (the
International Grid Trust
Federation)

8. @EGI_eInfrawww.egi.eu 09/05/2019 8
• 12 regional federations, 27 countries
• The helpdesk – GGUS – is proposed as
central incident Management system
• New implementations of the interoperability
framework – the Grid Middleware -
undergoes major design, implementation
and re-engineering to cope with scale and
user requirements. Standards are introduced
where missing (SRM, GLUE, StAR .. )
• Security frameworks and policies are defined
2003: Operations are in preparatory stage

9. @EGI_eInfrawww.egi.eu 09/05/2019 9
Federated Operations

10. @EGI_eInfrawww.egi.eu 09/05/2019 10
2004: First accounting data is published!
CERN, Netherlands, Spain, Germany… with Canada and the Asia Pacific region
https://accounting.egi.eu/
Q1
Q1
Q2
Q3

11. @EGI_eInfrawww.egi.eu 09/05/2019 11
2010: The EGI Foundation is created
First EGI Council, March 2010

12. @EGI_eInfrawww.egi.eu 09/05/2019 12
• Multi-cloud IaaS with Single Sign-On
• Federation features:
 Common VM image catalogue
 Discovery, accounting, SLO monitoring
 Unified GUI dashboard
May 21, 2014: EGI Federated Cloud is launched
in Helsinki
Cloud Compute
Cloud Container
Compute BETA Training Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA
EGI Services powered by the Cloud Federation

13. @EGI_eInfrawww.egi.eu 09/05/2019 13
EGI Federated Cloud today
IaaS
providers
Federation Services
Orchestration
Platforms
Check-in : Common AuthN and AuthZ across all layers
Research Platforms
Operators
Research Communities
Research Communities

14. @EGI_eInfrawww.egi.eu 09/05/2019 14
Federated Cloud Infrastructure (May 2019)
Finalising integration!New!

15. @EGI_eInfrawww.egi.eu 09/05/2019 15
Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
Direct API
Access
Cloud Architecture
EGI Federation features:
Accounting, Monitoring, Conf. DB, Info Discovery,
AppDB
AppDB VMOps
GUI Access with
(Dashboards &
Data Analytic Platforms
IaaS Federated Access Tools
Federated
Access
Developers/
Advanced users
AAI: Check-in
GUI Users

16. @EGI_eInfrawww.egi.eu 09/05/2019 16
2019: Towards an hybrid cloud
• The EC3/IM of the Applications On Demand platform extended its
framework in order to connect to the HNSciCloud contractors for scaling
applications on the hybrid cloud infrastructure
• CERN and STFC are sponsoring the long-tail of science using the Exoscale
vouchers (1 voucher = 250 EUR worth of compute and storage)
• Integration with OCRE vouchers in 2019 in the context of the EOSC Early Adopter
Programme (https://www.eosc-hub.eu/eosc-early-adopter-programme)
• Federation of Copernicus Data and Information Access Services - DIAS

17. @EGI_eInfrawww.egi.eu 09/05/2019 17
Distributed scientific computing enables
scientific discoveries…
LIGO-VIRGO CollaborationLHC Collaboration
1700 open access publications / year

18. @EGI_eInfrawww.egi.eu 09/05/2019 18
Operational infrastructure behind
the scenes
Tens of operations team members
47 (potential) vulnerabilities reported, 28 advisories/alerts issued
Verified middleware distributions
Training and FitSM certification

19. @EGI_eInfrawww.egi.eu 09/05/2019 19
EGI: State of Play

20. @EGI_eInfrawww.egi.eu 09/05/2019 20
The EGI Federation (May 2019)
4.4 Billion CPU
core wall time
(2018)
> 1 Million
computing
cores in 2019
> 740 PB disk
& tape
2,915 service
end-points

21. @EGI_eInfrawww.egi.eu 09/05/2019 21

22. @EGI_eInfrawww.egi.eu 09/05/2019 22
A global system of e-Infrastructures worldwide

23. @EGI_eInfrawww.egi.eu 09/05/2019 23
European members
May 2019: Collaboration agreement under finalization
with Austria/Tech University of Wien

24. @EGI_eInfrawww.egi.eu 09/05/2019 24
Installed Compute Capacity 2011-2019

25. @EGI_eInfrawww.egi.eu 09/05/2019 25
CPU wall time consumed 2009-2018
+20% annual increase (2018)

26. @EGI_eInfrawww.egi.eu 09/05/2019 26
2 3 4Co-development. Prototype setup Operations
1Early engagement
AGINFRA+ (agricultural sciences)
• Notebooks and Galaxy in Applications on Demand services
• Offered through D4Science
• Pilot runs and assessment in three areas
• Agro-climatic & Economic Modelling
• Food Safety Risk Assessment
• Food Security
NBIS (National Bioinformatics Infrastructure Sweden)
• Contacted EGI via the support channel
• PconsC2, Pcons, TOPCONS2 servers on Federated Cloud –
With SLA
• TOPCONS2 alone predicted more than 2 million protein
sequences by 5000 users from 69 countries
SeaDataNet (ocean observation)
• Testing applicability of DataHub for data access in cloud
• Use cases:
• Migration of legacy applications to the cloud
• Reducing redundant data transfers
• Virtual space for user data storage and delivery
• Interface for distributed search
• Involved: CINECA, CNAF, CYFRONET, IN2P3
SKA (SA)
• Piloting of distributed data access performance in an
international research cloud federation
EGI Research Community Support

27. @EGI_eInfrawww.egi.eu 09/05/2019 27
Support to the Photon/Neutron Ris - PaNOSC
• ESRF (France)
• ILL (France)
• ESS (Sweden)
• ELI-DC (Belgium)
• XFEL (Germany)
• CERIC-ERIC (Italy)
• EGI Foundation (Netherlands)
• Setup an EGI Notebooks instance
• Connect with community-specific AAI
• Connect with community-specific storage

28. @EGI_eInfrawww.egi.eu 09/05/2019 28
NextGEOSS: Contributing to UN Sustainable
Development Goals
Innovative Pilots Business Pilots

29. @EGI_eInfrawww.egi.eu 09/05/2019 29
Scientific Disciplines/HTC
BELLE, DUNE (+16%, 2016), LSST (+267%, 2016), CTA (+57%, 2014),
LIGO-VIRGO (+90%, 2017), HESS (+9%, 2013), SKA (2018)

30. @EGI_eInfrawww.egi.eu 09/05/2019 30
Scientific Disciplines/Cloud
Humanities: CLARIN (2018), DARIAH (+28%, 2017),
Life Sciences: ELIXIR (+104%, 2017)
Environment: EMSO (2018), LifeWatch (+45%, 2015)
BIOISI (+45%, 2016), BIOMED (+575%, 2016), CHIPSTER (+256%, 2016),
GEOHAZARDS (+19%, 2016), OpenCOAST (2018), EXTRAS (2018),

31. @EGI_eInfrawww.egi.eu 09/05/2019 31
The challenges of tomorrow
• Difficult cross-border access due to different funding models, access and
provisioning policies
 Data and service provisioning to international user communities possible only when supported by
sound business models or existing collaboration agreements. Today only a few structured int.
research groups have achieved this.
• Needs of large investments for the creation, processing, preservation, access
and reuse of research data  will the funding match the anticipated needs of
future data-intensive science?
 Opportunities for economies of scale and aggregation of demand can arise with joint provisioning
of infrastructure common components
• Major separation between data preservation and data exploitation
infrastructures in many disciplines
 Ris and e-Infrastructures should collaborate to support the entire research workflow of an
experiment

32. @EGI_eInfrawww.egi.eu 09/05/2019 32
Towards the European Open Science Cloud
A federation of e-Infrastructure and Research Infrastructure
facilities offering
research data, computing, applications and other open science
resources, responding to the problem of scalable access to
research data through a new data provisioning service
approach that is complementary to the traditional data
download model.
Provide access to the data & data products close to processing
facilities while avoiding duplication of local data

33. @EGI_eInfrawww.egi.eu 09/05/2019 33
The federated infrastructure and supporting initiative
providing
all researchers, innovators, companies and citizens
with seamless access to an open-by-default, efficient and
cross-disciplinary environment
for storing, accessing, reusing data, tools, publications and
other scientific outputs for research, innovation and
educational purposes
European Open Science Cloud

34. @EGI_eInfrawww.egi.eu 09/05/2019 34
EGI Assets for the EOSC implementation
Federating services, activities, know-howTechnical infrastructure

35. @EGI_eInfrawww.egi.eu 09/05/2019 35
More on Wed 8 May: Closing Plenary!

36. This work by the EGI Foundation
is licensed under a Creative Commons
Attribution 4.0 International License.
Questions?
Thank you
for your attention.
www.egi.eu
@EGI_eInfra
EGI: Advanced Computing for Research
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme