The European Middleware Initiative Advanced services for Distributed Computing Infrastructures

1. The European Middleware Initiative
Advanced services for Distributed Computing Infrastructures
TRINITY COLLEGE DUBLIN Befo
B f r EMI
Befor EMI 3 years
ears After EMI
A
After EMI
Applications
Integrators, System
Administrators
Specialized services,
professional support
and customization
EMI Reference
Services
Standards,
New technologies (clouds)
Users and Infrastructure
Requirements
EMI Middleware Evolution
CERN • www.chern.ch, CESGA • www.cesga.es, CESNET • www.cesnet.cz, CINECA •
EMI brings together the major European that the European middleware evolves in syn-
www.cineca.it, CSIC • www.csic.es, DESY • www.desy.de, FOM-Nikhef • www.nikhef.nl, Middleware providers – ARC, gLite, UNICORE chrony with its global environment.
and dCache – to establish a sustainable model
JUELICH • www.fz-juelich.de, GRNET • www.grnet.gr, INFN • www.infn.it, Lund University • for the support, harmonisation and evolution of The consolidated services resulting from
www.lu.se, NIIF • www.niif.hu, STFC • www.stfc.ac.uk, SWITCH • www.switch.ch, core middleware services for deployment in the EMI will close the gap between different kinds
European Grid Infrastructure (EGI), the Part- of infrastructures, allowing end users to access
Trinity College Dublin • www.tcd.ie, Technische Universität Dresden • www.tu-dresden.de, nership for Advanced Computing in Europe resources such as high throughput compute
(PRACE) and other computing and data infra- farms and advanced super computers in a co-
University of Copenhagen • www.ku.dk, Helsinki Institute of Physics • www.hip.it, University of structures. herent, secure and seamless fashion, offering
Oslo • www.uio.no, UPJS • www.upjs.sk, Uppsala Universitet • www.uu.se, ICM • the potential for deploying new scientific appli-
EMI is working with other global middleware cations.
icm.edu.pl, KISTI • www.kisti.re.kr, ASGC • www.twgrid.org providers and technology partners to ensure

2. EMI-Data
The project implements practices com- ment activities are articulated by Product The fundamentals of EMI-Data are to reduce These data will be eventually collected and pro-
mon in commercial software development; Team, within four main technical areas: expenses in operating large storage manage- cessed , significantly reducing operational costs.
in particular, practical support and develop- Compute, Data, Infrastructure and Security. ment by standards adoption. EMI-Data will
combine the production-proofed components EMI
Experiment EMI SE
specific
coming from the extensive experience of ARC, LFC
catalogues
gLite, UNICORE, with the dCache Storage
Namespace
EMI-Compute Technology, to form a powerful and consistent API FTS
storage software distribution. Catalogues
Monitoring
Current job management services have been through heterogeneous mechanisms. The com- U
API
designed and implemented using different and pute team will also consolidate the different • Namespace integration enabled by POSIX N ARC/
Scheduled Data Transfer
I gLite
incompatible approaches, often referring to pro- clients, to improve usability, maintainability and access to data via NFS 4.1 (pNFS) protocol, Namespace
Synchronisation C
O Worker
by Message
prietary solutions. To address this issue, EMI- portability. with consistency ensured between the different Passing R
E
Node
(EMI Infrastructure)
Compute is defining the interface for an EMI storage, metadata and data-location services.
execution service (EMI-ES) that will be imple- Finally, EMI-compute will address the execution • All data available via Web protocols http(s) Central EMI
Monitoring System
mented in these services. By means of a single of parallel and MPI jobs, providing a standard and WebDav.
job description language and a common job unified layer, targeted to the end user and • Fine grained remote data management con-
management interface, differences among responsible for managing parallel applications. trol with Storage Resource Manager (SRM) Namespace
API
NFS 4.1 WebDAV FTP
Access Protocols
Monitoring
API
different job management services will be trans- protocol. Access Layer
parent to users, enabling true interoperability More at www.eu-emi.eu/Compute • Remote protocols generally secured by X509/
dCache DPM
SSL web authentication standard. STORM
Storage Layer
EMI-Security Data Components will report accounting GPFS/
Lustre
information in a common format, offering fine Disks Disks Disks
The innovations in Grid security will focus on ware, and supporting both TLS/SSLv3 with grained billing of storage and transfer requests.
improvements in usability and transparency for standard X.509 certificates as well as HTTP EMI Storage Element
users and a transition to a common security authentication (username/password) methods. As all EMI Component, monitoring informa- Tape Storage Tape Storage
solution based on industry standards. Currently tion will be provided through a standard inter-
there are different Authorization mechanisms
Argus Integration and Deployment face, respectively API. More at www.eu-emi.eu/Data
Argus as a site service to manage consitent authorization
used throughout the middleware. As Harmoni- policy based decisions.
zation and Evolution steps within the middle-
ware a single Authorization service that provides PAP PDP EMI-Infrastructure
user-friendly features will be integrated through-
PEP daemon
out. This is Argus, a site central service that can Global EMI-Infrastructure manages the middleware providers that allow the retrieval of Grid services
Banning
administer and enforce authorization policies List PAP services that provide common information and status information and service state manage-
in a hierarchical manner. All EMI components, management functionality to deployed Grid ment; the Logging and Bookkeeping services
including job and data management services,
EES gridmap
services. that allows collecting, aggregating and archiving
will benefit of this centralization of policies avoid- Site job execution information; the accounting func-
ing inconsistent authorization decisions, thus Administrator
These include the Information System and tionality to collect, distribute and publish infor-
easing service deployment and maintenance. Service Registry, which provide a view of the mation concerning the usage of resources. This
PEP client
PEP client
PEP client
services available on the Grid together with area also deals with internal infrastructure com-
Batch
Standardization of mechanisms will be pur- CREAM
BLAH gLExec
System
gLExec Data their characteristics; the messaging infrastruc- ponents, such as service containers that are
sued for authentication purposes too, through WN Management ture, that allows to collect and distribute mes- required for middleware services.
the adoption of a common set of libraries provid- sages generated by Grid services or user
ing a consistent set of decisions to the middle- More at www.eu-emi.eu/Security tasks; the service monitoring and management More at www.eu-emi.eu/Infra