20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...

Diego Chialva 27-05-2019
Diego Chialva
ERC Executive Agency, Unit A1
Research evaluation:
the unseized opportunities of the open
science era and the possible technology
and methodology approaches
27 May 2019
The European Research Council

Diego Chialva 27-05-2019 │ 2
Outline
Motivation and Perspective
Issues and Opportunities
Technological and Methodological Aspects

Motivation and Perspective

Research and research policy:
evaluation and monitoring
The perspective I will adopt here:
From workshop abstract: “explore mechanisms for research policy monitoring and
indicators, and how to link these to infrastructure and services. [….] first day will focus on
open science indicators”
The perspective here (a slightly tilted one): let’s look at the potentialities from Open Science,
Open Data and the interconnected communication landscape (Internet → Linked (Open) Data)
for monitoring and evaluation
→ effects at large spectrum: data collection, management and analysis, for Open Science
monitoring and evaluation, but also beyond that

Research and research policy:
evaluation and monitoring
OECD, 2009, “Measuring Government Activity”, OECD Publishing, Paris
Monitoring: “a continuing function that uses systematic collection of data on specified
indicators to provide [….] stakeholders of an ongoing [….] intervention with indications
[….]”
Evaluation: “the systematic and objective assessment of an ongoing or completed project,
programme or policy, its design, implementation and results. The aim is to determine the
relevance and fulfillment of objectives, [....] efficiency, effectiveness, impact and
sustainability.”

Issues and Opportunities

Research evaluation: a few notable
features
Two directions in the study for the assessment of efficiency, effectiveness,
impact and sustainability
Vertical: funding-laboratory-society/world/….
Horizontal: bench-marking

Vertical direction: examples
An examples of questions for the “vertical” direction:
what research and research policy contributed most to those medicines and
the related societal improvement?

Horizontal direction: examples
Example of questions for the “horizontal” direction:
how does the results of research activity (or policy!) implemented by A
compare to the activity (or policy) of B?
A B

Monitoring and evaluation and Linked
Open Data
Monitoring and evaluation
Collecting and
processing large
amount of data
A multiplicity
of questions
A multiplicity
of stakeholders
Working with
data from
different sources
Relationships
between data form
complex networks

The issues
Open Science as a paradigm helps in dealing with these issues
(open publications, data, lab notes allow to track and create chains
of evidence).
But relevant issues appear at a more core level.
(And Open Science needs to be monitored and evaluated as well).

The issues
Indeed, the above situation brings upon several (related) issues,
such as:
 data available to a single actor/analyst is limited and often
specific to a single entity/operation
 data processing done in isolation by each actor and on a ad-
hoc basis → often it is re-processing
difficult contextualisation, duplication of efforts

The issues
 data produced/collected/organised by different entities may be
categorised/classified differently
 data models and formats are not standardized
difficult contextualisation, barriers to automation and to
re-use and interoperability of data

Monitoring and Evaluation today
Today: 1) looking at repositories and other sources

Today: 2) processing the raw data

Today: 3) obtaining useful data

Today: 4) perform the analysis and find their connections

Today: somebody else has to do it again for him/herself

Today: but crucial/useful (part of the) data and information are
easily lost/inaccessible

The issues
Overall consequences:
limits to the analysis reach, ineffective evaluation

Monitoring and evaluation and Open
Linked Data
But there are new unseized opportunities and solutions to each of
these issues
Linked Open Data
Semantic Web
Knowledge Graphs

Linked Open Data and the Semantic Web
How do we move toward
Linked Open Data, and in particular the Semantic Web?

Creating Linked Open Data and in particular a Semantic Web is a
task that has technical aspects to it, but which is largely a
responsibility of data owners (research institutes), creators,
curators, and policy makers (that is, overall, data publishers)

Technological and Methodological Aspects

The Semantic Web does not only yield a bunch of interconnected data, but it adds
advantages:
the possibility to uniquely identify a resource (owl:sameAs, skos:**Match)
the possibility to identify/map concepts, classes, specifications
the possibility to locate the resource
information about the resource, and the data itself
relationships and location of related resources that can be queried (federatively)
the elimination of data storage and maintenance on a single site, by a single actor
the possibility to automatise the data collection and analysis

The advantages are bound to Openness and Web
Tim Berner Lee’s four qualities of Semantic Web https://www.w3.org/DesignIssues/
LinkedData.html
1) Use URIs as names for things.
2) Use HTTP URIs so that people can look up those
names.
3) When someone looks up a URI, provide useful
information using the standards
4) Include links to other URIs. so that they can
discover more things.
Metadata, semantics
Link, discover and locate
other data/resources on
different sites
Identify and locate the
resource

Linked Open Data and the Semantic Web:
technology and methodology approaches
For all data publishers and policy analysts: 5-star system road to good data
Available on the web (whatever format) but with an open licence, to be Open Data
Available as machine-readable structured data (e.g. excel instead of image scan
of a table) [→ and also eas-ish to automatise ]
As (2) plus non-proprietary format (e.g. CSV instead of excel)
All the above plus, Use open standards from W3C (RDF and SPARQL) to identify
things, so that people can point at your stuff [→also truly easi-er to automatise ]
All the above, plus: Link your data to other people’s data to provide context

The FAIR principles (a suggestion for good general database practices)
also help, but they are not equivalent to

Linked Data or Semantic Web

Open
[Wilkinson et al., 2016 ; Mons et al., 2017]

The three main steps for a data publisher are:
Map your data using some ontology

an ontology is a sort of “enhanced thesaurus”

ontologies are also a piece of data, so even if I and you make two different ones
for the same concepts, we can link them and clarify semantic
Write out your data in RDF format (“there’s an app for that”)
Publish the data (no need to maintain it alone)
There are toolchains and best practice guidelines for publishing data in
Linked Open Data format (but not enough time to present them here!).

https://lod-cloud.net/

at the ERC
What are we doing for this at the ERC?
Created an ontology for area of interest: DINGO (http://w3id.org/dingo)

extensible interoperable framework

DINGO conceptualizes and expresses relevant parts of the research and
research funding landscape

aims at minimising effort by users: minimum effort for integration in data
systems, satisfy complex requirements by easy extensions

already linked to other ontologies, and also extended for Wikidata use

The DINGO ontology graph

at the ERC
Liased with relevant stakeholders, for example

Wikidata: also creating models/ontologies for related
knowledge/conceptual areas (for impact analysis)

OpenAire: modeling and publishing data on EC research funding

at the ERC
Established best practices and toolchains
Organising a workshop (expected ~ October 2019)

Diego Chialva 27-05-2019 │
Thank you !
diego-valerio.chialva@ec.europa.eu .

20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie 20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...

Ähnlich wie 20190527_Diego Chialva_ Research evaluation: the unseized opportunities ... (20)

Mehr von OpenAIRE

Mehr von OpenAIRE (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...