cocreation coproduction enablers Cloud Computing public services evaluation quality of coproduced artefacts

1. 1
A Collaborative Environment to Boost Co-
Production of Sustainable Public Services
(co-authored by Diego López-de-Ipiña, Julen Badiola, Daniel Silva, Diego Casado-Mansilla,
Elena Not, Chiara Leonardi, Ana Ortiz-de-Guinea and Igone Porto)
7th International Conference on Smart and Sustainable Technologies (Splitech 2022)
5 – 8 July 2022, https://2022.splitech.org/
Dr. Diego López-de-Ipiña González-de-Artaza
dipina@deusto.es
http://paginaspersonales.deusto.es/dipina
http://www.morelab.deusto.es
@dipina

2. 2
Towards more citizen-centric and sustainable public services
• The INTERLINK H2020 project aims to overcome the barriers that hinder
administrations to reuse and share services with private partners (including
citizens) by combining the advantages of two often opposed approaches:
“bottom-up” approach in which
citizens self-organize and deliver
grassroot services where government
plays no active role in day-to-day
activities but may provide a facilitating
framework
“top-down” approach where
Government holds primary
responsibility for creating these
services compliant with EU directives,
sometimes seeking the support of
citizens for specific design or delivery
tasks

3. 3
INTERLINK design goals
• COLLABORATION & RE-USE
▪ The INTERLINK platform offers a digital environment that facilitates
co-production processes between Public Administrations, private
stakeholders and citizens and promotes the re-use of software for
delivery of public services.
• CO-DESIGN & CO-DELIVERY
▪ INTERLINK provides a step-by-step guidance for the co-design and co-
delivery of public services along with guidelines, tips and templates
that facilitate the collaboration of different actors.
• INTERLINKERs
▪ Pieces of knowledge or software that your team can re-use and
customize to deliver services.

4. 4
Teaming up co-producers: quadruple helix

5. 5
INTERLINK co-production methodology

6. 6
INTERLINKERs: coproduction enablers
• Common building blocks, provided as software tools or in the form of knowledge
offered digitally, that offer interoperable, re-usable, EU-compliant, standardized
functionality for public service co-production
– Devised to support the co-production of effective, participatory, and sustainable public services
• Applicable to the following purposes:
– To guide co-production: Co-production enablers that guide and support teams in the
collaborative execution of the co-production initiatives.
– To build capacity: Partnership tools and knowledge resources, which tackle the legal, social, &
business aspects to make co-delivered public services viable and feasible
– To aid service development: Technical enablers for co-delivered services, aligned with other
existing EU-wide initiatives to foster interoperable and sustainable public services.

7. 7
INTERLINK Collaborative Environment
• Designed to support the co-production methodology of INTERLINK and
facilitate its adoption and use in the co-production of novel public services
• It offers the following core functionalities:
– Co-producer team and project management;
– Guide for co-production process;
– Recommendation of INTERLINKERs most suitable to the problem profiles represented by
the chosen co-production task;
– Selection and registry of use (displaying result of using the enabler, e.g. instantiation of a
Business Plan) and
– Access to INTERLINKER catalogue
• Designed to maximize extensibility, adaptation and reusability by making use
of declarative models for INTERLINKERs and co-production schemas

8. 8
INTERLINKER catalogue
• This component provides a one-stop-shop for KNOW-
HOW enabling co-production.
– Populated with knowledge and software INTERLINKERs
leveraging resources from other projects
• Items can be filtered according to strings associated
to their metadata, to their nature (software or
knowledge), problem profile and their ranking.

9. 9
Extending Collaborative Environment: new co-production schemas &
INTERLINKERs
• JSON Schemas have been defined to
declaratively define Software and Knowledge
INTERLINKERs & co-production processes
• New co-production processes can be declared
to adjust to the specifics/purpose of a
concrete co-production endeavour:
– E.g., Hackathon organization and celebration or co-
refinement of public service descriptions
• Purpose-specific INTERLINKERs can be
defined and then recommended, e.g.
Hackathon Rules Template for INTERLINK
project

10. 10
Software INTERLINKER integration with Collaborative Environment
URI Method Description
/ GET redirects to swagger / redoc DOCS
/assets POST
[OPTIONAL] Posts data for asset creation and
return JSON of asset
/assets/instantiate GET GUI for asset creation
/assets/{ASSET_ID} GET JSON data of asset
/assets/{ASSET_ID} DELETE Deletes asset and returns No content
/assets/{ASSET_ID}/downl
oad
GET Download a representation of asset
/assets/{ASSET_ID}/view GET GUI for the interaction with the asset
/assets/{ASSET_ID}/clone POST
[OPTIONAL] Clones the asset and returns
JSON data

11. 11
Collaborative Environment: extendible and
customizable co-production environment

12. 12
Evaluation & Assessment
• Research question: “Will co-production model and supporting tools and co-produced assets in
INTERLINK enhance the quality, quantity, and reuse of public services among European public
administrations (PAs)?”
– INTERLINK evaluates the usability, acceptability, and adoption of the Collaborative Environment as a key
enabler for the co-design and co-delivery of public services.
• INTERLINK is going to be tested in 3 cross-European pilots:
– Ministry of Economy and Finance - Italy (MEF) – a mock-up of a Participatory Strategic Planning Module
(PSPM) which allows Public Bodies and staff to actively participate in the definition of the Strategic Plans
– Ministry of Environmental Protection and Regional Development of the Republic of Latvia (VARAM) and
its Latvian State Portal (https://latvija.lv/EN), which provides easy access to services delivered by state and
local government institutions.
– Zaragoza city (ZGZ) and its Center for Art and Technology (eTOPIA), where the aim is promoting
collaborative city-making facilities and programs and improving the process of Open Innovation

13. 13
Evaluation & Assessment strategy
• 2 iterations per pilot:
• 2 sub-phases per phase:
April - May 2022
(M16 - M17)
PRE-PILOT PILOT EXECUTION
PILOT ITERATION 2
CONSOLIDATION
PRE-PILOT PILOT EXECUTION
PILOT ITERATION 1
EXPLORATION
June - September 2022
(M18 - M21)
March- April 2023
(M27 - M28)
May - Sept 2023
(M29 - M33)
PRE-PILOT PILOT EXECUTION
GOAL: Instantiation of INTERLINK platform and
testing
TESTING GROUP: Selected group of testers
GOAL: Pilot open to all stakeholders (INTERLINKERs and services published and
available to authorized users)
TESTING GROUP: Open set of users: citizens, public administrations and local
businesses (up to 200 users are expected)
ITERATION I (M16 - M21)
April 2022 June 2022 September 2022

14. 14
Evaluation goals
• Global or pilot-agnostic goals:
A. INTERLINK USE and CO-PRODUCTION of SERVICES. The number of INTERLINKERs in use,
stakeholders involved with the INTERLINK solution during the two Pilots, as well as the co-
production of services enabled by INTERLINK, which correspond to the KPIs specified in the
Proposal and their targets
B. THE VALUE PROVIDED by INTERLINK. The value improvements provided by the INTERLINK
solution
C. THE USERS’ PERCEPTIONS of INTERLINK. Users’ perceptions of INTERLINK regarding
acceptance, usability, and trust
• Local or pilot-specific goals:
D. PILOT SPECIFIC KPIs. Custom made at each pilot site

15. 15
Evaluation dimensions and constructs
• INTERLINK pilots aim to assess whether the co-production model & supporting tools and co-
produced assets put forward by the project will enhance the quality, quantity, and reuse of
public services among European public administrations (PAs).
– What will be the degree of ADOPTION achieved by INTERLINK co-production model,
supporting tools and co-produced public services and INTERLINKERs?
Product-based
Quality
• Functional
suitability
• Performance
efficiency
• Compatibility
• Usability
• Reliability
• Security
• Maintanability
• Portability
User-based Quality
• Effectiveness
• Efficiency
• Usefulness
• Satisfaction
• Easy of use
• Flexibility
Value-based
Quality
• Inclusiveness
• Security/Privacy
• Democratic values
• Legitimacy
• Weberian criteria
of public
administration
TRUST >> ACCEPTANCE >> ADOPTION
Quality of co-production process

16. 16
Qualitative and quantitative measures for evaluation gathered at each pilot
• Logging mechanism for understanding user-behaviour in co-production and as end users
• Questionnaires for alpha testers (alpha scripts, activity satisfaction, UEQ and in-app questionnaires)
• Questionnaires for beta testers (for co-producers and end-users)
• KPIs tables: pilot wide and specific metrics
• Cognitive Walkthrough, Heuristic Evaluation, Think-Aloud, In-depth interviews or videos, and other
techniques
Pre-Pilot sub-phase Pilot Execution Phase
Aim → Instantiation of
INTERLINK services and
testing.
Testing group → Controlled
& selected group of alpha
testers (5-10 users/pilot
site)
Aim → Pilot open to all stakeholders (services published and
available to authorized users).
Testing group → open set of users: citizens, public
administration and local businesses (up to 200 users are
expected)
April (M16 M17)- May 2022
(M17)
June 2022 (M18) – September 2022
(M20 M21)
Data logs
Cognitive walkthrough
Heuristic evaluation
Alpha questionnaires
Data logs
In-app questionnaires
Online surveys
Think-aloud methods
Interviews
April (M16)-May (M17) 2022 June (M18)-September (M21) 2022
Pilot Execution sub-phase

17. 17
Conclusions & Further Work
• The INTERLINK collaborative environment has been released in BETA mode and is being tested in the 3
European pilots
• The evaluation strategy and measuring instruments, pilots’ workplan and engagement activitiesare
ongoing to carry out evaluation
• Alpha testing, heuristic and usability studies completed to transform ALPHA into BETA version of
Collaborative Environment
• Assessing whether INTERLINK co-production methodology and associated tools and artefacts contribute to
a wider adoption of co-production as a good practice to deliver more highly acceptable and usable public
services or not.
• Future work:
– Keep enriching the catalogue with new co-production enabling INTERLINKERs.
– Create new exemplary co-production schemas for different purpose co-production projects
– Enhanced usability and integration of co-production guide, timeline, tasks, and resources views to streamline the co-
production of innovative public services will be amended and the project again assessed in a 2nd iteration
– Collaborative process will also be empowered with incentivization and rewarding means through INTERLINK tokens.

18. 18
A Collaborative Environment to Boost Co-
Production of Sustainable Public Services
(co-authored by Diego López-de-Ipiña, Julen Badiola, Daniel Silva, Diego Casado-Mansilla,
Elena Not, Chiara Leonardi, Ana Ortiz-de-Guinea and Igone Porto)
7th International Conference on Smart and Sustainable Technologies (Splitech 2022)
5 – 8 July 2022, https://2022.splitech.org/
Dr. Diego López-de-Ipiña González-de-Artaza
dipina@deusto.es
http://paginaspersonales.deusto.es/dipina
http://www.morelab.deusto.es
@dipina