Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

A Framework to build Games with a Purpose for Linked Data Refinement

presentation of the Resources track paper at ISWC 2018 - Monterey

  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

A Framework to build Games with a Purpose for Linked Data Refinement

  1. 1. A Framework to build Games with a Purpose for Linked Data Refinement Gloria Re Calegari, Andrea Fiano and Irene Celino Monterey, October 11th, 2018 – 17th International Semantic Web Conference (ISWC 2018)
  2. 2. MOTIVATION: LINKED DATA REFINEMENT • Knowledge Graph Refinement (and, in general, linked data refinement) is an emerging and hot topic to (1) identify and correct errors and (2) add missing knowledge, often by means of statistical learning and/or machine learning • Machine learning approaches train automatic models on the basis of a training set, thus they require some partial gold standard, often also named “ground truth” • Ground truth is usually put together manually by experts, but sourcing training sets from humans is expensive! A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 2 Building a training set for linked data refinement = = asking people to execute a set of linked data refinement tasks
  3. 3. • Crowdsourcing is the process to outsource tasks to a “crowd” of distributed people (notable examples: Amazon Mechanical Turk, Figure Eight) • Human Computation is a computer science technique in which a computational process is performed by outsourcing certain steps to humans, usually when humans are very good at solving those tasks while computers are not (notable example: reCAPTCHA) • Games with a Purpose (GWAP) are a Human Computation application that lets to outsource some tasks to humans in an entertaining way (notable example: the ESP game) • Crowdsourcing, Human Computation and GWAPs have been used in the Semantic Web community for several tasks related to linked data management and refinement • Collection, verification, ranking, correction, … of resources, triples, ontologies, alignments, … • Those approaches differ in engagement and reward schemes for human participants APPROACH: CROWDSOURCING & HUMAN COMPUTATION A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 3 premium access money prizes knowledge recognition fun enjoyment
  4. 4. RESOURCE: GWAP ENABLER • This presentation is about the GWAP Enabler, which is a software framework to build Games with a Purpose designed to solve some linked data refinement task by involving participants as players • Why such a framework? • Designed to solve a specific family of tasks (linked data refinement) • Implements a simple yet effective game mechanics • Predefined aggregation algorithm for contributions (truth inference) • Coding effort saving!!! A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 4 execute linked data refinement tasks play games and have fun produce training set (ground truth) used to train machine learning models for automatic linked data refinement
  5. 5. lrs ro p • Data linking is the creation of links in the form of RDF triples (subject, predicate, object) • Within the same dataset (i.e. generating new connections between resources of the same dataset or knowledge graph) • Across different datasets (i.e. creating RDF links, as named in the linked data world) • Cases of Data Linking addressed in the GWAP Enabler: • Link creation: a link l = (rs,p,ro) is created A link score σ ∈ [0,1]can be attached to each existing/created link, indicating the confidence on the truth value of the link • Link ranking: a score σ ∈ [0,1] is assigned to each link l, representing the probability of the link to be recognized as true; links are ordered by their score σ (ranking) • Link validation: a score σ ∈ [0,1] is assigned to each link l, representing the actual truth value of the link; a threshold t ∈ [0,1] is tells apart true (σ ≥ t) from false links FAMILY OF TASKS: DATA LINKING A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 5 l1 l2 σ1 > σ2 σ1 > t “true” σ2 < t “false” lrs ro p σ l1 l2
  6. 6. INSIDE THE GWAP ENABLER: APPLICATION LOGIC • Game mechanics: • One action = one task: each game action correspond to a task execution, usually related to the processing of one link l = (rs,p,ro) • Double player game: players play in pairs, the game automatically/anonymously couple them • Output agreement: players win if they give the same solution to the same task (so they are motivated to give the “right answer”) • Gold standard tasks: players’ reliability measured through tasks with known solution • Truth inference algorithm • One contribution = one evidence: each player solution is counted as a supporting evidence on the truth value of a link • Weighted contributions: each input increments/decrements the respective score σ of the link, weighted by the respective player’s reliability (measured on the gold standard tasks) • Incremental truth inference: a task is assigned to different players until its score σ overcomes the threshold t ; this allows minimizing the task repetitions and helps identifying difficult tasks A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 6
  7. 7. ARCHITECTURE OF THE GWAP ENABLER A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 7 MySQL PHP API HTML5 + AngularJS + Bootstrap (mobile first design) GWAP API Database Server-side Client-side Tasks to solve Solved tasks https://github.com/STARS4ALL/gwap-enabler https://gwapenablerapi.docs.apiary.io/ Released with the Apache License, Version 2.0 rs l = (rs,p,ro) ro Predefined but highly customizable user interface Dynamically add new data linking tasks or get aggregated solutions Only requirement: resources have some “visual” representation (e.g., title, description, picture, map, …)
  8. 8. 8 • Input: set of all links <asset> foaf:depiction <photo> • Goal: assign score 𝜎 to rank links on their recognisability/ representativeness • The score 𝜎 is a function of Τ𝑋 𝑁 where 𝑋 is the no. of successes (=recognitions) and 𝑁 the no. of trials of the Bernoulli process (guess or not guess) realized by the game • Cultural heritage assets in Milano and their pictures APPLICATIONS: INDOMILANDO http://bit.ly/indomilando LINK RANKING Pure GWAP with hidden purpose Points, badges, leaderboard as intrinsic reward Link ranking is a result of the “agreement” between players The game has also an educational “collateral effect”  Irene Celino, Andrea Fiano, Riccardo Fino. Analysis of a Cultural Heritage Game with a Purpose with an Educational Incentive. 16th International Conference on Web Engineering, 2016 A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018
  9. 9. 9 • Input: set of links <land-area> clc:hasLandCover <land-cover> • Goal: assign score 𝜎 to each link to discover the “right” land cover class • Score 𝜎 of each link is updated on the basis of players’ choices (incremented if link selected, decremented if link not selected) • When the score of a link overcomes the threshold 𝜎 ≥ 𝑡 , the link is considered “true” (and removed from the game) • Two automatic classifications of land cover in disagreement: <land-cover-assigned-by-DUSAF> ≠ <land-cover-assigned-by-GL30> APPLICATIONS: LAND COVER VALIDATION GAME A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 http://bit.ly/foss4game https://youtu.be/Q0ru1hhDM9Q LINK VALIDATION Pure GWAP with not-so-hidden purpose (played by “experts”) Points, badges, leaderboard as intrinsic reward A player scores if he/she guess one of the two disagreeing classifications Link validation is a result of the “agreement” between players Maria Antonia Brovelli, Irene Celino, Andrea Fiano, Monia Elisa Molinari, Vijaycharan Venkatachalam. A crowdsourcing-based game for land cover validation. Applied Geomatics, 2017
  10. 10. 10 • Input: set of subject resources (pictures) and object resources (classification categories) • Goal: create links <picture> hasCategory <category> and assign score 𝜎 to each link • Score 𝜎 of each link is updated on the basis of players’ choices (incremented if link selected) • When the score of a link overcomes the threshold 𝜎 ≥ 𝑡 , the link is considered “true” (and the picture is removed from the game) • Identify pictures of cities from above between those taken on board of the ISS (the pictures are used then in a scientific process in light pollution research) APPLICATIONS: NIGHT KNIGHTS A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 http://nightknights.eu LINK CREATION AND VALIDATION Pure GWAP with not-so-hidden purpose (but played by anybody) Points, badges, leaderboard as intrinsic reward A player scores if he/she agrees with another player “Bonus” intrinsic reward with NASA pictures! Gloria Re Calegari, Gioele Nasi, Irene Celino. Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification. Human Computation Journal, vol. 5, issue 1, 2018. Gloria Re Calegari and Irene Celino: Interplay of Game Incentives, Player Profiles and Task Difficulty in Games with a Purpose, in proceedings of EKAW 2018.
  11. 11. • In order to help the interested potential adopters of the framework, we designed a full end-to-end tutorial to familiarize with the software framework and understand how to build a new GWAP on top of it • The tutorial supports the user step-by-step, both in preparing input data (getting restaurant information from OpenStreetMap) and in adjusting the code to the specific purpose (displaying a map and asking players to classify restaurants) THE GWAP ENABLER TUTORIAL 11A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 https://github.com/STARS4ALL/gwap-enabler-tutorial
  12. 12. EXTERNAL REUSE OF THE GWAP ENABLER: ARGO • The GWAP Enabler was adopted by a research group at the University of Milano Bicocca to build a game to collect analogies, to be used in an Analogical Reasoning experimentation for Knowledge Graph Embeddings • The result is ARGO: Analogical Reasoning Game with a purpOse A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 12 Credits: Francesco Barrera, Federico Bianchi, Matteo Palmonari
  13. 13. CONCLUSIONS AND POSSIBLE EXTENSIONS • The GWAP Enabler is a free and open source software framework • to build Games with a Purpose to solve different and frequent tasks of data linking • to collect valuable training set for linked data refinement with machine learning • to save time and effort in design and development and focus on research  • Possible extensions / next steps • Extend the framework to support additional data linking tasks (e.g. free text input by users) • Experiment in different domains and further challenges (e.g. NLP-related tasks) • Address your feedbacks and requirements!!! A Framework to build Games with a Purpose for Linked Data Refinement - ISWC 2018 13
  14. 14. MILANO viale Sarca 226, 20126, Milano - Italy LONDON 4° floor 57 Rathbone Place London W1T 1JU – UK NEW YORK One Liberty Plaza, 165 Broadway, 23rd Floor, New York City, New York, 10006 USA Cefriel.com A Framework to build Games with a Purpose for Linked Data Refinement Gloria Re Calegari, Andrea Fiano and Irene Celino This work was partially supported by the STARS4ALL project (H2020-688135) co-funded by the European Commission Icons made by Eucalyp from www.flaticon.com Contact me: Irene Celino Head of Knowledge Technologies Group Cefriel - Politecnico di Milano irene.celino@cefriel.com iricelino.org

×