Linked Data publishing on the Web is a stably growing phenomenon, but its effective usage depends on the ability of consumers to assess the trustworthiness and the relevance of the published data. Pure automatic techniques are often inadequate to this end. Crowdsourcing is often advocated as a valuable solution. In this presentation, we propose WikiFinder – a Games With A Purpose inspired by popular mobile puzzle games – and we report on its effectiveness in solving typical Linked Data Management tasks.

1. Emanuele Della Valle
DEIB - Politecnico di Milano
http://emanueledellavalle.org
Joint work with
• Irene Celino
CEFRIEL
• Roberto Gualandris
Politecnico di Milano
ON THE EFFECTIVENESS
OF A MOBILE PUZZLE
GAME UI TO
CROWDSOURCE
LINKED DATA
MANAGEMENT TASKS
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 1

2. Agenda
• Context: linked data
• Problem space: assess trustworthiness and relevance of linked data
• Solution space
• Automatic solutions
• Crowdsourcing alternative solutions (Focus on Game with a Purpose)
• Contribution: WikiFinder
• Analysis of results obtained with WikiFinder
• Distinguishing true from false facts
• Identifying relevant facts
• Other tasks
• Discussion
• Conclusions and future works
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 2

3. Context
Linked Data publishing: a growing phenomenon
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 3

4. Problem space
Assess trustworthiness and relevance of data
• Are the following two statements true?
• The airport of Milano Malpensa is in Milan
• http://dbpedia.org/page/Malpensa_Airport
dbpedia-owl:city
dbpedia:Milan
• Leornardo da Vinci was born in Vinci
• http://dbpedia.org/page/Leonardo_da_Vinci
dbpedia-owl:birthPlace
dbpedia:Vinci,_Tuscany
• Which is more relevant?
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 4
No, It is the airport of
Milan, but it is faraway
Yes, Vinci is a small
town in Tuscany
:-O

5. CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 5
State of the art solution
Automatic techniques can fail
• Are the following two statements true?
• The airport of Milano Malpensa is in Milano
• A map based system can provide the answer
• Leornardo da Vinci was born in Vinci
• Data quality tools and instance matching may give the right answer
• Which is more relevant?
• No heuristic for or dataset of human association strengths between Linked
Data entities is available

6. Innovative solution
Crowdsourcing
[source: http://blog.optimyth.com/wp-content/uploads/2013/01/microtask_crowdsourcing_forums1.jpg ]
Trustworthy
& relevant
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 6

7. Innovative solution
Game With a Purpose and Linked Data 1/3
Jörn Hees, Thomas Roth-Berghofer, Ralf Biedert, Benjamin Adrian, Andreas Dengel:
BetterRelations: Collecting Association Strengths for Linked Data Triples with a
Game. SeCO Book 2012: 223-239
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 7
Subject
Property
Object
Property
Object
BetterRelations

8. Innovative solution
Game With a Purpose and Linked Data 2/3
Jörn Hees, Mohamed Khamis, Ralf Biedert, Slim Abdennadher, Andreas Dengel:
Collecting Links between Entities Ranked by Human Association Strengths. ESWC
2013: 517-531
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 8
Subject
related to
Object
Object
Knowledge Test Game

9. CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 9
Innovative solution
Game With a Purpose and Linked Data 3/3
• Other similar approaches
• Siorpaes, K., Hepp, M.: OntoGame: Towards Overcoming the Incentive
Bottle- neck in Ontology Building. In: Proc. of the 3rd International IFIP
Workshop On Semantic Web & Web Semantics (SWWS), Vilamoura,
Portugal, Springer, LNCS (2007)
• Kny, E., Kölle, S., Töpper, G., Wittmers, E.: WhoKnows? (October 2010)
• Wolf, L., Knuth, M., Osterhoff, J., Sack, H.: RISQ! Renowned Individuals
Semantic Quiz – A Jeopardy like Quiz Game for Ranking Facts. In: Proc. of
the I-SEMANTICS, Graz, Austria, pp. 71–78. ACM (2011)
• Broader approach
• E. Simperl, B. Norton, and D. Vrandecic. Crowdsourcing Tasks within
Linked Data Management. In Proceedings of COLD2011, volume 782. CEUR-
WS.org, 2011.

10. CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 10
Contribution
Mobile puzzle games with a purpose 1/3
Subject Property
Object
Alternative 1

11. CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 11
Contribution
Mobile puzzle games with a purpose 2/3
Alternative 2
Property
Object
Subject

12. CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 12
Contribution
Mobile puzzle games with a purpose 3/3
Property
Object
Subject
Alternative 3

13. Contribution
WikiFinder: a mobile puzzle game with a purpose
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 13

14. Contribution
WikiFinder: building the puzzle
True triples
False triples
Negation as failure
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 14

15. Analysis
Statistics about WikiFinder
• Players: 44
• Game played: 440
• Triples proposed: 1127
• True triples: 246
• False triples: 881
• Total hours played: 7,3 hours
• Average life play per user: 10 minutes
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 15

16. Analysis
Distinguishing between true and false triples
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 16

17. Analysis
Triples “popularity” – cf. with manual rank
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 17

18. Analysis
Accuracy on property domain
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 18

19. Analysis
Accuracy on property range
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 19

20. Analysis
Wrapping up
• WikiFinder is effective in
• distinguishing true triples from
false ones
• identifying relevant triples
• identifying the domain of the
predicate
• WikiFinder is not effective in
• Identifying the range of the
predicates
• A bias introduced by the game?
• Shall the player be allowed to play
also from right to left?
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 20

21. Conclusions and future work
• WikiFinder approach
• is effective for some Linked Data Management tasks
• is worth further investigation
• Next steps
• Improve the game play
• Implement the other alternatives
• Focus the puzzle content
• Broaden the evaluation
• Run comparative evaluations against alternative mobile puzzle games
• Run comparative evaluations against other approaches
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 21

22. Irene Celino
CEFRIEL
Emanuele Della Valle
DEIB - Politecnico di Milano
Roberto Gualandris
Politecnico di Milano
THANK YOU FOR PAYING
ATTENTION
ANY QUESTION?
CrowdUI Workshop, AVI 2014, May 27, 2014 Emanuele Della Valle - http://emanueledellavalle.org 22