Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Lora Aroyo
DATA SCIENCE FOR
SMART CULTUR...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
TAKE HOME MESSAGE
2
data is essential to...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Information
Traditionally Heritage Organ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
CULTURAL HERITAGE
4
Before the Digital A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Interpretation
But they aim to be seen a...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DIGITAL HERITAGE
6
Bringing collections ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Knowledge Representation, Taxonomies, Th...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Linked Data, Semantic Web, Interoperabil...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
2005 - 2007 http://multimedian.project.c...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
2005 - 2007 http://multimedian.project.c...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
2005 - 2007 http://multimedian.project.c...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
2005 - 2007 http://multimedian.project.c...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
2005 - 2007 http://multimedian.project.c...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
RIJKSMUSEUM LINKED DATA
Using Linked Dat...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Linked Data, Semantic Web, Interoperabil...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
ADDRESSED THE WEB ACCESS & SCALE ISSUES ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
André Malraux, The Imaginary Museum of W...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Gravity (2013)
LOST IN CULTURAL SPACE MO...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 19
AT THE OTHER END IN THE MEDIA WORLD ....
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 20
AT SOME POINT IT ALL LOOKS THE SAME
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 21
CECI N'EST PAS … LA MONA LISA
Museum ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 22
CECI N'EST PAS … LA MONA LISA
Museum ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 23
Museum Website ...
CECI N'EST PAS … L...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 24
CECI N'EST PAS … LA MONA LISA
Online ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 25
IT’S A JUNGLE OUT THERE ...
Lots of m...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 26
IN THE VERY NEAR FUTURE
most visitors...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 27
1998 2006 2007 2009 2013
FROM DVD DIS...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 28
NETFLIX: BIG & DEEP DATA FOR ENGAGEME...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 29
NETFLIX: BIG & DEEP DATA FOR ENGAGEME...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 30
FROM BOOKS TO DATA-DRIVEN BUSINESS
19...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 31
DATA IS ESSENTIAL TO EVOLVE WITH YOUR...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 32
DATA AT THE CENTER OF EVERY PROCESS
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 33
… BUT
understanding your data is cruc...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 34
AS IT HAS ...
variety of meanings
mul...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 35
COMFORT ZONE
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 36
One truth: knowledge acquisition for ...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 37
DOES THIS IMAGE DEPICT A WOMAN?
YES o...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 38
DOES THIS IMAGE DEPICT A WOMAN?
YES o...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 39
DOES THIS IMAGE DEPICT A WOMAN?
YES o...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 40
WHAT DO EXPERTS SAY?
✓ ✓
Does This Im...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 41
WHAT DOES A LAY ANNOTATOR SAY?
Does T...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 42
WHAT DOES ANOTHER LAY ANNOTATOR SAY?
...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 43
WHAT DOES A THIRD LAY ANNOTATOR SAY?
...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 44
WHAT DOES THE CROWD SAY?
This applies...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
THE WORLD IS SMOOTH AND NOT BINARY
“Trut...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DISAGREEMENT IS SIGNAL
Variety of source...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
a spatial representation of truth & mean...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
gathering diversity of perspectives & op...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
COMFORT ZONE DISRUPTED
49
Encourage Disa...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
On the role of user-generated metadata i...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
VIDEO METADATA ENRICHMENT
The Netherland...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DIVE+
Explorative Search
DIVE into the e...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DEEP QA IN CULTURAL HERITAGE
Mauritshuis...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
NetwerkOorlogsbronnen
Linked Data & Crow...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
ADDING EVENTS TO THE NOB THESAURUS
Linke...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
EVENTS THESAURUS
Linked Data & Crowdsour...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
PERSONAL EVENTS
Linked Data & Crowdsourc...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
PEOPLE PORTAL
Linked Data & Crowdsourcin...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
CROWDTRUTH IN THE WILD ...
GOOGLE MAPS
59
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
CROWDTRUTH IN THE WILD ...
New York Time...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Chris Dijkshoorn, Victor De Boer, Lora A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Chris Dijkshoorn, Victor De Boer, Lora A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Chris Dijkshoorn, Victor De Boer, Lora A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Chris Dijkshoorn, Victor De Boer, Lora A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DigiBird: on the fly collection integrat...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DigiBird: on the fly collection integrat...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DigiBird: on the fly collection integrat...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DigiBird: on the fly collection integrat...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Chris Dijkshoorn, Victor De Boer, Lora A...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
632.953 artworks - 411.745 Rijksstudios
...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… BUT TOO MUCH FOCUS ON EXPERTS
71
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… MORE EXPERT VIEWS
72
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… AND MORE EXPERT VIEWS
73
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… AND EVEN MORE EXPERT VIEWS
74
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… AND EVEN MORE EXPERT VIEWS
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… WHILE USERS CAN TELL SO MUCH
76
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
… AND THEY TELL DIFFERENT STORIES
77
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
DATA SCIENCE WITH RIJKSSTUDIO COULD PROV...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
SUCCESS STORIES: OPEN DATA AT RIJKSMUSEU...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
SUCCESS STORIES: BEYOND THE RIJKSMUSEUM
...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
TRACK & PROMOTE CREATIVITY OF INDIVIDUAL...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
TAKE HOME MESSAGE
82
data is essential t...
http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org
Lora Aroyo
HARNESSING THE POWER OF
CROWD...
Nächste SlideShare
Wird geladen in …5
×

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

516 Aufrufe

Veröffentlicht am

My keynote at the ICAL2018 Conference @ Rijksmuseum

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

  1. 1. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Lora Aroyo DATA SCIENCE FOR SMART CULTURAL HERITAGE
  2. 2. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org TAKE HOME MESSAGE 2 data is essential to evolve with your users data should be at the center of every process there is no single notion of truth but a spectrum of context, opinions, perspectives & shades of grey harnessing the full spectrum of truth from experts & users creates more opportunities for serendipity, creativity & engagement
  3. 3. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Information Traditionally Heritage Organizations are seen as Inventories of the World André Malraux, The Imaginary Museum of World Sculpture, 1953 3
  4. 4. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org CULTURAL HERITAGE 4 Before the Digital Age Lots of manual effort Focus on internal collection management Focus on art historical significance Access targeted to researchers & professionals Small curated selection online for general audiences onsite
  5. 5. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Interpretation But they aim to be seen as a Places for People to Engage with the World André Malraux, The Imaginary Museum of World Sculpture, 1953 5
  6. 6. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DIGITAL HERITAGE 6 Bringing collections online Focus on massive digitization of heritage collections Getting large collections online Still need significant art historical understanding to get access Metadata not sufficient for the online presence
  7. 7. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Knowledge Representation, Taxonomies, Thesauri METADATA ENRICHMENT Shared structured knowledge Guus Schreiber, et al (2000). The CommonKADS Methodology 7
  8. 8. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Linked Data, Semantic Web, Interoperability, Standards METADATA ENRICHMENT Shift from metadata for internal use to metadata for online access Michiel Hildebrand, http://e-culture.multimedian.nl, 2009 8
  9. 9. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 2005 - 2007 http://multimedian.project.cwi.nl/ 9
  10. 10. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 2005 - 2007 http://multimedian.project.cwi.nl/ 10
  11. 11. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 2005 - 2007 http://multimedian.project.cwi.nl/ 11
  12. 12. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 2005 - 2007 http://multimedian.project.cwi.nl/ 12
  13. 13. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 2005 - 2007 http://multimedian.project.cwi.nl/ 13
  14. 14. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org RIJKSMUSEUM LINKED DATA Using Linked Data to Diversify Search Results a Case Study in Cultural Heritage Chris Dijkshoorn, Lora Aroyo, Guus Schreiber, Jan Wielemaker, and Lizzy Jongma, 2014 14
  15. 15. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Linked Data, Semantic Web, Interoperability, Standards METADATA ENRICHMENT Building community for shared knowledge creation, use & maintenance 2014, http://www.getty.edu/research/tools/vocabularies/lod/index.html 15
  16. 16. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org ADDRESSED THE WEB ACCESS & SCALE ISSUES ... through using automated methods to enrich & curate metadata André Malraux, The Imaginary Museum of World Sculpture, 1953 16
  17. 17. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org André Malraux, The Imaginary Museum of World Sculpture, 1953 BUT THAT WASN’T ENOUGH FOR TRUE ENGAGEMENT Focus on information support rather than interpretation support for online collections 17
  18. 18. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Gravity (2013) LOST IN CULTURAL SPACE MORE THAN EVER The sense of disconnect was now bigger as there has never been so much online information and so difficult to find ... 18
  19. 19. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 19 AT THE OTHER END IN THE MEDIA WORLD ...
  20. 20. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 20 AT SOME POINT IT ALL LOOKS THE SAME
  21. 21. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 21 CECI N'EST PAS … LA MONA LISA Museum experiences decrease in quality - crowded, reservations, timed, etc.
  22. 22. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 22 CECI N'EST PAS … LA MONA LISA Museum Website ...
  23. 23. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 23 Museum Website ... CECI N'EST PAS … LA MONA LISA
  24. 24. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 24 CECI N'EST PAS … LA MONA LISA Online visibility is a struggle - Louvre’s Mona Lisa was #14, now not even in top 50
  25. 25. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 25 IT’S A JUNGLE OUT THERE ... Lots of mobile museum apps - disconnected, static & limited access
  26. 26. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 26 IN THE VERY NEAR FUTURE most visitors will be digital-born not bound by time or location native to new forms of co-makership native to new media Siebe Weide, Max Meijer and Marieke Krabshuis (2012). Agenda 2026: Study on the Future of the Dutch Museum Sector
  27. 27. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 27 1998 2006 2007 2009 2013 FROM DVD DISTRIBUTION TO ORIGINAL CONTENT Granular Recommendations, Personalized Trailers, Posters, Content
  28. 28. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 28 NETFLIX: BIG & DEEP DATA FOR ENGAGEMENT engagement monitoring: - how many users watching show X finished it to the end of season Y? - what did the other users do? - how big of a ‘time gap’ between watching episodes? deep data with tracking EVENTS: - when people pause, rewind, fast forward, leave (and if ever come back) - when people watch; where they watch (zip code) & on what device - what people search for (~ 3 mil per day); how people browse & scroll; deep data within video - “in the moment” characteristics - how much users need to watch in order to be less likely to cancel, e.g. - if users watch at least 15 h/month they are 75% less likely to cancel. - If they drop below 5 hours, there is a 95% chance they will cancel
  29. 29. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 29 NETFLIX: BIG & DEEP DATA FOR ENGAGEMENT 1. Set objectives, pick metrics = shared across whole organization 2. Consider UX as mission-critical 3. Personalize UX as much as possible 4. Understand user’s lifestyle and context 5. Use interaction data then ask for feedback 6. Let users know your service is adapting to their tastes 7. Ensure metadata captures content nuances and is consistent 8. Give reasons to come back often 9. Run frequent UI experiments 10. Close the loop and base your decisions upon data https://www.slideshare.net/PancrazioAuteri/personalization-10-lessons-learned-from-netflix/50-Netflix_solutions_are_applicable_and
  30. 30. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 30 FROM BOOKS TO DATA-DRIVEN BUSINESS 1994 2003 2006 2011 2013
  31. 31. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 31 DATA IS ESSENTIAL TO EVOLVE WITH YOUR USERS
  32. 32. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 32 DATA AT THE CENTER OF EVERY PROCESS
  33. 33. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 33 … BUT understanding your data is crucial
  34. 34. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 34 AS IT HAS ... variety of meanings multitude of perspectives abundance of sources endless applications
  35. 35. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 35 COMFORT ZONE
  36. 36. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 36 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old 7 Myths about Human Annotation BINARY WORLD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  37. 37. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 37 DOES THIS IMAGE DEPICT A WOMAN? YES or NO?
  38. 38. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 38 DOES THIS IMAGE DEPICT A WOMAN? YES or NO?
  39. 39. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 39 DOES THIS IMAGE DEPICT A WOMAN? YES or NO?
  40. 40. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 40 WHAT DO EXPERTS SAY? ✓ ✓ Does This Image Depict A Woman? ✓
  41. 41. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 41 WHAT DOES A LAY ANNOTATOR SAY? Does This Image Depict A Woman? ✓ ✓ ✓
  42. 42. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 42 WHAT DOES ANOTHER LAY ANNOTATOR SAY? Does This Image Depict A Woman? ✘✓ ✘
  43. 43. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 43 WHAT DOES A THIRD LAY ANNOTATOR SAY? Does This Image Depict A Woman? ✓ ✓ ✘
  44. 44. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 44 WHAT DOES THE CROWD SAY? This applies to everything, e.g. interests, popularity, relevance, significance .. & the bigger the crowd the better & better to do it continuously and from various perspectives @ at various granularity levels Does This Image Depict A Woman? 95% 75% 50%
  45. 45. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org THE WORLD IS SMOOTH AND NOT BINARY “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty 45
  46. 46. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DISAGREEMENT IS SIGNAL Variety of sources for disagreement 46
  47. 47. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org a spatial representation of truth & meaning harnesses disagreement CROWDTRUTH.ORG “The Three Sides of CrowdTruth”, Journal of Human Computation 2014, L. Aroyo, C. Welty 47
  48. 48. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org gathering diversity of perspectives & opinions from the crowd expand the expert vocabularies with these provide new type of gold standard for machine intelligence CROWDTRUTH.ORG “The Three Sides of CrowdTruth”, Journal of Human Computation 2014, L. Aroyo, C. Welty 48
  49. 49. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org COMFORT ZONE DISRUPTED 49 Encourage Disagreement in Annotation Tasks & Develop New Quality Metrics Anca Dumitrache, Oana Inel, Benjamin Timmermans, Carlos Ortiz, Robert-Jan Sips, Lora Aroyo, Chris Welty (2018): Empirical Methodology for Crowdsourcing Ground Truth
  50. 50. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision http://waisda.nl
  51. 51. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision http://spotvogel.vroegevogels.vara.nl/ On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 51
  52. 52. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DIVE+ Explorative Search DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics: 52 http://diveplus.beeldengeluid.nl/
  53. 53. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DEEP QA IN CULTURAL HERITAGE Mauritshuis use case 53 Nikita Galinkin, Zoltán Szlávik, Lora Aroyo and Benjamin Timmermans (2017). Catch Them If You Can: A Simulation Study on Malicious Behavior in a Cultural Heritage Question Answering System. The 29th Benelux Conference on Artificial Intelligence (BNAIC 2017)
  54. 54. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org NetwerkOorlogsbronnen Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/ 54
  55. 55. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org ADDING EVENTS TO THE NOB THESAURUS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/ 55
  56. 56. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org EVENTS THESAURUS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/ 56
  57. 57. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org PERSONAL EVENTS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/ 57
  58. 58. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org PEOPLE PORTAL Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/ 58
  59. 59. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org CROWDTRUTH IN THE WILD ... GOOGLE MAPS 59
  60. 60. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org CROWDTRUTH IN THE WILD ... New York Times emotions crowd 60
  61. 61. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING: FINDING NICHES IN THE CROWD Accurator tool: SealincMedia Project http://sealincmedia.wordpress.com 61
  62. 62. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl 62
  63. 63. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl 63
  64. 64. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl 64
  65. 65. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo SUCCESS STORIES: NICHESOURCING EVENTS Part of the SealincMedia Project http://annotate.accurator.nl 65
  66. 66. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo DigiBird Project http://annotate.accurator.nl 66 SUCCESS STORIES: NICHESOURCING EVENTS
  67. 67. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo DigiBird Project http://annotate.accurator.nl 67 SUCCESS STORIES: NICHESOURCING EVENTS
  68. 68. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo DigiBird Project http://annotate.accurator.nl 68 SUCCESS STORIES: NICHESOURCING EVENTS
  69. 69. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage CREATING EXPERTS WITH GAMES Accurator tool http://annotate.accurator.nl 69
  70. 70. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org 632.953 artworks - 411.745 Rijksstudios SUCCESS STORY: RIJKSSTUDIO Crowdsourcing & Engagement https://www.rijksmuseum.nl/en/rijksstudio 70
  71. 71. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … BUT TOO MUCH FOCUS ON EXPERTS 71
  72. 72. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … MORE EXPERT VIEWS 72
  73. 73. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … AND MORE EXPERT VIEWS 73
  74. 74. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … AND EVEN MORE EXPERT VIEWS 74
  75. 75. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … AND EVEN MORE EXPERT VIEWS
  76. 76. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … WHILE USERS CAN TELL SO MUCH 76
  77. 77. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org … AND THEY TELL DIFFERENT STORIES 77
  78. 78. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org DATA SCIENCE WITH RIJKSSTUDIO COULD PROVIDE ... engagement monitoring: - how many users viewing artwork X? - in how many collections is artwork X? - what are the labels users give to their collections where artwork X is? - how many shares / downloads of artwork X? - what follow-up user study can be done with the most popular artworks in rijksstudio? - how to promote the less popular artworks? deep data with tracking EVENTS: - when people engage with the content (and if ever come back) - what day & date & time people engage with content; - where do they view it (zip code) & on what device - what people search for in the rijksstudio / website - how people browse & scroll; - what kind of in-museum engagement can be done for rijksstudio popular users? - how to link to current exhibitions the collections in rijksstudio? 78
  79. 79. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org SUCCESS STORIES: OPEN DATA AT RIJKSMUSEUM Rijksmuseum API 79
  80. 80. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org SUCCESS STORIES: BEYOND THE RIJKSMUSEUM Unleasing Creativity with Open Data 80
  81. 81. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org TRACK & PROMOTE CREATIVITY OF INDIVIDUAL USERS Utilise Users’ Creativity with Open Data … BUT ALSO 81
  82. 82. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org TAKE HOME MESSAGE 82 data is essential to evolve with your users data should be at the center of every process there is no single notion of truth but a spectrum of context, opinions, perspectives & shades of grey harnessing the full spectrum of truth from experts & users creates more opportunities for serendipity, creativity & engagement
  83. 83. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo http:://crowdtruth.org Lora Aroyo HARNESSING THE POWER OF CROWDS & MACHINES FOR SMART CULTURAL HERITAGE

×