Thesis Projects @ VU University Amsterdam and IBM Amsterdam

•

1 gefällt mir•685 views

JOIN our team
• Dr. Watson choose
Medical Synonyms Game Your
• Medical Detective Thesis
Gamiﬁcation of Medical project
Diagnosis

• To CATCH a Thief
SPAM detection in
call now:
lora.aroyo
crowdsourcing with no Gold
Standard @vu.nl
Monday, December 10, 12

Weitere ähnliche Inhalte

Andere mochten auch

For more information visit our website: http://chip-project.org This is a presentation of a MSc thesis by Ivo Roes, performed within the CHIP project, entitled: Personalized Museum Tour with Real-Time Adaptation on a Mobile Device with Multi-Point Touch Interface The CHIP project is a collaborative effort between the Rijksmuseum Amsterdam, VU University Amsterdam and Eindhoven University of Technology http://chip-project.org

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

WebSci2013 Harnessing Disagreement in Crowdsourcing

WebSci2013 Harnessing Disagreement in Crowdsourcing

WebSci2013 Harnessing Disagreement in Crowdsourcing

The Agora project is a collaboration between the History and Computer Science departments at the VU University Amsterdam, the Rijksmuseum Amsterdam and the Dutch national audiovisual archive Beeld en Geluid. The aim of Agora is to develop a social platform in which museum objects can be placed into an explicit (art)historic context. Through the (art)historic context, objects from highly diverse museum collections can be related, resulting in a more complete and illustrated description of historical events. End-users will also be allowed to create their own personal narratives which will lead to theoretical reflection on the meaning of digitally mediated public history in contemporary society. Check out our website http://agora.cs.vu.nl/ and our twitter feed @agora_project

Agora User Committee Meeting 2013

Agora User Committee Meeting 2013

Agora User Committee Meeting 2013

Finding relevant multimedia content is notoriously difficult, and the difficulty increases with the size and heterogeneity of the content collection. Linked cultural media collections are heterogeneous by nature and rapidly increase in size, mainly through enormous amounts of user-generated content and metadata that are placed on the Internet on a daily basis. Without mechanisms for keeping any part of these collections easily accessible by any user at any time and any use context, the value of these collections for the community will drop, just like their value as an economic asset. demo: http://2-dot-rma-accurator.appspot.com/#Intro website: http://sealincmedia.wordpress.com/

SealincMedia Accurator Demos

SealincMedia Accurator Demos

SealincMedia Accurator Demos

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

CrowdTruth @VU Faculty Colloquium (June 2015)

CrowdTruth @VU Faculty Colloquium (June 2015)

CrowdTruth @VU Faculty Colloquium (June 2015)

Tutorial on Question Answering Systems

Tutorial on Question Answering Systems

Tutorial on Question Answering Systems

Saeedeh Shekarpour

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Tasnim Ara Islam

Deep Learning Models for Question Answering

Deep Learning Models for Question Answering

Deep Learning Models for Question Answering

From TREC to Watson: is open domain question answering a solved problem?

From TREC to Watson: is open domain question answering a solved problem?

From TREC to Watson: is open domain question answering a solved problem?

Constantin Orasan

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

TEDx Manchester: AI & The Future of Work

TEDx Manchester: AI & The Future of Work

TEDx Manchester: AI & The Future of Work

Build Features, Not Apps

Build Features, Not Apps

Build Features, Not Apps

Natasha Murashev

Andere mochten auch (14)

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...

WebSci2013 Harnessing Disagreement in Crowdsourcing

WebSci2013 Harnessing Disagreement in Crowdsourcing

WebSci2013 Harnessing Disagreement in Crowdsourcing

Agora User Committee Meeting 2013

Agora User Committee Meeting 2013

Agora User Committee Meeting 2013

SealincMedia Accurator Demos

SealincMedia Accurator Demos

SealincMedia Accurator Demos

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

TEDx Navesink 2015: to be AND not to be - Quantum Intelligence

CrowdTruth @VU Faculty Colloquium (June 2015)

CrowdTruth @VU Faculty Colloquium (June 2015)

CrowdTruth @VU Faculty Colloquium (June 2015)

Tutorial on Question Answering Systems

Tutorial on Question Answering Systems

Tutorial on Question Answering Systems

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Presentation of Domain Specific Question Answering System Using N-gram Approach.

Deep Learning Models for Question Answering

Deep Learning Models for Question Answering

Deep Learning Models for Question Answering

From TREC to Watson: is open domain question answering a solved problem?

From TREC to Watson: is open domain question answering a solved problem?

From TREC to Watson: is open domain question answering a solved problem?

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

Question Answering over Linked Data: Challenges, Approaches & Trends (Tutoria...

TEDx Manchester: AI & The Future of Work

TEDx Manchester: AI & The Future of Work

TEDx Manchester: AI & The Future of Work

Build Features, Not Apps

Build Features, Not Apps

Build Features, Not Apps

Mehr von Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

Harnessing Human Semantics at Scale (updated)

Harnessing Human Semantics at Scale (updated)

Harnessing Human Semantics at Scale (updated)

Data excellence: Better data for better AI

Data excellence: Better data for better AI

Data excellence: Better data for better AI

CHIP Demonstrator presentation @ CATCH Symposium

CHIP Demonstrator presentation @ CATCH Symposium

CHIP Demonstrator presentation @ CATCH Symposium

Semantic Web Challenge: CHIP Demonstrator

Semantic Web Challenge: CHIP Demonstrator

Semantic Web Challenge: CHIP Demonstrator

Presentation at ISWC2018: http://iswc2018.semanticweb.org/sessions/the-rijksmuseum-collection-as-linked-data/ of our paper published originally in the Semantic Web Journal: http://www.semantic-web-journal.net/content/rijksmuseum-collection-linked-data-2 Many museums are currently providing online access to their collections. The state of the art research in the last decade shows that it is beneficial for institutions to provide their datasets as Linked Data in order to achieve easy cross-referencing, interlinking and integration. In this paper, we present the Rijksmuseum linked dataset (accessible at http://datahub.io/dataset/rijksmuseum), along with collection and vocabulary statistics, as well as lessons learned from the process of converting the collection to Linked Data. The version of March 2016 contains over 350,000 objects, including detailed descriptions and high-quality images released under a public domain license.

The Rijksmuseum Collection as Linked Data

The Rijksmuseum Collection as Linked Data

The Rijksmuseum Collection as Linked Data

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Presentation at the NYC Media Lab (NYCML2018). There is a growing demand for news videos online, with more consumers preferring to watch the news than read or listen to it. On the publisher side, there is a growing effort to use video summarization technology in order to create easy-to-consume previews (trailers) for different types of broadcast programs. How can we measure the quality of video summaries and their potential to misinform? This workshop will inform participants about automatic video summarization algorithms and how to produce more “representative” video summaries. The research presented is from the FAIRview project and is supported by the Digital News Innovation Fund (DNI Fund), which is part of the Google News Initiative.

FAIRview: Responsible Video Summarization @NYCML'18

FAIRview: Responsible Video Summarization @NYCML'18

FAIRview: Responsible Video Summarization @NYCML'18

Understanding bias in video news & news filtering algorithms

Understanding bias in video news & news filtering algorithms

Understanding bias in video news & news filtering algorithms

StorySourcing: Telling Stories with Humans & Machines

StorySourcing: Telling Stories with Humans & Machines

StorySourcing: Telling Stories with Humans & Machines

Data Science with Humans in the Loop

Data Science with Humans in the Loop

Data Science with Humans in the Loop

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Digital Humanities Benelux 2017: Keynote Lora Aroyo

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to the volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, this assumption often creates issues in practice. Previous experiments we performed found that inter-annotator disagreement is usually never captured, either because the number of annotators is too small to capture the full diversity of opinion, or because the crowd data is aggregated with metrics that enforce consensus, such as majority vote. These practices create artificial data that is neither general nor reflects the ambiguity inherent in the data. To address these issues, we proposed the method for crowdsourcing ground truth by harnessing inter-annotator disagreement. We present an alternative approach for crowdsourcing ground truth data that, instead of enforcing an agreement between annotators, captures the ambiguity inherent in semantic annotation through the use of disagreement-aware metrics for aggregating crowdsourcing responses. Based on this principle, we have implemented the CrowdTruth framework for machine-human computation, that first introduced the disagreement-aware metrics and built a pipeline to process crowdsourcing data with these metrics. In this paper, we apply the CrowdTruth methodology to collect data over a set of diverse tasks: medical relation extraction, Twitter event identification, news event extraction and sound interpretation. We prove that capturing disagreement is essential for acquiring a high-quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with a majority vote, a method which enforces consensus among annotators. By applying our analysis over a set of diverse tasks we show that, even though ambiguity manifests differently depending on the task, our theory of inter-annotator disagreement as a property of ambiguity is generalizable.

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Ambiguity in interpreting signs is not a new idea, yet the vast majority of research in machine interpretation of signals such as speech, language, images, video, audio, etc., tend to ignore ambiguity. This is evidenced by the fact that metrics for quality of machine understanding rely on a ground truth, in which each instance (a sentence, a photo, a sound clip, etc) is assigned a discrete label, or set of labels, and the machine’s prediction for that instance is compared to the label to determine if it is correct. This determination yields the familiar precision, recall, accuracy, and f-measure metrics, but clearly presupposes that this determination can be made. CrowdTruth is a form of collective intelligence based on a vector representation that accommodates diverse interpretation perspectives and encourages human annotators to disagree with each other, in order to expose latent elements such as ambiguity and worker quality. In other words, CrowdTruth assumes that when annotators disagree on how to label an example, it is because the example is ambiguous, the worker isn’t doing the right thing, or the task itself is not clear. In previous work on CrowdTruth, the focus was on how the disagreement signals from low quality workers and from unclear tasks can be isolated. Recently, we observed that disagreement can also signal ambiguity. The basic hypothesis is that, if workers disagree on the correct label for an example, then it will be more diﬃcult for a machine to classify that example. The elaborate data analysis to determine if the source of the disagreement is ambiguity supports our intuition that low clarity signals ambiguity, while high clarity sentences quite obviously express one or more of the target relations. In this talk I will share the experiences and lessons learned on the path to understanding diversity in human interpretation and the ways to capture it as ground truth to enable machines to deal with such diversity.

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

Software systems are becoming ever more intelligent and more useful, but the way we interact with these machines too often reveals that they don’t actually understand people. Knowledge Representation and Semantic Web focus on the scientific challenges involved in providing human knowledge in machine-readable form. However, we observe that various types of human knowledge cannot yet be captured by machines, especially when dealing with wide ranges of real-world tasks and contexts. The key scientific challenge is to provide an approach to capturing human knowledge in a way that is scalable and adequate to real-world needs. Human Computation has begun to scientifically study how human intelligence at scale can be used to methodologically improve machine-based knowledge and data management. My research is focusing on understanding human computation for improving how machine-based systems can acquire, capture and harness human knowledge and thus become even more intelligent. In this talk I will show how the CrowdTruth framework (http://crowdtruth.org) facilitates data collection, processing and analytics of human computation knowledge. Some project links: - http://controcurator.org/ - http://crowdtruth.org/ - http://diveproject.beeldengeluid.nl/ - http://vu-amsterdam-web-media-group.github.io/linkflows/

Data Science with Human in the Loop @Faculty of Science #Leiden University

Data Science with Human in the Loop @Faculty of Science #Leiden University

Data Science with Human in the Loop @Faculty of Science #Leiden University

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

"Video Killed the Radio Star": From MTV to Snapchat

"Video Killed the Radio Star": From MTV to Snapchat

"Video Killed the Radio Star": From MTV to Snapchat

Mehr von Lora Aroyo (20)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

Harnessing Human Semantics at Scale (updated)

Harnessing Human Semantics at Scale (updated)

Harnessing Human Semantics at Scale (updated)

Data excellence: Better data for better AI

Data excellence: Better data for better AI

Data excellence: Better data for better AI

CHIP Demonstrator presentation @ CATCH Symposium

CHIP Demonstrator presentation @ CATCH Symposium

CHIP Demonstrator presentation @ CATCH Symposium

Semantic Web Challenge: CHIP Demonstrator

Semantic Web Challenge: CHIP Demonstrator

Semantic Web Challenge: CHIP Demonstrator

The Rijksmuseum Collection as Linked Data

The Rijksmuseum Collection as Linked Data

The Rijksmuseum Collection as Linked Data

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

FAIRview: Responsible Video Summarization @NYCML'18

FAIRview: Responsible Video Summarization @NYCML'18

FAIRview: Responsible Video Summarization @NYCML'18

Understanding bias in video news & news filtering algorithms

Understanding bias in video news & news filtering algorithms

Understanding bias in video news & news filtering algorithms

StorySourcing: Telling Stories with Humans & Machines

StorySourcing: Telling Stories with Humans & Machines

StorySourcing: Telling Stories with Humans & Machines

Data Science with Humans in the Loop

Data Science with Humans in the Loop

Data Science with Humans in the Loop

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Digital Humanities Benelux 2017: Keynote Lora Aroyo

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

Data Science with Human in the Loop @Faculty of Science #Leiden University

Data Science with Human in the Loop @Faculty of Science #Leiden University

Data Science with Human in the Loop @Faculty of Science #Leiden University

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

"Video Killed the Radio Star": From MTV to Snapchat

"Video Killed the Radio Star": From MTV to Snapchat

"Video Killed the Radio Star": From MTV to Snapchat

Thesis Projects @ VU University Amsterdam and IBM Amsterdam

1. JOIN our team • Dr. Watson choose Medical Synonyms Game Your • Medical Detective Thesis Gamiﬁcation of Medical project Diagnosis • To CATCH a Thief SPAM detection in call now: lora.aroyo crowdsourcing with no Gold Standard @vu.nl Monday, December 10, 12