This presentation on understanding Images and Video detection is part of the ARCOMEM training curriculum. Feel free to roam around or contact us on Twitter via @arcomem to learn more about ARCOMEM training on archiving Social Media.

1. ARCOMEM Training Material
Understanding Images and Video

2. ARCOMEM Training Material
Introduction
• The web is increasingly multimedia in nature
• We concentrate on the multimedia aspects
• Image and video content analysis, indexing,
matching and annotation
• Exploit the web, social web and linked data
web

3. ARCOMEM Training Material
Exploring image and video reuse on
the Web

4. ARCOMEM Training Material
• Goal
– Use image analysis techniques to aggregate social contexts; in
particular to interlink the social discussion of events and topics
through the content of images and video.
• Motivation
– Media is often reused or reposted on social networks.
– Detection of near-duplicate multimedia artifacts provides a means to
investigate and explore many facets about the documents the media is
embedded within.
• Some examples include:
– Aggregating documents about the same subject/event/opinion
» Finding cases where media is used in differing contexts is also interesting.
– Exploring how different social groups talk about the same media

5. ARCOMEM Training Material

6. ARCOMEM Training Material
ARCOMEM US-Elections Crawl
Examples

7. ARCOMEM Training Material
Images
• image/gif 4371
• image/jpeg 667073
• image/png 46644
• image/svg+xml 1
• image/x-icon 2
• image/x-ms-bmp 167

8. ARCOMEM Training Material
122 Detected Duplicates
https://sphotos-a.xx.fbcdn.net/hphotos-ash3/63089_10151299959267177_1113898282_n.jpg
http://p.twimg.com/A7Oq4JRCEAA05uG.jpg
https://p.twimg.com/A7RMKgrCIAAvPdj.jpg
http://24.media.tumblr.com/tumblr_md4ttulK8d1qa5ex8o1_500.jpg
https://p.twimg.com/A7Oq4JRCEAA05uG.jpg
https://p.twimg.com/A7JYdoDCEAAktAb.jpg
https://p.twimg.com/A7JIdp7CcAAjsPF.jpg
http://p.twimg.com/A7JSUYCCUAEzW5O.jpg
http://p.twimg.com/A7NoTNOCIAEQ3ie.jpg
https://p.twimg.com/A7LsnVrCQAAWYFA.jpg
...
112 Detected Duplicates
http://25.media.tumblr.com/tumblr_mc0mmx2kTA1rxxq3ro1_500.png
http://p.twimg.com/A7Ep0CjCMAAon-f.jpg
http://p.twimg.com/A7N3lxrCAAE_gzP.jpg
https://p.twimg.com/A5ctYtVCMAE8_8e.jpg
http://p.twimg.com/A7NMvVACUAEL9aa.jpg
http://p.twimg.com/A7EkZK8CEAE0pYu.jpg
https://p.twimg.com/A7FOSwtCUAADws1.jpg
http://p.twimg.com/A7EiFd4CcAAZdT1.jpg
http://p.twimg.com/A7JwHHbCQAAvqNN.jpg
...

9. ARCOMEM Training Material
103 Detected Duplicates
https://p.twimg.com/A7K8WWiCAAE9Uap.png
https://p.twimg.com/A7Iq1DSCQAAyRBG.jpg
http://p.twimg.com/A7L7bQJCIAIaaqm.jpg
http://p.twimg.com/A7KSC5gCcAAcLTj.jpg
https://p.twimg.com/A7OqzarCYAA8L9N.jpg
https://p.twimg.com/A7OlKn-CQAA-j7X.jpg
https://p.twimg.com/A7J5CHPCUAEp-86.png
https://p.twimg.com/A7JOd9kCcAIy3YM.jpg
http://25.media.tumblr.com/tumblr_lyr1v4kp8J1qcjsjlo1_500.jpg
...
106 Detected Duplicates
https://p.twimg.com/A7MQiKeCMAE1rmX.jpg
https://p.twimg.com/A7IFNt_CUAAqYFV.jpg
https://p.twimg.com/A7DrPLMCIAAoKb8.jpg
http://p.twimg.com/A7Y5BW9CQAAqF3E.jpg
https://p.twimg.com/A2YzbudCQAE77pr.jpg
https://p.twimg.com/A7FQF7PCUAAgWWH.jpg
https://p.twimg.com/A7EO5YFCMAAMTxB.jpg
http://p.twimg.com/A7IWHTtCUAIAtH8.jpg
http://p.twimg.com/A7FcyZdCUAAO_Vg.jpg
http://25.media.tumblr.com/tumblr_m7c9nivzwF1qfep67o1_500.jpg
...

10. ARCOMEM Training Material

11. ARCOMEM Training Material
Multimedia Opinion Mining

12. ARCOMEM Training Material
Goals
• Investigate the use of facial analysis to classify
facial expressions in images and videos found on
the web.
– Can be used to indicate emotion of subject.
• Investigate course-grained automatic
classification using image features
– For abstract opinion-related concepts
(sentiment/privacy/attractiveness)
• Investigate correlations between images and
opinions mined from text
– Does the same image get reused in different
documents to illustrate the same (or different)
opinion?

13. ARCOMEM Training Material
Sentiment/privacy/attractiveness
• Experimental support for classifying visual
media with respect to sentiment, privacy and
attractiveness is being built into ARCOMEM.

14. ARCOMEM Training Material
Image-opinion correlation
• Correlations between images and opinions
extracted from the text can be explored by
querying the ARCOMEM database.
+ve -ve

15. ARCOMEM Training Material
Towards Facial Analysis in the Wild
• Detection and analysis of faces in multimedia
content can help us guide and contextualise a
crawl:
– Recognition and expression analysis can help us
determine if an image is relevant or interesting.
– Post-crawl the information can be used for visualisation.
• Current research very much based on images
taken in lab-conditions; how far can we take it?

16. ARCOMEM Training Material
Applying the model to crawled images