Deep Learning for Social Media Image Filtering During Crises
•
6 likes•934 views
The document describes an automatic image filtering pipeline for social media images posted during crises. The pipeline uses deep learning and perceptual hashing to (1) filter out irrelevant and duplicate images, (2) categorize remaining images into damage levels (severe, mild, little-to-none), and (3) save over 60% of annotation budget by reducing image volume. Evaluation shows the relevancy filter achieves 0.98 AUC and 0.98 F1, while damage assessment achieves 0.72 AUC and 0.67 F1. The pipeline aims to extract actionable information from images in real-time to assist crisis response.
Recommended
Recommended
More Related Content
Similar to Deep Learning for Social Media Image Filtering During Crises
Similar to Deep Learning for Social Media Image Filtering During Crises (20)
Recently uploaded
Recently uploaded (20)
Deep Learning for Social Media Image Filtering During Crises
- 7. Challenges of Social Media Image Processing • High velocity and high volume – Crowd cannot keep up • Sandy Hurricane – 20 millions tweets in 5 days – 16 thousands tweets per minute Background
- 8. Challenges of Social Media Image Processing • High velocity and high volume – Crowd cannot keep up • Sandy Hurricane – 20 millions tweets in 5 days – 16 thousands tweets per minute • Large por>on of redundant/duplicate and irrelevant content – We should not waste crowd’s >me and effort Background
- 13. Related Works • Very few: – Images with on-topic tweets are more relevant (Peters and Joao, 2015) – Visually relevant and irrelevant tweets (F1 70.5% in binary classifica>on task) (Chen et al., 2013) – Damage level from aerial Images (Turker et al., 2004; Fernandez et al. 2015) • Mostly used bag-of-visual-word features Background
- 16. Data: Damage Assessment Corpus Annota'on: • Severe damage: – Substan>al destruc>on of an infrastructure – Non-usable buildings, roads, bridges for example. • Mild damage: – Minor damages with up to 50% • LiQle-to-no damage Number of labeled images for each dataset and each damage category
- 17. Data: Relevancy Detec>on Corpus Annota'on Mapping: • Images has been selected from the data we annotated for damage assessment – {severe, mild} à relevant (randomly sampled 3518 images) • Resulted 7036 Images {relevant, irrelevant} Image annotated as LiQle-to-None ImageNet Object Detec'on Model Object category website, suit, lab coat, envelope, dust jacket, candle, menu, vestment, monitor, street sign, puzzle, television, cash machine, screen irrelevant Most frequently occurring Human Supervision
- 23. Relevancy Filtering (Cont.) Image Processing Pipeline • Deep Learning Model: Build a binary classifier – Transfer learning – Extracted features from VGG16, a pre-trained convolu>onal neural network, e.g., VGG16 (Simonyan et al., 2014 ) – Fine tuned the parameters 1x2 Image credit: hIps://blog.heuritech.com
- 24. Relevancy Filtering (Cont.) * Simonyan, K. and Zisserman, A. (2014). “Very deep convolu>onal networks for large-scale image recogni>on”. In: arXiv preprint arXiv:1409.1556 Image Processing Pipeline • Data Split for Experiments: – Training 60% – Valida>on 20% – Test 20% • Performance
- 27. Duplicate Filtering (Cont.) Image Processing Pipeline • Selected 1100 images for the experiment • Approach: – Compute perceptual hash for each image – Compute hamming distance between a pair of images Two images are duplicate if: distance(Vi, Vj) 10 Vi = pHash(Imagei) Vi ∈ Rn distance(Vi, Vj) = n k=0 |(Vi,k − Vj,k)| Vi = pHash(Imagei) Vi ∈ Rn distance(Vi, Vj) = n k=0 |(Vi,k − Vj,k)| hamming distance(Vi, Vj) = n k=1 (Vi,k ̸= Vj,k) (Lei et al. 2011)
- 33. Damage Assessment • Three-class classifica>on – Categories: severe, mild & liIle-to-none • Dis>nc>on between categories is ambiguous. Image Processing Pipeline Descrip'on Severe Mild None All Number of Images 1765 483 3751 6000
- 36. Summary • We propose a real >me image processing pipeline, which filters – duplicate, near duplicate, and – irrelevant images and categories – images into severe, mild and liIle-to-no damage • Relevancy filter model: AUC: 0.98; F1: 0.98 • Damage assessment model: AUC: 0.72; F1: 0.67 • We can save 62% of the whole budget Summary and Future Works 62% images are irrelevant and duplicates D. T. Nguyen, F. Alam, F. Ofli, M. Imran, “Automa>c image filtering on social networks using deep learning and perceptual hashing during crises”, ISCRAM 2017.
- 37. Future Work • Image processing pipeline – Please check: hIp://aidr.qcri.org • Fine-grained object analysis – Severity of injury, quality of aid/shelter, etc. • Mul>modal analysis – text + image + metadata Summary and Future Works