Lecture at the 2011 European Summer School on Information Retrieval (ESSIR 2011). 31st August 2011.

1. A brief introduction to extracting information from images Jonathon Hare University of Southampton

2. What can images tell us? How are images represented in digital computers How do we extract information from images Examples of some different extraction techniques Analogies with text Free software! Contents

3. Images CAN… the main roles of images in the communications process

4. Attract attention and make documents more appealing

6. Convey opinions and emotional messages

8. Convey information for documenting a claim

10. Representation and Understanding how a computer “sees”

11. digital image Representation 87 91 85 ... 86 86 81 ... 88 85 84 ... ... ... ... ... 137 145 144 ... 153 150 137 ... 148 139 123 ... ... ... ... ... 89 91 89 ... 84 88 90 ... 88 87 90 ... ... ... ... ...

12. Understanding An Image

13. Feature Extraction f(x) Feature extraction is the process of extracting “descriptors” from an image. Descriptors describe some aspect of the image content. Typically, a descriptor is a numerical vector called a “feature vector”, however other forms of descriptor are possible.

14. Higher-level features Directly interpretable by humans i.e. the number of faces in the image Either hand-crafted or trained with machine learning techniques Lower-level features Much more abstract; convey a notion of the image content i.e. the colour distribution of the image IMAGE Feature Morphology

15. Example High-Level Features faces, composition & photoshop disasters

16. The detection of faces in an image is a very useful feature for inferring information about an image Face detection is the first step of face recognition The most popular face detection algorithm is the “Viola-Jones” detector Conceptually simple Uses machine learning; Requires training (slow). Very fast detection High-level features: face Detection

17. Viola-Jones face Detection Bank of filters. Consider all possible position, scale and type parameters (very large numbers of features) For each feature create a simple (weak) binary classifier (a stump) Use ADABOOST to select the informative features P. Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. (first version appeared at CVPR 2001)

18. Viola-Jones face Detection P. Viola, M. Jones, Robust Real-Time Face Detection, IJCV, Vol. 57(2), 2004. (first version appeared at CVPR 2001)

19. Photographers use the “rule-of-thirds” to improve the composition of their photos. The basic idea is to place main subjects at roughly one-third of the horizontal or vertical dimension of the photograph. High-level features: Composition

20. High-level features: Composition It is possible to design features that look for the presence of composition using the rule-of thirds image saliency map segments + saliency map distance to closest power-point area of segment * saliency of segment Che-HuaYeh, Yuan-Chen Ho, Brian A. Barsky, and Ming Ouhyoung. "Personalized Photograph Ranking and Selection System". In ACM Multimedia 2010, pages 211–220, October 2010.

21. High-level features: Tampering

22. High-level features: Tampering

23. High-level features: Tampering A Political Advertisement for George W. Bush Automatic cloning detection (“copy-move” forgery)

24. Example LOW-Level Features colour histograms, segments and sift

25. Global features describe the content of an entire image One of the simplest global features is the “Global RGB Colour Histogram” Quantise each pixel into a discrete number of colours and then build a histogram. Low-Level Features: Global

26. Global features are useful for some tasks, but in many cases are not powerful enough Local features attempt to overcome this by breaking the image into smaller parts from which to extract features Three primary techniques for splitting up the image Low-level features: Local segmentation salient regions & interest points grids & blocks

27. Salient interest regions and their associated features are currently the most popular way of describing an image content. Extracting image features using interest regions is a two-part process: Find regions Extract feature to describe region properties Typically, the resultant image feature will have a variable length, dependent on the number of regions Salient interest regions

28. Important regions portray: Repeatability Saliency Corners and blobs have these qualities Detectable using various techniques Difference of Gaussian - corners Harris corner detector - corners MSER - blobs Salient interest region Location corners blobs

29. Good region descriptors portray: Resilience to image transforms Compactness Emphasise different image characteristics: Pixel intensities, colour, texture, edges etc. Common descriptors include: SIFT: histogram of edge orientation Shape context: histogram of edge location Salient interest region descriptors

30. SIFT: Scale Invariant Feature Transform

31. Analogies with text introducing the visual bag-of-words

32. In the computer vision community over recent years it has become popular to model the content of an image in a similar way to a “bag-of-terms” in textual document analysis. Bags of Visual Words

33. Features localised by a robust region detector and described by a local descriptor such as SIFT. A vocabulary of exemplar feature-vectors is learnt. Traditionally through k-means clustering. Local descriptors can then be quantised to discrete visual terms by finding the closest exemplar in the vocabulary. BoVW using local features

34. BOVW models have many applications Auto-annotation and object recognition Concept classification Large-scale indexing Applications of BOVW

35. open-source tools for image analysis and indexing introducing openimaj & imageterrier

36. http://www.openimaj.org Open-source (BSD Licence) libraries and tools for multimedia (image, video, sound) analysis and information extraction Implemented in Java; use with any JVM language Implementations of all the techniques mentioned in this tutorial Scalability of extraction using Hadoop with the included tools

37. http://www.imageterrier.org Extension to the Terrier retrieval system to allow indexing of images Collections and documents that read data produced from image feature extractors. New indexers and supporting classes to make compressed augmented inverted indices for visual term data. New distance measures implemented as WeightingModels. Geometric re-ranking implemented as DocumentScoreModifiers. Command-line tools for indexing and searching. Freely available under the Mozilla Licence