Visual Information Retrieval

Visual Information
Retrieval

Mathias Lux
Klagenfurt University
mlux@itec.uni-klu.ac.at

Agenda
http://www.uni-klu.ac.at

● Introduction & Motivation
● Content Based Image Retrieval
● Indexing & Search
● Selected Applications

2

Motivation

● Things get easier in digital age …
 Taking pictures & recording videos
 Storing thousands of MBs
 Publishing content to the web
 Entertainment at your fingertips

● Just some figures …

3

Digital Imaging Devices
(global) http://www.uni-klu.ac.at

● How many devices exist?

Device # in 2006

digital cameras 400 * 106

camera phones 600 * 106

Source: IDC Study “Expanding Digital Universe” http://www.emc.com/about/destination/digital_universe/

ITEC, Klagenfurt University, Austria 4

Number of Digital Photos
(global) http://www.uni-klu.ac.at

● Estimate 2006
 > 150 billion photos from cameras
 > 100 billion photos from camera phones

● Forecast 2010
 > 500 billion photos
 + increased resolution

Source: IDC Study “Expanding Digital Universe” http://www.emc.com/about/destination/digital_universe/


Digital Imaging Devices
(Germany) http://www.uni-klu.ac.at

Still image cameras sold in Germany (thousands)
9000

8000
analogue
7000

6000 digital
5000

4000

3000

2000

1000

0
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Source: Cewe Factbook, http://www.cewecolor.de


Photo prints market
(Western Europe) http://www.uni-klu.ac.at

● Photo prints forecast (in billions)

analogue:
- labs

digital:
- labs
- printers

Source: Cewe Factbook, http://www.cewecolor.de


Motivation

So how do we actually find images when we
need them?

● Using a clever directory structure?
● Using “sophisticated” applications?

8


9

Motivation

● Or even on the web?
 Flickr …

10


11


12


13


14

Motivation

Satisfied with the results?

● Actually there are some minor problems.

15

Sensory Gap

● Regarding the sensor
● Inability to record the scene

● Example:
 Too few colors, pixels
 Too low light, too small memory
 Too few fps

16

What is so special about
Semantic Gap
Mona Lisa’s smile?

● Inability of computers to interpret the
scene

17

Semantic Gap

● Limited understanding of computers
● Inability to interpret image content

18

Semantic & Sensory Gap

19

What is VIR?

It’s about finding an automated solutions to
the problem of finding and retrieving
visual information (images, videos) from
(large, distributed, unstructured)
repositories in a way that satisfies the
search criteria specified by their users,
relying (primarily) on the visual contents
of the media.

20

What is the problem
with VIR? http://www.uni-klu.ac.at

The fundamental difficulty in doing what we
want to do is related to the need to
encode, perceive, convey, and measure
similarity (e.g. between two images)

21

Similarity

● Are these two images similar?

taken from [Eidenberger 2004]

22

Similarity

● Which of the small images is most similar
to the big one?

23

Dimensions of the
Problem: User http://www.uni-klu.ac.at

From [Datta et al. 2008]

ITEC, Klagenfurt University, Austria – Multimedia Information Systems 24

Dimensions of the
Problem: System http://www.uni-klu.ac.at



Research issues …

26

Agenda


27

Content Based Image
Retrieval (CBIR) http://www.uni-klu.ac.at

● Text & structured text have already been
discussed
 So we leave metadata for today
● Focus on image content
 Given by pixels
 Within a raster
 Each pixel has a color value

28

Images

Real world Digitized

29

Sampling &
Quantization http://www.uni-klu.ac.at

● Size of a captured image:
 # of samples (width*heigth) * # of colors

30

Image Features

● Images are too “big” for retrieval
 Too many pixels & colors
● We need to extract
 The necessary minimum of information
 For meaningful similarity assessment
● Reduce the problem to a “lower
dimensional space”

31

Some numbers describing
the image? http://www.uni-klu.ac.at

(12, 83, 14, 2)

0.58

(18, 24, 11, 1)

32

What is meaningful?

● Reflecting human perception
● Invariant to certain transformations?

?

33

Transformation: Scale

?

34

Transformation:
Translate http://www.uni-klu.ac.at

?

35

Other Constraints: It’s
a metric … http://www.uni-klu.ac.at

● For a dissimilarity measure d(i,j)
 d(i,i)=0 … no dissimilarity for same image
 d(i,j)=d(j,i) … reflexive
 d(i,j)+d(j,k)>=d(i,k) … transitive

36

Common features

● Color histograms
● Dominant colors
● Color distribution
● Color correlogram
● Tamura features
● Edge histogram
● Local features
● Region based features
(CC) by Pixel Addict, flickr.com/photos/pixel_addict/1083928126/

37

Color Histogram

● Count how often which color is used
● Algorithm:
 Allocate int array h with dim = # of colors
 Visit next pixel -> it has color with index i
 Increment h[i]
 IF pixels left THEN goto line 2
● Example: 4 colors, 10*10 pixels
 histogram: [4, 12, 20, 64]


Color Histogram

● Strategies:
 Quantize if too many colors
 Normalize histogram (different image sizes)
 Weight colors according to use case
 Use (part of) color space according to domain
● Distance / Similarity
 Assumption: All images have the same colors
 L1 or L2 is quite common, JD works even better


Color Histogram

● Benefits
 Easy to compute, not depending on pixel order
 Matches human perception quite well
 Quantization allows to scale size of histogram
 Invariant to rotation, translation & reflection
● Disadvantages
 Distribution of colors not taken into account
 Colors might not represent semantics
 Find quantization fitting to domain / perception
 Image scaling might be a problem


Color Histogram

● Example: 4 images, 7 colors
 1: [0, 4, 12, 20, 64, 0, 0]
 2: [66, 4, 12, 20, 0, 0, 0]
 3: [0, 0, 0, 64, 0, 20, 16]
 4: [0, 0, 0, 0, 64, 20, 16]


Color Histogram

Histograms: Dissimilarity d: L1
● 1: [0, 4, 12, 20, 64, 0, 0] ● d(1,2) = 130
● 2: [66, 4, 12, 20, 0, 0, 0] ● d(1,3) = 160
● 3: [0, 0, 0, 64, 0, 20, 16] ● d(1,4) = 52
● 4: [0, 0, 0, 0, 64, 20, 16]


Dominant Color

● Reduce histogram to dominant colors
 e.g. for 64 colors c0-c63:
• image 1: c12 -> 23%, c33 -> 6%, c2 -> 2%
• image 2: c11 -> 43%, c2 -> 12%, c54 -> 10%
● Dissimilarity function in 2 aspects:
 Difference in amount (percentage)
 Difference between colors (c11 vs. c12)
● Further aspects:
 Diversity and distribution


Dominant Color

● Benefits:
 Small feature vectors
 Easily understandable & intuitive
 Invariant to rotation, translation & reflection
● Disadvantages
 Similarity of color pairs no trivial problem


Color Distribution

● Index dominant color in image segment
 e.g. 8*8 = 64 image segments
 feature vector has 64 dimensions
• One for each segment
 color index is the entry on segment dimension
• e.g. 16 colors [2, 0, 3, 3, 8, 4, ...]


Color Distribution

● Similarity
 L1 or L2 are commonly used
● Benefits
 Works fine for many scenarios
• clouds in the sky, portrait photos, etc.
 Mostly invariant to scaling
● Disadvantages
 Rotation, translation & reflection are a problem


Color Correlogram

● Histogram on
 how often specific colors occur
 in the neighbourhood of each other
● Histogram size is (# of colors)^2
 For each color an array of neighboring colors


Color Correlogram

● Extraction algorithm
 Allocate array h[#colors][#colors] all zero
 Visit next pixel p
 For each pixel q in neighborhood of p:
• increment h[color(p)][color(q)]
 IF pixels left THEN goto line 2
● Algorithm is rather slow
 Depends on size of neighborhood
 Typically determined by city block distance


Color Correlogram

● Similarity
 L1 or L2 are commonly used
● Benefits
 Integrates color as well as distribution
 Works fine for many scenarios
 Mostly invariant to rotation & reflection
● Disadvantages
 Find appropriate neighborhood size
 Rather slow indexing / extraction


Color Correlogram

● Auto Color Correlogram
 Just indexing how often color(p) occurs in
neighborhood of pixel p
 Simplifies the histogram to size # of colors


Color Correlogram

● Integrating different pixel features to
correlate
 Gradient Magnitude (intensity of change in
the direction of maximum change)
 Rank (intensity variation within a
neighborhood of a pixel)
 Texturedness (number of pixels exceeding a
certain level in a neighborhood)


Demo: LireDemo

52

Tamura Features

Features describing texture
● Coarseness
● Contrast
● Directionality
● Line-likeness
● Regularity
● Roughness

53

Tamura Features

● Coarseness
 Size of the texture elements
● Contrast
 More or less picture quality
● Directionality
 Focusing on the texture not the image
 Same angle but different orientation is
considered as same directionality

54

Edge Histogram

● Basic texture feature used in MPEG-7
 Divides into 64 sub images
 Classifies directionality of sub images
 Stores directionality values in histogram
● Dissimilarity
 L1-like

55

Edge & Texture Features

● Benefits
 Compact representation
 Captures “overall” texture
 Mostly invariant to scaling
● Disadvantages
 Not very intuitive in all domains
 Not invariant to rotation & translation


Local Features

● Index small sub images
 instead of global image
 e.g. 14x14 or 17x17 pixels
 typically 100-1000
 selection based on
local variance of
gray values
 idea of salience

from Gu et al. 1989: Comparison of Techniques for Measuring Cloud Texture in Remotely Sensed Satellite Meteorological Image Data.

57

Local Features

● Features are too big
 to reduce size PCA is applied
 for instance reduced to 40 dimensions
 still 1000*40*#bins
● Local features histograms
 Clustering a reasonable number of features
 Assigning numbers to clusters
 Create a histogram of clusters

58

Local Features Histogram

59

Local Features

● Benefits
 Work in general better than global features
 Especially good for image classification
 Invariant to translation
● Drawbacks
 Too big features (without clustering)
 Problems with scaling, rotation

60

Region Based Features

● Segmentation of the image
 roughly correlated to the objects in the image
 e.g. based on pixel clustering
● Extraction of features per region
 Note constraints of several features
• minimum size
• rectangular area
● Indexing of regions

61

Region Based Features

● Benefits
 Work better than global features
 Invariant to translation
 Mostly invariant to rotation & scaling
● Drawbacks
 Heavily depends on segmentation
 Segmentation is not a trivial problem

62

Regions of Interest

● Identify interesting patches in images
● Automatic extraction of ROIs
 Top-down, based on a model
 Bottom-up, e.g. stimulus-driven
● Applications
 Image re-targeting
 Image cropping

Src.: Borba, Gamba, Marques and Mayron , “Extraction of salient regions of interest using visual
attention models”, SPIE Conference on Multimedia Content Access: Algorithms and Systems III, 2009


Bottom-Up Visual
Attention http://www.uni-klu.ac.at

● Attention Models
 Find most interesting point in visual scene
 Direct gaze towards this point
 Selective or focal attention or attention for
perception
● Metaphor of a spotlight
 Sweeping the scene
 Highlighting most important parts


Model of Itti, Koch &
Niebur http://www.uni-klu.ac.at

● Biologically inspired
● Three low level dimensions of an image
 Color, orientation and intensity
● Features are extracted in different scales
 This results in feature maps
● Normalization -> conspicuity maps
● Normalization & summing -> saliency map
 Peaks are salient points
Itti, Koch & Niebur, “A Model of Saliency-based Visual
Attention for Rapid Scene Analysis”, PAMI 1998


Model of Itti, Koch &
Niebur http://www.uni-klu.ac.at

● In iterations
 Preserve prominent peaks
 Inhibit small peaks
● Number of iterations decides on the
outcome

Src.: Borba, Gamba, Marques and Mayron , “Extraction of salient regions of interest using visual
attention models”, SPIE Conference on Multimedia Content Access: Algorithms and Systems III, 2009


Model of Stentiford

● Suppress areas of repetitive color patterns
● For each pixel:
 Compare a number of randomly selected pixels
 Based on color in neighbourhood
 High value: low number of similar areas
 Low value: lots of similar areas
● Result added up to saliency map

F. W. M. Stentiford, “An estimator for visual attention through competitive novelty with application to
image compression,” Proc. Picture Coding Symposium, pp 101-104, Seoul, 24-27 April, 2001.


Model of Stentiford


Agenda


70

Intuitive Approach

● Query by Example (QBE)
 Extract indexed feature from query image
 Compare with each indexed image
 Using selected dissimilarity function
 Linear search
● Compare to text search
 Inverted list
 Search time depends on terms

71

Indexing Visual
Information http://www.uni-klu.ac.at

● Visual information expressed by “vectors”
 Combined with a metric capturing the
semantics of similarity
 Inverted list does not work here
 An “index of vectors” is needed


Indexing Visual
Information http://www.uni-klu.ac.at

● Vectors describe “points in a space”
 Space is n-dimensional
 n might be rather big
● Metric describes distance between points
 E.g. L1 or L2 …
● Query is also a vector (point)
 Searching for points (vectors) near to query
● Idea for index:
 Index neighbourhood …


Spatial Indexes

Using equally sized rectangles (Optimal for L1 …)


Spatial Indexes

Using overlapping rectangles …


Spatial Indexes

● Common data structures
 R Tree
• R*, R+, ….
• Overlapping rectangles
• Search is a rectangle

 Quadtree (Octtree)
• Equally sized regions, subdivided
• 4 quadrants or 8 octants
• Search selects quadrants


Spatial Indexes:
Drawbacks http://www.uni-klu.ac.at

● Data structures must minimize
 false negatives (-> maximizes recall)
 false positives (-> search time)
● Descriptors, metrics & parameters need to
be selected at index time
 Searches combining multiple descriptors are a
complicated issue
● Work best for small n
 MDS has to be applied …


Multidimensional
Scaling (MDS) http://www.uni-klu.ac.at

● Reducing the dimensions of a feature space
 E.g. From 64 dimensions to 8
 Without loosing too much information about
neighbourhoods
● Interpolation: FastMap
 Linear in terms of objects
 Used e.g. in IBM QBIC
● Iterative: Force Directed Placement
 Iterative optimization of initial placement
 Cubic runtime


Metric Index

● Hierarchical clustering is applied for
indexing
 Representative image for cluster
 Search in m<n clusters instead of n images
● Problems
 The same as for clustering
• How to get a balanced tree?
• Do clusters represent dissimilarity?

81

Hashing

● Finding a hash function, which
 Can be applied easily to features
 Reflects dissimilarity
• Similar images have roughly the same hash
• Dissimilar images have “distant” hashes
● Example
 Locality Sensitive Hashing (LSH)
 Works in Euclidean spaces

82

Metric Spaces

src. G. Amato & P. Savino, „Approximate

● M = (D,d) Similarity Search in Metric Spaces Using Inverted Files “,
Infoscale 2008

 Data domain D
 Total (distance) function d: D D R (metric
function or metric)
● The metric space postulates:
x, y D , d ( x, y ) 0
 Non negativity
 Symmetry x, y D , d ( x, y ) d ( y , x )
 Identity x, y D , x y d ( x, y ) 0
 Triangle inequality x, y , z D , d ( x, z ) d ( x, y ) d ( y , z )


Similarity Search in
Metric Spaces http://www.uni-klu.ac.at

● Objects close to one another see the space in a
“similar” way
● Choose a set of reference objects RO
● Orderings of RO according to the distances from
two similar data objects are similar as well
 Represent every data object o as an ordering of RO
from o
 Measure similarity between two data objects by
measuring the similarity between the
corresponding orderings



O1 := <5, 3, 4, 1, 2>
O2 := <1, 5, 3, 5, 2>
O3 := …



● Spearman Footrule Distance

SFD(S1 , S2 ) S2 (ro) S1 (ro)
ro RO


Agenda


88

Blobworld

89

Blobworld

90

Blobworld

91

Blobworld

92

Informedia

● Database for search and browsing
 Carnegie Mellon University, H.D. Wactlar
● Content based search in TV and radio
news
● ~ 1500 h video and audio
● Transcription, indexing and segmentation
 Speech Recognition,
 Image Analysis,
 Natural Language Processing

93

Informedia

Search

Storyboard

Results
94

Informedia „El Nino“

95

Retrievr

● Flickr images indexed
 Based on some color feature
● Query by sketch interface
 Ajax based implementation

96

Photosynth

● SIFT to identify salient points
● Reconstruction of 3D model
● Selection through social annotation

97

References

[Eidenberger 2004] Eidenberger, H., Introduction: Visual
Information Retrieval, Habilitation, 2004
[Datta et al. 2008] Datta, R., Joshi, D., Li, J., and Wang, J. Z.
2008. Image retrieval: Ideas, influences, and trends of the
new age. ACM Comput. Surv. 40, 2, Article 5 (April 2008)

98

Acknowledgements

● Thanks to Oge Marques for kindly offering
his slides!

99

Thanks …

… for your attention!

mlux@itec.uni-klu.ac.at

(CC) by prakhar, flickr.com/photos/prakhar/827192423/

100

Visual Information Retrieval

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Visual Information Retrieval

Ähnlich wie Visual Information Retrieval (20)

Mehr von dermotte

Mehr von dermotte (10)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Visual Information Retrieval