1. Outline
Introduction
Salient Point Detection
Challenges
Results
Salient Point Detection
Tyler Karrels
Department of Electrical and Computer Engineering
University of Wisconsin - Madison
April 22, 2009
Tyler Karrels Salient Point Detection

2. Outline
Introduction
Salient Point Detection
Challenges
Results
1 Introduction
Defining Saliency
The Saliency Experience
Human Visual System (HVS)
Psychology of Perception
Previous Work
2 Salient Point Detection
Mathematical Framework
Features
Clustering
Saliency
3 Challenges
4 Results
Tyler Karrels Salient Point Detection

3. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
What is saliency?
Definitions
1 SALIENT: “strikingly conspicuous; prominent; noticeable”
American Heritage Dictionary
Tyler Karrels Salient Point Detection

4. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
What is saliency?
Definitions
1 SALIENT: “strikingly conspicuous; prominent; noticeable”
American Heritage Dictionary
2 VISUAL SALIENCY: “. . . the distinct subjective perceptual
quality which makes some items in the world stand out from
their neighbors and immediately grab our attention.”
Laurent Itti, Scholarpedia
Tyler Karrels Salient Point Detection

5. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Tyler Karrels Salient Point Detection

6. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Popout Effect 1
Tyler Karrels Salient Point Detection

7. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Popout Effect 2
Tyler Karrels Salient Point Detection

8. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
What is salient?
Tyler Karrels Salient Point Detection

9. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Conjunction Test 1
Tyler Karrels Salient Point Detection

10. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Conjunction Test 2
Tyler Karrels Salient Point Detection

11. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
FORGET
(o.0)
Tyler Karrels Salient Point Detection

12. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Conjunction Test 3
Tyler Karrels Salient Point Detection

13. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
What is salient?
Tyler Karrels Salient Point Detection

14. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Phase Transition
Tyler Karrels Salient Point Detection

15. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
What is salient?
Tyler Karrels Salient Point Detection

16. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
The Eye: Physiology
Peripheral Vision, Wikipedia
Foveal Vision: attended location; line of sight
Peripheral Vision: surrounding locations
1-1 photoreceptor to ganglion in Foveal Vision
many-1 for Peripheral Vision (low res. compression)
50% Fovea + 50% Peripheral = 100% Data Sent to Brain!
Tyler Karrels Salient Point Detection

17. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
The Eye: Feature Detector
Bottom-Up Processing
Detects low-level features in parallel, e.g. color, orientation,
contrast, . . .
Occurs before brain perceives data
Feature detectors compete to direct attention to salient
locations
How do they compete, communicate, and cooperate?
Top-Down Processing
The brain’s expectations guide attention
Tyler Karrels Salient Point Detection

18. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Helmholtz Principle
“. . . whenever some large deviation from randomness occurs, a
structure is perceived.”
Desolneux, From Gestalt Theory to Image Analysis: A Probabilistic Approach
Tyler Karrels Salient Point Detection

19. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Gestalt Laws
Perceptual Grouping Principles
Closure
Similarity
Proximity
Symmetry
Continuity
Common Fate
Tyler Karrels Salient Point Detection

20. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Sha’asua [4]
Continuity & Closure
Detect edges
Form contours: Connect
edges
Maximize length
Minimize total curvature
Longer contours, more
salient
Disregards other gestalt laws
& image features
Tyler Karrels Salient Point Detection

21. Outline Defining Saliency
Introduction The Saliency Experience
Salient Point Detection Human Visual System (HVS)
Challenges Psychology of Perception
Results Previous Work
Itti [2]
Proximity & Similarity
Multiple scales encode
proximity
Center-surround
Measures local contrast
Fine scale ‘center’ minus
course scale ‘surround’
Normalization encodes
similarity
Feature map combination
Itti’s Biological determines success
Saliency Map Model
Tyler Karrels Salient Point Detection

22. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Salient Point Detection
Not constrained to be biologically plausible
Not image-processing; clustering/outlier detection in Rd
Pixels are salient, not objects or regions
Challenges
When is something salient? When not?
Can we quantify saliency?
Can we relate computer/human performance?
Can we improve on previous methods? Will we?
Tyler Karrels Salient Point Detection

23. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Framework
Data
Pixels {Xi }n
i=1
Xi = (yi , xi , ri , gi , bi )
Video? Include time coordinate: X = (y , x, r , g , b, t)
Feature Space
Feature Maps {Fj }m
j=1
Vi = (yi , xi , ri , gi , bi , Fi1 , . . . , Fim )
Feature Vectors {Vi }n
i=1
Feature Space Vi ∈ [0, 1]d
Tyler Karrels Salient Point Detection

24. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
The Process
Easy As 1,2,3?
1 Create feature maps
2 Cluster points in Rd
3 Choose the salient cluster
Tyler Karrels Salient Point Detection

25. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Proposed Features
Feature Maps {Fj }m
j=1
Salient Scenarios
1 Intensity
1 Intensity
2 Color
2 Colors [Red, Green, Blue]
3 Orientation
3 Edge orientations
[0 ◦ , 45 ◦ , 90 ◦ , 135 ◦ ]
4 Size
4 Scale Description
5 Location
5 Pixel Location
How does our data representation affect performance?
Tyler Karrels Salient Point Detection

26. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
2-D Example
Do we really need 2 dimensions? Is 1 sufficient?
Tyler Karrels Salient Point Detection

27. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
1-D Example
Background pixels: no
orientation?
Horizontal pixels: 0 ◦ or
180 ◦ ?
Tyler Karrels Salient Point Detection

28. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Feature Subset Selection
Choosing Salient Dimensions
Interpret variance
Projections onto feature subspaces
Projections
Pr[Vi = (v1 , . . . , vd )] empirical distribution
I = {i1 , . . . , il } index set
Project onto subset I , induce Pr[Vi = (vi1 , . . . , vil )]
Minimize the KL Divergence between Empirical and Subset
distributions
Tyler Karrels Salient Point Detection

29. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Vertical, Horizontal, Intensity, Red Example
Notice Pr[Red, VerticalBar ] ≈ Pr[RedBar ]
Tyler Karrels Salient Point Detection

30. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Clustering in Feature Space
Gaussian Mixture Clustering
Figueiredo’s algorithm determines best number of clusters [1]
Fits distribution in feature space to a mixture of Gaussians
Uses EM algorithm, results vary depending on initialization
Subspace Clustering
Ma’s algorithm provides distortion parameter [3]
Based on rate distortion theory
Deterministic, same results every time
Tight cluster requires low rate
Additional clusters increase rate
Tyler Karrels Salient Point Detection

31. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Subspace Clustering
Large → few clusters, Small → many clusters
As varies large → small, salient clusters emerge
Tyler Karrels Salient Point Detection

32. Outline
Mathematical Framework
Introduction
Features
Salient Point Detection
Clustering
Challenges
Saliency
Results
Salient Clusters
How to determine a cluster’s saliency?
Compare clusters for relative notion of saliency
Relative cluster size and variance
Small relative size indicates uniqueness
Small relative variance indicates similarity
‘Distant’ clusters have less in common.
Centroid Distance
Mahanalobis Distance
Outlier detection methods
Tyler Karrels Salient Point Detection

33. Outline
Introduction
Salient Point Detection
Challenges
Results
Existence of Salient Points
Tyler Karrels Salient Point Detection

34. Outline
Introduction
Salient Point Detection
Challenges
Results
Quantification of Salient Points
Tyler Karrels Salient Point Detection

35. Outline
Introduction
Salient Point Detection
Challenges
Results
Performance
Tyler Karrels Salient Point Detection

36. Outline
Introduction
Salient Point Detection
Challenges
Results
Orientation Test Results
Tyler Karrels Salient Point Detection

37. Outline
Introduction
Salient Point Detection
Challenges
Results
Google Tyler Karrels - Click Homepage - Click Papers
Tyler Karrels Salient Point Detection

38. Outline
Introduction
Salient Point Detection
Challenges
Results
References
M. A. T. Figueiredo and A. K. Jain.
Unsupervised learning of finite mixture models.
IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 381–396, 2002.
L. Itti and C. Koch.
Feature combination strategies for saliency-based visual attention systems.
Journal of Electronic Imaging, 10:161, 2001.
Y. Ma, H. Derksen, W. Hong, and J. Wright.
Segmentation of multivariate mixed data via lossy data coding and compression.
IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1546–1562, 2007.
A. Sha’asua.
Structural saliency: The detection of globally salient structures using a locally connected network, 1988.
ID: 1.
Tyler Karrels Salient Point Detection