Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Â
Sketching Out Your Search Intent
1. Sketching out Your Search Intent
- Towards bridging intent and semantic gaps in
web-scale multimedia search
Xian-Sheng Hua
Lead Researcher, Microsoft Research Asia
June 24 2010 â ACM Multimedia 2010 TPC Meeting â University of Amsterdam
2. Xian-Sheng Hua, MSR Asia
Evolution of CBIR/CBVR
Annotation Based
Commercial Search ~2006 ï
Engines
~2003 (1) Tagging Based
(2) Large-Scale CBIR/CBVR
Conventional CBIR ~2001
Direct-Text Based
1990â Academia
Prototypes
1970~
1980â
Manual Labeling Based
2
3. Xian-Sheng Hua, MSR Asia
Three Schemes
Example-Based
Annotation-Based
Text-Based
4. Xian-Sheng Hua, MSR Asia
Limitation of Text-Based Search
âą High noises
Ì” Surrounding text: frequently not related to the visual content
Ì” Social tags: also noisy, and a large portion donât have tags
âą Mostly only covers simple semantics
Ì” Surrounding text: limited
Ì” Social tags: People tend to only input simple and personal tags
Ì” Query association: powerful but coverage is still limited (non-clicked
images will never be well indexed)
A complex example:
Finding Lady Gaga walking on red carpet with black dress
5.
6. Xian-Sheng Hua, MSR Asia
Limitation of Annotation-Based Search
âą Low accuracy
Ì” The accuracy of automatic annotation is still far from satisfactory
Ì” Difficult to solve due
âą Limited coverage
Ì” The coverage of the semantic concepts is still far from the rich content
that is contain in an image
Ì” Difficult to extend
âą High computation
Ì” Learning costs high computation
Ì” Recognition costs high if the number of concepts is large
Still has a long way to go
7. Xian-Sheng Hua, MSR Asia
Limitation of Large-Scale Example-Based Search
âą You need an example first
Ì” How to do it if I donât have an initial example?
âą Large-scale (semantic) similarity search is still difficult
Ì” Though large-scale (near) duplicate detection is good
An example: TinEye
8. Xian-Sheng Hua, MSR Asia
Possible Way-Outs?
âą Large-scale high-performance annotation?
Ì” Model-based? Data-driven? Hybrid? âŠ
âą Large-scale manual labeling?
Ì” ESP game? Label Me? Machinery Turk? âŠ
âą New query interface?
Ì” Interactive search?
Or ⊠To combine text, content and interface?
9. Xian-Sheng Hua, MSR Asia
Simple Attempts
âą Exemplary attempts based on this idea
Ì” Search by dominant color (Google/Bing)
Ì” Show similar images (Bing)/Find similar images (Google)
Ì” Show more sizes (Bing)
10.
11.
12.
13.
14.
15.
16. Xian-Sheng Hua, MSR Asia
Two Recent Projects
âą Sketching out your search intent
Ì” Image Search by Color Sketch
Ì” Image Search by Concept Sketch
18. Xian-Sheng Hua, MSR Asia
Seems It Is Not New?
âą âQuery by sketchâ has been proposed long time ago
Ì” But on small datasets and difficult to scale up
Ì” And seldom work well actually
Ì” Not use to use â âobject sketchâ
19.
20. It is based on a SIGGRAPH 95âs paper:
C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995
on small scale image set and difficult to scale-up (0.44 second for 10,000 images)
23. Xian-Sheng Hua, MSR Asia
Our Approach
âą Color sketch âŠ
Ì” Is not âobject sketchâ
Ì” But only rough color distribution
Ì” Supports large-scale data
Ì” And uses text at the same time
24.
25.
26. Xian-Sheng Hua, MSR Asia
Our Approach
âą Color sketch âŠ
Ì” Is not âobject sketchâ
Ì” But only rough color distribution
Ì” Supports large-scale data
Ì” And uses text at the same time
âą Advantages of âColor Sketchâ
Ì” More presentative than dominant color (and text only)
Ì” More robust than âobject sketchâ
Ì” Easy to use compared with âobject sketchâ
Ì” Easy to scale up
66. Xian-Sheng Hua, MSR Asia
Summary â Why it Works
âą What are difficult
Ì” Semantics is difficult
Ì” Intent prediction is difficult
âą What are easier
Ì” Use color to describe imagesâ content
Ì” Use color to describe usersâ intent
content color sketch intent
Color sketch is an intermediate representation to connect imagesâ content and usersâ intent.
67. Xian-Sheng Hua, MSR Asia
Limitation of Color Sketch
âą Only works well for those semantics that can
be described by color distribution
âą Only works well when people is able to transfer
their desire into color distribution
A more complex example:
A flying butterfly on the top-left of a flower.
69. Xian-Sheng Hua, MSR Asia
Search By Concept Sketch
âą A system for the image search intentions concerning:
Ì” The presence of the semantic concepts (semantic constraints)
Ì” the spatial layout of the concepts (spatial constraints)
butterfly
flower
A concept map query Image search results of our system
70. Xian-Sheng Hua, MSR Asia
Framework
A concept map query A candidate image
1
A: butterfly Create Candidate
B: flower Image Set
butterfly
A: X=0.5, Y=0.5
B: X=0.8, flower
Y=0.8
2 3
Instant Concept Concept Distribution
Modeling Estimation
Concept Models Estimated distributions
butterfly butterfly
4 flower
Intent flower
Interpretation
Desired distributions
5
Spatial distribution
butterfly comparison
flower
Relevance score
71. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with âsnoopy appear at rightâ
snoopy
Text-based image search
âshow similar imageâ
snoopy
Image search by concept map
72. Xian-Sheng Hua, MSR Asia
Search Results
Task: searching for the images with âsnoopy appear at topâ / âsnoopy appear at leftâ
73. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with âa jeep shown above a piece of grassâ
jeep grass
74. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with âa car parked in front of a houseâ
house car
75. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with âa keyboard and a mouse being side by sideâ
keyboard mouse
76. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with âsky/Colosseum/grass appear from top to bottomâ
Colosseum grass
77. Xian-Sheng Hua, MSR Asia
Advanced Function: Influence Scope
Indication
âą Allow users to explicitly specify the occupied
region of a concept
house
Default
house
lawn lawn
house
lawn
Search for the long narrow lawn at the bottom of the image
78. Xian-Sheng Hua, MSR Asia
Advanced Function:
Influence Scope Adjustment
Searching for small windmill
Searching for large windmill
79. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
sky
house
lawn
Clear Search
Advanced Function
Visual Assistor
Previous Next
Allow users to indicate or narrow down
what a desired concept looks like (visual constraints)
80. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
Searching for
front-view jeeps
jeep
Searching for
side-view jeeps
Visual instances
81. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
Searching for
butterfly with
yellow flower
butterfly
flower
Searching for
butterfly with
red flower
Visual instances
82. Xian-Sheng Hua, MSR Asia
Conclusions
âą To bridge the gap between usersâ intent and
visual semantics of images by
Ì” Sketching out your search intent, and
Ì” Combining text and visual content
âą Two exemplary approaches introduced
Ì” Search by color sketch
Ì” Search by concept sketch