https://imatge.upc.edu/web/publications/efficient-exploration-region-hierarchies-semantic-segmentation
The motivation of this work is the efficient exploration of hierarchical partitions for semantic segmentation as a method for locating objects in images. While many efforts have been focused on efficient image search in large-scale databases, few works have addressed the problem of locating and recognizing objects efficiently within a given image. My work considers as an input a hierarchical partition of an image that defines a set of regions as candidate locations to contain an object. This approach will be compared to other state of the art algorithms that extract object candidates for an image. The final goal of this work is to semantically segment images efficiently by exploiting the multiscale information provided by a hierarchical partition, maximizing the accuracy of the segmentation when only a very few regions of the partition are analysed.
12. Related Work
12
Object Detection
and Recognition
Sliding Windows
Partition
e.g. Viola Jones
Hierarchical
Flat
e.g UCM
e.g CPMC
e.g Watershed Partition
Object Proposals
Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
13. Ultrametric Contour Map (UCM)
13
Original Image Ultrametric Contour Map Dendrogram
Arbelaez, P. (2006, June). Boundary extraction in natural images using ultrametric contour maps
root node
leavescosts
38. Conclusions
2. The class-dependent exploration of UCM regions is the best
configuration for a budget of a few regions.
38
Class-dependent tree
exploration
...
39. Conclusions
3. SDS descriptors extracted from a CNN obtain better results than O2P.
39
Deep learning features [SDS]hand-crafted features [O2P]
40. Future Work
● Class-dependent tree exploration using two classifiers
● Compare performance using different object candidates, such as MCG.
40
Is there a face
on this node?
Is this a face?
1
2
X. Giró, 2012, Part-based object retrieval with binary partition trees.
Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping
43. Related Work
43
Object Detection
and Recognition
Sliding Windows
Segmentation
e.g. Viola Jones
Hierarchical Segmentation
Flat segmentation
e.g UCM
Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
e.g CPMC
44. Related Work: Tree exploration
Regions generated from a hierarchical partition taking advantage of its multi-
scale information in order to guide an efficient exploration throughout the
tree.
X. Giró, 2012, Part-
based object
retrieval with binary
partition trees.
X. Giró, 2012, Part-based object retrieval with binary partition trees. 44
45. Constrained Parametric Min-Cuts (CPMC)
Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts. 45
46. Motivation
Local Feature Descriptors: SIFT HOG
Learned Descriptors
From hand-crafted to learned features
~1995 to ~2005 ~2005 to ~2010 ~2010 to ~2015
Feature visualization of convolutional net trained
on ImageNet from [Zeiler & Fergus 2013]
hand-crafted descriptors
46
47. Features: SDS features
[SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 47
48. Features: Second Order Pooling (O2P)
Average Pooling
Max Pooling
2nd
order
SIFT
O2PSIFT Second Order SIFT Pooling
Carreira, J., Caseiro, R., Batista, J., & Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. 48
51. Class-agnostic tree exploration: Costs
Big difference of
cost between node
and its father
Small difference of
cost between node
and one of its
children
51
SECOND
DERIVATIVE
53. Class-agnostic tree exploration: Costs
Big difference of
cost between node
and its father
Small difference of
cost between node
and one of its
children
53
55. Class-agnostic tree exploration: Indexes
Objects can be found in regions associated to indexes
that differ from the indexes of their adjacent regions
150
149
147
cost
easier to generate
than its sibling…
more
homogeneous
indexes
55
521
57. Class-agnostic tree exploration
Contours of UCM
Merging Sequence
INDEXES of
the merging
sequence
COSTS values
of the contours
Input image
Based on the structure of the UCM partition, defined by these two files:
1
2
57
59. Class-dependent tree exploration
Guide a top-down efficient exploration throughout the tree based on the
classifier’s decision.
X. Giró, 2012, Part-based object retrieval with binary partition trees.
Motivation
59
63. Related Work
63
Object Detection
and Recognition
Sliding Windows
Segmentation
e.g. Viola Jones
Hierarchical Segmentation
Flat segmentation
e.g UCM
Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
e.g CPMC
64. Related Work
64
Object Detection
and Recognition
Sliding Windows
Segmentation
e.g. Viola Jones
Hierarchical Segmentation
Flat segmentation
e.g UCM
Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.
e.g CPMC
70. Motivation
Goal: Guide a top-down exploration of a hierarchical
partition by answering the following question:
● Does this region contain the object
we are seeking?
● If so, does this region represent
the object we are seeking?
70