M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Laboratoire Hubert Curien, Saint-Etienne, FR Conference "Advanced Concepts for Intelligent Vision Systems " 2016

1. Introduction
Related Work
Methodology
Results and Discussion
Global Bilateral Symmetry Detection Using
Multiscale Mirror Histograms
M. ELAWADY1
, C. BARAT1
, C. DUCOTTET1
and P. COLANTONI2
1
Universit´e de Lyon, CNRS, UMR 5516, Laboratoire Hubert Curien,
Universit´e de Saint-´Etienne, Jean-Monnet, F-42000 Saint-´Etienne, France
2
Universit´e Jean Monnet, CIEREC EA n0
3068, Saint-´Etienne, France
ACIVS Conference, October 2016
UMR • CNRS • 5516 • SAINT-ETIENNE
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 1 / 31

2. Introduction
Related Work
Methodology
Results and Discussion
Table of Contents
1 Introduction
Background
Applications
Problem Definition
2 Related Work
Intensity-based Methods
Edge-based Methods
3 Methodology
Motivation
Algorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 2 / 31

3. Introduction
Related Work
Methodology
Results and Discussion
Background
Applications
Problem Definition
Table of Contents
1 Introduction
Background
Applications
Problem Definition
2 Related Work
Intensity-based Methods
Edge-based Methods
3 Methodology
Motivation
Algorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 3 / 31

4. Introduction
Related Work
Methodology
Results and Discussion
Background
Applications
Problem Definition
Bilateral Symmetry
1Image from book: The Photographer’s Eye by Michael Freeman
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 4 / 31

5. Introduction
Related Work
Methodology
Results and Discussion
Background
Applications
Problem Definition
Bilateral Symmetry in Computer Vision I
Medial Image Compression [1]
Depth Estimation [2]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 5 / 31

6. Introduction
Related Work
Methodology
Results and Discussion
Background
Applications
Problem Definition
Bilateral Symmetry in Computer Vision II
Object Segmentation [3]
Aesthetic Analysis [4]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 6 / 31

7. Introduction
Related Work
Methodology
Results and Discussion
Background
Applications
Problem Definition
Detection of Main Symmetry Axis
Axis Legend: Strong, Weak
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 7 / 31

8. Introduction
Related Work
Methodology
Results and Discussion
Intensity-based Methods
Edge-based Methods
Table of Contents
1 Introduction
Background
Applications
Problem Definition
2 Related Work
Intensity-based Methods
Edge-based Methods
3 Methodology
Motivation
Algorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 8 / 31

9. Introduction
Related Work
Methodology
Results and Discussion
Intensity-based Methods
Edge-based Methods
Baseline and its Successors I
The general scheme (Loy and Eklundh 2006 [5]) consists of:
Example:
1Second figure from [5]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 9 / 31

10. Introduction
Related Work
Methodology
Results and Discussion
Intensity-based Methods
Edge-based Methods
Baseline and its Successors II
Disadvantages:
Depending mainly on the properties of hand-crafted features (i.e. SIFT).
For example: (smooth objects with noisy background)
little feature points =⇒ lost symmetry.
(Mo and Draper 2011 [6]) proposed refinements in the general scheme in:
1 Selecting all symmetry candidate pairs instead of finding closest matches
for each point.
2 Using less complex hough voting scheme.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 10 / 31

11. Introduction
Related Work
Methodology
Results and Discussion
Intensity-based Methods
Edge-based Methods
State of Art
Instead of SIFT, the general idea (Cicconet et al. 2014 [7]) is extracting a
regular set of wavelet segments with local edge amplitude and orientation.
Disadvantages:
Lacking neighborhood’s information inside the feature representation.
Depending on the scale parameter of the edge detector.
For example: (high texture objects with noisy background)
inferior symmetrical info =⇒ incorrect symmetry.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 11 / 31

12. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Table of Contents
1 Introduction
Background
Applications
Problem Definition
2 Related Work
Intensity-based Methods
Edge-based Methods
3 Methodology
Motivation
Algorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 12 / 31

13. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Proposed Idea
Investigating Cicconet’s edge features [7] within Loy’s scheme [5] by
adding neighboring-pixel information.
Contributions:
1 Introducing a new local edge descriptor.
2 Using multiscale edge extraction exploiting the full resolution image.
3 Solving the orientation discontinuity problem in the voting space.
4 Introducing a symmetry dataset based on aesthetic analysis.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 13 / 31

14. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Symmetry Detection Algorithm
Main Steps:
(1) Mul�scale Edge
Segment Extrac�on
(2) Triangula�on based on
Local Symmetry Weights:
• Geometry Edge Orienta�ons (Cic)
• Local Texture Histogram (Loy)
(3) Vo�ng Space for Peak Detec�on with Handling
Orienta�on Discon�nuity.
θ
ρ
0
π
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 14 / 31

15. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Multiscale Edge Segment Extraction I
A feature point p and its local edge characteristics (Jp, τp) are extracted
within each cell using a Morlet wavelet ψk,σ of constant scale σ and
varying orientation {τk , k = 1 . . . n}.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 15 / 31

16. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Multiscale Edge Segment Extraction II
Jk (p) denote the modulus of wavelet coefficients at point p, in which
local edge characteristics Jp and τp are obtained by seeking the maximum
wavelet response and orientation over all orientations.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 16 / 31

17. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Multiscale Edge Segment Extraction III
Histogram count at a given orientation τk is:
hp(k) =
r∈N(p)
Jr δφk −φr (1)
where φk and φr are angles associated with τk and τr , and δx is the
Kronecker delta. hp is subsequently 1 normalized and circular shifted so
as the first bin corresponds to τp.
0 36 72 108 144
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
#106
Magnitude Histogram
108 144 0 36 72
0
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hp)
0 36 72 108 144
0
500
1000
1500
2000
2500
3000
Frequency Histogram
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 17 / 31

18. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Multiscale Edge Segment Extraction IV
In most images, relevant information about the visual content may
appear at different scales. Feature points are computed with respect to a
set of regular grids at different scales and a corresponding set of wavelet
scales {σl , l = 1..m}.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 18 / 31

19. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Triangulation: Local Texture Histogram
(Textural Information) Symmetry degree of the two regions around p and
q can be measured by comparing their corresponding local orientation
histogram hp and hq. Texture-based symmetry measure is given by:
dI (hp, h∗
q) =
n
k=1
min(hp(k), h∗
q(k)) (2)
108 144 0 36 72
0
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hp)
72 36 0 144 108
0
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hq*)
1 2 3 4 5
0
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Intersection
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 19 / 31

20. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Triangulation: Geometry Edge Orientation
(Edge Information) Pairwise symmetry coefficient f (p, q) is defined as [7]:
f (p, q) = |τqS(T⊥
pq)τp| (3)
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 20 / 31

21. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Triangulation: Symmetry Weights
Given a pair of feature points (p, q), the candidate axis T⊥
pq perpendicular
to (pq) is parametrized by the orientation of its normal θpq and its
distance to the origin ρpq.
Mirror symmetry histogram HS (ρ, θ) is defined as the sum of the
contribution of all pairs of feature points such as:
H(cx , cy , θ) =
p,q
p=q
JpJqf (p, q)dI (hp, hq)δ(cx ,cy )− p+q
2
δθ−θpq (4)
HS (ρ, θ) =
cx ,cy
H(cx , cy , θ)δρ−ρpq (5)
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 21 / 31

22. Introduction
Related Work
Methodology
Results and Discussion
Motivation
Algorithm Details
Voting Space and Peak Detection
A1 A2 A3 A4 A5
B
A1 A2 A3 A4 A5
B
B
A1 A2 A3 A4 A5
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 22 / 31

23. Introduction
Related Work
Methodology
Results and Discussion
Table of Contents
1 Introduction
Background
Applications
Problem Definition
2 Related Work
Intensity-based Methods
Edge-based Methods
3 Methodology
Motivation
Algorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 23 / 31

24. Introduction
Related Work
Methodology
Results and Discussion
Algorithm Evaluation
From Real-World Images Competition CVPR 2013 [10], a symmetry
detection is correct if: (1) θ < 15◦
and (2) d < 0.2 ∗ min(lenGT , lenR ).
R
GT
d
θ
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 24 / 31

25. Introduction
Related Work
Methodology
Results and Discussion
Quantitative Results$FFXUDFuD(t
M
CM
iM
cM
OM
uM
rM
M
M
M
CMM
'DWDVHWV
368iMCM 368iMCC 368iMCc $9$iMCr
/RiMMr
0RiMCC
LFiMCO
2XUiMCr
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 25 / 31

26. Introduction
Related Work
Methodology
Results and Discussion
Qualitative Results on PSU Datasets
(http://www.flickr.com/), around 200 images from PSU symmetry
detection challenges [9, 10] in ECCV2010, CVPR2011 and CVPR2013.
Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 26 / 31

27. Introduction
Related Work
Methodology
Results and Discussion
Qualitative Results on AVA Dataset
(http://www.dpchallenge.com/), around 250 images from Aesthetic
Visual Analysis “AVA” [8] with our global-axis symmetry groundtruth.
Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 27 / 31

28. Introduction
Related Work
Methodology
Results and Discussion
Conclusion
Summary:
1 A reliable global symmetry detection is developed among variants of visual cues.
2 A groundtruth of global symmetry axis is introduced and extracted from large
scale Aesthetic Visual Analysis (AVA) dataset.
Future work:
1 Real-world images is required to handle with large degrees of perspective view.
2 The proposed detection can be improved to avoid over-extended axes.
3 A stable balance measure can be introduced to describe the existence and degree
of global axes inside an image.
4 Possibility of integration within retrieval systems of visual arts.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 28 / 31

29. Introduction
Related Work
Methodology
Results and Discussion
References I
[1] V. Bairagi, “Symmetry-based biomedical image compression,” Journal of digital
imaging, pp. 1–9, 2015.
[2] L. Yang, J. Liu, and X. Tang, “Depth from water reflection,” Image Processing,
IEEE Transactions on, vol. 24, no. 4, pp. 1235–1243, 2015.
[3] C. L. Teo, C. Fermuller, and Y. Aloimonos, “Detection and segmentation of 2d
curved reflection symmetric structures,” in Proceedings of the IEEE International
Conference on Computer Vision, pp. 1644–1652, 2015.
[4] S. Zhao, Y. Gao, X. Jiang, H. Yao, T.-S. Chua, and X. Sun, “Exploring
principles-of-art features for image emotion recognition,” in Proceedings of the
ACM International Conference on Multimedia, pp. 47–56, ACM, 2014.
[5] G. Loy and J.-O. Eklundh, “Detecting symmetry and symmetric constellations of
features,” in Computer Vision–ECCV 2006, pp. 508–521, Springer, 2006.
[6] Q. Mo and B. Draper, “Detecting bilateral symmetry with feature mirroring,” in
CVPR 2011 Workshop on Symmetry Detection from Real World Images, 2011.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 29 / 31

30. Introduction
Related Work
Methodology
Results and Discussion
References II
[7] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetry
histograms for capturing geometric properties in images,” in Computer Vision
and Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 2981–2986,
IEEE, 2014.
[8] N. Murray, L. Marchesotti, and F. Perronnin, “Ava: A large-scale database for
aesthetic visual analysis,” in Computer Vision and Pattern Recognition (CVPR),
2012 IEEE Conference on, pp. 2408–2415, IEEE, 2012.
[9] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First symmetry
detection competition: Summary and results,” tech. rep., Technical Report
CSE11-012, Department of Computer Science and Engineering, The
Pennsylvania State University, 2011.
[10] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu,
“Symmetry detection from realworld images competition 2013: Summary and
results,” in Computer Vision and Pattern Recognition Workshops (CVPRW),
2013 IEEE Conference on, pp. 200–205, IEEE, 2013.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 30 / 31

31. Introduction
Related Work
Methodology
Results and Discussion
Questions?
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 31 / 31