《Scale and Object Aware Image Retargeting for Thumbnail》的PPT perillaroc@BIT

1. ICCV 2011
Scale and Object Aware Image
Retargeting for Thumbnail Browsing
Jin Sun, Haibin Ling
Temple University, Philadelphia, PA, 19122
By perillaroc

2. Introduction
• Image browsing tasks
• Tiny thumbnails: a fixed small size
• Scaling
– bring difficulties in searching and recognition
• Automatic image retargeting methods
– target size is comparable to that of the original image
• Several important issues for thumbnail
browsing
2012/6/19 2

3. Thumbnail Browsing
• Thumbnail scales
– Studies have shown that scales can have
significant effects on human visual perception
• Object completeness
– Low-level gradient-based information
– NOW! The object-level completeness
• Structure smoothness
– The contamination caused by pixel removal
methods, e.g. seam carving
2012/6/19 3

4. Scale and Object Aware Retargeting
• Thumbnail scales
– An image perceived by human vision system
– A new scale dependent saliency
• Object completeness
– integrating the objectness measurement recently
proposed by Alexe et al.
• Structure smoothness
– Use the thin-plate-spline(TPS) as warping function
– Use Cyclic seam Carving to guide the warping.
2012/6/19 4

5. Scale and Object Aware Retargeting
• A continuous retargeting algorithm but uses
discrete retargeting to guide the warping
transformation.
2012/6/19 5

6. Overview : Task Formulation
• Retargeting problem:
J  F ( I , S ( I ))
• Seam carving(SC)
 I (i, j ), if j  seam(i )
F 
sc
 I (i, j  1), if j  seam(i )
i, j
• First get retargeted images to a target size that is
comparable to the original image size.
• Then shrink the retargeted images to get the final
thumbnails.
2012/6/19 6

7. Overview : Framework Overview
• Scale and object aware image retargeting(SOAP): F so
• Warping function: thin-plate-spline (TPS)
• Landmark points : cyclic seam carving (CSC)
• Scale dependent saliency S scale ( I )
• Object awareness O( I )
• Scale and object aware saliency
S so ( I )  S scale ( I )  O( I )
• SOAP
J  F ( I , S ( I ))
so so
2012/6/19 7

8. Flow chart of the proposed method
2012/6/19 8

9. Scale-dependent Saliency
• Original image size
• Display size so s D
sd  , s p  d rp
• Projection size DPI D
• DPI(Dots Per Inch)
2012/6/19 9

10. Scale-dependent Saliency
• How well such a system preserves the image
information?
• We want to make the foreground object/
content/theme of the image as “clear” as
possible
• According to the study, not all patterns in an
image are recognizable by human
2012/6/19 10

11. Scale-dependent Saliency
• the perceived image, denoted as I p , can be derived as
 I (i, j ), if sd / D  [ ,  ]
I p ( I (i, j ))  
 I N (i, j ), otherwise
• I N (i, j ) is the mean value of N-connected neighbors of pixel I(i,j)
• N is determined by display device specifications.
• κ and ρ are the lower and upper bounds(in cycles per degree)
respectively of human visual acuity.
• κ and ρ define the limits at which the visual stimuli frequency
becomes too low or too high to be recognized by human.
• We use κ = 0.0175 and ρ = 0.83.
2012/6/19 11

12. Scale-dependent Saliency
• A pixel may become indistinguishable from its
neighbors to a human observer.
• An image patch that was salient in the original image
may not appear salient to a human observer.
• Scale dependent saliency
2012/6/19 12

13. Scale-dependent Saliency
• First, the original image is scaled in homogeneous
into the final thumbnail size, i.e. the display size,
60x60 pixels in our experiment.
• Then, the minimum recognizable pattern, denoted by

sd is determined by Eqn.4.
• Finally, the scale dependent saliency S scale ( I ) is defined
as
S scale ( I )  S ( I p )
where S(.) is the standard saliency (Itti 1998)
2012/6/19 13

14. Scale-dependent Saliency
• the minimum recognizable pattern sd 
• In our monitor : 1680X1050@65Hz , 120 DPI
• 
sd is in average 0.009 inches.
• The size is approximately the distance between two pixel lines
on the screen, i.e. N = 4 in Eqn.5.
• As a result, in the final thumbnail four adjacent neighbors of
one image pixel patch with value differences in color space
within certain threshold will be assigned their mean value,
which means those pixels are unable to be distinguished by
human.
• Differences : 50 in our experiment
2012/6/19 14

15. Scale-dependent Saliency
2012/6/19 15

16. Objectness Measure
• Preserve object completeness as much as possible
• Alexe et al. What is an object? (CVPR,2010).
– A novel objectness measure, which is trained to distinguish
object windows from background ones.
– For a rectangular window w = (i,j,w,h) with a top-left
corner at (i,j), width w and height h, its objectness is
defined as the probability p(w) that w contains an object.
2012/6/19 16

17. Objectness Measure
• First sample nw windows W  {wi }iw1 for an input image
• Then calculate the objectness map O as the average
objectness reponse at each pixel
1
O(i, j )  , j )w pobj (w)
 wW (i
Where   max O(i, j )
i, j
• nw = 10000 in our experiments
2012/6/19 17

18. Scale and Object Aware Saliency
S S
so scale
(i, j ) O(i, j )
2012/6/19 18

19. Cyclic Seam Carving
• In many cases a seam has no choice but to cross objects due
to the original definition of seam.
• Discontinuous seams
• Discontinuous seam-carving for video retargeting. CVPR, 2010
• Problem: object-structure damage, e.g. pixel shift problem
2012/6/19 19

20. Cyclic Seam Carving
• Cyclic Seams: warp the image
2012/6/19 20

21. Cyclic Seam Carving
• Energy function
Escale    Esc  (1   )  S so
where ρ is the weight and set to 0.3 .
• The improved energy is then combined with the CSC
algorithm to provide landmark point pairs needed for
estimating TPS warping.
2012/6/19 21

22. Cyclic Seam Carving
2012/6/19 22

23. Image Warping Function
• Many discrete retargeting methods generate
excellent results in general but they sometimes
create serious artifacts when the target has a size
much smaller than the input.
• We combine a continuous warping model with a
discrete retargeting guidance.
• Thin-plate-spline(TPS) as our warping function
2012/6/19 23

24. Thin-plate-spline(TPS)
• Landmark points P   pi  2
, i  1,2,
and , nl 
Q  qi  2 , i  1,2, , nl  , where p is mapped to q.
• the TPS transformation T is defined as the
transformation from P to Q that minimizes the
regularized bending energy E(f) defined as
E ( f )   qi  f (qi ) 
2
i
2 f 2 2 f 2 2 f 2
  ( 2 )  2( )  ( 2 ) dxdy
x xy y
• The TPS warping is defined as
T  arg min f E ( f )
2012/6/19 24

25. TPS Warping Function
• Landmark point pairs P and Q is derived from the CSC
retarget algorithm
• A two-way solution
• First, We sample randomly a landmark set b P (Fig.
6(a)) from original image and then trace their shifting
in the CSC process.
2012/6/19 25

26. TPS Warping Function
• Then, point set Q is re-sampled uniformly on the
target image, which is generated by CSC.
• Finally, a sample set P is generated by mapping Q to
the original image using warping estimated by Q and
P.
2012/6/19 26

27. Quantitative Experiments
• To find the thumbnail that matches the
description
• From 10 x 10 image thumbnails
• Different methods
– Scaling(SL)
– Seam carving (SC)
– Improved seam carving (ISC)
– The proposed SOAR algorithm (SOAR)
2012/6/19 27

28. Quantitative Experiments
2012/6/19 28

29. Quantitative Experiments
2012/6/19 29

30. Qualitative Experiments
• In general our method performs the best regarding
the (foreground) object size in the thumbnails.
2012/6/19 30

31. RetargetMe
2012/6/19 31

32. Limitations
• Saliency distribution is scattered
• Object is to big
2012/6/19 32

33. Conclusion
• A scale and object aware image retargeting
method for thumbnail browsing.
• Several new techniques
– Scale dependent saliency
– Objectness
– Cyclic seam carving
– A TPS-based continuous warping model
2012/6/19 33