K. Apostolidis, V. Mezaris, "A Fast Smart-Cropping Method and Dataset for Video Retargeting", Proc. 28th IEEE Int. Conf. on Image Processing (ICIP), Anchorage, Alaska, US, Sept. 2021. DOI:10.1109/ICIP42928.2021.9506390. Software and dataset available at https://github.com/bmezaris/RetargetVid In this paper a method that re-targets a video to a different aspect ratio using cropping is presented. We argue that cropping methods are more suitable for video aspect ratio transformation when the minimization of semantic distortions is a prerequisite. For our method, we utilize visual saliency to find the image regions of attention, and we employ a filtering-through clustering technique to select the main region of focus. We additionally introduce the first publicly available benchmark dataset for video cropping, annotated by 6 human subjects. Experimental evaluation on the introduced dataset shows the competitiveness of our method.

1. A fast smart-cropping method and dataset for
video retargeting
Konstantinos Apostolidis, Vasileios Mezaris
Information Technologies Institute (ITI), CERTH
Thessaloniki, Greece
This work was supported by the H2020 project ReTV (grant agreement No 780656)

2. Problem Statement
● Traditional TV and desktop computer monitors: landscape aspect ratios (16:9 or 4:3)
● Nowadays, mobile devices use different aspect ratios
● Video sharing platforms dictate the use of specific video aspect ratios
● Existing videos would have to be transformed to comply with their specifications

3. ● Straightforward approaches for transforming a video to a different aspect ratio:
○ static cropping of content
○ padding the frames with black borders
➢significant loss of visual content that might even be in the center of
attention
➢shrinks the original video by introducing large borders in the output video
Problem Statement
● The results of such simple approaches are often unsatisfactory
● Common video aspect ratio transformation methods of the literature:
○ Warping
○ Seam carving
➢Both introduce distortions and may alter the semantics of the video

4. Problem Statement
1. SmartVidCrop:
○ New video cropping method that minimizes the loss of semantically important
visual content
○ Selects the main focus out of the multiple possible salient regions of the video
○ Introduces a new filtering-through-clustering processing step
2. RetargetVid:
○ First publicly available benchmark dataset for video retargeting
○ Ground-truth video cropping results for 200 videos
○ Each video was annotated by 6 human annotators
○ Two target aspect ratios

5. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing

6. ow
oh
fh
fw
ow/oh = 16/9
fw/fh = 4/5
fh = oh
fw < ow
fh
ow
oh
fw
Original frame
Final frame
Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
ow/oh = 4/5
fw/fh = 16/9
fh < oh
fw = ow

7. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing

8. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing

9. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing

10. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing

11. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
yi
xi
t
Time-series of center of mass
displacement

12. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
t
Shot transitions Video
frames
Shot #1
Shot #2
Shot #3

13. Proposed Method
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Smoothed time-series
Inferred time-series
t
Smoothed time-series of center of mass
displacement

14. ● Read the first out of every five videos frames
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Proposed Method

15. RetargetVid dataset
● Motivation: Objective comparisons of video retargeting methods are difficult and
scarce in the literature
● Construction of a dataset for the task of video aspect ratio transformation
● Publicly available annotations
● 200 videos from the publicly available videos of the DHF1k dataset
● Annotations by 6 human subjects
● 1:3 and 3:1 target aspect ratios
● Dataset presents a good balance between inter-annotator agreement and diversity

16. Results
● Google AutoFlip: a cropping method for video retargeting with available source code
● We utilize the constructed RetargetVid dataset to compare our method to AutoFlip

17. Method Worst Best Mean t
Results for 1:3 target aspect ratio
AutoFlip 50.0 52.1 50.8 40
Ours (w/o clustering step) 45.3 47.4 46.1 17
SmartCrop 48.6 50.9 49.9 19
Results for 3:1 target aspect ratio
AutoFlip 70.9 74.7 72.2 41
Ours (w/o clustering step) 66.3 72.9 68.1 17
SmartCrop 70.1 73.6 71.4 20
Results

18. RetargetVid dataset and SmartVidCrop
● Source code of SmartVidCrop
● Ground-truth annotations for the RetargetVid dataset
● Updated results of our SmartVidCrop method, achieved by further optimizing it
are publicly available at:
https://github.com/bmezaris/RetargetVid

19. Try it yourself at:
Contacts:
Vasileios Mezaris, bmezaris@iti.gr
Konstantinos Apostolidis, kapost@iti.gr
This work was supported by the H2020 project ReTV (grant agreement No 780656)
http://multimedia2.iti.gr/videosmartcropping/service/start.html
Or ask us about the underlying algorithms
and how these can be integrated in your system