Presentation of the paper titled "A Web Service for Video Smart-Cropping", by K. Apostolidis, V. Mezaris, delivered at the IEEE Int. Symposium on Multimedia (ISM), Dec. 2021. The corresponding software and dataset are available at https://github.com/bmezaris/RetargetVid.

1. A Web Service for Video Smart-Cropping
Konstantinos Apostolidis, Vasileios Mezaris
CERTH-ITI
Thessaloniki, Greece
This work was supported by the H2020 project ReTV (grant agreement No 780656)
1
23rd IEEE International Symposium
on Multimedia (ISM 2021)

2. Problem Statement
● Traditional TV and desktop computer monitors: landscape aspect ratios (16:9 or 4:3)
● Nowadays, mobile devices use different aspect ratios
● Video sharing platforms dictate the use of specific video aspect ratios
● Existing videos would have to be transformed to comply with their specifications
2

3. ● Straightforward approaches for transforming a video to a different aspect ratio:
○ Static cropping of content
○ Padding the frames with black borders
Problem Statement
● The results of such simple approaches are often unsatisfactory
● Common video aspect ratio transformation methods of the literature:
○ Warping
○ Seam carving
➢Both introduce distortions and may alter the semantics of the video
3
➢Significant loss of visual content that might even be in the center of
attention
➢Shrinks the original video by introducing large borders in the output video

4. Problem Statement
● Several research works and some commercial software for video retargeting
available
● No easy and free video retargeting tools!
● Motivated by this, we built a freely accessible Web application for video retargeting
that consists of:
○ A REST service hosting the developed technologies for video cropping
○ An interactive user interface
● We used a modified version of the method in [1]
[1] Apostolidis, Konstantinos, and Vasileios Mezaris. "A Fast Smart-Cropping Method and
Dataset for Video Retargeting." In 2021 IEEE International Conference on Image
Processing (ICIP), pp. 2618-2622. IEEE, 2021. 4

5. Let’s quickly remind how the method of [1] works!
5

6. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
6

7. ow/oh = 16/9
fw/fh = 4/5
fh = oh
fw < ow
Original frame
Final frame
Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
7
fh
ow
oh
fw

8. ow
oh
fh
fw
ow/oh = 4/5
fw/fh = 16/9
fh < oh
fw = ow
Original frame
Final frame
Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
8

9. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
9

10. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
10

11. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
11

12. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
12

13. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
yi
xi
13

14. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing t
Time-series of center of mass
displacement

15. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
t
Shot transitions Video
frames
Shot #1
Shot #2
Shot #3
15

16. Method of [1]
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Smoothed time-series
Inferred time-series
t
Smoothed time-series of center of mass
displacement
16

17. ● Read the first out of every five videos frames
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Proposed Method
Our modifications:
Optimized set of parameters
17

18. ● Read the first out of every five videos frames
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Proposed Method
Our modifications:
Optimized set of parameters
Spatial sub-sampling
18

19. ● Read the first out of every five videos frames
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Proposed Method
Spatial sub-sampling
“Focus stability” mechanism
Our modifications:
Optimized set of parameters
19

20. ● Read the first out of every five videos frames
● Remove borders
● Calculate crop window dimensions
● Saliency detection
● Thresholding
● Filtering-through-clustering procedure
● Center of mass
● Shot detection
● Time-series smoothing
Proposed Method
Replace LOESS with a
Savitzky-Golay filter
Spatial sub-sampling
“Focus stability” mechanism
Our modifications:
Optimized set of parameters
20

21. Proposed Method
Deployed a Web application that:
1. Retrieves a video file
2. Analyzes the video
3. Transforms the video frames to the target aspect ratio
4. Renders the transformed video
21

22. Proposed Method
The REST service works through a 3-step process:
1. HTTP POST call to submit a video for analysis and the initiation of a relevant session
in the REST service
2. HTTP GET call to query the status of the initialized session and the progress of the
analysis
3. HTTP GET call to retrieve the results of a successfully completed session
22

23. User interface
● User can submit videos and
transform their aspect ratio
23

24. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
24

25. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
● Videos can be either available
on-line or locally stored
25

26. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
● Videos can be either available
on-line or locally stored
● The landing page includes 10
demo videos, and the ability to
provide feedback
26

27. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
● Videos can be either available
on-line or locally stored
● The landing page includes 10
demo videos, and the ability to
provide feedback
27

28. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
● Videos can be either available
on-line or locally stored
● The landing page includes 10
demo videos, and the ability to
provide feedback
● Analysis procedure monitoring
28

29. User interface
● User can submit videos and
transform their aspect ratio
● Predefined list of target aspect
ratios
● Videos can be either available
on-line or locally stored
● The landing page includes 10
demo videos, and the ability to
provide feedback
● Analysis procedure monitoring
● On-line inspection of the
results through the UI of our
tool or download the video file
29

30. Results
● We utilize the RetargetVid dataset and the evaluation protocol of [1] to compare:
○ Method of [1]
○ Method of [1] + modifications
30

31. Method Worst (↑) Best (↑) Mean (↑) T (% ↓)
Results for 1:3 target aspect ratio
[1] 48.6 50.9 49.9 19
[1] + proposed modifications 51.7 53.8 52.9 13
Results for 3:1 target aspect ratio
[1] 70.1 73.6 71.4 20
[1] + proposed modifications 74.4 77.0 75.3 14
Results (IoU; time as a percentage of the videos’ duration)
(↑: the higher the better; ↓: the lower the better)
31

32. Links
Instructional Video: https://youtu.be/_pdTDMWbIfs
Web Application: http://multimedia2.iti.gr/videosmartcropping/service/start.html
GitHub Repository: https://github.com/bmezaris/RetargetVid
● Source code of SmartVidCrop
● Source code of SmartVidCrop + our modifications
● Ground-truth annotations for the RetargetVid dataset
32

33. Try it yourself at:
Contacts:
Vasileios Mezaris, bmezaris@iti.gr
Konstantinos Apostolidis, kapost@iti.gr
This work was supported by the H2020 project ReTV (grant agreement No 780656)
http://multimedia2.iti.gr/videosmartcropping/service/start.html
Or ask us about the underlying algorithms
and how these can be integrated in your system
33