Invited presentation at the JOLT ETN virtual training event, June 2020.

1. Thessaloniki, June 2020
Video, AI and News: video
analysis and verification
technologies for supporting
journalism
Vasileios Mezaris
CERTH-ITI
1

2. Thessaloniki, June 2020Vasileios Mezaris
Video and the News
Fundamental need:
- Make good use of video (either own or 3rd-party content) to convey your
message!
Technology-wise, this requires:
- Verifying 3rd-party content; avoiding distributing falsehoods
- Understanding the content / making it dicoverable (especially own content)
- Adapting / re-purposing the (verified, discovered) content
This is where AI can step in!
2

3. Thessaloniki, June 2020Vasileios Mezaris
Verifying the content
Main requirements:
- Check if a supposedly new (breaking-news?) video is just “stock footage”, is
used out of context; the “easy fake”
- Check for manipulations / editings in the video
3

4. Thessaloniki, June 2020Vasileios Mezaris
Verifying the content
InVID Verification Plugin: a tool for the verification of selected newsworthy videos
4
A browser plugin to debunk fake
news and to verify videos and
images
>20.000
users
- Check prior video use: reverse video
search on the Web
- Check contextual information: Social-
media-based contextual analysis
- Keyframe/image inspection by
magnifying glass
- Check image (keyframe) forensics
Free! Get it from: https://www.invid-project.eu/verify

5. Thessaloniki, June 2020Vasileios Mezaris
Verifying the content
Web application for video fragmentation and reverse web search (try it!):
http://multimedia3.iti.gr/video_fragmentation/service/start.html
5

6. Thessaloniki, June 2020Vasileios Mezaris
Verifying the content
6
https://www.youtube.com/watch?v=OVAxQA3gMEo
“Video, the crowd in panic flees
from Notre Dame in Paris after
the attack of an armed man”

7. Thessaloniki, June 2020Vasileios Mezaris
Verifying the content
Behind the scenes:
- Efficient and effective video fragmentation [1][2]
- Can be extended with deep-learning based fragmentation techniques
A more global view of video verification for News can be found in [3]
7
[1] E. Apostolidis, V. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proc. IEEE Int. Conf. on
Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, May 2014. Software available at
https://mklab.iti.gr/results/video-shot-and-scene-segmentation/.
[2] K. Apostolidis, E. Apostolidis, V. Mezaris, "A motion-driven approach for fine-grained temporal segmentation of user-
generated videos", Proc. 24th Int. Conf. on Multimedia Modeling (MMM2018), Bangkok, Thailand, Feb. 2018.
[3] "Video verification in the fake news era", V. Mezaris, L. Nixon, S. Papadopoulos, D. Teyssou (Editors), Springer, 2019.

8. Thessaloniki, June 2020Vasileios Mezaris
Understanding the content / making it discoverable
Content fragmentation and annotation:
- Identify the different temporal fragments of a video (subshots/shots/scenes)
- Annotate each fragment with text that describes it: concept labels, sentiment
labels, descriptive captions
- Enable the retrieval of video fragments, using any of these labels, or free-text
queries
8

9. Thessaloniki, June 2020Vasileios Mezaris
Understanding the content / making it discoverable
9
Shot #15
Scene #4 Scene #5
Shot #11 Shot #12 Shot #13 Shot #14 Shot #16
Subshot #58 Subshot #59
Shot #17 Shot #18
Subshot #60
…
… …
……

10. Thessaloniki, June 2020Vasileios Mezaris
Understanding the content / making it discoverable
10
Sample video frame Top detected concepts

11. Thessaloniki, June 2020Vasileios Mezaris
Understanding the content / making it discoverable
Web application [4] for video analysis and search (try it with your video!):
http://multimedia2.iti.gr/onlinevideoanalysis_v5/service/start.html
Demo video:
https://youtu.be/mO-NRpIJ9UU
11
[4] C. Collyda, E. Apostolidis, A. Pournaras, F. Markatopoulou, V. Mezaris, I. Patras,
"VideoAnalysis4ALL: An on-line tool for the automatic fragmentation and concept-based
annotation, and the interactive exploration of videos", Proc. ACM ICMR 2017, Bucharest,
Romania, June 2017.

12. Thessaloniki, June 2020Vasileios Mezaris
Understanding the content / making it discoverable
Behind the scenes:
- Efficient and effective video fragmentation
- Elaborate deep-neural-network architectures for concept-based annotation,
e.g. [5][6] (and for video captioning, sentiment analysis, video retrieval using
free-text queries [7]; not shown in this demo)
[5] F. Markatopoulou, V. Mezaris, I. Patras, "Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation", IEEE
Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1631-1644, June 2019. DOI:10.1109/TCSVT.2018.2848458. Software available at
https://github.com/markatopoulou/fvmtl-ccelc.
[6] N. Gkalelis, V. Mezaris, "Subclass deep neural networks: re-enabling neglected classes in deep network training for multimedia classification", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
[7] D. Galanopoulos, V. Mezaris, "Attention Mechanisms, Signal Encodings and Fusion Strategies for Improved Ad-hoc Video Search with Dual Encoding Networks",
Proc. ACM Int. Conf. on Multimedia Retrieval (ICMR 2020), Dublin, Ireland, 2020.
12

13. Thessaloniki, June 2020Vasileios Mezaris
Adapting / re-purposing the content
Main requirements:
- Target distribution platforms & devices have varying requirements (e.g. the
optimal duration of a video differs from one platform to another)
- Target audiences have different preferences / information needs
Video summarization:
- Create editions of the content that are adapted to different platforms and
audiences
13

14. Thessaloniki, June 2020Vasileios Mezaris
Adapting / re-purposing the content
Example
- Original video (1’38’’)
- 14’’ summary
14

15. Thessaloniki, June 2020Vasileios Mezaris
Adapting / re-purposing the content
Web application [8] for video summarization (try it with your video!):
http://multimedia2.iti.gr/videosummarization/service/start.html
Demo video:
https://youtu.be/LbjPLJzeNII
15
[8] C. Collyda, K. Apostolidis, E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, "A
Web Service for Video Summarization", Proc. ACM Int. Conf. on Interactive Media
Experiences (IMX 2020), Barcelona, Spain, June 2020.

16. Thessaloniki, June 2020Vasileios Mezaris
Adapting / re-purposing the content
Behind the scenes:
- Elaborate generative adversarial learning architectures (GANs) for
unsupervised learning [9][10]
- Can be trained differently for different content, e.g. separate trained models
can be used for different shows; but, creating these models does not require
manually-generated training data (it’s (almost) for free)
[9] E. Apostolidis, A. Metsai, E. Adamantidou, V. Mezaris, I. Patras, "A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised
Video Summarization", Proc. 1st Int. Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV'19) at ACM Multimedia 2019, Nice, France,
October 2019.
[10] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, I. Patras, "Unsupervised Video Summarization via Attention-Driven Adversarial Learning", Proc. 26th Int.
Conf. on Multimedia Modeling (MMM2020), Daejeon, Korea, Jan. 2020.
16

17. Thessaloniki, June 2020Vasileios Mezaris
Some concluding thoughts
17
- AI tools for journalists are desperately needed!
- Even relatively simple approaches, such as video fragmentation for reserse
Web search, can have great impact (verification is becoming more important)
- The introduction of video understanding outputs (concept labels etc.) in video
archives can give new life to existing / continuously archived content
- Automatic generation of new content (e.g. video summaries) is a hot topic
- Complete automation is sometimes not desired! (AI + human symbiosis is key)
- New challenges & advances in AI (e.g. Explainable AI) will create new
opportunities for even greater use of AI in journalism

18. Thessaloniki, June 2020Vasileios Mezaris
Questions?
18
Contact: Dr. Vasileios Mezaris
Information Technologies Institute
Centre for Research and Technology Hellas
Thermi-Thessaloniki, Greece
Tel: +30 2311 257770
Email: bmezaris@iti.gr, web: http://www.iti.gr/~bmezaris/
This work was supported in part by the EU’s Horizon 2020 research and innovation programme under grant
agreements H2020-687786 InVID, H2020-732665 EMMA, H2020-693092 MOVING, and H2020-780656 ReTV.