In this presentation, our work in the context of the Ad-hoc Video Search (AVS) Task of TRECVID 2022 is presented. Our participation in the AVS task is based on a cross-modal deep network architecture, T x V ("T times V"), which utilizes several textual and visual features. As part of the retrieval stage, a dual-softmax approach is also utilized to revise the calculated text-video similarities.

1. CRiTERIA - Comprehensive data-driven Risk and Threat Assessment Methods for the Early and Reliable Identification, Validation and Analysis of migration-related
risks - has received funding from the European Union’s Horizon 2020 research and innovation action program under grant agreement No 101021866.
Combining textual and visual features into
multiple joint latent spaces, and introducing
dual softmax re-ranking, for Ad-hoc Video
Search
Damianos Galanopoulos, Vasileios Mezaris
Information Technologies Institute-CERTH
TRECVID 2022 Workshop
December 6 - 9, 2022

2. Problem statement
2
● Retrieving video shots using natural-language textual queries
○ “Find a shot of a man with a white beard”

3. Our 2021 model 1,2,3
3
1 K. Gkountakos, D. Galanopoulos, D. Touska, K. Ioannidis, S. Vrochidis, V. Mezaris, I. Kompatsiaris, "ITI-CERTH participation in ActEV and AVS Tracks of
TRECVID 2021", Proc. TRECVID 2021 Workshop, Dec. 2021.
2 D. Galanopoulos, V. Mezaris, "Hard-negatives or Non-negatives? A hard-negative selection strategy for cross-modal retrieval using the improved
marginal ranking loss", Proc. IEEE/CVF ICCV Workshops (ICCVW), pp. 2312-2316, Oct. 2021. DOI:10.1109/ICCVW54120.2021.00261.
3 D. Galanopoulos, V. Mezaris, "Attention Mechanisms, Signal Encodings and Fusion Strategies for Improved Ad-hoc Video Search with Dual Encoding
Networks", Proc. ACM Int. Conf. on Multimedia Retrieval (ICMR 2020), Dublin, Ireland, 2020. DOI:10.1145/3372278.3390737.

4. ● We utilize the T x V cross-modal network
● Combination of textual and visual features
● Multiple latent spaces development
● Multi-loss-based learning
● Similarity revision using a dual softmax operation
Our 2022 approach 4
4
4 D. Galanopoulos, V. Mezaris, "Are All Combinations Equal? Combining Textual and Visual Features with Multiple Space
Learning for Text-Based Video Retrieval", Proc. ECCV 2022 Workshop on AI for Creative Video Editing and
Understanding (CVEU), Oct. 2022. https://arxiv.org/abs/2211.11351.

5. T x V model
Our 2022 approach
5
BoW
BERT
W2V
CLIP
ATT
CLIP
encoder
ResNet152-11k
ResNeXt-101-wsl
ViT-B/32 CLIP

6. Dual Softmax Inference
6
● Query-video similarities revision
● Inspired by the Dual Softmax loss
● Two approaches:
○ Evaluation with a priori knowledge of all queries
○ Query-agnostic evaluation

7. Dual Softmax Inference
7
● Two different background queries strategies
● AVS 2022 queries
● AVS 2019-2020-2021 queries

8. ● Textual Features
○ Bag-of-words
○ BERT
○ Word2Vec
○ ViT-B/32 CLIP
● Textual Encoders
○ Attention-based dual encoding network (ATT)
○ ViT-B/32 CLIP encoder
Features and encoders
8

9. Features and encoders
9
● Visual Features
○ ResNet-152 trained on the ImageNet11k dataset
○ ResNeXt-101 re-trained by weakly supervised learning on web
images and fine-tuned on ImageNet
○ ViT-B/32 CLIP model

10. ● Datasets:
○ Training: MSR-VTT, TGIF, ActivityNet Captions and Vatex
○ Evaluation:
■ V3C2 evaluated on TRECVID AVS 2022 queries
■ V3C1 evaluated on TRECVID AVS 2019-2021 queries
■ IACC.3 evaluated on TRECVID AVS 2016-2018 queries
● Evaluation metric:
○ Mean extended inferred average precision (MXinfAP)
Datasets and metrics
10

11. ● ITI_CERTH.22_run_2:
○ T x V model with 2 textual encoders and 3 visual features
○ Late fusion of 6 trained models with different configurations
○ Dual softmax using all AVS2022 queries as background queries
● ITI_CERTH.22_run_1:
○ Similar to run #2
○ Dual softmax using AVS 2019-2021 queries as background
Submitted runs
11

12. Results
12
MXinfAP for all submitted runs and T x V baseline for the fully-
automatic AVS task.

13. Results
13
AVS 2022 ranking list of all submitted runs regarding the main task in
MXinfAP terms. Red bars indicate our submitted runs.

14. “A person is in the act of swinging”
Dual Softmax impact
14
T x V model T x V model with DS (run #2)

15. “A drone landing or rising from the ground”
Dual Softmax impact
15
T x V model T x V model with DS (run #2)

16. “A type of cloth hanging on a rack, hanger, or line”
Dual Softmax impact
16
T x V model T x V model with DS (run #2)

17. Conclusions
17
● We utilize the cross-modal T x V network
● We combine multiple textual and visual features
● The dual softmax operation boosts the performance
● The a priori knowledge of evaluation queries slightly benefits the
performance
● A query-agnostic dual softmax revision serves real-life scenarios,
with a small trade-off regarding the overall performance

18. Thank you!
18
Damianos Galanopoulos, Vasileios Mezaris
Information Technologies Institute-CERTH
dgalanop@iti.gr, bmezaris@iti.gr
www.iti.gr/~bmezaris