Andrii Boichuk: Video-based action recognition - past, present and future AI & BigData Online Day 2022 Autumn Website - https://aiconf.com.ua Youtube - https://www.youtube.com/startuplviv FB - https://www.facebook.com/aiconf

1. Video-based action recognition -
past, present and future
Andrii Boichuk

2. ● About me
● What is Action Recognition?
● Data for action recognition
● Pre-DL era models
● DL-era models
● Future prediction
● Q&A
Lecture structure
2

3. ● 10+ years in IT
● 5+ years in AI & CV
● research engineer
● Lviv Polytechnic National University - lecturer, PhD student
● Волонтер в
КомуШоТреба(https://www.facebook.com/komushotreba)
About me
3

4. #
What is action recognition?

5. ● Action classification seeks to assign the correct label (e.g. “cooking,”
“writing,” etc.) to a given image or video.
● Action localization, given a particular action and a video as input, seeks to
identify the correct location and timestamp in the video when the action is
being performed.
5
Action recognition

6. 6
Action recognition applications

7. #
DATASETS

8. 8
HMDB-51 - 7 000 clips

9. 9
UCF101 - 13320 videos

10. 10
Kinetics - 650 000 clips
label youtube_id time_start time_end split
pouring milk -2gUy2nHjDQ 3 13train
squeezing orange -3CGqhQXQTo 7 17train
card stacking -3Xt-82JfPQ 301 311train

11. 11
Kinetics

12. 12
Something-Something v. 2 - 220 000 clips
Sample classes
● Putting something on a surface
● Moving something up
● Covering something with something
● Pushing something from left to right
● Moving something down

13. 13
Something-Something v. 2

14. 14
Activity net
● 200 classes
● 648 hours
● 100 untrimmed videos per class
● 1.54 activities per video

15. 15
Activity net

16. #
Action recognition past

17. 17
PCA and hand made video cropping

18. 18
Space-time interest points

19. 19
Online action recognition from RGB-D cameras based on
reduced basis decomposition (2017!!)

20. Today

21. 21
Large-scale Video Classification with Convolutional Neural
Networks

22. 22
3D Convolutional Neural Networks for Human Action
Recognition

23. 23
Quo Vadis, Action Recognition? A New Model and the
Kinetics Dataset (I3D)

24. 24
Quo Vadis, Action Recognition? A New Model and the
Kinetics Dataset (I3D)

25. 25
MoViNets: Mobile Video Networks for Efficient Video
Recognition

26. 26
MoViNets: Mobile Video Networks for Efficient Video
Recognition

27. 27
Tips
● Don’t expect transfer learning to work
● Use robust feature extractors if possible
● The simpler the baseline the better
● Expect to need lots of resources. 1 Nvidia gpu isn’t
enough.
● You’ll need a lot of data. 20 hours won’t be enough.

28. Todayish

29. 29
MERLOT

30. 30
MERLOT Reserve

31. 31
MERLOT Reserve

32. 32
XClip

33. ● Transfer learning
● Model efficiency
● Transformers, 3D CNN or smth else?
33
Unanswered questions

34. Q&A

35. ● Tg @ThatAiGeek
● Donate to ЗСУ
● https://savelife.in.ua/
● https://www.facebook.com/komushotreba