https://imatge.upc.edu/web/publications/visual-question-answering-20 This bachelor's thesis explores dierent deep learning techniques to solve the Visual Question-Answering (VQA) task, whose aim is to answer questions about images. We study dierent Convolutional Neural Networks (CNN) to extract the visual representation from images: Kernelized-CNN (KCNN), VGG-16 and Residual Networks (ResNet). We also analyze the impact of using pre-computed word embeddings trained in large datasets (GloVe embeddings). Moreover, we examine dierent techniques of joining representations from dierent modalities. This work has been submitted to the second edition Visual Question Answering Challenge, and obtained a 43.48% of accuracy.

1. Visual Question Answering 2.0
Slides by Francisco Roldán
Bsc Thesis, UPC
12th July, 2017
Author: Francisco Roldán Sánchez
Advisors: Xavier Giró-i-Nieto, Santiago Pascual de la
Puente, Issey Masuda Mora

2. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
2

3. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
3

4. Visual Question Answering
4
AI System
Visual Question AnsweringVisual Question AnsweringVisual Question AnsweringVisual Question Answering

5. Why VQA?
5

6. Why VQA?
● Multidisciplinary task.
● Models need to tackle different sub-tasks at once
6
Natural Language
Processing
Knowledge
Representation
Computer Vision

7. VQA Challenge
7Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., & Parikh, D. Vqa: Visual question answering. ICCV (2015)

8. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
8

9. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
9

10. VQA: Common solution
10
WE

11. Convolutional Neural Networks (CNN)
11

12. Word embeddings
12

13. Recurrent Neural Networks (RNN)
13

14. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
14

15. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
15

16. VQA Dataset 2.0
16
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2016). Making the V in VQA matter: Elevating the role of image
understanding in Visual Question Answering. arXiv preprint arXiv:1612.00837.

17. VQA Dataset 2.0 Population
17
Train Validation Test
Images 82,783 40,504 81,434
Questions 443,757 214,354 447,793
Answers 4,437,570 2,143,540 4,477,930
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2016). Making the V in VQA matter: Elevating the role of image
understanding in Visual Question Answering. arXiv preprint arXiv:1612.00837.

18. Evaluation Metric
18

19. Loss Function
19

20. UPC 2016 Model
20
Yes/No Number Other Overall
66.05 29.77 20.35 40.25
Masuda, I., de la Puente, S. P., & Giro-i-Nieto, X. (2016). Open-Ended Visual Question-Answering. arXiv preprint arXiv:1610.02692.

21. Language-Only Model
21
Pennington, J., Socher, R., & Manning, C. D. Glove: Global vectors for word representation. In EMNLP (2014, October)

22. Language-Only Model
22
Freezing or fine-tuning GloVe embeddings:
Fine-tuning
Freezing
Yes/No Number Other Overall
66.14 30.42 24.47 42.31
66.15 31.17 24.87 42.59

23. 23
VQA
1.0
VQA
2.0

24. ResNet based model
24

25. ResNet based model
25

26. ResNet based model
26

27. ResNet based Model
27
Selection of the merge operand
Concat + FC
Concat
Product
Sum
Yes/No Number Other Overall
64.66 29.49 21.27 40.08
65.51 29.99 22.02 40.84
65.90 30.00 22.80 41.37

28. VGG based model
28

29. VGG based Model
29
Selection of the merge operand
Sum
Product

30. VGG based Model
30
Adding Average Pooling to VGG based model

31. VGG based Model
31
Adding Average Pooling to VGG based model
No pooling
Avg pooling

32. VGG based Model
32
Deciding between GloVe or learnable embeddings:
Glove
Learnable
Yes/No Number Other Overall
66.59 31.01 25.83 43.21
67.10 31.54 25.46 43.30

33. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
33

34. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
34

35. Results
35
0% 100%50%
43.48%
Our method
25.98%
UPC 2016*
Priors
40.25%
* Accuracy obtained from the test-dev split, where our method obtained a 43.30%
LV_NUS
69.71%

36. Qualitative Results
36

37. Qualitative Results
37

38. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
38

39. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
39

40. Conclusions
Summary:
- GloVe embeddings work better when frozen .
- Need of spatial information for VQA task.
- Models tend to learn language biases.
Goals achieved:
- Improve last year model’s performance by 3.05%.
- Participate in the VQA 2.0 Challenge, obtaining a 43.48% of accuracy.
- Explore many different Deep Learning techniques.
- Build a reusable modular software.
40

41. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
41

42. Roadmap
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Conclusion
6. Future Work
42

43. Visual Reasoning
43
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2016). CLEVR: A diagnostic dataset for
compositional language and elementary visual reasoning. arXiv preprint arXiv:1612.06890.

44. 44

45. Visual Reasoning
45
Johnson, J., Hariharan, B., van der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017). Inferring and
Executing Programs for Visual Reasoning. arXiv preprint arXiv:1705.03633.

46. LSTM
46