Region-oriented Convolutional Networks for Object Retrieval
•
1 like•2,835 views
This document describes research on using region-oriented convolutional neural networks for object retrieval. It discusses using local CNNs like CaffeNet, Fast R-CNN, and SDS to extract visual features from object candidates in images. These features are used to match against query descriptors. Pooled regional features are ranked to retrieve relevant shots. Fine-tuning pre-trained networks on larger datasets like COCO can improve retrieval accuracy. Combining global and local approaches through re-ranking provides an additional boost in performance.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Region-oriented Convolutional Networks for Object Retrieval
Similar to Region-oriented Convolutional Networks for Object Retrieval (20)
More from Universitat Politècnica de Catalunya
More from Universitat Politècnica de Catalunya (20)
Recently uploaded
Recently uploaded (20)
Region-oriented Convolutional Networks for Object Retrieval
- 1. REGION-ORIENTED CONVOLUTIONAL NETWORKS FOR OBJECT RETRIEVAL Eduard Fontdevila Amaia Salvador Xavier Giró-i-Nieto ADVISORSAUTHOR
- 2. ACKNOWLEDGEMENTS 2 Financial Support Technical Support Albert Gil Josep Pujal
- 3. OUTLINE 1. Motivation 2. State of Art 3. Local CNNs for Instance Search 4. Fine-tuning 5. Conclusions 3
- 4. visual Data is Big Data 4 motivation
- 6. ... and visual Data needs Computer Vision 6 COMPUTER VISION motivation
- 8. OUTLINE 1. Motivation 2. State of Art 3. Local CNNs for Instance Search 4. Fine-tuning 5. Conclusions 8
- 9. from shallow to deep learning 9 Bag of Words SIFT Histograms of gradients Convolutional Neural Networks (CNNs) “hand crafted” features state of art “learned” features
- 10. why deep learning now? 10 state of art large datasets Powerful GPUs ...
- 11. AlexNet 11 state of art Krizhevsky et al. (Toronto), ImageNet Classification with Deep Convolutional Neural Networks (2012)
- 12. CaffeNet 12 state of art CaffeNet architecture [Krizhevsky’12] data [Deng’09] framework [Jia’14] Slide credit: Xavier Giró-i-Nieto
- 13. CaffeNet 13 state of art input image Babenko et al. (Moskow), Neural Codes for Image Retrieval (2014)
- 14. CaffeNet 14 state of art convolutional layers Babenko et al. (Moskow), Neural Codes for Image Retrieval (2014)
- 15. CaffeNet 15 state of art fully connected layers Babenko et al. (Moskow), Neural Codes for Image Retrieval (2014)
- 16. object candidates 16 state of art Selective Search bounding boxes Uijlings et al. (Trento), Selective Search for Object Recognition (2013) MCG segments Arbeláez et al. (Berkeley), Multiscale Combinatorial Grouping (2014)
- 17. R-CNN 17 state of art Girshick et al. (Berkeley), Rich feature hierarchies for accurate object detection and semantic segmentation (2014) Object Detection network
- 18. fast R-CNN 18 state of art R. Girshick (Berkeley), Fast R-CNN (2015)
- 19. SDS 19 state of art Hariharan et al. (Berkeley), Simultaneous Detection and Segmentation (2014) Object Detection + Semantic Segmentation network
- 20. OUTLINE 1. Motivation 2. State of Art 3. Local CNNs for Instance Search 4. Fine-tuning 5. Conclusions 20
- 21. TRECVid Instance Search 21 local CNNs for instance search large collection of videos 464h shots ~470k frames 1/4 fps
- 22. TRECVid Instance Search 22 local CNNs for instance search large collection of videos 464h shots ~470k frames 1/4 fps ...in our case, subset of 13k shots (23k frames)
- 23. a Big Data scenario 23 local CNNs for instance search
- 24. query descriptors 24 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features visual features query set descriptors image bbox region
- 25. query descriptors 25 local CNNs for instance search query set examples of TRECVid query images
- 26. query descriptors 26 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features visual features query set descriptors image bbox region
- 27. object candidates main scheme 27 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features visual features query descriptors matching matching matching frames in 1 shot pooling pooling pooling ranking ranking ranking
- 28. object candidates pooling pooling visual features visual features main scheme 28 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features query descriptors matching matching matching frames in 1 shot pooling ranking ranking ranking global approach
- 29. poolingvisual features object candidates pooling pooling visual features visual features main scheme 29 local CNNs for instance search CaffeNet Fast R-CNN SDS query descriptors matching matching matching frames in 1 shot ranking ranking ranking global approach
- 30. visual features pooling object candidates pooling pooling visual features visual features main scheme 30 local CNNs for instance search CaffeNet Fast R-CNN SDS query descriptors matching matching matching frames in 1 shot ranking ranking ranking global approach
- 31. visual features pooling object candidates pooling pooling visual features visual features main scheme 31 local CNNs for instance search CaffeNet Fast R-CNN SDS query descriptors matching frames in 1 shot matching matching ranking ranking ranking global approach euclidean distance Babenko et al. (Moskow), Neural Codes for Image Retrieval (2014)
- 32. poolingvisual features object candidates pooling pooling visual features visual features main scheme 32 local CNNs for instance search CaffeNet Fast R-CNN SDS query descriptors matching matching matching frames in 1 shot ranking ranking ranking global approach Zhu et al. (NII), Multi-image aggregation for better visual object retrieval (2014) distance frame 1 distance frame 2 distance frame 3 average distance distance shot - query =
- 33. poolingvisual features object candidates pooling pooling visual features visual features main scheme 33 local CNNs for instance search CaffeNet Fast R-CNN SDS query descriptors matching matching matching frames in 1 shot ranking ranking ranking global approach only top1000 shots
- 34. object candidates main scheme 34 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features visual features query descriptors matching matching matching frames in 1 shot pooling pooling pooling ranking ranking ranking
- 35. visual features pooling object candidates main scheme 35 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features query descriptors matching matching matching pooling pooling ranking ranking ranking local approach frames in 1 shot
- 36. object candidates main scheme 36 local CNNs for instance search CaffeNet Fast R-CNN SDS frames in 1 shot local approach
- 37. visual features pooling object candidates main scheme 37 local CNNs for instance search CaffeNet Fast R-CNN SDS visual features visual features query descriptors matching matching matching pooling pooling ranking ranking ranking local approach frames in 1 shot
- 38. quantitative results: ranking 38 local CNNs for instance search mAP (%) SDS Fast R-CNN
- 39. re-ranking 39 local CNNs for instance search CaffeNet SDS / F-RCNN re-ranking global + local fusion
- 40. quantitative results: re-ranking 40 mAP (%) SDS Fast R-CNN CaffeNet local CNNs for instance search
- 41. quantitative results: re-ranking 41 mAP (%) SDS Fast R-CNN CaffeNet local CNNs for instance search adding context ~8%
- 42. qualitative results: re-ranking 42 query SDS Fast R-CNN local CNNs for instance search
- 43. qualitative results: re-ranking 43 query SDS Fast R-CNN local CNNs for instance search
- 44. as a reminder... 44 local CNNs for instance search Selective Search bounding boxes Uijlings et al. (Trento), Selective Search for Object Recognition (2013) MCG segments Arbeláez et al. (Berkeley), Multiscale Combinatorial Grouping (2014) Fast R-CNN SDS
- 45. OUTLINE 1. Motivation 2. State of Art 3. Local CNNs for Instance Search 4. Fine-tuning 5. Conclusions 45
- 46. training CNNs from scratch is costly... 46 fine-tuning
- 47. ... instead: fine-tuning 47 fine-tuning already trained network new dataset (novel domain) resume training
- 48. a quick trial 48 fine-tuning CaffeNet Pascal dataset
- 49. results on Pascal (global scale) 49 fine-tuning validation subset validation set accuracy (%) 59,31% 4,14% Histogram of images per category categories %ofimages
- 50. Microsoft COCO 50 fine-tuning ● Multiple objects per image ● 80 categories ● > 300k images (80k training) ● > 2M instances Lin et al. (Cornell - Microsoft), http://vision.ucsd.edu/sites/default/files/coco_eccv.pdf (2015)
- 51. fine-tuning SDS on COCO 51 fine-tuning SDS network COCO dataset resume training
- 52. fine-tuning SDS on COCO 52 fine-tuning SDS network COCO dataset resume training
- 53. ... but why? 53 fine-tuning the more objects the network knows, the better
- 54. OUTLINE 1. Motivation 2. State of Art 3. Local CNNs for Instance Search 4. Fine-tuning 5. Conclusions 54
- 55. about the results ● Although not outperforming CaffeNet: SDS good for localization! 55 conclusions maybe more suitable for TRECVid localization task?
- 56. about fine-tuning ● Networks trained on objects, but not on the objects to retrieve 56 conclusions fine-tuning on a larger dataset is clearly the next step
- 57. about object candidates ● Only 100 candidates decreseases likelihood to success ... but using a higher number 57 conclusions Fast SDS would be the key
- 58. thank you
- 59. visualizing CNNs’ features more class-specific information annex
- 60. SDS: Proposal Generation input image MCG object candidates segments, not only bounding boxes annex
- 62. SDS: Feature Extraction object candidate penultimate fully connected layers annex
- 63. SDS: Region Classification Linear SVM annex
- 65. basic pipeline for retrieval annex
- 66. interactive: Multi-image aggregation Query images for a topic was used with the min distance to each shot. The best option with SIFT-BoW is average, wheteher features (Avg-Pooling) or similarity scores (Sim-Avg) annex Zhu et al. (NII), Multi-image aggregation for better visual object retrieval (2014)