https://imatge.upc.edu/web/publications/demonstration-open-source-framework-qualitative-evaluation-cbir-systems
Evaluating image retrieval systems in a quantitative way, for example by computing measures like mean average precision, allows for objective comparisons with a ground-truth. However, in cases where ground-truth is not available, the only alternative is to collect feedback from a user. Thus, qualitative assessments become important to better understand how the system works. Visualizing the results could be, in some scenarios, the only way to evaluate the results obtained and also the only opportunity to identify that a system is failing. This necessitates developing a User Interface (UI) for a Content Based Image Retrieval (CBIR) system that allows visualization of results and improvement via capturing user relevance feedback. A well-designed UI facilitates understanding of the performance of the system, both in cases where it works well and perhaps more importantly those which highlight the need for improvement. Our open-source system implements three components to facilitate researchers to quickly develop these capabilities for their retrieval engine. We present: a web-based user interface to visualize retrieval results and collect user annotations; a server that simplifies connection with any underlying CBIR system; and a server that manages the search engine data.
User Interface for an Image Retrieval Engine System
1. UI for an image
retrieval engine
system
Paula Gomez Duran
A project carried out in The Insight Centre for Data Analytics , in DCU
Kevin McGuinness, Eva Mohedano, Xavier Giró-i-Nieto
3. Why is a UI useful for CBIR ?
● The importance of
visualizing the results
● Ability to capture the
user's intent
33
4. Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
4
5. Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
5
7. ReactJs | NodeJS | Python
● Framework of JavaScript
● Scalability, speed and simplicity.
● Fast
● Virtual DOM
● Fast and scalable network apps
● Single-thread using non-blocking I/O calls
● Capable of handling huge number of simultaneous
connections with high throughput.
● NOT able to handle CPU-intensive operations
● Focuses on code readability
● Large standard libraries
● SLOW with speed in request or
response petitions processing
DEVELOPING UI
7
12. CBIR system
● All images analysed and stored
● Compare query analysed with all the other queries (cosine)
** Mohedano, Eva, Kevin McGuinness, Noel E. O'Connor, Amaia Salvador, Ferran Marques, and
Xavier Giro-i-Nieto. "Bags of local convolutional features for scalable instance search." ACM ICMR,
2016.
DEVELOPING UI
12
13. INPUTS OF THE SYSTEM
URL
IMAGE
FROM
FILE
SYSTEM
EXAMPLES
DEVELOPING UI
13
14. INPUTS OF THE SYSTEM
DEVELOPING UI
URL
SYSTEM
EXAMPLES
IMAGE
FROM
FILE
14
15. Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
15
16. Functionalities of the system
● Explorer mode
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
● Query expansion mode ● Annotation mode
16
17. Explorer mode :
- motivation → Get to know the datasets and explore the system
- functioning → When the first query is received and the ranking of the similar images is
computed, whichever other image appearing below can be selected now as the new query to
search into the dataset.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
17
18. Query expansion mode :
- motivation → Get to know how the algorithm works in the system.
- functioning → Average of the multiple image descriptors selected providing richer representation.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
18
19. Annotation mode :
- motivation → Improve the accuracy of the automatic system by user’s interaction.
- functioning → Annotating images and submit to the system to it can train an SVM.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
19
20. Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
20
21. Feedback of users
● UI intuitive
● UI robust and consistent
● UI fully featured
● Understand the purpose of the UI
● Understanding modes with existing explanations
● Explorer mode useful regarding the ‘clickable’ function
● Query expansion mode useful to experiment without affecting
the systems accuracy .
● Annotation mode useful to improve accuracy of a trained model.
Questionnaire data represented in a graphic
Strongly agree
Agree
QUANTITATIVE AND QUALITATIVE EVALUATION
80%
100%
80%
80%
70%
70%
90%
90%
21
22. Query expansion mode:
QUANTITATIVE AND QUALITATIVE EVALUATION
● Low Average Precision → can improve the ranking
● High Average Precision → Just adds noise
22
23. Annotation mode :
QUANTITATIVE AND QUALITATIVE EVALUATION
● Possibility to give just the negative feedback
● Possibility to train a model in order of improve the
system by just annotating some images of the dataset.
23
25. Conclusions
● UI for an image retrieval system.
● User’s feedback was positive in the questionnaire done.
● UI works with 3 commonly used CBIR benchmarks :
❖ Oxford, Paris and Instre
● Annotation tool has been developed
● Quantitative and qualitative evaluation have been carried out.
● Structure in blocks → Can be adapted for other retrieval
algorithms.
25
26. FUTURE WORK
● Include a 'Crop' mode on the query images to specify the region of interest.
● Unify structure of all datasets
● Include a mechanism to measure the time expand per query image.
● Include the ability to search within all photos in the three datasets.
26
Hinweis der Redaktion
eina pels cientifics per millorar els seus systems
As I just said, the goal was to develop a UI for a Content Based Image Retrieval search engine system. These systems emerged as a research field in order to fix issues in text-based systems. VISUAL SEARCH
They aim of this systems is to structure the datasets based on the content instead of on the metadata associated.
They work through algorithms which kind of summarize the content of an image into a numerical vector called ‘image representation’.
It is really important to visualize the results. Most systems, base their evaluation in a quantitative assessment, for example, computing measures as mean Average Precision to perform objectives comparisons with a groundtruth. However, in a real life system there is no groundtruth but just the user. So, qualitative assessments are even more important due the fact that good and bad examples can be shown to understand better how the system works. If there is no groundtruth, visualizing the data can be the only way of testing the results and maybe the only opportunity to realize when a system is failing and correct it.
The ability of capture the user’s intent is also really important. This is because it can happen that the similarity concept of the automatic system does not fit with what the user thing is similar. For this reason, it is also really important to develop tools which allows to the user to annotate his perception of whether the results are correct or not.
relevance feedback
My contributions on the project can be summarized in 3 fields.
The first one would be the development of the user interface
the second one would be incorporate to the system different modes of interaction with the users
and the last one would be the quantitative and qualitative evaluation.
So, the development of the UI is very important because it will leads us the tool to provide a visualization of the results obtained to, eventually, also improve them by getting the user’s feedback.
To build the system we needed to build first an structure which was able to support all the requirements on the system. We thought about this:
We thought about developing a web application, which would be all the client side, and connect it with the image retrival system through a server.
Choosing the tools to develop a project is always a difficult task. However, taking into account the different needs of the project when we are in the designing part can make it easier.
3 languages are presented in the state of art of this project, and they are ReactJS, nodeJS and Python.
// EXPLICAR ELS 3
So, taking into account that the server should allow exchanging data with the client in order to provide fast speed and consistent data structures, we decided to built the front end or client part in reactjs and the server in nodejs, due its strength in processing request-response petitions.
However, we also needed a server able to support all the mathematical operations in order to calculate the rankings of a query. A solution for this was to build also another server, this one in python, that would incorpore all the image retrieval system code and maintain also a connection with the nodejs server. This connection will be done by ZERO_RPC, which is a library that allows to establish the connection between both programming languages.
The three datasets used are :
** these are very popular datasets in the scientific community working in CBIR.
- Oxford Buildings contains 5,063 images and Paris Buildings contains 6,412 images. These both datasets have collected the images from Flickr, a photo sharing community.
- In the other hand, we also have Instre dataset. This dataset is larger and contains 28,543 images collected from multiple
sources, such as search engines as Google, different social networks as facebook and also photo sharing communities like Flickr.
The aim of the project is to develop an UI capable to compute the list of similar images given a query. So when introducing a query into the system this page will be displayed where you can observe the most similar images at the top of the raking and the less similar ones at the bottom of it.
So, i want to brief a little bit how does the CBIR system works. The system i’m using is from the state of art . it was developed at DCU & UPC and it was awarded in a conference in NY in 2016.
In the CBIR system, first all of the images are analysed and stored. Analysing the images follows the next proceedment.
first of all, the image is introduced into a pre-trained CNN and decomposed in local deep learning descriptors. Then, this descriptors will be codified by a BoW model using a specific vocabulary. This descriptor vector is the one stored for each image.
So, when a query is received into the system, this image is also analysed in order to get the final descriptor vector as gotten above. This vector will be compared with all the other vector’s queries of the system by cosine similarity and then, the ranking will be generated by sorting the similarity scores in descending order.
When two images are really similar, the cosine similarity score is nearly 1.
Cosine similarity computedSimilarity scores in descending orderRanking computed
A web UI is designed so that different alternatives are provided when choosing an image to enter into the system. Thus, the interface will have the option of using some of the already existing images provided as an example by the system, as well as an option of experimenting with a user's own image, either uploading it from a file or providing its URL to the system.
In the second contribution to the project the aim was to incorporate to the system different modes of interaction with the users. Each mode built has different purposes
and it is explained in future slides.
The explorer mode’s motivation is to navigate through the system and get to know the different datasets. When a query is received and its ranking is computed, whichever of these images that appear below like in this image, are clickable to become the new query on which one start the new search.
The query expansion mode’s motivation is to check how the deep learning algorithm works by a qualitative evaluation. This mode works by selecting multiples images. The average of their descriptors is done and the search and ranking are computed again. A richer representation is provided to the system, allowing it to improve . In this image it is not very clear but in the demo at the end of the presentation it can be better appreciated.
relevance feedback
The annotation’s mode motivation is to improve the system’s accuracy by user’s interaction. Annotate successful and failure images of the ranking and submit the annotations to the system. It is thought to train a model in order to classify what is relevant of what is not. Then, the ranking is update it.
support vector machines (SVMs, also support vector networks[1]) are supervised learning models with associated learning algorithms that analyze data used for classificationand regression analysis.
functioning → annotate successful and failure images of the ranking and submit the annotations to the system. It is thought to train a model in order to classify what is relevant of what is not. Then, the ranking is update it.
The last contribution is the quantitative and qualitative evaluation of the system. It is done in the first place by the results of a questionnaire, and then through the comparison between query expansion and annotation modes.
An experiment is carried out by asking to 10 people if they can use the system for about 10 minutes and then, answer these questions. The questionnaire was measured with a likert scale where the users choose an option amongst : SD, D, N, A or SA.
In this slide it is shown a brief of the results obtained on the questionnaire.
The statistics concluded that the UI is intuitive, robust, consistent and built with a useful purpose. The aim of its different modes of operation is understood by the users although some hover info icons are advised in the comments.
The users also agreed on the fact that each mode has its own utility thus it is very useful either giving straight feedback to the system as well as being able to experiment with hypothetical situations without affecting it.
Last but not least, the fact that an explorer mode is developed with 'clickable' images, has been the most liked feature.
On the QE mode, the 2 next conclusions have been achieved.
The first one, is that taking a query with Low Average Precision makes the ranking improves either in the quantitative way as well as in the visualization of the results, what means the qualitative way.
The second one, as we can also see in quantitative and qualitative ways, is that when selecting an image with hight AP, it does not improve the accuracy but just introduce noise.
For this reason, if a really accurate ranking wants to be improved, its is much better to use the annotation mode. This is due the possibility of annotating specifically the negative images that doesn’t belong to the ranking .
It is also useful the possibility of training an svm because when we have thousands of images we don’t want to annotate the full dataset. Then, instead of annotating all, we can just annotate some of the images and then, train a mode, in this case an svm was trained, which will predict the classification for all the images.
The final conclusions are summarized in this points:
So, we built an UI for an image retrieval system and the feedback collect it by the users denoted that the UI is intuitive, robust, consistent and built with a useful purpose.
The UI works with 3 commonly used CBIR benchmarks that allow internal and external searches of queries. This are Oxford, Paris and instre dataset.
An annotation tool to collect the intent of the users is developed to improve the image retrieval system.
Quantitative and qualitative evaluations have been carried out and different scenarios have been studied in order to improve the system
The system is designed in a way that the structure can be used for other search engines just changing this module. For this reason, experimenting with another retrieval system could be done and that is why a possible open source contribution has been talked about.