This document describes CLEAR, a fully user-side image search system developed by Ryoma Sato at Kyoto University. CLEAR allows users to build and publish their own image search engines without backend servers by formulating image search as a multi-armed bandit problem and implementing the system using only JavaScript on the client-side. This overcomes limitations of traditional search engines which require extensive resources to operate. CLEAR demonstrates that ordinary users can now develop customized image search tools.
3. 3 KYOTO UNIVERSITY
Problem: each user wants different services
A’s ideal service
Bs’ ideal services
C’s ideal service
D’s ideal service
gap
Actual Service
4. 4 KYOTO UNIVERSITY
User-side systems realize users’ ideals
A’s ideal service
Bs’ ideal services
C’s ideal service
D’s ideal service
Actual Service
User-side systems fill this
gap on the user’s side
5. 5 KYOTO UNIVERSITY
Challenge: users cannot access to the database
Standard IR sysmtes
are implemented by Engineers of
Twitter, who have complete
access to the database.
User-side IR systems
are implemented by users, who don’t have direct access to the database.
Twitter Inc.
Database
Twitter Engineer
IR system
SQL
ANN
Twitter Inc.
Database
User
SQL
Search Query
API
6. 6 KYOTO UNIVERSITY
Flickr provides text search only, no image-based
↑put text here to search for images
Flickr supports tag
and text-based
searches, but no
image-based search
We want to find
images with similar
atmosphere,
similar composition,
and similar colors,
based on images
at hand
7. 7 KYOTO UNIVERSITY
Suppose we have a score function
Suppose we have a function that receives an image
and returns a scalar score
Ex) The more similar a image is to the image at hand,
the higher the score.
Similarity can be calculated based on composition,
color, etc. It is up to the user to decide which one to
use (depending on what the user wants to do).
0.882
8. 8 KYOTO UNIVERSITY
Maximize the score function at hand fron Flickr
Suppose we have a function that receives an image
and returns a scalar score
Problem setup: retrieve (in real time) the image with
the maximum score from all Flickr images when the
score function is passed by the user.
0.882
9. 9 KYOTO UNIVERSITY
Naive methods are intractable
Method 1: A very naive method is to crawl a large
number of images from Flickr, evaluate each one's
score, and return the image with the highest score
→ High communication cost, not finish in real time
Method 2: Pre-crawl and build an index
→ Storage and server costs are high, and is intractable
for an ordinary user.
10. 10 KYOTO UNIVERSITY
Architecture of CLEAR
CLEAR searches images on the fly
CLEAR runs on the frontend
No indices or crawling
No backend servers
11. 11 KYOTO UNIVERSITY
Formulate as a multi-armed bandit problem
Regard the search query as an arm and formulate the
problem as a multi-armed bandit problem
q = dog q = cat q = Persian cat q = goldfish
0.725
Issue a query
reward
...
Score function f
13. 13 KYOTO UNIVERSITY
Develop and publish your own search engine
Contribution: An ordinary user realizes a new search
engine for the existing service.
You can easily develop your own search engine.
↔ Traditionally, building search engines requires
a lot of storage, computational, and network
resources.
14. 14 KYOTO UNIVERSITY
CLEAR is general
CLEAR
score function A
content-based
user specifies
the function
16. 16 KYOTO UNIVERSITY
CLEAR is general
CLEAR
score function B
hue-based
you can change
the target service
17. 17 KYOTO UNIVERSITY
CLEAR is general
CLEAR
score function B
hue-based
you can change
the target service
18. 18 KYOTO UNIVERSITY
Develop and publish your own search engine
CLEAR is implemented fully on the client’s side
(with JavaScript)
No back-end servers
→ You can realize your own search engine by rewriting
the score function and opening it in a browser.
→ Deployment is easy
You can deploy in on GitHub Pages or Amazon S3.
→ The more people create and publish their own
search systems, the more options there are, and the
easier it will be to find a favorite search system.
↔ Only Flickr official ones and a few voluntary ones.
19. 19 KYOTO UNIVERSITY
Summary: First fully user-side image search
CLEAR is the first practical implementation of
fully user-side image search.
The system is implemented completely on the client-
side (Javascript).
You can build and publish your own search engine.
Code: https://github.com/joisino/clear
Demo: https://clear.joisino.net/
Fork the repository and
build your own search engine!
