Hear how Adobe scales, manages multiple use cases, and puts machine learning features to work with Elastic and learn about extensions to Elasticsearch that allow them to search at scale natively.

1. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Elastic @ Adobe: Making Search Smarter with Machine Learning at Scale
--- Baldo Faieta & Gaurav Kukal

2. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. 2
Adobe is unique in Search space as large
part of content is non textual in nature like
images, videos, 3d templates, psd, dcx but at
the same time billions of content pieces have
text as well ….

3. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
A buffet of Search Use Cases
3
1 2 3 4 5
Search Based
on Computer
Vision &
Metadata
Deep Textual
and hybrid
content Search
Video and
Richer format
Search
Enterprise
Search
Discovery and
Recommendation
s

4. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
“Computer vision is an interdisciplinary field that
deals with how computers can be made to gain
high-level understanding from digital
images or videos. From the perspective
of engineering, it seeks to automate tasks that
the human visual system can do”
Computer Vision, ML, AI to the rescue

5. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

6. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Landscape of Search Product Wise
6
Adobe Experience CloudAdobe Creative Cloud Adobe Document Cloud
ADOBE MARKETING
CLOUD
ADOBE ANALYTICS CLOUD
ADOBE ADVERTISING CLOUD
Experience
Manager
Campaign PrimetimeTarget
Audience
Manager
Analytics

7. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Landscape of Search Product Wise : Current State
7
Adobe Experience CloudAdobe Creative Cloud Adobe Document Cloud
ADOBE MARKETING
CLOUD
ADOBE ANALYTICS CLOUD
ADOBE ADVERTISING CLOUD
Experience
Manager
Campaign PrimetimeTarget
Audience
Manager
Analytics

8. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. 8
SEARCH EXPERIENCE
MATTERS MORE THAN
EVER

9. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Architecture Search @Adobe
Birds Eye View

10. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

11. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

12. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

13. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

14. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

15. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.

16. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Adobe Search Elasticsearch Stats
3
Geographical Regions
~10
Billion
Documents
18
Production
Elastic Search Clusters
* As of June 2018
200
Shards
Max # Shards
16
~400
Virtual Machines in AWS and
Azure
2
Public Clouds
~6000
Live Ingestion Rate/
second
~25000
Ingestion Rate/second
Capacity for Reindexing
~600
Queries Per Second

17. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Adobe Search Elasticsearch : Self Managed
* As of June 2018
17
Self Managed Clusters on public clouds1
Moved Lr Search from AWS Elasticsearch hosted to Self-
Managed, 3.4 billion docs
2

18. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Why Elasticsearch?
* As of June 2018
18
Because we have seen Solr Cloud code and cloud is an
after thought
1
Stability and resilience is very high (if done right )
2
Right balance of open-source with tight review process
3

19. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
Adobe Search Play with Elasticsearch
Elasticsearch as a white box ES Plugin for Image/Video similarity
& Search Ranking
* As of June 2018
19
1
Cost Saving: Hot & Cold Indices & Static and Live Indices
2
Index any Generic Entity into Elastic Search with zero code
change
3
Out of order Event : Optimistic locks using ES scripting
4
Custom ACLs in Elastic Search for Enterprise Searches
5

20. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
ML Deep dive

21. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Stock powered by Elasticsearch
License images used in projects, ads,
websites, etc.
• Over 130M images
• Images indexed by tags, price, type,
…
• Novel visual search applications to
surface more results as well as
differentiator
Do these better using deep learned
representations (DLR)

22. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Deep Learned Representation: Embedding
Project to continuous space where similarity of
particular property can be mapped to
(Euclidean) distance:
Dense, small (1k dimensions) vectors
Similarity score corresponds to square
distance
Dimensions usually don’t have meaning
Usually trained using (deep) neural
networks
Used to power deep search engine Image embedding space
Image Similar tags (“semantic”)
maps to
Word (word2vec) Similar word context
Face Similar face attributes

23. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Image Embedding
Trained using a neural network (CNN)
On an auxiliary task (e.g., classification)
Layer 1
Layer 4 Layer 5
CNN
Embedding
Layer
1 DOG
0 SURF
1 SUNSET
0 SUN
1 WOMAN
0 PALM
0 HEAD
0 50s
1 BEACH
0 STARFISH
0 SMILE
0
RETRIEVER
…
…
WOMAN, DOG, BEACH, …
Training:
• Feed image and network tries to
predict tag indicators. Errors
backpropagated
Embedding layer = layer before last
• Embedding is output of embedding
layer
• Captures abstract rep. of tags for
image
• 1k dimensions vector
• Learns filters activated when
patterns arise at various layers

24. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Deep Search Engine
24
Get embeddings for all images and index them
Deep
Search Engine
Embeddings
Query
embedding
Query
Query using an embedding corresp. to query image
Relevant results are those images whose embeddings
are closest to query embedding using Euclidean distance
Close by embeddings corresp. to ”semantically” similar
Finding similar images:
• Calculate similarity for all candidates
• Pick top k

25. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Towards a Reverse Index for Embeddings
Too expensive to calc. distance with 100M+ per query 3
1 4
8
15
18
20
14
16
17
56
7
12
9
10
13
2
19
11
14
9
10
13
query
…Bucket1
Bucket2
Bucket3
…
…
…
…
So, bucketize (cluster) embeddings assigning them
to one of 1k buckets
Find nearest 20 buckets to query from bucket
centroids
Only compare query with embeddings of 20 nearest
buckets (2% of corpus)
Reverse index:
• Bucket to image embedding

26. © 2018 Adobe Systems Incorporated. All Rights Reserved.
PQ-Codes
Embeddings are still too big to keep in index (4kb *
100M+)
• Also, lots of floating point distance calculations
1011
0110
1111
0100
1010
01010010
1110
1001
1101
0111
0011
1000
1100
0001
0000
1011
0110
1111
0100
1010
01010010
1110
1001
1101
0111
0011
1000
1100
0001
0000
1011
0110
1111
0100
1010
0101 0010
1110
1001
1101
0111
0011
1000
1100
0001
0000
1011
0110
1111
0100
1010
01010010
1110
1001
1101
0111
0011
1000
1100
0001
0000
s1 s2 s63 s64
1k floats / 4KB
64 bytes
…
…
Compress embedding using trained encoder:
• Subdivide embedding space in (64) subspaces
During search:
• Because codes quantize sub-vectors, we can pre-
calculate values dependent on quantized -centroids
and bucket centroids
• Distance to candidate is fast because we can leverage
LUTs calculated once per query per bucket
• Cluster each subspace in 256 clusters
• Encode every subspace-vector of
embedding with ID of nearest cluster (as
byte)
• PQ-Code is concatenation of subspace IDs
• Store pqcode, bucketID in index
tabQC – 2.ql + tl + 2.tcl
Query
&
quantized-centroids
Query
&
Bucket
Bucket Bucket
&
Sub-centroids
PrecalculatedPrecalculatedOnce per query
per bucket
Once per query

27. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Elasticsearch Pqcode Plugin
27
Has to work in conjunction with other asset data in ES
So, implement deep search as a plugin and store pqcodes in ES
Plugin implements comparison between query embedding and candidates pqcodes
CAS analyzer outputs query embedding
Reverse index used to limit search to nearest 20 buckets
Calculate scores for all candidates in buckets
CAS
Model Encode
r
Embeddin
g +
Nearest
Buckets
Elastic Search
ES Pqcode
Plugin
Encode
r
Images
index
pqcode
s

28. © 2018 Adobe Systems Incorporated. All Rights Reserved..
Demo – Find Similar Controls

29. © 2018 Adobe Systems Incorporated. All Rights Reserved..
Exploration/Refinement
• Majority 1, 2 word queries
Numberofsearches(M)
0
15
30
45
60
Number of words / query
1 2 3 4 5
• 1 word queries
• Banana, christmas, family, beach, food, flowers,
car, …
• Very general
• Exploration
• 2 word queries
• happy family, wood background, doctor patient,
business man,…
• Still too general
• Refinement by query rewrite
• Both modes can be helped by way of
clustering

30. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Clustering PQ Codes
Use pqcodes for top 5k results
to cluster using k-means:

31. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Clustering PQ Codes
Use pqcodes for top 5k results to
cluster using k-means:
Iteratively assign pqcodes to cluster
of closest centroid and recalculate
centroids until convergence

32. © 2018 Adobe Systems Incorporated. All Rights Reserved.
Clustering PQ Codes
Use pqcodes for top 5k results to
cluster using k-means:
Iteratively assign pqcodes to cluster
of closest centroid and recalculate
centroids until convergence
Centroids like queries and
assignments like find-similar
Decision to assign to a cluster uses
only additions and subtractions
s(y1,xi)−s(y2,xi) = tabQC1 + ql1
− tabQC2 − ql2
Distance to
Centroid Y1
Distance to
Centroid Y2
Once per
cluster centroid
Once per cluster centroid
per bucket

33. © 2018 Adobe Systems Incorporated. All Rights Reserved..
Demo – Clustering Stock Images