Infrared simulation and processing on Nvidia platforms
Our Tale from the Trail of Shadows at REI Co-op - Chris Phillips & Dale Smith, REI Co-op
1. Our Tale From the Trail of
Shadows
Chris Phillips and Dale Smith
REI Co-op
2. Who Are We?
Dale Smith
● REI Co-Op for 5 years
○ Lead Engineer: Search
Platform
● 40 years of software
engineering
● Boeing manufacturing
automation
● Seattle tech, including
○ Go2Net: Dogpile &
Metacrawler
@dasmith836
3. Who Are We?
Chris Phillips
● REI Co-op for 7 years
5 Years on Search
● 20 years in the industry
○ Half Life
○ Exchange Indexing
○ Launch of Xbox Kinect
@realchrispdad
5. Agenda
• What We Did
• Where We Were
• The Big Challenges
• Solutions – The Cool Stuff
• Questions
6. What We did
We replaced our SaaS search engine with an on-premise Solr stack
7. Where We Were
Stuck in the mud!!
● Not meeting customer or
internal expectations
● Slow searches: 750-900 ms
● Random outages
● Not flexible for our business
partners
● Limited visibility into the
servers
10. The Goal
Internal Expectations
● Fast and stable search platform
● Rapid development cycle
● Save budget on licensing and support
11. The Big Challenges
No changes to the existing API’s
● Website
● Mobile apps
● Point of Sale apps
● Internal tools
Limit SEO impact
12. The Big Challenges
We were on our previous search engine
for 4 years.
● Had to tune relevance and
merchandising rules to match in a
matter of months
● Internal users had to learn new tools
● Biggest Challenge: How do we verify
we haven’t broken something in
search?
13. 01
The Big Challenges
The Solr Stack Is Awesome, But Scary
● Lucidworks Fusion
● Innovent Solutions Findtuner
● Multi-term synonyms
● “47 knobs that all go to Volume 11!”
14. The Trail of Success
Build a Lean Dream Team
● 2 Senior Back-End Developers
● 1 Senior QA Analyst
● 1 SDET
● PM created a “bubble of safety”
● Product Owner is the search
expert at REI
15. The Trail of Success
Nurture the Team Dynamic
● Trusted by leadership
● We trusted each other
● Accountability
● Committed to quality
● We knew what success looked like
16. The Trail of Success
Embrace The Team Philosophy
17. The Trail of Success
Quality: The measure of the excellence of
your product.
● “Quality is not an act. It is a habit.” --
Aristotle
18. The Trail of Success
Be Serious About Who Owns
Quality
● Everyone owns quality
● Build good habits
19. The Trail of Success
Good Habits
● Co-locate
● Keep the business owner
involved
● Engineers are cool!
● Find bugs before code is written
○ Architect together
○ Mental lifting
○ Risks
○ Unit tests
○ Test scenarios
20. The Trail of Success
Good Habits
● Avoid legacy code with 3
questions
○ Testable
○ Supportable
○ Maintainable
21. Solutions - Use Great Tools
● Innovent Solutions FindTuner
● Lucidworks Fusion
● Solr Cloud
● These tools were designed to
work together
22. Solutions - Innovent Solutions FindTuner
● Manages search rules
○ Product promotion
○ Banners, redirects, etc.
○ Query adjustments
● UI For Business Users
● Eliminated the need to
implement this in-
house to replicate
SaaS solution
23. Solutions - Lucidworks Fusion
● Manages Solr Cloud
configuration
● Provides configurable
search pipelines
● Nice UI for business
users
● Many more features!
24. Solutions - A Great Software Ecosystem
● Continuous Integration Model
● Docker Platform
● Microservices Architecture
● High-functioning Monitoring &
Metrics
25. Solutions – Dynamic Facets
The Problem
● ~200,000 product variants
● ~250 “facet-worthy” attributes
● Asking Solr to calculate facets
for all of these attributes on
each request would lead to
significant performance issues
26. Because Two Queries Are Better Than
One!
● Indexed products with list of
important attributes
● First Query: read and calculate
10-15 key attributes for facet request
01
Solutions – Dynamic Facets
● Second Query: full request with
selected facets
● Solr query only ever includes the
10-15 most important facets
27. The Problem:
● How do we verify product results from our new search stack?
● How do we verify performance capabilities under production load?
Solutions – The Shadow Query
The Solution:
● “Shadow Query”: all production searches are sent to both
search engines, the results compared using special comparison
functions, and metrics recorded on everything.
● All while not affecting existing search performance!
● Basically, we let our customers help test the system
31. Solutions – Search API Normalization
REI search API is search-engine
agnostic
● Avoid exposure of search
engine-specific parameters or
attributes
● Preserve existing API contract
● Preserve existing SEO patterns
● Enable queries to multiple
search engines
32. Solutions – Search API Normalization
An Example
Normalized Refinement on REI.com:
r=features:Insulated;size:Large;color:Purple
fq=features:"Insulated"&fq={!tag=SIZE}size:"Large"&fq={!
tag=COLORFAMILY}color_family:"Purple"
Translated to Solr:
q1=insulated&x1=tab.ftrs&q2=large&x2=tab.size&q3=purple&x3=
colorfam
Translated to SaaS:
33. Solutions – Result ComparisonsComparing the Shadow Query Results
● Record query result times and deltas
● Record total number of results and deltas
● Record count of products in result A and not in result B, and vice-
versa
● Record Damerau-Levenshtein Distance, using product IDs instead
of letters
● Use graph dashboards to “visualize quality” as the search metrics
converged
34. Calculated based on operations transforming a list of products
from one sequence to another
● Insertion
● Deletion
● Substitution
● Transposition
● Each operation has a cost, weighted higher at the start of the
sequence
Solutions – Modified Damerau-Levenshtein
Distance
35. Preparing for ‘Solr Day’
● Load Test using recorded production queries replayed through the
shadow query at high load
● Monitored CPU and Memory - It’s all about that swap space
● Diagnose crashes from good bots doing bad things
36. It’s On!!!
We turned it on for our customers and…..
● Nobody noticed anything had changed
● They just enjoyed it being 600ms faster!
37. After The Party...
We continued to use the Shadow
Query to verify the new Solr stack
● Kept making calls to the SAAS
search engine in the
background
● Provided a safety net for an
important sale period
● Continued to use metrics to
validate rule refinements
38. Wisdom From The Trail
Pack It Out!
● Start with a solid team
● Trust matters in all directions
● Use the best tools
● Use data to validate everything
● Stop relying on hope and
heroics and instead bake QA
into the process and the
system