Presentation at Gothenburg University, the 8th of March 2012 by Svetoslav Marinov, Tobias Berg and Björn Klockljung-Johansson.

1. Implementing and designing search solutions
Gothenburg University – Gothenburg – 2012-03-08
© FINDWISE 2012

2. Agenda
• Introduction to Findwise
• Technical approach
• DIY UX design
• Research

3. About Findwise
• Founded in 2005
• Offices in Sweden, Denmark,
Norway and Poland
• 72 employees (February 2012)
• Our objective is to be a leading provider of Findability solutions utilising
the full potential of search technology to create customer business value

4. Technology independent
Creating search-driven Findability solutions based on market-leading
commercial and open source search technology platforms:
 Autonomy IDOL
 Microsoft (SharePoint and FAST Search products)
 Google GSA
 IBM ICA/OmniFind
 LucidWorks
 Apache Lucene/Solr (Open source)
 and more…

5. Findability Challenges
Employee productivity (DN article, March 2011):
”The effort to find the right information costs an average company 80,000
SEK per employee and year”
Customer Service quality and efficiency (Accenture report, March 2011):
“69% of agents don't have answers to help service customers”
E-commerce conversion rate (Google survey, December 2010):
“77% of those surveyed used search within an e-commerce website to find
products”

6. Information overload?

7. A search engine alone is not enough

8. Technical approach

9. RE-USE

10. STANDARD

11. Standard architecture

12. Search core

13. Search core - overview
Documents Inverted index
Title: Brown fox Term Documents
Content: The quick
brown fox jumps over
the lazy dog
Tokenization … …
Stemming
Author: Tobias Berg
Stop-word fox 1
…
jump 1,2
Title: My dog lazy 1
Content: My old dog
cannot jump anymore dog 1,2
Author: Svetoslav
Marinov tobias 1
berg 1
… …

14. Relevancy
Retrieved
documents Relevant
documents
• Precision – how many of the retrieved documents are relevant?
• Recall – how many of the relevant documents were retrieved?

15. Relevancy
Recall
find everything related to the query
Goal
- lemmatization Improve precision,
- synonyms without sacrificing recall
- wildcards
- anti-phrasing
- or-operator
Precision
find only entities related to the query
- exact word matching
- exact phrase matching
- and-operator

16. Search core – relevance score
• TF/IDF
• Field length
• Field weight
• Title *2
• Author *4
• Content *1
• Freshness
• …

17. Search Core
• Optimized for full-text search
{query} {result} • Sub-second responses
• Tunable relevance
• Scalable
Find Score
matching documents • Configurable & Extendable
documents

18. Standard architecture

19. Connectors

20. Connectors – fetch data
Id Product Description Price
name
1 Wheel Makes the
bus go
45 Database
round round
round
connector
2 Window A shield of 12
glass
Id Book name Abstract Author
1 Ulysses Irish novel James
Joyce
Database
2 Crime and Russion Dostoevsky, connector
Punishment novel Fyodor

21. Connector framework – code example
public void execute() {
//Insert code to fetch content
}
public void interrupt() {
//Insert code to handle interrupt signal
}
public void init() {
//Insert code to initialize connnector
}

22. Connector Frameworks
• Existing connectors
• Re-usable
• Configuration interfaces
http://incubator.apache.org/connectors/ • Standardized implementation
http://code.google.com/p/google-enterprise-
connector-manager/

23. Standard architecture

24. Pipeline

25. Pipeline - overview
• PDF/Office -> Text
• Lemmatization
• Language identification
• NER
• Phonetic search
• Keyword extraction
• External calls
• …

26. Pipeline framework – code example
protected void addAction(Document doc) throws PipelineException {
//Insert code
doc.addField(“Title”,”Hello world!”);
}
protected void updateAction(Document doc) throws PipelineException {
//Insert code
addAction(item);
}
protected void deleteAction(Document doc) throws PipelineException {
//Insert code
}

27. NLP tools and approaches
• Open source:
GATE, OpenNLP, UIMA, StanfordNLP, Mallet,
Apache Mahout
• Proprietary:
IBM LanguageWare
• Own components:
e.g. KeywordExtraction Service; LanguageIdentify
• POS taggers – Hunpos, OpenNLP, Mallet
• Dependency Parsers – MaltParser, StanfordParser
• NER – rule-based + statistical models
• Document summarization
• Document clustering

28. Pipeline – configuration example

29. Pipeline frameworks
• Re-usable stages
http://www.openpipeline.com/ • Configuration interface
• Focus on task
Findwise
Hydra
http://www.pypes.org/

30. Putting it all together

31. What the frell is UX design?

32. What the frell is UX design?
• Interaction design
• Usability Engineering
• Information Architecture
• Visual Design

33. Findwise UX design principles
Users want results
Dialogue not monologue
Participation builds trust
Answer frequent questions
Simple but powerful

34. Users want results

35. Dialogue not monologue

36. Participation builds trust

37. Answer frequent questions

38. Simple but powerful

39. Findwise UX design principles
Users want results
Dialogue not monologue
Participation builds trust
Answer frequent questions
Simple but powerful

40. DIY UX design

41. DIY UX design
Design research
Analytics
Usability tests
Iterate!

42. Design research
• Be easy to reach – keep contact
• Let users requests guide you when prioritizing new features
• Listen & try to discover the underlying problem
• Try to find out what the user needs not what they say they want

43. Analytics
• Web analytics
• Search analytics
• A/B testing

44. Usability tests
• Test early - test often
• Use sketches, paper prototypes, static prototypes and
working prototypes!
• Create real tasks or problems
• Don’t ask them how they would want it
• Test on friends and family or colleagues

45. Iterate!

46. Why UX design?
• Improved requirements
• Better feedback
• Eliminate bias
• Less development time

47. Summary
• Listen & try to discover the underlying problem
• Search analytics – Top queries
• Do usability tests early & often
• Iterate!

48. Research
• Collaboration with Universities
GU, Borås, KTH, Copenhangen U.
• EU projects
RUSHES
• Master’s Thesis supervision
Chalmers, KTH, Lund

49. Master’s Thesis projects
• A way to test ideas
• A way to recruit people
• A way to cooperate with Universities
• Keyword Extraction
• Document Clustering
• NER
• Document summarization
• Extracting structural information from text
• Query log analysis

50. Resources - books
• The design of everyday things
• Don’t make me think
• Search analytics for your site
• ManifoldCF in Action
• Taming Text

51. Shameless plug
twitter.com/findwise
slideshare.net/findwise
findabilityblog.se
findwise.com

52. Tobias Berg
tobias.berg@findwise.com
Thanks! Björn Klockljung Johansson
bjorn.klockljung.johansson@findwise.com
Svetoslav Marinov
svetoslav.marinov@findwise.com