Slides from webinar given by Ted Dunning and LucidWorks Chief Scientist, Grant Ingersoll on how search technology can be abused to implement apparently intelligent systems

1. 1©MapR Technologies - Confidential
Crowd Sourcing Reflected
Intelligence Using Search and Big
Data
Ted Dunning
Grant Ingersoll

2. 2©MapR Technologies - Confidential
Grant’s Background
 Co-founder:
– LucidWorks – Chief Scientist
– Apache Mahout
 Long time Lucene/Solr committer
 Author: Taming Text
 Background in IR and NLP
– Built CLIR, QA and a variety of other search-based apps

3. 3©MapR Technologies - Confidential
Ted’s Background
 Academia, Startups
– Aptex, MusicMatch, ID Analytics, Veoh
– Big data since before big
 Open source
– since the dark ages before the internet
– Mahout, Zookeeper, Drill
– bought the beer at first HUG
 MapR
– Chief Application Architect
 Founding member of Apache Drill

4. 4©MapR Technologies - Confidential
Agenda
 Intro
 Search Evolution and Search Revolution
 Reflected Intelligence Use Cases
 Building a Next Generation Search and Discovery Platform
– MapR
– LucidWorks
 1+1=3

5. 5©MapR Technologies - Confidential
Search is Dead, Long Live Search
 Search is a system building block
– text is only a part of the story
 If the algorithms fit,
use them!
 Embrace fuzziness!
 Scoring features are everywhere
Content
User
Interaction
Access
Content
Relationships

6. 6©MapR Technologies - Confidential
Search (R)evolution
 Search use leads to search abuse
– denormalization frees your mind
– scoring is just a sparse matrix multiply
 Lucene/Solr evolution
– non free text usages abound
– many DB-like features
– noSQL before NoSQL was cool
– flexible indexing
– finite State Transducers FTW!
 Scale
 “This ain’t your father’s relevance anymore”

7. 7©MapR Technologies - Confidential
Add (Lots of) Water
 Large-scale analysis is key to reflected intelligence
– correlation analysis
• based on queries, clicks, mouse tracks,
even explicit feedback
• produce clusters, trends, topics, SIP’s
– start with engineered knowledge,
refine with user feedback
 Large-scale discovery features
encourage experimentation
 Always test, always enrich!
Search
DiscoveryAnalytics

8. 8©MapR Technologies - Confidential
Social Media Analysis in Telecom
 Correlate mobile traffic analysis with social media analysis
– events cause traffic micro-bursts
– participants tweet the events ahead of time
 Deploy operations faster to predict outages and better handle
emergency situations
– high cost bandwidth augmentation can be marshaled as the traffic appears
– anticipation beats reaction

9. 9©MapR Technologies - Confidential
Provenance is 80% of value
 Analysis of social media to determine advertising reach and
response
 In one case the same untargeted advertising was worth 5x if sold
with supporting data.

10. 10©MapR Technologies - Confidential
Claims Analysis
 Goal
– Insurance claims processing and analysis
– fraud analysis
 Method
– Combine free text search with metadata analysis to identify high risk
activities across the country
– Integrate with corporate workflows to detect and fix outliers in customer
relations
 Results
– Questions that took 24-48 hours now take seconds to answer

11. 11©MapR Technologies - Confidential
Virginia Tech - Help the World
 Grab data around crisis
 Search immediately
 Large-scale analysis enriches data to find
ways to improve responses and
understanding
 http://www.ctrnet.net

12. 12©MapR Technologies - Confidential
Bright Planet - Catch the Bad Guys
 Online Drug Counterfeit detection
 Identify commonly used language indicating counterfeits
– you know it when you see it
– and you know you have seen it
 Feed to analyst via search-driven application
– enrich based on analysts feedback

13. 13©MapR Technologies - Confidential
Veoh - Cross Recommendations
 Cross recommendation as search
– with search used to build cross recommendation!
 Recommend content to people who exhibit certain behaviors
(clicks, query terms, other)
 (Ab)use of a search engine
– but not as a search engine for content
– more like a search engine for behavior

14. 14©MapR Technologies - Confidential
What Platform Do You Need?
 Fast, efficient, scalable search
– bulk and near real-time indexing
– handle billions of records with sub-second search and faceting
 Large scale, cost effective storage and processing capabilities
 NLP and machine learning tools that scale to enhance discovery
and analysis
 Integrated log analysis workflows that close the loop between the
raw data and user interactions

15. 15©MapR Technologies - Confidential
Shards
1
2
3 N
Search View
•Documents
•Users
•Logs
Document
Store
Analytic
Services
•View into
numeric/histo
ric data
•Classification
•Recommendation
Personalization &
Machine Learning
Services
Classification Models
In memory
Replicated
Multi-tenant
Discovery &
Enrichment
Clustering, classificat
ion, NLP, topic
identification, searc
h log analysis, user
behavior
Content Acquisition
ETL, batch or near
real-time
Access APIs
Data
• LucidWorks Search
connectors
• Push
Reference Architecture

16. 16©MapR Technologies - Confidential
MapR
 MapR provides the technology leading Hadoop distribution
– full eco-system distribution
– integrated data platform
– complete solution for data integrity
 MapR clusters also provide tight integration with search
technologies like LucidWorks
– integration is key for effective ops

17. 17©MapR Technologies - Confidential
LucidWorks
 LucidWorks provides the leading packaging of Apache Lucene and
Solr
– build your own, we support
– founded by the most prominent Lucene/Solr experts
 LucidWorks Search
– “Solr++”
• UI, REST API, MapR connectors, relevance tools, much more
 LucidWorks Big Data
– Big Data as a Service
– Integrated LucidWorks Search, Hadoop, machine learning with prebuilt
workflows for many of these tasks

18. 18©MapR Technologies - Confidential
LucidWorks Big Data Architecture
Big Data Operating System
• Administration
• Provisioning
• Monitoring
• Configuration
• Service Management
• Data Management
• Security
System
Management
Uniform ReST API
Search – Discovery – Analytics
• LucidWorks Search
• Machine Learning
(classification, clustering, recommendations)
• Natural Language Processing
• SQL (Hive) Interface
• Data Workflows (ETL, log analysis, common metrics)
• Extensible
• Enterprise
Repository
• Social Media
• Databases
• HDFS
• Cloud (S3)
• Push
Content
Acquisition
Hadoop/HBase Search
Indexes
Search
Logs

19. 19©MapR Technologies - Confidential
Easy Wins
 Analyze logs from application stored in MapR
 Seamlessly store search indexes in MapR
– and feed to Pig, Mahout and others
– use mirrors + NFS to directly deploy indexes
 Snapshots make backups a snap
 LucidWorks 2.5 (2013 Q1) easily connects with MapR

20. 20©MapR Technologies - Confidential
1 + 1 = 3

21. 21©MapR Technologies - Confidential
Learn More
 More information
http://www.mapr.com/company/events/lucidworks-12-13-2012
 Vote for this topic for Hadoop Summit EU:
http://bit.ly/128tLQe
 Talk to Ted
@ted_dunning
tdunning@maprtech.com
 Talk to Grant
@gsingers
 MapR and Lucid Works
http://www.mapr.com
http://www.lucidworks.com