Rising of social networks like Twitter and Facebook has been flooding the web with user-generated contents in the amount that was never seen before. As we become more depending on the social media for daily information gathering and social interaction, we have started to demand more from search engines, such as real time conversations and social connections. Such new environment is forcing search engine technology to evolve. This presentation will first review the traditional search engine technologies and architecture. Then, we will talk about some of the new architectures for supporting real time search and social search. The last part of the presentation is devoted to discussion of the emerging real time web monitoring system, the cousin of search engine. Social media monitoring is predicted to become a mainstream tool for consumers and businesses because the inherent collective intelligece has the potential to further increase the efficiency of finding information and making decision as what search engine has done.

1. SDForum Event
Building Search Engine in the
Social Media Era
AJ Chen
ajchen@web2express.org
twitter: @web2express
http://web2express.org
3/24/2010

2. Agenda
1. The changing search environment
2. Traditional search engine architectures
3. New architectures for real time search
and social search
4. Semantic search
5. Emerging real time web monitoring
6. Q/A

3. The Changing Search Environment

4. The Changing Search Environment

5. The Changing Search Environment

6. Agenda
1. The changing environment
2. Traditional search engine
technologies and architecture
3. New architectures for supporting real
time search and social search
4. Semantic search
5. Emerging real time web monitoring
system
6. Q/A

7. Open Source Lucene/Solr
from Grant Ingersoll

8. Lucene/Solr Search for Web App

9. Lucence Indexing/Searching

10. Solr Server Architecture
from Yonik Seeley

11. Scaling Lucene and Solr
from Mark Miller

12. Agenda
1. The changing environment
2. Traditional search engine technologies
and architecture
3. New architectures for supporting
real time search and social search
4. Semantic search
5. Emerging real time web monitoring
system
6. Q/A

13. Near Real Time Search
Lucene 2.9:
IndexReader (old + new records) <-- IndexWriter
Zoie (LinkedIn):
Swap Indexes in memory

14. Google Real-time Search

15. Google Social Search

16. Google Aardvark Social Q/A Engine

17. Agenda
1. The changing environment
2. Traditional search engine technologies
and architecture
3. New architectures for supporting real
time search and social search
4. Semantic search
5. Emerging real time web monitoring
system
6. Q/A

18. Bing/Powerset NLP Search Engine
from Barney Pell

19. Sindice: Retrieving semantic data
index API:
sindice.com
search engine:
sig.ma

20. Google Semantic Search

21. Hybrid Search Engine

22. Hybrid (Universal) Search

23. Agenda
1. The changing environment
2. Traditional search engine technologies
and architecture
3. New architectures for supporting real
time search and social search
4. Semantic search
5. Emerging real time web
monitoring system
6. Q/A

24. Social Media Monitoring
Benefits for businesses:
understand what people are talking about the company,
brands, products, and competitions, etc.
identify leads for marketing and sales;
engage with customer and community conversations;
support customers;
cultivate product advocates;
use customer feedback to improve products and brands;
Differences from search engine:
search engine for filtering
real-time text analysis and semantic analysis
auto-discovery of trends/patterns
collective intelligence
rule engine for routing right information to right person

25. Basic Monitoring

26. Scaling up Social CRM
Problem:
Today's social media program does not scale
Solution:
Build a pipeline of social context flow and then apply
advanced text analytics to automate routing of social
contexts.

27. Intelligent Monitoring Pipeline

28. Social CRM Integration in the Clouds

29. Turning CRM into Social CRM

30. Agenda
1. The changing environment
2. Traditional search engine technologies
and architecture
3. New architectures for supporting real
time search and social search
4. Semantic search
5. Emerging real time web monitoring
system
6. Q/A