This keynote presentation describes the critical role that search and Lucene has in building next generation products that understand reputation and relevance. We also describe how data science and machine learning have been applied at LinkedIn to collect, interpret, and index data around topical reputation. Lucene Revolution is the biggest open source conference dedicated to Apache Lucene/Solr.

1. Skills, Reputation, and Search
Pete Skomoroch
Principal Data Scientist, LinkedIn

2. Vision: Create Economic Opportunity for Every Professional
2
Location

3. LinkedIn: The Professional Profile of Record
©2012 LinkedIn Corporation. All Rights Reserved. 3
200+MMembers 200M Member
Profiles

4. LinkedIn Search: Connecting Talent with Opportunity
4

5. Skills Correlated with the Job Title “Data Scientist”
5

6. Skills Related to “Big Data”
6

7. Information Retrieval
7

8. Soul Retrieval
8

9. 9

10. Lucene on LinkedIn
10

11. Lucene Endorsement Graph
11

12. Solr on LinkedIn
12

13. Solr Endorsement Graph
13

14. Reputation: Building the Endorsement Graph
14

15. 15
Viral Growth: 1 Billion Endorsements in 5 Months

16. How Did We Gather this Data?
16
1. Desire + Social Proof
2. Viral Loops + Network Effects
3. Data Foundation + Recommendation Algorithms

17. 17
1) Desire & Social Proof

18. A
endorses
B
B
notified
B “accepts”
endorsement
B
endorses
C
B
endorses
D
Endorsement
recommendations
Email NotificationNews Feed
2) Viral Loops & Network Effects

19. 3) Data Foundation: Skills & Suggested Skills
19

20. Data Foundation: LinkedIn Skills
20

21. Social Tagging Accelerates Adoption
Suggested
endorsements
Skill recommendations
Skill marketing
©2012 LinkedIn Cororation. All Rights Reserved.
Virality only

22. Outline
22
Skill discovery
Skill tagging
Skill recommendations
Suggested endorsements

23. Skill Discovery: Unsupervised Topics from Profiles
23
Extract

24. Topic Clustering & Phrase Sense Disambiguation
24

25. Deduplication Signals from Mechanical Turk
25

26. Sample Task for Mechanical Turk Workers
26

27. Skill Phrase Deduplication
27

28. Outline
28
Skill discovery
Skill tagging
Skill recommendations
Suggested endorsements

29. Lead designer and engineer for the implementation of a user-
centric, fully-configurable UI for data aggregation and reporting.
Developed over 20 SaaS custom applications using Python,
Javascript and RoR.
Tagging Skill Phrases
 Tagging: Extract potential skill phrases from text
 Standardize unambiguous phrase variants
29
JavaScript RoR SaaS Python
ror
rubyonrails
ruby on rails development
ruby rails
ruby on rail
Ruby on Rails
Document
(ex: Profile)
Tokenization
Skills Tagger
Phrases
(up to 6 words)
Skills Classifier
Skills
(unordered)
Skills
(ranked by relevance)

30. Outline
30
Skill discovery
Skill tagging
Skill recommendations
Suggested endorsements

31. Skill Inference
 How suggested/inferred skills work:
– The skill likelihood is a conditional model
– Probabilities are combined using a Naïve Bayes
Classifier
 If you are an engineer at Apple, you probably know
about iPhone Development.
31
Profile
Extract
attributes
- Company ID
- Title ID
- Groups ID
- Industry ID
- …
Skills Classifier
Skills
(ranked by likelihood)
Feature
Vectors

37. Skill Recommendations for Your LinkedIn Profile
37
49% Conversion
4% Conversion

38. Outline
38
Skill discovery
Skill tagging
Skill recommendations
Suggested endorsements

39. Social Tagging via Skill Endorsements
39

40. Social Tagging Accelerates Adoption
Skill endorsements
Skill recommendations
Skill marketing
©2012 LinkedIn Cororation. All Rights Reserved.

41. Data Amplifies Desire
41
1. Desire + Social Proof
2. Viral Loops + Network Effects
3. Data Catalyst + Recommendation Algorithms

42. Over 58 Million Profiles are now Tagged with Skills
42

43. All This Data Flows Back Into Our Lucene Index
43

44. Helping us Connect Talent & Opportunity
44
Location

45. Questions?
We’re hiring: data.linkedin.com
@peteskomoroch
©2012 LinkedIn Corporation. All Rights Reserved. 45

46. CONTACT
Pete Skomoroch
@peteskomoroch
http://data.linkedin.com