When recommendation is described in mathematical terms as a matrix equation, a striking symmetry in the form of the equation becomes apparent. Exploiting this symmetry allows us to build search engines that don't need meta-data and self-organizing web-sites.

1. Recommendation as Search
Reflections on Symmetry
©MapR Technologies - Confidential 1

2. Company Background
 MapR provides the industry’s best Hadoop Distribution
– Combines the best of the Hadoop community
contributions with significant internally
financed infrastructure development
 Background of Team
– Deep management bench with extensive analytic,
storage, virtualization, and open source experience
– Google, EMC, Cisco, VMWare, Network Appliance, IBM,
Microsoft, Apache Foundation, Aster Data, Brio, ParAccel
 Proven
– MapR used across industries (Financial Services, Media,
Telcom, Health Care, Internet Services, Government)
– Strategic OEM relationship with EMC and Cisco
– Over 1,000 installs
©MapR Technologies - Confidential 2

3. What is Hadoop?
 A new style of computation
 A new style of combining computation and storage
 Allows very large computations
 Used by all large internet companies, many other industries
 Fundamentally changes the economics of large-scale computation
©MapR Technologies - Confidential 3

4. Why Big Data?
 Because we can
 Because we can learn new things
 Because new economics of computation favors large scale
 Because big data can be simpler than small data
©MapR Technologies - Confidential 4

5. Recommendations
 Often known as collaborative filtering
“People who bought x also bought y”
 Actors (people) interact (bought) with items (x and y)
– observe successful interaction
 We want to suggest additional successful interactions
 Observations are inherently very sparse
©MapR Technologies - Confidential 5

6. Examples
 Customers buying books (Linden et al)
 Web visitors rating music (Shardanand and Maes) or movies
(Riedl, et al), (Netflix)
 Internet radio listeners not skipping songs (Musicmatch)
 Internet video watchers watching >30 s (Veoh)
 iTunes song purchases or plays (Apple)
©MapR Technologies - Confidential 6

7. Fundamental Algorithm
 History matrix A has the shape of actors x items
 Cooccurrence matrix K has the shape of items x items
an actor interacted with both x and y
sum over all actors
 A is also a linear operator
 K tells us “users who interacted with x also interacted with y”
©MapR Technologies - Confidential 7

8. … Warning …
©MapR Technologies - Confidential 8

9. … Warning …
Mathematics ahead
©MapR Technologies - Confidential 9

10. Fundamental Algorithmic Structure
 Cooccurrence
K=A A T
r = AT (Ah) = (AT A)h
 For very large data-sets
r = sparsify(A A)h
T
©MapR Technologies - Confidential 10

11. But Wait ...
Does it have to be that way?
©MapR Technologies - Confidential 11

12. But why not ...
T
(A A)h
©MapR Technologies - Confidential 12

13. But why not ...
T
(A A)h
Why just dyadic learning?
©MapR Technologies - Confidential 13

14. But why not ...
T
(B A)h
Why just dyadic learning?
Why not triadic learning?
©MapR Technologies - Confidential 14

15. But why not ...
T
(B A)h
Why just dyadic learning?
Why not p-adic learning?
©MapR Technologies - Confidential 15

16. For example
 Users enter queries (A)
– (actor = user, item=query)
 Users view videos (B)
– (actor = user, item=video)
 A’A gives query recommendation
– “did you mean to ask for”
 B’B gives video recommendation
– “you might like these videos”
©MapR Technologies - Confidential 16

17. The punch-line
 B’A recommends videos in response to a query
– (isn’t that a search engine?)
– (not quite, it doesn’t look at content or meta-data)
©MapR Technologies - Confidential 17

18. Real-life example
 Query: “Paco de Lucia”
 Conventional meta-data search results:
– “hombres del paco” times 400
– not much else
 Recommendation based search:
– Flamenco guitar and dancers
– Spanish and classical guitar
– Van Halen doing a classical/flamenco riff
©MapR Technologies - Confidential 18

19. Real-life example
©MapR Technologies - Confidential 19

20. Real-life example
©MapR Technologies - Confidential 20

21. Hypothetical Example
 Want a navigational ontology?
 Just put labels on a web page with traffic
– This gives A = users x label clicks
 Remember viewing history
– This gives B = users x items
 Cross recommend
– B’A = label to item mapping
 After several users click, results are whatever users think they
should be
©MapR Technologies - Confidential 21

22. Resources
 Me
tdunning@maprtech.com
@ted_dunning
 Slides and such:
– http://info.mapr.com/ted-paris-05-2012
 The original paper
– Accurate Methods for the Statistics of Surprise and
Coincidence
– (check on citeseer)
 Source code
– Mahout project
– contact me
©MapR Technologies - Confidential 22