This document summarizes Mark Levy's presentation on using Hadoop for algorithms on user data at Last.fm. It describes how Last.fm uses Hadoop to compute music charts from over 1 billion user scrobbles per month and calculate royalties. It also discusses using Hadoop for topic modeling of user profiles and documents through LDA, graph recommendations through label propagation on the user-item graph, and processing audio files to extract metadata through a map-reduce approach.

1. Algorithms on Hadoop at Last.fm Mark Levy, 14 April 2011

3. Hadoop dfs keeps them safe

5. Hadoop dfs keeps them safe

7. cluster adds them upand so on...

9. Graph Recommendation

10. Audio Analysis

11. LSH indexingand so on...

12. Topic Modelling learning topics from documents

14. use trained model for:

15. inference

16. smoothing

17. many applications

19. labelling

25. use Gibbs Sampling (MCMC):

26. initialise all parameters to random values

27. loop till convergence:

28. consider one parameter at a time

29. compute a sampling distribution based on current values of all other parameters

32. learn distributions p(z|w)= (C(w,z)+β)/(C(z)+V β) ∝ C(z,d)+α

34. initialise randomly

35. iterate:

36. sample a new topic for each word

38. copy word-topic matrix to each machine

39. sample based on local copy

41. Topic Modelling: AD-LDA class GibbsSamplingMapper: init(): load current word-topic matrix map(docID,doc): for w,z in doc: compute p(z|w) from matrix,doc sample new_z from p(z|w) doc[w] = new_z yield docID,doc for w,z in doc: yield (w,z),1

42. Topic Modelling: AD-LDA class Reducer: reduce(key,val): if val is a docID: # save new topic assignments yield key,val else: # update word-topic matrix matrix[key] += val

45. initial iterations slower, later fasteronly sample “likely” topics

47. 200 topics, 76M documents, 670M words

48. 80 map tasks

49. initially 25 minutes per training iteration

50. falls to 5 minutes by iteration #50

55. many short routes from U to t ⇒ recommend!t U

58. user nodes are labelled with known items

59. each label has an associated weight

60. iterate:

61. propagate labels to adjacent nodes

62. accumulate and renormalise at each node

63. prune number of labels held at each node

65. labels at item nodes are similar items

70. no global state

71. state at node recomputed from scratch on each iteration from incoming messages

72. examples:

73. breadth-first search

74. page rank

76. initialize state at each node, write to diska 2 U,[(a,2),(b,4),(c,4)] 4 U b 4 c

78. output: updated state at each node

79. map(nodeID,value):

80. join adjacency list and state

81. emit a message to each node in ajacency list

82. reduce(nodeID,messages):

83. process messages at each node

85. Label Propagation class Reducer: reduce(nodeID,msgs): # accumulate labels = defaultdict(lambda:0) for msg in msgs: for label,w in msg: labels[label] += w # normalise, prune normalise(labels,MAX_LABELS_PER_NODE) yield nodeID,labels

87. mimic abandoning random walk:

88. propagate a dummy label

89. base its weight on degree of source node

91. well... just a small part of it:

92. 1M artists, 1.8M users, 350M edges

93. set MAX_LABELS_PER_NODE = 150

95. after some iterations mappers propagate MAX_LABELS_PER_NODE updates along every edge

96. lots of disk for mapper output

99. key estimation

100. chord sequence estimation

101. energy

102. music/speech?

104. pull audio from its own dfs

105. run C++ analysis code on it 

106. write verbose output somewhere

107. don’t take too long

109. zip up binary and libs

110. extract them on each machine

113. Label Propagation: Mark Levy

114. dumbaudio: Marcus Holland-Moritz

115. with help from James Grant, Klaas BosteelsThanks for listening! mark@last.fm @gamboviol