Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Andere mochten auch
Andere mochten auch (9)
Ähnlich wie Meetup#4, Smart.Data@OK.ru
Ähnlich wie Meetup#4, Smart.Data@OK.ru (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Meetup#4, Smart.Data@OK.ru
- 2. What!is!it!about?! • About/us/and/the/size/ • Smart!data!vs.!big!data! • Smart!data!tasks! • Smart!data!deck! • Open!data! 2!
- 3. OK.ru!is!about! • Family,!friends,!classmates! • Sea!of!posiEve!and!humor! • Enormous!collecEon!of!mulEmedia!content! • Largest!plaHorm!for!online!games! 3!
- 4. OK.ru!is!! ! ! The!largest!entertainment!social! network!between!AtlanEc!ocean!and! Amur!river! 4!
- 5. OK.ru!in!numbers! • 200!000!000!registered!users! • 10!000!000!communiEes! • 10!000!000!000!connecEons! • Daily:! – 40!000!000!unique!users! – 250!000!000!messages! – 25!000!000!posts! – 44!000!000!photos! – 7!000!000!friendships! – 9!000!000!000!entries!added!to!news!feeds! – …! 5!
- 6. OK.ru!in!technical!numbers! • 7000+!servers! • 10Tb!of!posts! • 20Tb!of!likes! • 80Tb!messages!in!discussions! • 400Gb!social!connecEons! • …! • And!6Tb!of!new!data!in!our!Hadoop!each!day! 6!
- 7. What!is!it!about?! • About!us!and!the!size! • Smart/data/vs./big/data/ • Smart!data!tasks! • Smart!data!deck! • Open!data! 7!
- 9. Big!Data!vs.!Smart!Data! • 100!gigabytes!is!enough!to! – Auto[generate!music!catalog! – Triple!acEvity!in!the!music!layer! – Double!acEvity!in!the!video!«related»!area! • 400!gigabytes:! – Plus!50%!for!clicks!at!the!communiEes!page! – Extra!1М!communiEes!visits!from!the!“recommended! communiEes”!portlet! • 10Тб!–!plus!20%!for!clicks!at!“Like”!at!news!feed! 9!
- 11. Be!smart!! 1. Set!the!Goal! 2. Find!the!Model! 3. Select!the!Toolset! 4. Collect!the!Data!needed! 5. Mine!the!Data,!train!the!Model,!apply!results! 6. Profit!! 11!
- 12. Be!smart!! 1. Set!the!Goal! 2. Find!the!Model! 3. Select!the!Toolset! 4. Collect!the!Data!needed! 5. Mine!the!Data,!train!the!Model,!apply!results! 6. Profit!! 7. Repeat/from/step/1/ 12!
- 13. What!is!it!about?! • About!us!and!the!size! • Smart!data!vs.!big!data! • Smart/data/tasks/ • Smart!data!deck! • Open!data! 13!
- 15. Smarter!data!for!music! • Stage!1:!mining!collaboraEve!correlaEons! – Similar!arEsts!and!“ArEst!radio”! • Stage!2:!mining!temporal!correlaEons!and! combining!with!metadata!and!collaboraEve! part! – PersonalizaEon!of!the!main!page! – The!most!affecEng!feature!(+100%!to!playbacks)! • Stage!3:!segmenEng!users’!tastes! – “My!Radio”!feature! 15!
- 18. Smarter!data!for!communiEes! • Stage!2:!extend!recommender!with!metadata!(tags)! and!regional/demography!data! – Improved!CTR!for!recommend!communiEes! – PersonalizaEon!of!the!communiEes!page! – +50%!to!clicks!at!the!communiEes!page! • Stage!3:!mine!communiEes!content!for!their! semanEcs! – Implemented!fancy!distributed!Robust!LDA!model!for! communiEes!post! – Not!in!producEon!yet!–!waiEng!for!one!of!you,!guys,!to! complete!;)! 18!
- 19. Smart!data!for!video! • The!data! – Likes!for!videos! – “View!events”! • Stage!1:!mine!collaboraEve!correlaEons!! – Double!clicks!at!the!“related!video”!area! – Likes!perform!be0er!then!“view!events”! • Stage!2:!advances!collaboraEve!models!for!top! personalizaEon! – Different!variaEons!of!SVD! – Limited!effect.!To!be!conEnued!(may!be!with!one!of! you!;)!)! 19!
- 20. Smart!data!for!news!feed! • The!data:! – News!feed!impressions!(the!largest!our!data!set)! – Likes,!comments!and!clicks! • Stage!1:!improve!ranking!using!CTR! – We!managed!to!construct!the!infrastructure!capable!to!calculate!CTR! for!all!our!content!at!real!Eme!(at!the!speed!of!up!to!4!000!000!events! per!second)! – +10%!for!CTR!when!considering!CTR!!! • Stage!2:!improved!models!for!news!feed!ranking! – SVD[based!collaboraEve!approach!is!running!in!experimental!seungs! and!looks!promising! – More!models!are!waiEng:!content!based!(LDA!or!whatever),!social! based,!ensembles! – Join!the!movement!! 20!
- 21. Smart!data!for!anEspam! • Stolen!accounts!detecEon! • Pornography!detecEon! • Textual!spam!classificaEon! • AutomaEc!registraEon!detecEons! • …! 21!
- 23. More!areas!willing!to!become!smart!! • Help!users!to!find!friends!(people!you!may! know!and!more)! • Presents! • Games! • Photos! • OperaEonal!data! • …! 23!
- 24. What!is!it!about?! • About!us!and!the!size! • Smart!data!vs.!big!data! • Smart!data!tasks! • Smart/data/deck/ • Open!data! 24!
- 25. Our!technologies! • Hadoop!2.x! • Apache!Pig! • Apache!Spark! • Apache!Kaxa! • Apache!Samza! • Apache!Cassandra! • Python,!R,!Tableau! 25!
- 27. What!is!it!about?! • About!us!and!the!size! • Smart!data!vs.!big!data! • Smart!data!tasks! • Smart!data!deck! • Open/data/ 27!
- 28. Likes!data!set! • The!first!dataset!we!opened!for!public! • Contains!posts!made!in!communiEes! • Contains!users’!likes!for!the!post! • Get!it!at!h0p://likesdataset.sh2014.org/index/! • Try!yourself!at!the!mini[contest!for!predicEng! likes! 28!
- 29. SNA!hakaton!dataset! • Large!dataset!(100+Gb)! • Contains! – Fragment!of!social!graph! – Users’!posts,!likes!and!logins! – CommuniEes!posts! – Demography!data! – Complaints! • To!get!it! mailto:elena.mikhaylova@corp.mail.ru! 29!
- 30. New!users!dataset! • The!freshest!one!(data!up!to!2015!April!8[th)! • Contains! – New!users!registraEons!(40K)! – All!acEviEes!related!to!those!users!(both!own!and!others)! – Social!graph!for!all!involved!users!(4M)! – Demography!data!for!the!users! • PotenEal!tasks:!friends!recommendaEons,!bot! detecEon,!profile!data!correcEon…! • To!get!it!mailto:dmitry.bugaychenko@corp.mail.ru! ! 30!
- 31. What!is!it!about?! • About!us!and!the!size! • Smart!data!vs.!big!data! • Smart!data!tasks! • Smart!data!deck! • Open!data! • A/bit/of/humor/(author/personal/opinion)/ 31!