Using MongoDB for great extendability and scalability for our project musweet.com

1. musweet.com
Handling Humongous Data Sets
from the Social Web
Grischa Andreew & Nader Cserny, compuccino

2. Agenda
• Einführung
• Technik
• Daten im Detail
• Abfragen
• Tools & Debugging
• Fragen

4. Einführung

5. Was ist musweet?

6. Media Stream

7. Themen

8. Analytics

9. Pro l

10. Statistik
7002 KÜNSTLER
20.848 3.914.259
SOCIAL MEDIA PROFILE STREAM ITEMS
~3450
READ QUERIES / SEC
309.855
MEDIEN (AUDIO, VIDEO, FOTO)
~15.000 INSERTS / DAY
3.75 GB DATA SIZE

12. Technik

13. Was brauchen wir?
KÜNSTLER Name, City, Genre, Bild
SOCIAL PROFILE Plattform (z.B. twitter), Link
MEDIA POSTS Bilder, Videos, Audios, Statusmeldungen
PROFIL INFO Freunde, Follower, Datum, Webseiten,
Pro lbild, Biographie, Label, etc.

14. MySQL Schema
Artist
id
name
Indexs
name
Numbers
artist
socialprofile
outgoing Socialprofile artist_genres
incoming id artist
feedback artist genre
push url Indexes
Service Informations Twitter
artist Indexes service artist_genre
lang, artist_outgoing Indexes
verified artist_incoming artists_service
location, artist_feedback
id, artist_push
url, artist
created_at,
description, Stream
time_zone, artist Genres
profile_image_url, socialprofile id
screen_name Service Informations Facebook message name
Indexes artist created_at Indexes
artist category, Indexes name
name, message
fan_count, artist_created_at
Service Informations Myspace bio, created_at
artist url,
website, username,
genre, record_label,
location, location,
art_des_labels, profile_image_url, Stream Informations Facebook
Stream Informations Twitter
headline, band_members, stream Stream Informations Myspace
stream
created_at website, name, stream
source,
id, ink, caption, category,
in_reply_to_status_id,
profile_image_url, pinnwand_posts, link, image,
in_reply_to_user_id,
label genre, likes, link,
truncated,
Indexes friends, type, source
deleted
artist id icon Indexes
Indexes
Indexes Indexes stream
stream
artist stream

15. MongoDB Schema
Artist
id (Object Id)
name (str)
genres (strict array)
socialprofiles (strict array)
service (dbref)
url (str)
numbers (strict array)
incoming
outgoing
push
feedback
date
meta (array)
(unterschiedliche Felder, ja nach Plattform)
Indexes
name,
genres,
socialprofiles.service,
socialprofiles.numbers
Stream
id (Hash aus facebook / myspace / twitter id)
socialprofile (dbref)
genres (strict array) (redundanz der genres vom
artists um den stream direkt über genres
abzufragen)
data (array)
( data from plattforms,
field message is a must have)
created_at (datetime)
Indexes
socialprofile,
genres,
data.message

16. Wie kommen wir an die Daten? (Einfach)
Crawler musweet
• Verarbeitung von Links Links • Darstellung der Inhalte
• Extraktion von Medien • Zuordnung Artist / Service
• Aufbereitung der Inhalte
Daten

18. Daten im Detail

19. Künstler Pro l bei MySpace
"numbers" : {
"outgoing" : 221665,
"incoming" : 770355,
"feedback" : 36862603,
"push" : 0,
"date" : "Wed Sep 29 2010 02:00:00 GMT+0200 (CEST)",
},
"meta" : {
"website" : "http://www.snoopdogg.com",
"genre" : "Hip Hop / Rap / R&B",
"location" : "Long Beach, California Vereinigte Staaten von Amerika",
"art_des_labels" : "Major",
"headline" : "",
"created_at" : "Sat Dec 11 2004 01:00:00 GMT+0100 (CET)",
"id" : 6344278,
"profile_image_url" : "http://c1.ac-images.myspacecdn.com/images02/130/
m_9857dcca155247b69e1260e6e34cce3c.jpg",
"label" : "Doggystyle / Priority"
}

20. Künstler Pro l bei twitter
"numbers" : {
"outgoing" : 1204,
"incoming" : 2030350,
"feedback" : 22750,
"push" : 3145,
"date" : "Wed Sep 29 2010 02:00:00 GMT+0200 (CEST)"
},
"meta" : {
"lang" : "en",
"verified" : true,
"location" : "LBC",
"id" : 3004231,
"url" : "http://www.snoopdogg.com",
"created_at" : "Fri Mar 30 2007 21:05:42 GMT+0200 (CEST)",
"description" : "More Malice CD + DVD IN STORES NOW",
"time_zone" : "Pacific Time (US & Canada)",
"profile_image_url" : "http://a3.twimg.com/profile_images/1096549203/snoop_normal.jpg",
"screen_name" : "SnoopDogg"
}

21. Künstler Pro l bei facebook
"numbers" : {
"outgoing" : 0,
"incoming" : 2930860,
"feedback" : 0,
"push" : 0,
},
"meta" : {
"category" : "Musicians",
"name" : "Snoop Dogg",
"fan_count" : 2930860,
"bio" : "The offices at the top of the Capitol Records building in Hollywood are home to some
of Southern California’s most awe-inspiring views. ....",
"url" : "http://www.facebook.com/snoopdogg?v=info",
"username" : "snoopdogg",
"record_label" : "Priority/Doggystyle ",
"location" : "Long Beach, CA",
"profile_image_url" : "http://profile.ak.fbcdn.net/hprofile-ak-snc4/
hs622.snc3/27524_11455644806_1192_s.jpg",
"band_members" : "Snoop Dogg",
"website" : [
"http://www.snoopdogg.com",
"http://www.myspace.com/snoopdogg",
"http://twitter.com/snoopdogg"
],
"link" : "http://www.facebook.com/snoopdogg",
"pinnwand_posts" : 0,
"genre" : "Hip Hop / Rap / R&B",
"friends" : 0,
"id" : "11455644806"
}

23. Abfragen

24. MySQL vs. MongoDB (1)
Alle Social Media Pro le mit Follower-Zahlen von einem Artist
MySQL MongoDB
SELECT db.artist.find( { "name": "Snoop Dogg" } )
n.incoming,
a.id as artist,
a.name as artist_name, Dauer: 0.0001 Sek.
s.id as socialprofile,
s.url as socialprofile_url,
FROM
numbers as n
JOIN socialprofile as s on s.id =
n.socialprofile
JOIN artist as a on a.id = n.artist
WHERE
a.name = "Snoop Dogg"
ORDER n.incoming DESC
Dauer: 0.0288 Sek.

25. MySQL vs. MongoDB (2)
10 HipHop Musiker mit den meisten Followern
MySQL MongoDB
SELECT db.artist.find( {
n.incoming, "genre": DBRef("genre","hiphop")
a.id as artist, } ).sort( {
a.name as artist_name, "socialprofiles.numbers.incoming": -1
s.id as socialprofile, } ).limit(10)
s.url as socialprofile_url,
FROM
Dauer: 0.0230 Sek.
numbers as n
JOIN artist_genres as ag on
ag.artists = n.artist
JOIN genres as g on g.id = ag.genre
JOIN socialprofile as s on s.id =
n.socialprofile
JOIN artist as a on a.id = n.artist
WHERE
g.name = "Hip/Hop"
ORDER BY
n.incoming DESC
LIMIT 10
Dauer: 0.8741 Sek.

26. MySQL Index
• Index wird von links nach rechts gelesen
Reihenfolge wichtig
Felder: „artist“, „incoming“, „push“, „date“
SELECT * FROM numbers WHERE artist = 1 Funktioniert
SELECT * FROM numbers WHERE incoming = 1 Funktioniert nicht
SELECT * FROM numbers WHERE artist = 1 AND push < 10 Funktioniert nicht
SELECT * FROM numbers WHERE artist = 1 AND push < 10 AND incoming > 0 Funktioniert
• Index Debugging
EXPLAIN SELECT * FROM numbers WHERE artist = 1

27. MongoDB Index
• Index Reihenfolge ist egal
kann ein Feld mitten im Index verwenden
db.artist.ensureIndex( {"name":1, "numbers": -1 } );
db.artist.find( { "name": "Snoop Dog" } ) Funktioniert
db.artist.find( { "socialprofiles.numbers.incoming": { "$gte": 10 } } ) Funktioniert
db.artist.find( {
"name": "Snoop Dogg",
"socialprofiles.numbers.incoming": { "$gte": 0 }
} ) Funktioniert
• Index Debugging
db.artist.find( { "name": "Snoop Dogg" } ).explain()

29. Tools & Debugging

30. MongoDB Fehlermeldungen
• Sortierte Abfrage ohne Limit:
Fehler: „too much data for sort() with no index. add an index or
specify a smaller limit“
Lösung: Feld in den Index aufnehmen
• Duplicate Key Error:
Fehler: in älteren Versionen (< 1.6.0) schmiert DB bei zu vielen
Duplicate Key Errors ab
Lösung: Upsert verwenden

31. db.serverStatus()
Wieviel memory-Verbrauch, wieviele Connections, ...
globalLock Wie lange Collections gesperrt waren, ...
connections Wieviel Verbindungen offen / verfügbar, ...
backgroundFlushing Wann war der letzte Flush auf die Festplatte, ...
... Mehr Info in der Dokumentation:
http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics

32. Pro ling
db.setPro lingLevel(0) off
log slow operations (>100ms), optional „slow“
db.setPro lingLevel(1)
de nieren mit db.setPro lingLevel(1, 10)
db.setPro lingLevel(2) log all operations
system.pro le
{ "ts" : "Tue Aug 10 2010 00:25:33 GMT+0200 (CEST)", "info" : "update musweet.media query:
{ _id: ObjId(4c607ef2a68299079400b5ea) } nscanned:1 moved ", "millis" : 0 }
{ "ts" : "Tue Aug 10 2010 00:25:33 GMT+0200 (CEST)", "info" : "update musweet.media query:
{ _id: ObjId(4c607ef2a68299079400b468) } nscanned:1 moved ", "millis" : 0 }
{ "ts" : "Tue Aug 10 2010 00:25:33 GMT+0200 (CEST)", "info" : "update musweet.media query:
{ _id: ObjId(4c607fd9a68299079400c067) } nscanned:1", "millis" : 0 }
{ "ts" : "Tue Aug 10 2010 00:25:33 GMT+0200 (CEST)", "info" : "update musweet.media query:
{ _id: ObjId(4c607ef2a68299079400b6a6) } nscanned:1 moved ", "millis" : 0 }

33. Collection Objekte analysieren
Download: http://github.com/compuccino/mongodb-ac

35. Abschließend...

36. Abschließend...
• Fragen?
• Mehr über uns:
http://compuccino.com
http://facebook.com/compuccino
• Personen:
Grischa Andreew, @grischaandreew
Nader Cserny, @nadr