A deck presented at the MRS 'Maximising the Value of Big Data' conference in London, January 2013.
Presents my view of big data and the potential it gives us for mapping the systems that we deal with on a day-to-day basis. Big data holds the promise of providing us with a meta-view of the systems that we all think we are so familiar with. I think we will find that the woods look nothing like the trees.
How to Troubleshoot Apps for the Modern Connected Worker
Big Data: Mapping Twitter Communities
1. Mapping Twitter Communities
Presented at the MRS ‘Maximizing the Value of Big Data’ conference, London, January 2013
Kyle Findlay (TNS)
Paul Oosterveld (TNS)
Timothy Wilson (Independent)
2. My understanding of ‘big data’
Lots of Resource- Resource-
intensive to intensive to
data collect process
(often unstructured)
Meta view of A new world of …through
insights, technique
old, familiar s, understanding…
primary
systems research???
3. The tools
Machine learning Integration of
Text and artificial Next-gen
multiple data
analysis intelligence
sources databases
Etc . etc.
etc.
4. What we are really …is mapping complex
doing here though… systems
11. Who are the popular Who is influencing What is the nature What sub-
and/or influential whom within a of their influence i.e. communities exist
individuals in my specific topic of how are they within my online
network?
interest to me? influencing people? community?
What topics are What are people saying What do these sub- Who is sharing
about my communities believe
people talking brand, political and how do they relate my content
about? party, charity, music, et to my brand/ online?
c.? organisation/etc?
What are the Are they saying How far is word of How is my
emerging topics mostly positive or my brand or messaging being
and trends? negative things? negative publicity shared online?
being spread?
Relevance?
12. A paradigm shift…
Participants not respondents
Market (CASRO)
researchers “Private gardens” vs. public spaces
misunderstand
social media “Representative sample” what?
research…
Source: Barber, Michael, 2011
14. Tweets data summary Number of tweets collected:
@helenzille = 28,500
@PresidencyZA = 3,500
@SAPresident = 2,900
Time period: Sept 2011 – March 2012
15. Target
nodes
Source nodes
Collected 64,357
Cleaned 58,349
Distance <= 4 37,695
2.5 million
Distance <= 3 20,460
Distance <= 2 12,753
Distance <= 1 4,479
Distance = 0 3
Time period: April 2012
Twitter user data summary
17. Topic modelling
Hashtags
Types of analyses
Structural network
Betweeness centrality
Authority (HITS)
Community detection
Influence networks
Image sources: Knowledge Matters. http://www.durantlaw.info/category/miscellaneous/strategy
Gutiérrez-Pérez, JA, et al. 2011. Application of graph-spectral methods in the vulnerability assessment of water supply networks
31. Top 10 sportsmen & celebrities
Follower count
Helen Zille Bryan Habana John Smit Albie Morkel Mark Boucher
157,642 100,141 84,529 94,593 84,564
Cricket South South African Trevor Immelman Steve Hofmeyr 94.7 Highveld
Africa 61,536 Rugby 60,280 53,777 39,351 Radio 37,554
32. Top 10 sportsmen & celebrities
Betweeness centrality
Rob van Vuuren Arno Carstens Cape Talk 567 Alex van Tonder Sam Wilson
16,610 11,562 11,685 10,065 6,756
Karen Zoid Lead SA Jeannie D Creative Cape The Foodie (David)
14,410 18,786 38,858 Town 6,259 5,219
33. Top 10 news media
Follower count
Yaseen Theba Mail & Guardian South African BBC Africa Redi Tlhabi
77,191 57,320 Presidency 53,413 49,425 45,866
SA Breaking News City Press allAfrica.com Mandy Wiener Evita Bezuidenhout
38,487 36,376 32,788 31,296 30,139
34. Top 10 news media
Betweeness centrality
Mandy Wiener Sam Mkokeli Ferial Haffajee Mail & Guardian Robyn Clark
31,296 3,033 29,530 57,320 1,598
Sipho Hlongwane US South African Verashni Pillay The Big Issue Stephen Grootes
9,093 Embassy 17,803 8,304 4,363 19,708
35. Top 10 technology
Follower count
Aki Anastasiou Jaco van Wyk Simon Dingle Dave Duarte Toby Shapshak
28,063 20,087 15,386 9,563 9,073
Finance24 Mxit Matthew Buckland Pieter Uys Duncan McLeod
8,184 7,651 7,033 6,654 6,194
36. Top 10 technology
Betweeness centrality
Matthew Buckland Dave Duarte Aki Anastasiou Raoul de Jongh Toby Shapshak
7,033 9,563 28,063 4,133 9,073
Saul Kropman Mike Sharman Uno de Waal Pete Flynn Cathryn Reece
2,530 4,680 2,469 1,008 2,302
37. Top 10 black influentials
Follower count
Bonang Matheba Julius Malema Dineo Ranaka Jacob Zuma Black Coffee
231,696 (unconfirmed) 192,482 142,745 136,403 97,302
David Kau Terry Pheto Pabi Moloi Claire Mawisa Kuli Roberts
83,169 64,117 66,841 54,562 57,352
38. Top 10 black influentials
Betweeness centrality
Khaya Dlanga Simphiwe Dana TimesLIVE Julius Malema Xolisa Dyeshana
46,101 33,986 52,706 (unconfirmed) 192,482 4,747
Jason Von Berg David Kau Zama Ndlovu Bonang Matheba Mvelase Peppetta
4,445 83,169 6,718 231,696 2,804
39. Top 10 proudly SA & DA
Follower count
Lindiwe Mazibuko SA The Good News Brand South Africa SouthAfrica.info City of Cape Town
30,460 12,483 8,549 8,575 7,725
Ryan Coetzee Mmusi Maimane Tim Harris DA Youth Andrew Boraine
3,528 4,917 2,792 3,239 2,214
40. Top 10 proudly SA & DA
Betweeness centrality
Lindiwe Mazibuko Ryan Coetzee Climate Smart Tim Harris Mmusi Maimane
30,460 3,528 Cape Town 867 2,792 4,917
Solly Malatsi Gareth van Mbali Ntuli Cape Town Phumzile Van
853 Onselen 2,085 1,903 Green Map 1,408 Damme 1,588
48. A new kind of research Complex systems
Influence and dynamics
Public data
MR = computer science?
CONCLUSIONS
Primary research
Paradigm shift Representative what!?
Participants, not respondents
3rd party disruptive innovation
51. Top 10 sportsmen & celebrities
Authority
Helen Zille Rob van Vuuren John Smit Arno Carstens Cape Talk 567
Bryan Habana Alex van Tonder Sam Wilson Karen Zoid Vanessa Raphaely
52. Top 10 news media
Authority
South African
Mail & Guardian Ferial Haffajee Mandy Wiener Stephen Grootes
Presidency
Evita Bezuidenhout Redi Tlhabi Zapiro Max du Preez Philip de Wet
53. Top 10 technology
Authority
Aki Anastasiou Matthew Buckland Toby Shapshak Dave Duarte Simon Dingle
Duncan McLeod Memeburn Cherryflava Media Pieter Uys Mike Sharman
54. Top 10 black influentials
Authority
Julius Malema
Jacob Zuma TimesLIVE Khaya Dlanga Leanne Manas
(unconfirmed)
Riaad Moosa Zakes Mda David Kau Simphiwe Dana Pabi Moloi
55. Top 10 proudly SA & DA
Authority
Lindiwe Mazibuko Ryan Coetzee SA The Good News City of Cape Town Mmusi Maimane
Gareth van
Tim Harris David Maynier Andrew Boraine Gareth Morgan
Onselen