You may not be sure how Lord Varys collects information from his little birds, but in this talk you will hear how we can collect information from our little birds. @kristw shares a behind-the-scenes view of his latest data visualization project, which shows how each #GameOfThrones episode was discussed on Twitter. Using data visualization, we can extract and reveal the stories of every episode from fans’ Tweets. https://interactive.twitter.com/game-of-thrones These slides are from a talk given at Bay Area d3 User Group meetup on June 9, 2016. http://www.meetup.com/Bay-Area-d3-User-Group/events/231281298

1. from fans’ conversations
Krist Wongsuphasawat / @kristw
Reveal the talking points
of every episode of

2. Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Information Visualization
Univ. of Maryland
IBM
Microsoft
Data Visualization Scientist
Twitter
Krist Wongsuphasawat / @kristw

3. interactive.twitter.com open-source visualization visual analytics tools

4. Problem is coming.
CHAPTER I

5. “Problem first, not solution backward”
— Brian Caffo (via Ron Brookmeyer)

6. “If all you have is a hammer,
everything looks like a nail.”
— Abraham Maslow

7. Problem
Want to know what the audience
talk about a TV show

8. Problem
Want to know what the audience
talk about a TV show
from Tweets

9. HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.

10. Brief Story

11. A King dies.
A lot of contenders wage a war
to reclaim the throne.

12. Minor characters with no claim to the throne
set their own plans in action to gain power
when all the major characters end up killing each other.

13. Brave/Honest/Honorable characters die.
Intelligent but shady characters
and characters who know nothing
continue to live.

14. On another continent,
there is a princess who was supposed to be
rightful heir to the throne.
The dead king mentioned earlier,
overthrew her father
and killed her entire family.
She escaped.
Now she is finding her way back
and she has dragons.

15. While humans are busy killing each other,
ice zombies “White walkers” are invading from the North.
The only group who seems to care about this
is neutral group called the Night’s Watch.

16. HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.
Many characters.
Anybody can die.
6 seasons (57 episodes) so far
Multiple storylines in each episode

17. Problem
Want to know what the audience
talk about a TV show
from Tweets

18. Ideas
Common words
Too much noise

19. Ideas
Common words
Too much noise
Characters
How o"en each character were mentioned?

20. I demand a trial by prototyping.
CHAPTER II

21. Prototyping
Pull sample data
from Twitter API
Entity recognition and counting
naive approach

22. List of names
Daenerys Targaryen,Khaleesi
Jon Snow
Sansa Stark
Tyrion Lannister
Arya Stark
Cersei Lannister
Khal Drogo
Gregor Clegane,Mountain
Margaery Tyrell
Joffrey Baratheon
Bran Stark
Theon Greyjoy
Jaime Lannister
Brienne
Eddard Stark,Ned Stark
Ramsay Bolton
Sandor Clegane,Hound
Ygritte
Stannis Baratheon
Petyr Baelish,Little Finger
Robb Stark
Bronn
Varys
Catelyn Stark
Oberyn Martell
Daario Naharis
Davos Seaworth
Jorah Mormont
Melisandre
Myrcella Baratheon
Tywin Lannister
Tommen Baratheon
Grey Worm
Tyene Sand
Rickon Stark
Missandei
Roose Bolton
Robert Baratheon
Jojen Reed
Jeor Mormont
Tormund Giantsbane
Lysa Arryn
Yara Greyjoy,Asha Greyjoy
Samwell Tarly,Sam
Hodor
Victarion Greyjoy
High Sparrow
Dragon
Winter
Dothraki

23. Sample Tweet

24. Sample Tweet

25. Sample data
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
Bran Stark 3000
… …
*These numbers are made up for presentation, not real data.

26. When you play the game of vis,
you iterate or you die.
CHAPTER III

27. Where to go from here?

28. + episodes
The Guardian & Google Trends
http://www.theguardian.com/news/datablog/ng-interactive/2016/apr/22/game-of-thrones-the-most-googled-characters-episode-by-episode

29. + emotion

30. + connections

31. + connections

32. Gain insights from a single episode
emotion & connections

33. Better matching
Regular Expression
/gregor([ ]?clegane)?|(the mountain)/

34. Sample data
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
… …
INDIVIDUALS CONNECTIONS
+ top emojis + top emojis
*These numbers are made up for presentation, not real data.

35. Graph
NODES LINKS
+ top emojis + top emojis
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 1000
Jon Snow 500
Daenerys 400
… …
*These numbers are made up for presentation, not real data.

36. Network Visualization
Node-link diagram
Force-directed layout
http://blockbuilder.org/kristw/762b680690e4b2b2666dfec15838a384

37. Issue: Hairball

38. Why?
Too many nodes & edges
nodes = nodes.filter(n => n.count > 100)
links = links.filter(l => l.count > 100)
The force is (too) strong.
force
.charge(…)
.gravity(…)
.linkDistance(…)
.linkStrength(…)

39. Issue: Occlusions

40. Tried: Fixed positions

41. + Collision Detection
http://blockbuilder.org/kristw/2850f65d6329c5fef6d5c9118f1de6e6

42. + Community Detection
https://github.com/upphiminn/jLouvain

43. + Collision Detection (with clusters)
https://bl.ocks.org/mbostock/7881887

44. Issue: Convex hull
http://bl.ocks.org/mbostock/4341699
d3.geom.hull(vertices)

45. x & y only, no radius

46. Example

47. Fix it

48. Fix it

49. Let’s get other episodes.

50. Hadoop remembers.
CHAPTER IV

51. More data
Hadoop
Rewrite the scripts in Scalding
to get archived data

52. How much data do we need?
Whole week?
5 days?
2 days?
A day?
etc.

53. How much data do we need?

54. Must work for every episode
Too many nodes & edges
nodes = nodes.filter(n => n.count > 100)
links = links.filter(l => l.count > 100)
Is 100 a good number for every episode?

55. Transitions

56. … Jumpy …

57. Too fast?
Slow down
force.friction(f)
0 = stop
0.1 = very slow
1 = very fast (frictionless)

58. A$er switching episode
1. Store old positions for existing objects.
2. Assign new initial positions.*

59. Initial positions
Default: random
Better starting points
Heuristics based on degree of nodes

61. A$er switching episode
1. Store old positions for existing objects.
2. Assign new initial positions.*
3. Run simulation without updating <svg> for n rounds
4. Animate objects from old to new positions.
5. Resume simulation and update <svg> every tick.

62. Animate Nodes & Links
Remove
delay
Move & Change size/thickness
Add new

63. const selection = svg.selectAll('g.node')
.data(nodes, d => d.entity.id);
selection.exit()
.transition()
.duration(1000)
.style('opacity', 0)
.remove();
const sEnter = selection.enter().append('g')
.classed('node', true)
.attr('transform', d => `translate(${d.x},${d.y})`)
.style('opacity', 0)
.call(force.drag);
sEnter.append('circle')
.attr('r', d=>d.r)
.style('fill', d => options.colorScale(d.entity.group));
const sTrans = selection.transition()
.delay(1000)
.duration(2000)
.attr('transform', d => `translate(${d.x},${d.y})`)
.style('opacity', 1)
sTrans.select('circle')
.attr('r', d=>d.r)
Add “enter” nodes
with opacity 0
After 1s delay,
use transition to move nodes
and fade in new nodes
Fade “exit” nodes to opacity 0
and remove
Create selection

64. Animate Communities
Remove
delay
Move & Change shape*
Add new
http://blockbuilder.org/kristw/f9ffe87dd8b4038b5867e853c27cebb7

65. Default
t=0 t=1

66. Smoother
t=0 t=1t=0.5 t=0.51

67. Code
// original
path.attr('d', hull);
// with custom interpolation
path.attrTween('d', (d,i,currentAttr) =>
interpolateHull(d, currentAttr)
)

68. Colors
Default: d3.category10()
Distinct but nothing about the context
Custom palette
Colors related to the groups/houses.
Black = Night’s Watch
Blue = North
Red = Daenerys
Gold = Lannister
…

69. Hold the vis.
CHAPTER V

70. The vis is not enough.

72. Legend

73. Navigation

74. Top 3

75. Adjust threshold

76. Recap

77. Filtered Recap
Tooltip

78. Demo
https://interactive.twitter.com/game-of-thrones

79. Mobile Support

80. A visualizer always evaluates his work.
CHAPTER VI

81. “Feedback is the breakfast of champion.”
— Ken Blanchard

82. Self & Peer
Does it solve the problem?

83. Google Analytics
Pageviews
Visitors
Actions
Referrals
Sites/Social

84. Feedback

85. Feedback

86. You know nothing, algorithm.
CHAPTER VII

87. Pros
List of Characters + Tweets = Vis.
No story or relationship as input

88. Cons
Weak assumption about connections
Can misclassify
Noise

89. All talk must end.
CHAPTER VIII

90. Summary
Problem first, not solution backwards
Prototype, Iterate, Scale & Adapt
Identify characters from Tweets + Network visualization
Vis is important, but there are other parts.
Feedback
Understand the pros & cons
Rooms for improvement
kristw.yellowpigz.com
Krist Wongsuphasawat / @kristw

91. Robert Harris, Miguel Rios, Elaine Filadelfo
and many colleagues at Twitter;
Elijah Meeks for his network vis tutorial;
Mike Bostock for D3 and examples
Lastly, to my wife for taking care of our baby, so I had time to prepare these slides.
Acknowledgement

92. Resources
Quora: How would you explain the plot of Game of Thrones in Brief from the beginning?
https://www.quora.com/How-would-you-explain-the-plot-of-Game-of-Thrones-in-brief-from-the-beginning
Convex Hull
https://en.wikipedia.org/wiki/Convex_hull
Images
Aemon - http://tinyurl.com/zqdzc2a
Cersei - http://tinyurl.com/gumcs7g
Crow - http://tinyurl.com/ju6up5g
Eddard - http://tinyurl.com/jc23mel
Feast - http://tinyurl.com/zcms4lq
Hammer - http://tinyurl.com/hz8emrp
Hodor - http://tinyurl.com/j748jfr
House sigils - http://tinyurl.com/jywgcjx
House Stark - http://tinyurl.com/jtdtrdy
Jon Snow - http://tinyurl.com/h4hofe8
Tyrion - http://tinyurl.com/z7z2uow
Tyrion - http://tinyurl.com/hvw4u89
Tyrion - http://tinyurl.com/jkrvqtb
Watercolor Map by Stamen Design

93. Thank you

94. Questions?