Almost every application needs data to function - and if you don't know how to be nice to your data, then things will start to go wrong. This talk aims to convince JavaScript developers that they do need to care about statistics, and then talk about how to do so. We look at some theory and lots of case studies and real-world advice to deal with a range of scenarios. The talk aims to touch on the entire data life cycle: We'll dive into data modelling and how the shape and size of your data affects your architecture, and how to build these architectures using JavaScript. Once the data is in the front-end, we'll touch on the wide range of libraries that allows your code to react based on the data, and the wrappers on top that aid visualisation and readability.

1. STAT I ST I CA L P R O G RA M M I N G
I N JAVAS C R I PT
D av i d S i m o n s
@ Swa m Wi t h Tu rt l e s

2. slides:
www.tinyurl.com/stats-js

3. demos:
swamwithturtles.github.io/js-statistics
code:
github.com/SwamWithTurtles/js-statistics

4. W H O A M I ?
Freelance
Software
Developer
@SwamWithTurtles
Java and
JavaScript
Afraid of goats?

5. W H O A M I ?
DATA
NERD

6. C O N T E N T S
T H E O RY CA S E S T U D I E S
JAVA S C R I P T
A P P L I CAT I O N
W H AT I S
DATA ?
G A I N I N G
I N S I G H T S
R A N D O M N E S S S I M U L AT I O N
L E A R N I N G T H R O U G H
Reward: What shape is the internet?

7. Data

9. B E H I N D T H E H O O D
A P I
D B
A D M I N
I N T E R F A C E
S C H E D U L E D
T A S K S
3 R D
P A R T Y
A P I S

10. W H AT D ATA
WA S T H E R E ?
S O …

11. W H AT D ATA
WA S T H E R E ?
• Counts of lists (e.g. brands,
products etc.)
• Stock levels and prices of
products
• Days an item has been out
of stock

12. W H AT D ATA
WA S T H E R E ?
• Non-functional data
• Numbers of users
• Performance for users
• Performance of third
party APIs
• Robustness of system
(Uptime, status codes,
frequency of errors)

13. T H E R E I S D ATA
E V E RY W H E R E
T H E L E S S O N ?

14. What is data?

15. What is good data?

16. W H AT D ATA
S H O U L D I C A R E
A B O U T ?
• Data you get repeatedly
• Data you can extract
‘information’ from
• Normally this means
numerical data, though
NLP is getting big!
• Data that answers valuable
questions

17. Gaining Insights

18. A d a t a s e t :
Identification WIND CEILING TEMP DEWPT RHX
USAF NCDC Date HrMn I Type QCP Dir Q I Spd Q Hgt Q I I Temp Q Dewpt Q RHx
865300,99999,19860401,0000,4,FM-12, ,110,1,N, 7.2,1,22000,1,C,N, 21.6,1, 19.2,1, 86,
865300,99999,19860401,0300,4,FM-12, ,110,1,N, 5.1,1,22000,1,C,N, 19.4,1, 18.5,1, 95,
865300,99999,19860401,0600,4,FM-12, ,070,1,N, 7.2,1,03600,1,C,N, 19.2,1, 999.9,9,999,
865300,99999,19860401,0900,4,FM-12, ,070,1,N, 6.2,1,00120,1,C,N, 19.2,1, 18.9,1, 98,
865300,99999,19860401,1200,4,FM-12, ,070,1,N, 7.7,1,03600,1,C,N, 21.6,1, 18.3,1, 82,
865300,99999,19860401,1500,4,FM-12, ,040,1,N, 9.8,1,03600,1,C,N, 23.0,1, 18.8,1, 77,
865300,99999,19860401,1800,4,FM-12, ,030,1,N, 6.2,1,03600,1,C,N, 19.6,1, 19.0,1, 96,
865300,99999,19860401,2100,4,FM-12, ,050,1,N, 6.7,1,03600,1,C,N, 19.0,1, 18.7,1, 98,
865300,99999,19860402,0000,4,FM-12, ,340,1,N, 7.2,1,03600,1,C,N, 20.0,1, 19.4,1, 96,
865300,99999,19860402,0300,4,FM-12, ,360,1,N, 4.1,1,03600,1,C,N, 19.4,1, 19.1,1, 98,
865300,99999,19860402,0600,4,FM-12, ,999,1,C, 0.0,1,03600,1,C,N, 19.2,1, 18.9,1, 98,
865300,99999,19860402,0900,4,FM-12, ,999,1,C, 0.0,1,00210,1,C,N, 19.0,1, 18.7,1, 98,
865300,99999,19860402,1200,4,FM-12, ,200,1,N, 2.6,1,00210,1,C,N, 20.4,1, 20.1,1, 98,
865300,99999,19860402,1500,4,FM-12, ,210,1,N, 5.1,1,00750,1,C,N, 23.2,1, 19.3,1, 79,
865300,99999,19860402,1800,4,FM-12, ,200,1,N, 3.1,1,00750,1,C,N, 26.4,1, 18.4,1, 62,
865300,99999,19860402,2100,4,FM-12, ,999,1,C, 0.0,1,22000,1,C,N, 26.2,1, 17.1,1, 57,
865300,99999,19860403,0000,4,FM-12, ,140,1,N, 4.1,1,22000,1,C,N, 19.2,1, 17.0,1, 87,
865300,99999,19860403,0300,4,FM-12, ,999,1,C, 0.0,1,22000,1,C,N, 15.8,1, 15.2,1, 96,
865300,99999,19860403,0600,4,FM-12, ,999,1,C, 0.0,1,22000,1,C,N, 15.4,1, 14.0,1, 91,
865300,99999,19860403,1200,4,FM-12, ,060,1,N, 5.1,1,22000,1,C,N, 21.0,1, 19.8,1, 93,
865300,99999,19860403,1500,4,FM-12, ,060,1,N, 4.1,1,00900,1,C,N, 24.8,1, 21.3,1, 81,
865300,99999,19860403,1800,4,FM-12, ,050,1,N, 7.7,1,09000,1,C,N, 28.0,1, 21.4,1, 67,
865300,99999,19860403,2100,4,FM-12, ,040,1,N, 5.1,1,09000,1,C,N, 25.4,1, 21.4,1, 79,
865300,99999,19860404,0000,4,FM-12, ,060,1,N, 6.2,1,03600,1,C,N, 22.2,1, 21.3,1, 95,
865300,99999,19860404,0300,4,FM-12, ,050,1,N, 5.1,1,09000,1,C,N, 21.0,1, 20.7,1, 98,
865300,99999,19860404,0600,4,FM-12, ,060,1,N, 6.2,1,22000,1,C,N, 20.2,1, 19.9,1, 98,
865300,99999,19860404,1200,4,FM-12, ,040,1,N, 5.1,1,00120,1,C,N, 20.4,1, 19.5,1, 95,
865300,99999,19860404,1500,4,FM-12, ,020,1,N, 7.7,1,00420,1,C,N, 24.2,1, 20.4,1, 79,
865300,99999,19860404,1800,4,FM-12, ,250,1,N, 4.1,1,00750,1,C,N, 25.6,1, 20.7,1, 74,
865300,99999,19860404,2100,4,FM-12, ,250,1,N, 5.1,1,00750,1,C,N, 23.6,1, 20.4,1, 82,
865300,99999,19860405,0000,4,FM-12, ,180,1,N, 6.2,1,00420,1,C,N, 20.2,1, 19.6,1, 96,

19. s u m m a r y s t a t i s t i c s

20. S U M M A RY
S TAT I S T I C S
• A statistic is a function of
the data we have inputed
• It aims to capture
information about values
to make it more
understandable

21. T H E FA M O U S
O N E :
• Mean (‘average’)
• Sum all of the data
and divide by the
number of items
• Gives a sense of ‘size’

22. Group 1:
Group 2:

23. O T H E R
S TAT I S T I C S
• “Location”
• Mean, Mode, Median
• “Spread”
• Standard Deviation
• “Shape”
• Skew, Kurtosis

24. D E M O

25. Distributions

26. What is a random variable?

27. Discrete Variables
Can be any of a list of values, each with its own probability
H E A D S 0 . 5
TA I L S 0 . 5
2 1 / 3 6
3 2 / 3 6
4 3 / 3 6
5 4 / 3 6
6 5 / 3 6
7 6 / 3 6
8 5 / 3 6
9 4 / 3 6
1 0 3 / 3 6
1 1 2 / 3 6
1 2 1 / 3 6

28. This makes sense:
X = Result of a coin flip
H E A D S 0 . 5
TA I L S 0 . 5 But:
X won’t always have the
same value

29. R A N D O M VA R I A B L E S
X = Result of a coin flip
H E A D S 0 . 5
TA I L S 0 . 5
X is a
Random Variable
This is its distribution

30. D E M O …

31. Continuous
A numerical variable,
that can be any number
(sometimes within a range)
height
weight
Math.random()

32. H O W D O W E D E F I N E T H E
D I S T R I B U T I O N ?
Math.random() height

33. D E M O

34. S O W H AT ?
E R R R …

35. • When we do data analysis,
we’re really looking at the
range of values a random
variable can be…
• … and asking questions
about its distribution.

36. Y O U ’ R E A N
A U D I T O R
I M A G I N E …

37. A U D I T I N G A
L E D G E R
• Make a list of all ingoing
and outgoing transactions
• These are random
variables.
• What is their distribution?
Does it deviate from what
we expect?

38. B E N F O R D ’ S L A W
http://www.journalofaccountancy.com/Issues/1999/May/nigrini

39. I N T U I T I V E
U S E R I N P U T S
D E S I G N I N G

40. O U R TA S K …
• Designing a system that
tries to understand what
happens under financial
system “shocks”
• So: a user would input a
shock, its impacts would
propagate and we would
see our bottom line.

41. O U R F I R S T AT T E M P T
• Shock ‘sliders’ that scaled linearly
0 %
2 5 %
B O O M
9 0 %
B U S T

42. D I S T R I B U T I O N O F F I N A N C I A L
C H A N G E S

43. S O …
• Shock ‘sliders’ that scaled linearly
0 %
8 %
B O O M
1 0 5 %
B U S T
Change that happens
with 75% chance
Change that happens
with 10% chance

44. Randomness

45. M A K I N G R A N D O M VA R I A B L E S

46. S O M E
WA R N I N G S
• Exactly what randomness
means is a fuzzy question.
• These numbers are not
‘cryptographically’
random.

47. J AVA S C R I P T ’ S
E N T RY T O
R A N D O M N E S S
• Different runtimes can
implement it differently.
• V8 implements Multiply-With-
Carry:
• Take a sequence of ‘seed’
values
• Iteratively perform modular
arithmetic-based operations
• Extend the initial seed values
to a longer sequence.
Math.random()

48. W H AT A B O U T
O T H E R
D I S T R I B U T I O N S ?
B U T …

49. T H E S H O R T A N S W E R
Math.random()= f( )

50. T H E S H O R T A N S W E R
=
H E A D S 0 . 5
TA I L S 0 . 5
=

51. W H AT ’ S T H E F U N C T I O N ?
jStat
beta
centralF
cauchy
chi-squared
exponential
gamma
inverse gamma
kumaraswamy
lognormal
normal
pareto
student t
uniform
weibull
binomial
negative binomial
hypergeometric
poisson
triangular
OR

52. U S I N G R A N D O M N E S S

53. w hy w o u l d i w a n t
t o u s e
R A N D O M N E S S
?

54. S T U B B E D
T E S T D ATA
• Avoid coupling yourself to
specific test
implementations
• Spin-up life-like
environments for load
testing

55. N O N -
D E T E R M I N I S T I C
A L G O R I T H M S
• Modelling underlying or
random data
• Solving a problem that is
expensive or impossible to
solve perfectly

56. P I T FA L L S

57. C H O O S I N G T H E
D I S T R I B U T I O N
• What if a ‘uniform’
distribution isn’t enough?
• What if we want random
data that isn’t just
numbers?

58. E X A M P L E : S O C I A L N E T W O R K

59. E X A M P L E : S O C I A L N E T W O R K
11 Traversals

60. D E M O

61. B a r a b a s i - A l b e r t
R a n d o m M o d e l

62. B A R A B A S I - A L B E R T
R A N D O M M O D E L
• Start with two linked
objects
• Add one new object at a
time
• Link that object to one
existing object, with
already ‘popular’ objects
more likely to be chosen.

63. T H I S
M O D E L S …
• Academic Citations
• Actor filmographies
• Spread of Infectious
diseases
• Social Networks

64. C O N T E N T S
T H E O RY CA S E S T U D I E S
JAVA S C R I P T
A P P L I CAT I O N
W H AT I S
DATA ?
G A I N I N G
I N S I G H T S
R A N D O M N E S S S I M U L AT I O N
L E A R N I N G T H R O U G H
Reward: What shape is the internet?

65. We’reOUTof
TIME

66. • Data is any information we collect. Not all data is
valuable.
• Seeing trends in lots of numbers is hard. Summary
statistics and charts help us unpick its meaning.
• Data can be treated as random ‘realisations’ from a
backing distribution.
• Making random variables is easy, and can be done in
different shapes for different purposes.
W H AT I S
DATA ?
G A I N I N G
I N S I G H T S
R A N D O M N E S S S I M U L AT I O N

67. L I B R A R I E S W E U S E D
G E N E R A L L I B R A R I E S
K N O C K O U T. J S
R E Q U I R E . J S
B O O T S T R A P
D ATA M A N I P U L AT I O N
L O D A S H
J S TAT
D ATA I M P O RT PA PA PA R S E
C H A RT I N G
D 3
C H A R T. J S

68. T H A N K YO U
D av i d S i m o n s
@ Swa m Wi t h Tu rt l e s