So you’re a big data and distributed systems “expert”, you’ve collected 500 billion data points, thrown it into sci-lib-of-the-week, you’re using Hadoop, backing onto those cool AWS GPU instances, let it grind away for days and its spit out the answer to life the universe and everything. But is it really better than a coin toss? How do you validate whether your data analysis algorithm works? Are you learning a solution to your problems or just the data you already have? What problems can you encounter when analysing your data? How do you solve them, and what can you do easily under the time pressures of a business environment?

1. ARE YOU BETTER THAN AARE YOU BETTER THAN A
COIN TOSS?COIN TOSS?
BY JOHN OLIVER AND RICHARD WARBURTONBY JOHN OLIVER AND RICHARD WARBURTON

3. WHO ARE WE?WHO ARE WE?

4. Why you should care
The Fundamentals
Practical Problems
Applying the Theory

8. 'EXPERTS" AREN'T VERY GOOD'EXPERTS" AREN'T VERY GOOD

10. BIG DATA SOLVES ALLBIG DATA SOLVES ALL
KNOWN PROBLEMSKNOWN PROBLEMS

11. BIG DATABIG DATA SOLVES ALLSOLVES ALL
KNOWN PROBLEMSKNOWN PROBLEMS
... HELPS... HELPS

12. VALIDATION = TESTSVALIDATION = TESTS
FOR DATAFOR DATA

13. PART 1: FUNDAMENTALSPART 1: FUNDAMENTALS

14. NULL HYPOTHESISNULL HYPOTHESIS
Untilproven otherwise there is no relationship between
phenomena

15. WHEN YOU HEAR "WOLF!" THERE IS A WOLF NEARBYWHEN YOU HEAR "WOLF!" THERE IS A WOLF NEARBY
Cry"Wolf!" StayQuiet
Wolf Nearby Ok False Negative
Its reallyachicken! False Positive Ok

16. WHY IS THIS IMPORTANT?WHY IS THIS IMPORTANT?

17. Itis better thatten guilty persons escape than
thatone innocentsuffer
-William Blackstone

19. STATIC ANALYSISSTATIC ANALYSIS

20. COST BENEFIT ANALYSISCOST BENEFIT ANALYSIS
Costs alotto jailan innocentman
Costs verylittle to show someone an inappropriate house
Credibility, Liberty, Moralityare also costs

21. CHOOSE THE RIGHT MEASUREMENTCHOOSE THE RIGHT MEASUREMENT
There's more than one conceptof accuracy

22. RECALLRECALL
number of true positives /number of actuallytrue values

23. PRECISIONPRECISION
number of true positives /predicted true value

24. F MEASUREF MEASURE

25. CASE STUDY: MEMORY LEAKSCASE STUDY: MEMORY LEAKS
About~10%of our datasethad memoryleaks
Predict"never leaks memory"~= 0.9 accuracy, butF1 = 0
Our algorithm ~= 0.9 accuracyand F1 ~= 0.9

26. PROBLEM: RELIABILITY OF MEASUREMENTPROBLEM: RELIABILITY OF MEASUREMENT

27. RULE OF THUMBRULE OF THUMB
If itlooks like random noise, itprobablyis random noise.

28. SOLUTION: CHECK YOUR DATASOLUTION: CHECK YOUR DATA
Low Standard Deviation
Coefficientof Variation = Standard Deviation /Mean

29. CAVEAT: NON-NORMAL DISTRIBUTONSCAVEAT: NON-NORMAL DISTRIBUTONS

31. SOLUTION: GO MADSOLUTION: GO MAD

32. MEDIAN ABSOLUTE DEVIATIONMEDIAN ABSOLUTE DEVIATION

33. PROBLEM: EXPERIMENTAL FLUKESPROBLEM: EXPERIMENTAL FLUKES

34. IS YOUR A/B TEST A HEISEN TEST?IS YOUR A/B TEST A HEISEN TEST?

35. SOLUTION: P-VALUESOLUTION: P-VALUE

36. SCIENCE WORKS - B****ES!SCIENCE WORKS - B****ES!

37. PRACTICAL PROBLEMSPRACTICAL PROBLEMS
PART 2PART 2

38. PROBLEM: FALSE PROPHETSPROBLEM: FALSE PROPHETS

39. I'M AN EXPERT, LISTEN TO ME!I'M AN EXPERT, LISTEN TO ME!

40. SOLUTION: ESTABLISH GOALS AND HYPOTHESIS THEN TESTSOLUTION: ESTABLISH GOALS AND HYPOTHESIS THEN TEST
SOLUTIONSSOLUTIONS

41. PROBLEM: CODE QUALITYPROBLEM: CODE QUALITY
The math works :-) the code does not:-(
@headinthebox

42. GROWTH IN A TIME OF DEBTGROWTH IN A TIME OF DEBT

43. SOLUTION: SOFTWARE ENGINEERING PRACTICESSOLUTION: SOFTWARE ENGINEERING PRACTICES

44. Everyone Lies
-House

45. SOLUTION: UNDERSTAND BIASES AND DESIGNSOLUTION: UNDERSTAND BIASES AND DESIGN
AROUND THEMAROUND THEM

46. Gay couples should have an equal rightto get
married, notjustto have civil partnerships
Populus: 65%vs 27%
Marriage should continue to be defined as alife-
long exclusive commitmentbetween aman and
awoman
Comres + Catholic Voices: 22%vs 70%

47. ACQUIESCENCE BIASACQUIESCENCE BIAS
Answer yes if there’s apositive connotation

48. REMOVAL OF PARTICULAR ADVERTISING AND SPONSORSHIP BANSREMOVAL OF PARTICULAR ADVERTISING AND SPONSORSHIP BANS
FOR: 1045
AGAINST: 731
ABSTAIN: 121
Motion Carried
MAINTAINING AN ETHICAL UNION BY REAFFIRMING ADVERTISING AND SPONSORSHIP BANSMAINTAINING AN ETHICAL UNION BY REAFFIRMING ADVERTISING AND SPONSORSHIP BANS
FOR: 858
AGAINST: 755
ABSTAIN: 166
Motion Carried

49. SOLUTION: PHRASE QUESTIONS NEUTRALLYSOLUTION: PHRASE QUESTIONS NEUTRALLY
And onlyhave one question

50. SOCIAL DESIRABILITYSOCIAL DESIRABILITY
Poor people overestimate their income, rich people under
estimate it.

51. SOLUTIONSSOLUTIONS
Anonymisation
Confidentiality
Randomized Response
Bogus Pipeline

52. BIAS TOWARDS THE FIRST ANSWER OF A QUESTIONBIAS TOWARDS THE FIRST ANSWER OF A QUESTION
Make sure to randomise the order of answers

53. WHAT WILL THE NEXT CRISIS IN WASHINGTON BE?WHAT WILL THE NEXT CRISIS IN WASHINGTON BE?
Fightover the debtceiling
Difficultyavertingautomatic cuts to the Pentagon
Failure to pass basic budgetbills
Allof the above
http://www.foxnews.com/politics/elections/2012/you-decide/what-will-next-crisis-washington-be

54. PROBLEM: CORRELATION DOESN’T IMPLY CAUSALITYPROBLEM: CORRELATION DOESN’T IMPLY CAUSALITY

55. DATABASE AND NETWORK ACTIVITY CORRELATINGDATABASE AND NETWORK ACTIVITY CORRELATING
Performance Diagnosis: was actuallyaGC Problem.

56. SOLUTION: DOMAIN KNOWLEDGESOLUTION: DOMAIN KNOWLEDGE

58. SOLUTIONSSOLUTIONS
Use domain knowledge -ask Pilots
Stratified sample sets
Measure outcomes -are planes survivingmore?

59. BE RIGOROUSBE RIGOROUS

60. PART 3: APPLYING THEPART 3: APPLYING THE
THEORYTHEORY

61. CORRELATIONCORRELATIONA MEASURE OF THE STRENGTH OF DEPENDENCE BETWEEN TWO VARIABLESA MEASURE OF THE STRENGTH OF DEPENDENCE BETWEEN TWO VARIABLES

62. PEARSON CORRELATIONPEARSON CORRELATION
Err...Justlook itup
(Assumes linear relationship)

63. Range Strength
<0.4 Weak/No Correlation
<0.7 Some Correlation
>0.7 StrongCorrelation

64. CASE STUDY: PERFORMANCE PROBLEM WITH HIGH SYSTEMCASE STUDY: PERFORMANCE PROBLEM WITH HIGH SYSTEM
TIMETIME
Hypothesis: caused byDisk I/O

65. Correlation Strength: 0.78453

66. MACHINE LEARNINGMACHINE LEARNING
Application of statistics to learn arelationship

67. HOW MANY CLUSTERS?HOW MANY CLUSTERS?

68. WHERE'S THE ELBOW?WHERE'S THE ELBOW?

69. FITTINGFITTING

70. FITTINGFITTING

71. SOLUTION:SOLUTION:
CROSS VALIDATIONCROSS VALIDATION

73. CHOOSE CROSS VALIDATION DATA WISELYCHOOSE CROSS VALIDATION DATA WISELY

74. SELF VALIDATINGSELF VALIDATING
Ensemble methods -Train lots of weak classifiers and merge

75. RANDOM FOREST AND BAGGINGRANDOM FOREST AND BAGGING
Divide the datainto bootstrap sets
Use the restfor calculatingerror

76. LEARNING CURVESLEARNING CURVES

79. HOW MUCH IS TOO MUCH?HOW MUCH IS TOO MUCH?

84. MONITOR PRODUCTION DATA...IT CHANGESMONITOR PRODUCTION DATA...IT CHANGES
Does itlook like the same datathatyou learntwith?

85. A/B TEST NEW SYSTEMSA/B TEST NEW SYSTEMS
Satisfaction/Profit/Traffic...

86. COMMON THREADSCOMMON THREADS
Trainingseterrors are misleading
Cross Validation, Production Monitored Values are the ones
thatreallymatter
Visualise and compare these errors

87. CONCLUSIONCONCLUSION
Analytics are increasinglyimportant
Wide varietyof statisticaland practicaltips to getthem right
Have fun and Bestof luck!

88. @johno_oliver
@RichardWarburto
QUESTIONS?QUESTIONS?
http://insightfullogic.com