We are all becoming adherents of the data-driven life. Big data, data-driven decisioning, the quantified self are the buzzwords of today. Arguments not backed by data just do not cut it, whether it is in decision making, a business meeting, or a casual social conversation. The Powerpoint Presentation of Suresh Shankar's Speech at Big Data Asia World 2013, Singapore.

1. Big Data World Asia 2013
Suresh Shankar, Crayon Data
13 September 2013 1
The Paradox of Big Data:
Misery or Magic?

2. 13 September 2013 2
Before we start, a personal perspective
Loyalty is dead
Big Data will flourish

3. 3
We live in the Age of MORE
MORE
MORE
MORE Connected channels
to interact with consumers
Transparency
driven by ubiquity of information
Sources
of influence
Real-time
environment than ever
Personalisation
expected by consumers
Choices
available to consumers

4. 4
Big data is changing
life for businesses…
and consumers
Volume
Velocity
Variety
Data drives the age of more

5. 13 September 2013 5
Reality: Data on anything & everything at our fingertips…
Food & Beverage TravelHealthcare Fitness

6. 13 September 2013 6
Reality: we are all adherents of the “data-driven life”
o MedHelp - One of the largest
forums for health information
o More than 30,000 new personal
tracking projects are started by
users every month
Source: The New York Times, “The Data-Driven Life, By Gary Wolf, April 28, 2010
o Fitbit – a personal activity tracker
that measures the steps you walk,
quality of sleep, calories burned and
other personal fitness metrics.
o Achieve and share targets / goals
o Airbnb – platform for short-term
lodging for guests
o Collects data across 250k listings in
30k cities to personalize stays based
on preferences, social connections,
rental history, reviews etc.

7. 13 September 2013 7
We are now at an inflection point in the big data revolution
Each of us now leaves a trail of digital
exhaust, an infinite stream of phone
records, texts, browser histories, GPS
data and other information, that will
live on forever
Every animate & inanimate object on
earth will soon be generating data,
including our homes, our cars, and YES,
even our bodies.
An average person today processes more data in a single
day than a person in the 1500s did in an entire lifetime
Source: The Human
Face of Big Data

8. 8
...but data can easily
become misery
Decision Paralysis
Post-purchase dissonance
Default to the easiest option
Source: The Paradox of Choice, Barry Schwartz

9. 13 September 2013 9
At work…
The latest
numbers show that…
…let’s look into the
numbers in greater detail,
let’s resolve this difference,
let’s get accurate & up-to-date
numbers next time
Result: No conclusions, postponed
decisions

10. 13 September 2013 10
At home…
74 Formulas
Liquids / Powders
/ Tablets
Cotton / Polyester 20x stains
Plant based
/ Chemical
Premium / Value
Front loading
/ Top Loading
Hot vs. Cold
Source: http://www.goodhousekeeping.com/product-reviews
Over 1,000 possible outcomes…
Result: Buying default product because
there were too many to choose from

11. 13 September 2013 11
At play…
Which team is performing better?
Who will win the match / series?
What do the statistics tell us?
Result: Selective use of facts.
Who is the best player?

12. 13 September 2013 12
We are inundated with
more and more data and we have
less and less time to navigate and
make sense of this data
The Paradox of More

13. 13 September 2013 13
How can we make data magic?
We need a different
mind-set
‘what is the meaning of this
analysis?’
‘can we see some more
data?’
‘what if we analyse it
differently?’
‘…can we run some
more analysis?’
‘…is the data
right?’
‘why did this
happen?’
Traditional mind-set

14. 13 September 2013 14
Big Data can set your mind free
Stop trying to understand why something
happened, and let statistical algorithms
find patterns we cannot detect
Leverage intelligent algorithms to reduce
the number of choices to a relevant few
More data beats better algorithms Look for confidence level, not certainty
in predicting outcomes

15. 13 September 2013 15
Correlation is enough.
We can stop looking for models.
We can analyze the data without
hypotheses about what it might show….
The new availability of huge amounts
of data, along with the statistical tools
to crunch these numbers,
offers a whole new way of
understanding the world.
Chris Anderson,
Wired Magazine
Correlation = Causation
Source: Wired Magazine, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete, Chris Anderson, June 2008

16. 13 September 2013 16
Correlation is enough.
We can stop looking for models.
We can analyze the data without
hypotheses about what it might show….
The new availability of huge amounts
of data, along with the statistical tools
to crunch these numbers,
offers a whole new way of
understanding the world.
Chris Anderson,
Wired Magazine
Correlation Trumps Causation
Source: Wired Magazine, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete, Chris Anderson, June 2008

17. 13 September 2013 17
More data beats better algorithms
In a nutshell, having more data allows the “data
to speak for itself,” instead of relying on
unproven assumptions and weak correlations
More data can reveal non-linear relationships
within a dataset

18. 13 September 2013 18
When people are given a
moderate number of options (4 to 6)
rather than a large number (20 to 30),
they are more likely to make a choice,
are more confident in their decisions,
and are happier with what
they choose.
Sheena Iyengar,
The Art of Choosing
Less = More

19. 13 September 2013 19
The era of probabilistic evidence-based answers is here
§ Extracts facts and understands the relationships in vast
quantities of data
§ Judges each possible answer in real time and decides
which one is most likely the correct answer
§ Maps possible answers to a probabilistic estimate that
the response is correct with a “level of confidence”
confidence

20. 13 September 2013 20
To make big data magic, we need to resolve the
“Paradox of More”
Algorithms that help data
find data
Bigger data sets merging
external & enterprise data
Make data visual and easy
to understand

21. 13 September 2013 21
What if consumer’s decisions were easier?
Enable consumers to make choices based on
discovery, interest & serendipity…
Her
tastemakers /
Friends…?
Who / What
influences her?
Her tastes?
Her past
behaviour?
Her context -
location,
weather,
time…?

22. 13 September 2013 22
What if business decisions were easier?
Mix and utilize vast amounts of internal
& external datasets to find relationships
Derive range of outcomes / possibilities
at various confidence levels
Predict the outcomes of decisions

23. 13 September 2013 23
Example of making big data magic:
A Choice Engine to map & predict the world’s choices

24. Suresh Shankar
suresh@crayondata.com
+65 96639144
24
Thank you