Presentation from Aaron Kimball, WibiData #dataconf More at http://event.gigaom.com/structuredata/

1. ANALYZING LARGE-SCALE USER DATA
SPEAKER: Aaron Kimball
CTO
WibiData
Friday, July 27, 2012

2. Friday, July 27, 2012

3. Analyzing
Large-­‐Scale
User
Data
with
Hadoop
and
HBase
Aaron
Kimball
–
CTO
WibiData,
Friday, July 27, 2012

4. We
can
now
collect
more
data
than
at
any
Dme
in
history.
Friday, July 27, 2012

5. Yesterday’s
engineering
challenge:
FiJng
the
problem
into
the
hardware.
Friday, July 27, 2012

6. Today’s
constrained
resource
is
understanding.
Friday, July 27, 2012

7. How
do
we
best
apply
data
…to
beMer
serving
our
Friday, July 27, 2012

8. The
best
products
are
user-­‐
• IntuiDve
UI
• ConDnuously
learning
– Guided
search
– Smarter
recommenda1ons
• More
effec1ve
service
Friday, July 27, 2012

9. What
are
we
building
Friday, July 27, 2012

10. What
are
we
building
Friday, July 27, 2012

11. What
are
we
building
Friday, July 27, 2012

12. What
are
we
building
Friday, July 27, 2012

13. What
are
we
building
Friday, July 27, 2012

14. What
are
we
building
Friday, July 27, 2012

15. What
are
we
building
Friday, July 27, 2012

16. What
are
we
building
Friday, July 27, 2012

17. What
are
we
building
Friday, July 27, 2012

18. What
are
we
building
Friday, July 27, 2012

19. Requirements
1.
Understand
the
user
populaDon
Friday, July 27, 2012

20. Requirements
2.
Respond
to
users
in
real
Dme
Friday, July 27, 2012

21. Requirements
3.
Support
graceful
data
evoluDon
Friday, July 27, 2012

22. Large-­‐scale
data
science
is
• What
does
a
user
look
like?
– What
data
is
available
about
the
user?
– Which
features
are
important?
– Which
features
are
correlated?
• How
do
I
model
this
in
MapReduce?
• How
do
I
serve
results
in
a
Dmely
Friday, July 27, 2012

23. Friday, July 27, 2012

24. Tools
of
the
trade
• Store
all
data
about
a
user
in
one
place
• Support
real-­‐Dme
get/put,
as
well
as
MapReduce
Friday, July 27, 2012

25. Tools
of
the
trade
• Use
complex
data
types
to
model
complex
data
• Support
extended
data
models
over
Dme
Friday, July 27, 2012

26. Tools
of
the
trade
• Abstract
computaDonal
model
away
from
MapReduce
• Support
computaDon
over
all
users…
or
one
user
at
a
Dme
Friday, July 27, 2012

27. Friday, July 27, 2012

28. Friday, July 27, 2012

29. Friday, July 27, 2012

30. Friday, July 27, 2012

31. Friday, July 27, 2012

32. Friday, July 27, 2012

33. Friday, July 27, 2012

34.
:
for
set-­‐top
boxes
Viewing/recording
history
Friday, July 27, 2012

35.
:
for
set-­‐top
boxes
Viewing/recording
history
Friday, July 27, 2012

36.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Friday, July 27, 2012

37.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Friday, July 27, 2012

38.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Analysis
for
product
roadmap
Friday, July 27, 2012

39.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Analysis
for
product
roadmap
Friday, July 27, 2012

40.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Analysis
for
product
Tech
support
roadmap portal
Friday, July 27, 2012

41.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Analysis
for
product
Tech
support
roadmap portal
Friday, July 27, 2012

42.
:
for
set-­‐top
boxes
Libraries
Device
and
User
Analysis
Viewing/recording
history
Personalized
offers
and
recommenda=ons
Improve
Analysis
for
d
reports
product
Tech
support
for
roadmap portal
Friday, July 27, 2012 adver=se

43. The
future
• More
personalizaDon
• AdapDve
UIs
(self
arranging
dashboards)
• Targeted
content,
ads
• More
effecDve
customer
service
Friday, July 27, 2012

44. Conclusions
• ApplicaDons
are
becoming
increasingly
user-­‐centric
• Data
drives
this
capability,
but
harnessing
it
requires
a
new
distributed
architecture
Friday, July 27, 2012

45. www.wibidata.com
/
Aaron
Kimball
–
aaron@wibidata.com
Friday, July 27, 2012

46. Friday, July 27, 2012