Today, billions of activities and interactions happen online. The level of interactions with online applications are getting more complex as well. Within the UX, we have the opportunity to understand collective behavior and various experiences through big data. Specifically, large scale and strategical directions to products can be determined and evaluated through big data behavioral analysis. In this talk, I will go through various types of research objectives, appropriate methodologies and explain how we can use quantitative methodologies to solve UX and user behavior problems and drive product strategy. In this presentation, I will go through a couple of example case studies and topics where behavioral data can help us better understand users and inform strategic product development. Presented by Saide Bakhshi

1. How to use big data to drive UX
strategy?
1
Saeideh Bakhshi
Quantitative UX Researcher, Facebook
bakhshi@fb.com
UXPA 2018

2. At Georgia Tech College of Computing
Started PhD in CS
2009 2013
Focused on behaviors that drove users'
engagement with photos
Started working at Y! Labs
As a Quantitative UX Researcher
Started working at Facebook
2016
2012
Got interested in HCI and started
working on user engagement
Changed thesis topic
2014
And started to work full time at Yahoo
Labs
Graduated with PhD

3. Agenda
3
1
How does big data fit into UX?
2
Which big data methods are
useful for UX?
3
What are some examples of
using big data methods?
4
How is this different from
what data scientists/analysts
do?

4. How does big data fit into UX?
1
4

5. 5
In 1995,Victor Minichiello
described how qualitative and
quantitative research differ in
concept and methodology.

6. Concept
Qualitative
Concerned with
understanding human
behavior from end user's
perspective
Assumes dynamic and
negotiable reality
Used to gain an
understanding of
reasons, opinions, and
motivations
Less generalizable
Quantitative
Concerned with
understanding behavior
from researcher's
perspective
Assumes a fixed and
measurable reality
Uses measurable data to
formulate facts and
uncover patterns
More generalizable
Minichiello, 1995
Method
Qualitative
Data is collected
through participant
observation or
interviews
Data is presented
through the language
of the end user
More in-depth
information on a few
cases
primary inductive
process used to
formulate theory or
hypotheses
Quantitative
Data is collected
through measuring
things
Data is presented
through the language
of statistical analysis
Less in-depth but more
breadth of information
across a large number
of cases
Primary deductive
process used to test
pre-specified concepts,
constructs, and
hypotheses that make
There's been a lot of progress
made in quantitative research
methods since 1995

7. Advantages of using big data
7
1Actions may tell a different
story than words. 2We can find new patterns that
were previously hidden.
3
Technology use cases are
getting more complex and
harder to find patterns. 4A lot of this data is already
available.

8. Big data methods can be useful at various stages of user
research
StrategicFoundational
Predictive and Hypothesis
driven
Strategic research can specially
benefit from big data methods
where we need to test
hypotheses about behaviors or
predict trends.
Exploratory & Generative
Variety of big data methods (e.g.
unsupervised methods) can help
us explore behaviors and
attitudes associated with product.
Characterize use and
measure success
After a product is launched, big
data methods can help us
characterize behaviors, measure
success and connect the two.
Evaluative
8

9. 9
Theorize/Predict what's
next: We can use past
behavior to predict future
behavior and strategize
products around that.
Characterize
behaviors and
estimate success
Use unsupervised
methods to find
patterns and trends Conceptual model
around behavior: Based
on findings, we can come up
with theories on how/why it
is happening
Test
hypotheses
Analysis: We can analyze
large scale behavioral data to
find patterns around the product
or topic we are interested in
How can behavioral data help us with strategy?

10. What are some useful machine learning
methods we can use with large data?
2
10

11. 11
Problem type Big data methods Types of data
Exploratory
exploratory data analysis, unsupervised
methods (clustering, assoication rules),
text analysis
behavioral or user generated data or self
reported data
Deep dive in an existing behavior
supervised methods (regression,
classification)
behavioral + attitudinal
Hypothesis validation
Hypothesis testing, supervised methods
(regression, classification)
behavioral + attidunal
Trends forecasting
Timeserie analysis, Forecasting,
prediction
behavioral

12. Data Preparation,
exploration and
reduction: Collection,
Cleaning and preparing
data, bivariate analysis,
visualization,
dimensionality reduction
Prediction: Linear
regression, K-nearest
neighbors, Regression trees,
Neural networks, Ensembles
Time-series analysis:
Regression based,
Smoothing methods
Theorize/Predict
what's next: Sometimes,
we can use past behavior
to predict future behavior
and strategize around that
Classification: K-nearest
neighbor, Naive-bayes,
Classification trees, Logistic
regression, Neural networks,
Discriminant analysis, Ensembles
What goes together:
Association rules,
collaborative filtering
Segmentation:
Clustering
Model evaluation
and selection:
accuracy and
performance evaluations
Descriptive analysis:
bivariate and
multivariate analysis,
Data visualization
Insights: Gather insights
How to use machine learning in UX to drive strategy?

13. Two Examples of using big data methods
3
13

14. Are the social signals of a review indicative of the
polarity of the text?
How do users use various voting options on
Yelp reviews?
Are the social signals of a review indicative of how much
the rating differs from the mean rating for that business?
14

15. 15
We obtained data that was shared by Yelp.
11,537 businesses
43,873 reviewers
229,907 reviews
It contained, data about businesses in Phoenix,AZ area.
Yelp data

16. We created the response variable to indicate the difference between
review's rating and the average rating
Business's avg rating Review's rating
Funny, cool and useful
votes
Different set of variables we considered
16

17. 17

18. Take-aways
18
1
Funny votes imply lower
ratings and higher negativity
in the tone. 2
Useful votes imply lower
ratings and lower positivity in
the tone.
3
Cool votes imply higher
ratings and higher positivity
in the tones. 4
Community signals carry
implied meaning beyond
their specific labels.
We used a big data model to better understand how our users use the features that are available to them on the site.

19. Connection: What is the structure of social connection
on Pinterest? For example, are women or men more
connected?
Comparison: How does behavior on Pinterest compare
to behavior on other social network sites? Letting Twitter
be our point of reference, do Pinterest and Twitter users
systematically differ in what they talk about?
What is Pinterest and what do people do
there?
Activity: What drives user activity on Pinterest: what
about a pin—and its pinner—“grabs the attention” of other
users? For example, what role does gender play? Do pins
from women receive more, less, or equal attention than
those from men?
19

20. Data collection process
20
steps taken in this work to collect and prepare data for modeling

21. Pins
21
Pin is a unit of analysis for content on Pinterest
• We collected data around pins:
• repins: how many times this pin was repinned
• likes: how many other users liked this pin
• comments: textual comments included by the pin’s
pinner
• text: the pin’s description and comments by others

22. Users on Pinterest are called pinners.They can create
boards and pin images to their boards.
Pinners
22
Aenean eu leo quam Vehicula Etiam sed diam eget risus varius
blandit sit amet non magna.
• We collected data around pinners:
• followers: how many pinners follow this pinner
• follows: how many users this pinner follows
• boards: how many boards this pinner has created
• pins: how many pins this pinner has created

23. we also collected additional data features that could
help us understand Pinterest activities better.
Additional data
23
• location: a free-text description of a pinner’s location,
via either Pinterest or Facebook; if they specified it in
both places, we used the Pinterest location
• gender: a pinner’s self-reported gender via their
Facebook profile page
• tweet text: via their linked Twitter handle, the text of
all tweets written by this pinner in the last year

24. Take-aways
24
1Being American or British
earns repins. 2Being female means fewer
followers.
3Pinterest’s verbs: use, look,
want, need. 4Twitter: lol, watching now.

25. Geographic distribution of users we used in the analysis
25

26. We run a negative binomial regression with number
of followers as a dependent variable.
Understanding follows
26
Main findings:
Gender: being female is associated with lower number
of followers
Country: Being American or British earns repins.While
we were unable to analyze every country in our dataset
because of sparsity, we do find that pinners from the
United States and the United Kingdom attract more
repins than the rest of the world.

27. We plot the distributions of followers for each
gender:
Gender follow up analysis
27
while the median male follower
count is lower than the female median follower
count (67 vs. 86), the mean male follower count is
substantially higher than the mean female follower
count (1,063 vs.270).

28. What differentiates posts on Pinterest from the posts on Twitter?
28
Pinterest’s verbs: use, look, want, need.

29. What differentiates posts on Pinterest from the posts on Twitter?
29

30. What differentiates posts on Pinterest from the posts on Twitter?
30
Twitter: lol, watching now.

31. How is this different from Data Science/
Analytics?
4
31

32. Basic unit is interactionBasic unit is task Basic unit is action
32
Qualitative methods DS big data methodsUX big data methods
Large dataBasic unit is task Large data
Data from replication of real
world usage
Data from real world usage Data from real world usage
Human centric Human centric Business centric
Answers why
Answers what, sometimes
why
Answers what

33. Footercontent20pts.Loremipsumdolorsitamet,erosverteremsalutatusnamte.Viscutamquamconclusionemque.
With Large scale behavioral
analysis we can be both
human centric and business
centric.
We can connect people's
needs and behaviors to
business metrics and propose
new metrics using behavior.
Small scale
Large scale
Business centric Human centric
Qualitative
UX methods
Data Science
Methods Large-Scale
Behavioral
Methods
Survey
Methods
Large scale machine learning methods can help us report business
metrics based on actual behavior

34. Accuracy
Explainability
UX models are often
intended to be
descriptive
Business models often
aim to be predictive
Explainability vs accuracy trade-off

35. 1 2
Make sure to learn and understand how data
is collected and what are the limitations of it.
3 4 Having access to big data and being able to
analyze that data will not do you a bit of good
if you are unable to translate those efforts into
successful actions.
Keep it simple: that any modeling exercise
has inherent risk.Although advanced
statistical methods indisputably make for
better models, complex models are often not
practical.
Big data can't answer all types of research questions
Start with a hypothesis
Complexity is not
always better
Understand your data
Turn insights into
actions
It's easy to get lost with big data. Often
projects are exploratory so there is always a
need to approach these projects with
established hypotheses.
Things to remember
35

36. Be ready to triangulate with
qualitative methods
36
Algorithms are not as good as Humans in telling
the full story

37. How to use big data to drive UX
strategy?
37
Saeideh Bakhshi
Quantitative UX Researcher, Facebook
bakhshi@fb.com
UXPA 2018