Presented at PyCon 2017 Data science introductory courses might give you the impression that dealing with data is neat, tidy, and simple. They present you with a simplistic dataset and the scikit-learn or Pandas documentation, and a day or so later, you're done! Piece of cake, right? The real world of data isn't that easy! As a data scientist who has worked in the industry for several years, I have had a lot of experience dealing with messy, inaccurate, incomplete data, and I want to share those experiences with you. I'll talk my way through three real-world situations where I've had to analyze and build models on untidy and complex data, going through how I've preprocessed the data and prepared it for modeling. You'll leave with an understanding of how a data scientist thinks about data and what she does when the data is complicated.

1. The Wild West of Data
Wrangling
Sarah Guido
PyCon 2017
@sarah_guido

2. This talk:
• A day in the life
• Three examples of dealing with uncooperative data
• Not ground truth!

3. Who am I?
• Senior data scientist at Mashable
• Mashable == internet culture media!
• Data sciencing in Python
• Twitter: @sarah_guido

4. Iris Dataset

5. Iris Dataset

8. Example 1: Predicting building sales
• The problem: can we predict if a building will sell the
following year?
• The data: floors, location, square footage, price per sqft,
etc
• The goal: provide valuable insight to platform users

9. Example 1: Predicting building sales
• First thought: logistic regression using scikit-learn
• Binary classification: sale/no sale

10. Problem…
Data: 95% no sale, 5% sale
Logistic regression: 95% accurate
DONE!

12. Problem: Class imbalance
Class imbalance
When the values you are trying to predict are not equal, this
can create bias in classification models.

13. Solution: Gradient boosting
Gradient boosting
Produces a prediction model in the form of an ensemble of
weak prediction models, typically decision trees.

14. Example 2: Clustering user interactions
The problem: how can we identify similar patterns based on
click data?
The data: time, geolocation, cookie, browser useragent
string, referrer
The goal: understand how people interact with content over
time

15. Why Scala?

16. Problem: Clustering user interactions
K-means clustering
An unsupervised learning method of grouping data together
based on a distance metric.

17. Problem: Clustering the data
• Only look at users with 5 or more interactions
• Each user has a different number of interactions
• Each data point ends up in a different cluster

22. Solution: Transform the data

23. Solution: Transform the data
date: 2017-04-09, 2017-04-13, 2017-04-30, 2017-05-01,
2017-05-12
Length of interactions: 5
Average time between interactions: ~8 days

24. Solution: Transform the data
referrer: facebook, twitter
One-hot encode and transform to matrix
• Facebook: [1, 0]
• Twitter: [0, 1]

25. Solution: Transform the data

26. Example 3: Understand audience composition
The problem: how can we effectively describe our audience?
The data: anonymized demographic and psychographic data
The goal: audience segmentation and channel analysis

27. Problem: insufficient data
• Google Analytics data – 1/3 of urls
• Finicky API
• Semi-useless psychographic data

28. Solution: accept defeat

29. Solution: accept defeat make it work!

30. Solution: make it work!
• Theory of highly-performant links
• Segmentation through archetypal analysis
• Go get more data!

31. General strategy
• What problem are you trying to solve?
• What’s wrong with your data?
• What do you need that you don’t have?

32. Keep in mind…
• Data your company collects is complicated
• What you do to your data will affect the model
• Creativity is your friend
• Lots of ways to solve the problem

34. Thank you!
@sarah_guido