You love python. You love Data Science. But the size of your data set keeps crashing your code. Is it time to bring in big data tools or simply code smarter? Lee is going to show you efficiency hacks, drawn from top Kaggle competitors, to get python to work on large data sets. Skip the hassle of creating a Big Data infrastructure. Let’s find out how far we can push our home laptop first.

1. Doing Your First
Kaggle - Python for
Big Data Sets
Lee Trawick, ASA
#datapopup

2. What We’ll Cover

3. About me.

4. How is this relevant?

5. Tell me again how this is relevant?

6. Pokemon vs Big Data
Name! That! Pokemon!
Hadoop
Cloudera
Bulbasour
Mongo
Talend
Qubole
Statwing
Magikarp
Flink
Impala

7. It’s easy to submit to Kaggle.

8. Download someone’s code and submit.

9. We’re on the leaderboard!

10. About the data.

11. My approach.

12. First, I load the data.

13. How they load the data.

14. Why be memory-conscious?

15. Other ways to save memory
Delete objects when you’re done with them
Specifying data types on every new column. Not just for loading files

16. My approach to the “One Vector Problem”

17. Kaggle solution to the “One Vector Problem”

18. The map() trick to joining large tables

19. Kaggle Solution - 3 Key Things

20. Gameshow…Will! That! Work?!
Looping through big tables

21. Challenge 1

22. Challenge 2

23. Challenge 3

24. It’s the same code

25. That’s All Folks.