這是 Data Science 課程報告的投影片 內容再分享一些 Data Science 會使用的 Tool demo code: https://github.com/JIElite/ikdd-course-presentation

1. Data Science Toolchain
presented by Jie-Han Chen
slide: https://goo.gl/1hXBGk

2. Language & Software
Python
R
Java
Matlab
Octave
Jupyter Notebook

3. Python
Open Source Community
Package
Web Service
Good Readability
Machine Learning

4. R
Open Source Community
Built-in Statistics Package
Standalone computing &
data analysis
Slower than Python

5. High Performance
Big Data
Poor Visualization,
Modeling
Java

6. Matlab & Octave
Powerful built-in math functions
Simple Data Visualization tool
Prototyping
-5
0
5
10
-10 -10
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
-5
0
5
10

7. Jupyter Notebook
Support 40+ programming language.
eg: Python, R, Scala...
Excellent for sharing your experiments
Markdown, Latex
example1
example2

8. Language & Software
Python
R
Java
Matlab
Octave
Jupyter Notebook

9. Data Science Roadmap

10. Data Science Toolchains
Data Collection
Data Visualization
Data Storage
Algorithm & Modeling

12. Data Collection
Using API: Facebook, Wikipedia
Web Scraper

13. Web Scraper

14. Web Scraper
HTTP request + HTML Parser

15. HTTP: python-requests
Better than built-in urllib
Sessions with Cookie Persistence
Thread-safety

16. HTTP: python-requests

17. HTTP: python-requests

18. Web page

19. Parser!
Regular Expression?

20. BeautifulSoup
HTML/XML parser

21. BeautifulSoup
Ptt

22. HTML parser

23. More Powerful Tool?

24. Scrapy
An open source and collaborative framework for
extracting the data you need from websites.In a
fast, simple, yet extensible way.

25. Scrapy
$ scrapy startproject tutorial

26. Scrapy

27. path: /scrapy/dmoz.py
crawler name: dmoz
Scrapy

28. Scrapy

29. $ scrapy crawl dmoz
Scrapy

31. robots.txt

32. youtube.com/robots.txt

33. "I believe that visualization is one of the most
powerful means of achieving personal goals."
Harvey Mackay

34. Data Visualization

35. Data Visualization
Matplotlib, ggplot2
D3.js
Bokeh
Tableau
PlotDB
Leaflet

36. Matplotlib

37. ggplot2

38. D3.js
Data Visualization Project
Interacive
Web frontend
example1
example2

40. Bokeh
Python, R, Scala, Julia
Interactive
Jupyter Notebook

41. Tableau

42. Tableau
( )
code
Data Source

44. Data Visulization
Code

47. Programming

48. Using GeoJSON with Leaflet
, Configurable

49. Using GeoJSON with Leaflet

57. S3
1. Key-value
2. Permission
3. Data Visualization
4. Big Data (Spark)

58. Algorithm
&
Modeling

59. Algorithm & Modeling
python-numpy + python-pandas + scikit-learn
libsvm
spark-Mlib
Weka
Deep Learning

60. Numpy + Pandas
+ Scikit-learn

61. Numpy
C

62. Numpy - data structure
ndarray (n-dim array)
ndim
size
shape
dtype

63. Numpy
generate matrix

64. Numpy
generate matrix

65. Numpy
generate matrix

66. Numpy
generate matrix

67. Numpy
generate matrix

68. Numpy - linalg

69. numpy
Series, DataFrame
: csv, json ...
nan

70. Series -

71. Series -

72. Series -

73. Series -

74. DataFrame - Series

75. Pandas - import

76. Pandas - import

77. Pandas - import

78. Pandas - import

79. Pandas - NaN

80. Pandas - NaN

81. Pandas - NaN

82. Pandas - operation
Merge
Grouping
Reshaping
. . .

84. Dataset
Feature Engineering
Modeling
Evaluation

86. LIBSVM
C
Easy to use
Support many programming languages
Dataset

87. LIBSVM - install
$ git clone

88. LIBSVM - install
$ make

89. LIBSVM - workflow

90. LIBSVM - data format
label
index , attribute
value , attribute

91. LIBSVM - data format

92. LIBSVM - toy

94. MLlib

95. MLlib
, Hadoop
Java, Scala, R, Python

96. MLlib

97. MLlib
, Hadoop
Java, Scala, R, Python

98. Classification: logistic regression, naive Bayes,...
Regression: generalized linear regression, survival regression,...
Decision trees, random forests, and gradient-boosted trees
Recommendation: alternating least squares (ALS)
Clustering: K-means, Gaussian mixtures (GMMs),...
Topic modeling: latent Dirichlet allocation (LDA)
Frequent itemsets, association rules, and sequential pattern
mining
MLlib

99. Feature transformations: standardization,
normalization, hashing,...
ML Pipeline construction
Model evaluation and hyper-parameter tuning
ML persistence: saving and loading models and
Pipelines
MLlib

100. MLlib
, Hadoop
Java, Scala, R, Python

101. MLlib

103. Weka
Java library
Big Data
Support GUI

104. Deep Learning
Theano
Pylearn2
Keras
Tensorflow
Caffe
Deeplearning4J
...

105. Theano
Base on Numpy
Implemented by Cython
Dynamic C code generation
GPU & CUDA
tensor, math expression
A CPU and GPU Math Compiler in
Python
Theano tutorial:
http://www.slideshare.net/SergiiGavrylov/theano-tutorial

106. Keras
Theano, Tensorflow
Support GPU
prototype
High-level neural networks library

112. Tool ?

114. Homework
Github repo Data science
Database, Social Network Analytics, ML library, Deep
Learning Platform ...
READM.md: Repo
Demo Code
email: ita3051@gmail.com
Google
https://goo.gl/forms/PQPz8u2glyunQvfM2​