Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Observe Changes of Taiwan Big Data Communities with Small Data

用小數據看
台灣大數據生態系的
社群流轉
Observe Changes of Taiwan Big Data Communities with Small Data
Shared at 2018-06-13 < 交通大學數據科學與工程研究所 Seminar >

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Observe Changes of Taiwan Big Data Communities with Small Data

  1. 1. 2018-06-13 ( ) 15:30~16:20 1001 Initiator and Chair, TDEA Data Architect, TenMax AVP, Product Management, Etu Associate Researcher, NCHC Observe Changes of Taiwan Big Data Communities with Small Data Small Data , 2013 2018 Indeed ( D i g i t a l Transformation) Social Network Analysis
  2. 2. My Journey of “Innovation” 
 ( aka “From Zero to One” ) 用小數據看 台灣大數據生態系的 社群流轉 Observe Changes of Taiwan Big Data Communities with Small Data Jazz Yao-Tsung Wang Initiator and Chair, TDEA Data Architect, TenMax Shared at 2018-06-13 < Seminar >
  3. 3. Hello! I am Jazz Wang Co-Founder of Hadoop.TW Initiator and Chair of Taiwan Data Engineering Association (TDEA) Hadoop Evangelist since 2008. Open Source Promoter. System Admin (Ops). - 11 years (2002/08 ~ 2014/02) Associate Researcher in HPC field. - 2 years (2014/03 ~ 2016/04) Assistant Vice President (AVP), 
 Product Management of ‘Big Data Platform Management’ - 2 years (2016/04 ~ Now) Data Architect of Real-Time Bidding You can find me at @jazzwang_tw or
 https://fb.com/groups/dataengineering.tw 
 https://slideshare.net/jazzwang 3
  4. 4. 4 Hadoop ~ ~ 2006 2 2008 4 ~$ whois hadoop.tw Domain Name: hadoop.tw Record created on 2008-04-28 (YYYY-MM-DD)
  5. 5. 5 ~ - Data Engineer – Vendor Site – – – - Data Scientist – User Site – – –
  6. 6. 6 
 
 
 ● ● ● ● 
 
 HR 
 Do you believe in “Data-Driven” ??
  7. 7. 7 
 / Q1 - - ? ? ? ? ? Q2 - (Dimensions) (Measures) - ? ? ? - ? ? ?
  8. 8. 8 Hadoop • Indeed Hadoop • • 2013/02/07~2018/06/11 http://www.indeed.com/jobtrends/q-Hadoop.html http://tw.indeed.com/jobs?q=Hadoop&l=Taiwan
  9. 9. 9 Indeed • • — • 1998 :) • PS. / ( ) • Unix, Linux DOS AI/DO Linux … https://plus.nctu.edu.tw/courses/22241
  10. 10. 10 Hadoop • 1,255 ( Gmail mbox Python • 5,647 • 500 (
  11. 11. - Indeed.mbox 11https://takeout.google.com/settings/takeout/custom/gmail
  12. 12. mbox Python 12https://github.com/jazzwang/show-mbox Binary File Git Repo Indeed.mbox CSV Git Repo
  13. 13. (1) 80% 13 Job Company 1. 2016-11-17 E-mail HTML 2.2016-11-17 CSS class=“sg-paragraph-large” class=“sg-paragraph-large db” case 3. 2016-11-17 HTML Tag “n” “ ”
  14. 14. (2) 80% 14 Job Location 1. 2016-11-17 E-mail HTML 2.2016-11-17 CSS class=“sg-paragraph-large db” 3. 2016-11-17 HTML Tag “n” “ ” 4. 2016-11-17 HTML Tag ( - )
  15. 15. (3) 80% 15 Job Title 1. 2016-11-17 E-mail HTML 2. 2016-11-17 HTML Tag ‘ “> ‘ 3. 2016-11-17 HTML Tag “n” “ ”
  16. 16. Tableau 16 Hadoop E-mail PS. URL ( ) 2017 11 2016 11 2018 Hadoop
  17. 17. 17
  18. 18. 18
  19. 19. 19
  20. 20. https://public.tableau.com/profile/jazz.wang#!/vizhome/Hadoop_Job_Taiwan_2018-06/Dashboard1 20 197 2018
  21. 21. 2016 12 21
  22. 22. 22
  23. 23. ▷ Top 15 ○ ■ → → / 
 → → → (?) ▷ Top 15 ○ Job Title URL ? ○ Job Title URL 
 
 → / 23
  24. 24. Tableau Public ▷ Mac OS brew cask install tableau-public 24https://public.tableau.com/profile/jazz.wang#!/vizhome/Hadoop_Job_Taiwan_2018-06/Dashboard1
  25. 25. Google Data Studio 25 https://datastudio.google.com
  26. 26. ## Part 1 : Indeed Job Analytics 1. http://blog.chunnorris.cc/2015/04/python-2x-unicode.html 2. https://stackoverflow.com/questions/3806874/how-to-merge-two-files-line- by-line-in-bash 3. https://stackoverflow.com/questions/466345/converting-string-into- datetime 4. https://stackoverflow.com/questions/19351541/excluding-unwanted- results-of-findall-using-beautifulsoup 5. https://pythonadventures.wordpress.com/2010/10/11/chomp-functionality- in-python/ 6. https://www.cnblogs.com/pylemon/archive/2011/05/18/2050179.html 26
  27. 27. 27 
 / Q1 - - ? ? ? ? ? Q2 - (Dimensions) (Measures) - ? ? ? - ? ? ?
  28. 28. “ LinkedIn Profile 28 (HR 4.0?)
  29. 29. 29 Hadoop • Cloudera (CCDH/CCAH) • LinkedIn • • 2015-05-12 LinkedIn API • LinkedIn • • 62 Cloudera LinkedIn
  30. 30. 30 LinkedIn Advanced People Search • 3837 • Cloudera (119) Apple ( 79)
 Cognizant (68) eBay (55)
 Hortonworks (47) • • IT
  31. 31. 31 Hadoop • Hadoop • (2012) → / / / → / / (2015) • (SI)
  32. 32. { } 32 解讀雲端大數據新趨勢 Big Data Stack on The Cloud Shared at 2018-05-16 < iThome Cloud Summit 2018 >
  33. 33. 33
  34. 34. 34
  35. 35. — 35 https://www.facebook.com/groups/dataengineering.tw https://www.facebook.com/groups/datasci.tw
  36. 36. Facebook Graph API 2018 36 https://github.com/HadoopTW/communities-dashboard Facebook Graph API 2.10 3.0 : Selenium
  37. 37. 37 Hadoop.TW • • Hadoop.TW 5,534 (@2018-06-13) • Facebook Graph API • HTML
  38. 38. Selenium Python Code 38 https://github.com/jazzwang/snippet/tree/master/selenium_lab
  39. 39. 39 Hadoop.TW • 2013 SNA • http://www.slideshare.net/jazzwang/ss-50602379 • • (post) (comment) (like) •
  40. 40. ▷ Selenium Facebook member ○ “ (group_name)”, “ (user_id)”, “ (join_time)” ▷ https://github.com/HadoopTW/communities-dashboard Gephi Social Network Analytics ○ https://gephi.org/ ▷ member ▷ D3.js Sankey Diagram ○ https://d3js.org/ ○ http://benlogan1981.github.io/VerticalSankey/UBS.html 40
  41. 41. ## Part 2 : Hadoop.TW member Analytics 1. https://www.youtube.com/watch?v=49Mwqbu2cMo 2. https://github.com/mozilla/geckodriver/releases 41
  42. 42. ▷ ( ) ○ - ○ - ○ - ▷ ( ) ○ Python / Shell Script ■ Data Science at the Command Line
 https://www.datascienceatthecommandline.com/ ○ Crawler - Ex. Selenium, Scrapy ○ Data Visualization - Ex. Tableau, Google Data Studio, D3.js ▷ - ○ Intern 
 ○ Kaggle ■ Ex. Streaming Data Input → Data Quality / Data Governance 
 → Datawarehouse → BI Report 42
  43. 43. 43 https://goo.gl/2z9BGK
  44. 44. Thanks! Any questions? You can find me at @jazzwang_tw or
 https://fb.com/groups/dataengineering.tw 
 https://slideshare.net/jazzwang 44

×