Presented at http://2015.geecon.org/

1. Analysing GitHub commits with R
Barbara Fusinska
@BasiaFusinska
barbarafusinska.com

2. About me
Programmer
Manager
Sweet tooth

3. Goals
Analysing
GitHub data
Methods of data
exploration & analysis
Learn R

4. Agenda
• Data analysis
• R basics
• Data exploration & processing
• Big Data
• Outside GitHub:
– Data visualisation
– Libraries & tools

5. Data analysis process
Raw Data
Processed
Data
Data Analysis
& Visualization
Exploratory
Data Analysis

6. What do we want to find out?

7. GitHut Visualisation
http://githut.info/

8. GitHub Archive
https://www.githubarchive.org/

9. Google BigQuery
https://cloud.google.com/bigquery/what-is-bigquery

10. GitHub API
https://developer.github.com/v3/

11. DEMO: R basics
• RStudio
• Data types
• Data filtering

14. DEMO: Active repositiories
• What is an active repository?
• Where can we get data from?
• How to get useful information?
• How to handle missing data?

15. GitHub Archive CreateEvent

16. GitHub Archive PushEvent

17. GitHub Archive PullRequestEvent

18. Expected result

19. Read Pull Requests

20. Summary Output

21. Unique results

22. Language information

23. Missing information

24. Filter missing

25. Plotting

26. Languages

27. DEMO: Processing various data
sources
• Active repositories – from Create, Push and
PullRequest events
• Missing language information:
– Google BigQuery
– GitHub API

28. Expected result

29. Google BigQuery

30. GitHub API from R

31. GitHub API Search

32. Combining data

33. Sorted data

34. Languages

35. Big Data in R
• What’s _Big_Data_ anyway?
• R processes data in memory
• Bring down only the data
you need
• Streaming the data from
database

36. R, Hadoop, and How They Work
Together
• Hadoop Streaming – utilities available as R scripts
• ORCH (Oracle R Connector for Hadoop) - provides access to a
Hadoop cluster from R
• RHIPE (R and Hadoop Integrated Programming Environment)
– techniques designed for analysing large sets of data
• Rhadoop - R packages that allow users to manage and analyse
data with Hadoop
https://blog.udemy.com/r-hadoop/

37. Azure Machine Learning Studio
http://channel9.msdn.com/Blogs/Windows-Azure/R-in-Azure-ML-Studio

38. DEMO: Processing multiple files
• Reading files line by line
• Reading multiple files
• Saving files

39. Analysing week

40. Read multiple files line by line

41. Push Events during the week

42. DEMO: Plotting in R
http://www.harding.edu/fmccown/r/

43. Qplot - dots

44. qqplot - lines

45. qqplot - bars

46. Libraries & Tools
• DataScience
• JSON processing
• Http requests
• Summarising
• Plotting

47. DEMO: Downsampling
https://github.com/javiljoen/LTTB

48. DEMO: Anomalies detection
https://github.com/twitter/AnomalyDetection

49. To summarize…
• GitHub Archive, GitHub API
• R language – RStudio, types, data exploration,
I/O operations, etc.
• Big Data in R
• 3rd party & build-in libraries
• Visualization

50. Thank you
Barbara Fusinska
@BasiaFusinska
barbarafusinska.com
https://github.com/BasiaFusinska/RTalk