This document provides an overview and introduction to big data. It discusses the technical challenges of big data including issues of volume, variety, velocity and veracity. It also discusses solutions like Hadoop, MapReduce, and big data databases. Additionally, it covers big data analytics including different levels of analytics maturity and techniques like data mining, machine learning, and predictive analytics. Finally, it provides resources for learning more about big data including online courses, sandbox environments, open source tools, and public datasets.

1. Big Data, First Steps
First Steps on Big Data
Alexandre Simundi
Apr 2016

2. Agenda
• What’s all this fuzz about?
• Technical Challenges & Solutions
• Big Data Analytics
• Where to find knowledge?

3. Software Engineer / Solutions Architect
Over 11 years of experience in the IT market
Passionate about distributed architecture and
big data technologies
Linkedin: https://cl.linkedin.com/in/simundi
Contact: simundi@bdatalabs.com
Twitter: @simundi
Alexandre
Simundi

5. 1890s motion
picture invented

6. New experience through multiplication of data

7. New industry
New profession
New _ _ _ _ path

9. The real revolution is not in the machine
that calculates data but in the data itself
and how we use it
Big Data: a revolution that will transform how we live, work and think

10. 5%

11. 95% of data is unstructured
Name Country Football team Favorite Food
Alexandre Brazil Grêmio Barbeque
Structured
Unstructured
Hi, my name is Alexandre and
I’m from brazil. I support the
best football team in the world:
Grêmo and my favorite food is
barbeque.

12. If you can’t analyze all your data, you are blind to its
opportunities

13. Volume Variety
Velocity Veracity
Value

14. Source: http://www.halogensoftware.com/uploads/blog/whats-your-credit-score-using-predictive-analytics-with-hr-
data/_thumb/848x450/whats-your-credit-score-using-predictive-analytics-with-hr-data.png
Datafication

15. N = all

16. Farecast recorded 200 billion flight
information to predict the best
moment to buy a ticket.
accuracy: 75%
average saving: $50

17. Correlation
shows what, not why.
knowing what is often good enough.

18. Big Data, First Steps
Technical Challenges & Solutions

19. MapReduce Paper
http://research.google.com/archive/mapreduce-osdi04-slides/index.html

20. Mapreduce necessity:
Reduce cost of distributed
computation

21. • Distributed file system HDFS
• Computation framework (Map Reduce)
• Resource Management (Yarn)

22. Hadoop Ecosystem
Master
Worker
result
data
data
data
data
data
data
data

23. Big Data Databases

25. Big Data, First Steps
Big Data Analytics

26. “We are drowning in data, but
starving for knowledge!”
(John Naisbitt, 1982)

27. Data Mining
Combination of AI and statistical analysis to discover
information that is “hidden” in the data

28. Data Analysis Insights Decision Action
Business Value

29. Analytics Maturity Levels

31. Machine Learning
• Statistics + AI
• Predictive Methods
–Use some variables to
predict some unknown or
future values of other
variables
• Descriptive Methods
–Find human –
interpretable patterns
that describe the data

32. Supervised vs Unsupervised
• Supervised
– Learning in a presence of an expert/teacher
– Training data set is labeled with a class value
– Goal: Predict a class or value label
Unsupervised
– No knowledge of the output class/value
– Data is NOT labeled
– Goal: learn patterns/groupings

33. Cross Industry Standard Process for
Data Mining

34. Big Data, First Steps
Where to find knowledge?

36. https://www.coursera.org/specializations/big-data
Content:
• Introduction to Big Data
• Hadoop Platform and Application Framework
• Introduction to Big Data Analytics
• Machine Learning With Big Data
• Graph Analytics for Big Data
• Big Data - Capstone Project

37. Things to learn in the Hadoop Zoo
• HDFS
• Map Reduce
• Apache Pig
• Apache Hive
• Spark

39. http://www.cloudera.com/downloads/quickstart_vms/5-5.html

40. http://hortonworks.com/downloads/#sandbox

41. Machine Learning
• Python
• R https://www.r-project.org
• Mahout http://mahout.apache.org/
• MlLib http://spark.apache.org/mllib
• H2O http://www.h2o.ai
• Dato https://dato.com
• KNIME https://www.knime.org
• bdatalabs (soon) http://www.bdatalabs.com

42. https://www.coursera.org/specializations/machine-learning
Content:
• Machine Learning Foundations: A Case Study
Approach
• Machine Learning: Regression
• Machine Learning: Classification
• Machine Learning: Clustering & Retrieval
• Machine Learning: Recommender Systems &
Dimensionality Reduction

43. Google Machine Learning

44. Microsoft Azure

46. Open Datasets
• https://www.kaggle.com/datasets
• https://github.com/caesar0301/awesome-
public-datasets#machine-learning

47. https://www.kaggle.com/datasets

48. Take away
1. Big Data is a new section in the industry
(Photography vs Film)
2. N = All
3. Build up your strengths, catch up with your
weakness
4. Take advantage of open source power
5. Give it back