Big Data technologies and applications in Vietnam

1. Big Data in Action
Ngon Pham, Lana Engineer

2. Introduction
●
●
●
●
●
Introduction
Problem
Approach
Demo
Big Data in Vietnam

3. Introduction
● Internet-enabled devices
○ Tons of data generated every second
● Hardware becomes much cheaper
○ We can now store and process much more data

4. Problem
● How to process 10TB, how long and how
much?
○ Assume
■ Amazon EC2
■ HDD read at 50MB/s
■ Computation time is less than I/O time

5. Problem
● 1 machine, 1 core, 1 HDD
○ Time: 55.56 hours
○ Amazon Cost: $0.12 x 55.56 = $6.67
● 10 machines, 40 cores, 40 HDD
○ Time: 1.39 hours
○ Amazon Cost: $0.48 x 10 x 1.39 = $6.67
⇒ The same cost but 40x faster

6. Question
● How to divide data/process between
machines?
● How to make each process read data inside
the machine directly instead of another?
● How to replicate data, restore the process if
there is failure?
● Lots of task management questions...

7. Approach
● Hadoop
● MongoDB
● Spark

8. Hadoop Approach
● Storage
○ HDFS

9. Hadoop Approach
● Computation
○ MapReduce

10. MongoDB Approach
● Storage
○ Document

11. MongoDB Approach
● Computation
○ SQL
○ Aggregation
○ MapReduce

12. Spark Approach
● Storage
○ Resilient distributed
dataset (RDD)
○ Persistent backed by
HDFS / HBase...

13. Spark Approach
● Computation
○ Mixed
○ In-memory
computing

14. Demo
● Hadoop
○ Run script to create Amazon cluster
○ Play with Hadoop / HDFS / Spark
○ Process Wikipedia data
● MongoDB
○ Collect data from different sources and analyze

15. Big Data in Vietnam

16. Big Data in Vietnam
● Why is MongoDB popular?
○ Lots of PHP developers prefer
○ Simple to setup and use
○ Similar to MySQL

17. Big Data in Vietnam
● Hadoop is used by a few big local online
companies & international startups
○ Analyze tons of data
○ Create new competitive advantage
⇒ But there is a big shortage of skilled engineers

18. Q&A
Q&A