RT-BigData Architecture Proposal

1. Real-Time Big Data
handarusakti@gmail.com

2. What is Big Data?

3. Bussiness Data
• Structured data
• Unstructured data (not less important than
structured data)

4. Data Analysis:

5. Predictive Analysis

6. Objectives
• Depend on our context
• Objective first, plan later

7. These Three Trends
• A shift to scalable, elastic computing
infrastructure.
• An explosion in the complexity and variety of
data available.
• The power and value that come from
combining disparate data for comprehensive
analysis.

8. What is Hadoop?
• A file store, HDFS (Hadoop Distributed File
System)
• A distributed processing system:
– 1.0: MapReduce
– 2.0: Yarn (a distributed operating system)
• Process comes to data

9. Hadoop 1.0 vs. Hadoop 2.0

10. HDFS
• Designed to distributing store very large data
sets reliably, and to stream those data sets at
high bandwidth to distributing computation
• HDFS Comics

12. YARN
• A cluster management technology
• YARN combines a central resource manager
that reconciles the way applications use
Hadoop system resources with node
manager agents that monitor the processing
operations of individual cluster nodes

13. YARN

14. Spark
• Doing large scale stream processing
• Achieve low latency
• Comparasion:
– Spark Streaming: 670k records/second/node
– Storm: 115k records/second/node
– Apache S4: 7.5k records/second/node

15. Spark
• Spark offers an integrated framework for
advanced analytics, including a machine
learning library (MLLib), a graph engine
(GraphX), a streaming analytics engine (Spark
Streaming) and a fast interactive query tool
(Shark)

16. Spark

17. Flume
• A distributed, reliable, and available service for
efficiently collecting, aggregating, and moving
large amounts of streaming data into the Hadoop
Distributed File System (HDFS)
• It has a simple and flexible architecture based on
streaming data flows; and is robust and fault
tolerant with tunable reliability mechanisms for
failover and recovery

18. Sqoop
• A tool designed for efficiently transferring bulk
data between Hadoop and structured
datastores such as relational databases

19. RT-BigData Proposal
Log Flume
RDBMS Sqoop HDFS
S
Spark Streaming
Shark GraphXMLLib
Dashboards
Spark
Spark SQL
MESOS

21. Images taken from:
• http://www.datameer.com/images/product/big_data_
hadoop/img_bigdata.png
• http://www.kdnuggets.com/websites/cartoons.html
• http://www.alexjf.net/blog/distributed-
systems/hadoop-yarn-installation-definitive-guide
• http://hadoop.apache.org/docs/r1.2.1/hdfs_design.ht
ml
• http://hortonworks.com/hadoop/yarn/