Real time big data applications with hadoop ecosystem

Real-time Big Data Applications
with Hadoop Ecosystem
Chris Huang
Sr. Manager, Core Tech
2014/9/24
1 9/25/2014 Confidential | Copyright 2013 TrendMicro Inc.

About – Chris Huang
• Chris Huang
– SPN Solution Developer Manager
– SPN Hadoop Architect
– Hadoop.TW Active Member
• Believes Cloud, Service, Software, Big
Data are critical factors for Taiwan’s
future economic development
9/25/2014 Confidential | Copyright 2013 TrendMicro Inc. 2

Conference Talks

Hot Keywords in Hadoop Community
Real-time
• Impala, Stinger
Computing Framework
• YARN, Tez
In Memory
• Spark
Streaming
• Kafka, Storm

Big Data Applications
• Operational
– Real-time
– Near Real-time
• Analytical
– Batch
– Interactive
– Near Real-time
– Streaming

An Online Music Example
• Operational
– Recent N login time (listen duration)
– Recent N album/artist user browses
– Recent N keyword user search
– Recent N song/album/artist user listens (buys)
– Recent N month user’s purchase amount
• Analytical
– Recommend right song/album/artist to right user at right time
– Correlate similar song/album/artist (CDDB or user behavior)
– Know seasonal music trending (X’max, Valentine’s Day, New Year)
– Know regional music trending
– Calculate regional leaderboard
– Connect user with social network

An Online Banking Example
• Operational
– Recent N login time / frequency
– Recent N items purchased by credit card
– Recent N month balance amount
– Recent N transfer in/out amount
– Recent N investment event
– Recent N month investment balance
• Analytical
– Know user’s profile more (assets/debts/shopping habits/family)
– Recommend right product to right user (investment, credit card, loan)
– Know seasonal trending (tax month/year end/back to school/X’mas)
– Know regional investment product leaderboard (by different age)
– Recommend product by similar user profile

Building Your Big Data Applications
• Think about your data
– Entity or Event?
• Think about your use case
– Operational or Analytic?
• Think about your data user
– External or Internal?

Think About Your Data
Slides from “Apache HBase Application Archetypes”,
HBaseCon 2014
You can Replace HBase with similar alternatives, but
concepts are the same

Think About Your Use
Case

Operational Use Case 1
MR /
Spark
Real-time
MR /
Spark
Real-time
Batch
Batch
Real-time
HDFS

HBase: No Secondary Index (yet)
• Search index building (row key)
• Use Solr to make text data searchable
– Snapshot & clone table
– Index column qualifier text
– Record row-key in Solr document
– Use HBase client to fetch data
• Usually less than few seconds

Operational Use Case 2 (SPN)
Get, Scan
Solr Client
low latency
high throughput
Index Query
MapReduce
Pig
HDFS
Flume
Feed App
Real-time
Real-time
Batch

Operational Use Case 3 (Mixed)
Real-time
Put, Incr,
Append
Get, Scan
Solr Client
low latency
high throughput
Index Query
Gets
Short scan
MapReduce
Pig
HDFS
Flume
Feed App
Real-time
Batch
HBase Client
HBase Client
Bulk Import
HBase Client
MR /
Spark Batch
HBase
Replication
Solr
MR /
Batch Spark

HBase or HDFS?
• Depends on what’s your data
– Entity or Event?
• Depends on your workload
– Low latency?
– Random read/write?
– Short/full scan?
– Sequential read/write?
– Update?

Wait…
Batch for
Operational?

Yes,
Why not?

Operational: Batch + Real-time
• Bridge the gap between batch and now
• 80/20 rule
– HDFS/MapReduce/Spark solves 80% easily
– Remaining 20% takes 80% of the efforts
• Go as close as possible, don’t overdo it!

What is Real-time?
• Real-time is NOT always “faster than batch”
– If you have really BIG DATA
• Most of the time, we want Timely Information
• Minimize the gap between scheduled batch jobs
Hourly Job
Hourly Job
Hourly Job
How to get result at 1:33?

Analytical Use Case
Batch/streaming compute
Near real-time/interactive deliver

Near Real-time Interactive

Recommendation
System

The Online Music Example
• Operational
– Recent N login time (listen duration )
– Recent N album/artist user browses
– Recent N keyword user search
– Recent N song/album/artist user listens (buys)
– Recent N month user’s purchase amount
Do you really want to analytical result
• Analytical
(recommendation)
EVERY 50 millisecond?
– Recommend right song/album/artist to right user at right time
– Correlate similar song/album/artist (CDDB or user behavior)
– Know seasonal music trending (X’max, Valentine’s Day, New Year)
– Know regional music trending
– Calculate regional leaderboard
– Connect user with social network

Analytical Use Case 1
Batch
HDFS
Index Query
Solr Client
Real-time

Analytical Use Case 2 (SPN)
“A Graph Service for Global Web Entities Traversal and Reputation Evaluation Based on HBase”,
HBaseCon 2014

You Need an
Interactive
Analytic Engine

Stinger

Impala Architecture
Datanode
Tasktracker
Regionserver
impala
daemon
NN, JT, HM
Active
NN, JT, HM
Standby
Datanode
Tasktracker
Regionserver
impala
daemon
Datanode
Tasktracker
Regionserver
impala
daemon
State store
Catalog
Datanode
Tasktracker
Regionserver
impala
daemon
Hive
Metastore

Apache Pig (MapReduce)
• Do hourly count on akamai log
– A = load 'date://2014/07/20/00'
using AkamaiRCLoader();
B = foreach (group A all) COUNT_STAR(A);
dump B;
– …
0% complete
100% complete
(194202349)
Too Slow for
Interactive

Using Impala
• No memory cache
– > select count(*) from akafast
where day=20140720 and hour=0
– 194202349
• with OS cache
• Do a further query:
– select count(*) from akafast where day=20140720
and hour=00 and c='US';
– 41118019
Make Sense
Now

Don’t Connect
Analytic
Engine with
Operational
Use Case

Analytical Use Case 3
low latency
high throughput
Real-time
Put, Incr,
Append
Gets
Short scan
HBase Client
Impala/Stinger
HDFS
Flume
Feed App
Real-time
Interactive
HBase Client
Bulk Import
HBase Client
MR /
Spark Batch
Customer
Analyst

Streaming Use Cases

TME – Trend Message Exchange
http://trendmicro.github.io/tme/

Streaming Operational Use Case
Real-time
Gets
Short scan
Kafka/Storm
Put, Incr,
Append
HBase Client
Kafka/Storm
low latency
HDFS
high throughput
HBase Client
Streaming
Index Query
Solr Client
Streaming

Streaming Analytical Use Case
Put, Incr,
Append
HBase Client
Kafka/Storm
low latency
HDFS
high throughput
Flume
Feed App
Gets
Short scan
HBase Client
Impala/Stinger
Interactive
Analyst
Real-time
Customer
Streaming

Think About Your Data User

Data User
• External
– Customer
– Partner
• Internal
– Business report user
– Data researcher
– Data analyst
– Algorithm developer
• They want instant response
• They don’t know (and don’t care) if
the recommendation is computed 1
hour ago or 50 ms ago
• Interactive or near real-time is
enough
• Sometimes even wait for batch (make
data small and analyze)
• Of course, everyone wants result
faster, but it depends on your
investment $$

No Silver Bullet
For Real-time,
Or Big Data Application

Q&A

Real time big data applications with hadoop ecosystem

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie Real time big data applications with hadoop ecosystem

Ähnlich wie Real time big data applications with hadoop ecosystem (20)

Mehr von Chris Huang

Mehr von Chris Huang (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Real time big data applications with hadoop ecosystem

Hinweis der Redaktion