講者：Etu 資深協理 | 陳育杰 簡介：過去這兩年內，Big Data 在企業的應用架構已逐漸形塑出來，我們看到，不同的產業，陸續開始運用 Hadoop 來解決不同的問題，而背後的 IT 架構，其實都具有一些共通性。我們將透過這些共通性的架構來探索 Big Data / Hadoop 具體展現的企業應用。

1. Monetize Enterprise Data
Big Data 在台灣的經典應用與行動
陳育杰 Eric Chen
Senior AVP, Etu Business Development
eric_chen@etusolution.com

2. 的特性
2
傳統平行運算架構
The
Old
Way:
Bringing
Data
to
Compute
Hadoop
架構
=
平行運算 +
分佈式儲存
The
New
Way:
Bringing
Compute
to
Data
運算
儲存
MapReduce
HDFS

3. Big Data 新應用架構
3
RDB
Business
Intelligence
ETL
Business
Analycs
Voice file
Video file
Image file
Doc file
Txt file
XML file
Web Logs
Click event
Social
network
Associated
map News
Feeds
Sensor
Embedded
RFID Tags
Geographic
GPS
Event
Others
MapReduce
HDFS
HBase
HIVE
Impala
Mahout
Pig

4. Big Data 新應用架構
Hadoop as a “Data Store”
4
RDB
Business
Intelligence
ETL
Business
Analycs
Voice file
Video file
Image file
Doc file
Txt file
XML file
Web Logs
Click event
Social
network
Associated
map News
Feeds
Sensor
Embedded
RFID Tags
Geographic
GPS
Event
Others
MapReduce
HDFS
HBase
HIVE
Impala
Mahout
Pig

5. Big Data 新應用架構
Hadoop as a “Data Pre-
processing Platform”
5
RDB
Business
Intelligence
ETL
Business
Analycs
Voice file
Video file
Image file
Doc file
Txt file
XML file
Web Logs
Click event
Social
network
Associated
map News
Feeds
Sensor
Embedded
RFID Tags
Geographic
GPS
Event
Others
HDFS
HBase
HIVE
Impala
Mahout
MapReduce
Pig
HIVE
QL
Join,
Aggrega,on,
Filter,
Sor,ng,
Correla,on
……..
Data
Integra4on

6. Big Data 新應用架構
Hadoop as a “DB”
6
RDB
BI
ETL
Business
Analycs
Voice file
Video file
Image file
Doc file
Txt file
XML file
Web Logs
Click event
Social
network
Associated
map News
Feeds
Sensor
Embedded
RFID Tags
Geographic
GPS
Event
Others
MapReduce
HDFS
HBase
HIVE
Impala
Mahout
Pig
ODBC
API

7. Big Data 新應用架構
Hadoop as a “Data Analytics
Engine”
7
RDB
Business
Intelligence
ETL
Business
Analycs
Voice file
Video file
Image file
Doc file
Txt file
XML file
Web Logs
Click event
Social
network
Associated
map News
Feeds
Sensor
Embedded
RFID Tags
Geographic
GPS
Event
Others
MapReduce
HDFS
HBase
HIVE
Impala
Mahout
Pig

8. Use Case
Reference Architecture
l Real-time Query (電信)
l Customer Services (電信)
l Yield Rate Improvement (電子製造)
l Recommendation Behavior Analysis (電
商、媒體、零售)
l Smart Retailing (Internet of Things)
8

9. Real-time Query (電信)
Use Case Reference Architecture
9
A10
/
F5
Etu
Hadoop
Cluster
HDFS
HBase
Syslog
/
TCP
Plug-­‐in
Etu
Data
Flow
MSISDN
IP
/
Time
IP
by
MSISDN
IP
by
MSISDN
MSISDN
by
IP
Real-­‐me
Query
End-­‐to-­‐End
5
sec
Historical
Query
Internet

10. Customer Services (電信)
Use Case Reference Architecture
10
DPI
Syslog
UDP
Internet
Border
Router
Etu
Log
Collector
200K
EPS
HDFS
FTP
1
TB
/
Day
0.7
TB
/
Day
Table
A
Table
B
Table
C
Etu
Hadoop
Cluster
for
Correla4on
(Regional)
HDFS
HBase
Etu
Hadoop
Cluster
for
Query
(180
days)
(Central)

11. 11
Etu TQuery
電信巨量資料多樣查詢最佳化解決方案

12. Yield Rate Improvement (電子製造)
Use Case Reference Architecture
12
組裝包產線
SMT產線
製程資料
組裝包產線
SMT產線
SMT產線
SPC
SFCS
SMT
SFCS
SPI
1. 生產問題及時發現: 資料處理與計算時間大幅縮短，可以提升品質判斷速度，減少產線損失
2. 運算效能佳: 採用平行運算與分散式檔案系統，減少過多Temp Files與資料轉換，生成統計表提供查詢
3. 容量擴充成本低: 因應產線擴充，機台測試資料增加與保存時間延長可線性擴充 (Scale out)
Etu IProspects
統計分析加以確認
統計分析
製作圖表
良率低
主因素
RDB
N
mins
MPP
DB
資料探勘
特徵規則 (平行運算)
HDFS
No
SQL
SPC
SMT
SFCS
SMT
Data
Files
及時
算得快
擴充成本低

13. Etu Recommender
Etu Insight
Recommendation Behavior
Analysis (電商、媒體、零售)
Use Case Reference Architecture
13
Consumer Discovery
Consumer Connect
Customer Profile Discovery
Data
Converter
Customer
Behavior
Data Warehouse
HIVE
JDBC/ODBC
Driver
Event
Collector
Customer Behavior
Data Store
Analytics core
Etu Hadoop Platform
HDFS
HBase
HIVE
User Tracking
Dashboard
Recommendation
Recommendation
List
商品推薦
內容推薦
廣告推薦
EDM
整合
CRM
整合
第三方
分析工具
連結

14. Recommendation Behavior
Analysis (電商、媒體、零售)
Use Case Reference Architecture
14
Etu Recommender
商品
內容
廣告
Consumer Connect
Customer
Behavior
Recommenda4on
List
Etu Recommender
商品
內容
廣告
Consumer Connect
Customer
Behavior
Recommenda4on
List
Consumer Discovery
DW
CRM
推薦運算叢集
3600
Customer
View
推薦運算叢集
客戶行為分析叢集
Customer Profile Discovery
Data
Converter
Customer Behavior
Data Warehouse
HIVE JDBC
/ODBCDriver
Event
Collector
Customer Behavior
Data Store
Analytics core
Event
Collector
Customer Behavior
Data Store
Analytics core

15. Smart Retailing (IoT)
Use Case Reference Architecture (Proof-of-Concept)
15
Etu Recommender
商品
內容
廣告
Consumer Connect
Customer
On-­‐line
Behavior
Recommenda4on
List
Consumer Discovery
DW
CRM
3600
Customer
View
Customer Profile Discovery
Data
Converter
Customer Behavior
Data Warehouse
HIVE JDBC
/ODBC Driver
Event
Collector
Customer Behavior
Data Store
Analytics core
Customer
Off-­‐line
Behavior

16. 318,
Rueiguang
Rd.,
Taipei
114,
Taiwan
T:
+886
2
7720
1888
info@etusolu4on.com
www.etusolu4on.com
Thank
you