#LAMP人#新一代互联网行为定向广告技术的挑战与优化- 品友互动专场

1. 12
- -
www.LAMPER.cn
QQ 83304912
http://weibo.com/lampercn
Copyright@2012 iPinyou All Rights Reserved.

2. Hadoop ETL
—Flume
Copyright@2012 iPinyou All Rights Reserved.

3. Copyright@2012 iPinyou All Rights Reserved.

4. Web Web
Log
Server Server
Web Web
Server Server
Copyright@2012 iPinyou All Rights Reserved.

5. Copyright@2012 iPinyou All Rights Reserved.

6. Copyright@2012 iPinyou All Rights Reserved.

7. ——Scribe
Copyright@2012 iPinyou All Rights Reserved.

8. ——Chukwa
Copyright@2012 iPinyou All Rights Reserved.

9. ——Flume
Copyright@2012 iPinyou All Rights Reserved.

10. ——Flume
• Flume
-Nodes
-Nodes Source Sink
-Nodes
-
- Sources Sinks
- Nodes
Copyright@2012 iPinyou All Rights Reserved.

11. ——Flume
Copyright@2012 iPinyou All Rights Reserved.

12. ——Flume
• Flume
1.Flume
- Source Sink APIs
-
- Sources Sinks
Decorators
Copyright@2012 iPinyou All Rights Reserved.

13. ——Flume
• Flume
2.Connector
-Sources
Console Exec Syslog Scribe IRC Twitter
-Sinks
Console Local files HDFS S3
-Decorators
Sinks
Wire batching compression sampling throughput
throttling
Copyright@2012 iPinyou All Rights Reserved.

14. ——Flume
• Flume
3.
Copyright@2012 iPinyou All Rights Reserved.

15. ——Flume
• Flume
1. agentBESink
Copyright@2012 iPinyou All Rights Reserved.

16. ——Flume
• Flume
2. agentDFOSink
Copyright@2012 iPinyou All Rights Reserved.

17. ——Flume
• Flume
3. agentE2ESink
Copyright@2012 iPinyou All Rights Reserved.

18. ——Flume
• Flume
1.
-Web Page
-Flume Shell
Copyright@2012 iPinyou All Rights Reserved.

19. ——Flume
• Flume
2.
Configuring FlumeNode:
tail(“file”) | filter [ console, roll(1000)
{ dfs(“hdfs://namenode/user/flume”) } ] ;
Copyright@2012 iPinyou All Rights Reserved.

20. ——Flume
Copyright@2012 iPinyou All Rights Reserved.

21. – Why Flume?
Scribe Server Scribe Server Scribe Server
Scribe Store Scribe server
Agent Collector
Chukwa
Agent Collector Collector Store
Flume 3
Copyright@2012 iPinyou All Rights Reserved.

22. – Why Flume?
Agent Thrift Client
Scribe Collector Thrift server
Chukwa Agents Collectors
Agents Collector Decorators
Flume
Copyright@2012 iPinyou All Rights Reserved.

23. – Why Flume?
Scribe
Chukwa
Flume Web Flume Shell
Copyright@2012 iPinyou All Rights Reserved.

24. – Why Flume?
Copyright@2012 iPinyou All Rights Reserved.

25. Flume
1.Master
2.Collector
3.HDFS Small Files
4.
5.CPU
6.
Copyright@2012 iPinyou All Rights Reserved.

26. Flume
• Master
Copyright@2012 iPinyou All Rights Reserved.

27. Flume
• Collector
1.Agents Collectors
Copyright@2012 iPinyou All Rights Reserved.

28. Flume
• Collector
2. failover
Collector
Collector
Copyright@2012 iPinyou All Rights Reserved.

29. Flume
• HDFS small files
HDFS
- NameNode
- Map
Copyright@2012 iPinyou All Rights Reserved.

30. Flume
• HDFS small files
• CollectorSink rollmillis
• flume-site.xml flume.collector.roll.millis
Copyright@2012 iPinyou All Rights Reserved.

31. Flume
•
1. Batch
- batch(n,maxlatency) Decorator Event
batch customdfs
- Agent batch Collector
unbatch Event Event
Copyright@2012 iPinyou All Rights Reserved.

32. Flume
•
2.Compression
- gzip Decorator Event
- batch
batch gzip customdfs
- Agent gzip Collector
gunzip Event
gzip 80%
Copyright@2012 iPinyou All Rights Reserved.

33. Flume
•
3. checksum
- Collector Checksum
Checksum
- Checksum 33%
Copyright@2012 iPinyou All Rights Reserved.

34. Flume
• CPU
TailDirSource CPU
- 200ms
- CPU
-
-
- Log
CPU 2/3
Copyright@2012 iPinyou All Rights Reserved.

35. Flume
•
TailDirSource
-TailDirSource Direct Buffer
MaxDirectMemorySize Direct Memory
1/2
Copyright@2012 iPinyou All Rights Reserved.

36. Flume –
——
Copyright@2012 iPinyou All Rights Reserved.

37. Hadoop Redis Hbase Hive Pig
Oozie Ganglia Flume Lucene LIBSVM
Mahout Zookeeper…………
Zookeeper…………
Optimus Folo8
!!!!
http://www.ipinyou.com.cn
http://weibo.com/pinyouhudong
http://weibo.com/pinyouhudonghr !!!!
hr@ipinyou.com
Copyright@2012 iPinyou All Rights Reserved.

38. Copyright@2012 iPinyou All Rights Reserved.