Thrift+scribe实现分布式日志收集，并与log4j集成

1. 刘中兵 sohu-tpc
lzbbox@hotmail.com

2.  大型系统=多台服务器，监控体系=分布式
汇报+统一处理
 Facebook每天log上百亿条信息(每秒log上
百万条)
 Scribe不依赖中心服务器，中心服务器不可
用时log在本地，当中心服务器可用后又会
同步到中心服务器
 Scribe本身由C++实现， RPC通信为Python
实现(基于Thrift)

3.  1.Thrift简介
 2.Scribe简介
 3.安装Thrift+Scribe
 4.简单演示样例
 5.部署for log4j

5.  Facebook的Thrift：基于C++，开源，跨语
言C++、C#、Erlang、Java、Perl、PHP、
Python、Ruby、HTML、XSD、Smalltalk、
Cocoa、OCaml、Haskell
 Google的Protocol Buffers：基于C++，开源，
基于C++, Java, Python
 Baidu使用的ICE基于C++，开源
 Renren使用ACE基于C++(比ICE底层)
Facebook与Google比较：http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers

6. Thrift传
输
通
信
协
议
服
务
端
代
码
客
户
端
代
码
Java
C++
Python
Erlang
Java
C++
Python
Erlang

7.  bool true, false
 byte 8位的有符号整数
 i16 16位的有符号整数
 i32 32位的有符号整数
 i64 64位的有符号整数
 double 64位的浮点数
 string UTF-8编码的字符串
 binary 字符数组
 struct 结构体
 list<type> 有序的元素列表，类似于STL的vector
 set<type> 无序的不重复元素集，类似于STL的set
 map<type1,type2> key-value型的映射，类似于STL的map
 exception 是一个继承于本地语言的exception基类
 service 服务。包含多个函数接口(纯虚函数)

8.  定义数据类型
 struct User {
 1: i32 uid,
 2: string name
 }
 定义服务接口
 service UserService {
 void store(1: User user),
 User retrieve(1: i32 uid)
 }

9.  导出命令
 thrift –r –gen py service.thrift
 可导出语言配置
 cocoa(Cocoa)、cpp(C++)、csharp(C#)、
erl(Erlang)、hs(Haskell)、html(HTML)、
java(Java)、ocaml(OCaml)、perl(Perl)、
php(PHP)、py(Python)、rb(Ruby)、st(Smalltalk)、
xsd(XSD)

11. Central
Server
Client
Client
Local
Server
Local
Server

12.  enum ResultCode
 {
 OK,
 TRY_LATER
 }
 struct LogEntry
 {
 1: string category,
 2: string message
 }
 service scribe extends fb303.FacebookService
 {
 ResultCode Log(1: list<LogEntry> messages);
 }

13.  启动Scribe Server:
 scribed config.conf
 停止Scribe Server:
 scribe_ctrl stop 11810
 监控Scribe Server:
 scribe_ctrl {command} 11810
▪ status – ALIVE表示运行
▪ version – 版本
▪ alive – 服务器启动时的时间(s)
▪ reload – 重载配置文件
▪ counters – 统计项
▪ received good, received bad, sent, denied for queue size, denied for rate, retries,
requeue, lost, received blank category

14.  配置项：
 port: 端口，默认0
 max_msg_per_second: 默认100,000
 max_queue_size: 默认5,000,000 bytes
 check_interval: 检查每个store间隔，默认5s
 new_thread_per_category: 默认yes，为每一个catagory创建一个新
的线程
 num_thrift_server_threads: 收取消息的监听线程数，默认1
 例如:
 port=1463
 max_msg_per_second=2000000
 max_queue_size=10000000
 check_interval=3

15.  file – 写入文件local or nfs.
 network – 发送到Scribe Server
 buffer – 主从存储，primary and a secondary
store
 bucket – 按照消息头分类hash存储
 null – 忽略
 thriftfile – 类似于file的ThriftTFileTransport
 multi – 同时写多个
参考：http://wiki.github.com/facebook/scribe/scribe-configuration

16. autoconf-2.60
automake-1.10.1
libtool-2.2.6b
zlib-1.2.5
libevent-2.0.4
python-2.6.5
boost-1.36.0(依赖python)
erlang-R13B01
thrift-0.2.0(依赖automake/libtool/python/erlang)
scribe-2.2(依赖libevent/python/boost/thrift)
scribe
thriftboost
python erlang
zlib
libevent
libtoolautomake
autoconf
安装包参见：192.168.0.1:/opt/soft

17.  autoconf-2.60(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 autoconf --version
 automake-1.10.1(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 automake --version

18.  libtool-2.2.6b(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 libtool --version
 zlib-1.2.5(http://www.zlib.net/)
 ./configure
 make
 make install
 ll /usr/local/include

19.  libevent-2.0.4(http://monkey.org/~provos/libevent/)
 ./configure
 make
 make install
 ll /usr/local/lib/
 python-2.6.5(http://python.org/)
 ./configure
 make(稍长时间，忽略Tcl/Tk错误)
 make install
 python –V, ctrl+D退出
 输出库路径：import sys, print sys.path

20.  boost-1.36.0(http://www.boost.org/)
 ./configure --prefix=/usr/local/boost_1_36_0 --
without-icu
 make install(较长时间)
 ll /usr/local/boost_1_36_0/
 erlang-R13B01(http://erlang.org/)
 ./configure
 make (较长时间)
 make install
 erl, ctrl+C

21.  thrift-0.2.0(http://incubator.apache.org/thrift/)
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make (较长时间)
 make install
 ll /usr/lib/python2.6/site-packages/thrift
 fb303(在thrift里：cd contrib/fb303)
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make
 make install
 ll /usr/lib/python2.6/site-packages/fb303
MONO错误需要pkg-config0.20

22.  scribe-2.2(http://github.com/facebook/scribe)
 vi /etc/profile
▪ export BOOST_ROOT=/usr/local/boost_1_36_0
▪ export LD_LIBRARY_PATH=/usr/local/boost_1_36_0/lib::/usr/lib:/usr/local/lib
 . /etc/profile
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make
 make install
 ll /usr/lib/python2.6/site-packages/scribe
 拷贝python库
 cp -r /usr/lib/python2.6/site-packages/* /usr/local/lib/python2.6/site-
packages
 cp /usr/local/bin/python /usr/bin/python
 验证：python, import thrift, import scribe

23.  启动服务端： /opt/soft/scribe
 scribed examples/example1.conf
 启动客户端： /opt/soft/scribe
 echo "hello world" | ./examples/scribe_cat test
 查看日志
 more /tmp/scribetest/test/test_current
 hello world

24. Thrift样例：Java、Python通信
Scribe Java客户端：pyserverjavaclient

25.  安装apache-ant-1.8.0
 export ANT_HOME=/opt/soft/apache-ant-1.8.0
 export PATH=$PATH:$ANT_HOME/bin
 编译libthrift.jar
 ant
 安装slf4j-1.6.0

26.  配置文件：/usr/liuzhongbing
 struct User {
 1: i32 uid,
 2: string name
 }
 service UserService {
 void store(1: User user),
 User retrieve(1: i32 uid)
 }
 生成：/usr/liuzhongbing
 thrift -r -gen java user.thrift
 产生：User.java、UserService.java

27.  编译：/usr/liuzhongbing/gen-java
 javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ *.java
 启动：/usr/liuzhongbing/gen-java
 服务端：java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaServer 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
 客户端：java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaClient localhost 9091
▪ retrieve: liuzhongbing

28.  生成：/usr/liuzhongbing
 thrift -r -gen py user.thrift
 启动：/usr/liuzhongbing/gen-py
 服务端：python PythonServer.py 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
 客户端：python PythonClient.py localhost 9091
▪ retrieve: liuzhongbing

29.  生成：/opt/soft/scribe/if
 thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen java
scribe.thrift
 thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen py
scribe.thrift
 编译：/opt/soft/scribe/if/gen-java
 javac -classpath
/usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-
api-1.6.0.jar:./ *.java

30.  启动Python服务端：/opt/soft/scribe
 scribed examples/example1.conf
▪ [Thu May 13 15:39:29 2010] "[test] Creating new category from model default"
▪ [Thu May 13 15:39:29 2010] "store thread starting"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/scribetest/test/test_00000> for
writing"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/test/test_00000> for writing"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <DISCONNECTED> to
<SENDING_BUFFER>"
▪ [Thu May 13 15:39:29 2010] "[test] successfully read <0> entries from file
</tmp/test/test_00000>"
▪ [Thu May 13 15:39:29 2010] "[test] No more buffer files to send, switching to streaming
mode"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <SENDING_BUFFER> to
<STREAMING>"
 启动Java客户端：/opt/soft/scribe/if/gen-java
 java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ JavaClient localhost 1463

31. scribe-log4j部署结构
部署Scribe Server
引用scribe-log4j
测试Scribe Server

32. scribe-
log4j.jar
Central
Server:11810
web1
service2
Local
Server:
11811
Local
Server:
11811
service1

33.  Central Server：192.168.0.1:11810
 启动：/usr/local/scribe/conf/central_start.sh
 停止：/usr/local/scribe/conf/central_stop.sh
 查询：/usr/local/scribe/conf/central_status.sh
 统计：/usr/local/scribe/conf/central_counters.sh
 日志目录：/opt/scribelog/central, 暂存
/opt/scribelog/tmpcentral
 Local Server：192.168.0.2:11811(App所在主机)
 启动：/usr/local/scribe/conf/local_start.sh
 停止：/usr/local/scribe/conf/local_stop.sh
 查询：/usr/local/scribe/conf/local_status.sh
 统计：/usr/local/scribe/conf/local_counters.sh
 远程主机：192.168.0.1:11810，暂存/opt/scribelog/tmplocal
配置文件参见：192.168.0.1:/usr/local/scribe/conf/

34.  引用scribe-log4j
 <dependency>
 <groupId>${project.groupId}</groupId>
 <artifactId>scribe-log4j</artifactId>
 <version>${scribe-log4j.version}</version>
 </dependency>
 log4j.properties中增加了几个功能配置：
 # scribe
 log4j.logger.com.mycompany=DEBUG,scribeAppender
 log4j.appender.scribeAppender=org.apache.scribe_log4j.ScribeAppender
 log4j.appender.scribeAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.scribeAppender.layout.ConversionPattern=[%p] - [%d{yyyy-MM-dd
HH:mm:ss}] - [%c.%t(%L)]: %m%n
 log4j.appender.scribeAppender.host=localhost
 log4j.appender.scribeAppender.port=11811
 log4j.appender.scribeAppender.reconnectInterval=10000
 log4j.appender.scribeAppender.category=scribe-service
 注意：自动重连信息输出在当前App标准stderr.log中

35.  修改host指向Local Server
 log4j.appender.scribeAppender.host=192.168.0.1
 注意：App中必须host=localhost
 运行scribe-log4j测试类
 Test.java
 查看服务端收到的日志：
 tail -f /opt/scribelog/central/scribe-service/scribe-
service_current
在/commons/scribe-log4j下进行

36.  安装并启动服务端： /opt/soft
 ./setup.sh > setup.log
 使用scribe-log4j测试
 Test.java