Weitere ähnliche Inhalte Ähnlich wie Plugins by tagomoris #fluentdcasual (20) Mehr von SATOSHI TAGOMORI (20) Kürzlich hochgeladen (20) Plugins by tagomoris #fluentdcasual2. @tagomoris
NHN Japan Corp.
Web Service Business Division
Development Department 2
Development Platforms
5. Fluentd Cluster status updates (2)
ruby 1.9.2 + glibc malloc
scribeline
ruby 1.9.3 + jemalloc
fluent-agent-lite
9. Fluentd Cluster status updates (6)
Before: 10 plugins
in_forward, out_forward, in_scribe, out_scribe
out_copy, out_roundrobin, out_exec_filter
out_hoop
out_flowcounter, out_growthforecast
After: +9 plugins
out_route, in_gc_stat, in_object_space
out_mongo, out_sampling_filter
out_amplifier_filter, out_datacounter
out_file_alternative, out_forest
11. gem search -r fluent-plugin
41 plugins
fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1)
fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2)
fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1)
fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)
fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0)
fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2)
fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0)
fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)
fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2)
fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1)
fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1)
fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8)
fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)
fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1)
fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1)
fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)
fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6)
fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1)
fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)
fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3)
fluent-plugin-mongo (0.6.7)
12. gem search -r fluent-plugin
13/41 plugins
fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1)
fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2)
fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1)
fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)
fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0)
fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2)
fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0)
fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)
fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2)
fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1)
fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1)
fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8)
fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)
fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1)
fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1)
fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)
fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6)
fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1)
fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)
fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3)
fluent-plugin-mongo (0.6.7)
13. file-alternative (1/13)
out_file
time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
out_file_alternative
time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] {"key1":"value1", "key2":"value2",... }
tag [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] tag [TAB] value1 [TAB] value2 ...
time,tag,value1,value2 ...
14. hoop (2/13)
Hoop
Fluentd
Server HDFS
output format compatible with out_file_alternative
15. mysql (3/13) alpha!
Fluentd
MySQL
JSON into single column
or
values into columns
18. flowcounter (6/13)
messages
count_keys *
flowcounter unit minute
{"count":300, "bytes":3660,
"count_rate":5, "bytes_rate":61}
19. datacounter (7/13)
messages
unit minute
datacounter aggregate all
count_key referer
pattern1 google google.com
{"google_count":120,
"google_rate":2,
"google_percentage":20,
"unmatched_count":480, ...}
20. alpha!
numeric-monitor (8/13)
messages
unit minute
aggregate all
numeric_monitor monitor_key duration
percentiles 90,95
{"min":100131,"max":483120,
"avg":181343,
"percentile_90":283201,
"percentile_95":341210}
21. notifier (9/13) alpha!
10 Notify msg
12
suppressed
10 notifier in specified interval...
warn 10
crit 30
15
13 Notify msg
24. beautiful real-time
monitoring world
Streaming data
Sampling
Counting
Amplifying
Generatig
Notification
Output
(Graph, IRC, ...)
25. parser (12/13) alpha!
{"log":"2012/05/18:16:21 DRINK1 1"}
format /^(?<time>S+) (?<item>S+)
(?<num>d+)$/
numeric_monitor time_format %Y/%m/%d:%H:%M:%S
key_name log
reserve_data yes
log_based_time message
{"log":"2012/05/18:16:21 DRINK1 1",
"item":"DRINK1", "num":"1"}
26. <match input.A>
forest (13/13)
type foo
....
<match input.*>
conf_a xxxx.A.yyyy
type forest
option_x 30
subtype foo
....
remove_prefix input
</match>
<template>
<match input.B>
....
type foo
conf_a xxxx.__TAG__.yyyy
....
....
conf_a xxxx.B.yyyy
</template>
option_x 30
<case {C,D}>
....
option_x 100
</match>
</case>
<match input.C>
<case *>
type foo
option_x 30
....
</case>
conf_a xxxx.C.yyyy
</match>
option_x 100
....
</match>