1. What s new in v11
Sadayuki Furuhashi
fluentd.org

2. Non-stop restart
v10 (old):
supervisor process
listen TCP socket
worker process
heartbeat

3. Non-stop restart
v10 (old):
supervisor process
Downtime!
heartbeat
worker process
restart
initializing...

4. Non-stop restart
v11:
supervisor process
listen TCP socket
worker process
heartbeat

5. Non-stop restart
Zero downtime!
v11:
supervisor process
listen TCP socket
heartbeat
worker process
restart
initializing...

6. Multiprocess
supervisor process
heartbeat
worker process
<worker>
...
</worker>
<worker>
...
</worker>
<worker>
...
</worker>

7. Multiprocess
worker process
<worker>
<worker>
...
</worker>
<worker>
･ in_tail
･ in_forward
･ out_groupcounter
･ out_VERY_UNSTABLE
･ out_webhdfs
</worker>
</worker>
Separate important
logging from
experimental plugins

8. Error stream
v10 (old):
<source>
<match **>
{“event”: 1, ...}
chunk
event
OK
{“event”: 2, ...}
ERROR!
{“event”: 3, ...}
?
{“event”: 4, ...}
chunk
ERROR!
{“event”: 5, ...}
?
{“event”: 6, ...}
?
Error handling
depends on plugin
(throw away! etc.)

9. Error stream
v10 (old):
<source>
特殊ラベル
<match **>
{“event”: 1, ...}
event
chunk
OK
{“event”: 2, ...}
ERROR!
{“event”: 3, ...}
OK
{“event”: 4, ...}
chunk
ERROR!
{“event”: 5, ...}
OK
{“event”: 6, ...}
OK
<label @ERROR>
<match **>
...
OK
OK
Fluentd core
handles error
events

10. Plugin version management
v11:
$ cat /etc/td-agent/Gemfile
gem “fluent-plugin-s3”, “~> 0.11.3”
gem “fluent-plugin-webhdfs”, “~> 0.2.2”
gem “fluentd”, “~> 0.11.2”
$

11. Log level per plugin
v11:
<source>
<match **>
type forward
type mongo
log_level error
log_level debug
</source>
</match>

12. Variables in a config file
v11:
<source>
type forward
port “#{ENV[‘FLUENTD_LISTEN_PORT’]}”
</source>

13. Variables in a config file
v11?：
<server>
@set port 9800
@include “/etc/fluentd/server.conf”
</server>
<server>
@set port 9801
@include “/etc/fluentd/server.conf”
</server>

14. Streaming processing without tag hacking
v10 (old):
<match raw.**>
type flowcounter
add_tag_prefix counted
</match>
<match counted.**>
type growthforecast
</match>
v11:
<match **>
type flowcounter
<match **>
type growthforecast
</match>
</match>

15. v11
Reliability &
Non-stop restart
Performance
Multiprocess
Error stream
Management
Plugin version management
Log level per plugins
Configuration
Variables in a config file
Streaming processing
without tag hacking