Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
ServerEngine Integration & Windows support
Ritta Narita
Fluentd Meetup 2016
About me.
Ritta Narita
joined Treasure Data after internship in 2015
2
@naritta
@narittan
Fluentd v0.14
ServerEngine is integrated
What’s ServerEngine?
A framework to implement robust multiprocess servers like Unicorn
worker
worker
worker
super
visor
Se...
how to use ServerEngine
worker
super
visor
Server
module MyServer
def before_run
@sock = TCPServer.new(
config[:bind], confi...
how to use ServerEngine
worker
super
visor
Server
se = ServerEngine.create(MyServer, MyWorker, {
daemonize: true,
log: 'my...
module MyWorker
def spawn(process_manager)
process_manager.spawn(env, config[:spawn_cmd])
end
end
ServerEngine.create(MySer...
What’s the advantages for integrating serverengine into fluentd?
serverengine advantages
Auto restart
live restart
socket manager
signal handler
log rotation
Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages
if worker died unexpectedly…
worker
worker
super
visor
Server
worker
if worker died unexpectedly…
worker
worker
worker
super
visor
Server
if worker died unexpectedly…
worker
worker
super
visor
Server
It’s detected by heartbeat and server launch worker again.
w...
Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages
worker
super
visor
Server
tcp tcp
send SIGHUP
live restart
super
visor
Server worker
tcp
kill the worker, leaving server’s condition
live restart
worker
super
visor
Server worker
tcp
start new worker and read config again
live restart
worker
super
visor
Server worker
tcp
tcp
share the TCP with living zero downtime
live restart
Signal and RPC server
api/config.reloadlive restart/config reload HUP
/api/processes
.interruptWorkers
INT
/api/processes
.k...
worker
super
visor
Server
you can reload system config without restart
system config reload
send USR2
check mtime and previo...
Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages
Socket Manager
worker
Socket
Manager
client
Socket
Manager
server
Server
26
①socket listen request②find or create listening...
how to use Socket Manager
worker
Socket
Manager
client
Socket
Manager
server
Server
27
#generate path for UDS/TCP for serv...
worker
Socket
Manager
client
Socket
Manager
server
Server
28
#get socket manager client from socket manager path
socket_ma...
What’s the merit of Socket Manager?
worker
worker
worker
Socket
Manager
client
Socket
Manager
client
Socket
Manager
client...
easy to use multicore power fully
30
with SocketManagerwith in_multiprocess plugin
<source>
type multiprocess
<process>
cm...
Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
send USR1
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
USR1
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT
USR1
send QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT
run!
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT run!
Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages
support new features
(These are already ported to ruby core.)
・multiprocess aware log rotation
・reopening of log file
・'tra...
summary
windows
・signal → rpc handler
・no socket FD and UDS
・fork based → spawn based
・live & auto restart
・multicore with...
Thank you!
appendix
unix→fork based damon
windows→spawn based damon
damonize
fluentd
server
engine
①set windows_daemon_cmdline
in configuration
...
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Nächste SlideShare
Wird geladen in …5
×

Fluentd Meetup 2016 - ServerEngine Integration & Windows support

5.549 Aufrufe

Veröffentlicht am

In Fluentd new version v0.14, ServerEngine is integrated.
I'll explain about the advantages of ServerEngine.

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

Fluentd Meetup 2016 - ServerEngine Integration & Windows support

  1. 1. ServerEngine Integration & Windows support Ritta Narita Fluentd Meetup 2016
  2. 2. About me. Ritta Narita joined Treasure Data after internship in 2015 2 @naritta @narittan
  3. 3. Fluentd v0.14 ServerEngine is integrated
  4. 4. What’s ServerEngine? A framework to implement robust multiprocess servers like Unicorn worker worker worker super visor Server Dynamic reconfig
 live restart Heartbeat via pipe auto restart multi process/thread
  5. 5. how to use ServerEngine worker super visor Server module MyServer def before_run @sock = TCPServer.new( config[:bind], config[:port]) end attr_reader :sock end module MyWorker def run c = server.sock.accept c.write "Awesome work!" end def stop c.close end end 1. write server module and worker module
  6. 6. how to use ServerEngine worker super visor Server se = ServerEngine.create(MyServer, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, bind: '0.0.0.0', port: 9071, }) se.run 2. write configuration and run
  7. 7. module MyWorker def spawn(process_manager) process_manager.spawn(env, config[:spawn_cmd]) end end ServerEngine.create(MyServer, MyWorker, { worker_type:‘spawn’, spawn_cmd: cmd }) ServerEngine.create(MyServer, MyWorker, { worker_type:‘thread' }) workerServer workerServer workerServer spawn thread processServerEngine.create(MyServer, MyWorker, { worker_type:‘process' }) worker type
  8. 8. What’s the advantages for integrating serverengine into fluentd?
  9. 9. serverengine advantages Auto restart live restart socket manager signal handler log rotation
  10. 10. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  11. 11. if worker died unexpectedly… worker worker super visor Server worker
  12. 12. if worker died unexpectedly… worker worker worker super visor Server
  13. 13. if worker died unexpectedly… worker worker super visor Server It’s detected by heartbeat and server launch worker again. worker
  14. 14. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  15. 15. worker super visor Server tcp tcp send SIGHUP live restart
  16. 16. super visor Server worker tcp kill the worker, leaving server’s condition live restart
  17. 17. worker super visor Server worker tcp start new worker and read config again live restart
  18. 18. worker super visor Server worker tcp tcp share the TCP with living zero downtime live restart
  19. 19. Signal and RPC server api/config.reloadlive restart/config reload HUP /api/processes .interruptWorkers INT /api/processes .killWorkers /api/plugins. flushBuffers system config reload USR2 TERM USR1flush buffers shutdown
  20. 20. worker super visor Server you can reload system config without restart system config reload send USR2 check mtime and previous load time, if there is new one, reload! serverengine config cash new config
  21. 21. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  22. 22. Socket Manager worker Socket Manager client Socket Manager server Server 26 ①socket listen request②find or create listening socket ③share the socket FD(unix)/ duplicated socket info(windows) UDS(unix)/ TCP(windows) ④get socket and accept in worker
  23. 23. how to use Socket Manager worker Socket Manager client Socket Manager server Server 27 #generate path for UDS/TCP for server connection socket_manager_path = ServerEngine::SocketManager::Server.generate_path #open socket manager server socket_manager_server = ServerEngine::SocketManager::Server.open(socket_manager_path) In server
  24. 24. worker Socket Manager client Socket Manager server Server 28 #get socket manager client from socket manager path socket_manager = ServerEngine::SocketManager::Client.new(socket_manager_path) #get listening socket from socket manager lsock = socket_manager.listen_tcp(bind, port) In worker how to use Socket Manager
  25. 25. What’s the merit of Socket Manager? worker worker worker Socket Manager client Socket Manager client Socket Manager client Socket Manager server super visor Server 29 we can use multicore easily in future
  26. 26. easy to use multicore power fully 30 with SocketManagerwith in_multiprocess plugin <source> type multiprocess <process> cmdline -c /etc/td-agent/td-agent-child1.conf </process> <process> cmdline -c /etc/td-agent/td-agent-child2.conf </process> </source> #/etc/td-agent/td-agent-child1.conf <source> type forward port 24224 </source> #/etc/td-agent/td-agent-child2.conf <source> type forward port 24225 </source> <source> type forward port 24224 </source> setting when using 2 core
  27. 27. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  28. 28. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 send USR1
  29. 29. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 USR1
  30. 30. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT USR1 send QUIT
  31. 31. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT
  32. 32. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT run!
  33. 33. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT
  34. 34. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT run!
  35. 35. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  36. 36. support new features (These are already ported to ruby core.) ・multiprocess aware log rotation ・reopening of log file ・'trace' level, which is lower level than ‘debug' ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1*1024*1024, })
  37. 37. summary windows ・signal → rpc handler ・no socket FD and UDS ・fork based → spawn based ・live & auto restart ・multicore with socket manager ・sophisticated signal handler ・easier log rotation
  38. 38. Thank you!
  39. 39. appendix
  40. 40. unix→fork based damon windows→spawn based damon damonize fluentd server engine ①set windows_daemon_cmdline in configuration ②run windows_daemon_cmdline when damonize ③windows_daemon_cmdline runs daemon.rb ④run_server with servermodule and workermodule

×