Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Fluentd meetup logging infrastructure in paa s

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 29 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Anzeige

Ähnlich wie Fluentd meetup logging infrastructure in paa s (20)

Weitere von Rakuten Group, Inc. (20)

Anzeige

Aktuellste (20)

Fluentd meetup logging infrastructure in paa s

  1. 1. Logging infrastructure in PaaS with Fluentd Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team 1 Rakuten. Inc, 22/08/2012
  2. 2. About us §  ささきと申します。日本人です。 §  ワリと申します。メキシコ人です。 (Sasaki, from Japan) (Wally, from Mexico) §  Living in Japan since 1982. §  Living in Tokyo since 2010. §  Tech Lead of the PaaS Dev&Ops §  Member of the PaaS Dev&Ops team team at Rakuten, Architect of at Rakuten, programming in Ruby. Rakuten PaaS. §  Emacs user (org-mode rocks, try .org §  Node.js user / node-fluentd-logger markup in Github) §  CouchDB, …etc §  OpenStreetMap (Mapnik rocks) 2
  3. 3. Agenda §  Cloud Foundry §  Cloud Foundry + Fluentd §  Fluentd Pros & Cons §  Misc 3
  4. 4. Cloud Foundry •  Open Application Platform as a Service project •  By VMware •  Written in Ruby •  OpenSource http://github.com/cloudfoundry/vcap •  Hosted version http://www.cloudfoundry.com/ 4
  5. 5. Why Cloud Foundry? •  Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html http://www.slideshare.net/rakutentech/cloud-foundry-10423477 •  Main points: –  Open Source –  Separation of concerns from IaaS –  Scalable architecture –  Multi language 5
  6. 6. What Cloud Foundry does? •  Easy to deploy applications!! $ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3 •  Not easy to deploy business ready applications yet. –  We are now solving our own issues by modifying Cloud Foundry source –  One critical thing is Logging! –  Now we’ve solved this matter basically, let’s share our case study. 6
  7. 7. Current logging issue in Cloud Foundry $ vmc push yssk22-myapp! ! $ curl http://yssk22-myapp.cloudfoundry.com/! ! $ vmc logs yssk22-myapp! ! [Wed, 22 Aug 2012 08:03:48 GMT] Hello World! ! ! $ vmc update yssk22-myapp! ! $ vmc logs yssk22-myapp! 7
  8. 8. Current logging issue in Cloud Foundry $ vmc push yssk22-myapp! ! $ curl http://yssk22-myapp.cloudfoundry.com/! ! $ vmc logs yssk22-myapp! ! [Wed, 22 Aug 2012 08:03:48 GMT] Hello World! ! ! $ vmc update yssk22-myapp! ! $ vmc logs yssk22-myapp! Oops, logs are gone. 8
  9. 9. Persisting logs is a must for business applications! •  Monitoring •  Analysis •  Visualization Ask a data scientist => Sorry we lost everything :P http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist 9
  10. 10. Cloud Foundry does not persists the logs from apps… •  Enter Fluentd! 10
  11. 11. Brief overview of Cloud Foundry architecture Droplet Execution Agent Source: http://www.slideshare.net/derekcollison/design-of-cloud-foundry 11
  12. 12. Approach Attach a Fluentd collector to one application instance upon dispatch. DEA app fluentd app fluentd Log Storage DEA app fluentd app fluentd 12
  13. 13. Collecting the logs from the apps DEA •  Log collector monitors changes in the files app fluentd –  Stdout, stderr logs in general –  Catalina, manager, host-manager logs for Java –  error.log for PHP apps app fluentd app fluentd •  Also makes available: –  VCAP_LOG_PORT env variable to flush logs directly app fluentd –  VCAP_LOG_STREAM_PORT to inspect the logs in realtime app fluentd 13
  14. 14. Collecting the logs from the apps DEA app fluentd Log Storage Fluent::Logger => VCAP_LOG_PORT Sends logs directly to Fluentd making unnecessary having to write to file. VCAP_LOG_STREAM_PORT Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’} 14
  15. 15. Small setup for collecting the logs DEA CloudController app fluentd Log Storage Server app fluentd /logs/:path fluentd Rack app Collected log files DEA Log Storage Server CloudController app fluentd Rack app app fluentd Collected log files /logs/:path 15
  16. 16. How is it setup? app/ ! fluentd/! Attached to an application instance on |-- stderr.stat (1)! startup by DEA |-- stdout.stat (2)! •  (1) and (2) is a “stat file” used by the input plugin, |-- fluentd.conf (3)! which allows it to pick up from the last position it |-- startup (4)! had monitored. |-- stop (5)! fluentd_child.pid (6)! •  (3) is the configuration of the plugins dynamically generated by the DEA fluentd.pid (7)! logs/! •  (4) and (5) are the scripts executed to manage the |-- fluentd.log (8)! fluentd process |-- stdout.log! •  (6) and (7) are the supervisor and main process |-- stderr.log! ids from Fluentd ruby! run.pid! •  (8) is where any logs produced by the logging daemon are stored startup! stop! 16
  17. 17. fluentd.conf :: Input configuration ! Set up VCAP_LOG_PORT to flush the logs <source>! directly type forward! port 12345! </source>! ! <source>! type cf_app_logs! instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a! runtime ruby19! framework sinatra! tag logs.storage! </source>! ! ! Specify runtime and dir to monitor files 17
  18. 18. fluentd.conf :: Output configuration <match logs.**>! type copy! Settings for when forwarding <store>! to the log storage, we include user type cf_app_forward! information here so that it cannot be flush_interval 5s! overrided by a logging library. buffer_type file! buffer_path fluentd/fluentd_buffer! ! cf_app_user foo@example.com! cf_app_name appname! cf_app_instance_id 3e9676485562c64d81a61de2df05903a! cf_app_instance_index 0! 18
  19. 19. fluentd.conf :: Output configuration <server>! name logserver! Location of the log storage server: host 192.168.2.1! -  We use a active-standby setup at the port 4224! moment. </server>! -  When primary Logserver is down, logs <server>! are forwarded to the one on standby. name backup_logserver! host 192.168.2.2! port 4224! When both are down, fluentd buffers to standby! files the logs locally until one of them is </server>! available. </store>! <store>! type cf_app_logs_streaming! port 43208! </store>! </match>! Set up VCAP_LOG_STREAM port to enable tail –f like support. 19
  20. 20. …And now that we managed to collect the logs, we also can support the following 2 commands: 20
  21. 21. vmc tail 21
  22. 22. vmc log-storage 22
  23. 23. What we could do •  Persisting the logs in Cloud Foundry is a must! •  Support to persist the logs is a key issue in adoption of Cloud Foundry for developing application services. •  Thanks to Fluentd we could enable users to have log archives in a straightforward way. •  Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy. 23
  24. 24. What we could not do •  It was difficult to let the user have access to the logs from Nginx (used as a Router in CF) # Sample release process! vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp! vmc map app-v1 olympics.rakuten.co.jp => Released!! vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp! vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps! vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.! •  Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user. •  Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs. 24
  25. 25. Fluentd: Pros and Cons •  Pros –  Easy to include in Cloud Foundry compared to other solutions •  Possible to include it like gem 'fluentd' in a Ruby component –  Easy to change storage technology depending on scale –  Easy to extend –  Growing community o/ •  Cons –  Fluentd needs better testing tools –  Better materials about Fluentd from the community? •  Learning curve is already low •  Just we need to share more about how we are using Fluentd •  e.g. 'The Little Fluentd Book', 'Advanced Fluentd Recipes’ –  Cool.io is not maintained 25
  26. 26. Misc: Playing with the message bus •  Cloud Foundry is built with Eventmachine. •  Fluentd is built with Cool.io. •  Cool.io and Eventmachine do not play along very well. •  The message bus from Cloud Foundry named NATS is also built using Eventmachine. 26
  27. 27. Misc: NATS input plugin # This works, though a warning is displayed from Cool.io overriding Eventmachine! require 'nats/client'! ! module Fluent! class NatsInput < Input! Plugin.register_input('nats', self)! def run! NATS.start(:uri => @nats_connection_uri) do! ! # Triggered when an url changing event occurs (remap, change)! NATS.subscribe('dea.update') do |msg|! Engine.emit('logs.dea_update', Time.now.to_i, msg)! end! ! # Triggered when an application is started or dea is started! NATS.subscribe('dea.*.start') do |msg|! Engine.emit('logs.dea_start', Time.now.to_i, msg)! end! end! end! 27
  28. 28. Misc: Adding new forwarding nodes dynamically through NATS? # DEA @droplets.each_value do |instance| begin instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port]) instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 }) rescue => e @logger.error "Error: #{e} -- #{e.backtrace}" end end # Extended Forward plugin module Fluent class CloudfoundryAppForwardOutput < ForwardOutput Plugin.register_output('cf_app_forward', self) def emit(tag, es, chain) es.each do |emit_time, record| # Add a new forwarding server if record.has_key?('type') and record['type'] == 'serverConf’ add_node(record['host'], record['port’]) next end 28
  29. 29. FIN You can enable log collection to your Cloud Foundry by merging this patch: https://github.com/rakutentech/dea/commit/ 64d271904de1e195cc90ca1958eadf704f530d94 Checkout our github repo for more Cloud Foundry related things: https://github.com/rakutentech/ That’s all, thanks! ご清聴ありがとうございました! 29

×