OSMC 2019 | Monitoring your Logs with Fluent by Toshaan Bharvani
•
0 gefällt mir•140 views
This presentation shows how to setup Icinga2 with Fluent and Grafana, for logging, monitoring, dashboarding and notifications. In the first part the presentation show how to setup FluentD the server part of Fluent for log aggregation, Fluentbit is the client that ships logs to the log server for both systems and applications. In the second part of the presentation the setup of Grafana for dashboarding is explained. In the third part the setup of Icinga2 for monitoring and notifications is explained. And finally the integration between these part is explained so you can get an integrated solution. At the end of the presentation a demo will show how this works with some examples.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie OSMC 2019 | Monitoring your Logs with Fluent by Toshaan Bharvani
Ähnlich wie OSMC 2019 | Monitoring your Logs with Fluent by Toshaan Bharvani (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
OSMC 2019 | Monitoring your Logs with Fluent by Toshaan Bharvani
- 1. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Monitoring your Logs with Fluent Fluentd with fluentbit Open Source Monitoring Conference 2019 6 November 2019 Toshaan Bharvani - VanTosh bvba <toshaan@vantosh.com> Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 1 / 28
- 2. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler $ whoami Toshaan Bharvani • From Antwerp, Belgium • Self-employed engineer/trainer (available for hire) http://www.vantosh.com • Involved with Enterprise Linux & RPM packaging : PEL, CentOS, RHEL, IBM AIX, BSD, SLES, . . . • Likes to keep everything secure : SELinux, WebSec, . . . • Lives in a virtual container world : KVM, Podman, PowerVM, z/VM, LXC, Xen, . . . • Likes automation CfgMgmt / DevOps : Ansible, Foreman, Puppet, . . . • Involved with hardware, software and conferences • Wants to take over the universe • Twitter : @toshywoshy • Blog : http://www.toshaan.com • Social : @toshywoshy Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 2 / 28
- 3. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler 1 Introduction Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 3 / 28
- 4. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Monitoring • Checks the current state of a subject • Checks the current utilization of a subject • Checks the current return of a subject Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 4 / 28
- 5. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Metrics • Measures and keeps the history of the states of a subject • Measures and keeps the history of the utilization of a subject • Measures and keeps the history of the return of a subject Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 5 / 28
- 6. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Logs • Logs are the records of the events of the system • Logs are the records of the messages of the system • Logs are the records of the transactions of the system Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 6 / 28
- 7. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler 2 Components Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 7 / 28
- 8. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler What is FluentD • A unified logging layer in between • Decouples backend from frontend • Streaming data system • JSON payload • Many inputs and output plugins • HA setup • Reduces complexity Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 8 / 28
- 9. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentD M+N Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 9 / 28
- 10. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler What is FluentBit • Log processor & forwarder • Light-weight daemon • Written in C • Many inputs and output plugins • Collecting from different sources and sending to different destinations • TLS connections • Reduces complexity Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 10 / 28
- 11. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Graphite - Carbon - Whisper • Written in Python • Graphite • Django-based web application that renders graphs and dashboards • Carbon • Metric processing daemons • Whisper • Time-series database library Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 11 / 28
- 12. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Grafana • Written in Golang • Dashboard tool • Visualization • Log viewing • Metrics view Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 12 / 28
- 13. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Icinga • Monitoring tool • Alerting tool • Notification tool • Business Overview Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 13 / 28
- 14. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Setup Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 14 / 28
- 15. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Ansible • Orchestration / CfgMgmt / Remote Execution Tool • Python based uses SSH, WinRM, or API • Server based, agentless1 • host information in flat files, CMDB, scripts, . . . • executes the task on the host side • Playbook : combination of tasks with meta information • Roles : abstract playbook with tasks, variables and handlers • Templates : Jinja2 • works where Python works2 1 Abstraction of SSH / WinRM to connect with the node and in certain setup an agent might be required, but by default it is not necessary 2 Ansible can actually run without Python on the remote host, however it is not fully supported Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 15 / 28
- 16. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler 3 Build Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 16 / 28
- 17. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentD Basic Setup 1 <source> 2 @type forward 3 port 24224 4 </ source> 5 6 <source> 7 @type http 8 port 9880 9 </ source> 10 11 <match myapp.access> 12 @type file 13 path /var/log/fluent/access 14 </ match> Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 17 / 28
- 18. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentD Application Setup 1 <source> 2 @type forward 3 </ source> 4 5 <source> 6 @type tail 7 @label @SYSTEM 8 </ source> 9 10 <filter access .**> 11 @type record_transformer 12 <record> 13 </ record> 14 </ filter> 15 <match **> 16 @type elasticsearch 17 </ match> 18 19 <label @SYSTEM> 20 <filter var.log.middleware .**> 21 @type grep 22 </ filter> 23 <match **> 24 @type s3 25 </ match> 26 </ label> Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 18 / 28
- 19. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentD Syslog 1 <source> 2 @type syslog 3 port 5140 4 bind 0.0.0.0 5 tag system 6 </ source> 7 8 <filter system .**> 9 @type grep 10 <regexp> 11 key ident 12 pattern /^ sudo$/ 13 </ regexp> 14 <regexp> 15 key message 16 pattern /COMMAND/ 17 </ regexp> 18 </ filter> 19 20 <filter system .**> 21 @type parser 22 key_name message 23 <parse> 24 @type regexp 25 expression /USER =(?< sudoer >[^ ]+) ; COMMAND =(?< command >.*)$/ 26 </ parse> 27 </ filter> Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 19 / 28
- 20. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentD Syslog 1 <source> 2 type carbon 3 tag tcp.events 4 format /^(?< key >S+)s+(?< val >S+)s+(?< time >S+)$/ 5 port 2003 6 time_format %s 7 bind 0.0.0.0 8 </ source> Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 20 / 28
- 21. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler FluentBit 1 SERVICE] 2 Flush 5 3 Daemon Off 4 Log_Level info 5 Parsers_File parsers.conf 6 Plugins_File plugins.conf 7 HTTP_Server Off 8 HTTP_Listen 0.0.0.0 9 HTTP_Port 2020 10 11 [INPUT] 12 Name cpu 13 Tag cpu.local 14 Interval_Sec 1 15 16 [OUTPUT] 17 Name stdout 18 Match * Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 21 / 28
- 22. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Carbon 1 [default] 2 pattern = .* 3 retentions = 10s:1d,1m:7d,10m:70d ,30m:180d,1h:5y Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 22 / 28
- 23. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Carbon 1 search_index : /var/lib/graphite/index 2 finders: 3 - graphite_api .finders.whisper. WhisperFinder 4 functions: 5 - graphite_api .functions. SeriesFunctions 6 - graphite_api .functions. PieFunctions 7 whisper: 8 directories: 9 - /var/lib/graphite/whisper 10 time_zone: UTC Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 23 / 28
- 24. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler 4 Automation Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 24 / 28
- 25. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Ansible Extentions • Install a new VM 1 ansible -playbook ~/. ansible/playbook/vm -install.yml -e ’host =newvm0 ’ • Setup FluentD and configure plugins for Graphite Carbon 1 ansible -playbook ~/. ansible/playbook/fluentd.yml • Setup FluentBit on a host and configure shipping to FluentD 1 ansible -playbook ~/. ansible/playbook/fluentbit.yml -l vmachine0 Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 25 / 28
- 26. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler Conclusion We can build an integrated open source monitoring and logging solution. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 26 / 28
- 27. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler The End Thank You for your attention Toshaan Bharvani - VanTosh bvba <toshaan@vantosh.com> http://www.vantosh.com/ Made with Beamer LATEX a TEXbased Presentation program Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 27 / 28
- 28. Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba Introduction Components Build Automation The End Spoiler CfgMgmtCamp 2020 3 → 5 February 2020 https://cfgmgmtcamp.eu/ Monitoring your Logs with Fluent Toshaan Bharvani - VanTosh bvba () 28 / 28