Brief information in comparing different logging and monitoring techniques

1. Следете актуалните обяви за DevOps
Партньори:
Monitoring & Logging
Marian Marinov
mm@yuhu.biz

2. Следете актуалните обяви за DevOps
Партньори:
Who am I?
● Director of Engineering at Web Hosting Canada
● Former partner and Head of DevOps at SiteGround
● A SysAdmin and System Architect

3. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● 13 physical linux machines
○ Storage capacity (df/df -i)
○ S.M.A.R.T. of the drives
○ RAID (HW or Soft)
○ Network (routes, traffic and usage)
○ Performance (CPU, Mem, I/O, Processes)
○ Kernel logs
○ Service logs

4. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● 1 UPS
● 2 APC PUDs
● 2 Switches (SNMP statistics)
● 2 Thermostat (traffic, temp, humidity)
● 40+ LXC containers
○ Performance (CPU, Mem, I/O, Processes)
○ Storage capacity (df/df -i)
○ Service logs
● 2-3 Wifi access points
○ number of attached devices
○ traffic per-device

5. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● A few things for which I want traffic and power on time
○ 3 TVs
○ 3 Amplifiers
○ 4 Cameras
○ 1 Washing machine
○ 1 Dryer

6. Следете актуалните обяви за DevOps
Партньори:
What I wanted
● Single solution for log and metrics collection
● Single central interface

7. Следете актуалните обяви за DevOps
Партньори:
What I ended up having
● multiple grafana dashboards
● monitor events, instead of reading logs
● a bunch of different log collectors

8. Следете актуалните обяви за DevOps
Партньори:
What tested
● syslog-ng
● rsyslog
● Filebeat
● Prometheus node_exporter
● Loki
● Fluentd
● Clolectd
● StatsD
● Graylog
● PostgreSQL+timescale
● Grafana

9. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs

10. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs
● collectd was the easiest
○ with the most metrics out-of-the-box

11. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs
● collectd was the easiest
○ with the most metrics out-of-the-box
● ElasticSearch + Kibana require too much resources
○ Not usable for smaller setups
● Graylog uses a lot of CPU for the work it does
○ alerts can be based on number of events instead of parsing logs

12. Следете актуалните обяви за DevOps
Партньори:
Installation / Setup
● basic apt-get:
○ rsyslogd, syslog-ng, fluentd, collectd, filebeat, loki, node_exporter
○ statsd wanted full npm

13. Следете актуалните обяви за DevOps
Партньори:
Pros and Cons
● Syslog pros
○ can easily ingest netconsole kernel logging
○ very good performance
○ well documented and standardized interface
● Syslog cons
○ fire and forget
○ the syslog protocol
○ not enough parsing flexibility
○ syslog-ng was heavier then rsyslogd

14. Следете актуалните обяви за DevOps
Партньори:
Pros and Cons
● Loki/Node_exporter/filebeat/fluentd
○ very good parsing capabilities
○ filebeat was the easiest for me
○ reliable log delivery
○ different integrations
○ ready made grafana dashboards
● Loki/Node_exporter/filebeat/fluentd
○ very heavy on CPU
○ Loki did not have sysv init script :)

15. Следете актуалните обяви за DevOps
Партньори:
Interesting
● OAIEvals Collector - by Nikolay Stankov

16. Следете актуалните обяви за DevOps
Партньори:
DB integrations
1. Prometheus node-exporter
2. Fluentd
3. filebeat
4. syslog

17. Следете актуалните обяви за DevOps
Партньори:
Not out of the box
● Custom local collectors still have to go directly to your metrics DB
● Having a producer/subscriber greatly reduces the performance hit
● Fluent and fliebeat were the only one supporting kafka out of the box
○ https://github.com/hikhvar/mqtt2prometheus
○ https://github.com/toyokazu/fluent-plugin-mqtt-io

18. Thank you!
СЛЕДВАЩО СЪБИТИЕ
Лектор Дата Език
Следете актуалните обяви за DevOps
Партньори:
Monitoring & Logging
Marian Marinov 19.Mar.2024 Български
Contacts:
Marian Marinov
Github profile
Facebook profile

19. Следете актуалните обяви за DevOps
Партньори:
What do I have on the containers?
● NextCloud
● Home Assistant
● Mirrors
● VPNs
● NetBox
● Monitoring (Grafana, StatPing)
● Games (Minecraft, CS, PVPGN)
● IRC (server, bouncers, bots)
● Matrix, Mattermost
● Backups
● Streaming (FOSDEM streamer setup)
● DBs (PostgreSQL, MySQL, Redis, DragonFly, Timescale, InfluxDB, Mongo)
● Vitess, ProxySQL
● MPI (Gearman, MQTT, Kafka, RabbitMQ)
● Web stuff - Wiki, HAproxy, Nginx, Varnish
● OpenShift, OpenStack, K8s on VMs and physical
● A lot of other experiments

20. Следете актуалните обяви за DevOps
Партньори:
What storage do I use?
● Local + LVM
● DRBD+OCFS2
● iSCSI
● cLVM + iSCSI
● GlusterFS
● OrangeFS
● I had in the past:
○ Ceph
○ NFS
○ cLVM + ATAoE
○ cLVM + NBD