Atmosphere 2014: Centralized log management based on Logstash and Kibana - case study - Dariusz Eliasz
•
1 gefällt mir•1,691 views
Nowadays cloud enviroments are primary platform for applications. We no longer have multipurpose machines, rather multiple smaller virtual servers with dedicated roles. Therefore there is a need to have one place where we can manage applications and system logs. I wish to share my experience gained while building centralized log managment system using Nxlog, Logstash and Kibana. With that tools we are building cost effective and scalable log managment platform. Dariusz Eliasz - Works in Allegro Group as a Solution Architect and is responsible for organizing cooperation with infrastructure teams, also leads some of the infrastructure projects. Earlier as an Expert System Administratorhe was related with building and maintaining the infrastructure shared services (i.e. image hosting platform) within Allegro Group.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Atmosphere 2014: Centralized log management based on Logstash and Kibana - case study - Dariusz Eliasz
Ähnlich wie Atmosphere 2014: Centralized log management based on Logstash and Kibana - case study - Dariusz Eliasz (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Atmosphere 2014: Centralized log management based on Logstash and Kibana - case study - Dariusz Eliasz
- 1. Centralized log management based on Logstash and Kibana - case study Dariusz Eliasz 20.05.2014 Atmosphere Conference
- 2. • What’s the problem ? • Solutons • Transport format • Architecture – Sender – Log router – Log collector – Full text search engine – GUI • Use case Agenda
- 3. What’s the problem ??
- 6. Solutons ??
- 11. Transport format - syslog • RFC3164 (BSD syslog ) • limited size - 1kB • format of a syslog message: – PRIORITY (calculated from severity and facility) – HEADER (tmestamp + hostname or IP) – MSG (tag + content) <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
- 12. Transport format - json • JavaScript Object Notaton • lightweight text-data interchange format • language independent • self-describing
- 13. Transport format - json { "LogType": "access_log", "Vhost": "atmosphere-conference.com", "HtpsOn": "false", "Xrealip": "1.2.3.5", "Clientp": "91.17.13.28", "UserAgent": "Mozilla/4.0 (compatble; MSIE 6.0; Windows NT 5.1)", }
- 14. Architecture
- 15. Sender
- 16. Sender • nxlog (htp://nxlog-ce.sourceforge.net/) • multple input types: – tcp socket – udp socket – fle input – unx socket • multple parser types: – bsd syslog – json
- 17. Sender Good practce: • make as much as possible processing on sender site, eg: apache access logs in json format • automate confguraton management
- 18. Log router
- 20. Log router ### JSON PARSER parser p_json { json-parser (prefx("_json.")); }; ### FILTERS flter f_someflter { ("${_json.SomeJsonField}” == ”abc.com”) }; ### INPUTS & OUTPUTS ### LOG PATHS log { source(s_network_json); destnaton(d_udp_logstash); destnaton(d_tcp_hadoop); #fags(fow-control); # disabled to separate destnatons };
- 21. Log router Good practce: • good separaton of destnatons • calculate enough redis size – it’s yor bufer • batch events writes to redis
- 22. Log collector
- 23. Log collector • Logstash htp://logstash.net/ • collectng, parsing and storing logs tool • plugins: Inputs • fle • gelf • tcp • log4j • redis • varnishlog Codecs • json • line • msgpack • netlow • multline Filters • grok • alter • cidr • geoip • grep • mutate Outputs • elastcsearch • graphite • jira • tcp • zeromq • zabbix
- 24. Log collector Good practce: • keep up2date version of java & logstash • use batch & multthread read from redis • read logs • bulk writes to elastcsearch
- 25. Full text search engine • Elastcsearch htp://www.elastcsearch.org/ • distributed, real-tme search and analytcs engine • store documents as a JSON • high availability • schema free • index mult-tenancy • on top of Lucene
- 26. Full text search engine • every index is replicated • every index sharded • index parttoning – tme based • data retenton – tme based
- 27. Full text search engine Good practce: • half memory for ES (<30GB), half for system cache • bootstrap.mlockall: true • gateway.recover_afer_nodes • indices.felddata.cache.size • authorizaton via proxy • curator • Marvel plugin
- 28. GUI • Kibana 3 htp://www.elastcsearch.org/overview/kibana/ • search, graph & analyze logs • JavaScript based (AngularJS) • only simple htp server needed
- 30. Q&A