Opensource, Highly available and Scalable solution that can accommodate your Log Management needs with a centralized Dashboard with Filtering capabilities using elastic search
1. How To Deploy Logstash 1.1.13 on
Centos 6.x
Author : Kanwar Batra
Audience : System Administrators, NOC Monitoring Team, DBA's, Developers
Relevance : This document outlines the deployment of Logstash server components
What is Logstash
Logstash is a tool for managing events and logs. You can use it to collect logs, parse them,
and store them for later use (like, for searching). Speaking of searching, logstash comes
with a web interface for searching and drilling into all of your logs.
How to Download the Software
The software can be downloaded here
Software Details
This document is based on a 2 node deployment as a POC without redundancy. Logstash is
recommended to be deployed as a HA Cluster for redundancy and avoid loss of log data
due to individual node outages.
First Node (LogStash Master Node)
o Centos 6.4 64 bit
o Logstash 1.1.13
o Elasticsearch v0.90 or higher
o Java v1.6 or Higher
o redis 2.6
o httpd 2.4
o apr 1.4.6
o grok 1.2
o geoip-geolite 2013.04.1
Second Node (Elasticsearch Node)
o Centos 6.4 64 bit
o Logstash 1.1.13 (For Agent)
o Elasticsearch v0.90 or higher
o Java v1.6 or Higher
O/S Configuration Changes
On Centos 6.4 Server modify the following files
/etc/sysctl.conf add to bottom of file
o sudo vi /etc/sysctl.conf
vm.overcommit_memory = 1
/etc/security/limits.conf
o * soft core unlimited
o * soft nofile 65535
o * hard nofile 65535
o elsearch soft memlock unlimited
o elsearch hard memlock unlimited
o elsearch soft nofile 256000
o elsearch hard nofile 256000
o elsearch soft rss unlimited
o elsearch hard rss unlimited
o elsearch soft stack unlimited
o elsearch hard stack unlimited
2. o elsearch soft cpu unlimited
o elsearch hard cpu unlimited
o elsearch soft nproc unlimited
o elsearch hard nproc unlimited
o elsearch soft as unlimited
o elsearch hard as unlimited
/etc/sysctl/selinux
o SELINUX=disabled
/etc/sysconfig/iptables & ip6tables
o Modify the files and add relevant ports. This document is created based on
iptables being disabled.
o service iptables stop
o service ip6tables stop
o chkconfig iptables off
o chkconfig ip6tables off
Reboot the Host after above Changes
Pre-Install Checks
o service iptables status ( output - iptables: Firewall is not running)
o service ip6tables status ( output - ip6tables: Firewall is not running)
o sestatus ( output - SELinux status: disabled)
Software Install
Logstash Node (Install rpm's in the following order )
sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm
sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13-1.el6.noarch.rpm
redis-2.6.13-1.el6.x86_64.rpm grok-1.20110708.1-1.el6.x86_64.rpm
sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm
Backup the default Logstash file logstash.conf in /etc/logstash directory to
logstash.conf.default
Create logstash.conf
Modify the elastic search yml file also and update it with relevant node details
if you are using GeoIP license change the logstash GOIP to ls
/usr/share/GeoIP/GeoIPCity.dat if using lite us the value in the attached logstash.conf
Install sudo yum install httpd-* apr-*
Create a link to /usr/lib64 in /etc/httpd
Modify the httpd.conf Please pay special attention to the LoadModules .
Unzip the kibana software downloaded earlier and move the directory to /var/www/html
Change directory to location of your kibana (/var/www/html/kibana3), copy kibana3.conf
to /etc/httpd/conf.d
kibana conf should be configured
config.js is updated as
To have all services startup at boot run chkconfig
o chkconfig httpd on
o chkconfig elasticsearch on
o chkconfig logstash on
This completes the setup of Logstash software on the First Host. The second host is
configured as an elastic search server.
Elasticsearch Node (Install rpm's in the following order )
sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm
sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13-
1.el6.noarch.rpm grok-1.20110708.1-1.el6.x86_64.rpm
sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm
3. Backup the default Logstash file logstash.conf in /etc/logstash directory to
logstash.conf.default
Create logstash.conf as for the agent
Update the elastic search yml as
To have all services startup at boot run chkconfig
o chkconfig httpd on
o chkconfig elasticsearch on
o chkconfig logstash on
Now we have a running Logstash environment. At this time you can access the Kibana
frontend
Run the curl command for template mapping from logstash server.
Configuration Files for references
https://drive.google.com/folderview?id=0B2jSbXbYuSe_MVotR3ZDdzlwaFE&usp=sharing
Disclaimer:
The install of this product and opinions are listed above are solely based on my experience in the
implementation of Logstash for a Customer and is a working solution copy from that experience..
You can reference and use this document and send questions which I can answer based on my
experience.
This however is not an official document from Logstash team and they have not evaluated this
document for it’s accuracy.