1.
Log Scaling and Analytics With
Logstash
Richard Viet
Principal Engineer
Cloud Elements

2.
Problems

Logging to a database or filesystem

Logging has placed a load on the database and
filesystem.

Multiple log formats

No easy way to search logs

No easy method to gather statistics

3.
Logstash

Open source, Apache licence

Written in JRuby. Runs on jvm.

Plugins easily written in Ruby.

Part of the Elasticsearch family.

www.logstash.net

4.
Logstash

Scalable: Elasticsearch for indexing, search
and retrieval

Process multiple log formats

Receive logs from multiple sources

Output logs to multiple destinations

Kibana provides web interface for search and
analytics

Easily extended with plugins written in Ruby

5.
Logstash Architecture
Shipper
Broker Indexer
Search
Storage
Shipper
Shipper
Web
Interface

6.
Logstash Pipeline

Input → filters → output

Separate threads

Filters are applied in order of config file

Outputs processed in order of config file

7.
Logstash Plugins

Input – read input stream
– File input
– Log4j
– Redis
– Syslog

Codecs – decoding log messages
– Json
– Multiline

8.
Logstash Plugins

Filters – processing messages
– Csv – define fields in a csv
– Date – define date field formats
– Mutate – change date type
– Xml – extract xml
– Grok – parses arbitrary text

9.
Logstash Plugins

Output
– Elasticsearch
– Elasticsearch_http
– Mongodb
– Email
– Nagios

10.
Indexer

Send message to Elasticsearch for indexing

An index is created for each day

Each index split into 5 shards by default

Original message is stored

Each field indexed

11.
Elasticsearch

Apache Lucene search engine

An elasticsearch index is made up of multiple
shards

Each shard is a lucene index

Primary shard and at least one replica

Shards are moved between servers when
servers are added or removed

12.
Elasticsearch Configuration

Self discovery
– Multicast
• Simplest if all nodes on same network
– Unicast
• Provide a list of servers
– Combination

13.
Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

14.
Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

15.
Kibana

Browser based analytics for time-stamped data

Included in the logstash jar

Connect to the logstash server port 9292

Sends multiple requests to avoid overloading
the server.

16.
Log4j to Logstash
App
Logstash Redis
Elasticsearch
Cluster
App
App Logstash

17.
Logstash Log4j Server

Configure logstash as a Log4j server
input {
log4j {
mode => "server"
port => 9501
}
}

18.
Send to a broker

Configure broker
output {
stdout {}
redis {
host => "redis1"
data_type => "list"
key => "logstash"
}
}

19.
Indexing
input {
redis {
host => “redis”
data_type => “list”
key => “logstash”
}}
output {
elasticsearch {
cluster => “logstash”
host => "elasticsearch"
port => "9200"
}}

20.
Scaling
Broker
Indexer
Search
Storage
Shipper
Web
Interface
Broker
Indexer

21.
Sending to Broker
output {
stdout {}
redis {
host => ["redis1", “redis2”]
data_type => "list"
shuffle_hosts => true
key => "logstash"
}
}

22.
Indexing
input {
redis {
host => “redis1”
data_type => “list”
key => “logstash”
redis {
host => “redis2”
data_type => “list”
key => “logstash”
}
}
output { ...

23.
Quick Start

Logstash, elasticsearch and kibana configured
to run from the logstash jar

Download and untar

bin/logstash agent -f config.file

bin/logstash web