2. What
is
a
log
• Oxford
Dic4onary
– a
thick
piece
of
wood
that
is
cut
from
or
has
fallen
from
a
tree
– (also
logbook)
an
official
record
of
events
during
a
par4cular
period
of
4me,
especially
a
journey
on
a
ship
• 4me
+
data
3. In
theory,
life
cycle
of
log
Record
Transmit
Analyze
Store
Delete
6. Problems
• Logging
to
a
database
or
filesysytem
• Logging
has
placed
a
load
on
the
database
and
filesystem
• Mul4ple
log
formats
• No
easy
way
to
search
logs
• No
easy
method
to
gather
sta4s4cs
8. Why
use
Logstash?
• A lot choices!
• But we want a free & high-integrality &
easy to use solution
• splunk (finding your faults, just like mom)
• facebookarchive/scribe (2682 ★)
• Graylog2(Server+WUI 1683 ★)
• fluentd (2038 ★)
• logstash (2689 ★)
12. Elas4cSearch
• A
response
to
the
claim
:
“Search
is
hard”
• Powerful
indexing
&
search
tool
• search
&
index
data
available
Rescully
as
JSON
over
HTTP
15. How logstash works?
• logstash process events, not (only)
loglines!
• “The logstash agent is a processing
pipeline with 3 stages:
– inputs -> filters -> outputs.”
– separate threads
• “Inputs generate events, filters modify
them, outputs ship them elsewhere.”
• -- [the life of an event in logstash]
17. In
fact,
Event
Life
Cycle
event
(Input
-‐>
output)
event
-‐-‐-‐-‐-‐-‐-‐-‐-‐
input
filter
output
18. Logstash
is
a
wooden
tube
Input
Input
Input
filter
output
codec
filter
filter
output
output
19. Logstash
plugins
Workflow
• inputs
– How
events
get
into
LogStash.
• codecs
–
convert
an
incoming
format
into
an
internal
representa4on
• filters
–
processing
ac4ons
on
events
:
modify
events
or
drop
events
• outputs
– How
output
events
from
LogStash
21. What
is
an
event!?
• A
@4mestamp
(ISO
8601
4mestamp)
• A
messsage
field
(
data
)
• A
@version
• host
(
the
host
of
sender)
• type(
syslog,
irc,
etc)
27. logstash-‐forwader
• ♫
I'm
a
lumberjack
and
I'm
ok!
I
sleep
when
idle,
then
I
ship
logs
all
day!
I
parse
your
logs,
I
eat
the
JVM
agent
for
lunch!
♫
• WriRen
in
Go
• lumberjack
is
reserved
for
protocol
• Resource
Usage
Concerns
• Need
an
SSL
CA
to
verify
the
server
33. powerful
grok
• Parse
arbitrary
text
and
structure
it.
• The
syntax
for
a
grok
paRern
is
– %{SYNTAX:SEMANTIC}
• 55.3.244.1
GET
/index.html
15824
– %{IP:client}
– %{WORD:method}
– %{URIPATHPARAM:request}
– %{NUMBER:bytes}
• hRps://github.com/elas4csearch/logstash/blob/
v1.4.2/paRerns/grok-‐paRerns
