1. LOGONOMICS:
The
Hidden
Side
of
Blackboard
Logs
by
steve
feldman
@PerfForensics

2. Logging
Doesn’t
Suck

5. It’s
Like
Fishing
in
the
Night…

8. So
Why
Don’t
We
Talk
About
Logs
More
OJen?

9. At
least
20%
of
all
people
in
this
room
don’t
know
where
to
find
their
logs.

10. At
least
50%
of
all
people
in
this
room
don’t
look
at
their
logs.

11. At
least
60%
of
all
people
in
this
room
don’t
visualize
their
log
data.

12. At
least
75%
of
all
people
in
this
room
don’t
correlate
data
between
logs.

13. At
least
90%
of
all
people
in
this
room
don’t
standardize
the
management
of
logs
to
a
centralized
service.

14. At
least
95%
of
all
people
in
this
room
don’t
alert
IT
staff
based
on
a
specific
log
event.

15. If
a
System
Doesn’t
Output
to
a
Log
Do
We
Assume
Nobody
is
Using
it?

16. If
a
System
ConZnuously
Spews
Data
to
a
Log
Do
We
Ignore
it?

18. What
We
Can
Do
With
Our
Log
Data
LOGONOMICS:
The
Hidden
Side
of
Blackboard
Logs

19. Trending
and
Intelligence
Service
Levels
Threats
and
VulnerabiliZes
Responsiveness
Reliability

20. Primer
Data
Points
Everyone
Should
Know
Unique
Requests
Time
Series
of
Requests
ConcentraZon
of
Request
Types
Origin
of
Requests
Quick
Averages
Cascading
Issues
Across
Logs

21. Combining
Other
Data
with
Log
Data
CorrelaZon
Root
Cause
InterpretaZon
CompleZon
of
Message
Full
Picture
Sequence
and
Timelines

22. Types
of
Data
We
Can
Get
LOGONOMICS:
The
Hidden
Side
of
Blackboard
Logs

23. Business
AnalyZcs:
AdopZon
and
Growth

24. System
Health

25. Capacity
Planning

26. Security
and
Threat
Analysis

27. Quality
and
Experience:
MeeZng
SLAs

28. Replay
and
Benchmarking

29. Insight
into
the
BbLogs
LOGONOMICS:
The
Hidden
Side
of
Blackboard
Logs

30. Four
Horseman
of
Logs

31. Bablefield
of
Other
Logs
• AuthenZcaZon
• Plugins
Directory
• NauZlus
for
events
• Monitoring
(System
Logs)
– Syslogs
and
Rsyslogs
(/var/messages)
– Windows
Event
Logs

32. Is
there
a
Most
Important
Log?

33. Access
Log
Log
Formafng
Mabers
Log
Levels
(INFO,
WARN,
ERROR)
mod_log_forensic
Use
%k,
%T
and
%D
Decompose
the
URI
Log
Formafng
Mabers

34. Is
there
a
2nd
Most
Important
Log?

35. Tomcat
and
Java
Logs
Stack
Traces
Startup
OpZons
GC
Events
GC
Pauses
and
Status

36. Tools
We
Should
Consider
LOGONOMICS:
The
Hidden
Side
of
Blackboard
Logs

37. It’s
All
About
the
Right
Fishing
Rod

38. CAT!
GREP!
TAIL!
SED!AWK!
SORT!

39. GROK!

40. SomeZmes
a
Net
is
Beber
to
Cast

41. Log
CentralizaZon

42. Please
Take
All
My
Logs
Format
Lots
of
Log
Data
Send
it
Down
the
River

44. • amqp
• exec
• file
• gelf
• redis
• stdin
• stomp
• syslog
• tcp
• twiber
• xmpp
• zeromq
• amqp
• elasZcsearch
• elasZcsearch_
river
• file
• ganglia
• gelf
• graphite
• internal
• loggly
• mongodb
• nagios
• date
• dns
• gelfify
• grep
• grok
• grokdisco
very
• json
• mulZline
• mutate
• split
• null
• redis
• statsd
• stdout
• stomp
• tcp
• websocket
• xmpp
• zabbix
• zeromq
Inputs
Filters
Outputs

45. Configure
Apache
for
JSON
log
• hbp://cookbook.logstash.net/recipes/apache-­‐
json-­‐logs/

46. Configure
Tomcat
for
MulZ-­‐Line
Filter

47. Setup
Bb
to
feed
logstash

49. What
We
Use
Logstash
Log
AggregaZon
Non-­‐FuncZonal
Requirements
Event
NoZficaZon
IntegraZon
with
Zabbix
Kibana
Front-­‐End
Redis
Inputs
&
Outputs
Indexing

50. Simple
Challenge
to
All
• Setup
Logstash
architecture
(All
Single
Node)
• Start
shipping
basic
log
files
– Apache
2.X
access
log
or
IIS
web
server
log
– Tomcat
Catalina
log
file
• Output
results
to
statsD
(Etsy
Project)
– Simple
Use
Case:
IncremenZng
HTTP
codes
(200,
300,
400)
• Visualize
statsD
data
with
Graphite

51. Bonus
Challenge
to
All
• Take
the
Vagrant
VM
and
integrate
Logstash
shipper
with
configuraZon
files.
• Add
Postgres
support
(Development
Only)
• Basic
syslog
funcZonality
for
CentOs
• Custom
Log
Interface
for
a
B2

52. Let’s
Add-­‐on
to
the
IniZaZve
developer.blackboard.com