Vortrag an unserem Super WebGate Day am 17. März 2016 zum Thema Log Management mit dem ELK Stack

1. 1
Wie man aus langweiligen
Logdateien Gold gewinnen
kann

2. 2
About me
• Klaus Bild
• Senior System Architect
• IBM Connections/Sametime/TDI
• Monitoring/Log Management
• Infrastructure (Cloud, Docker…)
• Blog: http://kbild.ch
• http://linkedin.com/in/kbild
• https://www.xing.com/profile/Klaus_Bild

3. 3
Logdatei
Eine Logdatei (auch Ereignisprotokolldatei; englisch log file) enthält das automatisch
geführte Protokoll aller oder bestimmter Aktionen von Prozessen auf einem
Computersystem. Die korrekte Bezeichnung dafür ist deshalb Protokolldatei.
Wichtige Anwendungen finden sich vor allem bei der Prozesskontrolle und
Automatisierung. Prinzipiell werden alle Aktionen mitgeschrieben, die für eine spätere
Untersuchung (Audit) erforderlich sind oder sein könnten. Der Flugschreiber in
Flugzeugen ist ein Beispiel für kontinuierliche Protokollierung, die jedoch selten
ausgewertet wird, zum Beispiel nach einem Unfall.
Im Bereich der Datenbanken bezeichnet Logfile die Protokolldatei, in der Änderungen
an der Datenbank von korrekt abgeschlossenen Transaktionen (per Commit
abgeschlossen) festgehalten werden, um im Fall eines Fehlers (z. B. Systemabsturz) den
aktuellen Datenbestand wiederherstellen zu können.
https://de.wikipedia.org/wiki/Logdatei

4. 4
When do you consult logs?
Never:
• You are not an admin or developer
If something went wrong
(and a user reported it):
• What happened?
• Where?
• When?
• Why?

5. 5
But…
Multi-tier systems:
• Multiple servers
• Multiple applications
• Multiple databases
• Multiple systems
• …

6. 6
Log Sources
Applications / APIs
• Requests
• Error handling
• Successes
• Failed attempts
• Privilege changes
• Object manipulation
Appliances
• Routers
• Switches
• Firewalls
Databases
• Queries
• Errors
Infrastructure
• Servers
• Containers
• Web servers
• Load balancers
• Paas / IaaS
Front End
• Log-ins
• Form
completions
• Important click
events
Tools
• Configuration
Automation
• Analytics tools
• Alerting tools
• Chat tools
Sensors
• IoT
• Industrie 4.0
• Home
automation

7. 7
Log examples:
• [01988:00243-3598456576] 18.01.2016 08:49:35 Opened session for WGMob01/WGC/CH (Release 9.0.1FP4)
• [41732479.416668] [INT_2_VYATTA-default-D]IN=bond1 OUT=bond1.2036 MAC=00:00:5e:00:01:01:00:08:e3:ff:fd:90:08:00
SRC=95.26.112.172 DST=81.95.156.246 LEN=106 TOS=0x00 PREC=0x00 TTL=55 ID=27102 PROTO=ICMP TYPE=3 CODE=3
[SRC=81.95.156.246 DST=95.26.112.172 LEN=78 TOS=0x08 PREC=0x20 TTL=235 ID=62876 DF PROTO=UDP SPT=15798 DPT=53
LEN=58 ]
• 220.160.156.109 - - [18/Jan/2016:01:54:22 -0600] "POST /saveNewSubmit.do HTTP/1.1" 200 6687 "http://
www.logfilesarecool.net/createSubmit.do?submitId=4418324" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; MATP; rv:11.0) like
Gecko”
• [1/18/16 8:46:05:061 CET] 000001b6 IndexBuilderQ I com.ibm.connections.search.admin.index.impl.IndexBuilderQueue build
CLFRW0285I: Search is starting to build the index for wikis.
Booooring
Logs

8. 8
Visualization of Logs = Gold

9. 9
Visualization of Logs
Gives you:
• Operational Visibility
Gain end-to-end visibility across your operations and break down silos across your infrastructure
• Search and Investigation
Find and fix problems, correlate events across multiple data sources and automatically detect
patterns across massive sets of data
• Proactive Monitoring
Monitor systems in real time to identify issues, problems and attacks before they impact your
customers, services and revenues
• Business Insights
Make better-informed business decisions by understanding trends, patterns and gaining
operational intelligence from machine data

10. 10
Visualization of Logs
The Solution - ELK Stack
}

11. 11
The ELK stack
Elastic Search:
• Lucene based search engine (Java Stack)
• Distributed capability
• REST API over HTTP
• Data share using JSON format
Logstash:
• Ruby Agent application
• Agent to collect log data in numerous input formats
• Filters can be applied
• Many Output formats supported
Kibana:
• Flexible analytics and visualization platform

12. 12
WebGate environment
Agents/Shipper Broker
Filebeat
Filter/Indexer
Search/StorageWeb Interface/
Visualizer
Docker containers

13. 13
Logstash
Input:
• beats, couchdb_changes, drupal_dblog, elasticsearch, exec, eventlog, file,
ganglia, gelf, generator, graphite, github, heartbeat, heroku, http, http_poller, irc,
imap, jdbc, jmx, kafka, log4j, lumberjack, meetup, pipe, puppet_facter, relp, rss,
rackspace, rabbitmq, redis, snmptrap, stdin, sqlite, s3, sqs, stomp, syslog, tcp,
twitter, unix, udp, varnishlog, wmi, websocket, xmpp, zenoss, zeromq
Output:
• boundary, circonus, csv, cloudwatch, datadog, datadog_metrics, email,
elasticsearch, elasticsearch_java, exec, file, google_bigquery,
google_cloud_storage, ganglia, gelf, graphtastic, graphite, hipchat, http, irc,
influxdb, juggernaut, jira, kafka, lumberjack, librato, loggly, mongodb,
metriccatcher, nagios, null, nagios_nsca, opentsdb, pagerduty, pipe, riemann,
redmine, rackspace, rabbitmq, redis, riak, s3, sqs, stomp, statsd, solr_http, sns,
syslog, stdout, tcp, udp, webhdfs, websocket, xmpp, zabbix, zeromq

14. 14
Logstash
Filter:
• aggregate, alter, anonymize, collate, csv, cidr, clone, cipher, checksum, date,
de_dot, dns, drop, elasticsearch, extractnumbers, environment, elapsed,
fingerprint, geoip, grok, i18n, json, json_encode, kv, mutate, metrics, multiline,
metaevent, prune, punct, ruby, range, syslog_pri, sleep, split, throttle, translate,
uuid, urldecode, useragent, xml, zeromq
Log Entry/Message
84.74.43.46 - - [15/Mar/2016:08:41:00 +0100] "GET /files/basic/api/myfilesync/feed?page=1&pageSize=500&includeConflict=true HTTP/1.1" 200 1323 "-" "IBM-LC-
IBM Connections sync/1602.3033.1103 (Mac OS X 10.10.5)"
Document
Field 1
i.e. Source IP
Field 2 Field 3 Field 4 Field 5
Filters

15. 15
Logstash
Example (HTTP access log):
• 84.74.43.46 - - [15/Mar/2016:08:41:00 +0100] "GET /files/basic/api/myfilesync/feed?page=1&pageSize=500&includeConflict=true HTTP/
1.1" 200 1323 "-" "IBM-LC-IBM Connections sync/1602.3033.1103 (Mac OS X 10.10.5)"
filter {
if [type] == "apache_access" {
grok {
match =>
{ "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?:
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} }
…
clientip :
84.74.43.46
timestamp:
15/Mar/2016:08:41:00 +0100
verb:
GET
request:
/files/basic/api/myfilesync/feed?
page=1&pageSize=500&includeConflict=true
httpversion:
1.1
response:
200
bytes:
1323
referrer:
-
agent:
"IBM-LC-IBM Connections sync/1602.3033.1103
(Mac OS X 10.10.5)"

16. 16
Logstash
Example (HTTP access log):
• 84.74.43.46 - - [15/Mar/2016:08:41:00 +0100] "GET /files/basic/api/myfilesync/feed?page=1&pageSize=500&includeConflict=true HTTP/
1.1" 200 1323 "-" "IBM-LC-IBM Connections sync/1602.3033.1103 (Mac OS X 10.10.5)"
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "agent"
add_tag => [ "browser" ]
}
}
}
timestamp:
15/Mar/2016:08:41:00 +0100
… geoip.country_code3:
CHE
geoip.location:
8.298599999999993, 47.06030000000001
clientip :
84.74.43.46
…
agent:
"IBM-LC-IBM Connections sync/1602.3033.1103
(Mac OS X 10.10.5)"
os_name :
Mac OS X
name :
Other
os_major :
10
os_minor :
10

17. 17
Logstash

18. 18
Visualization of Logs
Gives you:
• Operational Visibility
Gain end-to-end visibility across your
operations and break down silos across your
infrastructure
• Search and Investigation
Find and fix problems, correlate events across
multiple data sources and automatically
detect patterns across massive sets of data
• Proactive Monitoring
Monitor systems in real time to identify
issues, problems and attacks before they
impact your customers, services and revenues
• Business Insights
Make better-informed business decisions by
understanding trends, patterns and gaining
operational intelligence from machine data
• IBM Solutions Log Management
• Centralized Log Management
• Security Monitoring
• Performance Monitoring
• Data Analysis
{

19. 19
Costs
• All ELK Stack products are
• Installation and configuration: Couple of days

20. 20
Lassen Sie uns
zusammen Gold
finden
Vielen Dank!