SlideShare ist ein Scribd-Unternehmen logo
1 von 40
Downloaden Sie, um offline zu lesen
Application Logging
With Logstash
Ben Waine
• Worked With
PHP For 5Years
• Software
Engineer -
Sainsbury’s
• Dabbles in
devops
https://joind.in/talk/view/13369
System Logs
Application Log
Debug Information - Errors (connections,
uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls,
Event Triggers
Business Events - Purchases, Logins,
Registrations, Unsubscribes
Keeping Track Of All This....
ssh webserver@mydomain.net
tail -f /var/log/nginx/my-site.access.log
tail -f /var/log/my.application.log
ssh data@mydomain.net
tail -f /var/log/mysql/mysql.log
ssh q@mydomain.net
tail -f /var/log/rabbitmq/nodename.log
The Elk Stack
Visualizing Log Data
PHP Logging Tools
1) Monolog
2) Everything else....
Basic Logging Examples
1) Monolog: Loggers And Handlers
2) Monolog:Tags & Formatters
3) Logging business events
use MonologLogger;
use MonologHandlerFingersCrossedHandler;
use MonologHandlerStreamHandler;
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);
$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
// Set A Log Level
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger
$appLog = new Logger('AppLog');
$strHandler
= new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler
= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');
$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
EG 2:Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
// Create A Logger
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
Log Levels
2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424
Log Levels
2013 - PSR03 - PHP Logging Interface Standard
Phrase / Severity
emergency Emergency: system is unusable
alert Alert: action must be taken immediately
critical Critical: critical conditions
error Error: error conditions
warning Warning: warning conditions
notice Notice: normal but significant condition
info Informational: informational messages
debug Debug: debug-level messages
http://www.php-fig.org/psr/psr-3/
http://imgs.xkcd.com/comics/standards.png
EG 3: Event Logging
use MonologLogger;
use SymfonyComponentEventDispatcherEventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener(
"business.registration.post",
function () use ($busLog) {
$busLog−>info("Customer registered");
}
);
$dispatcher−>dispatch("business.registration.post");
Logstash Architecture
1. Logstash Shipper ships logs to
logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic
Search
4. Kibana exposes a web interface
to Elastic Search data
Logstash Architecture
https://joind.in/talk/view/13369
Why not rate the talk now BEFORE
the demo?
Logstash Demo
https://github.com
/LoveSoftware/
application-logging-with-logstash
Application Logging With Logstash
Application Logging With Logstash
Application Logging With Logstash
Application Logging With Logstash
Application Logging With Logstash
Logstash Config
Logstash Collecting
{
"network": {
"servers": [ "logs.logstashdemo.com:5000" ],
"timeout": 15,
"ssl ca":
"/etc/pki/tls/certs/logstash−forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/nginx/helloapp.access.log"
],
"fields": { "type": "nginx−access" }
}
]
}
Logstash Processing
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate =>
"/etc/pki/tls/certs/logstash−forwarder.crt"
ssl_key =>
"/etc/pki/tls/private/logstash−forwarder.key"
}
}
Input
Logstash Processing
Filtering
filter {
if [type] == "nginx−access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
Logstash Processing
Output
output {
elasticsearch { host => localhost }
}
Groking
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}
%{WORD:method}
%{URIPATHPARAM:request}
%{NUMBER:bytes}
%{NUMBER:duration}
Logging Ideas
Release Marker
Error rates of various applications over time
Latency in various percentiles of each application tier
HTTP Responses: 400 series responses
HTTP Responses: 500 series responses
Auto git blame production errors
Auth and Syslogs
Go Forth And Log....
BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS
Questions?
https://joind.in/talk/view/13369

Weitere ähnliche Inhalte

Was ist angesagt?

{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4琛琳 饶
 
Managing Your Security Logs with Elasticsearch
Managing Your Security Logs with ElasticsearchManaging Your Security Logs with Elasticsearch
Managing Your Security Logs with ElasticsearchVic Hargrave
 
Logstash family introduction
Logstash family introductionLogstash family introduction
Logstash family introductionOwen Wu
 
Life of an Fluentd event
Life of an Fluentd eventLife of an Fluentd event
Life of an Fluentd eventKiyoto Tamura
 
Ansible tips & tricks
Ansible tips & tricksAnsible tips & tricks
Ansible tips & tricksbcoca
 
Aaron Mildenstein - Using Logstash with Zabbix
Aaron Mildenstein - Using Logstash with ZabbixAaron Mildenstein - Using Logstash with Zabbix
Aaron Mildenstein - Using Logstash with ZabbixZabbix
 
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0Zabbix
 
Fluentd - CNCF Paris
Fluentd - CNCF ParisFluentd - CNCF Paris
Fluentd - CNCF ParisHorgix
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with phpElizabeth Smith
 
More than syntax
More than syntaxMore than syntax
More than syntaxWooga
 
Monitoring with Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring with Syslog and EventMachine (RailswayConf 2012)Wooga
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUNCong Zhang
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talkLocaweb
 
Using Ansible Dynamic Inventory with Amazon EC2
Using Ansible Dynamic Inventory with Amazon EC2Using Ansible Dynamic Inventory with Amazon EC2
Using Ansible Dynamic Inventory with Amazon EC2Brian Schott
 

Was ist angesagt? (20)

Elk stack
Elk stackElk stack
Elk stack
 
Fluentd meetup #2
Fluentd meetup #2Fluentd meetup #2
Fluentd meetup #2
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4
 
Webscraping with asyncio
Webscraping with asyncioWebscraping with asyncio
Webscraping with asyncio
 
Managing Your Security Logs with Elasticsearch
Managing Your Security Logs with ElasticsearchManaging Your Security Logs with Elasticsearch
Managing Your Security Logs with Elasticsearch
 
Logstash family introduction
Logstash family introductionLogstash family introduction
Logstash family introduction
 
Life of an Fluentd event
Life of an Fluentd eventLife of an Fluentd event
Life of an Fluentd event
 
Elastic stack
Elastic stackElastic stack
Elastic stack
 
Puppet Camp 2012
Puppet Camp 2012Puppet Camp 2012
Puppet Camp 2012
 
Ansible tips & tricks
Ansible tips & tricksAnsible tips & tricks
Ansible tips & tricks
 
Aaron Mildenstein - Using Logstash with Zabbix
Aaron Mildenstein - Using Logstash with ZabbixAaron Mildenstein - Using Logstash with Zabbix
Aaron Mildenstein - Using Logstash with Zabbix
 
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
 
Fluentd - CNCF Paris
Fluentd - CNCF ParisFluentd - CNCF Paris
Fluentd - CNCF Paris
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
 
More than syntax
More than syntaxMore than syntax
More than syntax
 
Monitoring with Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring with Syslog and EventMachine (RailswayConf 2012)
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUN
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
 
Using Ansible Dynamic Inventory with Amazon EC2
Using Ansible Dynamic Inventory with Amazon EC2Using Ansible Dynamic Inventory with Amazon EC2
Using Ansible Dynamic Inventory with Amazon EC2
 

Andere mochten auch

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
Monitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAMonitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAACA IT-Solutions
 
MySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELKMySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELKYoungHeon (Roy) Kim
 
Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Ronny López
 
Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoringVinay Krishna
 
Fluentd and PHP
Fluentd and PHPFluentd and PHP
Fluentd and PHPchobi e
 
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchFrom Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchSematext Group, Inc.
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaAmazee Labs
 

Andere mochten auch (10)

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
 
Monitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSAMonitor your Atlassian stack like the NSA
Monitor your Atlassian stack like the NSA
 
MySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELKMySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELK
 
Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2Integrando Redis en aplicaciones Symfony2
Integrando Redis en aplicaciones Symfony2
 
Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoring
 
Fluentd and PHP
Fluentd and PHPFluentd and PHP
Fluentd and PHP
 
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & ElasticsearchFrom Zero to Hero - Centralized Logging with Logstash & Elasticsearch
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
 
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementFluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log Management
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
 

Ähnlich wie Application Logging With Logstash

TDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensTDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensJackson F. de A. Mafra
 
Apache2 BootCamp : Logging and Monitoring
Apache2 BootCamp : Logging and MonitoringApache2 BootCamp : Logging and Monitoring
Apache2 BootCamp : Logging and MonitoringWildan Maulana
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsGraham Dumpleton
 
Php through the eyes of a hoster
Php through the eyes of a hosterPhp through the eyes of a hoster
Php through the eyes of a hosterCombell NV
 
Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Combell NV
 
Software Engineering - RS4
Software Engineering - RS4Software Engineering - RS4
Software Engineering - RS4AtakanAral
 
PSR-3 logs using Monolog and Graylog
PSR-3 logs using Monolog and Graylog PSR-3 logs using Monolog and Graylog
PSR-3 logs using Monolog and Graylog OCoderFest
 
Review unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpReview unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpDamien Seguy
 
Php through the eyes of a hoster: PHPNW10
Php through the eyes of a hoster: PHPNW10Php through the eyes of a hoster: PHPNW10
Php through the eyes of a hoster: PHPNW10Combell NV
 
Introduction to web and php mysql
Introduction to web and php mysqlIntroduction to web and php mysql
Introduction to web and php mysqlProgrammer Blog
 
The why and how of moving to php 8
The why and how of moving to php 8The why and how of moving to php 8
The why and how of moving to php 8Wim Godden
 
Art of Web Backdoor - Pichaya Morimoto
Art of Web Backdoor - Pichaya MorimotoArt of Web Backdoor - Pichaya Morimoto
Art of Web Backdoor - Pichaya MorimotoPichaya Morimoto
 
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry Pi
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry PiGrâce aux tags Varnish, j'ai switché ma prod sur Raspberry Pi
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry PiJérémy Derussé
 
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce Diane Mueller
 
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin
 
Troubleshooting Plone
Troubleshooting PloneTroubleshooting Plone
Troubleshooting PloneRicado Alves
 
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)James Titcumb
 
Orange@php conf
Orange@php confOrange@php conf
Orange@php confHash Lin
 

Ähnlich wie Application Logging With Logstash (20)

TDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensTDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit Happens
 
Apache2 BootCamp : Logging and Monitoring
Apache2 BootCamp : Logging and MonitoringApache2 BootCamp : Logging and Monitoring
Apache2 BootCamp : Logging and Monitoring
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Php through the eyes of a hoster
Php through the eyes of a hosterPhp through the eyes of a hoster
Php through the eyes of a hoster
 
Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11
 
Php logging
Php loggingPhp logging
Php logging
 
Software Engineering - RS4
Software Engineering - RS4Software Engineering - RS4
Software Engineering - RS4
 
PSR-3 logs using Monolog and Graylog
PSR-3 logs using Monolog and Graylog PSR-3 logs using Monolog and Graylog
PSR-3 logs using Monolog and Graylog
 
Review unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpReview unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphp
 
Php through the eyes of a hoster: PHPNW10
Php through the eyes of a hoster: PHPNW10Php through the eyes of a hoster: PHPNW10
Php through the eyes of a hoster: PHPNW10
 
Introduction to web and php mysql
Introduction to web and php mysqlIntroduction to web and php mysql
Introduction to web and php mysql
 
The why and how of moving to php 8
The why and how of moving to php 8The why and how of moving to php 8
The why and how of moving to php 8
 
Art of Web Backdoor - Pichaya Morimoto
Art of Web Backdoor - Pichaya MorimotoArt of Web Backdoor - Pichaya Morimoto
Art of Web Backdoor - Pichaya Morimoto
 
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry Pi
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry PiGrâce aux tags Varnish, j'ai switché ma prod sur Raspberry Pi
Grâce aux tags Varnish, j'ai switché ma prod sur Raspberry Pi
 
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce
 
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...
 
Troubleshooting Plone
Troubleshooting PloneTroubleshooting Plone
Troubleshooting Plone
 
Php manish
Php manishPhp manish
Php manish
 
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
 
Orange@php conf
Orange@php confOrange@php conf
Orange@php conf
 

Mehr von benwaine

DPC 2016 - 53 Minutes or Less - Architecting For Failure
DPC 2016 - 53 Minutes or Less - Architecting For FailureDPC 2016 - 53 Minutes or Less - Architecting For Failure
DPC 2016 - 53 Minutes or Less - Architecting For Failurebenwaine
 
The Road To Technical Team Lead
The Road To Technical Team LeadThe Road To Technical Team Lead
The Road To Technical Team Leadbenwaine
 
PHPNW14 - Getting Started With AWS
PHPNW14 - Getting Started With AWSPHPNW14 - Getting Started With AWS
PHPNW14 - Getting Started With AWSbenwaine
 
Business selectors
Business selectorsBusiness selectors
Business selectorsbenwaine
 
The Art Of Application Logging PHPNW12
The Art Of Application Logging PHPNW12The Art Of Application Logging PHPNW12
The Art Of Application Logging PHPNW12benwaine
 
Behat dpc12
Behat dpc12Behat dpc12
Behat dpc12benwaine
 
Acceptance & Integration Testing With Behat (PBC11)
Acceptance & Integration Testing With Behat (PBC11)Acceptance & Integration Testing With Behat (PBC11)
Acceptance & Integration Testing With Behat (PBC11)benwaine
 
Acceptance & Integration Testing With Behat (PHPNw2011)
Acceptance & Integration Testing With Behat (PHPNw2011)Acceptance & Integration Testing With Behat (PHPNw2011)
Acceptance & Integration Testing With Behat (PHPNw2011)benwaine
 
Say no to var_dump
Say no to var_dumpSay no to var_dump
Say no to var_dumpbenwaine
 

Mehr von benwaine (9)

DPC 2016 - 53 Minutes or Less - Architecting For Failure
DPC 2016 - 53 Minutes or Less - Architecting For FailureDPC 2016 - 53 Minutes or Less - Architecting For Failure
DPC 2016 - 53 Minutes or Less - Architecting For Failure
 
The Road To Technical Team Lead
The Road To Technical Team LeadThe Road To Technical Team Lead
The Road To Technical Team Lead
 
PHPNW14 - Getting Started With AWS
PHPNW14 - Getting Started With AWSPHPNW14 - Getting Started With AWS
PHPNW14 - Getting Started With AWS
 
Business selectors
Business selectorsBusiness selectors
Business selectors
 
The Art Of Application Logging PHPNW12
The Art Of Application Logging PHPNW12The Art Of Application Logging PHPNW12
The Art Of Application Logging PHPNW12
 
Behat dpc12
Behat dpc12Behat dpc12
Behat dpc12
 
Acceptance & Integration Testing With Behat (PBC11)
Acceptance & Integration Testing With Behat (PBC11)Acceptance & Integration Testing With Behat (PBC11)
Acceptance & Integration Testing With Behat (PBC11)
 
Acceptance & Integration Testing With Behat (PHPNw2011)
Acceptance & Integration Testing With Behat (PHPNw2011)Acceptance & Integration Testing With Behat (PHPNw2011)
Acceptance & Integration Testing With Behat (PHPNw2011)
 
Say no to var_dump
Say no to var_dumpSay no to var_dump
Say no to var_dump
 

Kürzlich hochgeladen

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 

Kürzlich hochgeladen (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 

Application Logging With Logstash

  • 2. Ben Waine • Worked With PHP For 5Years • Software Engineer - Sainsbury’s • Dabbles in devops
  • 5. Application Log Debug Information - Errors (connections, uncaught exceptions, resource exhaustion) Narrative Information - Methods Calls, Event Triggers Business Events - Purchases, Logins, Registrations, Unsubscribes
  • 6. Keeping Track Of All This.... ssh webserver@mydomain.net tail -f /var/log/nginx/my-site.access.log tail -f /var/log/my.application.log ssh data@mydomain.net tail -f /var/log/mysql/mysql.log ssh q@mydomain.net tail -f /var/log/rabbitmq/nodename.log
  • 9. PHP Logging Tools 1) Monolog 2) Everything else....
  • 10. Basic Logging Examples 1) Monolog: Loggers And Handlers 2) Monolog:Tags & Formatters 3) Logging business events
  • 11. use MonologLogger; use MonologHandlerFingersCrossedHandler; use MonologHandlerStreamHandler; $logEnv = getenv('LOG_LEVEL'); $level = empty($logLevel) ? $logEnv : Logger::WARNING; $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level); $appLog−>pushHandler($fcHandler); $appLog−>debug('LOGGING!'); EG1: Loggers And Handlers
  • 12. // Set A Log Level $logEnv = getenv('LOG_LEVEL'); $level = empty($logLevel) ? $logEnv : Logger::WARNING; // Create A Logger $appLog = new Logger('AppLog');
  • 13. $strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level); // Create Handlers $appLog−>pushHandler($fcHandler); $appLog−>debug('Start Logging!'); $appLog−>emergency('Something Terrible Happened'); // Push The Handler And Start Logging
  • 14. EG 2:Tagging Formatting $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/lg.lg', $level); $formatter = new LogstashFormatter("helloapp", "application"); $strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler)); $id = $_SERVER('X_VARNISH'); $tag = new TagProcessor(['request−id' => $id]) $appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
  • 15. // Create A Logger $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/lg.lg', $level); $formatter = new LogstashFormatter("helloapp", "app"); // Create A Handler & Formatter // Set Formatter Onto Handler $strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler)); //Push Handler Onto Logger
  • 16. $id = $_SERVER('X_VARNISH'); $tag = new TagProcessor(['request−id' => $id]) $appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!"); // Capture A Unique Id, Create A Tag Processor, Push
  • 17. Log Levels 2009 - RFC 5424 - Syslog Protocol Code / Severity 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages https://tools.ietf.org/html/rfc5424
  • 18. Log Levels 2013 - PSR03 - PHP Logging Interface Standard Phrase / Severity emergency Emergency: system is unusable alert Alert: action must be taken immediately critical Critical: critical conditions error Error: error conditions warning Warning: warning conditions notice Notice: normal but significant condition info Informational: informational messages debug Debug: debug-level messages http://www.php-fig.org/psr/psr-3/
  • 20. EG 3: Event Logging use MonologLogger; use SymfonyComponentEventDispatcherEventDispatcher; $dispatcher = new EventDispatcher(); $dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); } ); $dispatcher−>dispatch("business.registration.post");
  • 21. Logstash Architecture 1. Logstash Shipper ships logs to logstash 2. Logstash processes them 3. Logstash Inserts Into Elastic Search 4. Kibana exposes a web interface to Elastic Search data
  • 23. https://joind.in/talk/view/13369 Why not rate the talk now BEFORE the demo?
  • 32. Logstash Collecting { "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }
  • 33. Logstash Processing input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key" } } Input
  • 34. Logstash Processing Filtering filter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }
  • 36. Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns http://grokdebug.herokuapp.com/ 55.3.244.1 GET /index.html 15824 0.043 %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
  • 37. Logging Ideas Release Marker Error rates of various applications over time Latency in various percentiles of each application tier HTTP Responses: 400 series responses HTTP Responses: 500 series responses Auto git blame production errors Auth and Syslogs
  • 38. Go Forth And Log.... BUT Remember log rotation Beware running out of space Beware file logging on NFS