Presentation to php[tek]2018

1. EASY AS FALLING OFF A LOG
(OR WRITINGTO ONE)
Brent Laminack
brent@laminack.com

2. TOPICS
Why Log
Where to Log
Basic PHP Logging
Syslog
MonoLog
in Laravel
In MySQL
Log Catchers - Loggers

3. WHY LOG?
Immutable record of
what happened when
Audit Trail
Security/Forensics
Compliance
Performance
Debugging Complex Systems

4. WHERETO LOG?
Local file
Typically in /var/log
Pro:Very Easy
Problem: Multiple Files per Machine Makes Correlation Difficult
Problem++: Log Files on Different Machines
Makes it Even Harder
12-Factor App Says to Log to stdout: https://12factor.net/logs. I take issue.

5. BASIC PHP LOGGING
error_log
http://php.net/manual/en/function.error-log.php
Can write to:
file
email
syslog
Depends on error_log directive in php.ini

6. BETTER WAY:
CENTRALIZED LOGGING
Provides UnifiedView
More Secure
Easier Searching
Less Disk Space Management
IMPORTANT: ntp is your friend!
Central
Logging
Server
PHP MySQL
Apache Firewall

7. THE STANDARD: SYSLOG
The Old: https://tools.ietf.org/html/rfc3164
rfc3164 Written In 2001 BSD/Cisco which was obsoleted by
https://tools.ietf.org/html/rfc5424
rfc5424 from March 2009
Even Then, Not the Greatest RFC I’ve Ever Seen

8. SYSLOG CONCEPTS
WHO is saying something: Facility
HOW IMPORTANT it is: Severity
Combined they form the Priority
Network on Port 514 UDP
On Linux now interacts with systemd-journald

9. Numerical Code Facility
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem ← really?
8 UUCP subsystem ← really *= 2 ?
9 clock daemon ← NTP?
10 security/authorization messages ← deja vu?
11 FTP daemon

10. 12 NTP subsystem ← evidently clock != NTP
13 log audit
14 log alert
15 clock daemon (note 2) ← what?!? another clock? Where is note 2?!?
16 local use 0 (local0)
17 local use 1 (local1)
18 local use 2 (local2)
19 local use 3 (local3)
20 local use 4 (local4)
21 local use 5 (local5)
22 local use 6 (local6)
23 local use 7 (local7)

11. Numerical Code Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately ← isn't 'alert' a facility?
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages
lower number
=
higher importance
priority = facility * 8 + severity

12. LIMITATIONS
24 Facilities x 8 Severities = 192 Combinations of Messages
CAN’T Expand or Extend
Antiquated/Redundant Facilities
“syslog transport receivers need only support receiving up to and
including 480 octets”
“SHOULD be able to accept messages of up to and including 2048 octets”
Sucks

13. COUNTER-EXAMPLE
Forgot a ; in a larvel class
Entry in storage/logs/laravel.logs
15k+
[2018-02-25 17:10:51] laravel.EMERGENCY: Unable to create configured logger. Using emergency
logger. {"exception":"[object] (ParseError(code: 0): syntax error, unexpected
'$logger' (T_VARIABLE) at /var/www/vhosts/laminack.com/subdomains/demo/laraveldemo/app/
Logging/CreateCustomLogger.php:21)
[stacktrace]
#0 /var/www/vhosts/laminack.com/subdomains/demo/laraveldemo/vendor/composer/
ClassLoader.php(301): ComposerAutoloadincludeFile('/var/www/vhosts...')
#1 [internal function]: ComposerAutoloadClassLoader->loadClass('AppLoggingCre...')
#2 [internal function]: spl_autoload_call('AppLoggingCre...')
#3 /var/www/vhosts/laminack.com/subdomains/demo/laraveldemo/vendor/laravel/framework/src/
Illuminate/Container/Container.php(767): ReflectionClass->__construct('AppLogging
Cre...')

14. GELF - GRAYLOG EXTENDED
LOG FORMAT
Syslog++
Compressed
8K bytes
JSON Format
Pro:Wide Support, even in MonoLog
Con: Non-RFC, e.g. Non-Standard

15. COMMAND-LINE LOGGING
Some use nc
Better is logger.
Beware! Many distros ship with broken logger that won’t log to
remote machines
Best to compile yourself. util-linux-2.31
You know the drill: configure && make

16. APACHE LOGGING
<VirtualHost *:80>
ServerAdmin brent@laminack.com
DocumentRoot "/var/www/vhosts/laminack.com/subdomains/demo/laraveldemo/
public"
ServerName demo.laminack.com
ErrorLog "| /usr/local/bin/logger -d -n log.laminack.com -p local3.info"
CustomLog "| /usr/local/bin/logger -d -n log.laminack.com -p
local4.info" combined
</VirtualHost>
Apply Logger to Apache Logging
Note:Apache mod_syslog only logs ERROR, not ACCESS

17. MONOLOG PHP LIBRARY
https://github.com/Seldaek/
monolog
Probably most popular PHP
Logging Library
8,000+ stars
Built into Laravel

18. MONOLOG CLI EXAMPLE
<?php
include_once 'vendor/autoload.php';
use MonologLogger;
use MonologHandler;
use MonologHandlerSyslogUdpHandler;
putenv('HOSTNAME=demo.laminack.com');
$log = new Logger('testlog');
$remote_logger = 'log.laminack.com';
$log->pushHandler(new SyslogUdpHandler($remote_logger, 514, LOG_USER,
Logger::INFO, true, 'cronlogs'));
$log->info('testing 123 ' . date('r') );
?>

19. MONOLOG HANDLERS
Stream (e.g. file)
RotatingFileHandler (daily files)
SysLogHandler (local syslog)
GelfHandler (for GELF, obviously)
SqsHandler (for you AWS Types)
SyslogUdpHandler (what we’ll use)

20. MONOLOG CAN SENDTO:
HipChat
Slackbot
Slack Webhook
Mandrill
SendGrid
IFTTT

21. MONOLOGTO DATABASE
Redis
MongoDB
DynamoDB
ElisticSearch
CouchDB
MySQL via 3rd Party Handler

22. MYSQL ERRORSTO SYSLOG
Can’t write to remote syslog
Can write to local syslog
Local syslog daemon can forward
https://dev.mysql.com/doc/refman/5.7/en/error-log-syslog.html

23. MYSQL LOGGING
MySQL Can’t Write to Syslog
Can Write to Files and FIFOs
GRANT FILE ON *.* TO user;
A Long Way from Writing to a File to the Network
Doesn’t Work on Stock MySQL or MariaDB
https://bugs.mysql.com/bug.php?id=44835
Does Work in Persona
https://blueprints.launchpad.net/percona-server/+spec/into-outfile-pipe-and-socket

24. UNLESSTHE FILE ISN’T
We use a named pipe, a fifo:
Create via mknod
Acts like a regular pipe
But can be read from another process
prw-rw-rw- 1 root root 0 Feb 7 15:52 /var/lib/mysql-files/logpipe

25. READ FROMTHE FIFO,
WRITETO SYSLOG
putenv('HOSTNAME=database_machine');
$remote_logger = 'log.laminack.com';
$fifo = '/var/lib/mysql-files/logpipe';
// read from the fifo and write to the log
while(true){
// create a log channel
$log = new Logger('cronlog');
$log->pushHandler(new SyslogUdpHandler($remote_logger, 514,
LOG_USER, Logger::INFO, true, 'mysql_logs'));
if(!$fp = fopen($fifo, 'r')){
die("can't open $fifo for reading");
}
while($line = fgets($fp)){
$log->info($line);
}
fclose($fp);
}

26. KEEP IT GOING
$ cat /etc/init/send-to-syslog.conf
description "Read a fifo via monolog and send to a remote syslog server"
author "Brent Laminack"
start on startup
stop on shutdown
respawn
script
cd /home/brent/monolog; php -f ./fifolog.php
end script
SELECT 'This is a log message' INTO OUTFILE '/var/lib/mysql-files/logpipe';
Grand Finale:

27. SET UP LARAVEL
CUSTOM LOGGER
'channels' => [
'stack' => [
'driver' => 'stack',
'channels' => ['single', 'custom'],
],
…
/* custom logger per:
https://laravel.com/docs/5.6/logging#creating-custom-channels */
'custom' => [
'driver' => 'custom',
'via' => AppLoggingCreateCustomLogger::class,
in config/logging.php

28. THE CUSTOM LOGGER CLASS
<?php
namespace AppLogging;
use MonologLogger;
use MonologHandler;
use MonologHandlerSyslogUdpHandler;
class CreateCustomLogger
{
/**
* @param array $config
* @return MonologLogger
*/
public function __invoke(array $config)
{
$logger = new Logger('laravel_log');
$remote_logger = env('LOG_HOST', 'localhost');
$logger->pushHandler(new SyslogUdpHandler($remote_logger, 514, LOG_USER,
Logger::INFO, true, 'laravelogs'));
return $logger;
}
}

29. CALLINGTHE LOGGER
Route::get('/', function() {
Log::info('someone just hit the demo root endpoint');
return 'this is a demo ' . date('r');
});

30. TO WHERE SHALL WE SYSLOG?
Separate Machine
Hardened for Security
Specialized Logging/Reporting
Software
I like Open Source Solutions
Maybe with Commercial Support
Central
Logging
Server
PHP
Monolog
MySQL
fifo & Monolog
Apache
Logging
Firewall
Profile

31. LOGGING SOFTWARE FEATURES
Store
Search
Graph
Report
Whole New
Generation
Elastisearch
Web Interface
APIs

32. OPEN SOURCE
LOGGING SOFTWARE
https://www.loggly.com/
http://www.logzilla.net
https://www.graylog.org/
https://www.elastic.co/products/
logstash
All Have Commercial
Support Options

33. COMMERCIAL LOGGING
https://papertrailapp.com/
https://logz.io/
https://timber.io/
https://logmatic.io
https://www.sumologic.com/

34. GRAYLOG SEARCHING
Loggers: Basically Search Engines
Time Frame
Relative/Absolute/Keyword
What To Look For
Fields/Values/Booleans
Origin
Single System/Group/All

35. MORE SEARCHING
Multiple Words default to OR
“Exact Phrase”
AND OR and NOT work (CAPITALS!!!)
Wildcards (not leading!)
>, <, <=, >=
Fuzziness: HTP~ via Levenshtein

36. EXTRACTORS
Helps parse data into searchable fields
Can be RegEx
or GROK Patterns DANGER: only returns STRINGS!
or JSON
or Key=Value pairs

37. OTHERTOPICS
Stability
Clustering
Log Retention
Alerts
Notifications

38. DEMOS

39. Q&A

40. NOTES
Graylog AMI
http://docs.graylog.org/en/2.4/pages/installation/aws.htmls
Overview of Logging Protocols:
https://github.com/evanphx/eventd-rfc
GELF and Docker - harder than you thought
https://blog.docker.com/2017/02/adventures-in-gelf/
Basic Graylog Setup
https://blog.no42.org/post/centralized-logging-graylog2/