Dylan Ferreira from FuseMail will share how they use the Syslog Telegraf plugin to help them troubleshoot their systems faster and with more success. Dylan will go over how to set up Rsyslog and Telegraf to filter logs then configure Kapacitor to help you look for interesting things in your raw logs to trigger alerts to your team. He will then bring this all together in a dashboard for your teams to use.

1. Finding OOMs with Telegraf
Dylan Ferreira
@dylanferreira

3. Agenda
● A whole bunch of dry stuff
● Maybe snacks?

4. Agenda.real
● Our TICK stack layout
● Configuring Rsyslog & Telegraf
● Doing more with Telegraf plugins
● Using Kapacitor to search &
sanitize log data
● Exploring this data with Grafana

5. How did we get here?
● Multi-dimensional TSDB.
● Very high performance.
● All data is aggregated on ingest.
● Can store raw metrics.
● Multi-dimensional TSDB.
● Complex aggregations (CQs).
● Multiple fields!
● Can store raw metrics.
● Flat structure with metadata stored
in the metric name (pre-v1).
● Fixed per-series aggregations

6. What is Telegraf?
“Telegraf is InfluxData's open source
plugin-driven server agent for collecting
and reporting metrics.”
● Inputs
● Processors
● Aggregators
● Outputs
● Stream Buffer
●● Metrics Router

7. TICK Stack Layout

8. Applications
Host Setup
Applications sending log data to syslog
Syslog relaying log data to Telegraf

9. Applications
Applications
Applications
Single DC Setup

10. Multi DC Setup
DC1
DC2

11. Syslog into Telegraf

12. The OOM Killer
/**
* oom_badness - heuristic function to determine which candidate task to kill
* @p: task struct of which task we should calculate
* @totalpages: total present RAM allowed for page allocation
* @memcg: task's memory controller, if constrained
* @nodemask: nodemask passed to page allocator for mempolicy ooms
*
* The heuristic for determining which task to kill is made to be as simple and
* predictable as possible. The goal is to return the highest value for the
* task consuming the most memory to avoid subsequent oom failures.
*/

13. Armouring Telegraf Against the OOM Killer
[Service]
OOMScoreAdjust=-600
/etc/systemd/system/telegraf.service.d/oom_score_adj.conf
value between -1000 and +1000/proc/$PID/oom_score_adj
/proc/$PID/oom__adj value between -17 and +15

14. Telegraf Config
[[inputs.syslog]]
server = "tcp://:6514"
[[outputs.influxdb]]
urls = [ "http://influxdb-syslog.service.consul:8086" ]
database = "syslog"
retention_policy = "autogen"
precision = "ns"
user_agent = "telegraf"
namepass = ["syslog"]

15. Rsyslog Config
$ActionQueueType LinkedList
$ActionQueueFileName telegraf
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
# forward over tcp with octet framing according to RFC 5425
*.* @@(o)localhost:6514;RSYSLOG_SyslogProtocol23Format
# all logs that contain the string "**WARNING**"
if ($msg contains '**WARNING**') then
@@(o)localhost:6514;RSYSLOG_SyslogProtocol23Format

16. What do you get?

17. Telegraf Output
● syslog
○ tags
■ severity (string)
■ facility (string)
■ hostname (string)
■ appname (string)
○ fields
■ version (integer)
■ severity_code (integer)
■ facility_code (integer)
■ timestamp (integer): the time recorded in the syslog message
■ procid (string)
■ msgid (string)
■ sdid (bool)
■ Structured Data (string)
○ timestamp: the time the messages was received
Original timestamp
Timestamp given by Telegraf

18. Timestamps & Ingest Latency
var data = stream
|from()
.database('syslog')
.retentionPolicy('autogen')
.measurement('syslog')
|groupBy('hostname')
|window()
.align()
.period(1m)
.every(1m)
|eval(lambda: unixNano("time") - "timestamp")
.as('timestamp_lag_ns')
var mean_latency = data
|mean('timestamp_lag_ns')
.as('val')
mean_latency
|log()
Typical Latency: under 1ms

19. Getting more from Telegraf

20. Enable Counters
# Create per-minute counts of syslog data
[[aggregators.basicstats]]
period = "1m"
drop_original = false
stats = ["count"]
name_suffix = "_counts"
# filtering
namepass = ["syslog"]

23. Adding More Tags From Your Log Data
[[processors.parser]]
parse_fields = ["message"]
drop_original = false
merge = "override"
data_format = "json"
tag_keys = ["method","uri"]
namepass = ["syslog"]
[processors.parser.tagpass]
appname = ["crazy-cool-api"]
{
"hostname": "ab704bf3336f" ,
"ip": "10.0.0.30",
"method": "GET",
"msg": "request completed" ,
"start": "2018-10-23T07:00:03Z" ,
"status": 200,
"time": "2018-10-23T07:00:03.942327Z" ,
"uri": "/health",
"uri_query": "",
"user_agent": "Consul Health Check"
}
Log Message Telegraf Config

25. Data Sanitization

26. var ipv4_address = /d{1,3}.d{1,3}.d{1,3}.d{1,3}/
|eval(lambda: regexReplace(ipv4_address, "message", '<ipv4-address>'))
.as('message')
.keep()
Rewriting your data with regexReplace
var email_address = /w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/
|eval(lambda: regexReplace(email_address, "message", '<email-address>'))
.as('message')
.keep(
Email Addresses
IP Addresses

27. Email Address Regex (RFC 5322)
(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[
x01-x08x0bx0cx0e-x1fx21x23-x5bx5d-x7f]|[x01-x09x0bx0cx0e
-x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*
[a-z0-9])?|[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|
2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[x01-x08x0bx0cx0e
-x1fx21-x5ax53-x7f]|[x01-x09x0bx0cx0e-x7f])+)])

28. Email Address Regex / Pain Amplifier
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?
[t])*))*>(?:(?:rn)?[t])*)|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:
rn)?[t])*)*:(?:(?:rn)?[t])*(?:(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]
))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn
)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*|
(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t]
)*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^(
)<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:"
.[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-0
31]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?:
(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?
:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+
(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:r
n)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*>(?:(?:rn)?[t])*)(?:,s*(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)
?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:
rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?
:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])
+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|
"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t])*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[
]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^
[]r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|
.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:
rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]
))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn
)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*
>(?:(?:rn)?[t])*))*)?;s*)

29. Finding Events
With Kapacitor

30. Tick Template
var where_filter = lambda: TRUE
var appname_lambda lambda
var event_type string
var data = stream
|from()
.database('syslog')
.retentionPolicy('autogen')
.measurement('syslog')
|where(lambda: "facility" == 'kern')
|where(where_filter)
|eval(appname_lambda)
.as('appname')
.tags('appname')
.keep()
|eval(lambda: regexReplace(/w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/, "message", '<email-address>'))
.as('message')
.keep()
|eval(lambda: unixNano("time") - "timestamp")
.as('timestamp_lag_ns')
.keep()
|log()
/etc/kapacitor/load/templates/template-syslog_kern_service_events.tick

31. Task Definition : OOM
template-id: template-syslog_kern_service_events
vars:
Where_filter:
type: lambda
value: >
"message" =~ /.+ Memory cgroup out of memory: Kill process .+/
appname_lambda:
type: lambda
value: >
regexReplace(/^[.+] Memory cgroup out of memory: Kill process .+? ((.+?)) score .+/, "message", '$1')
event_type:
type: string
value: "OOM"
/etc/kapacitor/load/tasks/syslog_kern_service_events-OOM.yaml

32. Task Definition : Segfault
/etc/kapacitor/load/tasks/syslog_kern_service_events-segfault.yaml
template-id: template-syslog_kern_service_events
vars:
Where_filter:
type: lambda
value: >
"message" =~ /.+: segfault at .+/
appname_lambda:
type: lambda
value: >
regexReplace(/^[.+] (.+?)[.+]: segfault at .*/, "message", '$1')
event_type:
type: string
value: "segfault"

33. Memory cgroup out of memory: Kill process 3056 ( upstart-socket-) score 1057 or sacrifice childn
The kernel truncates process names down to 15 chars (16 chars -1 NUL)
and stores this in /proc/<PID>/comm
include/linux/sched.h
/* Task command name length: */
#define TASK_COMM_LEN 16
e.g. upstart-socket-bridge becomes upstart-socket-
The Kernel & Process Names

34. metric_relabel_configs:
- source_labels: ['name']
regex: '([[:ascii:]]{1,15}).*)'
target_label: 'name_short'
replacement: '$1'
Linkage Workarounds
Prometheus
[[processors.regex]]
[[processors.regex.tags]]
key = "<tag>"
pattern = "^([[:ascii:]]{1,15}).*$"
replacement = "${1}"
result_key = "short_<tag>"
Telegraf

35. Dashboards

37. Ad-hoc Variable

38. Group By Using A Custom Variable
GROUP BY time(1m), "[[group]]" fill(0)

39. Regex Search Using A Constant
WHERE ("message" =~ / $search/)

40. Annotations

41. Questions
Dylan Ferreira
@dylanferreira