Telegraf is an agent for collecting, processing, aggregating, and writing metrics. With over 200 plugins, Telegraf can fetch metrics from a variety of sources, allowing you to build aggregations and write those metrics to InfluxDB, Prometheus, Kafka, and more. In this talk, we will take a look at some of the lesser known, but awesome, plugins that are often overlooked; as well as how to use Telegraf for monitoring of Cloud Native systems.

1. The Telegraf Toolbelt
It can do that, really?

2. © 2019 InfluxData. All rights reserved.2
David McKay
Developer Advocate
at InfluxData
@rawkode
🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scottish
🐭 Has 9 Pets
💙 Esoteric Programming Languages
☸ Kubernetes Release Team
🤔 Stoic

3. © 2019 InfluxData. All rights reserved.3

4. © 2019 InfluxData. All rights reserved.4
Introduction Demo

5. © 2019 InfluxData. All rights reserved.5
Telegraf is plugin based

6. © 2019 InfluxData. All rights reserved.6 © 2019 InfluxData. All rights reserved.6
Over 200 Plugins
➔ 169 Inputs
➔ 35 Outputs
➔ 15 Processors
➔ 14 Parsers
➔ 9 Serializers
➔ 8 Aggregators

7. © 2019 InfluxData. All rights reserved.7
➔ Outputs
◆ Google Cloud PubSub
➔ Serializers
◆ Nowmetric
◆ Carbon2
V1.10 (May)
➔ Inputs
◆ Google Cloud PubSub
◆ Kinesis Consumer
◆ Kube Inventory
◆ Neptune Apex
◆ Nginx Upstream Checks
◆ Multifile
◆ Stack Driver

8. © 2019 InfluxData. All rights reserved.8
➔ Outputs
◆ Health
◆ Syslog
➔ Serializers
◆ Wavefront
➔ Aggregators
◆ Final
V1.11 (June)
➔ Inputs
◆ bind9
◆ Cisco GNMI
◆ Cisco MDT
◆ ECS & Fargate
◆ GitHub
◆ OpenWeatherMap
◆ PowerDNS

9. © 2019 InfluxData. All rights reserved.9
V1.12 (September)
➔ Inputs
◆ apcupsd
◆ Docker Logs
◆ Fireboard
◆ Logstash
◆ MarkLogic
◆ OpenNTPD
◆ uWSGI
➔ Outputs
◆ Exec
➔ Parsers
◆ Form
➔ Processors
◆ Date
◆ Pivot
◆ Unpivot
◆ Tag Limit

10. © 2019 InfluxData. All rights reserved.10

11. © 2019 InfluxData. All rights reserved.11
There are 3249 telegraf.conf
files on GitHub

12. © 2019 InfluxData. All rights reserved.12

13. © 2019 InfluxData. All rights reserved.13
I used a sample of 1000
telegraf.conf files from GitHub

14. © 2019 InfluxData. All rights reserved.14 © 2019 InfluxData. All rights reserved.14
Output Plugins
➔ 72% InfluxDB
➔ 5% File
➔ 2% Prometheus Client
➔ .9% Graphite
➔ .6% Kafka

15. © 2019 InfluxData. All rights reserved.15
Interval
73% Use 10s (Default)
5.6% use 1s
4% use 5s
2% use 1m
1% use 30s

16. © 2019 InfluxData. All rights reserved.16
Round
Interval
90% True

17. © 2019 InfluxData. All rights reserved.17
Jitter 90% None

18. © 2019 InfluxData. All rights reserved.18
Omit
Hostname
90% False

19. © 2019 InfluxData. All rights reserved.19 © 2019 InfluxData. All rights reserved.19
Output Plugins
➔ 71% 1 Output
➔ 5% 2 Outputs
➔ 2% 0 Outputs
➔ .6% 3 Outputs

20. © 2019 InfluxData. All rights reserved.20 © 2019 InfluxData. All rights reserved.20
Input Plugins
➔ 17% 1 Input
➔ 12% 9 Inputs
➔ 10% 8 Inputs
➔ 5% 10 Inputs
➔ 5% 11 Inputs
➔ 5% 6 Inputs
➔ 5% 7 Inputs
➔ 1 56 Inputs ?!?!

21. © 2019 InfluxData. All rights reserved.21 © 2019 InfluxData. All rights reserved.21
Input Plugins
➔ 58% CPU
➔ 53% Mem
➔ 52% Disk
➔ 51% System
➔ 47% DiskIO
➔ 47% Swap
➔ 40% Process
➔ 31% Kernel
➔ 28% Docker
➔ 23% Net

22. © 2019 InfluxData. All rights reserved.22
Multiple Outputs

23. © 2019 InfluxData. All rights reserved.23 © 2019 InfluxData. All rights reserved.23
Multiple Outputs
[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
[[outputs.influxdb_v2]]
urls = ["http://influxdb2:9999"]

24. © 2019 InfluxData. All rights reserved.24
Remote Configuration

25. © 2019 InfluxData. All rights reserved.25 © 2019 InfluxData. All rights reserved.25
Remote Configuration
telegraf --config <http_uri>

26. © 2019 InfluxData. All rights reserved.26 © 2019 InfluxData. All rights reserved.26
Remote Configuration
telegraf --config
https://raw.githubusercontent.com
/influxdata/telegraf/master/etc/telegraf.conf

27. © 2019 InfluxData. All rights reserved.27 © 2019 InfluxData. All rights reserved.27
Remote Configuration
SOME_VAR=abc123 telegraf --config <http_uri>

28. © 2019 InfluxData. All rights reserved.28 © 2019 InfluxData. All rights reserved.28
Remote Configuration
[agent]
interval = “${INTERVAL}”
[[outputs.influxdb_v2]]
token = “${TOKEN}”

29. © 2019 InfluxData. All rights reserved.29
Output Resiliency

30. © 2019 InfluxData. All rights reserved.30 © 2019 InfluxData. All rights reserved.30
Output Resiliency
Use
metric_buffer_limit
to handle downtime of your outputs

31. © 2019 InfluxData. All rights reserved.31
InfluxDB
Telegraf
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Buffer Batch

32. © 2019 InfluxData. All rights reserved.32
Telegraf
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Buffer Batch

33. © 2019 InfluxData. All rights reserved.33
InfluxDB
Telegraf
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Buffer Batch

34. © 2019 InfluxData. All rights reserved.34 © 2019 InfluxData. All rights reserved.34
Output Resiliency
influxdb_listener
http_listener

35. © 2019 InfluxData. All rights reserved.35 © 2019 InfluxData. All rights reserved.35
influxdb_listener
Allows Telegraf to serve as a proxy for the /write endpoint of the
InfluxDB HTTP API.
Output Resiliency

36. © 2019 InfluxData. All rights reserved.36
influxdb_listener
[[inputs.influxdb_listener]]
service_address = ":8086"

37. © 2019 InfluxData. All rights reserved.37

38. © 2019 InfluxData. All rights reserved.38
v1 Client Libraries

39. © 2019 InfluxData. All rights reserved.39
http_listener_v2
Allows Telegraf to accept metrics over HTTP in any supported
format
Output Resiliency

40. © 2019 InfluxData. All rights reserved.40
http_listener_v2
[[inputs.http_listener_v2]]
service_address = ":8080"
data_format = "json"

41. © 2019 InfluxData. All rights reserved.41
JSON
[[inputs.http_listener_v2]]
data_format = "json"
json_name_key = "name"
tag_keys = [“go_version”]
Docs

42. © 2019 InfluxData. All rights reserved.42
Bring Your Own Telegraf

43. © 2019 InfluxData. All rights reserved.43
Plugins come at a cost

44. © 2019 InfluxData. All rights reserved.44
➜ docker image ls
byot-sample 15.2MB
telegraf 254MB

45. © 2019 InfluxData. All rights reserved.45 © 2019 InfluxData. All rights reserved.45
Bring Your Own Telegraf
FROM rawkode/telegraf:byo AS build
FROM alpine:3.7 AS telegraf
COPY --from=build /etc/telegraf /etc/telegraf
COPY --from=build /binary /bin/telegraf
ENTRYPOINT [ "/bin/telegraf" ]

46. © 2019 InfluxData. All rights reserved.46 © 2019 InfluxData. All rights reserved.46
Bring Your Own Telegraf
BYOT
BYOT Example
Click Me! Click Me!

47. © 2019 InfluxData. All rights reserved.47
InfluxDB
Telegraf
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Buffer Batch

48. © 2019 InfluxData. All rights reserved.48
InfluxDB
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Telegraf
Telegraf
Telegraf
Telegraf

49. © 2019 InfluxData. All rights reserved.49
Internal

50. © 2019 InfluxData. All rights reserved.50
Internal
➔ Keep an eye on buffer_size
➔ Make sure you alert on metrics_dropped

51. © 2019 InfluxData. All rights reserved.51
Health Output

52. © 2019 InfluxData. All rights reserved.52
@app.route("/health")
def healthcheck():
return "OK”

53. © 2019 InfluxData. All rights reserved.53
@app.route("/health")
def healthcheck():
return "OK”

54. © 2019 InfluxData. All rights reserved.54 © 2019 InfluxData. All rights reserved.54
Health Output
[[outputs.health]]
service_address = "http://:5559"
namepass = ["web-metrics"]
[[outputs.health.compares]]
field = "response_time"
lt = 0.300

55. © 2019 InfluxData. All rights reserved.55
Kafka In, Kafka Out

56. © 2019 InfluxData. All rights reserved.56
InfluxDB
Application
Software
Dependencies
Bare Metal
/
VMs
Network
Telegraf
Telegraf
Telegraf
Telegraf
Kafka
Telegraf
Telegraf

57. © 2019 InfluxData. All rights reserved.57
Serverless?

58. © 2019 InfluxData. All rights reserved.58
Demo

59. © 2019 InfluxData. All rights reserved.59
We’ve covered less than 10 of the
200 plugins

60. https://speakerdeck.com/rawkode
https://github.com/rawkode/influxdb-examples
@rawkode