Suivez le périple d'une log depuis son émission, jusqu'à sa destination finale. Un parcourt semé d’embûches pour sauver la belle application du méchant NullPointor. Des courses poursuites à plus 10000 messages/s dans le dédale des systèmes distribués. Acteurs: Logback le faillot, Filebeat l'espion, Kafka le transporteur, Elasticsearch.... Costumes et Maquillage: Logstash Il s'agit d'un retour d'expérience sur la configuration et l'exploitation d'un pipeline de données (des logs, des métriques...). A partir des problèmes rencontrés en production, j'expliquerai le fonctionnement interne des outils utilisés, pour vous permettre d'éviter les même écueils. Je m’intéresserai à des problématiques comme la fiabilité (perte de messages), la résilience, la performance ou la sécurité. MixIT conference 2017 https://mixitconf.org/2017/l-odyssee-de-la-log

1. L'ODYSSÉE DE LA LOG
1

2. QUI ?
@gerald_quintana
2

3. ITINÉRAIRE
3 . 1

4. DONNÉES TEMPORELLES
Logs
Mesures
Evénements
(2017-04-20 18:17:16, Data)
3 . 2

5. COLLECTER
Applica ons
Agents
3 . 3

6. STOCKER
Bases de données
Système de fichier
Cloud
3 . 4

7. POUR COMMENCER
Collecter Stocker
3 . 5

8. TRANSFORMER, FILTRER, ENRICHIR
Logs processing
Stream processing
3 . 6

9. POUR CONTINUER
Collecter Transformer Stocker
3 . 7

10. TRANSPORTER, BUFFERISER
3 . 8

11. TRANSPORTER, BUFFERISER
Aler ng, SIEM...?
3 . 9

12. POUR FINIR
Collecter Bufferiser Transformer Stocker
3 . 10

13. COLLECTER
4 . 1

14. LOG
2017-03-20 22:42:03 [main] INFO Bonjour à tous
4 . 2

15. FORMAT
{
"@timestamp":"2017-03-20T22:42:03.522+01:00",
"logger":"mixit",
"level":"INFO",
"message":"Bonjour à tous",
"thread":"main",
"host":"laptop-gerald",
"user":"gerald",
"transactionid": 4567,
"talk":"log-odyssey"
}
4 . 3

16. EMISSION
Fichier vs TCP/UDP
4 . 4

17. APPLICATIONS & JSON
...
4 . 5

18. APPLICATIONS & KAFKA
... →
4 . 6

19. BEATS & JSON
filebeat.prospectors:
- input_type: log
document_type: logback
paths:
- /var/log/log-odyssey/application.*.log
json:
keys_under_root: true
output.elasticsearch:
hosts: ["elasticsearch:9200"]
4 . 7

20. BEATS & KAFKA
filebeat.prospectors:
- input_type: log
...
output.kafka:
hosts: ["kafka:9092"]
topic: logstash
4 . 8

21. DOCKER
--log-driver=json-file|syslog|gelf|fluentd|splunk...
4 . 9

22. TRANSPORTER
5 . 1

23. KAFKA
5 . 2

24. PRODUCER / BROKER / CONSUMER
5 . 3

25. MESSAGE / RECORD
Key Value TS
5 . 4

26. PRODUCER
n1
n2
n3
5 . 5

27. PARTITIONNEMENT
5 . 6

28. BEATS & KAFKA
filebeat.prospectors:
- input_type: log
...
output.kafka:
hosts: ["kafka:9092"]
topic: logstash
partition.round_robin:
reachable_only: false
5 . 7

29. CONSUMER
n1
n2
n3
5 . 8

30. COMMITFAILEDEXCEPTION
CommitFailedException: Commit cannot be completed since the group has already
rebalanced and assigned the partitions to another member.
5 . 9

31. EQUILIBRAGE DES CONSUMERS
<[... 14:27:40,752] ...: Preparing to restabilize group logstash with old generation 0>
<[... 14:27:40,753] ...: Stabilized group logstash generation 1>
<[... 14:27:40,773] ...: Assignment received from leader for group logstash for generation
<[... 14:27:48,243] ...: Preparing to restabilize group logstash with old generation 1>
<[... 14:27:49,837] ...: Stabilized group logstash generation 2>
<[... 14:27:49,845] ...: Assignment received from leader for group logstash for generation
<[... 14:27:54,969] ...: Preparing to restabilize group logstash with old generation 2>
<[... 14:27:56,621] ...: Stabilized group logstash generation 3>
5 . 10

32. CONSUMER
n3
poll
commit
subscribe
session.timeout.ms
partition.max.fetch.size max.poll.records
5 . 11

33. PROTOCOLE
EOFException: null at o.a.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:
SchemaException: Error reading field 'throttle_time_ms': java.nio.BufferUnderflowException
InvalidRequestException: Error getting request for apiKey: 3 and apiVersion: 2
5 . 12

34. PROTOCOLE
Version Client ≤ Version Serveurs
h p://ka a.apache.org/protocol.html
5 . 13

35. FILTRER, TRANSFORMER
6 . 1

36. PIPELINE LOGSTASH
filterinput output
6 . 2

37. CONFIGURATION
input {
kafka {
bootstrap_servers => "kafka:9092"
codec => json
topics => ["logstash"]
}
}
filter {
if [type] == "jetty" {
grok {
match => { "message" =>
"%{COMBINEDAPACHELOG} (?:%{NUMBER:latency:int}|-)" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
6 . 3

38. KAFKA INPUT/OUTPUT
Ka a Logstash Plugin
0.8 2.0 - 2.x <3.0
0.9 2.0 - 2.3 3.x
0.9 2.4 - 5.x 4.x
0.10.0 2.4 - 5.x 5.x
0.10.1 2.4 - 5.x 6.x
6 . 4

39. PIPELINE LOGSTASH
input
input
batcher filterfilter output
batcher filterfilter output
pipeline.workers -w
pipeline.batch.size -b
6 . 5

40. MONITORER
6 . 6

41. MONITORER
filter {
ruby {
init => "require 'time'"
code => "start_time=Time.now.to_f*1000.0;
event.set('[@metadata][start_time]', start_time);"
}
# Filtrage
#....
ruby {
init => "require 'time'"
code => "end_time=Time.now.to_f=1000.0;
start_time=event.get('[@metadata][start_time]');
event.set('[logstash_duration]', end_time - start_time)"
}
metrics
6 . 7

42. MONITORER
6 . 8

43. PIPELINE ELASTICSEARCH
PUT _ingest/pipeline/jetty
{ "description": "Jetty Access Logs",
"processors": [
{ "grok": {
"field": "message",
"patterns": [
"%{COMBINEDAPACHELOG} (?:%{NUMBER:latency:int}|-)" ] } },
{ "date": {
"field": "timestamp",
"formats": [
"dd/MMM/yyyy:HH:mm:ss Z" ] } },
{ "date_index_name": {
"field": "@timestamp",
"index_name_prefix": "logs-",
"date_rounding" : "d" } }
6 . 9

44. STOCKER
7 . 1

45. ELASTICSEARCH
7 . 2

46. SCHEMALESS ?
Normaliser les champs
7 . 3

47. MAPPINGS
{ "jetty": {
"properties": {
"champ": {
"type": "text|keyword|integer|...",
"index": false?,
"norms": false?
} },
"_all": { "enabled": false }
} }
7 . 4

48. ROUTING & MAPPING
bulk
map
7 . 5

49. WRITES
segment
refresh
buffer
translog segment
flush
segment
...
index.refresh_interval 1s
index.translog.flush_threshold_size 512mb
7 . 6

50. index.merge.scheduler.max_thread_count CPU/2
MERGES
7 . 7

51. ARRIVÉE
8 . 1

52. QUESTIONS
?
8 . 2

53. MERCI
@gerald_quintana
8 . 3