More Related Content Similar to Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly) (20) Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)28. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
whisper calc
29. Retention and size
• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc
33. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
34. Retention and size
• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
• can be assign per metric
35. How
• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rocker (https://github.com/grammarly/rocker)
• rocker-compose (https://github.com/grammarly/rocker-compose)
45. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
46. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
47. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
• defaults
48. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND,
MAX_CREATES_PER_MINUTE = inf
• defaults
• almost 1.5h to get limit :(
56. Try to tune!
• WHISPER_SPARSE_CREATE = true
(don’t allocate space on creation)
non-linear IO load.
• CACHE_WRITE_STRATEGY =
sorted (default)
59. Try to tune!
• CACHE_WRITE_STRATEGY = max
will give a strong flush preference to
frequently updated metrics and will
also reduce random file-io.
62. Try to tune!
• CACHE_WRITE_STRATEGY = naive
just flush. Better with random IO.
76. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
77. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
78. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
79. Start load testing
• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
• almost 1h to get limit :(
89. try to tune!
remaining := whisper.Size() - whisper.MetadataSize()
whisper.file.Seek(int64(remaining-1), 0)
whisper.file.Write([]byte{0})
chunkSize := 16384
zeros := make([]byte, chunkSize)
for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {
// return nil, err
// }
remaining -= chunkSize
}
if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err
}
114. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
115. Start load testing
• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections
139. TSDB с тегами
• influxDB
• openTSDB (hbase)
• cyanite (cassandra)
• newTS (cassandra)
• Prometheus
168. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club
169. feel free to ask
• Vsevolod Polyakov
• ctrlok@gmail.com
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club
Мы хайрим!