The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.

1. © 2022 VictoriaMetrics
VictoriaMetrics:
scaling to 100 million metrics per second
Aliaksandr Valialkin, CTO @ VictoriaMetrics

2. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics

3. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics
● I like writing programs in Go

4. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics
● I like writing programs in Go
● I like simple and clear code
doSimpleThing1()
doSimpleThing2()

5. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics
● I like writing programs in Go
● I like simple and clear code
● I hate over-engineered code, useless abstractions and bloated dependencies
abstractSingletonFabricProducerVisitorOperatorPrototype

6. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics
● I like writing programs in Go
● I like simple and clear code
● I hate over-engineered code, useless abstractions and bloated dependencies
● I like performance optimizations (fasthttp, fastjson, quicktemplate, fastcache)

7. Let’s meet
● I’m Aliaksandr Valialkin - core developer @ VictoriaMetrics
● I like writing programs in Go
● I like simple and clear code
● I hate over-engineered code, useless abstractions and bloated dependencies
● I like performance optimizations (fasthttp, fastjson, quicktemplate, fastcache)
● https://github.com/valyala/

8. What is VictoriaMetrics?
● Open source monitoring solution and time series database

9. What is VictoriaMetrics?
● Open source monitoring solution and time series database
● Supports popular data ingestion protocols - Prometheus, InfluxDB, Graphite,
DataDog, OpenTSDB, CSV, JSON

10. What is VictoriaMetrics?
● Open source monitoring solution and time series database
● Supports popular data ingestion protocols - Prometheus, InfluxDB, Graphite,
DataDog, OpenTSDB, CSV, JSON
● Can discover and scrape Prometheus targets (Kubernetes too)

11. What is VictoriaMetrics?
● Open source monitoring solution and time series database
● Supports popular data ingestion protocols - Prometheus, InfluxDB, Graphite,
DataDog, OpenTSDB, CSV, JSON
● Can discover and scrape Prometheus targets (Kubernetes too)
● Easy to setup and operate

12. What is VictoriaMetrics?
● Open source monitoring solution and time series database
● Supports popular data ingestion protocols - Prometheus, InfluxDB, Graphite,
DataDog, OpenTSDB, CSV, JSON
● Can discover and scrape Prometheus targets (Kubernetes too)
● Easy to setup and operate
● Low resource usage

13. What is VictoriaMetrics?
● Open source monitoring solution and time series database
● Supports popular data ingestion protocols - Prometheus, InfluxDB, Graphite,
DataDog, OpenTSDB, CSV, JSON
● Can discover and scrape Prometheus targets (Kubernetes too)
● Easy to setup and operate
● Low resource usage
● High performance

14. VictoriaMetrics kinds
● Single-node - scales vertically

15. VictoriaMetrics kinds
● Single-node - scales vertically
● Cluster - scales horizontally

16. VictoriaMetrics kinds
● Single-node - scales vertically
● Cluster - scales horizontally
● Single-node and cluster share the same core code

17. VictoriaMetrics single-node: scaling data ingestion
● Read incoming data in blocks
Client
data Read data
blocks
VictoriaMetrics

18. VictoriaMetrics single-node: scaling data ingestion
● Read incoming data in blocks
● Process blocks in parallel on multiple CPU cores
Client
data Read data
blocks
CPU_1
CPU_2
CPU_N
…
Process blocks
blocks
VictoriaMetrics

19. VictoriaMetrics single-node: scaling data ingestion
● Put the parsed data into independent buffers
CPU_1
CPU_2
CPU_N
…
Parse blocks
Buffer_1
Buffer_2
Buffer_M
…
In-memory buffers
Buffer parsed data
Tech details

20. VictoriaMetrics single-node: scaling data ingestion
● Put the parsed data into independent buffers
● Periodically store buffers to disk as independent LSM parts
Part_1
Part_2
Part_P
…
LSM parts
CPU_1
CPU_2
CPU_N
…
Parse blocks
Buffer_1
Buffer_2
Buffer_M
…
In-memory buffers
Compress and store data
Buffer parsed data
Tech details

21. ● VictoriaMetrics stores data in compressed blocks
VictoriaMetrics single-node: scaling querying path
block_1 block_N1
…
series_1
block_1 block_NM
…
series_M
…
block_1 block_N2
…
series_2

22. ● VictoriaMetrics stores data in compressed blocks
● Selected blocks are unpacked in parallel on available CPUs
VictoriaMetrics single-node: scaling querying path
block_1 block_N1
…
series_1
block_1 block_NM
…
series_M
…
CPU_1
CPU_P
…
blocks
block_1 block_N2
…
series_2

23. ● VictoriaMetrics stores data in compressed blocks
● Selected blocks are unpacked in parallel on available CPUs
● Selected time series are processed in parallel on available CPUs
VictoriaMetrics single-node: scaling querying path
block_1 block_N1
…
series_1
block_1 block_NM
…
series_M
…
CPU_1
CPU_P
…
blocks
CPU_1
CPU_P
…
series
block_1 block_N2
…
series_2

24. VictoriaMetrics single-node: scalability limits
● The performance is limited by a single host (CPU, RAM, disk)

25. VictoriaMetrics single-node: scalability limits
● The performance is limited by a single host (CPU, RAM, disk)
● Benchmark numbers:
○ Data ingestion: 300k samples/sec per CPU
○ Active time series: 1 million per GB of RAM
○ Query path: 50 million samples/sec per CPU

26. VictoriaMetrics single-node: scalability limits
● The performance is limited by a single host (CPU, RAM, disk)
● Benchmark numbers:
○ Data ingestion: 300k samples/sec per CPU
○ Active time series: 1 million per GB of RAM
○ Query path: 50 million samples/sec per CPU
● Production numbers:
○ Data ingestion: 2 million samples/sec
○ Active time series: 100 millions
○ Query path: 1 billion samples/sec
○ Total samples: 15 trillions

27. Scaling VictoriaMetrics cluster
● VictoriaMetrics cluster consists of three components:
○ vminsert - accepts incoming data
vminsert_1
vminsert_2
vminsert_M
…
HTTP
load
balancer
Incoming
data

28. Scaling VictoriaMetrics cluster
● VictoriaMetrics cluster consists of three components:
○ vminsert - accepts incoming data
○ vmselect - processes incoming queries
vminsert_1
vminsert_2
vminsert_M
…
HTTP
load
balancer
vmselect_1
vmselect_2
vmselect_P
…
Incoming
data
HTTP
load
balancer
Incoming
queries

29. Scaling VictoriaMetrics cluster
● VictoriaMetrics cluster consists of three components:
○ vminsert - accepts incoming data
○ vmselect - processes incoming queries
○ vmstorage - stores the data
vmstorage_1
vmstorage_2
vmstorage_N
…
vminsert_1
vminsert_2
vminsert_M
…
data
HTTP
load
balancer
vmselect_1
vmselect_2
vmselect_P
…
queries
Incoming
data
HTTP
load
balancer
Incoming
queries

30. Scaling VictoriaMetrics cluster
● VictoriaMetrics cluster consists of three components:
○ vminsert - accepts incoming data
○ vmselect - processes incoming queries
○ vmstorage - stores the data
● Each component can run on the most suitable hardware
vmstorage_1
vmstorage_2
vmstorage_N
…
vminsert_1
vminsert_2
vminsert_M
…
data
HTTP
load
balancer
vmselect_1
vmselect_2
vmselect_P
…
queries
Incoming
data
HTTP
load
balancer
Incoming
queries

31. Scaling VictoriaMetrics cluster
● VictoriaMetrics cluster consists of three components:
○ vminsert - accepts incoming data
○ vmselect - processes incoming queries
○ vmstorage - stores the data
● Each component can run on the most suitable hardware
● Each component can scale independently to any number of instances
vmstorage_1
vmstorage_2
vmstorage_N
…
vminsert_1
vminsert_2
vminsert_M
…
data
HTTP
load
balancer
vmselect_1
vmselect_2
vmselect_P
…
queries
Incoming
data
HTTP
load
balancer
Incoming
queries

32. VictoriaMetrics cluster: scaling data ingestion
● An http load balancer spreads incoming data among vminsert nodes
● Data ingestion performance scales with the number of vminsert nodes
HTTP load
balancer
vminsert_2
vminsert_1
vminsert_N
…
incoming data

33. VictoriaMetrics cluster: scaling data ingestion
● vminsert automatically shards incoming data among available vmstorage nodes
via consistent hashing
● Each vmstorage node has its own subset of time series (ideally)
● Data ingestion performance scales with the number of vmstorage nodes
vminsert vmstorage_2
vmstorage_1
vmstorage_M
…
sharding

34. VictoriaMetrics cluster: scaling querying path
● An http load balancer spreads incoming queries among vmselect nodes
● QPS scales with the number of vmselect nodes
HTTP load
balancer
vmselect_2
vmselect_1
vmselect_P
…
incoming queries

35. VictoriaMetrics cluster: scaling querying path
● vmselect fetches the needed data from every vmstorage node in parallel
● Querying performance scales with the number of vmstorage nodes
vmselect vmstorage_2
vmstorage_1
vmstorage_N
…
compressed data

36. VictoriaMetrics cluster: scaling querying path
● vmselect fetches the needed data from every vmstorage node in parallel
● Querying performance scales with the number of vmstorage nodes
● vmselect unpacks the fetched data in parallel on available CPUs
● Querying performance scales with the number of vCPUs at a single vmselect node
vmselect vmstorage_2
vmstorage_1
vmstorage_N
…
compressed data

37. VictoriaMetrics cluster: scalability limits
● CPU?

38. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs

39. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM?

40. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM? No - cluster capacity scales with RAM

41. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM? No - cluster capacity scales with RAM
● Disk?

42. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM? No - cluster capacity scales with RAM
● Disk? No - cluster capacity scales with disk space and io

43. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM? No - cluster capacity scales with RAM
● Disk? No - cluster capacity scales with disk space and io
● Network?

44. VictoriaMetrics cluster: scalability limits
● CPU? No - data ingestion and querying performance scales with CPUs
● RAM? No - cluster capacity scales with RAM
● Disk? No - cluster capacity scales with disk space and io
● Network? Yes!

45. 100M benchmark
● Can VictoriaMetrics cluster accept 100 million samples per second in
production?
● Can VictoriaMetrics cluster handle a billion of active time series
● How much resources does it need?

46. Benchmarketing?
● Artificial data?

47. Benchmarketing?
● Artificial data?
● Limited amounts of data?

48. Benchmarketing?
● Artificial data?
● Limited amounts of data?
● Limited benchmark duration?

49. Benchmarketing?
● Artificial data?
● Limited amounts of data?
● Limited benchmark duration?
● Special configs?

50. Benchmarketing?
● Artificial data?
● Limited amounts of data?
● Limited benchmark duration?
● Special configs?
● Optimized hardware?

51. No!

52. Prometheus-benchmark
● Helm chart for testing Prometheus-like systems

53. Prometheus-benchmark
● Helm chart for testing Prometheus-like systems
● Uses production-like workload for data ingestion and querying

54. Prometheus-benchmark
● Helm chart for testing Prometheus-like systems
● Uses production-like workload for data ingestion and querying
● Pushes the real node-exporter metrics to the tested systems
vmagent
node_exporter
scrape
load generator
Prometheus-like system
remote_write

55. Prometheus-benchmark
● Helm chart for testing Prometheus-like systems
● Uses production-like workload for data ingestion and querying
● Pushes the real node-exporter metrics to the tested systems
● Allows using the real alerting rules for node-exporter metrics
vmagent
node_exporter
scrape
load generator
Prometheus-like system
remote_write
vmalert
alerting rules
read
queries

56. Prometheus-benchmark
● Helm chart for testing Prometheus-like systems
● Uses production-like workload for data ingestion and querying
● Pushes the real node-exporter metrics to the tested systems
● Allows using the real alerting rules for node-exporter metrics
● https://github.com/VictoriaMetrics/prometheus-benchmark
vmagent
node_exporter
scrape
load generator
Prometheus-like system
remote_write
vmalert
alerting rules
read
queries

57. 100M benchmark: requirements
● Stable ingestion rate: 100.000.000 samples/sec

58. 100M benchmark: requirements
● Stable ingestion rate: 100.000.000 samples/sec
● Active time series: 1.000.000.000 (1 billion)

59. 100M benchmark: requirements
● Stable ingestion rate: 100.000.000 samples/sec
● Active time series: 1.000.000.000 (1 billion)
● Duration: 24 hours

60. 100M benchmark: requirements
● Stable ingestion rate: 100.000.000 samples/sec
● Active time series: 1.000.000.000 (1 billion)
● Duration: 24 hours
● Total samples: 100M*3600s*24h=8.640.000.000.000 (8.64 trillions)

61. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)

62. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)
● Scrape targets (node_exporter v1.4.0): 16*51.250=820.000

63. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)
● Scrape targets (node_exporter v1.4.0): 16*51.250=820.000
● Each scrape targets exposes around 1220 metrics

64. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)
● Scrape targets (node_exporter v1.4.0): 16*51.250=820.000
● Each scrape targets exposes around 1220 metrics
● Total number of metrics (aka active series): 820K*1220=1 billion

65. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)
● Scrape targets (node_exporter v1.4.0): 16*51.250=820.000
● Each scrape targets exposes around 1220 metrics
● Total number of metrics (aka active series): 820K*1220=1 billion
● Scrape interval: 10 seconds

66. 100M benchmark: prometheus-benchmark configs
● 16 load generator pods (8vCPU, 25GB RAM each)
● Scrape targets (node_exporter v1.4.0): 16*51.250=820.000
● Each scrape targets exposes around 1220 metrics
● Total number of metrics (aka active series): 820K*1220=1 billion
● Scrape interval: 10 seconds
● Scrape rate: 1 billion / 10 seconds = 100M samples/sec

67. 100M benchmark: VictoriaMetrics cluster configs
● Runs in Google Kubernetes Engine via the official VictoriaMetrics helm charts

68. 100M benchmark: VictoriaMetrics cluster configs
● Runs in Google Kubernetes Engine via the official VictoriaMetrics helm charts
● vmstorage: 46 x (16 vCPU, 55GB RAM, 2200 GB hdd-based disk)

69. 100M benchmark: VictoriaMetrics cluster configs
● Runs in Google Kubernetes Engine via the official VictoriaMetrics helm charts
● vmstorage: 46 x (16 vCPU, 55GB RAM, 2200 GB hdd-based disk)
● vminsert: 18 x (16 vCPU, 55GB RAM)

70. 100M benchmark: VictoriaMetrics cluster configs
● Runs in Google Kubernetes Engine via the official VictoriaMetrics helm charts
● vmstorage: 46 x (16 vCPU, 55GB RAM, 2200 GB hdd-based disk)
● vminsert: 18 x (16 vCPU, 55GB RAM)
● vmselect: none (wait for the next talk)

71. 100M benchmark: allocated resources
● Prometheus-benchmark resources:
○ vCPU cores: 16*8=128
○ RAM: 16*25GB=400GB

72. 100M benchmark: allocated resources
● Prometheus-benchmark resources:
○ vCPU cores: 16*8=128
○ RAM: 16*25GB=400GB
● VictoriaMetrics cluster resources:
○ vCPU cores: (46vmstorage+18vminsert)*16=1024
○ RAM: (46vmstorage+18vminsert)*55=3520GB
○ Disk: 46 x 2200GB = 101.2 TB

73. 100M benchmark: allocated resources
● Prometheus-benchmark resources:
○ vCPU cores: 16*8=128
○ RAM: 16*25GB=400GB
● VictoriaMetrics cluster resources:
○ vCPU cores: (46vmstorage+18vminsert)*16=1024
○ RAM: (46vmstorage+18vminsert)*55=3520GB
○ Disk: 46 x 2200GB = 101.2 TB
● Kubernetes cluster:
○ 36x e2-standard-32 nodes (32 vCPU, 128GB RAM each)
○ Total: 1152 vCPU, 4608GB RAM

74. 100M benchmark: used resources
● vminsert: 206vCPU, 26GB RAM

75. 100M benchmark: used resources
● vminsert: 206vCPU, 26GB RAM
● vmstorage: 510vCPU, 600GB RAM, 101.2TB disk

76. 100M benchmark: used resources
● vminsert: 206vCPU, 26GB RAM
● vmstorage: 510vCPU, 600GB RAM, 101.2TB disk
● Total: 716vCPU (70%), 626GB RAM (18%), 7.5TB disk (7.5%)

77. 100M benchmark: used resources
● vminsert: 206vCPU, 26GB RAM
● vmstorage: 510vCPU, 600GB RAM, 101.2TB disk
● Total: 716vCPU (70%), 626GB RAM (18%), 7.5TB disk (7.5%)
● Network: 140Gbit/s (can be reduced to 20Gbit/s at the cost of 10% CPU)

78. 100M benchmark: used resources
● vminsert: 206vCPU, 26GB RAM
● vmstorage: 510vCPU, 600GB RAM, 101.2TB disk
● Total: 716vCPU (70%), 626GB RAM (18%), 7.5TB disk (7.5%)
● Network: 140Gbit/s (can be reduced to 20Gbit/s at the cost of 10% CPU)
● Disk IO: 3GB/s write, 450MB/s read

79. 100M benchmark: results
● Stable data ingestion at 100M samples/sec during 24 hours

80. 100M benchmark: results
● Stable data ingestion at 100M samples/sec during 24 hours
● Active time series: 1 billion

81. 100M benchmark: results
● Stable data ingestion at 100M samples/sec during 24 hours
● Active time series: 1 billion
● Total samples ingested: 8.77 trillions

82. 100M benchmark: results
● Stable data ingestion at 100M samples/sec during 24 hours
● Active time series: 1 billion
● Total samples ingested: 8.77 trillions

83. 100M benchmark: results
● Stable data ingestion at 100M samples/sec during 24 hours
● Active time series: 1 billion
● Total samples ingested: 8.77 trillions
● Average sample size: 0.85 bytes

85. 100M benchmark: key takeaways
● VictoriaMetrics cluster performance and capacity scales linearly to 100 nodes
and more

86. 100M benchmark: key takeaways
● VictoriaMetrics cluster performance and capacity scales linearly to 100 nodes
and more
● A single VictoriaMetrics cluster can collect metrics from a million of hosts
vmagent
host_1
host_2
host_1.000.000
…
scrape
VictoriaMetrics cluster
remote_write
a million of hosts
scrape_interval=10s

87. 100M benchmark: key takeaways
● VictoriaMetrics cluster performance and capacity scales linearly to 100 nodes
and more
● A single VictoriaMetrics cluster can collect metrics from a million of hosts
● Cluster stability improves with the number of nodes

88. 100M benchmark: key takeaways
● VictoriaMetrics cluster performance and capacity scales linearly to 100 nodes
and more
● A single VictoriaMetrics cluster can collect metrics from a million of hosts
● Cluster stability improves with the number of nodes
● HDD-based disks are enough - there is no need in SSD-based disks
HDD
$40/TB/month
SSD
$170/TB/month
vs

89. 100M benchmark: key takeaways
● VictoriaMetrics cluster performance and capacity scales linearly to 100 nodes
and more
● A single VictoriaMetrics cluster can collect metrics from a million of hosts
● Cluster stability improves with the number of nodes
● HDD-based disks are enough - there is no need in SSD-based disks
● VictoriaMetrics handles large workloads with default configs

90. Reproduce the 100M benchmark on yourself!
● https://github.com/VictoriaMetrics/prometheus-benchmark/tree/bm-100

91. Reproduce the 100M benchmark on yourself!
● https://github.com/VictoriaMetrics/prometheus-benchmark/tree/bm-100
● Benchmark configs

92. Reproduce the 100M benchmark on yourself!
● https://github.com/VictoriaMetrics/prometheus-benchmark/tree/bm-100
● Benchmark configs
● VictoriaMetrics cluster configs

93. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?

94. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?
● A billion samples/sec benchmark?

95. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?
● A billion samples/sec benchmark?
● 10 billions of active time series?

96. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?
● A billion samples/sec benchmark?
● 10 billions of active time series?
● Kubernetes-like time series churn rate?

97. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?
● A billion samples/sec benchmark?
● 10 billions of active time series?
● Kubernetes-like time series churn rate?
● A month-long benchmark (needs $$$)?

98. What’s next?
● Benchmark querying performance (50M samples/sec per vCPU processing
speed)?
● A billion samples/sec benchmark?
● 10 billions of active time series?
● Kubernetes-like time series churn rate?
● A month-long benchmark (needs $$$)?
● Share your results!

99. Questions?