Simple practices in performance monitoring and evaluation

1. Simple Practices in
Performance Monitoring and Evaluation
Schubert Zhang
2016.3.24

2. SLA
Service Level Agreements
https://en.wikipedia.org/wiki/Service-level_agreement
SLAs commonly include segments to address:
a definition of services, performance measurement, problem management, customer duties,
warranties, disaster recovery, termination of agreement.

3. •
•
• API
IM SLA
•
• Performance
• Performance
performance oriented SLA

4. Metrics
SLA Performance SLA
Performance Metrics
e.g.1: API
•
• (99%)
•
e.g.2: Call Center
• Abandonment Rate: Percentage of calls abandoned while waiting to be answered.
• ASA (Average Speed to Answer): Average time it takes for a call to be answered
by the service desk.
• TSF (Time Service Factor): Percentage of calls answered within a definite
timeframe, e.g., 80% in 20 seconds.
• FCR (First-Call Resolution): Percentage of incoming calls that can be resolved
without the use of a callback or without having the caller call back the helpdesk to
finish resolving the case.
• TAT (Turn-Around Time): Time taken to complete a certain task.
Metrics
Performance Metrics

5. Benchmarking
the quality of a service must be measured, evaluated,
… benchmarked.
and we must have a set of approaches for benchmarking.

6. Metrics to be monitored

7. Throughput
QPS TPS CPS
in seconds, in minutes, in hours …

8. Concurrency

9. Latency
Response Time Round-Trip Time(RTT) …
Average Median Min. Max. Percentile …

10. Quantile / Percentile
refers to Google Sawzall Paper

11. A Summary of these Concepts
Client-1
Client-2
Client-3
Client-N
Work Thread
Work Thread
Work Thread
Work Thread
Work Thread
ThroughputLatency Concurrency
Clients Server

12. A Life-World Example

13. Example-1
Paper Amazon Dynamo

16. Average
99.9%, quantile

17. Example-2
Evaluation Report to a NoSQL DB
Cassandra

18. Benchmark for Write API
Benchmark for Writes Cluster overview
Throughput
Latency
• Each node runs 6 clients (threads), totally 54 clients.
• Each client generates random CDRs for 50 million users/phone-numbers,
and puts them into DaStor one by one.
– Key Space: 50 million
– Size of a CDR: Thrift-compacted encoding, ~200 bytes
ü Throughput: average ~80K ops/s; per-node: average ~9K ops/s
ü Latency: average ~0.5ms
p Bottleneck: network (and memory)

19. Benchmark for Read API
• Each node runs 8 clients (threads) , totally 72 clients.
• Each client randomly uses a user-id/phone-number out of the 50-million
space, to get it’s recent 20 CDRs (one page) from DaStor.
• All clients read CDRs of a same day/bucket.
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61
100ms
percentage of read ops
ü Throughput: average ~140 ops/s; per-node: average ~16 ops/s
ü Latency: average ~500ms, 97% < 2s (SLA)
p Bottleneck: disk IO (random seek) (CPU load is very low)
average
97%
quantile

20. Total & Delta
Total:
Delta:

21. Generate the metrics and
monitor them

22. • In server side
• Add a operation-count and the time-
cost for every client call
• For every monitor interval, pull and
push the current Throughput and
Latency the monitor-tool(ganglia/
zabbix) or console.
• Throughput = sum of count / time interval
• Latency = average(sum of latency / sum of count),
max, min, quantile …
Code in Gitlab and Gerrit

23. Code for Spring Project

24. • Java
• JMX (Java Management Extensions, a simple example at https://github.com/schubertzhang/jsketch)
• javaagent (java -javaagent:jar path [= premain ] )
• jmxetric (use JMX and javaagent to display metrics to Ganglia, https://github.com/schubertzhang/jmxetric)
•
• Ganglia
• Zabbix
• …

25. Ganglia Zabbix etc.

26. Performance Benchmark
Programing
Demo
Test and Evaluation the Throughput and Latency of http://www.fangdd.com

27. Demo Time …

28. demo screenshots

29. demo screenshots
Average 95%
The long tail …

30. Statistical Monitoring for Outlier
usually for trouble-shooting

31. Captured from UTStarcom mSwitch R5 system, Guangxi Site, 2004.
The magic matrix:

32. •
• Redis Memcache
• Just add at a point, very low-cost
•
• Very
• Logs ELK

33. Heavy Logs & ELK
It’s another topic!

34. Thank You!