Time is money. Understanding application responsiveness and latency is critical not only for delivering good application behavior but also for maintaining profitability and containing risk. But good characterization of bad data is useless. When measurements of response time present false or misleading latency information, even the best analysis can lead to wrong operational decisions and poor application experience.
This presentation discusses common pitfalls encountered in measuring and characterizing latency. It demonstrates and discusses some false assumptions and measurement techniques that lead to dramatically incorrect reporting and covers ways to do a sanity check and correct such situations.
142. How bad can this get?
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
143. How bad can this get?
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
144. How bad can this get?
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
145. How bad can this get?
Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
146. How bad can this get?
Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
Avg. is 50 sec.
over next 100 sec
147. How bad can this get?
Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
Avg. is 50 sec.
over next 100 sec
Overall Average response time is ~25 sec.
148. How bad can this get?
Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
Avg. is 50 sec.
over next 100 sec
Overall Average response time is ~25 sec.
149. Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
150. Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
151. Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
Naïve Characterization
152. Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
Naïve Characterization
10,000 @ 1msec 1 @ 100 second
153. Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
Naïve Characterization
10,000 @ 1msec 1 @ 100 second
99.99%‘ile is 1 msec! Average. is 10.9msec! Std. Dev. is 0.99sec!
(should be ~100sec) (should be ~25 sec)
154. Proper measurement
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
155. Proper measurement
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
156. Proper measurement
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
157. Proper measurement
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
158. Proper measurement
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
Coordinated
Omission