When raw data becomes overwhelming, we turn to abstraction to understand our world. In examining the performance of our systems, the data is always overwhelming. Solutions like summary statistics have come to our rescue, and they are good—up to a point. In order to truly understand our systems, we need to know when and how to sidestep those abstractions,to get deep, detailed performance insight. At this meetup, I’ll explore techniques for visualizing the underlying structure of performance data and how this empowers drilling down to populations and individual samples in the data set.

1. BEYOND AVERAGES
Dan Kuebrich / @dankosaur

2. Beyond Averages
• Abstraction: summary statistics for performance data
• Why performance data is hard
• Visualizing data distributions

5. A few of my favorite abstractions

6. A few of my favorite abstractions
• Abstraction lets us trade information for
actionability

7. A few of my favorite abstractions
• Abstraction lets us trade information for
actionability
• Min, max, average (“mean”), quantiles, stdev

8. A few of my favorite abstractions
• Abstraction lets us trade information for
actionability
• Min, max, average (“mean”), quantiles, stdev
• That’s a great trade!
• ... right?

9. Averages: average at best

10. Averages: average at best

11. Averages: average at best

12. Averages: average at best

13. Averages vs Percentiles
[1, 16, 17, 19, 13, 5, 20, 3, 10, 14, 8]

14. Averages vs Percentiles
[1, 3, 5, 8, 10, 13, 14, 16, 17, 19, 20]

15. Averages vs Percentiles
Average (Mean): 11.54
[1, 3, 5, 8, 10, 13, 14, 16, 17, 19, 20]
Median (50th Percentile): 13
90th Percentile: 19

16. Percentiles: 1 of 100 slices
X
95%

17. Percentiles: 2 of 100 slices
X
95%
10%
Y

18. Percentiles: 2 of 100 slices
95%
10%

19. Percentiles: 2 of 100 slices
95%
10%

20. Percentiles: 2 of 100 slices
95%
10%

21. Computers are hard
• Rarely do we have a single distribution underlying the
data
• Different users, different requests, different
resources, different instances, different times

22. Percentiles vs Distributions
http://en.wikipedia.org/wiki/Percentile

23. Percentiles vs Distributions
[13, 13, 13, 13, 13, 13, 19, 19, 19, 19, 19]
Median (50th Percentile): 13
90th Percentile: 19

24. Rarely do we have a single normal
distribution underlying the data

26. Median
Mean
90th

30. Median
Mean
90th

31. The Log-Normal Distribution
(source: http://www.geo.mtu.edu/volcanoes/vc_web/background/S_chem.html)

32. Log-Normal Distribution
(source: http://en.wikipedia.org/wiki/File:Comparison_mean_median_mode.svg)

33. Log-Normal Distribution
(source: http://en.wikipedia.org/wiki/File:Comparison_mean_median_mode.svg)

34. Log-Normal Distribution

35. Is there a place between Averageland
and “A Beautiful Mind”?
http://now-here-this.timeout.com/2012/10/07/crazy-walls-of-clues-from-tv-film-reviewed-by-carrie-from-homeland/

36. (eg. # of calls)
Frequency
Histograms
Value
(eg. latency)

37. Populations revisited
95%
10%

38. (eg. # of calls)
Frequency
Histograms
Value
(eg. latency)

39. Populations re-revisited
95%
?
10%

40. (eg. # of calls)
Frequency
3d Histograms?
Value
(eg. latency)

41. (eg. # of calls)
Frequency
3d Histograms?
Time
Value
(eg. latency)

42. (eg. # of calls)
Frequency
Heatmaps
Value
(eg. latency)

43. (eg. # of calls)
Frequency
Heatmaps
Value
(eg. latency)

44. (eg. # of calls)
Frequency
Heatmaps
Value
(eg. latency)

45. (eg. latency)
Value
Heatmaps
Time

46. (eg. latency)
Value
Heatmaps
Time

47. Latency in the wild…
http://sciencefiction.com/2013/10/24/throwback-thursday-jurassic-park/

48. Average, or Absolute?

49. Average, or Absolute?

50. Multi-modal Data

51. Multi-modal Data

52. Multi-modal Drill-down

53. Multi-modal Drill-down

54. Multi-modal Drill-down

55. Long Tails and Outliers

56. Long Tails and Outliers
bottom 98%

57. Long Tails and Outliers
bottom 98%

58. Long Tails and Outliers
all of it

59. Added Population

60. Added Population

61. Added Population

62. Thanks!
Sign up for a free TraceView account at
appneta.com/products/traceview-free-account