SlideShare ist ein Scribd-Unternehmen logo
Know Your Data:
The stats behind the alerts
Dave McAllister
Sr. OSS Technologist
NGINX
©2023 F5
2
Quick:
What’s the difference between
Mean, Median and Mode?
©2023 F5
3
Quick:
What’s the difference between
Mean, Median and Mode?
And for extra credit, What’s the 9th Dedekind number?
©2023 F5
5
Monitoring is a numbers game
• Metrics are numbers that represent selected
behavior
• Generally
• Timestamped
• Key-Values
• Data, to be useful, must be
• Aggregated
• Analyzed
• Visualized
©2023 F5
6
• How do you deal with outliers (spikes) in
monitoring?
• How do you get a representative value when
values build on each other?
• How do you arrive at values to represent rate of
change over time?
Do you know what your alert is
really showing you?
Some questions to ponder
©2023 F5
7
Mean, Median, Mode
Median:
Represents the middle
value in a set of ordered
data
Mean:
Measure of central
tendency, represents
average value of a set of
data
Mode:
Value that appears most
frequently in a set of data
Data:
2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6,
3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4
Median = 6 Mode = 1
Mean = 5.444
©2023 F5
8
Mean, Median, Mode
Data:
2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6, 3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1,
4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4
Mean = 5.444
Or is it 4.130 or 2.791?
Median:
Represents the middle
value in a set of ordered
data
Mean:
Measure of central
tendency, represents
average value of a set of
data
Mode:
Value that appears most
frequently in a set of data
©2023 F5
9
Arithmetic, Harmonic, Geometric, Trimmed, Weighted, Moving
Means to an End
• Each has potential uses and drawbacks
• Often already implemented in monitoring software
• Can give very different results
• Can make like and unlike comparisons easier
©2023 F5
10
Also often called Average
• Most common
• Is the central point in a normal distribution
• This is not the 50% mark (mostly)
• Useful for comparing current to previous conditions
• May be aggregated into groups (time series)
In a time series, we usually calculate
constantly to incorporate new data
Arithmetic
Internet
Load
Balancer
300
Request/sec
200
Request/sec
100
Request/sec
Amean = (100 + 200 + 300) / 3
Amean = 200 Requests per second
©2023 F5
11
• Multiply all the items together, take the nth
root
• Often used for things growing exponentially
• In DevOps
• Average number of deploys per unit of
time
• Average lead time for changes
• MTTR
• Throughput
Geometric
DevOps team: optimizing app deployment
Sprint 1: 5% reduction
Sprint 2: 10% reduction
Sprint 3: 3% reduction
Sprint 4: 7% reduction
How well are they reducing deployment times?
X1=1-0.05 = 0.95, x2 = 0.90, X3 = 0.97, x4 = 0.93
GM = 0.937 or
Deployment has a 6.3% improvement
©2023 F5
12
Harmonic
• Divide n by the sum of the reciprocals
• Measure the performance where
multiple systems are involved
• Weights the lowest figure the highest
• In DevOps
• Performance within range
• Overall indication of latency or thruput
• Use in complex environments
• Especially useful for outliers
Internet
Load
Balancer
300
Request/sec
200
Request/sec
100
Request/sec
HM = 150 Requests per second
©2023 F5
13
Take a comparison look
What’s more important to the equation?
• Latency
• Thruput
• GM is the moral equivalent
• AM weights the larger number
• HM weights the lower value
Average
Latency
ms/sec
Average
Count/sec
Suggested
metric AM GM HM
250 40 10000 145.00 100.00 68.97
200 50 10000 125.00 100.00 80.00
400 25 10000 212.50 100.00 47.06
198 70 13860 134.00 117.73 103.43
105 99 10395 102.00 101.96 101.91
474 73 34602 273.50 186.02 126.52
195 97 18915 146.00 137.53 129.55
196 138 27048 167.00 164.46 161.96
133 10 1330 71.50 36.47 18.60
298 55 16390 176.50 128.02 92.86
0.00
50.00
100.00
150.00
200.00
250.00
300.00
1 2 3 4 5 6 7 8 9 10
Means overlay
AM GM HM
©2023 F5
14
• Amazingly underutilized!
• Center value of a sorted list
• Median is always the 50% point of a normal curve
Median
Mean 25.25
Median 26
©2023 F5
15
Choosing Between Mean and Median
• Mean can be impacted by outliers
• Resilience is better in median
• In DevOps
• Response time monitoring
• Anomaly Detection
• Capacity Planning
Old New
Outlier 250
25.54 Mean 31.15
26 Median 26.5
©2023 F5
16
Old New
Outlier 291
21.90 Mean 28.63
22 Median 22.5
18.61 Harmonic 19.06
©2023 F5
17
If you are using P95
you are using using a percentage value
Congrats!
©2023 F5
18
Slight sidetrack: Measure of Variability
How the numbers behave.
• Standard Deviation
• Range
• InterQuartile Range (IQR)
• Variance
• Clusters
• Outliers
Properly used, variability can
help you target outliers
©2023 F5
19
Those outside edges are good candidates for outliers
Outlier Formula, visually
Q2
IQR
Outlier Formula
Q3
Q1
©2023 F5
20
How about the Mode?
• The most commonly recurring
value in the set
• Often presented as a histograph
• Not commonly used in DevOps,
mostly inferential
• Log Analysis
• Security Monitoring
• User Behavior Analysis
0 2 4 6 8 10 12 14
one
two
three
four
five
six
seven
eight
nine
ten
Mode Chart
©2023 F5
21
Slight sidetrack: Descriptive versus Inferential stats
• Descriptive uses the whole data set to draw statistical conclusions
• Used for visualization
• Can define and extract trends
• Inferential uses a sampled set to draw conclusions
• Used for predictions or hypotenuse testing
• Can also visualize
• But this leads us to sampling
©2023 F5
22
Dealing with the data
• Monitoring is now a data problem
• Observability signals: Metrics, Traces, Logs
• Analysis is often
• Aggregated or Analyzed in segments: Time-defined
• Sampled and inferential
• Random sampling
• Stratified sampling
• Cluster sampling
• Systematic sampling
• Purposive sampling
©2023 F5
23
Sampling No Sampling
©2023 F5
24
• Changes behavior from Descriptive to Inferential
• Can hide outlier behavior
• Metrics are not usually sampled
• May make forensics tougher
• Lack of direct correlation
• A necessary evil
• But understand what you are giving up
Sampling
©2023 F5
25
Distributions
• Normal
• Data equally distributed
• Poisson
• used to model the occurrence of rare
events
• Beta
• Success/failure of binomial events
• Exponential
• Time between async events
• Weibull
• Likelihood of failure
• Log-normal
• Values based on many small events
©2023 F5
26
Slight sidetrack: Standard deviation
• Measures the variability of your data
• Identifies trends and outliers
• NOT percentage based
• Except with coefficient of variability
• CV=Mean / std dev X 100
• Useful for measurement ignoring range
• SRE cases
• Lead times
• Recovery times
• Anomalies (alerts)
• SLO / SLI
This Photo by Unknown Author is licensed under CC BY-NC-ND
©2023 F5
27
Deeper dive: Weibull
• Usually used for time-to-failure
• Defined by a Shape and a Scale
parameter
• This can be challenging
• Don’t ask the math
• F(t)=1−e−(t/λ)β
• R does it for you!
Component Time-to-Failure
Spinning Rust 500 hours
Memory 1000 hours
Power Supply 1500 hours
CPU 2000 hours
SSD 2500 hours
library(fitdistrplus)
data <- c(500, 1000, 1500, 2000, 2500)
fit.weib <- fitdist(data, "weibull")
summary(fit.weib)
Fitting of the distribution ' weibull ' by maximum likelihood
Parameters : shape 1.0624082 scale 2158.256
p.failure <- pweibull(300, shape = fit.weib$estimate[1],
scale = fit.weib$estimate[2])
1 - p.failure
Failure (disk, memory) = 64.81%
Failure (entire system) = 75.52%
Fdisk(300)≈1−e−0.6918≈0.50025
Fmemory(300)≈1− −0.3490≈0.2959Fmemory(300)≈1−e−0.3490≈0.2959
Fpower(300)≈1− −0.2327≈0.2092Fpower(300)≈1−e−0.2327≈0.2092
FCPU(300)≈1− −0.1745≈0.1593FCPU(300)≈1−e−0.1745≈0.1593
FSSD(300)≈1− −0.1396≈0.1298FSSD(300)≈1−e−0.1396≈0.1298
©2023 F5
28
Deeper dive: Exponential
• Models the “rate” (time between
events that are unrelated)
• Use cases
• Network performance
• User Requests (traces)
• Messaging service
• System failures
• f(x) = me–mx
• λ = 1/µ
• e ~ 2.71828182846
Average
Latency
ms/sec
Average
Count/sec
Suggested
metric AM GM HM
250 40 10000 145.00 100.00 68.97
200 50 10000 125.00 100.00 80.00
400 25 10000 212.50 100.00 47.06
198 70 13860 134.00 117.73 103.43
105 99 10395 102.00 101.96 101.91
474 73 34602 273.50 186.02 126.52
195 97 18915 146.00 137.53 129.55
196 138 27048 167.00 164.46 161.96
133 10 1330 71.50 36.47 18.60
298 55 16390 176.50 128.02 92.86
Latency
λ = 1/244.9
Probability density of 158 ms = 0.2142%
Cumulative density = 47.5422%
Expdist median ≈ 169.7 ms
Throughput
λ = 1/65.7
Probability density at 58 count = 0.628%
Cumulative density = 58.7%
Expdist median ≈ 45.6
©2023 F5
29
You may stumble upon:
“On scale, statistics are not your friend”
WRONG
On scale, probability is not your friend.
Slight Sidetrack
and a pet peeve
©2023 F5
30
Conditional Probability – The Monty Hall Problem
Bayes Theorem
• You have three doors
• Behind one of them is a car
• The other two are goats
• You pick Door # 1
1 3
2
✅
©2023 F5
31
Conditional Probability – The Monty Hall Problem
Bayes Theorem
• You have three doors
• Behind one of them is a car
• The other two are goats
• You pick Door # 1
• The host shows you door # 2 and
offers to let you switch
• Do you switch or not?
1 3
2
✅
©2023 F5
32
1 3
2
𝟏
𝟑
𝟏
𝟑
𝟏
𝟑
1 3
2
✔
𝟏
𝟑
𝟐
𝟑
𝟏
𝟑
𝟏
𝟑 + =
{ }
1 3
2
✔
𝟏
𝟑
𝟐
𝟑
𝟎
𝟑
• Mathematical framework for
updating probability for an event as
new information becomes available
• Based on this theorem
• Initial probability of Door 1 = 1/3
• Initial probability of Doors 2 or 3 = 2/3
• Since it isn’t behind Door # 2 then
Door # 3 is the remaining 2/3
• Conditional Probability
Bayes Theorem
©2023 F5
33
Bayes Theorem in DevOps
Incident Troubleshooting
• You're seeing an increase in error rates
a<er a recent deployment. Historically,
80% of such incidents have been Ged to
database issues a<er major updates.
• Bayes' theorem suggests that the
current issue is database-related
considering the recent deployment and
historical data.
Predictive Monitoring
• Historically there's a 5% chance that
your application will crash on any given
day. Now, you get alerts about
increasing memory consumption.
• Bayes' theorem indicates the probability
of an impending application crash.
Optimizing A/B Testing
• You're testing a new feature against a
baseline to determine its impact on
server load. Initially, both variants might
be assumed equally likely to have an
effect.
• Data from the test allows you to update
the probability distribution of the effect
of the new feature on server load.
Bayes' theorem provides a mathematical framework for incorporating new evidence into existing beliefs
©2023 F5
34
Common Pitfalls in Statistics
Looking at the
wrong central
measure
Ignoring Scale
Confusing
correlation with
causation
Getting causation
backwards
Failing to see
biases
©2023 F5
35
Correlation and Causation
©2023 F5
36
• Statistics are how we tend to analyze our metrics
• Statistics are aggregation and reduction to reveal central tendencies
• They do not show individual behavior
• Most choices make use of very few basics
• But other choices may show amazing inferential results
• And finally
Summary
©2023 F5
37
• Statistics are how we tend to analyze our metrics
• Statistics are aggregation and reduction to reveal central tendencies
• They do not show individual behavior
• Most choices make use of very few basics
• But other choices may show amazing inferential results
• And finally
Summary
The most effective debugging tool is still careful thought,
coupled with judiciously placed print statements.
-Brian Kernighan Unix for Beginners 1979
©2023 F5
38
D(9) = 286 386 577 668 298
411 128 469 151 667 598
498 812 366
©2023 F5
39
Thanks!
Linkedin:
in/davemc
NGINX Community Slack
Slides on
GitHub

Weitere ähnliche Inhalte

Ähnlich wie OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Maarten Smeets
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformDevOps.com
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration TestingNick van Beest
 
Designing apps for resiliency
Designing apps for resiliencyDesigning apps for resiliency
Designing apps for resiliencyMasashi Narumoto
 
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...Edge AI and Vision Alliance
 
Time Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureTime Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureMarco Parenzan
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationThomas Ploetz
 
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Peter Tröger
 
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchHeuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchGreg Makowski
 
GLM & GBM in H2O
GLM & GBM in H2OGLM & GBM in H2O
GLM & GBM in H2OSri Ambati
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyAleksandr Tavgen
 
Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Celumusa Godfrey Nkosi
 
Amaya_Presentation
Amaya_PresentationAmaya_Presentation
Amaya_PresentationIsaias Amaya
 

Ähnlich wie OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister (20)

Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS Platform
 
Time series
Time seriesTime series
Time series
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration Testing
 
Process Control
Process ControlProcess Control
Process Control
 
Training - What is Performance ?
Training  - What is Performance ?Training  - What is Performance ?
Training - What is Performance ?
 
Designing apps for resiliency
Designing apps for resiliencyDesigning apps for resiliency
Designing apps for resiliency
 
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...
“Deep Neural Network Training: Diagnosing Problems and Implementing Solutions...
 
Time Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureTime Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and Azure
 
Time-resource v&v for complex systems
Time-resource v&v for complex systemsTime-resource v&v for complex systems
Time-resource v&v for complex systems
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
 
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
 
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchHeuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient search
 
GLM & GBM in H2O
GLM & GBM in H2OGLM & GBM in H2O
GLM & GBM in H2O
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the ugly
 
Multivariate Analysis
Multivariate AnalysisMultivariate Analysis
Multivariate Analysis
 
Statistics
StatisticsStatistics
Statistics
 
Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)
 
Amaya_Presentation
Amaya_PresentationAmaya_Presentation
Amaya_Presentation
 
Six Sigma Overview
Six Sigma OverviewSix Sigma Overview
Six Sigma Overview
 

Kürzlich hochgeladen

Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).luckyk1575
 
05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community NetworkingMichael Orias
 
527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdfrajpreetkaur75080
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.bazilnaeem7
 
OC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaOC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaRahsaan L. Browne
 
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22LHelferty
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxFamilyWorshipCenterD
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnairebazilnaeem7
 
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfOracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfSkillCertProExams
 
Hi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxHi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxShivamM16
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024SkillCertProExams
 
The Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFThe Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFRahsaan L. Browne
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxMogul Press
 
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Rahsaan L. Browne
 
123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptxgargh1099
 

Kürzlich hochgeladen (15)

Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).
 
05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking
 
527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.
 
OC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaOC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa Ana
 
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptx
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnaire
 
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfOracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
 
Hi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxHi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptx
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024
 
The Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFThe Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDF
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
 
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
 
123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx
 

OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

  • 1. Know Your Data: The stats behind the alerts Dave McAllister Sr. OSS Technologist NGINX
  • 2. ©2023 F5 2 Quick: What’s the difference between Mean, Median and Mode?
  • 3. ©2023 F5 3 Quick: What’s the difference between Mean, Median and Mode? And for extra credit, What’s the 9th Dedekind number?
  • 4. ©2023 F5 5 Monitoring is a numbers game • Metrics are numbers that represent selected behavior • Generally • Timestamped • Key-Values • Data, to be useful, must be • Aggregated • Analyzed • Visualized
  • 5. ©2023 F5 6 • How do you deal with outliers (spikes) in monitoring? • How do you get a representative value when values build on each other? • How do you arrive at values to represent rate of change over time? Do you know what your alert is really showing you? Some questions to ponder
  • 6. ©2023 F5 7 Mean, Median, Mode Median: Represents the middle value in a set of ordered data Mean: Measure of central tendency, represents average value of a set of data Mode: Value that appears most frequently in a set of data Data: 2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6, 3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4 Median = 6 Mode = 1 Mean = 5.444
  • 7. ©2023 F5 8 Mean, Median, Mode Data: 2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6, 3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4 Mean = 5.444 Or is it 4.130 or 2.791? Median: Represents the middle value in a set of ordered data Mean: Measure of central tendency, represents average value of a set of data Mode: Value that appears most frequently in a set of data
  • 8. ©2023 F5 9 Arithmetic, Harmonic, Geometric, Trimmed, Weighted, Moving Means to an End • Each has potential uses and drawbacks • Often already implemented in monitoring software • Can give very different results • Can make like and unlike comparisons easier
  • 9. ©2023 F5 10 Also often called Average • Most common • Is the central point in a normal distribution • This is not the 50% mark (mostly) • Useful for comparing current to previous conditions • May be aggregated into groups (time series) In a time series, we usually calculate constantly to incorporate new data Arithmetic Internet Load Balancer 300 Request/sec 200 Request/sec 100 Request/sec Amean = (100 + 200 + 300) / 3 Amean = 200 Requests per second
  • 10. ©2023 F5 11 • Multiply all the items together, take the nth root • Often used for things growing exponentially • In DevOps • Average number of deploys per unit of time • Average lead time for changes • MTTR • Throughput Geometric DevOps team: optimizing app deployment Sprint 1: 5% reduction Sprint 2: 10% reduction Sprint 3: 3% reduction Sprint 4: 7% reduction How well are they reducing deployment times? X1=1-0.05 = 0.95, x2 = 0.90, X3 = 0.97, x4 = 0.93 GM = 0.937 or Deployment has a 6.3% improvement
  • 11. ©2023 F5 12 Harmonic • Divide n by the sum of the reciprocals • Measure the performance where multiple systems are involved • Weights the lowest figure the highest • In DevOps • Performance within range • Overall indication of latency or thruput • Use in complex environments • Especially useful for outliers Internet Load Balancer 300 Request/sec 200 Request/sec 100 Request/sec HM = 150 Requests per second
  • 12. ©2023 F5 13 Take a comparison look What’s more important to the equation? • Latency • Thruput • GM is the moral equivalent • AM weights the larger number • HM weights the lower value Average Latency ms/sec Average Count/sec Suggested metric AM GM HM 250 40 10000 145.00 100.00 68.97 200 50 10000 125.00 100.00 80.00 400 25 10000 212.50 100.00 47.06 198 70 13860 134.00 117.73 103.43 105 99 10395 102.00 101.96 101.91 474 73 34602 273.50 186.02 126.52 195 97 18915 146.00 137.53 129.55 196 138 27048 167.00 164.46 161.96 133 10 1330 71.50 36.47 18.60 298 55 16390 176.50 128.02 92.86 0.00 50.00 100.00 150.00 200.00 250.00 300.00 1 2 3 4 5 6 7 8 9 10 Means overlay AM GM HM
  • 13. ©2023 F5 14 • Amazingly underutilized! • Center value of a sorted list • Median is always the 50% point of a normal curve Median Mean 25.25 Median 26
  • 14. ©2023 F5 15 Choosing Between Mean and Median • Mean can be impacted by outliers • Resilience is better in median • In DevOps • Response time monitoring • Anomaly Detection • Capacity Planning Old New Outlier 250 25.54 Mean 31.15 26 Median 26.5
  • 15. ©2023 F5 16 Old New Outlier 291 21.90 Mean 28.63 22 Median 22.5 18.61 Harmonic 19.06
  • 16. ©2023 F5 17 If you are using P95 you are using using a percentage value Congrats!
  • 17. ©2023 F5 18 Slight sidetrack: Measure of Variability How the numbers behave. • Standard Deviation • Range • InterQuartile Range (IQR) • Variance • Clusters • Outliers Properly used, variability can help you target outliers
  • 18. ©2023 F5 19 Those outside edges are good candidates for outliers Outlier Formula, visually Q2 IQR Outlier Formula Q3 Q1
  • 19. ©2023 F5 20 How about the Mode? • The most commonly recurring value in the set • Often presented as a histograph • Not commonly used in DevOps, mostly inferential • Log Analysis • Security Monitoring • User Behavior Analysis 0 2 4 6 8 10 12 14 one two three four five six seven eight nine ten Mode Chart
  • 20. ©2023 F5 21 Slight sidetrack: Descriptive versus Inferential stats • Descriptive uses the whole data set to draw statistical conclusions • Used for visualization • Can define and extract trends • Inferential uses a sampled set to draw conclusions • Used for predictions or hypotenuse testing • Can also visualize • But this leads us to sampling
  • 21. ©2023 F5 22 Dealing with the data • Monitoring is now a data problem • Observability signals: Metrics, Traces, Logs • Analysis is often • Aggregated or Analyzed in segments: Time-defined • Sampled and inferential • Random sampling • Stratified sampling • Cluster sampling • Systematic sampling • Purposive sampling
  • 23. ©2023 F5 24 • Changes behavior from Descriptive to Inferential • Can hide outlier behavior • Metrics are not usually sampled • May make forensics tougher • Lack of direct correlation • A necessary evil • But understand what you are giving up Sampling
  • 24. ©2023 F5 25 Distributions • Normal • Data equally distributed • Poisson • used to model the occurrence of rare events • Beta • Success/failure of binomial events • Exponential • Time between async events • Weibull • Likelihood of failure • Log-normal • Values based on many small events
  • 25. ©2023 F5 26 Slight sidetrack: Standard deviation • Measures the variability of your data • Identifies trends and outliers • NOT percentage based • Except with coefficient of variability • CV=Mean / std dev X 100 • Useful for measurement ignoring range • SRE cases • Lead times • Recovery times • Anomalies (alerts) • SLO / SLI This Photo by Unknown Author is licensed under CC BY-NC-ND
  • 26. ©2023 F5 27 Deeper dive: Weibull • Usually used for time-to-failure • Defined by a Shape and a Scale parameter • This can be challenging • Don’t ask the math • F(t)=1−e−(t/λ)β • R does it for you! Component Time-to-Failure Spinning Rust 500 hours Memory 1000 hours Power Supply 1500 hours CPU 2000 hours SSD 2500 hours library(fitdistrplus) data <- c(500, 1000, 1500, 2000, 2500) fit.weib <- fitdist(data, "weibull") summary(fit.weib) Fitting of the distribution ' weibull ' by maximum likelihood Parameters : shape 1.0624082 scale 2158.256 p.failure <- pweibull(300, shape = fit.weib$estimate[1], scale = fit.weib$estimate[2]) 1 - p.failure Failure (disk, memory) = 64.81% Failure (entire system) = 75.52% Fdisk(300)≈1−e−0.6918≈0.50025 Fmemory(300)≈1− −0.3490≈0.2959Fmemory(300)≈1−e−0.3490≈0.2959 Fpower(300)≈1− −0.2327≈0.2092Fpower(300)≈1−e−0.2327≈0.2092 FCPU(300)≈1− −0.1745≈0.1593FCPU(300)≈1−e−0.1745≈0.1593 FSSD(300)≈1− −0.1396≈0.1298FSSD(300)≈1−e−0.1396≈0.1298
  • 27. ©2023 F5 28 Deeper dive: Exponential • Models the “rate” (time between events that are unrelated) • Use cases • Network performance • User Requests (traces) • Messaging service • System failures • f(x) = me–mx • λ = 1/µ • e ~ 2.71828182846 Average Latency ms/sec Average Count/sec Suggested metric AM GM HM 250 40 10000 145.00 100.00 68.97 200 50 10000 125.00 100.00 80.00 400 25 10000 212.50 100.00 47.06 198 70 13860 134.00 117.73 103.43 105 99 10395 102.00 101.96 101.91 474 73 34602 273.50 186.02 126.52 195 97 18915 146.00 137.53 129.55 196 138 27048 167.00 164.46 161.96 133 10 1330 71.50 36.47 18.60 298 55 16390 176.50 128.02 92.86 Latency λ = 1/244.9 Probability density of 158 ms = 0.2142% Cumulative density = 47.5422% Expdist median ≈ 169.7 ms Throughput λ = 1/65.7 Probability density at 58 count = 0.628% Cumulative density = 58.7% Expdist median ≈ 45.6
  • 28. ©2023 F5 29 You may stumble upon: “On scale, statistics are not your friend” WRONG On scale, probability is not your friend. Slight Sidetrack and a pet peeve
  • 29. ©2023 F5 30 Conditional Probability – The Monty Hall Problem Bayes Theorem • You have three doors • Behind one of them is a car • The other two are goats • You pick Door # 1 1 3 2 ✅
  • 30. ©2023 F5 31 Conditional Probability – The Monty Hall Problem Bayes Theorem • You have three doors • Behind one of them is a car • The other two are goats • You pick Door # 1 • The host shows you door # 2 and offers to let you switch • Do you switch or not? 1 3 2 ✅
  • 31. ©2023 F5 32 1 3 2 𝟏 𝟑 𝟏 𝟑 𝟏 𝟑 1 3 2 ✔ 𝟏 𝟑 𝟐 𝟑 𝟏 𝟑 𝟏 𝟑 + = { } 1 3 2 ✔ 𝟏 𝟑 𝟐 𝟑 𝟎 𝟑 • Mathematical framework for updating probability for an event as new information becomes available • Based on this theorem • Initial probability of Door 1 = 1/3 • Initial probability of Doors 2 or 3 = 2/3 • Since it isn’t behind Door # 2 then Door # 3 is the remaining 2/3 • Conditional Probability Bayes Theorem
  • 32. ©2023 F5 33 Bayes Theorem in DevOps Incident Troubleshooting • You're seeing an increase in error rates a<er a recent deployment. Historically, 80% of such incidents have been Ged to database issues a<er major updates. • Bayes' theorem suggests that the current issue is database-related considering the recent deployment and historical data. Predictive Monitoring • Historically there's a 5% chance that your application will crash on any given day. Now, you get alerts about increasing memory consumption. • Bayes' theorem indicates the probability of an impending application crash. Optimizing A/B Testing • You're testing a new feature against a baseline to determine its impact on server load. Initially, both variants might be assumed equally likely to have an effect. • Data from the test allows you to update the probability distribution of the effect of the new feature on server load. Bayes' theorem provides a mathematical framework for incorporating new evidence into existing beliefs
  • 33. ©2023 F5 34 Common Pitfalls in Statistics Looking at the wrong central measure Ignoring Scale Confusing correlation with causation Getting causation backwards Failing to see biases
  • 35. ©2023 F5 36 • Statistics are how we tend to analyze our metrics • Statistics are aggregation and reduction to reveal central tendencies • They do not show individual behavior • Most choices make use of very few basics • But other choices may show amazing inferential results • And finally Summary
  • 36. ©2023 F5 37 • Statistics are how we tend to analyze our metrics • Statistics are aggregation and reduction to reveal central tendencies • They do not show individual behavior • Most choices make use of very few basics • But other choices may show amazing inferential results • And finally Summary The most effective debugging tool is still careful thought, coupled with judiciously placed print statements. -Brian Kernighan Unix for Beginners 1979
  • 37. ©2023 F5 38 D(9) = 286 386 577 668 298 411 128 469 151 667 598 498 812 366