An introductory overview to an extended series about hypothesis testing. This lesson includes the general 4 step process used for hypothesis testing.

1. Section & Lesson #:
Pre-Requisite Lessons:
Complex Tools + Clear Teaching = Powerful Results
Hypothesis Testing: Overview
Six Sigma-Analyze – Lesson 9
An introductory overview to an extended series about hypothesis testing.
This lesson includes the general 4 step process used for hypothesis testing.
None
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted by any means
(electronic, mechanical, photographic, photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

2. An Example from our Judicial System
o To understand hypothesis testing, let’s first look at an example of our judicial system.
• An entire Lean or Six Sigma project can be compared to a courtroom trial as described below:
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
2
Jury = Sponsor/Champion
Make final decision and determine
consequences/actions.
Judge = Peers, Mentor or MBB
Reviews process to ensure methodology is
followed and sufficient evidence is provided.
Defendant = Root Cause
Represents the source of the
problem and why there’s
a case in the first place.
Prosecutor = Six Sigma Black Belt
Primary leader for evaluating evidence
and proving to the
jury the Defendant’s guilt.
Witness/Evidence = Data Analysis
The proof used by the Prosecutor to
demonstrate the Defendant’s guilt
beyond “a reasonable doubt” .

3. Hypothesis Testing Defined
o What is hypothesis testing?
• It’s a formal method of statistically testing and validating data for differences between factors.
• It’s a method of testing one factor (X) at a time to see if/how it influences the output (Y).
o What is the purpose of hypothesis testing?
• To find evidence that statistically proves what is the
root cause for the original problem in the project.
o How does it fit into our judicial system example?
• It represents all the testing and analysis done
on the evidence before the trial.
• The burden of proof is on the prosecution; they must
provide sufficient evidence to convict the Defendant.
• Likewise, hypothesis testing is used to validate if the
evidence is sufficient to prove what is the root cause.
o What if the prosecution doesn’t have evidence?
• Generally, if the prosecution can’t fulfill their burden of proof, their case is dismissed.
• Likewise, if there’s insufficient evidence to prove the root cause, the project may be at risk.
 At a minimum there is no data to validate the root cause so the risks of continuing should be considered.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
3

4. Gathering the Evidence
o The police gather and analyze evidence the same way we do in DMAIC.
• The DMAIC methodology of Six Sigma is used for solving a problem with an unknown root cause.
• Likewise, the police use a similar method to solve a crime having an unknown suspect.
o Remember, the Measure phase is often the most critical and most neglected phase.
• The Measure phase was when we gathered the data that may become evidence.
• In solving a crime, if the evidence is not reliable, then it can’t be used to convict.
• Likewise, validating the evidence using hypothesis testing is useless if the data isn’t reliable.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
4
Define
Measure
Analyze
Improve
Control
Understand
the problem
Gather
reliable info
Identify
root cause
Fix
root cause
Sustain
improvement

5. Examining the Evidence
o How do we examine the evidence?
• Again, the approach is very similar to how the police examine evidence as described below:
• Just as evidence is examined for each suspect, each factor (X) is examined in the data.
• Just as evidence must link the suspect to the crime, hypothesis testing may similarly prove the
“means, motive & opportunity”, but the compiled evidence must link the suspected factor (X) as
the true root cause of the problem.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
5
Q1: Who committed the crime?
Suspect A Suspect B Suspect C
Means
How did they
commit the crime?
Motive
Why did they
commit the crime?
Opportunity
When/Where did they
commit the crime?
Link to Crime
How is the Suspect linked to the actual crime?
Q2: Is Suspect B guilty?
It’s not enough to
prove means, motive
& opportunity….
…the suspect must
also be linked to the
crime to prove guilt.

6. Proof Beyond a Reasonable Doubt
o How much proof is needed for “convicting” the root cause?
• In our judicial system, a suspect is innocent until proven guilty “beyond a reasonable doubt”.
• Likewise, prudent data analysis presumes innocence on each factor (X) until proven “guilty”.
 Hypothesis testing does this by starting the test with a null (innocent) and alternative (guilty) hypothesis.
 It measures “reasonable doubt” using alpha (α) and beta (β) error types (more on this later).
o How much evidence is needed for proof beyond reasonable doubt?
• It doesn’t require 100% facts, but it depends on the “jury” (i.e., the Sponsor & Champion).
 Refer to the ABC model (from “Introduction – ABC Model”):
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
6
Consequences
What arguments would persuade
their beliefs that would lead them
to make the right conclusions?
Many decisions are based
on assumptions and
personal experience…
…but by measuring them
and doing hypothesis
testing on them…
…we can make them a more
solid & weighty argument
that can tip the Belief Scale.
BELIEF SCALE
Ignorance
(Nothing)
Theoretical
(Assumptions)
Empirical
(Experience)
Scientific
(Data)
Truth
(Facts)
Lighter Weight
Intangible
Not Repeatable
Limited Application
Limited Accessibility
More Emotions
Less Confidence
Heavier Weight
Tangible
Repeatable
Universal Application
Universal Accessibility
Less Emotions
More Confidence

7. Measuring Statistical Risk
o Statistics generally use two measurements for risk: Alpha (α) and Beta (β).
• To explain these, let’s assume that all possible evidence is represented as an empty glass.
 The glass is filled with what data/evidence we have.
 How full should the glass be for me to make a reasonable conclusion?
o What is the ideal level of risk I need?
• It depends on the inherent risks of what’s being analyzed.
 Would you want 5% risk for pharmaceutical testing, or surgical equipment, or nuclear engineering?
• Most statistical tests require a confidence level (1-α), but not many require a power level (1-β).
o Bottom Line: if the test results are unexpected or fall between the alpha & beta risks,
then test again with more samples.
7
Data
(Evidence)
α Alpha Risk threshold
• Type I Error (false positive); 5% default
• Confidence Level = 1 – α (or 95%)
• Judicial Ex. = convicting an innocent man
(lower threshold of “reasonable doubt”)
• Statistical Ex. = we say a factor causes a
difference when it really doesn’t
• Practical Ex. = we fix something that
isn’t broken
95%
90% β Beta Risk threshold
• Type II Error (false negative); 10% default
• Power Level = 1 – β (or 90%)
• Judicial Ex. = acquitting a guilty man
(upper threshold of “reasonable trust”)
• Statistical Ex. = we say a factor doesn’t
cause a difference when it really does
• Practical Ex. = we divert attention away
from the real root cause
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

8. Hypothesis Testing: 4 Step Process
o The 4 step process for hypothesis testing begins and ends with keeping it practical:
o Steps 2 & 3 are the heart of hypothesis testing where we’ll spend the most time.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
8
Practical Statistical
Problem Problem
Solution Solution
 

Practical Problem
State the problem as a
practical Yes/No question.
 Statistical Problem
Convert the problem to an
analytical question identifying
the statistical tool/method.

Practical Solution
Interpret the analytical
answer in a practical way.
Statistical Solution
Interpret the results of the
hypothesis test with an
analytical answer.


9. Practical Application
o Think of at least 2 prior projects or initiatives you worked in your organization.
• How confident are you that the results (i.e., your conclusions) from those examples were right?
 What are you basing that confidence on?
 If your confidence for either example is less than 95%, then what are some of the “reasonable doubts” you
have about the accuracy of those results?
 If you were asked to present those examples for a real legal case in court, would you have the same level of
confidence?
– If not, then why not?
 Would you change the data or analysis that was done before submitting it as evidence?
– What changes would you make and why?
– Are these changes indicative of risks that you or others overlooked in the project or initiative? If so, then why were these
overlooked and not further analyzed or validated to mitigate that risk?
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
9