Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness

Copyright © 2014, SAS Institute Inc. All rights reserved.
Exploring best practises in
Design of Experiments
A Data Driven Approach to DOE Increasing
Robustness, Efficiency and Effectiveness

Malcolm Phil
Julie
Who’s here from jmp
Bernard Luke

jmp helps you make better decisions, faster

We will show you how you can
§ Simplify and make DoE work for more people in
more situations
§ Make use of existing data to have better
informed experiments
§ Make better decisions in less time

What we will cover today
Time Topic Speaker
0940 Introduction to Design of Experiments (DoE) Malcolm Moore
1025 Identifying key factors and optimising
processes using the key factors Phil Kay
1100 Break
1130 Example of DOE in Service Industries Malcolm Moore
1155 Effective experimentation when we have
constraints on the factor combinations Phil Kay
1220 Data Driven DoE and Choice Experiments Malcolm Moore
1250 Summary and close Malcolm Moore
1300 Adjourn for lunch

Help us to help you . . .

How often is DoE used in your
organisation?
(Select one)
1. Never
2. Rarely
3. Often
4. The default approach for experimentation

What is you organisation’s general view
of DoE (not your view which can be
different)?
(Select one)
1. Committed to it
2. Unsure what it is
3. Not really bothered
4. Tried it but it didn’t work
5. Against it

Are your experimental problems ever
complex (factor constraints, disallowed
combinations)?
(Select one)
1. Never
2. Rarely
3. Often
4. Always
5. Don’t know

Do you have existing data that you would
like to use to inform future experiments?
(Select one)
1. Never
2. Rarely
3. Often
4. Always

Contents
§ Background to DOE
§ Why Use DOE?
§ Tips for Effective DOE with Classical Designs
§ Definitive Screening
§ Case Studies 1-3
§ Role of Statistical Modelling and DOE in Learning
§ Data Driven DOE
§ Case Study 4

BACKGROUND TO DESIGN OF
EXPERIMENTS (DOE)

FATHER OF DOE RONALD A. FISHER
Rothamstead Experimental Station, England – Early 1920’s

FISHER’S FOUR DESIGN PRINCIPLES
1. Factorial Concept - rather than one-factor-at-a-time
2. Randomization - to avoid bias from lurking variables
3. Blocking - to reduce noise from nuisance variables
4. Replication - to quantify noise within an experiment

AGRICULTURAL IMPACT
US corn yields
Cornell University, http://usda.mannlib.cornell.edu/MannUsda

WHY USE DOE?

Inputs
Factors
Machine
Operator
Temperature
Pressure
Humidity
Typical Process
The properties of products and processes are often affected
by many factors:
Typical
Process
Outputs
Responses
Yield
Cost
…
In order to build new or improve products and processes, we
must understand the relationship between the factors (inputs)
and the responses (outputs).

Traditional One-Factor-at-a-Time
§ A common approach is one-factor-at-a-time experimentation.
§ Consider experimenting one-factor-at-a-time to determine the
values of temperature and time that optimise yield.

§ One-factor-at-a-time
experimentation frequently
leads to sub-optimal
solutions.
§ Assumes the effect of one
factor is the same at each
level of the other factors, i.e.
factors do not interact.
§ In practice, factors frequently
interact.

Interaction between factors

Experimental Design
§ Most efficient way of investigating relationships.
§ Runs (factor combinations) chosen to maximize the information
§ Ideally balanced for ease of analysis and interpretation

ITERATIVE AND
SEQUENTIAL NATURE
OF CLASSICAL DOE

TIPS FOR EFFECTIVE DOE WITH
CLASSICAL DESIGNS

Stages of Experimental Design
§ Designing an experiment involves much more
than just selecting the sequence of experimental
runs:
Plan Design Conduct Analyse Confirm
§ Historically, improper planning is the most
common cause of failed experiments.

Some Planning Steps
§ Review what we know
• Have peer discussions
§ Determine new questions to answer
§ Identify factors and ranges to investigate
§ Define responses
• Easy and precise to measure

Common Experimental Objectives
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process RSM Design
Classical
Central
Composite
Optimise
Ingredients Mixtures
Classical
Simplex &
Extreme
Vertices

Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Sequential Experimentation Reduces Total Cost
Optimise
Process RSM Design
Classical
Central
Composite
Optimise
Classical
Simplex &
Extreme
Vertices

Definitive Screening Design Simplifies Experimental Workflow
Sequential Experimentation
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Optimise
Process RSM Design
Classical
Central
Composite
Optimise
Classical
Simplex &
Extreme
Vertices

Sequential Experimentation
Identify
Important
Factors
Screening
Design
Classical
Fractional
Factorial
Definitive Screening Design
Optimise
Process RSM Design
Classical
Central
Composite
Optimal Design Manages Experimental Constraints
Optimise
Classical
Simplex &
Extreme
Vertices

Determining the Appropriate Factors
§ Determining the factors to be included in your experiment is a
critical part of planning.
• Exploring too many factors may be costly and time
consuming.
• Exploring too few may limit the success of your experiment.
§ Prior knowledge and analysis of existing data are useful aids to
identifying and prioritising factors for study. Other methods may
include:
• Brainstorming
• Ishikawa

Selection of Factor Range is Critical With
Two Level Designs …

By experimenting at the two settings in
yellow, X would be declared unimportant

By using half and often times much less than
than half the factor range X is declared important

By using half and often times much less than
than half the factor range X is declared important
Often leads to narrow factor ranges
to force linear relationships but
consequence is high risk of
determining sub-optimal solution

Determining the Appropriate Responses
§ Selection of your responses will also be critical to the success of
your experiment. Whenever possible:
• Choose variables that correlate to internal or external
customer requirements
• Find responses that are easy to measure
• Make sure your measurement systems are precise, accurate,
and stable
§ Analysis of current data, prior knowledge, measurement systems
analysis are useful aids.

DEFINITIVE SCREENING

Fractional Factorials: Complex workflow
from many factors to optimum settings
Tempting to miss out
middle step which can
result in selection of
wrong factors and decisions

Definitive Screening Design
§ Identifies active main effects, uncorrelated with other
effects.
§ May identify significant quadratic effects, uncorrelated with
main effects and at worst weakly correlated with other
quadratic effects.
§ If few factors turn out to be important, can identify
significant two-way interactions uncorrelated with main
effects and weakly correlated with other higher order
effects.
§ One stage experiment if three or fewer factors important:
• progress straight to full quadratic model
• optimise process with no further experimentation
• otherwise augment DSD for optimization goals

New Class of Screening Design
§ Three-level screening
design
• 2m + 1 runs when m is even
• 2m + 3 runs when m is odd
• 1 additional run for
categorical factors
• based on m fold-over pairs
and an overall center point,
where m is number of factors
• the values of the ±1 entries in
the odd-numbered runs are
determined using optimal
design.
the structure illustrated in Table 1. We use xi,j to
denote the setting of the jth factor for the ith run.
For m factors, there are 2m + 1 runs based on m
fold-over pairs and an overall center point. Each run
(excluding the centerpoint) has exactly one factor
level at its center point and all others at the ex-tremes.
As described in the next section, the val-ues
of the ±1 entries in the odd-numbered runs of
TABLE 1. General Design Structure for m Factors
Factor levels
Foldover Run
pair (i) xi,1 xi,2 xi,3 · · · xi,m
1 1 0 ±1 ±1 · · · ±1
2 0 !1 !1 · · · !1
2 3 ±1 0 ±1 · · · ±1
4 !1 0 !1 · · · !1
3 5 ±1 ±1 0 · · · ±1
6 !1 !1 0 · · · !1
...
... ...
...
...
. . .
...
m 2m − 1 ±1 ±1 ±1 · · · 0
2m !1 !1 !1 · · · 0
Centerpoint 2m + 1 0 0 0 · · · 0
of linear and quadratic main-effects terms.
5. Quadratic effects are orthogonal to main effects
and not completely confounded (though corre-lated)
with interaction effects.
6. With 6 through (at least) 12 factors, the de-signs
are capable of estimating all possible full
quadratic models involving three or fewer fac-tors
with very high levels of statistical effi-ciency.
We use the term “definitive screening” because of
points one through five above. These are small de-signs
that, unlike resolution III and IV factorial de-signs,
permit the unambiguous identification of ac-tive
main effects, active quadratic effects, and, in the
presence of a moderate level of effect sparsity, active
two-way interactions.
In our view, another practical advantage of the
designs we propose is the explicit use of three levels.
It has been our experience that engineers and scien-tists
often feel some discomfort using two-level de-signs
for two reasons. First, statisticians advise them
to experiment boldly by choosing a substantial inter-val
between low and high values of each factor. But
their scientific training inculcates the notion that the
functional relationship between independent and de-pendent
variables is usually nonlinear, particularly
over a wide range. This leads to some cognitive dis-sonance
in considering the use of two-level designs.
Second, even in the early stages of a study, investiga-tors
frequently have an opinion regarding the “best”
Journal of Quality Technology Vol. 43, No. 1, January 2011

Use of Three Level Designs
Advantageous
§ Scientists and engineers are uncomfortable using two-level designs
• Restricting factor ranges may result in sub-optimal solutions
• Scientific/engineering judgment suggests relationships nonlinear over
wide ranges
§ Investigators frequently have an opinion regarding the “best” levels
of each factor for optimizing a response
• Experimental region centered at these levels.
• Two-level design might screen out an important factor when
experimental region centred at “best”
• Adding centre points allows test for curvature
• However ambiguity over factors causing curvature
• DSD avoids ambiguity by making it possible to uniquely identify the
source(s) of curvature.

CASE STUDIES

Case Study 1: Optimising a Chemical
Process
Why Consider Definitive Screening Designs?

Background
§ Five factors
§ One response yield
§ Goal optimise yield
§ Keep total cost of experimentation to minimum
§ Contrast traditional approach of main effect screening
design plus augmentation to RSM with DSD

§ Traditional screening approach correlates main
effects with two factor interaction effects
§ Cost constraint and inexperience with such
designs can lead to missed DOE steps
§ Investigator missed step of augmenting main
effect design to separate correlated interaction
effects from assumed important main effects
§ Resulted in wrong set of factors selected for
RSM design which results in wrong solution
Background

Traditional Approach with Missed Step

Resolution III Design Perfectly Correlates
Main Effects With Interaction Effects

Model Interpretation
§ Fitted Model
Y = b0 + b1*X1 + b2*X2 + b3*X3 + Error
§ Correct Interpretation of Fitted Model
Y = b0 + b1*(X1+X2X3) + b2*(X2+X1X3) + b3*(X3+X1X2) + Error

Missed Step Augments Initial Design to
Separate Main Effects From Interactions

Model Interpretation of Augmented Design
§ Correct Interpretation of Model Fitted to Augmented design
Y = b0 + b1*X1 + b2*X2 + b3*X3 + b12*X1X2 + b13*X1X3 + b23*X2X3 + Error
§ Allows clear separation of main and interaction effects
§ This step was missed in case study prior to modelling curvature

§ DSD results in correct identification of important
factors due to non correlated main and two factor
interaction effects
§ Because just three factors are important DSD
results in one step design:
• In addition to correctly identifying correct factors
• DSD requires no augmentation to identify optimal
settings of important factors
Background

CASE STUDY 1

Conclusions
§ Fractional factorial designs can lead to selection of
wrong factor set
§ Complex workflow for avoiding this risk which may be
misunderstood or not applied by users new to DOE
§ May lead to conclusion that DOE does not work for us!
§ DSD simplifies DOE process and removes risk of
selecting wrong factor set
§ Provides one step DOE when three or fewer important
factors
• Sufficient to identify correct factor set and determine best
settings of selected factors

Case Study 2: Optimising Marketing
Response Rate and Profitability
Definitive Screening Design for Efficiency

Background
§ Goal is to maximise return from credit card
marketing campaigns. Two outputs:
• Response rate - percentage mailed a credit card offer
who accept the offer;
• Indexed usage – average profit per individual over a
twelve month period.
§ Factors are balance transfer period, interest free
period for new purchases and %APR at end of
any introductory offers.
§ Goal: determine characteristics of credit card
offer that maximises response rate and
profitability.

CASE STUDY 2

Conclusions
§ DSD can be cost effective with few factors when
cost of experimental run is high
§ Tradeoff is greater uncertainty (reduced power)
in decisions

CASE STUDY 3

Case Study 3: Optimising Yield
What About Constrained Factor Spaces?

Background
§ From chapter 5 of Goos &
Jones
§ Chemical reaction
§ Goal: maximise yield
§ 2 factors: Temperature and
Time

Background
§ Expert knowledge tells us
• Certain conditions will give
poor results (hence,
constraints)
• Behaviour very non-linear
§ We will show
• Design where prior
knowledge is ignored.
• Fitting the design to the
problem

Example of Process Constraint

Shrink Experimental Range to Factorial

Optimal Design: Use Actual Factor Range

Optimal Design: Fit to Model
The process is not seen as a black box anymore…
… optimal designs allow investigation of complete factor
space properly adjusted for constraints
Typical
Process
Machine
Operator
Temperature
Pressure
Humidity
Yield
Cost
…
Inputs
Factors
Outputs
Responses
Model
Y = f(X)

Conclusions
§ Custom Design permits studying any:
• combination of factors with or without constraints,
• number of factor levels,
• blocking structure.
§ Build your design to suit the problem instead of
fitting the problem into a design

Case Study 4: Designing Products People
Want to Buy
Data Driven DOE

ROLE OF STATISTICAL MODELLING
AND DOE IN LEARNING

LEARNING IN THE FACE OF UNCERTAINTY
Data Driven DOE Integrates Incremental Learning
Across DOE and Observational Sources of Data
Able to Consistently Meet Customer Requirements
What is
really
happening
Y = F(X) + Error
Measurement
and Data
Collection
Situation
Appraisal
Situation
Appraisal
Adapted from Box, Hunter and Hunter Copyright © 2014, SAS Institute Inc. All rights reserved.
76
What we
think is
happening
Measurement
and Data
Collection Analysis
Situation
Appraisal
Measurement
and Data
Collection Design
Real World Model
Unable to Consistently Meet Customer Requirements

Simple Process of Statistical Learning
DOE Data ….…. Observational Data

Data Sources
§ DOE and/or observational (historical)
§ Potential problems with observational data:
• X’s are correlated – identification of “best” model
difficult
• Outliers (potential or real) - bias model estimation
• Missing data cells – result in loss of whole data rows
with traditional least squares based analysis
• Range over which X’s varied may be limited –
restricting model usefulness
• May not have measured all relevant X’s
§ In some situations these can also be issues with
DOE datasets

WHAT IS DATA DRIVEN DOE?

Data Driven DOE: Integrating Statistical
Modelling and DOE
§ Learning is incremental and effective statistical modelling of
observational data aids design of next experiment.
§ Analysis approach needs to manage real (messy) data simply
• Correlated X’s, outliers, missing cells
• Quickly deliver “best” current model to revise with new DOE data
• Aid better analysis of new experimental data when unexpected
occurs
• Build models based on individual datasets and aggregated data
§ Good statistical modelling integrated with DOE helps reduce
total learning time, effort and cost
§ It would be a shame to not use pre-existing data that comes
for free

JMP Statistical Discovery: Integrating
Statistical Modelling with DOE
Effectiveness
Of Learning
Statistical
Discovery
Speed of
Learning
Traditional
Approaches
§ Integrated methods
§ Ease of use
§ Manage messy data
§ Wide array of DOE
approaches
§ Satisfy (customer)
needs
§ Reduce learning time
§ Save effort and cost

DATA DRIVEN DOE EXAMPLE

Background
§ PC retailer is observing appreciable retail price
variation in its laptop computer line.
§ Goals:
• Investigate factors associated with retail price variation.
• Perform further experimentation in key factors to
optimise and standardise pricing across stores.

CASE STUDY 4

Conclusions
§ Analysis of prior data helps identify factors and
ranges to use in next DOE.
§ Analysis of prior data helps reduce risk and
increase efficiency and effectiveness of future
experiments.
§ Exploit prior data that comes for free to inform
next experiment.

Data Driven DOE: Integrated Statistical
Modelling and DOE
§ Supports wide range of user skills
§ Exploratory analysis and statistical modelling of historical
messy data simplifies and shortens whole DOE process.
§ Next generation DOE enables more staff to apply DOE with
reduced learning and implementation effort
§ Interact with model predictions to build consensus
§ Integrated simulation capabilities enables rapid progression
from models to decisions
§ Manage risk better by correctly identifying signal from noise

QUESTIONS?

We have shown you how you can
§ Reduce the risk of wrong decisions
• Make DoE work for more people in more situations
§ Fit the best design to your problem
• Find the best solution while managing system
constraints
§ Mine your “messy” data to inform future
experiments
• Make better decisions in less total time using Data
Driven DOE

Make better decisions, faster with jmp

Supplier of Digital Printing Materials
§ Needed to double capacity of a product line to meet
growing demand.
§ Poor understanding of key process step responsible for
increasing capacity.
§ Large number of potentially important variables and limited
budget for experimentation.
§ Definitive Screening Design enabled screening of all
factors and process optimisation in a small number of runs
to achieve doubling of production rate without additional
capital investment.
§ Saved £100,000s off development budget and enhanced
the credibility of the site as a location for cost-effective
high-value manufacturing within a multi-national
organisation.

Large Multi-National Chemical Company
§ Losing market share to start-ups who were faster
at introducing new products and more agile at
adapting to changing customer requirements.
§ Needed to get more products to market faster.
§ Instituted a culture of experimentation with JMP
Pro for variable selection and DOE to accelerate
cycles of learning, enabling more new products
to be introduced faster.
§ Helped retain and grow market share, facilitating
increased dividend growth to shareholders and
increased staff retention and satisfaction.

What are you going to do next?
Ask us to help you
Download a trial of JMP
§ Visit our website: www.jmp.com
Join our Design of Experiments Webcasts:
§ Exploring Best Practise in DoE: 14:00 on 20
November
§ Invite your colleagues
§ Mastering JMP on DoE: 1400 on 14:00 on 14
November
Register on our website

Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness

Ähnlich wie Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness (20)

Mehr von JMP software from SAS

Mehr von JMP software from SAS (18)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness