Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Using Designed Experiments to Improve Service Quality in a Customer Care Environment
Using Designed Experiments to Improve
Service Quality in a Customer Care
Environment
Rocky Mountain Quality Conference
Denver, June 2003
Ed Powers
VP Corp. Planning and Development
Quality
Center Partners, Inc.
Fort Collins, Colorado 80525

Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 ed.powers@centerpartners.com 2 of 62
Objective
This presentation helps quality professionals better
understand and apply Design of Experiments (DOE)
principles in non-manufacturing or customer service
environments. It describes Center Partners’ business
challenges and how DOE techniques have helped
determine the effectiveness and ROI of new software
and training solutions.

Agenda
• Center Partners—Who We Are, What We Do
• Client Expectations and Business Challenges
• DOE Overview
• Using DOE in a Service Environment
• Example 1: Using DOE to Evaluate Performance
Management Software on Quality and Average
Handle Time Improvement
• Example 2: Using DOE to Evaluate Monitoring
Software and Coaching Effectiveness on Quality
Improvement
• New DOE Applications
• Summary
• Q&A

About Center Partners
• Founded 1997, in Fort Collins, Colorado, as a call
center outsourcer specializing in high-touch customer
care for complex products
• 8000% growth in first five years to over $80M in 2002
billables
• Purchased in 2001 by the WPP Group
• Currently answering over 2 million calls a month on
behalf of clients like Qwest Communications, Xerox,
Agilent and Comcast
• Hassle Free Contact Center Services. Done Right.
On Time.

Center Partners’ Basic Client Expectations
• Meet contract metrics:
– Service Level
– Average Handle Time (AHT)
– Quality
– Sales/Retention Goals
– Others

About Service Level…

About Average Handle Time…

Center Partners’ Business Challenges
• Meet or exceed contract metrics
• Delight clients
• Make money
To meet these challenges,
continuous service and
process improvement is not
optional!
To meet these challenges,
continuous service and
process improvement is not
optional!

Design of Experiments Defined
The arrangement in which an experimental program
is to be conducted, and the selection of the versions
(levels) of one or more factors or factor combinations
to be included in the experiment.
Source: ASQ Quality Press,Glossary and Tables for Statistical Quality Control, Second Edition, 1983, 160 pages

General Process Model
ProcessInputs Outputs
Controllable Factors
Uncontrollable Factors
x1 x2 x3
z1 z2 z3
Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages

Example: Golf
Factor Level
Driver Oversized or regular size
Ball Balata or three-piece
Conveyance Walk and carry clubs or use golf cart
Refreshments Beer or water
Time of day Morning or afternoon

Typical Experimentation
• “One factor at a time”
– What if something changes??
– Were there any interactions?
• “Best guess”
– What factor(s) caused the result??
– How do we know this is the best solution?
DOE tests many variables at
once, quickly and efficiently
with more useful results
DOE tests many variables at
once, quickly and efficiently
with more useful results

ANOVA—Workhorse of DOE
• ANalysis Of VAriance: Statistical method to separate
causes (factors) and effects (response) by
accommodating systemic randomness (experimental
errors)
• Tests statistical hypotheses and provides confidence
levels in conclusions.

How ANOVA Works
Group 1 Group 2
100 101
105 98
98 96
99 99
101 89
110 91
103 93
101 92
90 100
Compare means by
analyzing variation within
and between groups.
Between
Within

Industrial Example
• Engineer studying effect of varying cotton weight
percent in synthetic fiber on tensile strength of cloth
material
• Randomized experiment with a single factor at
multiple levels
• Hypothesis (H1): tensile strength will be different for
different percentages of cotton; “Null Hypothesis”
(H0): there is no effect

Example Data
Weight %
Cotton
Observed Tensile Strength (lb/in2
)
1 2 3 4 5 Avg.
15 7 7 15 11 9 9.8
20 12 17 12 18 18 15.4
25 14 18 18 19 19 17.6
30 19 25 22 19 23 21.6
35 7 10 11 15 11 10.8
Variation
Within
Treatments
Variation Between Treatments

ANOVA Computations
Source of
Variation
Sum of
Squares
Degrees of
Freedom
Mean
Square
F0 P-Value
Cotton
Weight %
475.76 4 118.94 14.76 <0.01
Error 161.20 20 8.06
Total 636.96 24
H0 is rejected; cotton weight % DOES affect tensile strength

Experimental Design and Analysis
• Fixed effects
• Random effects
• Regression
• Analysis of Covariance
• Randomized Complete
Block
• Latin Squares
• Graeco-Latin Squares
• Balanced Incomplete
Block
• Two-Factor Factorial
• 2k
, 3k
• Confounding
• ½ Fraction; ¼ Fraction
• General 2k-p
, 3k-p
• Multi-Factor Factorial
with Random Factors
• Nested and Split-Plot
• Multiple Regression

Using DOE in a Services Environment
• Fewer metrics; results often less tangible
• Many more potential variables—many uncontrolled
• Environments can be highly dynamic and may
influence testing
• Higher PEOPLE content

Process Factors
ProcessInputs Outputs
x1 x2 x3
z1 z2 z3
•People
•Methods
•Materials
•Equipment
•Environment
•Information

Continuum of Observed Performance
Performance is
Due to Chance
Alone
Performance is
Due People
Factors Alone
Process Individual
What % is the mix
in our business?

Detecting Non-Random Events
What is the minimum number of times would you
need to flip a coin to determine if it were not “fair”?

Detecting Non-Random Events
What is the minimum number of times would you
need to flip a coin to determine if it were not “fair”?
Solution: 7. Use the binomial distribution. Assume r = n (you get either all
“heads” or all “tails”). To be >99% that the effect is non-random,
determine n when y <=0.01:
y = pr
(1-p)n-r
n!
r!(n-r)!
y n
.5 1
.25 2
.125 3
0.0625 4
0.03125 5
0.015625 6
0.0078125 7
When n=r, y reduces to:
y = pn
With a perfectly
balanced coin,
there is less than
1% chance of
getting 7 heads or 7
tails in a row.
<1%>99%

Identifying LIKELY “High Performers”
Quality
AHT
Other-
Save
Rate?
y n
0.25 1
.0625 2
.0156 3
y n
0.33 1
.1089 2
.0359 3
.0119 4
p=0.25 p=0.33
y n
.10 1
.01 2
p=0.10
Example: 2 metrics both
in upper 10% yields a
99.0% certainty of non-
randomness
y n
.20 1
.04 2
.008 3
p=0.20

Good Performance Two Months in a Row
(One metric, July and August 2002, Population Average 55 Agents/Mo.)
July and
August
QA AHT
Upper 10%
(E=1)
Christy, Charles, Robert Christy, Andrea, Brian, Graham
Upper 20%
(E=2)
Christy, Charles, Robert, Rebecca,
Thomas
Christy, Andrea, Brian, Graham, Nancy,
Matthew
Upper 25%
(E=3)
Thomas
Matthew, Victor, Kyle
Upper 33%
(E=6)
Thomas, Trula, Jami
Matthew, Victor, Kyle, Rigoberto,
Jennifer, Stephanie, Robyn
In the Agent population, 13%
exhibit non-random behavior
for QA, 22% for AHT when
considering top 1/3 of Agents
In the Agent population, 13%
exhibit non-random behavior
for QA, 22% for AHT when
considering top 1/3 of Agents

Good Performance in the Same Month
(2 metrics, July and August 2002, Population Average 55 Agents/Mo.)
QA and AHT July August
Upper 10%
(E=1)
Christy Christy
Upper 20%
(E=2)
Christy Christy
Upper 25%
(E=3)
Christy Christy
Upper 33%
(E=6)
Christy, Stephanie, Robyn Christy, Sarah
Less than 6% of Agents
exhibit non-random behavior;
we can be at least 99% sure
Christy was a stand-out
Less than 6% of Agents
exhibit non-random behavior;
we can be at least 99% sure
Christy was a stand-out

Results
Agent
Performance is
Due to Chance
Alone
Agent
Performance is
Due Agent
Factors Alone
Process Individual
Quality AHT
At best, Agent factors
account for about 50% of
observed performance in
Center Partners’ business
At best, Agent factors
account for about 50% of
observed performance in
Center Partners’ business

More About People Factors
• People VARY from person to person
• People are HABITUAL
• People RESPOND DIFFERENTLY when obvious
supervision is present

Side-By-Side vs. Remote QA Scores
Analysis of Variance (jump start data.sta)
Marked effects are significant at p < .05000
SS df MS SS df MS
Effect Effect Effect Error Error Error F p
SCORE 3802.597 1 3802.597 21562.79 98 220.0285 17.28229 .000069
Summary Table of Means (jump start data.sta)
N=100 (No missing data in dep. var. list)
SCORE
R 78.29268
S 90.83051
All Grps 85.69000
Min-Max
25%-75%
Medianvalue
Box&WhiskerPlot:SCORE
HOW
SCORE
10
30
50
70
90
110
R S
>99.9% confidenceThere is a real difference in
quality scores between
remote and side-by-side
observation
There is a real difference in
quality scores between
remote and side-by-side
observation

Example 1:
Using DOE to Evaluate Performance Management
Software on Quality and Average Handle Time
Improvement

Structure of a Simple DOE Study
“Control” Group
“Test” Group
Production Group

The Brain-EKP®
Experiment
• The PDCA Cycle
• Plan: Issue, Measures, Causes
• Do: Experiment
• Check: Results
• Act: Learning and Next Steps

The Improvement Cycle
Plan
Do
Check
Act
•Understand the issue
•Understand the process
•Define the measures
•Uncover the root cause(s)
•Determine the solution
•Establish the goals
•Plan the project
•Execute the plan
•Implement the solution
•Compare results to goals
•If successful,
document, share
knowledge and
leverage solutions
•If unsuccessful,
revisit root causes
and redo the cycle

Plan: Situation Analysis (June, 2002)
• Issue: Center Partners performance below quality
goal
• Process: Call Handling (Center Partners Key
Business Process 6.4)
• Measures: QA score goal 80%; current process
average 76-78% and in a state of statistical process
control

Plan: Causal Analysis
Wrong support structure assumptions
Lack of sr. mgt. reinforcement
QA scores
not
improving
“Best
Practices”
issues
Client
scoring
differences
Agents not
motivated
Agents not
knowledge
-able
Not a priority
Too much time
spent on other tasks
Don’t know how
Lack time
management skills
Lack of automation
“Special” projects
Lack of support
“Emergencies”
Competing
responsibilities
Too few support people
Process defects
Agent life issues
Sporadic events/outages
Low support productivity
Don’t know job priorities
Not a personal priority
Lack of CSM
reinforcement
No BFTs
Client politics/ structure
Bias
Bad calibration
Complacency
Not reinforced by Coaches
Don’t know importance
Conflicting metrics/rewards
Don’t agree with forms
Changes not
communicated
Training
ineffective
Coaches don’t
communicate
Changes frequently
Not enough QA
focus
No Jump Start /
Base Camp
Bad data format
No ownership
Can’t find
information
Too much
reliance on
memory
Difficult to navigate
Time/AHT pressure
Changes frequently
Unaware of changes
High point weighting
Too ‘lazy’ to look
Skills not
habitual
Good habits not formed
Cause and Effect analysis
identified Best Practices and
Agent Motivation as potential
primary causes
Cause and Effect analysis
identified Best Practices and
Agent Motivation as potential
primary causes

Best Practices
Complete/Accurate Notes
Just Ask
Proactive Education
Assurance to help
Hold/Silence
Additional Assistance
Recap
Others
323 216 188 177 120 104 88 87 277
20.4 13.7 11.9 11.2 7.6 6.6 5.6 5.5 17.5
20.4 34.1 46.0 57.2 64.8 71.4 77.0 82.5 100.0
0
500
1000
1500
0
20
40
60
80
100
Defect
Count
Percent
Cum %
Percent
Count LVD SOC June 1-24th Pareto
Plan: Pareto Analysis
Data validation showed Best
Practices and
were 34%, coaching issues
were 41% of the issue
Data validation showed Best
Practices and
were 34%, coaching issues
were 41% of the issue

Plan: Solution Design
• Solution:
– Improve coaching process (a separate project)
– Address “Best Practices” issue with The Brain-EKP®
• Goal:
– Brain-EKP®
: 3% improvement in average QA scores
• Project Plan:
– Designed experiment with equally balanced Test and
Control groups
– Treatment with/without The Brain-EKP®
– Also study impact on Average Handle Time (AHT)

Example 1 Experimental Model
Call Handling
Process
Customer Calls
Resolved Issues
•QA Score
•AHT
Controllable Factor
Brain-EKP®
Coaching Client Changes

Plan/Do
Task Duration Completion
Kickoff 1 day 6/27/02
Experimental Design 3 days 7/12/02
Installation and Testing 11 days 7/17/02
Knowledge Model Development 7 days 7/24/02
Training and Roll-out 20 days 7/26/02
Experimental Runs 8 weeks 9/20/02
Progress Checks Weekly Weekly
Conclusions 1 week 9/20/02

Control
Test
0.7
0.8
0.9
1.0
Group
QAScore
Boxplots of QA Score by Group
(means are indicated by solid circles)
One-way ANOVA: QA Score versus Group
Analysis of Variance for QA Score
Source DF SS MS F P
Group 1 0.04995 0.04995 7.84 0.006
Error 155 0.98816 0.00638
Total 156 1.03811
Individual 95% CIs For Mean
Based on Pooled StDev
Level N Mean StDev --+---------+---------+---------+----
Control 77 0.83519 0.07752 (--------*--------)
Test 80 0.87088 0.08202 (-------*--------)
--+---------+---------+---------+----
Pooled StDev = 0.07985 0.820 0.840 0.860 0.880
Check: QA Analysis
99.4% certainty of a 3.6%
difference
Boxplots and ANOVA show
that the Test group
outperformed the Control
group
that the Test group
group

Others
Confidence
Vantive #
Best Practices
Used Systems Correctly
Empathy
Willingness to help
Upsell
Follow-up
Greeting
Pro-active education
62.5016.3022.0026.5026.7029.1033.7038.0039.7041.3356.10
15.94.25.66.86.87.48.69.710.110.514.3
100.084.179.974.367.560.753.344.735.024.914.3
400
300
200
100
0
100
80
60
40
20
0
Defect
Count
Percent
Cum %
Percent
Count
LVD SOC - Aug 19 - Sept 9 Brain Pilot Group
Check: Pareto Analysis
Best Practices was reduced
to the #8 issue;
was now out of the top 10
Best Practices was reduced
to the #8 issue;
was now out of the top 10

Test
Control
15
10
5
Group
AHT
Boxplots of AHT by Group
(means are indicated by solid circles)
One-way ANOVA: AHT versus Group
Analysis of Variance for AHT
Source DF SS MS F P
Group 1 19.17 19.17 3.44 0.066
Error 145 808.46 5.58
Total 146 827.64
Individual 95% CIs For Mean
Based on Pooled StDev
Level N Mean StDev ------+---------+---------+---------+
Control 71 10.469 2.061 (----------*----------)
Test 76 9.746 2.611 (----------*----------)
------+---------+---------+---------+
Pooled StDev = 2.361 9.50 10.00 10.50 11.00
Check: AHT Analysis
93.4% certainty of 43.4 second
AHT reduction
that the Test group
group
that the Test group
group

Act: Learning
• The Brain-EKP® provided a better user interface for
call handling than the one provided by our client,
resulting in better QA scores and AHT
• Technology alone is not sufficient—adequate
coaching and floor support is needed to ensure tool
usage and help modify Agent habits

Example 2:
Using DOE to Evaluate Monitoring Software and
Coaching Effectiveness on Quality Improvement

Plan: Situation Analysis (January, 2002)
• Issue: Center Partners performance at quality goal
but wanted incentive levels; quality monitoring
software supplier made claims that their system
would improve quality
• Process: Call Handling (Center Partners Key
Business Process 6.4)
• Measures: QA at 80% and in a state of statistical
process control; goal was to increase 5%

Plan: Pareto Analysis
Pareto of Failure Causes
0
5
10
15
20
25
30
35
40
H
abit
A
ttitude/O
ther
Learning
Category
Count
0
20
40
60
80
100
120
Cumulative%

Plan: Theory & Assumptions
• Rewards programs have proven unsustainable in the
past
• To change a habit, people need to focus on the issue
and receive feedback multiple times
• Many more observations and feedback in a shorter
time will help change the habits
• Seeing audio and video playback of calls during
coaching sessions will improve results
• Using visual aids will help remind Agents to conform
to QA expectations

Plan: Solution Design
• Solution:
– Run a statistically designed experiment across three sites to
test effects of quality monitoring software, intensive
feedback, and visual aids on QA scores
• Goal:
– 10% improvement in average QA scores
• Project Plan:
– 23
design

Example 2 Experimental Model
Call Handling
Process
Customer Calls
Resolved Issues
•QA Score
Site
Differences
Client
Changes
Software FeedbackVisual
Aids

Experiment Outline
• Identify 36 individuals to be part of the program.
Individuals will be chosen based on average score –
80% +/- 5%.
• 8 combinations (Feedback, Visuals, Software) tested
3 times each in Fort Collins (FTC) and Loveland
(LVD), 4 combinations (Feedback, Visuals) tested 3
times each in Idaho Falls (IDF)

Experimental Design (LVD, FTC)
REPLICAT NAME FEEDBACK VISUALS SOFTWARE LOCATION CHANGE COACH
1 Low Low Low LVD
1 High Low Low LVD
1 Low High Low LVD
1 High High Low LVD
1 Low Low High FTC
1 High Low High FTC
1 Low High High FTC
1 High High High FTC
2 Low Low Low FTC
2 High Low Low LVD
2 Low High Low LVD
2 High High Low LVD
2 Low Low High FTC
2 High Low High FTC
2 Low High High FTC
3 Low Low Low FTC
3 High Low Low LVD
3 Low High Low LVD
3 High High Low LVD
3 Low Low High FTC
3 High Low High FTC
3 Low High High FTC
24 Agents total
Examples of treatment:
High – Low – Low =
Multiple observations
Low – Low – Low =
Control Group
Low – High – Low =
Delivery of QA as normal
leave visual reminder on
their computer about
areas they missed
Low – Low – High =
include viewing software
as part of the “normal”
feedback session

Plan/Do
Task Duration Completion
Approval 1 day 1/11/02
Experimental Design and Planning 3 days 1/14/02
Kickoff 1 day 1/15/02
Training, Calibrations, and Roll-out 2 days 1/17/02
Experimental Runs 3 weeks 2/7/02
Analysis 2 days 2/11/02
Presentation of Conclusions 1 day 2/14/02

Check: FTC/LVD Results
Marginal Means (Unweighted); variable: DIFFERENCE
Design: 2**(3-0) design
NOTE: Std.Errs. for means computed from MS Error=.0062194
Pooled Overall Std.Err.
Feedback Visual Witness Means Std.Dev. Std.Dev. N for Mean
High Low Low .089333 .111142 .111142 3 .045532
High Low High .172967 .061224 .061224 3 .045532
High High Low .066667 .043716 .043716 3 .045532
High High High .043800 .118826 .118826 3 .045532
Low Low Low -.053800 .045048 .045048 3 .045532
Low Low High -.002467 .072750 .072750 3 .045532
Low High Low -.048333 .054243 .054243 3 .045532
Low High High .055333 .075755 .075755 3 .045532
The group exposed to high
feedback and monitoring
software but not visuals
improved 17%
The group exposed to high
feedback and monitoring
software but not visuals
improved 17%

Check: FTC/LVD Factor Analysis
Effect Std.Err. t(17) p
Mean/Interc. .040437 .016098 2.51198 .022392
(1)FEEDBACK .105508 .032196 3.27709 .004444
(2)VISUAL .022142 .032196 .68772 .500905
(3)WITNESS .053942 .032196 1.67543 .112144
1 by 2 .053775 .032196 1.67025 .113177
1 by 3 .023558 .032196 .73172 .474305
2 by 3 .013542 .032196 .42060 .679313
11% of the 17% gain can be
attributed to feedback
11% of the 17% gain can be
attributed to feedback

Check: FTC/LVD Factor Analysis
ParetoChartofStandardizedEffects;Variable:DIFFEREN
2**(3-0)design;MSResidual=.0062194
DV:DIFFEREN:=v9-v8
EffectEstimate(AbsoluteValue)
-.420604
-.687721
.7317222
1.670252
1.675429
-3.27709
p=.05
2by3
(2)VISUAL
1by3
1by2
(3)WITNESS
(1)FEEDBACK
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Check: Summary of All Sites
• Out of the 17 agents who received high feedback, 13
showed improvement
• Results show three out of four high feedback agents
showed an average of 9.7% improvement in their
scores in four days
• Model shows about a 1% gain per high frequency
feedback session on average
• Subsequent to the experiment, much of the gains
were lost in the first month!

Act: Learning
• High feedback is the primary cause of higher scores;
software and visuals contributed a minimum amount
—software supplier’s claims need refinement!
• Habits are more difficult to change than originally
anticipated

Desire
(Want to)
Knowledge
(What to do
/Why)
Forming Habits
Skill
(How to)
Habit
Source: Covey, S., The Seven Habits of Highly Effective People, 1990, 319 pages
Creating or changing a habit
requires work in all three
dimensions
Creating or changing a habit
requires work in all three
dimensions

New DOE Applications
• Training tactics and tools
• New technologies
• Incentive programs
• Marketing tactics

Summary
• DOE is an efficient and effective means for
understanding cause and effect relationships
• Simple DOE techniques can be used effectively in
non-manufacturing or service environments
• PEOPLE factors tend to be significant—especially
when changes require a change of habit

Author Biographical Information: Ed Powers
• VP Corporate Planning and Development for Center
Partners, Inc.
• 16 years of experience in sales, marketing, quality
management, and consulting
• Formerly with Hewlett-Packard, Sorcia
• BSEE 1987 Illinois Institute of Technology
• HP Quality Maturity System Reviewer, ASQ Certified
Quality Manager, 2003 Baldrige Examiner
• Published in AMA Marketing News, Call Center
Solutions magazines

Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Ähnlich wie Using Designed Experiments to Improve Service Quality in a Customer Care Environment (20)

Mehr von Ed Powers

Mehr von Ed Powers (9)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Hinweis der Redaktion