Got Results? Evaluating Training Impact

Got Results?
Evaluating Training Impact

Karen Carleton, MEd, MS
Performance Solutions Corp.
www.performcorp.ca

CSTD Symposium
Edmonton
May 31, 2012
Copyright © 2012 Karen Carleton

Poll
How certain are you that your Training is
getting the desired results?
A) Very certain
B) Moderately certain
C) Somewhat certain
D) Not certain at all
E) N/A

“Intangible assets represent
…30-50% of a company’s
market value.”
-Ernst & Young (in Gayeski, 2005)

Human capital


2. Training
Evaluation
Standard
(old & new)

1. Learning
Objectives
&
Performance
Objectives
3. ROE vs. 4. Transfer
ROI & Impact
4


2. Training
Evaluation
Standard
(old & new)

1. Learning
Objectives
&
Performance
Objectives
ROI & Impact
5

Train

Skill-based

Results
focused,
practical


Educate

Knowledge-
based

Capacity
building

7

Learn Relevant

Immediately
meaningful


KSAs (Competencies)
Knowledge – cognitive
learning

Skills - behavior learning

Attitudes – social
learning
9


Training Caution the Gap”
“Closing

Learning Objectives

SWBAT/LWBAT
Student/Learner
Will Be Able To…

e.g. Create Excel
spreadsheets


PCC Method
Performance
Criteria
Conditions
e.g.
P – Will type
C – 45 wpm with 2% or less error rate
C – after 2 hours of daily practice for 3 weeks.
“Will be able to type.”

Performance Objectives

Example (L or P)
Trainees will be able to
operate the crusher after
practical training.

Learning objective

A) Learning Objective
OR
B) Performance Objective


Example (L or P)
Employees will
correctly
demonstrate
techniques for
resolving conflict, by
the end of the 2-day
workshop and doing
guided practice
exercises.
OR Performance
B) Performance Objective objective

Example (L or P)
Welding apprentices
shall learn how to
prepare an edge before
welding a plate edge or
pipe.
Learning objective
OR
B) Performance Objective Copyright © 2012 Karen Carleton

Example (L or P)
Following an in-class
activity and video on
new sterilization
procedures, the
nursing aides will
accurately
demonstrate the
proper procedure.

Performance
objective OR
Copyright © 2012 Karen Carleton B) Performance Objective

Learning vs. Performance Objectives
1. Most common? A)Learning Objectives
B)Performance
2. Preferable? Why? Objectives


Training Design

Target
Needs Draft
Audience


ADDIE Model

Feedback loop
19

Evaluation - determining value or worth;
examining, judging; giving an appraisal.
Threat OR Learning
opportunity



2. Training
Evaluation
Standard
(old & new)

1. Learning
Objectives
&
Performance
Objectives
ROI & Impact
21

Training Evaluation
Gold Standard
Kirkpatrick Model

Kirkpatrick Model

Kirkpatrick Model
4 Levels of 50+ year Universally
Evaluation old model recognized;
CSTD/ASTD


Kirkpatrick’s 4 Levels

1
• Learner Reaction

• Learning (KSAs)
2

• Behavior change (applied
3 learning; on-the-job)
• Organizational results ($,
4 time, quality, etc.) Dr. Donald
Kirkpatrick

4 Levels Stats
Level 1 – most companies
“Smile sheets”, forms, verbal

Level 4 - 5-10% companies
Recommend: evaluate all 4 levels
Increased complexity & cost with levels
Which Training Evaluation level(s) does your company use?
A) 1 B) 2 C) 3 D) 4 E) Combination of levels

NEW 4 Levels Model - KBPM
“IMPACT METRICS”
Level 4 – Results
“CONSUMPTIVE
For Org. (outcomes)
METRICS”
Level 3 – Behaviour
(applied learning)

Level 2 – Learning of
knowledge/skills (degree)
Level 1 – Reaction
favourably to learning
event
Dr. Jim Kirkpatrick

Desired Results

“The Ultimate intent of the Kirkpatrick
Four Levels…is to show the value and
worth of training”
–Kirkpatrick & Kayser Kirkpatrick (2009)

“Begin with the end in mind”
-Stephen Covey (1989)


KPBM Model

http://www.kirkpatrickpartners.com/


KBPM Model
Don’t assume “Transfer”

Consistently reinforce
learning & application
- e.g. coaching/mentoring,
feedback, refreshers, job aids
A) Training/L & D is “On Trial” internally
B) Training/L & D is “On Trial” as a profession
C) Both
D) Neither
http://www.kirkpatrickpartners.com/ Copyright © 2012 Karen Carleton


2. Training
Evaluation
Standard
(old & new)

1. Learning
Objectives
&
Performance
Objectives
ROI & Impact
29

Kirkpatrick Business Partnership
Model

Return On Expectations (ROE)
Stakeholder value
Contributions to “Success”

Business Partnership
Cooperation
Compelling “chain of evidence”
http://www.kirkpatrickpartners.com/ Copyright © 2012 Karen Carleton

Return on Investment (ROI)

PhD HRD, U Alabama;
~35 books about ROI in
~33 languages

ROI Institute
http://www.roiinstitutecanada.com/

Dr. Jack Phillips

ROI Methodology
1. Reaction, Satisfaction, Planned Action
2. Learning
3. Application/Implementation
4. Business Impact
5. Return on Investment (ROI)
6. Intangibles

$$$ “Level 5” $$$

(Program Benefits – Program Costs) = ROI
Program Costs
 Net profit/savings; business case
 Less than 5% organizations do ROI
Why?


Bottom-line results Isolation techniques
Training impact (program vs. other
causes) e.g. trends,
before/after test
ROI challenges
- Resources, know-
how
- Linking results back
to program
(Gayeski, 2005)


Jack vs. Jim
ROI ROE
Dr. Jack Phillips Dr. Jim Kirkpatrick

http://talentmgt.com/articles/view/beware-vague-learning-jargon/1
A) ROE
B) ROI
35
C) Both Copyright © 2012 Karen Carleton


2. Training
Evaluation
Standard
(old & new)

1. Learning
Objectives
&
Performance
Objectives
ROI & Impact
36

Transfer of Learning

Learn ? Apply

“That almost magical link between
classroom performance and something
which is supposed to happen in the real
world.” -J.M. Swinney

Plan Transfer -
avoid wasted
resources

Training
Scrap


Fostering Connections
Simulations,
role-plays,
guided
practice
-Realistic
practice
-High fidelity


“…training is unlikely to improve job
performance without a work
environment supportive of learning
transfer.”
-Reid Bates in
Improving Learning Transfer
in Organizations


Transfer Supports
 Discussion
 Case studies;
scenarios
 Learner goals

BEFORE During After

Transfer Supports
 Expert guests
 Examples, analogies
 Feedback & guidance;
reflection
 Share application tips

Before DURING After


Transfer Supports Others?
 Checklists, action plans, Q & A, discussion

Before During AFTER

Transfer Example
Truck driver trainees

 Read manual
 Simulator
 Drive with trainer
 Drive under
observation
Drive alone

Transfer Approaches
Training Focus Performance Focus
Training Performance needs
Some needs analysis Thorough analysis of
for training (often) performance needs
Focus: Transfer Focus: Results (*may
include training)


Clear Performance Standards

Necessary Support

Appropriate Consequences

Useful Feedback

Individual Capacity

*Needed Skills & Knowledge
*Adapted from Rummler & Brache, 1995

Top Transfer Tips
Identify measurable Ensure transfer
performance resources ($, time,
standards people, etc.)
Involve Managers & Assess trainee
Supervisors perceptions of
Communicate transfer factors
training-PI link to Invite stakeholder
Trainees feedback


Evaluating Impact
Evaluation is
essential
1. Post-training
2. On-the-job
3. Program level


Gauging Transfer


Transfer Supports used internally:

A. Post-training Survey
B. Follow-up calls/visits
C. Workplace Observations
D. Skills demos/Tests
E. Other
F. Combination G. Not sure (yet!)


Summary Quiz
1. Three parts to writing performance objectives.
Performance
1) _______________
Criteria
2) _______________
Conditions
3) _______________
Knowledge, Skills, & Attitudes
2. KSAs= _________________________________.
3. The Four Levels of Training Evaluation are:
Reaction, Learning, Behavior change, Business Results
______________________________________________.
4. Training/Learning and Development professionals should
Business Partners
be strategic ___________________.


Summary Quiz
5. The main difference between the old and new
Kirkpatrick model is:
Model is flipped, now begins with business needs
__________________________________________.
Why?
6. Sponsored training should have a Return On
xpectations nvestment
E__________ AND a Return on I_________.
7. Example of a Transfer support?
8. Quick way to gauge Training Impact?


Value-added Training
Performance Objectives
Plan for Transfer
Evaluate at Levels 1 & 2
at least
Have an ROE & ROI
mindset


“Transfer interventions will be
most successful when the
explicit goal is Performance
Improvement.”
-Holton & Baldwin in Improving Learning
Transfer in Organizations (2003)


Presenter
Karen Carleton
Performance Solutions Corp.
Karen@performcorp.ca
www.performcorp.ca Back of Your Business
Card:
(780) 705-7701 #1 – YES – (Send me
 Training program evaluation “Three Critical Keys to
 Performance needs analysis & Adult Learning”)
custom solutions #2 – YES (I’d like a FREE
 Seminars & workshops Training Evaluation
Strategy Chat for my
Copyright © 2012 Karen Carleton company)

References
Carliner, S. (2003). Training design basics. Alexandria, VA: ASTD.
Designed for Learning. Retrieved January 23, 2012 from,
http://tarunagoel.blogspot.com/2009/11/transfer-of-learning-
theories-and.html.
Gayeski, D. (2005). Chapter 7: Translating Data to Dollars. in
Managing Learning and Communication Systems as Business
Assets. Upper Saddle River NJ: Pearson Prentice Hall.
Kirkpatrick and Partners. Retrieved January 23, 2012 from,
http://www.kirkpatrickpartners.com
Kirkpatrick, J. D. & Kayser Kirkpatrick, W. (2010). Training on trial:
How workplace learning must reinvent itself to remain
relevant. New York, NY: AMACOM Books, A Division American
Management Association.

References
Bates, R.A. (2003). Chapter twelve -- Managers as Transfer Agents
in 243-270. In E. F. Holton III and T. T. Baldwin (Ed.s),
Improving Learning Transfer in Organizations (pp. 243-270).
San Francisco: Jossey-Bass.

ROI Institute Canada, Retrieved May 19, 2012 from,
http://www.roiinstitutecanada.com/

Warren, M.W. (1979). Training for Results: A systems approach to
the development of the human resources in industry (2nd
ed.). Menlo Park, CA: Addison-Wesley Publishing Company.


Got Results? Evaluating Training Impact

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Got Results? Evaluating Training Impact

Similar to Got Results? Evaluating Training Impact (20)

More from Performance Solutions Corp.

More from Performance Solutions Corp. (7)

Recently uploaded

Recently uploaded (20)

Got Results? Evaluating Training Impact