This presentation gives a fundamental understanding about Kirkpatrick's four levels of evaluation model. It also includes a brief of the fifth level of evaluation by Philip that forms the Kirkpatrick-Philip Model.
1. +
KIRKPATRICK’S
LEVELS OF
EVALUATION
Cindy Yew
Curriculum Project Manager
Knowledge Universe, CIRC-AIC
Asian International College, Singapore
31 January 2012
2. Dr Donald
+
Kirkpatrick’s
ideas on
measuring
training
effectiveness
were originally
published in
1959.
Four level model.
Simple.
Flexible.
Comprehensive.
Iae or : im. ro
mg suc g . m
e pc
3. The four levels of evaluation…
+
kirkpatrick’s model
Level 4:
Level 3:
Level 2:
Level 1 :
Image source: uncw.edu
6. Level 1: Reaction
+
measures how those who participate in the programme
react to the learning experience
“a measure of customer satisfaction” - Dr Don Kirkpatrick
Image source: wordpress.com
7. Level 1: Reaction
+
Level 1 evaluates reaction
e.g.,
Questionnaire Content
Methods
Media
Trainer style
Facilities
Course materials
• Did the participants like the training?
• Did they consider the training relevant?
• Did they find the trainer engaging?
Iae or : ma rgnloto
mg suc es inad yu m
e u a .c
8. Level 1: Reaction
+
How to collect reactions?
6.
Measure
reactions against
the standard &
follow-up where
necessary
10. Level 2: Learning
+
the extent to which participants change attitudes,
improve knowledge, and/or increase skill
as a result of attending the programme
“in order to
evaluate
learning, the
specific
objectives must
be determined”
- Dr Don Kirkpatrick
Image source: cartoonstock.com
11. Level 2: Learning
+
Level 2 measures learning
e.g.,
Interviews
Surveys
Tests (pre- and post-)
Observations
Combinations
• Did the participants learn what was intended to
be taught?
• Did the participants experience what was
intended for them to experience?
Iae or : ma rgnloto
mg suc es inad yu m
e u a .c
14. Level 3: Behaviour
+
the extent to which change in behaviour has occurred
because the participant attended the training programme.
One primary question for training
at this level is if there is a transfer
of knowledge to application at
daily activities?
Iae or : b oso g . o
mg suc o k olcm
e .g e
15. Level 3: Behaviour
Level 3 refers to behaviour
+
e.g.,
Survey the people who
observes the participant in Monetary
action back on the job Efficiency
Morale
Teamwork
etc…
• Did the participants put their learning into
effect when back on the job?
• Were the relevant skills and knowledge used?
• Was there a measurable change in the
activity and performance of the participants
when back in their roles?
Iae or : ma rgnloto
mg suc es inad yu m
e u a .c
18. defined as the final results that Level 4: Results
+
occurred because the participants
attended the programme.
Iae or : b g o.cm
mg suc l s to
e op
19. Level 4: Results
Level 4 looks at results
+
e.g.,
Key Performance Indicators Volume
Values
Percentages
Timescales
Return on Investment
Other quantifiable aspects of
organisational performance,
e.g. no. of attrition, wastage,
quality rating, achievement of
quality and standards, etc.
• Did the KPI improve?
• To what extent has it improved?
Iae or : ma rgnloto
mg suc es inad yu m
e u a .c
25. +
Credits
Sources:
Evaluating Training Programs: The Four Levels (3rd Edition) - http://bit.ly/uothqS
Kirkpatrick's Four-Level Model of Training Evaluation - http://bit.ly/vUuY6c
Kirkpatrickpartners -
www.kirkpatrickpartners.com/OurPhilosophy/tabid/66/Default.aspx
Kirkpatrick’s Four Level of Evaluation - http://youtu.be/oBpI0B5PtaA
ROI in Education: Increasing Innovation and Choice -
http://youtu.be/qRm9htSI3FM
Hinweis der Redaktion
Each level is different. Level 1 evaluates Reaction ; Level 2 measures Learning; Level 3 refers to Behavior. Level 4 looks at Results.
At this level, it is important to: collect reactions to instructor, programme, and the learning environment communicate to participants that their feedback is valued Rationale for this level: Reaction can make or break a training programme. Positive reactions are important for trainers who run both in-house and public programmes, e.g., what participants say is often reflected to the managers, who make decisions about the future of training programmes.
Level 1 evaluates Reaction, hence, it is important that people give a positive impression when relating their experience to others who might decide whether to experience the same
Kirkpatrick identified 3 ways of learning that can be accomplished as a result of attending training programmes: Change attitudes - programmes dealing with topics like diversity in the workforce aim primarily at changing attitudes. Improve knowledge - understand new concepts, principles, and/or techniques being taught Increase skills - technical programmes/OJT aim at developing/improving skills. Programmes on topics like leadership, motivation, and communication can aim at all three objectives Rationale for level 2 Learning has taken place when one or more of the following occurs: * Attitudes are changed * Knowledge is increased * Skills is improved
Level 2 measures Learning.
Rationale for level 3: Two reasons for evaluation of behaviour: 1) Benefit of the change 2) Identify if change of behaviour did not occur
Level 3 refers to Behavior, such as “ What behaviour/impact has the training achieved? ” Impacts can be monetary, efficiency, morale, teamwork, etc.
The final results can include increased production, improve quality, decreased costs, reduced frequency, and/or severity of accidents, increased sales, reduced turnover, and higher profits. These are the important reasons for having training programmes. Therefore, state these terms in the final objectives of the training program. These terms can be on long-term basis.
Level 4 looks at Results
No shortcut for the Kirkpatrick’s 4 levels of evaluation. Evaluation bypassing levels 1 & 2 in order to measure changes in behaviour (level 3) is a serious mistake. For e.g., suppose no change in behaviour is discovered. Assumed conclusion is that the programme was ineffective and should be discontinued. This conclusion may not be accurate. Reactions may have been accomplished, but the level 3 or 4 conditions may not have been present. In order for change to occur, four conditions are necessary: 1. The person must have a desire to change 2. The person must know what to do and how to do it 3. The person must work in the right climate 4. The person must be rewarded for changing The training programme can accomplish the first two requirements by creating a positive attitude toward the desired change and by teaching the necessary knowledge and skills. The third condition, right climate, refers to the participant ’s immediate supervisor. Without the right climate, participants may not attempt to use what they have learned. Ways to create a positive job climate: * involve bosses in the development of the programme, for e.g., asking bosses to help determine the needs of subordinates. Such involvement helps to ensure that a programme teaches practical concepts, principles, and techniques. *present the training programme to the bosses/supervisors before the participants are trained.
The fifth level “ROI” was a recent addition to the model by Phillip. It refers to evaluating the return on investment; but this was not in Kirkpatrick’s original model. Putting into perspective ROI in education, what does it mean? How do we know if the training investment pay off? Think about that.
This is an hour long video about ROI in education at the 2010 Social Enterprise Conference, where Professor Jonah Rockoff leads a discussion of how capital is being raised for education reform, and where is it being used to develop innovative, scalable, and sustainable solutions in education. At this point, as ROI is not the focus for this Kirkpatrick’s 4 levels of evaluation presentation, I do encourage you to watch it at your own time. This video is definitely informative and will expand your knowledge about the understanding for the returns on investment in education. See the relevance of what you are instilling in the training evaluation model and you will see the returns on investment soon to come.