LTEC 4440/ATTD 5440: Training Program Evaluation Module 4

1. Training Program
Evaluation
An Overview: Evaluate The Entire
Training Program Right

2. Why Evaluate?

3. Goals of Evaluation
• Basis for Planning and Directing
• To Assess Progress, Achievement
• To Evaluate Curriculum/Programs/
Staff/Facilities
• To Certify/Rate/Accredit
• To Monitor Expenditure of $$$
• To Improve Programs/Products

4. Goals of Evaluation
• To Determine if Training/ Education
makes a Difference (vs. cost)
• Justify Existence
• Accountability is a Trend Today
• Self-Satisfaction
• Pressure from Other Organizations
• Marketing
• MAKES GOOD ECONOMIC SENSE

5. Limitations of Evaluation
• Evaluation (generally) Does Not Solve
Problems,
> It Serves to Identify Them.

6. Evaluation Myths
#1#1 I canI can’t measure the results of my training’t measure the results of my training
#2#2 I donI don’t know what information to collect’t know what information to collect
#3#3 If I canIf I can’t calculate my return-on’t calculate my return-on
investment (ROI), then it is useless toinvestment (ROI), then it is useless to
evaluate the programevaluate the program
#4#4 Measurement is only effective in theMeasurement is only effective in the
production and financial areasproduction and financial areas

7. Evaluation Myths
#5#5 My Chief-Executive OfficerMy Chief-Executive Officer (CEO)(CEO)
does n0tdoes n0t
require evaluation, so why should I do it?require evaluation, so why should I do it?
#6#6 There are too many variables affectingThere are too many variables affecting
thethe
behavior change for me to evaluate thebehavior change for me to evaluate the
impact of trainingimpact of training
#7#7 Evaluation will lead to criticismEvaluation will lead to criticism
#8#8 I donI don’t need to justify my existence – I’t need to justify my existence – I
have a proven track record!have a proven track record!

8. Evaluation Myths
#9#9 The emphasis on evaluation should be theThe emphasis on evaluation should be the
same in all organizationssame in all organizations
#10#10 Measuring progress toward objectives isMeasuring progress toward objectives is
an adequate evaluation strategyan adequate evaluation strategy
#11#11 Evaluation would probably cost too muchEvaluation would probably cost too much
#12#12 The evaluation results will be used againstThe evaluation results will be used against
me or my program!me or my program!

9. Points to Remember
• Trainers must begin to take an “Evaluation-
Oriented Approach” to training.
• Not everything is quantifiable.
• If it’s worth doing, it’s worth evaluating.
• Consider evaluation EARLY.
• Cost should be proportional to importance.
• Time is the most important evaluation
component.
• Some benefits cannot be measured
immediately.

10. Terminology
• Cost/Benefit: Estimation of the projected cost
vs. projected benefits of the program (before
program).
• Return-On-Investment (ROI): Estimation of the
actual cost vs. actual benefits plus projected
benefits

11. 1.ConductaNeedsAnalysisandDevelop
TentativeObjectives
2.IdentifyPurposesofEvaluation
3.EstablishBaselineData
4.SelectEvaluationMethod/Design
5.DetermineEvaluationStrategy
6.FinalizeProgramObjectives
7.EstimateProgramCost/Benefits
8.PrepareandPresentProposal
9.DesignEvaluationInstruments
10.DetermineandDevelopProgramContent
11.DesignorSelectTrainingand
DevelopmentMethods
12.TestProgramandMakeRevisions
13.ImplementandConductProgram
14.CollectDataatProperStages
15.AnalyzeandInterpretData
16.MakeProgramAdjustments
17.CalculateReturnonInvestment
18.CommunicateProgramResults
Before AfterDuring
ACompleteResults
OrientedHRDModel

12. Data (Where do I find it?)
Hard Data
– Output (in basic work units)
– Quality (quality of the output)
– Cost (operating, administrative,
and capital expenditures)
– Time (savings)
*** If these are not available ***
try to convert soft data

13. Data (Where do I find it?)
Soft Data
– Work Habits
– New Skills
– Work Climate
– Development/Advancement
– Feelings/Attitudes
– Initiative

14. Evaluation Instrument
Design

15. Design Considerations
• Types of Evaluation Instruments
• Tests
• Focus Groups
• Observations
• Performance
Records
• Work Samples
• Questionnaires/Surveys
Telephone
Face-to-Face
• Attitude Surveys
• Interviews
Face to Face
Telephone

16. Preliminary Questions
• How will the data be used?
• How will the data be analyzed?
• Who will use the information?
• What facts are needed?
• Should the instrument be tested?
• Is there a standard instrument?
• What are the consequences of wrong answers
or biased information?

17. Characteristics of Effective Evaluation
Instruments
• Validity
– A valid instrument measures what the person
using the instrument wishes to measure. The
degree to which it performs this function
satisfactory is usually called the relative validity.
– YOU BUILD EVIDENCE OF…….

18. Characteristics of Effective Evaluation
Instruments
• Four basic approaches to determine validity:
– Content validity
– Construct validity
– Concurrent validity
– Predictive validity

19. 1. Determine the format that will address
your needs; i.e. numbers or percentages
versus open-ended questions.
2. Utilize a listing of tasks that are essential
to complete the job.
3. Write clear, straightforward directions and
questions.
Developing Data Collection
Instruments

20. 4. Ensure that questions address the most
important issues.
5. Provide space for comments.
6. Underline or capitalize key words.
7. Include descriptions for response scales.
8. Cluster related questions together.
Developing Data Collection
Instruments (cont.)

21. Developing Data Collection
Instruments (cont.)
9. Ensure the instrument is free of any racial,
sexual, or job denigration biases.
10. Run a pilot study on a sample group to
check for clarity of terms, instructions, and
response scales.
11. Develop a tabulation and scoring system.
12. Maintain confidentiality of respondents.

22. Questionnaires
• Most Common form of evaluation.
– They can be short reaction forms to detailed
follow-up instruments.
• If you can create a good (reliable & valid)
questionnaire, you can create almost any type
of instrument.

23. Types of Questions
Basically, there are five types of questions:
Open-ended Checklist
Two-way (yes/no)
Multiple choice
Ranking Scales

24. Questionnaire Design
• Determine the information needed
• Select the type(s) of questions
• ALWAYS remember reliability & validity
• Develop the questions
• Test the questions
• Develop the completed questionnaire and
prepare a data summary

25. One of the most common and most popular
data collection techniques.
Can be used to gather opinions, reactions,
responses in different environments.
Used primarily with large numbers vs. small
numbers of individuals.
Must understand advantages and limitations
before selecting questionnaires as instrument
to use.
Questionnaires

26. Advantages of Questionnaires
• Inexpensive to construct and administer
• Data can be tabulated without much effort.
• Confidentially can be maintained.
• Requires little training to administer.
• Many standardized attitude, aptitude, and
interest questionnaires have already been
developed.

27. Limitations of Questionnaires
• Difficult to design
• Different meanings may be read into
questions
• May appear to be an impersonal approach to
the collection of information
• Questionnaires can be a one-way process of
communication

28. Limitations of Questionnaires
• The true reasons behind problems can be
missed depending on how the questionnaire
items are constructed and how employees
perceive the questions.
• Low response rates can hinder the value of
the data
– This limits your ability to generalize the results to
the larger population.

29. Observation Procedures
1. Identify personnel to be observed.
2. Develop a list of items to observe.
3. Provide advance notice of your visit to
supervisors.
4. Observe employees without causing
disruptions.
5. Take appropriate notes when necessary,
but do not take on the impression of being
a “spy.”

30. Observation Procedures (cont.)
6. Share your observations with individual
employees, if appropriate.
7. Check validity of findings by comparing
results with other data collection
approaches.
8. Check validity of findings by comparing
results with results of other observers.

31. Advantages of Observations
• More Flexible
• Provides a picture of interpersonal
relationships
• Sparks employee ownership and interest in the
needs analysis process.

32. Face-to-face Interviews
1. One-on-One or in groups
2. Informal or formal
3. Permits expansion of thoughts, ideas,
comments, opinions, etc...
4. Use more than one interviewer
5. Follow guidelines when conducting
interviews

33. 1. Do a background study on the problem,
perhaps by reviewing work samples or
observation.
2. Develop interview questions in
advance.
3. Conduct the interview in a non-
threatening, private environment.
Guidelines for Conducting Interviews

34. 4. Determine the number of individuals to be
interviewed to obtain appropriate input.
Max. of 10 -12 participants is recommended
in group approach.
5. If the group approach is selected, be certain
that people from similar job classifications or
descriptions are grouped together.
Guidelines for Conducting Interviews

35. 6. Prepare your interviewees for your
session by explaining the purpose of the
interview.
7. Ask permission to take notes or to record
the interview.
8. Begin the interview with general
questions to set the right environment and
then move on to specifics.
Guidelines for Conducting Interviews

36. 9. Do not make judgments regarding
responses.
10. Be certain to keep the discussion on track,
emphasizing key points and asking for
clarifications when necessary. With the
group approach be certain to give
everyone an opportunity to contribute.
Guidelines for Conducting Interviews

37. 11. Be a good listener.
12. Summarize key points from time to time.
13. Keep responses confidential.
14. End the interview session on a positive note.
15. Analyze results.
16. Conduct follow-up sessions, if necessary.
17. Prepare a summary of the results of each
interview as soon as possible.
Guidelines for Conducting Interviews

38. Advantages of Face-to-Face Interviews
• Expands on earlier findings
• Reveals attitudes
• Builds rapport
• Observes non-verbal behaviors
• Personal approach vs. impersonal

39. Disadvantages of Face-to-Face
Interviews
• Racial/Sexual bias
• Time consuming
• Difficult to tabulate
• Poor interview skills
• Sympathy Syndrome
• Requires basic work-process knowledge

40. Conducted like face-to-face interviews
Advantages
Inexpensive
Good participation
Honest responses
Good follow-up
Limitations
Cannot see non-
verbal responses
Awkward start
Can interrupt

41. 1. Check the individual’s availability to
speak. If timing is not convenient,
make arrangements for a more
suitable time.
2. Establish rapport with the
respondent as soon as possible.
3. Speak clearly and at a pace that is easily
understood.

42. 4. Be certain you are talking with the right
individual.
5. Repeat questions if necessary.
6. Do not mislead respondents.
7. Do not coach answers.
8. Do not comment on responses.
9. Keep the discussion on the topic.
10. Keep accurate records of individuals
called.

43. 1. Copies of memos typed by secretary,
sample welds, curriculum, etc...
2. Permits in-depth study of products
3. Examine more than one sample
4. Validates other findings

44. Advantages
• Identifies bottom-line
problem areas
• Doesn’t interrupt
• Represents actual
work
• Inexpensive to collect
Limitations
• Requires skill to
interpret
• Takes time to
interpret
• Random sampling
difficult

45. Methods to Improve Validity
• Include an ample number of appropriate
items.
• Reduce response bias. (reverse answers)
• Be objective in administering the instrument.
• Recognize the weak link between attitude
and behavior (attitude may not equal
behavior).
• Others?

46. Methods to Improve Reliability
• Test/Retest
– Administer the same test to the same group at
two different time periods and calculate the
correlation of the scores.
• Alternative Form
– Construct two similar instruments and administer
to the employees at the same time. Analyze the
correlation between the two scores.

47. Methods of Improving Reliability
(continued)
• Split Half
– Involves splitting the instrument into two equal
parts and comparing the results. The two halves
are compared, are their correlations are
compared.
• Inter-item Correlation
– Calculate the correlations between the items on
the instrument

48. Methods of Improving Reliability
(continued)
• Ease of Administration
– This will increase the likelihood that it will be
administered consistently among different
individuals.
• Simple and Brief
– This will keep participants interested. Remember
that the participant’s time is valuable. Make the
items understandable
• Economical (Cost vs. Benefit)

49. Improving Return Rate
• Remember consent - If the subject has
informed consent, they are more likely to
complete and return the instrument.
• Remember confidentiality - If the subject has
assurances of confidentiality, they are more
likely to give honest statements. Honesty is
directly related to validity.
• Ease of use - If they can’t read or understand
it, they won’t complete the instrument

50. The Results
• What will the results tell you about your
subjects?
• How easy is the instrument to tabulate?
• Do you have the ability (or equipment) to
analyze the results?
• What will you recommend from the findings?
• Are the findings meaningful?
• Can the organization use the results to
improve?

51. “Customized” or “Canned”
• Creating an instrument that is customized for
your organization:
– The results are more meaningful to YOUR
organization.
– You know what the question are asking, and how
the results should be interpreted.
• Buying an instrument:
– Can be compared with other organizations
– Standardized scores, better generalizability.