501

1. REVISING INSTRUCTIONAL
MATERIALS CHAPTER 11
CARLY YASSES
APT501

2. OBJECTIVES
• Describe various methods for summarizing data obtained from
formative evaluation studies.
• Summarize data obtained from formative evaluation studies.
• Given summarized formative evaluation data, identify weaknesses in
instructional materials and instructor-led instruction.
• Given formative evaluation data that indicates problems in a set of
instructional materials, identify problems in the materials and
suggest revisions.

3. BACKGROUND
• Examine almost any instructional design model and you find
major emphasis on the concept of formative evaluation—that
is, on collecting data to identify problems and to revise
instructional materials. During the revision process, designers
must keep a systems perspective on their work and remain
open to possibilities that revisions may be warranted in any
stage of the design process.

4. CONCEPTS
• There are many different ways in which the data collected in a
formative evaluation may be summarized to point to areas of
learner difficulties and possible revisions. The methods we
describe here are merely suggestions. As you begin to work
with your own data, you may find other techniques that help
you derive more insight from them. We first look at what you
can do with the data and information from a one-to-one
formative evaluation, and then consider the small-group and
field-trial phases.

5. DATA ANALYSIS FOR ONE-TO-ONE TRIALS
• Following the one-to-one formative evaluation, the designer
has very little data because information typically is available for
only three to five learners. Because these learners were selected
based on their diversity, the information they provide will, in all
likelihood, be very distinct, rather than blending into some type
of group average

6. THE DESIGNER HAS FIVE KINDS OF BASIC
INFORMATION AVAILABLE:
• (1) learner characteristics and entry skills
• (2) direct responses to the instruction
• (3) learning time
• (4) posttest performance
• (5) responses to an attitude questionnaire

7. DATA ANALYSIS FOR SMALL-GROUP AND FIELD
TRIALS
• The small-group formative evaluation provides the designer
with a somewhat different data summary situation. The data
from eight to twenty learners are of greater collective interest
than individual interest—that is, these data can show what
problems and reactions this representative group of learners
had.

8. INDIVIDUAL ITEM INFORMATION IS REQUIRED
FOR THREE REASONS:
• Item information can be useful in deciding if there are particular problems with the item
or if it is measuring the performance described in its corresponding objective effectively
(described later).
• Individual item information can be used to identify the nature of the difficulties learners
are having with the instruction. Not only is it important to know that, for example, half
the learners missed a particular item, but it is also as important to know that most of
those who missed it picked the same distractor in a multiple-choice item or made the
same type of reasoning error on a problem-solving item.
• Individual item data can be combined to indicate learner performance on an objective,
and eventually, on the entire test. Sometimes, the criterion level for an objective is
expressed in terms of getting a certain percentage of items correct on a set of items. The
individual item data can be combined not only to show the percentage of items correct for
an objective, but also the number and percentage of learners who achieved mastery.

9. GRAPHING LEARNERS’ PERFORMANCES
• Another way to display data is through various graphing
techniques. A graph may show the pretest and posttest
performance for each objective in the formative evaluation
study.

10. OTHER TYPES OF DATA
• There are other kinds of data to summarize and analyze in
addition to learners’ performance on objectives. It has been
found that a good way to summarize data from an attitude
questionnaire is to indicate on a blank copy of the
questionnaire the percentage of learners who chose each
alternative to the various questions.

12. SEQUENCE FOR EXAMINING DATA
• After generally examining the data, we suggest that you use the data
in the following sequence:
1. Instructional Analysis and Entry Skills
2. Objectives, Pretests, and Posttests
3. Examine the pretest scores
4. Learning Components of Instructional Strategy and Materials
5. Learning Time
6. Media, Materials, and Instructional Procedures

13. INSTRUCTIONAL ANALYSIS AND ENTRY
SKILLS
• First, after removing data for any defective items, you should
examine the remaining data with regard to the entry skills of
learners. Did the learners in the formative evaluation have the
entry skills you anticipated? If so, did they succeed with the
instructional materials? If they did succeed but did not have the
required skills, then you must question whether you have
identified critical entry skills.

14. OBJECTIVES, PRETESTS, AND POSTTESTS
• The second step is to review the pretest and posttest data as
displayed on the instructional analysis chart. If you sequenced
the materials appropriately and if you identified skills that are
hierarchically dependent on each other, then learners’ pretest
performances should decrease as you move upward through
the hierarchy—that is, there should be poorer learner
performance on the terminal objective than on the earlier skills.

15. EXAMINE THE PRETEST SCORES
• Examine the pretest scores to determine the extent to which
individual learners, and the group as a whole, had already
acquired the skills that you were teaching. If they already
possess most of the skills, then you receive relatively little
information about the effectiveness of the instruction or how it
might be improved. If they lack these skills, you have more
confidence in the analyses that follow.

16. LEARNING COMPONENTS OF INSTRUCTIONAL
STRATEGY AND MATERIALS
• The next step is to examine the instructional strategy
associated with the various objectives with which learners had
difficulty. Was the planned strategy actually used in the
instructional materials? Are there alternative strategies that
might be used? The final step is to examine the materials
themselves to evaluate the comments about problem areas
made by learners, instructors, and SMEs.

17. LEARNING TIME
• An important concern in any formative evaluation is the amount
of time required by students to complete the instructional
materials. It may be necessary for you to revise the materials to
make them fit within a particular time period. This is an
extremely difficult task, and it must be done with great care.

18. MEDIA, MATERIALS, AND INSTRUCTIONAL
PROCEDURES
• Data that relate to the implementation of the instructional
materials must also be examined. We suggested earlier that
you might gather misleading data because of the faulty
operations of media equipment.

19. REVISION PROCESS
• We suggest that as you begin the revision process, you
summarize your data as suggested in this chapter. The next
slide contains a template designers can use for organizing the
information gathered from all sources, and it uses the elements
of the instructional strategy as the framework. In the table, the
components of the instructional strategy