A talk given at the Universidad Autonoma de Coahuila, August 2013

1. Oral Exams: Rubrics
JoAnn Miller
Macmillan Publishers
miller@room20.org

2. A simple way to grade
• Situation:
– Oral examination of 30 students in 50 minutes.
– Examiners had 5 or 6 groups to grade in one or
two days.
– Goal: Simplicity, fairness, rapidity.

3. Oral Exam Organization
• Students (pairs) given
role cards as they enter
classroom
• Told to prepare—when
finish exam, can leave
• Can use any reference
or ask questions
• Come to the front of
the class, talk only to
teacher
• Perform
• Either stay together or
separate for second
role-play

4. 5 point scale
Accuracy (deliberate
performance-prepared)
1 ---Little or no language
produced.
2 --Poor vocabulary, serious
mistakes in grammar, poor
pronunciation.
3 --Adequate vocabulary, mistakes
in grammar, adequate
pronunciation.
4 --Good vocabulary, occasional
errors in grammar, good
pronunciation.
5 --Wide vocabulary, very few
errors in grammar, very good
pronunciation.
Fluency (effortless
interaction-not prepared)
1 ---Little or no communication.
2 ---Very hesitant and brief
utterances, sometimes difficult
to understand.
3 ---Communicates ideas, but
hesitantly and briefly
4 ---Effective communication, but
does not elaborate on
response.
5 ---Easy and efficient
communication. Elaborates on
responses.

5. I. Pair Roleplay. Used to grade accuracy. (5 points)
Instructions:
• Give each pair of SS a roleplay card. Give them at least 5 minutes to
prepare their talk.
• Call pairs of Ss to the front. They perform without written notes.
• Grade them based on the following scale: [5-point scale presented here.]
Note: Please take S's level into account. A Course 1 student cannot
produce as much language as a Course 3 student. To get 3
points, the student should be able to use structures and vocabulary
taught in the course he/she just finished. However, expect errors
since the student has not fully acquired the material. To get 5
points, the student may still make a few isolated errors, but will
speak much above a typical student at the same level.

6. II. Roleplay with the teacher. Use to grade fluency. (5
points)
Instructions:
• Choose one of the situations given below.
• If three people are needed in the situation, keep the
pair together. If not, separate Ss and grade individually.
• S does not see situations.
• Explain the situation to the S and perform your part of
the roleplay.
• Grade S on the following scale: [5-point scale presented
here.]
Note: Students should be graded considering their course
level.

7. Student A: Imagine you are at
a meeting and not in an exam.
All of your classmates are at
the meeting too. Your partner
doesn’t know anyone. Tell him
who the people are.
Student B: Imagine you are at
a meeting and not in an exam.
All of your classmates are at
the meeting too. You don’t
know anyone. Ask your partner
who the people are.
Student A: Tell your partner
about an accident you or some
member of your family had.
When he/she tells you, ask
some intelligent questions or
make relevant comments.
Student B: Tell your partner
about an accident you or some
member of your family had.
When he/she tells you, ask
some intelligent questions or
make relevant comments.
Student A: You are making a
survey about what people think
they will be able to do with
telecommunications in twenty
years. Ask your partner at least
three questions about the topic.
Student B: Your partner is making
a survey. Answer his/her questions.

8. II. Roleplay with the teacher. Use to grade fluency.
(5 points) (Do not show these questions to the Ss. )
1. Keep students together. Ask them what they plan to do when
they finish school. Then ask them to tell you the pros and
cons of that job.
2. Separate students for a moment. You are going to give one
student a message for the other student. For example, ask
Student A to tell Student B you are going to meet him/her
after class. Then have the student pass on the message.
Make the messages a little bit complicated. When you
finish, give Student B messages for Student A.
3. Separate students. Tell student to imagine his girlfriend / her
boyfriend is angry. Ask him / her what he / she will do. Then
ask a “what if” question: What if he doesn’t believe you?
What if he goes out with someone else?, etc.

9. II. Oral Exam (10 points)
Role Play 1: 5 4 3 2 1 0
Role Play 2: 5 4 3 2 1 0
Total Oral points: ___/10
Part I is the listening comprehension
exam. It is on the same page.

10. But what if you want more….?

11. Rubrics-Definition(Wikipedia)
• an attempt to communicate expectations of
quality around a task.
• the criteria are public; teachers and students
understand the criteria
• can also provide a basis for self-
evaluation, reflection, and peer review.
– accurate and fair assessment
– fostering understanding
– indicating a way to proceed
– ongoing assessment or formative assessment.

12. Criterion-Referenced
a student's aptitude on a task is determined by
matching the student's performance against a
set of criteria to determine the degree to which
the student's performance meets the criteria for
the task.

13. Holistic vs. Analytic Rubrics (Mertler)
Holistic
• requires the teacher to
score the overall
process or product as a
whole, without judging
the component parts
separately
Analytic
• the teacher scores
separate, individual
parts of the product or
performance first, then
sums the individual
scores to obtain a total
score

14. Holistic (Mertler)
When errors in some part of the process can be tolerated provided the overall quality
is high

15. Analytic (Mertler)
Performance tasks in which there may be one or two acceptable responses
and creativity is not an essential feature of the students' responses.

16. Why use rubrics (Andrade)
• they help define “quality”—Ss know what
they are supposed to do
• help students become more thoughtful judges
of the quality of their own and others’ work
• reduce the amount of time teachers spend
evaluating student work
• their “accordion” nature allows them to
accommodate heterogeneous classes
• easy to use and to explain

17. Levels of Performance (1, 2, 3)Criteria
Descripters

18. Descriptors
• spell out what is expected of students at each
level of performance for each criterion.
• tells students more precisely what
performance looks like at each level and how
their work may be distinguished from the
work of others for each criterion.
• help the teacher more precisely and
consistently distinguish between student
work.

19. Content Validity
Assessment should be based on a
content-analysis of the text being
used

20. Content Validity
• You must test only material students
have seen
• The only common denominator is the
textbook

21. Goals and Objectives
• Goal: A statement of what you expect the student
to be able to do on the exam.
– Students will be able to order a hamburger
successfully.
• Objective: A description of the most important
components that will lead to a successful
completion of the goal.
– Vocabulary: demonstrate correct use of functions of
asking for something, clarifying, thanking
– Pronunciation: demonstrate ability to speak clearly
enough to be understood.

22. Goals and
Objectives
(Canisius College)

23. Goals and Objectives, cont.(Moskal)
1.stated goals and objectives for the performance assessment
should be clearly aligned with the goals and objectives of
instruction.
2.Both goals and objectives should reflect knowledge and
information that is worthwhile for students to learn.
3.The relationship between a given goal and the objectives
that describe that goal should be apparent.
4.All of the important aspects of the given goal should be
reflected through the objectives.
5.Objectives should describe measurable student outcomes.
6.Goals and objectives should be used to guide the selection
of an appropriate assessment activity.

24. Design Goals and Objectives
• First: What are you going to grade in an oral
exam:
pronunciation, grammar, vocabulary, spelling,
organization, fluency, listening
comprehension, ???????
• Think… what are you evaluating somewhere
else? What is important for oral
communication and can’t be tested
elsewhere.

25. Example (openMind 2, Unit 1)
• Before Unit 1---”In openMind 2 you will….” = oral
objectives????
• Plus the Speaking section of the unit..odd
numbered units
Goal:
On the completion of Unit 4, students will be able to
describe events in the past
describe people’s reactions to past events
use follow up questions.

26. The exams
• Have a conversation about a computer disaster. Use follow-up questions.
• Have a conversation about last year. Use follow-up questions.
• Have a conversation about activities in a specific year. Use follow-up questions.
• Have a conversation about an accident. Use follow-up questions.
• Have a conversation about a vacation. Use follow-up questions.
• Have a conversation about birthday celebration. Use follow-up questions.
• Have a conversation about a difficult day Use follow-up questions.
• Have a conversation about a trip. Use follow-up questions.
• Have a conversation about a funny experience. Use follow-up questions.

27. Objectives
• First reflect on what aspects of the objectives is
important and can me measured:
– (1) ability to communicate the description of an
event in past clearly; (2) distinguish between past
tense and past progressive tense use; (3) use of
vocabulary related to describing reactions; (4) use
logical follow-up questions showing understanding
of story.
– (1) and (2) could be combined.
1-describe events in the past
2-describe people’s reactions to past events
3-use follow up questions.

28. • Now, write one or two goals for Unit 5. Begin
each one with…”On the completion of Unit
5, students will be able to…..”
• When you finish, for each goal, write 2-3
specific objectives. Begin with “Demonstrate
(the ability to…).” You can combine related
objectives as I did…..

29. Share your goals and objectives with
another group.
Feel free to make changes….

30. Analytic Rubrics
Now we will begin writing the rubrics using your
goals and objectives.
Each goal will be one or more criteria and the
objectives will be in the descriptors.
Additional criteria could be common to all
exams:
Pronunciation (as it affects communication);
Communication

31. (Mertler)
How many levels of performance will you have? 3, 4?

32. How can you organize the exams if you
use the openMind test builder?
The exams are built
around units- 9 exams
per unit.
If you’re going to give
ONE exam to
students, you could
choose the unit at
random….
Or….you could give
each student 4
exams, but that would
take forever….

33. Review: Steps (Mertler)
1.Re-examine the learning goals and objectives
to be addressed by the task.
2.Identify specific observable attributes and
behaviors that you want to see (as well as
those you don’t want to see) your students
demonstrate in their product, process, or
performance.
For example: the language they should be able to
produce (vocabulary), etc.

34. 3. Brainstorm characteristics that describe each
attribute. Identify ways to describe above
average, average, and below average performance for
each observable attribute identified in Step 2. (Do this
for each objective.)
4.Write thorough narrative descriptions for excellent
work and poor work for each individual attribute.
Describe the highest and lowest levels of performance
using the descriptors for each attribute separately.
5.Complete the rubric by describing other levels on the
continuum that ranges from excellent to poor work for
each attribute. Write descriptions for all intermediate
levels of performance for each attribute separately.

35. And later
7. Collect samples of student work that
exemplify each level. These will help you
score in the future by serving as benchmarks.
8.Revise the rubric, as necessary. Be prepared to
reflect on the effectiveness of the rubric and
revise it prior to its next implementation.

36. Thank you very much…
JoAnn Miller
miller@room20.org
Handouts available at:
www.efltasks.net/Presentations
Power Point available at:
www.slideshare.net/jabbusch

37. Grading Oral Exams
JoAnn Miller, Macmillan Publishers
Bibliography
Andrade, Heidi Goodrich, Understanding Rubrics, http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm
Carnegie Mellon, Rubrics, http://www.cmu.edu/teaching/designteach/teach/rubrics.html
Mertler, Craig A. “Designing Scoring Rubrics for Your Classroom.” Practical Assessment, Research & Evaluation.
http://pareonline.net/getvn.asp?v=7&n=25
Moskal, Barbara K. “Recommendations for Developing Classroom Performance Assessments and Scoring Rubrics.” ”. Practical Assessment, Research
& Evaluation. http://pareonline.net/getvn.asp?v=8&n=14
Moskal, Barbara K. “Scoring Rubrics: What, When and How?” Practical Assessment, Research & Evaluation.
http://pareonline.net/getvn.asp?v=7&n=3
Mueller, Jon. Authentic Assessment Toolbox, http://jfmueller.faculty.noctrl.edu/toolbox/rubrics.htm
Rezaei, Ali Reza, Michael Lovorn. “Reliability and validity of rubrics for assessment through writing. Assessing Writing. Vol. 15(1), 2010. Pp. 18*39.
http://www.sciencedirect.com/science/article/pii/S1075293510000048
Rubistar, http://rubistar.4teachers.org/
Rubrics: Definition, Tools, Examples, References, The TLT Group, http://www.tltgroup.org/resources/flashlight/rubrics.htm
Sample Rubrics, Association for the Assessment of Learning in Higher Education, http://course1.winona.edu/shatfield/air/rubrics.htm
Teacher Planet, Rubrics for teachers, http://www.rubrics4teachers.com/
TeacherVision, Creating Rubrics. http://www.teachervision.fen.com/teaching-methods-and-management/rubrics/4521.html
teAchnology, Rubrics and Rubric Makers, http://www.teach-nology.com/web_tools/rubrics/