The following is a presentation by Dr Salosh Govender titled: Creating reliable and valid assessments for a digitally-enabled future at the VUT

2. PRESENTER DR SALOSH GOVENDER
ADOVH (ACADEMIC DEVELOPMENT OPEN VIRTUAL HUB PROJECT) –UNISA
Curriculum development and learning specialist
EMAIL: govenp2@unisa.ac.za
Cell contact: 0845557707
TOPIC: Creating reliable and valid assessments for a digitally-enabled future at the VUT

3. Overview of presentation
PURPOSE
To empower academics with practical
strategies for incorporating a variety of
valid and reliable assessment methods
into a digitally enabled teaching and
learning environment
CONTENT
1. Affordances of technology enhanced assessment
2. Designing for assessment integrity
3. Approaches in creating valid and reliable online
assessments
• Constructive alignment
• Bloom’s taxonomy
• Authentic assessment
4. Applying the utility formula in selecting appropriate
assessment methods
5. Types of online assessment methods
6. Strategies for designing valid and reliable multiple
choice questions
3

4. Affordances of technology enhanced assessment.
• Greater variety and authenticity in the design of assessments
• Improved learner engagement, for example through interactive formative assessments with
adaptive feedback
• Choice in the timing and location of assessments
• Capture of wider skills and attributes not easily assessed by other means, for example through
simulations, e-portfolios and interactive games
• Efficient submission, marking, moderation and data storage processes
• Consistent, accurate results with opportunities to combine human and computer marking
• Immediate feedback
• Increased opportunities for learners to act on feedback, for example by reflection in e-portfolios
• Innovative approaches based around use of creative media and online peer and self-
assessment
• Accurate, timely and accessible evidence on the effectiveness of curriculum design and
delivery

5. Assessment integrity and online assessment
• Assessment integrity is not just about preventing plagiarism or academic misconduct but also about upholding best
pedagogical practices to enhance learning and teaching efficacy.
• Assessments should require students to explain their approach, logic, or thinking in their assessments.
• This can involve short-answer items, written work, annotated portfolios or recorded and annotated presentations to ask
students to talk through how they came to an answer.
• Through higher-order thinking in assessment, educators can improve student reflection on course content, which
reduces opportunities for potential misconduct and supports deeper understanding.
• Focus on student knowledge (UC Berkeley, 2020) and consider eliminating components that do not inform learning. As
stated above, this means designing assessments that measure higher-order thinking.
• Consider what may be the most appropriate assessment to use for your course content and the online context that
supports student learning and academic integrity

6. Assessment validity
• Valid: Measures what it is intended to measure.
The assessment should align with the intended
learning outcomes (Constructive alignment)
• The assessment level should match the level of the
learning outcome (Blooms taxonomy)
• Authentic assessment

7. Reliability of assessment
• Reliable: The results are consistent and dependable
• Do different graders evaluate the student work the same
way?
• Are students likely to perform similarly if they retook the
assessment in a few days? Does the assessment depend
on students happening to study minor details in the
course or broad course themes?

8. Constructive alignment
• Assess students at a level that is
consistent with the learning
outcomes provided AND the
learning activities used
• Assessments should measure
student development and
competency in the learning
outcomes you have identified, and
to the same level you have indicated
in those learning outcomes.
• The types of tasks you ask students
to perform in the assessments
should have been practiced and
modeled in the learning activities for
your course. This does not mean
simply repeating the same examples
or practice problems from class on
an assessment; it means there
should be no surprises in terms of
what is expected of students on
assessments and exams.

9. Bloom’s taxonomy
9
Image from Purdue University, Reflections on Teaching and Learning Webpage:
http://blogs.itap.purdue.edu/learning/2012/05/04/review‐of‐idc‐tools‐to‐assess‐blooms‐taxonomy‐of‐cognitive‐domain/

10. BLOOMING HOT MCQs
• High ordered thinking skills (HOTS) refers to
thinking that goes beyond the basic recall
of memorized information
• Requires cognitive processing such as
critical thinking, problem solving,
application, analysis, evaluation, and
creation.
• General perception that MCQs are
incapable of assessing cognitive processes
beyond recall or recognition of knowledge.
10

11. The aspect of authenticity is an important
one. Authentic assessment can be defined as:
“An assessment requiring students to use the […]
competencies, or combinations of knowledge, skills,
and attitudes that they need to apply in the criterion
situation in professional life.”
(Gulikers, Bastiaens, & Kirschner, 2004, p. 69)
Authentic assessment
11

12. Characteristics of authentic
assessment
12
Ask students to perform, create,
produce or do something
Tap higher-level thinking and
problem-solving skills
Use tasks that represent
meaningful instructional activities
Invoke real-world applications
(Herman, Aschbacher and Winters, 1992)

13. Transforming Assessment in HE to Enhance
Student Transferrable Skills
• Authentic and sustainable
assessment: focuses on
assessment tasks that have
applicability to the world
outside the classroom and that
foster autonomous learning
• Foster metacognitive skills &
focus on deep, sustainable,
authentic learning (Kearney &
Perkins, 2011); provide
proficiencies and skills required
for job markets
1
3
Image source: 2018 iCanProgress

14. Using the utility formula to choose valid assessment methods
• Utility of assessment influenced by five factors:
• Repetitive
sessions of the
assessments will
generate the
same results
Reliability
• Does the
assessment
measure what we
want to measure?
Validity
• To what extent
does it influence
the learning
process
• Good feedback
Educational
effects
• Accepted by
students,
assessors,
teachers
Acceptance of
stakeholders
• Cost of location,
technology, costs
for staff, etc.
• Data
Cost
efficiency
Factors influencing the utility formula
Used to assist with the selection of the appropriate online method of
assessment and assessment design

15. Evaluating the assessment method: five factors of the utility
formula in the selection of the assessment method
A + sign means that factor is included within the
method of assessment, +/- is partially included and
- means the factor is not included. Within this
example, the overall utility of these assessments
together is strong, as all factors are considered to
assess a course i.e. all factors have at least one +
sign associated with a method of assessment.
IS YOUR ASSESSMENT METHOD FIT FOR PURPOSE
?
List 2-3 assessment methods used in the
programme that you teach on.
• Analyse: What are their purposes? i.e. what do
they measure/provide feedback on. How
aligned are they with principles of good
assessment?
• Evaluate: Are they fit for purpose or ideally
would you like to improve them?

16. Take home exams/ open
book
E PORTFOLIOS
Quizzes
Patchwork assessments
Self/ peer assessments
Group work
Wikis
Examples
of
innovative
online
assessments
Forum discussions
Reflective journals
Video reflection blogs

17. Designing valid and reliable multiple choice questions
Why Advanced skills?
❑ Builds on the foundational knowledge
covered in the presentation on basic
skills in MCQs
❑ Includes authentic assessments aimed
at producing graduates who are
responsive to the labour market.
❑ Involves Integrated assessments:
Application of practical, theoretical
and foundational competencies
❑ Promotes problem solving, critical
thinking which is essential for every
profession.
❑ Application of knowledge minimises
the risk of cheating
About randomization
❑ A number of questions are randomly
drawn from the pool . Each student is
presented with a unique test which
minimizes the chances of cheating
during the exam. The auto submit
function in the LMS deters students
from copying their question set and
passing it along to their peers who may
not have started the exam
❑ MCQs can be used to assess
students’ knowledge and skills by
incorporating lower, middle and higher
order thinking questions, as set out in
Bloom’s Taxonomy.
❑ HOT MCQs coupled with an open book,
can challenge students and reduce the
risk of cheating.

18. Potential of MCQS to support student learning
• Powerful teaching tool to support student learning in an online environment.
• Reservations about the suitability of MCQs:
✓ Assess only recall and memorisation type of content
✓ suitable only for a specific disciplines/ subjects.
• Carefully crafted MCQs aligned to learning outcomes are powerful tool to enable online
formative assessment to support student learning
• Formative assessment takes place during instruction to support the learning process (Oosterhof
et al., 2008; Vonderwell et al., 2007).
• FA activities are intrinsic parts of instruction that allow learning to be controlled and the
instruction to be modified until the desired learning goals have been achieved (Gikandi et al.,
2011
• MCQs provide students with opportunities to cover the module content/ core concepts
• Cementing students understanding: Can be used to cement students understanding of the more
subtle points in the module content, through the use of carefully constructed feedback.
• Well designed, high quality MCQs deepens students understanding.
18

19. Blooming MCQS: What are higher
ordered MCQs • Higher ordered thinking
skills (HOTS) refers to
thinking that goes beyond
the basic recall of
memorized information
• Requires more cognitive
processing such as critical
thinking, problem solving,
application, analysis,
evaluation, and creation.
• Misconception that MCQs
can only test the lower
levels of Bloom’s taxonomy
19

20. Criteria for quality MCQ assessments
Constructive alignment Adherence to quality criteria of
test theory
Meets the quality criteria of test theory : Objectivity ,
reliability and validity
• Objectivity : Provides opportunities for all
students to solve the MCQs under the same
conditions and these are assessed under the same
conditions
• Reliability : Difficulty of test, content and format is
appropriately determined for the target group
.Under the same conditions for the same unit of
competency, all assessors should reach the same
decision as to whether the candidate is competent,
based upon the evidence collected.
• Validity- Assessment content is aligned with the
learning outcomes, the respective level of difficulty
and the learning environment.
20

21. Planning for quality randomized MCQS
21
•Align to module outcomes
•Align to NQF
•Covers all content
•Content mapping
Choosing the content
and level of difficulty
•Higher ordered
•Different types
•Question stem and plausible
distractors
Formulating question
•Quality measures
•Objectivity
•Reliability
•Validity
Evaluating question
•Complete the MCQs on the
Online assessment tool
•Create Question pools
•Create Parts for randomization
•Prepare detailed feedback for
formative use
Compiling the exams

22. PLANNING MCQS
DESIGNANDPLANNINGOFQUALITYASSESSMENTTASKS:CHECKLISTFORCOGNITIVELEVELS
ThefollowingSpecs(Table1)formodulechangeswereproposedbyEXCOandwouldinclude
lowerorder,middleorderandhigherordertasksinthefollowingproportions.Basedonthe
abovedecisions,Bloom’staxonomyhasreference.
Table1:BloomsTaxonomy:RecommendedlevelsforNQFsasproposedbyEXCO
Descriptionofcognition NQF5 NQF6 NQF7 NQF8
Level1-Remembering (Knowledge)
Lowerorder
level1&2
40 30 20 10
Level2-Understanding (
comprehension)
Level3-Applying(Application) Middleorder
Level3and4
40 40 40 40
Level4-Analysing (analysis)
Evaluating(evaluation) Higherorder
Levels5and6
20 30 40 50
Creating(syntheses)

23. Strategies for HOT MCQs
Multi-logical
thinking
Uses visuals:
Graphs,
cartoons,
pictures
Real world
scenarios
2 Tier
questions

24. Strategy 1: Two tier multiple choice question
TWO TIER MCQ
• The first tier pertains to a knowledge or comprehension statement and the second tier facilitates testing of higher-order
thinking.
• It is in effect asking a lower-order question and then asking students to provide a reason (which requires higher-order
thinking).
• Two-tier multiple-choice questions can also be seen as scaffolded questions
EXAMPLE
Suppose you are given two clay balls of equal size and shape. The two clay
balls also weigh the same. One ball is flattened into a pancake-shaped piece.
Select the correct option.
1st tier: knowledge or comprehension statement
a. The pancake-shaped piece weighs more than the ball
b. The two pieces still weigh the same
c. The ball weighs more than the pancake-shaped piece
. because ( 2nd tier facilitates thinking of higher order thinking)
d. the flattened piece covers a larger area.
e. the ball pushes down more on one spot.
g. when something is flattened it loses weight.
h. clay has not been added or taken away.
i. when something is flattened it gains weight.
1. a and d
2. b and h
3.c and i
4. a and g

25. Strategy 2: Use real world scenarios/ case studies
EXAMPLE
After ( higher order)
You arrive at the scene of an accident where
people are panicked and yelling. Three people
appear to be injured. Which person will you
take care of first?
• One person has scrapes and bruises,
• A second is lying on the ground moaning in
pain and
• A third is losing blood quickly from a skull
injury.
One of the best way to promote
and assess high order thinking is
to use scenario-based questions.
This changes a question from
requiring simple recall to, at the
very least, requiring the
application of knowledge
Before ( lower order)
What is the first concern of an emergency
worker?

26. Strategy 3: Analysis of visuals
USING VISUALS FOR MCQS
• Analyse and interpret visuals from diagrams or
graphs require the application of critical, deep
thinking to solve.
• Cartoons that depict a situation or certain
behaviour can be used.
EXAMPLE
This is a section from a counting count. As you
can see many of the numbers are missing. Fill in
the numbers to complete the section. Select the
correct answer for the block marked A
1. A = 42
2. A = 18
3. A = 52
4. A = 32

27. Example 1 of Visuals: Stages of data handling
Data interpretation is an important step in solving the problem “What was the most popular
food/ drink sold on the 4 February”. The data presented below, has a key showing that each
item represents a quantity of 20.
Which option represents a common misconception which results when a key is not included in
the data
1. More bananas were sold than hot drinks.
2. 60 cold drinks were sold because each can represent 20 cans
3. 20 more cold drinks were sold than hot drinks.
4. Most learners preferred hot meals instead of salads.

28. Strategy 4: Multi-logical thinking
MULTILOGICAL THINKING
• Multi-logical thinking questions are
questions with more than one premise,
from which a student must make the correct
conclusion.
• Requires a student to take several facts and
aspects in account before making a decision.
• In the example given, the student is required
to both diagnose the symptoms and decide
on what the best way would be to break the
news to a parent.
EXAMPLE
Tim’s second grade teacher is concerned because of the following
observations about Tim’s behaviour in class:
• Withdraws from peers on the playground and during group
work
• Often confuses syllables in words (ex: says mazagine instead of
magazine)
• Often confuses b and d, p and q, etc. when writing or
recognizing letters
• Loses his place when reading
The teacher has arranged a meeting with Tim’s mother to discuss
these concerns. Which of the following statements is best for the
teacher to say to Tim’s mother?
• A. Tim needs extra practice reading and writing problematic
letters and words at home at least 30 minutes per day.
B. Please discuss the importance of schoolwork to Tim so that
he will increase his efforts in classwork.
C. These are possible symptoms of dyslexia so I would like to
refer him to a specialist for diagnosis.
D. Please adjust Tim’s diet because he is most likely showing
symptoms of ADHD due to food allergies.

29. Item Shells: Lower/ middle order MCQs
29

30. Activity: Using the item shells, design
suitable MCQs related to your
module
30

31. Avoid unnecessary details
You want to use your wood-burning fire pit for warmth on
chilly evenings and to roast marshmallows. To take advantage
of the shade under the gazebo, could you safely place the fire
pit underneath? (Select the best answer.)
1.Yes, if the gazebo ceiling is high enough to allow for heat
dispersal
2.Yes, if you use screening material over top of the fire pit to
reduce sparks
3. No, because the smoke and fumes from the fire pit may
damage the gazebo ceiling
4. No, because only propane fire pits may be used in a gazebo
31

32. Answer choices are not the same length
You are using your outdoor word-burning ceramic
fire pit. Which of the following should you do to
safely operate the fire pit? (Select the best answer.)
1. Burn pine or other soft woods in the fire pit.
2. Make sure that there aren’t any combustible
materials, such as dead vegetation, adjacent to the
fire pit.
3. Douse the fire with water from the hose to put it
out
32

33. Checklist criteria
33

34. 34
Good teaching is more a giving of right questions than a
giving of right answers
Josef Albers